April 2009
|
April 2009 // Volume 47 // Number 2 // Research In Brief // v47-2rb1
Organizational Restructuring and Its Effect on Agricultural Extension Educator Satisfaction and Effectiveness
Abstract
As the Cooperative Extension Service faces mounting pressure to meet the challenges of change, there is growing concern about the negative impact changes will have on employee satisfaction and performance. Many studies look at the impact of Extension restructuring on employee satisfaction and performance. However, these studies primarily offer a formative view of employee change. New research, focused on the effects of substantial restructuring at the University of Minnesota in 2004, offers a more summative view-indicating that Regional Extension Educators gained significantly in 13 of 16 related satisfaction and performance related variables while Local Extension Educators gained in five.
Introduction
The Cooperative Extension Service is challenged by powerful forces that have been mounting for many years. Markets, public priorities, and technology have changed the environment in which Extension functions, and many Extension systems are faced with making substantial changes to remain relevant and effective within decreasing budgets. This challenge is not new to Extension.
In the 1980s there was a need for Extension to better define its relevance, mission, priorities, and capabilities (Knutson, 1986). A few years later, the USDA Extension Service described the needed adjustment as a "rebirth" and implemented a number of initiatives to address the growing challenges (Johnsrud & Rauschkolb, 1989). A few years after this it was declared that "No state is exempt from restructuring" (Harriman & Daugherty, 1992). By year 2000, articles were exploring whether or not Extension was at the brink of existence and speculating what lied beyond that brink (King & Boehlje, 2000). Four years later, an article title posed the question "Is Extension an Idea Whose Time Has Come-and Gone?" (McDowell, 2004).
Many states have already taken substantial measures to address the mounting challenge for change, including Minnesota, North Carolina, South Carolina, Oregon, and Colorado (McDowell, 2004; Morse, 2006; Morse & O'Brien, 2006). Other states are considering how best to address the pressing inevitability of substantial change. These states have the advantage of learning from the experience of states that have made changes before them.
One important concern related to instituting changes in Extension has been employee satisfaction. Job satisfaction, generally defined as a person's attitude about work roles, influences employee motivation and output (Vroom, 1969). Change and instability in organizations tends to reduce satisfaction and lower employee output and output quality (Nester & Leary, 2000). Employee responses to organizational change can significantly diminish the value of instituted change.
Extension's concern about restructuring and its effect on employee satisfaction and performance has been the source of many studies and strategies (Bartholomew & Smith, 1990; Boltes, Lipkey, & Gregory, 1995; Huerta & Smith, 1994; Hutchins, 1992; Jayaratne & Gamon, 1998; Schmiesing, Safrit, & Gliem, 2003; Scott, Swortzel, & Taylor, 2005; Taylor-Powell & Richardson, 1990). Although these studies have focused on employee satisfaction, attitudes, and performance occurring before, during, or after a restructuring process, their focus has primarily been on formative information related to the period of restructuring. What has been lacking in the research is a summative comparison of employee satisfaction and performance levels occurring before and after the restructuring process.
In 2004, the University of Minnesota (UM) Extension instituted substantial restructuring to address the growing demands described above (Morse, 2006; Morse & O'Brien, 2006). A concern of UM Extension was employee satisfaction and work effectiveness resulting from restructuring. Before 2004, all agricultural Educators were located in county offices and had dual accountability to their associated counties and to UM Extension. Counties prioritized technical assistance responsibilities, while Extension prioritized educational opportunities.
Under the new structure, starting in 2004, Extension educators were redefined into two groups, Regional Extension Educators and Local Extension Educators. Regional Extension Educator (REE) positions were located in one of 18 Regional Extension Centers. These REE positions, supported with state and federal funds, were focused on educational programming around specialized discipline areas. Concurrently, Local Extension Educator (LEE) positions, supported solely with county funds, were located in county offices. Their work was focused on providing technical assistance and implementing local programming to counties that supported them.
UM Extension recently measured changes in agriculture educator satisfaction and work effectiveness resulting from the UM Extension restructuring. A summative assessment of change was generated in 16 related variables, spanning a 6-year period, 3 years before and after restructuring was implemented. The findings offer a new view on substantial Extension restructuring and its effect on employee satisfaction and effectiveness.
Methods
A retrospective pretest approach was used for this study (Pratt, McGuigan, & Katzev, 2000). An instrument was designed to measure the direction and amount of change that participants perceived as occurring in a variety of aspects of their own work as a result of Extension's organizational restructuring. All agriculture-related Extension staff who were Extension educators 3 years before the effective implementation date (January 1, 2004) of Extension's new regional/local structure and were still employed as educators 3 years after this implementation date were included in this study. A five-point Likert-type scale (much less, less, equally, more, much more) was used to measure degree of perceived change from 3 years before to 3 years after the restructuring. Using this five-point scale, participants were asked to finish a sentence that referred to changes in an aspect of their work due to restructuring. Sixteen items were included that represented five categories-skills, collaboration, autonomy and supervision, research and teaching, and satisfaction.
The instrument was face and content validated with several senior administrators. The survey was not piloted due to the relatively small population and the concern of affecting the results with pre-dissemination of the survey. Upon completion, participant responses indicated the reliability of the instrument scales were high, generating a Cronbach's alpha of 0.901.
All participants were e-mailed an invitation letter to complete this anonymous online survey within two weeks. This letter was sent by the associate dean who is responsible for the agricultural Extension program. After 8 days, a prompt was e-mailed to the population by the associate dean. The invitation letter and prompt letter included a hyperlink from which the participant could immediately access the Web-based survey. These implementation methods followed best practice guidelines for Web surveys (Umbach, 2004). An excellent response rate was measured for this survey study. Out of 32 Educators who met the participation criteria, 94% (N=30) completed the survey-100% of REEs (N=16) and 88% of LEEs (N=14).
Findings and Discussion
The study reported here measured Extension educator perceptions of changes in 16 variables related to the work of Extension educators. These 16 variables represented five broad categories: 1) work skills; 2) collaborations; 3) supervision and autonomy; 4) teaching and research; and 5) overall satisfaction with the new Extension structure.
The findings of the study indicate that agricultural REEs and LEEs perceived significant gains in many work attributes as a result of the change to a new Extension structure, with REEs reporting significant positive changes in 13 of 16 items and LEEs in five of 16 items (Table 1).
Means | Response Distribution Percentages | |||||||||||
Item | Sig. Dif Between Groups | Grp | N | Dif. From "Same" | Sig. Dif "Same" | Mean | SD | Much Less | Less | Equally | More | Much More |
Skills | ||||||||||||
Effective use of time 3A | .04♦ | LEE | 14 | 0.21 | .27 | 3.21 | 0.70 | 0% | 6% | 25% | 38% | 31% |
REE | 16 | 0.94 | .00** | 3.94 | 0.93 | 0% | 14% | 50% | 36% | 0% | ||
Leadership skills (effective use) 3B | .08 | LEE | 14 | 0.07 | .81 | 3.07 | 1.07 | 6% | 0% | 25% | 50% | 19% |
REE | 16 | 0.75 | .01** | 3.75 | 1.00 | 7% | 21% | 36% | 29% | 7% | ||
Instructional skills (effective use) 3C | .00♦ | LEE | 14 | 0.21 | .27 | 3.21 | 0.70 | 0% | 6% | 6% | 44% | 44% |
REE | 16 | 1.25 | .00** | 4.25 | 0.86 | 0% | 7% | 71% | 14% | 7% | ||
Programming skills (effective use) 3D | .01♦ | LEE | 14 | 0.21 | .43 | 3.21 | 0.97 | 0% | 6% | 13% | 31% | 50% |
REE | 16 | 1.25 | .00** | 4.25 | 0.93 | 0% | 29% | 29% | 36% | 7% | ||
Collaboration | ||||||||||||
Collaboration w/Field Faculty 5A | .82 | LEE | 14 | 0.14 | .55 | 3.14 | 0.86 | 0% | 31% | 25% | 31% | 13% |
REE | 16 | 0.25 | .36 | 3.25 | 1.06 | 0% | 21% | 50% | 21% | 7% | ||
Collaboration w/Campus Faculty 5B | .00♦ | LEE | 14 | -0.71 | .00** | 2.29 | 0.61 | 0% | 0% | 25% | 63% | 13% |
REE | 16 | 0.88 | .00** | 3.88 | 0.62 | 7% | 57% | 36% | 0% | 0% | ||
Collaboration w/Non-Ag Faculty 5C | 1.00 | LEE | 14 | -1.00 | .00** | 2.00 | 0.68 | 19% | 63% | 19% | 0% | 0% |
REE | 16 | -1.00 | .00** | 2.00 | 0.63 | 21% | 57% | 21% | 0% | 0% | ||
Collaboration w/Industry 5D | .31 | LEE | 14 | 0.79 | .00** | 3.79 | 0.70 | 6% | 0% | 19% | 38% | 38% |
REE | 16 | 1.00 | .00** | 4.00 | 1.10 | 0% | 0% | 36% | 50% | 14% | ||
Collaboration w/Public 5E | .24 | LEE | 14 | 0.64 | .01* | 3.64 | 0.84 | 19% | 6% | 38% | 25% | 13% |
REE | 16 | 0.06 | .85 | 3.06 | 1.29 | 0% | 7% | 36% | 43% | 14% | ||
Supervision | ||||||||||||
Super. Underst: Work Involves 7B | .01♦ | LEE | 14 | 0.79 | .00** | 3.79 | 0.70 | 0% | 13% | 31% | 38% | 19% |
REE | 16 | 1.56 | .00** | 4.56 | 0.89 | 7% | 7% | 36% | 36% | 14% | ||
Super. Aware: Work being done 7C | .02♦ | LEE | 14 | 0.93 | .00** | 3.93 | 0.62 | 0% | 6% | 6% | 13% | 75% |
REE | 16 | 1.56 | .00** | 4.56 | 0.81 | 0% | 0% | 36% | 50% | 14% | ||
Autonomy to make decisions 7A | .70 | LEE | 14 | 0.43 | .16 | 3.43 | 1.09 | 0% | 0% | 19% | 6% | 75% |
REE | 16 | 0.63 | .02* | 3.63 | 0.96 | 0% | 0% | 21% | 64% | 14% | ||
Focus | ||||||||||||
Teaching (time used for) 9A | .00♦ | LEE | 14 | -0.07 | .77 | 2.93 | 0.92 | 0% | 0% | 0% | 69% | 31% |
REE | 16 | 1.31 | .00** | 4.31 | 0.48 | 0% | 36% | 43% | 14% | 7% | ||
Applied Research (involvement) 9B | .02♦ | LEE | 14 | 0.64 | .03* | 3.64 | 1.01 | 0% | 0% | 13% | 25% | 63% |
REE | 16 | 1.50 | .00** | 4.50 | 0.73 | 0% | 14% | 29% | 36% | 21% | ||
Overall Perception | ||||||||||||
Satisf. with system over prior 11A | .00♦ | LEE | 14 | -0.07 | .77 | 2.93 | 0.92 | 0% | 6% | 13% | 25% | 56% |
REE | 16 | 1.31 | .00** | 4.31 | 0.95 | 0% | 43% | 21% | 36% | 0% | ||
Effectiveness of syst. over prior 11B | .01♦ | LEE | 14 | 0.14 | .58 | 3.14 | 0.95 | 0% | 0% | 25% | 31% | 44% |
REE | 16 | 1.19 | .00** | 4.19 | 0.83 | 0% | 29% | 36% | 29% | 7% | ||
Satisfied with Current Position | Strongly Agree | Agree | Neutral | Disagree | Strongly Dis-agree | |||||||
.10 | LEE | 14 | NA | NA | 2.21 | 0.89 | 21% | 43% | 29% | 7% | 0% | |
REE | 16 | NA | NA | 1.69 | 0.79 | 44% | 50% | 0% | 6% | 0% | ||
Notes: Mean difference refers to the mean difference from no change (point 3 on scale), positive or negative. Significant mean difference refers to difference from point of no change, using the one-sample T-Test. Confidence intervals =95% * = p<.05 (Significant difference from no change, using the one-sample T-Test.) ** = p<.01 (Significant difference from no change, using the one-sample T-Test.) ♦ = p<.05 (Significant difference between groups, using the Independent T-Test.) |
Effective Use of Work Skills
Regional Extension Educator and LEE perceptions regarding changes in their effective use of four work skills were measured. These included the effective use of the employees': 1) time; 2) leadership skills; 3) instruction skills; and 4) programming skills. Regional Extension Educators perceived significant gains in the effective use of each of these four skills with restructuring (Figure 1A). The largest of these gains related to the effective use of instructional skills and programming skills. Local Extension Educators reported no significant change in the effective use of any of these work skills.
The difference in the scores between REEs and LEEs was significant for three of the four work skills measured-only perceptions of change in leadership skills was not statistically different between the two groups (Figure 1A).
The results regarding perceived changes in the effective use of work skills are likely attributable to the overall change in position descriptions associated with restructuring. The position descriptions for LEEs remained very similar to the previous county Extension educator role; thus, the effective use of work skills would be expected to be perceived as similar. In contrast, the work skills associated with the new REE positions required much more specialized educational deliverables requiring greater time management, leadership, and programming skills. The increased demand for these skills is consistent with REE perceptions of an increase in effective use of these skills. Thus, 3 years after the change in everyone's role, the two groups of employees' perceptions of changes in effectiveness reflected the changes in their positions.
Figure 1.
Effectiveness in Collaboration
Regional Extension Educator and LEE perceptions regarding changes in their effectiveness in collaborating with five Extension related groups were measured. The five groups were: 1) agricultural Extension educators located throughout Minnesota (field faculty); 2) agricultural Extension faculty located on the St. Paul campus (campus faculty); 3) Extension colleagues in non-agricultural program areas (non-Ag faculty); 4) industry colleagues (industry); and 5) the public (public).
Regional Extension Educators and LEEs both reported similarly significant positive and negative changes regarding their effectiveness in collaborating with three of the five groups of collaborators (Figure 1B). The only collaborative group to which REEs and LEEs significantly differed related to their collaborations with campus faculty.
The REEs and LEEs diverged significantly in their perceptions of change regarding their effectiveness in collaborating with campus faculty (Figure 1B). Regional Extension Educators perceived their effectiveness in collaborating with campus faculty as having significantly increased (0.88), while LEEs perceived it as having significantly decreased (-0.71). Regional Extension Educator perceptions of increased effectiveness in collaborating with campus faculty was anticipated, as the REE's position descriptions emphasized greater applied research expectations-thus collaborations with campus faculty were desired. LEEs' perception of decreased effectiveness in their collaborations with campus faculty, however, was unexpected and of concern. It is speculated that LEEs perceived campus faculty as having shifted time away from them to focus more on REE collaborations.
Neither REEs nor LEEs reported significant changes in their effectiveness in collaborating with field faculty from restructuring (Figure 1B). This response was predicted as many REEs and LEEs had existing personal relationships with field faculty (as they have worked together in the same or nearby offices for many years) that continued to remain strong after restructuring.
Regional Extension Educator and LEE perceptions regarding their effectiveness in collaborating with non-agriculture faculty still within the Extension organization indicated that, of those variables measured, this was the item most negatively affected by restructuring (Figure 1B). Both REEs and LEEs perceived a mean decrease of 1.00. This perception was an unanticipated consequence of restructuring. Before 2004, all Extension educators, including all program areas, were supervised by district directors within a geographic region. This structure had encouraged good collaborations among Educators throughout a region, regardless of program area. After 2004, both LEEs and REEs were supervised within their separate program areas, each focused on the specific content and expectations associated with their particular specializations. As a result, the perceived effectiveness of collaborations between program areas decreased.
Both REEs and LEEs perceived significant gains in their effectiveness in collaborating with industry (Figure 1B). Regional Extension Educators reported a mean increase of 1.00, and LEEs reported a mean increase of 0.79. Note that REE responses included a high standard deviation (Table 1).
Local Extension Educators perceived significant gains (0.64) in their effectiveness in collaborations with the public (Figure 1B). Regional Extension Educators, however, perceived no significant change regarding their effectiveness in this collaborative relationship. Although the two groups reported different levels of change in this respect, their responses were not statistically different from each other due to a high standard deviation among REEs (Table 1). Local Extension Educator perceptions of an increase in their effectiveness in collaborating with the public can be attributed to the new funding model after 2004 that shifted total accountability of LEEs to county funding committees who represent the public, thus tightening their collaborative relationship.
Supervision and Autonomy
Regional Extension Educator and LEE perceptions regarding their supervision and their related decision-making autonomy was measured. Supervision represented the area of largest perceived gains of this study. Regional Extension Educators and LEEs were asked to compare their current and past (prior to 2004) supervisors regarding their supervisors' awareness and understanding of their work. Both groups were also asked to report on the amount of change in their level of autonomy to make decisions.
Both groups perceived significant increases in supervisor awareness of the work they do and understanding of what their work involves (Figure 2A). Although both groups perceived large gains in these respects, the REEs and LEEs significantly differed in the amount of increase they reported. Regional Extension Educators had a mean increase of 1.56 for both supervisor awareness of the work they do and understanding of what their work involves, whereas LEEs reported a mean increase of 0.93 for supervisor awareness and 0.79 for supervisor understanding, respectively.
From an organizational perspective, the significant gains in these supervisor elements were extremely positive after a major restructuring. Prior to 2004, supervision was administered by a district director, and after 2004 it was administered by a program leader within the same content discipline as the supervisee. Thus, the awareness and understanding of the REE and LEE work would be expected to be greater. Note, this does not necessarily indicate supervisory quality or effectiveness.
Regarding REE and LEE perceptions of change in their decision-making autonomy, only REEs perceived a significant gain (0.63), while LEEs perceived no significant change (0.43) (Figure 2A). There was, however, no significant difference between the mean responses of the two groups. Note that LEE responses included a high standard deviation (Table 1).
Figure 2.
Teaching and Research Focus
Regional Extension Educator and LEE perceptions regarding changes in their involvement in applied research and demonstration projects and their time used for teaching curriculum were measured. Both REEs and LEEs perceived significant gains in their involvement in applied research and demonstration projects, with REEs reporting the largest gains (1.50) and LEEs reporting an increase of 0.64.
When asked about changes in time used for teaching curriculum, REE and LEE reports were significantly different (Figure 2B). Regional Extension Educators reported significant increases in time used to teach curriculum (1.31), while LEEs reported no significant change in this respect (-0.07).
The revised position descriptions in 2004 for REEs stressed teaching and applied research efforts, thus the perceived increase in REE research efforts was predicted. The significant increase in applied research efforts perceived by LEEs was not expected. Regional Extension Educator and LEE perceptions of increased efforts toward teaching curriculum are consistent with changes in their position descriptions.
Effectiveness and Satisfaction
Regional Extension Educator and LEE perceptions regarding their general effectiveness in the new Extension structure compared with the previous structure were measured. Both groups were also asked about their general satisfaction with the new structure compared with the previous structure. Regional Extension Educators and LEEs significantly differed in their perceptions of change in their general effectiveness and satisfaction with the new structure. While REEs reported significant increases in their overall effectiveness (1.19) and satisfaction (1.31), LEEs reported no significant change in their overall effectiveness (0.14) or satisfaction (-0.07) with the new structure. Note that LEE responses regarding satisfaction included a high standard deviation (Table 1).
Regional Extension Educators and LEEs were also asked a related question that did not include a comparison between old and new structures, but instead focused only on their satisfaction with their current position using a five-point Likert-type scale (1=strongly agree/5=strongly disagree) (Figure 2C). Both REE and LEE groups reported they were satisfied with their positions, with no significant difference between the two groups (Table 1). However, REEs reported greater satisfaction with their positions than LEEs, with 94% of them reporting they were satisfied or strongly satisfied with their current position. Sixty-four percent of LEEs reported they were satisfied or strongly satisfied (Figure 3).
These results are again consistent with changes in restructured position descriptions. The revised position description of REEs became much more similar to campus Extension specialist position descriptions, and it is speculated that many REEs viewed their new positions as promotions. Thus, perceptions of effectiveness and satisfaction-both compared to one's previous position or in general-were rated very positively by REEs. In contrast, the LEE positions were much more similar to their existing positions, and it is hypothesized that many LEEs viewed this change as more of a reassignment than a promotion. As a result, perceptions of effectiveness and satisfaction did not significantly change.
Figure 3.
Satisfaction with Current Position ["I am satisfied with my position as an REE or LEE"]
Conclusion
Extension programs across the United States are being challenged to change to better accommodate shifts in markets, public priorities, technology, and funding levels. Many states have already made substantial changes to their structures and operations. An important concern at UM Extension during its major restructuring was the negative effect that change may have on employee satisfaction, work effectiveness, and resulting outputs. During the restructuring in 2004, Extension educator positions at UM Extension were rewritten. Many educator positions were transformed into Regional Extension Educator (REE) positions that focused on specialized content education and applied research, while the relatively unchanged Local Extension Educator (LEE) positions, which had broader agricultural expectations, emphasized broad education and technical assistance.
The study reported here indicates that substantial organizational restructuring of the UM Extension system was accomplished while boosting overall staff perceptions of work effectiveness and satisfaction. In fact, counter to common assumption, staff in the study who were most directly affected by restructuring perceived the greatest gains in effectiveness and satisfaction. Gains in work effectiveness were also perceived by staff who incurred little change in their position description; however, this group was more volatile, encompassing a larger range of perceptions about effectiveness and satisfaction.
The UM Extension experience indicates that the most vulnerable aspect of staff perceptions of effectiveness and satisfaction due to restructuring related to internal colleague collaboration. Structural change often involves changes in working relationships. The shift toward specialization severely reduced perceived effectiveness in collaboration with non-agriculture faculty. Additionally, changes in the structure, including the installation of the new REE category of staff, apparently displaced LEEs perceived collaborative relationship with campus faculty. Although staff mostly perceived gains or no significant change resulting from UM Extension restructuring, internal collaboration appeared to be the most vulnerable aspect of restructuring regarding staff perceptions of the work effectiveness and satisfaction measures studied.
As state Extension organizations consider ways to evolve toward the future, the UM Extension restructuring experience can offer important insights concerning staff response to Extension structural changes. Although Extension organizations differ in their situations and, perhaps, their choices, the UM Extension experience indicates that the work-related effects of substantial structural change can be perceived by staff as largely positive. This is good news for a system that is under pressure to make substantial organizational change.
References
Bartholomew, H., & Smith, K. (1990). Stresses of multicounty agent positions. Journal of Extension [On-line], 28(4) Article 4FEA2. Available at: http://www.joe.org/joe/1990winter/a2.php
Bolts, B., Lippke, L., & Gregory, E. (1995). Employee Satisfaction in Extension: A Texas Study. Journal of Extension [On-line], 33(5). Available at: http://www.joe.org/joe/1995october/rb1.html
Harriman, L., & Daugherty, R. (1992). Staffing Extension for the 21st Century. Journal of Extension [On-line], 30(4) Article 5RIB1. Available at: http://www.joe.org/joe/1995october/rb1.php
Huerta, J., & Smith, K. (1994). Attitudes of county Extension agents toward specialization in Ohio. Journal of Agricultural Education, 35(3), 6-10.
Hutchins, G. (1992). Evaluating county clustering. Journal of Extension [On-line], 30(1) Article1FEA5. Available at: http://www.joe.org/joe/1992spring/a5.php
Johnsrud, M., & Rauschkolb, R. (1989). Extension in transition: Review and renewal. Journal of Extension [On-line], 27(1) Article 1TP1. Available at: http://www.joe.org/joe/1989spring/tp1.php
Jayaratne, K., & Gamon, J. (1998). Effects of restructuring on the job performance of Extension educators: Implications for in-service training. Journal of Agricultural Education, 39(4).
King, D., & Boehlje, M. (2000). Extension: On the brink of extinction or distinction? Journal of Extension [On-line], 38(5) Article 5COM1. Available at: http://www.joe.org/joe/2000october/comm1.html
Knutson, R. (1986). Restructuring agricultural economics Extension to meet changing needs. American Journal of Agriculture Economics, 68(5), 1297-1306.
McDowell, G. (2004). Is Extension an idea whose time has come-and gone? Journal of Extension [On-line], 42(6) Article 6COM1. Available at: http://www.joe.org/joe/2004december/comm1.php
Morse, G. (2006). Minnesota Extension's regional and county delivery system: Myths and reality. Journal of Extension [On-line], 44(4) Article 4COM1. Available at: http://www.joe.org/joe/2006august/comm1.php
Morse, G., & O'Brien, P. (2006). Minnesota Extension's mixed regional/county model: Greater impacts follow changes in structure. Retrieved September 9, 2007, from: http://agecon.lib.umn.edu/cgi-bin/pdf_view.pl?paperid=21212&ftype=.pdf
Nestor, P., & Leary, P. (2000) The relationship between tenure and non-tenure track status of Extension faculty and job satisfaction. Journal of Extension [On-line], 38(4) Article 4RIB1. Available at: http://www.joe.org/joe/2000august/rb1.php
Pratt, C., McGuigan, W., & Katzev, A. (2000). Mesuring program outcomes: Using retropsective pretest methodology. American Journal of Evaluation, 21(3).
Schmiesing, R., Safrit, R., & Gliem, J. (2003, April). Factors affecting O.S.U. Extension agents' perceptions of organizational justice and job satisfaction: Critical insights into emerging trends and existing policies in Extension human resource management. Paper presented at the 2003 Association for International Agricultural and Extension Education (AIAEE) Annual Conference, Raleigh, North Carolina, USA.
Scott, M., Swortzel, K., & Taylor, W. (2005). Extension agents' perceptions of fundamental job characteristics and their level of job satisfaction. Journal of Southern Agricultural Education Research, 55(1).
Taylor-Powell, E., & Richardson, B. (1990). Issues programming changes Extension. Journal of Extension [On-line], 28(2) Article 2FEA4. Available at: http://www.joe.org/joe/1990summer/a4.php
Umbach, P. (2004) Web surveys: Best practices. New Directions for Institutional Research, 121(Spring), 23-38.
Vroom, V. H. (1967). Work and motivation. 3rd. ed., New York NY: John Wiley, & Sons, Inc.