Evaluating Extension Program Impacts Through Comparison of Knowledge and Behavior of Extension Clientele Versus Others

Introduction

There is increasing pressure on Cooperative Extension services to demonstrate program effectiveness and improve accountability in the face of resource limitations (Hachfeld, Bau, Holcomb, & Craig, 2013; Jayaratne, 2016). As a result, more Extension educators have begun to incorporate impact evaluation into program planning (Workman & Scheer, 2012).

Different evaluation methods have been suggested for individual programs (Jayaratne, 2016). Formats differ depending on the approach used to document changes in knowledge and behavior. A typical approach is to survey or interview program participants immediately after a program and several months following the program to measure changes in their knowledge and behavior (Hachfeld et al., 2013). Evaluation of Extension programs on a broader scale, such as on a statewide basis, however, generally has been overlooked (Wise, 2017). Meanwhile, Extension professionals continue to expand and innovate program delivery methods, including through use of the Internet and social media (Al-Kaisi, Elmore, Miller, & Kwaw-Mensah, 2015; Kelsey, & Stafne, 2012). As a result, Extension information and programming in a given subject area can be delivered in different formats and through the use of different media platforms. Given the plethora of delivery channels, evaluating Extension programming impact on a broader scale in particular subject areas should be very informative.

This article presents a new method for evaluating statewide Extension programming impacts that is based on testing whether Extension clientele differ from others in their knowledge and behavior. As Extension plays a pivotal role in promotion of soil conservation practices (Smart, Bauman, Boltz, & Hemenway, 2017; Drost, Long, Wilson, Miller, & Campbell, 1996), I applied this new methodology to evaluate impacts of South Dakota State University (SDSU) Extension programming on farmers' knowledge and behavior regarding soil conservation practices. The same approach may be used for assessing broad-scale influences of other Extension programming areas, such as 4-H, Supplemental Nutrition Assistance Program Education, and personal finance, to name but a few examples.

Data and Methodology

During the period of January to March 2018, a research associate and I sent a 16-page survey to 3,000 farmers from 34 counties in the East River area of South Dakota. The survey included multiple sections with questions pertaining to farming decisions, farm management practices, benefits and challenges associated with those practices, and perceptions of costs and profits. The list of farmers was obtained from South Dakota Farm Service Agency; 640 people responded by indicating that they were not farming, and thus they were ineligible for the survey. Of 2,360 eligible participants, 708 farmers responded to the survey.

Survey respondents were organized in two groups on the basis of their reporting of the role of SDSU Extension in their farm-related decision making (Table 1). Group 1 comprised those who reported that SDSU Extension was not important to their decision making and those who did not use SDSU Extension. Group 2 comprised those who thought SDSU Extension was at least somewhat important to their decision making. In total, 699 farmers rated the importance of SDSU Extension.

Table 1.
Respondent Perceptions of Level of Importance of South Dakota State University (SDSU) Extension

Role of SDSU Extension in decision making	Number of respondents	Percentage of respondents	Grouping
Not used	118	16.9%	Group 1
Not important	63	9.0%
Slightly important	131	18.7%	Group 2
Somewhat important	250	35.8%
Very important	137	19.6%
Total	699	100%

Effectiveness of SDSU Extension in education and promotion of soil conservation practices was assessed through analysis of responses to two survey items: (a) "How knowledgeable are you about the following practices? [four practices listed]" and (b) "How many years have you been using the following practices? Please check the relevant box if you haven't adopted certain practices yet or have discontinued usage of certain practices. [same four practices listed]." For the second question, producers who identified themselves as current users, regardless of the duration of use, were categorized as adopters. Those who had never used or had stopped using a practice were classified as nonadopters. It should be noted that less than 2% of respondents reported discontinuation of any soil conservation practice.

I performed Duncan's multiple range tests to examine whether significant differences existed between the groups regarding knowledge levels, adoption rates, and farm/farmer characteristics. In addition, to test whether Extension played a significant role in farmers' adoption decisions, I estimated four logistic regressions with adoption decisions for different soil conservation practices as dependent variables and the role of Extension and some key farm and farmer characteristics as independent variables for each regression.

Results and Discussion

Potential Influence of SDSU Extension on Knowledge

Overall, respondents differed in their knowledge of the four soil conservation practices (Figure 1). Proportions of respondents who rated themselves as moderately or very knowledgeable varied across the four practices. For example, approximately 20% of farmers in Group 1 and close to 30% of farmers in Group 2 believed that they were very knowledgeable about conservation tillage; however, only 10% in each group rated themselves as very knowledgeable about growing cover crops, and most farmers believed they had very little knowledge about growing cover crops. Compared to conservation tillage, cover cropping is a relatively new practice (Conservation Technology Information Center, 2017). Therefore the length of time a conservation practice has been promoted could be important in determining farmers' knowledge about it, regardless of information sources used.

For all practices, levels of knowledge about conservation practices differed between the two groups (Figure 1). Overall, Group 2 farmers were more likely to identify themselves as moderately knowledgeable or very knowledgeable about a practice. By contrast, Group 1 farmers were more likely to say that they were unfamiliar with or only a little knowledgeable about the practices. To describe such differences more clearly, I calculated average knowledge levels for Groups 1 and 2 for each practice (Table 2). Except for the practice of performing integrated crop and livestock management, average self-reported knowledge levels of Group 2 were statistically higher than those for Group 1 at the 5% significance level (Table 2).

Figure 1.
Comparison of Farmers’ Levels of Knowledge of Four Conservation Practices

Note: Group 1—Farmers who reported that South Dakota State University (SDSU) Extension was not important to their farm-related decision making or that they did not use SDSU Extension. Group 2—Farmers who reported that SDSU Extension was at least somewhat important to their farm-related decision making.

Table 2.
Farmer Groups' Average Levels of Knowledge of Four Soil Conservation Practices

	Average knowledge level rating
Conservation practice	Group 1	Group 2
Conservation tillage	2.68a	2.96b
Cover crops	2.32a	2.49b
Diversified crop rotation	2.40a	2.68b
Integrated crop and livestock management	2.34a	2.45a
Note. Group 1—Farmers who reported that South Dakota State University (SDSU) Extension was not important to their farm-related decision making or that they did not use SDSU Extension. Group 2—Farmers who reported that SDSU Extension was at least somewhat important to their farm-related decision making. Knowledge was assessed on a scale of 1 (unfamiliar with) to 4 (very knowledgeable). Superscripts denote the results of Duncan's multiple range test. Means followed by different superscripts within a row are statistically different at the 5% significance level, and means followed by the same superscript within a row are not statistically different at the 5% significance level.

Potential Influence of SDSU Extension on Behavior

Research has shown that farmers informed about conservation practices and their economic on-farm benefits are motivated to implement them (Coffey, Jennings, & Humenik, 1998). The results of the study reported here corroborate that finding. Adoption rates were generally higher for those practices associated with higher knowledge levels. For example, more farmers perceived themselves as having little knowledge about growing cover crops as compared to the other practices (see Figure 2), and adoption of cover cropping was lower than adoption of the other practices (Table 3).

Additionally, for all four practices, average rates of adoption for Group 2 were statistically higher than those for Group 1 at the 5% significance level (Table 3). In particular, there existed a dramatic difference in adoption rate of growing cover crops, with 52% of respondents in Group 2 having adopted this practice, in contrast to 32% of respondents in Group 1. It is apparent that farmers who viewed SDSU Extension as important in their decision making were more likely to change their behavior by adopting soil conservation practices as compared to those who regarded SDSU Extension as not important.

Table 3.
Farmer Groups' Average Rates of Adoption of Four Soil Conservation Practices

	Average adoption rate
Conservation practice	Group 1	Group 2
Conservation tillage	65%a	81%b
Cover crops	32%a	52%b
Diversified crop rotation	56%a	65%b
Integrated crop and livestock management	50%a	60%b
Note. Group 1—Farmers who reported that South Dakota State University (SDSU) Extension was not important to their farm-related decision making or that they did not use SDSU Extension. Group 2—Farmers who reported that SDSU Extension was at least somewhat important to their farm-related decision making. Superscripts denote the results of Duncan's multiple range test. Means followed by different superscripts within a row are statistically different at the 5% significance level, and means followed by the same superscript within a row are not statistically different at the 5% significance level.

Farm and Farmer Characteristics That Affect Use of SDSU Extension

To find out which types of farmers are more likely to use SDSU Extension, I also compared Group 1 farmers with Group 2 farmers on the basis of demographic characteristics (Table 4). The two groups were comparable with regard to years of experience as the primary decision maker on the farm and percentage of income from off-farm employment. However, Group 2 farmers had higher annual gross operation sales than Group 1 farmers. In addition, farmers in Group 2 had higher levels of education and were more likely to have completed an agricultural major or minor in college. In other words, farmers with higher gross sales values were more likely to view SDSU Extension as important in their decision making, as were the farmers with higher education levels and those who had completed an agricultural major or minor in college. Similar to these findings, Arbuckle (2013) found that Iowa farmers operating larger farms were more likely to use Extension than other Iowa farmers.

Table 4.
Comparison of Farm and Farmer Characteristics Between Farmer Groups

Farm/farmer characteristic	Group 1	Group 2
Years of decision making on the farm	26.91a	26.45a
Percentage of off-farm income	2.35a	2.17a
Gross sales value	2.75a	3.55b
Education level	2.92a	3.21b
Agricultural major or minor	0.41a	0.60b
Note. Group 1—Farmers who reported that South Dakota State University (SDSU) Extension was not important to their farm-related decision making or that they did not use SDSU Extension. Group 2—Farmers who reported that SDSU Extension was at least somewhat important to their farm-related decision making. Categories for percentage of off-farm income were 1= less than 20%, 2 = 20%–40%, 3 = 41%–60%, 4 = 61%–80%, 5 = 81% or more. Categories for gross sales value were 1 = less than $50,000, 2 = $50,000–$99,999, 3 = $100,000–$249,999, 4 = $250,000–$499,999, 5 = $500,000–$999,999, 6 = over $1,000,000. Categories for education level were 1 = Less than high school, 2 = High school diploma, 3 = Some college/technical school, 4 = College degree, 5 = Postgraduate degree. Superscripts denote the results of Duncan's multiple range test. Means followed by different superscripts within a row are different at the 5% significance level, and means followed by the same superscript within a row are not different at the 5% significance level.

Relative Effects of SDSU Extension on Adoption Decisions

While educational efforts of SDSU Extension were likely to increase adoption of conservation practices, I anticipated that certain farm and farmer characteristics could affect adoption decisions as well. Therefore, it was necessary to assess the role of SDSU Extension while controlling for other influencing factors. Applying the findings related to farm and farmer characteristics, I included in a regression model, in addition to the role of SDSU Extension, the three farm/farmer characteristic variables that were statistically different between the two groups at the 5% significance level: gross sales value, education level, and completion of an agricultural major or minor. Odds ratio estimates of the logistic regressions demonstrate the relative importance of SDSU Extension and other characteristics in influencing adoption (Table 5).

Table 5.
Odds Ratio Estimates for Logistic Regressions Regarding Four Soil Conservation Practices

Variable	Conservation tillage	Cover crops	Diversified crop rotation	Integrated crop and livestock management
Role of Extension	1.656**	2.092***	1.180	1.410*
Gross sales value	1.556***	1.209***	1.365***	1.143**
Education level	1.382*	1.074	0.891	1.046*
Agricultural major or minor	0.924	0.931	0.858	0.804
Percent concordant	69.6%	61.0%	64.0%	55.8%
Note. The odds ratio is defined as the relative odds of Y (i.e., adoption of soil conservation practices such as conservation tillage) when the value of X (e.g., role of Extension) increases by 1 unit. For example, an odds ratio estimate of 2.092 for role of Extension in the cover crops regression means that, for Extension clientele, the odds of adopting the practice of growing cover crops are 2.092 times as large as the odds of non-Extension clientele doing so. An odds ratio greater than 1 means a higher value of X is associated with higher odds of Y; an odds ratio less than 1 means a higher value of X is associated with lower odds of Y. *p < .01. **p < .05. ***p < .01.

When controlling for the other three influencing factors, I found that Extension still had a remarkably positive effect on adoption of conservation tillage, cover cropping, and integrated crop and livestock management. Among all practices, the impact of Extension was most important for adoption of cover cropping. A possible explanation for this observation is that the role of Extension is more pronounced for practices that have been under farmers' consideration for a relatively short period of time. For such practices, Extension plays a pivotal role in providing farmers with research-based knowledge and technology (Al-Kaisi et al., 2015; Smart et al., 2017). After a practice has been promoted over a longer period of time, farmers can also learn from their peers who have already adopted the practice, a scenario that can diminish the role of Extension as an information source.

There is a likely interaction and mutual reinforcement between farmers' perceptions of Extension's role and their adoption decisions. On one hand, use of Extension could promote adoption of conservation practices; on the other hand, farmers who have already adopted conservation practices may view the role of Extension as more important in their decision making. In both cases, it can be inferred that the Extension service plays a very important role in farmers' use of soil conservation practices, either before or after adoption.

Conclusion

Although most evaluation methods focus on knowledge and behavior change among people who use Extension services, this new evaluation method, comparing those who use Extension information in their decision making and those who do not, effectively documented the influence of Extension on a statewide basis. Even after controlling for farm and farmer characteristics between the two groups, I found that use of SDSU Extension was still associated with higher adoption rates of soil conservation practices such as conservation tillage, cover cropping, and integrated crop and livestock management. It can be inferred that SDSU Extension plays an important role in South Dakota farmers' decision-making processes regarding the adoption of soil conservation practices, especially for practices that are relatively new. This new evaluation methodology also can be used in other Extension programming areas to examine whether Extension clientele differ from others in terms of knowledge and behavior.

Acknowledgments

The work described in this article was financially supported by the U.S. Department of Agriculture Natural Resources Conservation Service (grant no. G17AC00337) and the South Dakota Corn Utilization Council.

References

Al-Kaisi, M. M., Elmore, R. W., Miller, G. A., & Kwaw-Mensah, D. (2015). Extension agriculture and natural resources in the US Midwest: A review and analysis of challenges and future opportunities. Natural Sciences Education, 44(1), 26–33.

Arbuckle, J. G., Jr. (2013, March). Iowa farmers' use of information sources and preferred ways of receiving information. Presented at ISU Agriculture and Natural Resources Extension Professional Development Meeting, Ames, IA. Retrieved from http://www.soc.iastate.edu/staff/arbuckle.html

Coffey, S., Jennings, G., & Humenik, F. (1998). Collection of information about farm management practices. Journal of Extension, 36(2), Article 2FEA4. Available at: http://www.joe.org/joe/1998april/a4.php

Conservation Technology Information Center. (2017). Annual report 2016–2017 cover crop survey. West Lafayette, IN: Conservation Technology Information Center, North Central Region Sustainable Agriculture Research and Education Program, and American Seed Trade Association. Retrieved from https://www.sare.org/Learning-Center/Topic-Rooms/Cover-Crops/Cover-Crop-Surveys

Drost, D., Long, G., Wilson, D., Miller, B., & Campbell, W. (1996). Barriers to adopting sustainable agricultural practices. Journal of Extension, 34(6), Article 6FEA1. Available at: https://joe.org/joe/1996december/a1.php

Hachfeld, G. A., Bau, D. B., Holcomb, C. R., & Craig J. W. (2013). Multiple year Extension program outcomes & impacts through evaluation. Journal of Extension, 51(1), Article 1FEA2. Available at: https://joe.org/joe/2013february/a2.php

Jayaratne, K. S. U. (2016). Tools for formative evaluation: Gathering the information necessary for program improvement. Journal of Extension, 47(3), Article 1TOT2. Available at: https://www.joe.org/joe/2016february/tt2.php

Kelsey, K., & Stafne, E. (2012). A model for evaluating eXtension communities of practice. Journal of Extension, 50(5), Article 5FEA1. Available at: https://www.joe.org/joe/2012october/a1.php

Smart, A. J., Bauman, P. J., Boltz, S., & Hemenway, J. (2017). Demonstration-based education generates behavior change related to conservation practices. Journal of Extension, 55(3), Article 3IAW8. Available at: https://www.joe.org/joe/2017june/iw8.php

Wise, D. K. (2017). Evaluating Extension impact on a nationwide level: Focus on programs or concepts? Journal of Extension, 55(1), Article 1COM1. Available at: https://www.joe.org/joe/2017february/comm1.php

Workman, J. D., & Scheer, S. D. (2012). Evidence of impact: Examination of evaluation studies published in the Journal of Extension. Journal of Extension, 50(2), Article 2FEA1. Available at: https://www.joe.org/joe/2012april/a1.php