Four Approaches to Building Extension Program Evaluation Capacity

Extension professionals are increasingly expected to more fully evaluate their programs (Nichols, Blake, Chazdon, & Radhakrishna, 2015; Rennekamp & Engle, 2008). In response, evaluation capacity building (ECB) is a necessary component of Extension educator professional development (Taylor-Powell & Boyd, 2008). One size rarely fits all; ECB is more likely to succeed when it is well aligned with the educator’s evaluation needs, derived from the type of educational effort he or she is using.

Four approaches to Extension work have been documented—service, facilitation, content transmission, and transformative education (Franz, 2013; Franz & Townson, 2008). These approaches require different evaluation measures and therefore different forms of ECB (Figure 1).

Figure 1.
Extension Educational Approaches and Their Implications for Evaluation Capacity Building

Intentional Program ECB

Adapting the four quadrants proposed by Franz and Townson (2008) to include guidance on the types of ECB required by educators can help one develop intentional ECB for each type of educational approach. For example, if an educator engages in content transmission (IIa), an ECB approach limited to measuring customer satisfaction (Ib) would be inappropriate and ineffective. This differs from many past ECB approaches focused solely on general evaluation concepts such as engaging stakeholders, focusing the evaluation, collecting and analyzing data, and reporting evaluation results. Tailored ECB matched with educational approaches also moves away from implementing one-size-fits-all evaluation whereby the Extension professional seeks a survey used by another educator that he or she can simply adopt with few or no changes without determining first whether the survey fits the situation at hand.

Service-Focused ECB

Extension systems as publicly funded entities have the responsibility or mandate to determine whether their educational services benefit tax payers (Franz, 2011). This context often requires evaluating the quality of customer service provided by Extension professionals. In some Extension systems, standardized surveys have been implemented across services and geography to aggregate customer service feedback statewide to report to legislators and other stakeholders. ECB related to gaining useful customer service feedback should address

• the rationale for collecting service feedback from customers,
• intended uses of the data (i.e., local service quality, legislative mandate, Extension professional performance, etc.),
• alignment of feedback collection methods and analysis with the evaluation rationale and uses,
• identification of customer feedback catalyzers and inhibitors to improve the amount and quality/authenticity of feedback, and
• ways to integrate the collecting and using of customer service feedback into processes for providing services.

Facilitation-Focused ECB

Extension professionals have increasingly become known for their ability to facilitate group processes (French & Morse, 2015). This approach to Extension education requires evaluation that measures the success of facilitated processes used and related outputs to help the group reach its goals. ECB related to determining the effectiveness of facilitated education should include

• ways to determine whether facilitation contributed toward the group's reaching its goals,
• the effectiveness of group process and engagement techniques,
• determination of whether the makeup or perspectives of the group are appropriate for the group to reach its goals (i.e., inclusion or exclusion of particular views or experiences),
• the unique roles or value of Extension in facilitating group processes, and
• the value of outputs created by facilitation (i.e., participation, inclusion, products).

Content Transmission–Focused ECB

This type of ECB is most common in Extension focused on a specific discipline, topic, communication method, or program. With this type of evaluation practice, evaluators have relied mostly on conducting preprogram and postprogram surveys to determine knowledge gain from an educational opportunity. Sometimes the intent to change behavior by the learner is measured. ECB related to determining successful content transmission focuses on

• reasons for collecting content dissemination feedback (i.e., to improve information quality, to improve dissemination methods, to change opinions or behavior, to fulfill funding requirements, to change conditions, etc.) and aligning evaluation accordingly;
• the degree to which information is accessible (methods, language, etc.) and used;
• what happens when learners use the information provided;
• timing of information dissemination for enhanced use of information;
• effectiveness and value of dissemination methods; and
• the value of Extension's unique content (i.e., research based, best practices, etc.).

Transformative Education–Focused ECB

Many Extension professionals are increasingly interested in articulating the impact of their educational efforts on economic, environment, and social condition changes at the individual, family, group, business, or community level. ECB related to determining impact on condition change requires that Extension professionals adopt evaluative thinking (Buckley, Archibald, Hargraves, & Trochim, 2015) as part of program design, implementation, and evaluation as this type of change requires attention to both Extension educational processes and content. ECB related to measuring condition change through transformative education addresses

• strong needs assessment that determines the current situation, the desired condition, and ways Extension education can bridge the gap;
• the building, maintaining, and evaluating of strong program evaluation partnerships (i.e., condition change requires a multipronged approach involving many educational efforts, often by many entities);
• integration of evaluative thinking into all aspects of the Extension program development model;
• gathering of impact data at multiple points in time to show when and for how long change takes place; and
• creation and sharing of condition change stories.

Summary

The types of evaluation conducted by Extension professionals need to match the educational approaches they use. Achieving this goal requires that ECB for Extension professionals focus more specifically on fit with educational approaches and less on basic evaluation steps or procedures. More intentional ECB should result in more effective and efficient program evaluation for all evaluation stakeholders. One-size ECB does not fit all Extension professionals or their approaches to improving program quality or determining program impact.

References

Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375–388. doi:10.1177/1098214015581706

Franz, N. (2011). Advancing the public value movement: Sustaining Extension during tough times. Journal of Extension, 49(2), Article 2COM2. Available at: https://www.joe.org/joe/2011april/comm2.php

Franz, N. (2013). Improving Extension programs: Putting public value stories and statements to work. Journal of Extension, 51(3), Article 3TOT1. Available at: http://www.joe.org/joe/2013june/tt1.php

Franz, N., & Townson, L. (2008). The nature of complex organizations: The case of Cooperative Extension. In M. T. Braverman, M. Engle, M. E. Arnold, & R. A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension (pp. 5–14). New Directions for Evaluation, 120. San Francisco, CA: Jossey-Bass. doi:10.1002/ev.272

French, C., & Morse, G. (2015). Extension stakeholder engagement: An exploration of two cases exemplifying 21st century adaptations. Journal of Human Sciences and Extension, 3(2), 108–131.

Nichols, A., Blake, S., Chazdon, S., & Radhakrishna, R. (2015). From farm results demonstrations to multistate impact designs: Cooperative Extension navigates its way through evaluation pathways. Journal of Human Sciences and Extension, 3(2), 83–107.

Rennekamp, R. A., & Engle, M. (2008). Case study for change: Evaluation in Cooperative Extension. In M. T. Braverman, M. Engle, M. E. Arnold, & R. A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension (pp. 15–26). New Directions for Evaluation, 120. San Francisco, CA: Jossey-Bass. doi:10.1002/ev.273

Taylor-Powell, E., & Boyd, H. (2008). Evaluation capacity building in complex organizations. In M. T. Braverman, M. Engle, M. E. Arnold, & R. A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension (pp. 55–69). New Directions in Evaluation, 120. San Francisco, CA: Jossey-Bass. doi: 10.1002/ev.276