A Framework for Introducing Program Evaluation to Extension Faculty and Staff

There is an increasing requirement by university administrations and funding agencies for Cooperative Extension Services to demonstrate that their programs are making an impact on the populations served. Many in Extension have little background in program evaluation, thus making it a daunting task.

At Washington State University we developed the Life Skills Evaluation System (http://ext.wsu.edu/lifeskills/) to assist Extension 4-H Youth Development and Family Living staff with this process. We realized that in order for staff to effectively use the evaluation system, which measures life skill outcomes, they would need a background in program evaluation. Therefore, as part of the training on the system, we conduct presentations introducing evaluation. The purpose of the presentation is to demystify program evaluation and provide a foundation by which staff can develop and conduct evaluations of their programs. Jacob's (1988) Five-Tiered approach to evaluation and the quality standards for program evaluation (Joint Committee on Standards for Educational Evaluation, 1994) were used in developing a framework to present the information.

We open each training with a learning activity based on Patton's (1997) premise that evaluation becomes meaningful when there is a shared definition and commitment to the process. One activity asks participants to describe evaluation through a musical instrument. This helps them express their feelings about and see the varied benefits of evaluation. Another activity asks participants to identify their favorite store and list why it is their favorite. This assists in illustrating that we are continuously evaluating in our everyday lives. Next, we discuss the framework, using examples from the participants' programs to explain the concepts.

Jacob's Five-Tiered Approach to Evaluation

Jacobs' (1988) five-tiered approach to evaluation is frequently used in prevention program evaluation design. The model is easy to understand and emphasizes the importance of using evaluation as a part of program planning and development.

• Preimplementation occurs during the initial stages of program planning to assess community needs.
• Accountability is the level at which staff account for the programming that took place such as the number of sessions offered and the number of participants who attended.
• Program clarification assesses the current status of the program, the program's strengths, and where improvement is needed.
• Progress towards objectives begins to assess whether or not short-term outcomes are occurring as a result of program participation.
• Program impact assesses whether systems-wide impact has occurred as a result of the program.

The tiers in Jacobs' (1988) model build upon one another as the program becomes more established. Jacobs emphasizes that a program should be evaluated at an appropriate level. By using Jacobs' model, we were able to assist staff in understanding that evaluation is effective and useful at all stages of development.

Program Evaluation Standards

The second component of the training is to present criteria for conducting quality evaluations. For this process, we draw upon the program evaluation standards developed by the Joint Committee on Standards for Educational Evaluation (1994). These standards are incorporated under the following four domains and consist of 30 criteria for planning and implementing quality evaluations.

• Utility emphasizes the need for useful evaluations that involve and respect stakeholders.
• Feasibility discusses the need to conduct evaluations that are practical and cost effective.
• Propriety is concerned with the ethical issues of conducting research using human subjects.
• Accuracy covers the importance of gathering and reporting reliable, valid data.

We present and discuss the four major domains and the implications for conducting evaluations by using fun and interactive methods, such as skits and role plays. In this manner, workshop participants are able to gain a greater understanding of the evaluation process and how the process impacts results.

Application of the Framework

After presentation of the tiers of program evaluation and criteria for quality evaluations, we then demonstrate the Life Skills Evaluation System. This step appears to help participants take the framework presented and put it into action during the workshop, reinforcing what was learned. Others using this framework might introduce an evaluation instrument at this point.

We have used this framework for numerous workshops across the state of Washington. Feedback from faculty and staff indicates that using the framework has assisted them in thinking about program evaluation by providing them with:

• Appreciation for evaluation research,
• Understanding of why a specific process needs to be followed,
• Awareness of the ethical issues involved in gathering information from program participants,
• Knowledge of the basics of program evaluation, and
• Understanding of incorporating evaluation as part of program planning.

We have learned firsthand the importance of providing a framework for conducting evaluation prior to implementing a specific evaluation. In one workshop, where we had limited time, we chose to give an abbreviated version of the framework described above and moved right into the logistics of using the Life Skills Evaluation System. It was clear during and after the workshop that without a foundation of the basics of evaluation, participants did not fully understand how or why the system was useful.

As a greater emphasis is placed on outcomes for Cooperative Extension programs, there is an increased need to provide staff training on the basics of evaluation. A framework using Jacobs' (1988) Five-Tiered model of program evaluation and the Joint Committee on Standards for Educational Evaluation (1994) can provide a useful foundation for introducing evaluation and demonstrating how it can be used to document program processes and outcomes.

References

Jacobs, F. H. (1988). The five-tiered approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs, New York: Aldine DeGruyter.

Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards (2^nd ed.). Thousand Oaks, CA: Sage Publications.

Patton, M. Q. (1997). Utilization-focused evaluation. (3^rd ed.). Thousand Oaks, CA: Sage Publications.