April 2005 // Volume 43 // Number 2 // Tools of the Trade // 2TOT3
 |  |
 |
A Simple Method to Evaluate Series-Type Extension Programs
Abstract
This article describes how to evaluate the impact
of a series-type Extension program. Evaluating program impact is essential
for Extension accountability. The evaluation method described in this article
is simple and effective in documenting the impact of one Extension program
taught as a series. This approach can be used to evaluate other series-type
Extension programs by modifying the behavior section of the instrument presented
in this article to match the program content and objectives. This evaluation
tool not only helps Extension agents document impact but also helps them
to focus on the program objectives during the program delivery process.
Introduction
Evaluation of program impact is critical for securing the limited resources available to the Extension Service in an era of financial accountability. Extension stakeholders are keen on results of educational programs. Therefore, it is important to develop, deliver, and evaluate high impact Extension programs.
The levels of impact vary with the type of Extension program. According to Williams, Dickey, and Hergert, Extension programs range from short-term to in-depth programs (As cited in Rockwell, Jha, & Krumbach, 2003). Some Extension programs are very short presentations, some are daylong workshops, and others are presented as a series of lessons.
The series-type Extension programs are more effective than one-time programs in achieving significant impact because there is repeated interaction with the same client over a period of time. However, series-type Extension programs demand more time than short-term Extension programs. Therefore, it is important to evaluate the impact of series-type programs to justify the time investment and to secure stakeholder support.
Extension agents need a simple method to evaluate program impact due to their busy schedules. This article presents a simple evaluation approach used to document the impact of a series-type nutrition and exercise Extension program presented in Georgia.
How Does This Evaluation Method Work?
This evaluation approach is based on Prochaska, Norcros, and DiClemente 's (2004) change theory and Rodger's (1983) adoption theory. It focuses on tracking the clients' behavior changes toward the planned direction. Change theory describes "not ready to change," "getting serious," "have a plan," "take action," "keep at it," and "change" stages during the changing process (Prochaska, Norcros, & DiClemente, 2004). According to Rodgers (1983), people go through awareness, interest, evaluation, testing, and adoption stages before changing their current practices.
Extension programs facilitate clients' changing and adoption process. The Likert-type scale of this instrument was designed to capture the client's process of change during the Walk-a-Weigh nutrition and exercise Extension program. The following example illustrates how this method was used to document the impact of the Walk-a-Weigh program in Georgia.
Evaluation of a Series-Type Nutrition & Exercise Extension Program
-
Program Objective: Participants will reduce excess body weight by adopting a healthy life style.
-
Evaluation Steps
-
Identifying the major behavior changes needed to achieve the objective: Changes in dietary and exercise habits are the major behavior changes this program promotes.
-
Designing the instrument to record the extent to which the clients practice each of the identified behaviors. The planned behaviors are listed on the instrument as illustrated in Table 1 with a Likert scale to record the change process.
-
Recording and comparing each of the participants' responses at the beginning and end of the program to assess the number of individuals that improve their behavior. The example in Table 1 illustrates 10 behaviors critical for weight reduction. The movement of an individual's response at the beginning of the program from a column on the left side of the instrument to a column on the right side of the instrument at the end of the program indicates his or her behavior improved. An individual's improvement in behavior can be quantified by subtracting the pre-test reading from the post-test reading of each of the recorded behaviors.
-
The results can be summarized as a percentage of participants who improve their practices and a comparison of pre- and post-test means as illustrated in the Table 2.
| Planned Behavior |
1. Not Important to me |
2. I'm considering this |
3. I'm doing this occasionally |
4. I'm doing this regularly |
5. This is now a part of my life |
|---|---|---|---|---|---|
| Making a conscious effort to limit fat to 30% of total calories. |
|||||
| Eating at least 3 vegetables each day. |
|||||
| Eating at least 2 fruits each day. |
|||||
| Eating at least 2 servings of low-fat or non-fat dairy products each day. |
|||||
| Eating chicken or turkey without skin. |
|||||
| Eating low-fat snacks and desserts (Pretzels, fruits, vegetables, or reduced-fat products). |
|||||
| Using low-fat condiments (low-fat or nonfat mayonnaise, low-fat margarine, mustard, catsup). |
|||||
| Modifying recipes to lower fat by using less fat or substituting low fat ingredients. |
|||||
| Reading nutrition labels to help make food choices. |
|||||
| Exercising three times a week for at least 30 minutes at a time. |
| Desired Dietary Behavior |
Percentage of Participants Who Improved Their Behavior |
Mean at the Pre-Test |
Mean at the Post-Test |
Significance p |
|---|---|---|---|---|
| Making a conscious effort to limit fat to 30% of total calories. |
70% |
2.9 |
3.9 |
.000* |
| Eating at least 3 vegetables each day. |
68% |
3.2 |
4.0 |
.000* |
| Eating at least 2 fruits each day. |
60% |
3.1 |
3.9 |
.000* |
| Eating at least 2 servings of low-fat or non-fat dairy products each day. |
55% |
3.1 |
3.9 |
.000* |
| Eating chicken or turkey without skin. |
61% |
3.3 |
4.1 |
.000* |
| Eating low-fat snacks and desserts (Pretzels, fruits, vegetables, or reduced-fat products). |
64% |
3.1 |
3.8 |
.000* |
| Using low-fat condiments (low-fat or nonfat mayonnaise, low-fat margarine, mustard, catsup). |
65% |
2.9 |
3.8 |
.000* |
| Modifying recipes to lower fat by using less fat or substituting low fat ingredients. |
69% |
2.7 |
3.9 |
.000* |
| Reading nutrition labels to help make food choices. |
64% |
3.3 |
4.1 |
.000* |
| Exercising three times a week for 30 minutes at a time. |
49% |
3.2 |
3.8 |
.000* |
| * Mean difference is statistically significant at p=0.01 level (2-tailed). | ||||
Discussion and Implications
The task of an Extension program evaluation is to help Extension agents document program impacts as well as to improve educational programs. Program evaluations should be accurate and useful for making decisions about the program (Patton, 1997).
This evaluation instrument was able to document the impact of the Walk-a-Weigh program and to prove its effectiveness by comparing pre- and post-test mean values. Table 2 shows that the participants significantly improved their dietary and exercise behaviors. The means of the desired behaviors shifted from the lower numbers in the left hand columns at pre-test toward the higher numbers in the right hand columns at post-test, indicating that the program made a significant impact on participants. Compared to pre-test means close to 3, most of the post-test means of the planned behaviors were close to 4, indicating that the participants' response had changed to "I'm doing this regularly" from "I'm doing occasionally."
The comparison of each of the participants' pre- and post-test means was used to calculate the percentage of individuals who improved their behaviors. For example, over 60% of the participants improved most of the dietary behaviors advocated by the program. However, only 49% of the participants improved their exercise habits. Agents can use this information to structure future Walk-a-Weigh programs to include more information and strategies to help participants to improve exercise habits.
Listing the specific program objectives on the evaluation instrument helps the Extension agent and the participant to focus on the objectives during the teaching and learning process. This uses the program time and educational resources more efficiently while contributing to the cost effectiveness of the Extension program.
The major implication of this type of evaluation approach is its usefulness in evaluating Extension programs taught as a series. It can be used to evaluate other Extension programs taught as a series by modifying the behaviors listed on the instrument to match the program content and objectives. For example, if the instrument were modified for a financial literacy program, "keeping track of personal spending" could be a potential behavior that could be measured. This evaluation tool is very versatile and can provide many opportunities to show impact and program effectiveness in Extension programming.
References
Patton, M. Q. (1997). Utilization focused evaluation: The new century text. (3rd ed.) Thousand Oaks, California: SAGE Publications.
Prochaska, J., Norcross, J., & DiClemente, C. (2004). Change how you change. Available at: http://www.proactive-coach.com/change/index.htm
Rockwell, S. K., Jha, L., & Krumbach, E. (2003). Success outcome markers in extension (SOME): Evaluating the effects of transformational learning programs. Journal of Extension [On-line], 41(5). Available at: http://www.joe.org/joe/2003october/a4.shtml
Rogers, E. M. (1983). Diffusion of innovations. New York: Free Press.