Evaluating Curriculum Effectiveness by Asking the Users

The traditional method for testing the effectiveness of a curriculum involves assessment of knowledge or practices before and after material use. If significant change occurs, it is assumed the curriculum is "effective." However, acceptability of the materials to teachers (leaders) and youth (members) affects use. And if materials are not used, the potential for improving knowledge and practices is not realized. Specifically, 4-H materials should be easy for leaders and fun for members to use.

To evaluate the effectiveness of a food safety curriculum for use with 4-H youth (ages 9-11) and their family members, acceptability was assessed, in addition to change in food handling knowledge and practices. The educational program evaluated (Operation RISK) includes four lessons, a computer game, an audiotape and videotape, and two lessons for use at home with family members. The content of the materials teaches youth the what, whys, and hows of safe food handling. During the pilot testing of this curriculum, opinion surveys about acceptability were completed by nine leaders, 21 parents, and nine 4-H groups (77 members).

The survey for leaders included questions about the ease of use, amount learned by members, and level of members' enjoyment. Regarding ease of use, one lesson received much lower ratings than the other three. This lesson was primarily information given from the leader to members (lecture style). In the leaders' opinions, it also resulted in the lowest "amount learned" and was enjoyed the least of the four lessons. Youth input was collected using a group interview format. They were asked how much they enjoyed each of the materials and how much they thought they learned from them. Their opinions were consistent with those of the leaders. The same lesson received a low rating (regarding amount learned and enjoyed) by students and leaders. Based on these findings, the lesson was modified to include less relating of facts and more "hands on" youth involvement.

Information from parents was collected by enclosing a one-page survey in each of the take-home lessons. Parents were asked about use of the activities, understandability of instructions, amount learned, and whether the activity was fun, interesting, and successful in accomplishing its objective. Input from parents led to changes in the lessons, and in suggestions for their use. For instance, we found that materials need to remain at home several days to maximize their use; time was a barrier, not lack of interest. According to parents, all lessons included "easy to understand" directions and were judged to meet their objectives. However, one lesson was rated much lower than the other two regarding "new information," and how interesting it was. It was considered to be boring to 35% of the parent users. Based on these findings, this take-home lesson was omitted from the final version of the curriculum.

Other interesting results of these surveys included that: food safety is considered by parents to be an important topic for their children to learn about and that most parents (82%) did not have previous knowledge about the recommended temperature for refrigerators. Reported temperatures of their refrigerators (as taken by parents and members) varied from 30 to 57 degrees Fahrenheit. Many (24%) parents had "never thought about" how to keep foods cold in a packed lunch, and only 6% reported using freezer packs when packing a lunch. Some open-ended questions were included on the survey to obtain general information about what parents learned from the take-home activities. Parents were impressed and surprised that their children were so knowledgeable about food safety. Apparently, an unanticipated benefit of the take-home lessons was that it provided members with an opportunity to "show off" and perhaps to update their parents' knowledge about safe food handling. Numerous comments indicated that use of the materials had increased parents' awareness, or provided a reminder, about the importance of proper food handling practices. Parents seemed to learn the most about proper handwashing (takes 20 seconds, proper method) and how to keep cold foods cold when packed in a lunch for school (including freezer packs, pre-freezing sandwiches, or frozen juice boxes).

By asking the users (leaders, parents, members) of educational materials how easy and fun they are to use, what they learned from them, and how they can be improved, a wealth of important information is available that can be used to adapt/improve materials for use in 4-H programs. Although knowledge surveys help to identify what and how much youth learn, assessment of acceptability of materials provides specific information to help plan, develop, and implement educational programs that are used, enjoyed, and effective with the target populations.

Inquiries about this curriculum and surveys can be directed to Ms. Patricia Hammerschmidt, 4-H Youth Programs, 6H Berkey Hall, Michigan State University, East Lansing, MI 48824-1111, or e-mail to .

This project received support from the USDA (Special Project No. 91-ESFQ-1-4009).