Impact Evaluation of Food Safety Self-Study Extension Programs: Do Changes in Knowledge Relate to Changes in Behavior of Program Participants?

Introduction

Food safety education is an important part of Extension programming because it targets the education of consumers and foodservice personnel as a strategy for preventing foodborne illnesses. Family and Consumer Sciences Extension agents present food safety educational programs to various client groups such as foodservice personnel, childcare providers, caregivers, homemakers, older adults, and youth throughout the United States. Impact evaluation of these programs "is becoming more important as accountability becomes more crucial and financial support becomes less available" (Novakofski, Boeckner, Canton, Clark, Keim, Britten, & McClelland,1997, ¶ 1).

Extension stakeholders are demanding high-level impacts such as economic impacts (O'Neill, 2008) and adoption of improved practices. Clements (1999) stated that impacts mean behavior changes to legislators. Because of stakeholders' expectation for high-level impacts, reporting program results with lower-level indicators of the impact hierarchy can be subjected to the "so what?" criticism of stakeholders.

When Extension agents engage in higher-level impact evaluations, Barge (2007) found they generally evaluate their educational programs by conducting pre- and post- knowledge tests. Although changes in participants' knowledge are higher-level indicators of impact than participants' levels of satisfaction with the program, there are only limited research findings that measure connections between participants' knowledge and changes in their behavior. For instance, Roberts, Barrett, Howells, Shanklin, Pilling, and Brannon (2008) described that "little research has been conducted exploring actual behavior of foodservice employees before and after food safety training" (p. 253).

Knowledge is useful only if it is transformed into actions (Büssing, Herbig, & Latzl, 2006). Extension stakeholders want to know whether the programs are effective in changing participants' behavior and practices leading to the planned end results. Because of this interest in evaluating outcomes, Extension educators are seeking information to establish the relationship between program participants' knowledge change and their behavior changes. Therefore, exploration of this relationship has a practical significance in predicting the direction of participants' behavior changes based on the knowledge they acquire.

There is some evidence to support the notion that change in knowledge has positive impact on changes in behavior. For instance, Edmiston and Gillett-Fisher (2006) conducted a research project with Federal Reserve Bank employees and concluded that there was a positive relationship between employees' financial knowledge and their retirement saving behavior. Similar to this finding, Belcher, Watkins, Johnson, and Ialongo (2007, ¶ 1) reported that "caregivers with more knowledge of child development had more optimal parenting behavior." Examining the relationship between nutrition knowledge and behavior among Finish homemakers, Palojoki (2007) concluded that knowledge had a weak relationship with healthy food behavior.

A study conducted in Arizona with Expanded Food and Nutrition Education Program (EFNEP) participants indicated that food safety knowledge had a small, positive effect on their food safety practices (Meer & Misner, 2000). However, Roberts et al. (2008) studied the outcomes of a food safety education program presented to food service employees and concluded that "an increase in knowledge does not necessarily mean that a change in behavior will take place" (p. 259). These contrasting observations indicate that there is a need for further investigation of the relationship between the changes in program participants' knowledge and changes in their subsequent behavior.

Purpose

The purpose of the research reported here was to study the relationship between the changes in the knowledge and behavior of childcare providers participating in a food safety self-study course. More specifically, the research investigated the correlation between childcare providers' knowledge change after participating in a food safety self-study course and their subsequent change in safe food handling behaviors.

Methods

The evaluation study used an ex-post facto research design. The study sample included all 77 in-home childcare providers who elected to participate in the pilot program of a childcare self-study Extension course focused on safe food handling in family childcare homes. The concepts and behaviors targeted in the self-study course were developed based on current research in safe food handling practices, as well as feedback collected during focus groups with in-home care providers at three locations statewide and through a survey of licensing professionals who work directly with family childcare providers.

The curriculum used is a self-study program focused around a series of 12 short video segments on food safety practices in in-home childcare settings. Each video segment has two parts, an initial scenario showing a childcare provider making common food handling mistakes and a follow-up segment showing the same provider preparing the food using proper food handling techniques. Participants viewed the first part, were instructed to pause the video and to identify mistakes they observed by writing them on an accompanying handout included in the self-study materials, and then viewed the second part of the segment to see the proper technique illustrated. A voiceover explained the steps in proper food handling. Additional resources on each topic were included in the self-study materials.

The impact of the food safety self-study course was evaluated using a pre- and post-test design. Two instruments, one for testing knowledge and one for recording behaviors, were developed for data collection. The instrument designed for testing participants' knowledge contained 25 multiple-choice questions on food safety concepts. Of the 25 questions, the correctly answered percentage was computed and used as the knowledge test score for analysis. The knowledge test score can be ranged from 0% to 100% on this scale.

The instrument designed to test participants' behavior change contained 19 food safety-related core practices as a pre- and post-course checklist. Practices included were based on common proper food handling and storage practices, as well as input from the initial focus groups and survey responses of the licensing personnel. Observers used this checklist to record whether or not each practice was adopted in the family childcare home. Of the 19 food safety practices, the percentage of food safety practices being adopted was computed and used for analysis. The Chronbach reliability alpha of the knowledge testing and behavior testing instruments were, respectively, .80 and .69. According to Nunnally (1994), this moderate level of reliability coefficient is sufficient in pilot testing.

Extension agents were trained in the content, delivery methods, and use of the evaluation instruments as part of a 4-hour training prior to program implementation. This included a timeline for pre- and post-evaluations, content administration, and follow-up with participants. Agents were guided through the behavior checklist as a part of the training.

The food safety self-study course for family childcare providers was advertised, and potential participants were enrolled in the self-study program. Before the training, every participant was administered the 25 multiple-choice question pre-knowledge test. Each participant then viewed a series of short video segments during the self-study course. They first saw a segment demonstrating unsafe food handling practices. They completed an activity sheet where they identified the food safety problems in the video. Then they viewed a subsequent clip showing the correct handling practices and read accompanying handouts describing safe food handling practices.

At the end of the training the same multiple-choice question test was administered as the knowledge post-test. Home visits were made once again to the convenience sample by the same person who had inspected their home before the course began and recorded participants' food safety behaviors on the checklist.

Pre- and post-tests and inspection forms were matched for each participant to assess the change in knowledge and behaviors. After data was collected, knowledge tests were graded, and the percentage of correct answers was computed. Percentage of the food safety behaviors adopted was computed for pre- and post- food safety behavior checklists. Data were entered into the SPSS program for analysis. Descriptive statistics and correlations were used to analyze data.

Results and Discussion

The study received 77 usable knowledge pre- and post-tests. Only 27 childcare providers agreed to participate in home food safety inspections conducted by Extension agents or program assistants. These trained observers used the food safety inspection checklist before and after the program to record participants' food safety practices. The number of childcare providers who expressed their willingness to have their kitchens audited before and after participation in the course and Extension agents' available time for observations were the two factors that mainly limited the number of childcare providers in the sample to 27. Of these, 23 pre- and post- behavior observation checklists were completed.

Evaluation data indicate that participants improved their food safety knowledge and behavior significantly, as summarized in Table 1. The mean food safety knowledge pre-test score was 70.5%, and the mean post-test score was 88%. Participants followed an average of 67.7% of the recommended food safety behaviors during pre-training home inspections. After completing the self-study course, participants' food safety behavior scores increased from 67.7% to 93.8% on this scale.

Table 1.
Participants' Food Safety Knowledge and Behaviors Before and After the Training Program

Type of Impact Indicators	n	Pre-Test		Post-Test		t	p
		M	SD	M	SD
Knowledge test score	77	70.5	17.266	88.0	15.948	8.14	.000*
Behavior checklist score	23	67.7	16.207	93.8	7.384	7.68	.000*
* Note. Significant at p<.001 (2-tailed)

Each participant's knowledge and behavior pre-test scores were subtracted from the post-test scores to measure the changes in knowledge and behaviors in order to achieve the research objective. The Pearson Correlation Coefficient between participants' food safety knowledge changes and their food safety behavior changes was .497, and it is significant at the .05 level. The results indicate that increases in childcare providers' food safety knowledge are related to positive changes in their safe food handling practices.

These findings are similar to those in a food safety study conducted in Spain with school children, which reported that generally, the students with more knowledge about food safety had better hygiene practices (Garayoa, Córdoba, García-Jalón, Sanchez-Villegas, & Vitas, 2005). Further, a study conducted by Meer and Misner (2000) in Arizona found that EFNEP participants' food safety knowledge had a small, positive effect on their food safety practices.

By conducting a meta-analysis of previous research studies, Hornik, Cherian, Madansky, and Narayana (1995) concluded that there was a strong positive correlation (r=.54) between knowledge and recycling behavior. Referring to recycling behavior, Shultz described that "the overwhelming finding from the research is that knowledge is a strong and consistent predictor of recycling behavior (2002, p.70)."

Research findings about positive correlation between knowledge and behavior change can be found in literature from different content areas such as knowledge about asthma and asthma management behavior (Rubin, Bauman, & Lauby, 1989), nutrition knowledge and healthy eating behavior (Palojoki, 2007), and financial knowledge and personal financial behavior (Edmiston & Gillett-Fisher, 2006). Available literature, as well as, findings of this research support the notion that "knowing and not doing is equal to not knowing" (Clements, 1999, p. 1)

Conclusions

The results of this study warrant the following conclusions.

• The food safety self-study course significantly improved home-based family childcare providers' food safety knowledge and behavior.
• There is a moderately positive correlation between the participants' food safety knowledge change and their subsequent food safety behavior changes.
• Change in childcare providers' knowledge of safe food handling practices is a reasonable indicator for predicting their subsequent change in food handling behaviors.

Implications and Limitations

Child-care providers' food safety knowledge improvement can be considered as a reasonable impact indicator in determining the direction of their food safety behavior change. The study reported here was conducted using self-study materials with home-based family childcare providers who care for small groups of children in their own homes during the day. The findings of the study can be applied for this client group and type of training. However, results might be different for center-based childcare providers or in situations where instructors present the same material in a classroom setting.

The main implication of the study is that Extension professionals may be able to use knowledge change as a reliable impact indicator of potential behavior changes of childcare food safety self-study program participants. Additional research into knowledge and behavior changes of self-study food safety programs is needed to determine if the findings of this study apply to other audiences.

The use of a convenience sample for recording participants' food safety behavior and the small sample size is a limitation of this study in generalizing the findings to other childcare providers or to other participants in Extension food safety programs. Future research with control groups and other types of Extension programs is needed to further explore the relationship between participants' knowledge change and their behavior changes in order to generalize the findings for other Extension programs.

References

Barge, G. A. (2007). Pre- and post-testing with more impact. Journal of Extension [On-line], 45(6) Article 6IAW1. Available at: http://www.joe.org/joe/2007december/iw1.php

Belcher, H. M. E., Watkins, K., Johnson & Ialongo, N. (2007). Early head start: Factors associated with caregiver knowledge of child development, parenting behavior and parenting stress. NHSA Dialog, 10, (1), 6-19. Abstract retrieved August 6, 2008, from: http://www.informaworld.com/smpp/content~content=a788027696~db=all

Büssing, A., Herbig, B., & Latzl, A. (2006). Knowledge, experience, action: A new model to analyze their mutual relation. In D. Frey, H. Mandl & L. V. Rosenstiel (Eds.), Knowledge and action. Cambridge, MA: Hogrefe & Huber Publishers.

Clements, J. (1999). Results? Behavior change. Journal of Extension [On-line], 37(2) Article 2COM1. Available at: http://www.joe.org/joe/1999april/comm1.php

Edmiston, K. D., & Gillett-Fisher, M. C. (2006). Financial Education at the Workplace, Part I: Knowledge and Behavior. Community Affairs Working Paper 06-02. Retrieved August 6, 2008, from: http://www.kansascityfed.org/comaffrs/GillettFisher_Edmiston_Financial_Education_April_2006.pdf

Garayoa, R., Córdoba, M., García-Jalón, I., Sanchez-Villegas, A. & Vitas, A. I. (2005). Relationship between consumer food safety knowledge and reported behavior among students from health sciences in one region of Spain. Journal of Food Protection, 68, (12), 2631-2636. Abstract retrieved February 8, 2009, from: http://cat.inist.fr/?aModele=afficheN&cpsidt=17317719

Hornik, J., Cherian, J., Madansky, M., & Narayana, C. (1995). Determinants of recycling behavior: A synthesis of research results. Journal of Socio-Economics, 24(1), 105-127. Retrieved April 22, 2009, from: http://www.sciencedirect.com/science//journal/10535357

Meer, R. R., & Misner, S. L. (2000). Food safety knowledge and behavior of Expanded Food and Nutrition Education Program participants in Arizona. Journal of Food Protection, 63, (12), 1725-1731. Abstract retrieved August 06, 2008, from: http://apt.allenpress.com/perlserv/?request=get-abstract&
doi=10.1043%2F0362-028X%282000%29063%5B1725%3AFSKABO%5D2.3.CO%3B2

Novakofski, K. C., Boeckner, L. S., Canton, R., Clark, C. D., Keim, K., Britten, P., & McClelland, J. (1997). Evaluating evaluation—What we've learned. Journal of Extension [On-line], 35(1) Article 1RIB2. Available online at: http://www.joe.org/joe/1997february/rb2.php

Nunnally, J. C. (1994). Psychometric theory (3^rd ed.). New York: McGraw-Hill.

O'Neill, B. (2008). Calculating the economic impact of health education programs: Five tools for Extension educators. Journal of Extension [On-line], 46(1) Article 1TOT4. Available at: http://www.joe.org/joe/2008february/tt4.php

Polojoki, P. (2007). The relationship between nutrition knowledge and food behavior among Finnish homemakers. International Journal of Consumer Studies, 20 (4), 327-338.

Roberts, K. R., Barrett, B. B., Howells, A. D., Shanklin, C. W., Pilling, V., & Brannon, L. (2008). Food safety training and foodservice employees' knowledge and behavior. Food Protection Trends, 28, (4), 252-260. Retrieved August 06, 2008, from: http://krex.k-state.edu/dspace/bitstream/2097/806/1/RobertsFPTApr2008.pdf

Rubin, D. H., Bauman, L. J., & Lauby, J. L. (1989), The relationship between knowledge and reported behavior in childhood asthma. Journal of Developmental and Behavioral Practices, 10, (6), 307-312.

Schultz, P. W. (2002). Knowledge, information and household recycling: Examining the knowledge-deficit model of behavior change. In T. Dietz & P. C. Stern (Eds.), New tools for environmental protection: Education, information and voluntary measures (pp. 67-82). Washington, D C: National Academy Press.