Evaluation of On-Farm Food Safety Programming in Pennsylvania: Implications for Extension

Introduction

Microbial contamination of farm produce has been identified as a source for foodborne disease outbreaks. The number of outbreaks related to contamination of produce has increased from 1973 to 1997 (Sivapalasingam, Friedman, Cohen, & Tauxe, 2004). Citing data from the Center for Disease Control and Prevention (CDC) on outbreaks associated with fresh produce commodities, the U.S. Food and Drug Administration (FDA) reported approximately 131 produce-related outbreaks from 1996 to 2010 (FDA, 2013a). The CDC reported that during 2012, several multistate foodborne illnesses associated with contaminated lettuce (CDC, 2012), organic spinach, and spring mix blend as well as cantaloupes resulted in hospitalizations and even deaths (CDC, 2013). In 1998, FDA proposed a set of agricultural production practices to reduce the risk of microbial contamination of fresh produce (FDA, 1998). These food safety practices, known as Good Agricultural Practices (GAPs), address farm workers' health and hygiene; the quality of agricultural water; the use of domesticated animals; potential contamination by wild animals; sanitation standards for equipment, tools, and buildings; and traceability/recall.

The Food Safety Modernization Act (FSMA) of 2011 has mandated the FDA to take preventive measures to ensure a safer food supply. The proposed food safety rules for fresh produce under FSMA (FDA, 2013b) would establish mandatory food safety guidelines for produce growers (Taylor, 2011). At the same time, private retailers such as supermarkets are increasingly implementing their own food safety standards that require their produce suppliers to provide evidence of their compliance with GAPs. Tobin, Thomson, LaBorde, and Bagdonis (2011) reported that in Pennsylvania, different supermarkets require various forms of evidence of GAP compliance from their produce suppliers. Examples of different forms of evidence include a written food safety plan for farm operations or third party-certification (TPC) verifying growers' compliance with on-farm food safety practices. Failing to provide evidence of GAP compliance to their produce buyers as a precondition of purchase may affect growers' market viability (Hatanaka, Bain, & Busch, 2005).

Despite the uncertainty in food safety policies, produce growers still need educational training and technical assistance on GAP skills—not only to document and implement GAPs on their farms (Eggers, Ackerlund, Thorne, & Butte, 2010)—but also to comply with current and future food safety policy requirements (Tobin, Thomson, LaBorde, & Radhakrishna, 2013). Responding to the growing concerns of food safety issues related to fresh produce and the growers' need to carry out food safety regulations to maintain market viability, Penn State Extension offered food safety workshops for these growers. These educational programs are intended to provide growers with the knowledge and skills needed to comply with and document GAPs, conduct a food safety self-inspection on their farms, and apply for a TPC.

Evaluations of these food safety workshops were carried out to ensure that the food safety programming was serving the participating produce growers' GAPs information needs. Tobin et al. (2013) indicated that evaluation is essential for Extension to ensure that on-farm food safety programming is addressing the different requirements and priorities of produce growers and their buyers. Systematic evaluation is valuable to indicate program strengths and limitations, as well as identify possibilities to improve programming (Radhakrishna & Relado, 2009). The evaluation findings can also be used to identify programming content that needs more attention (Chapman-Novakofski et al., 2004). The purpose of the study reported here was to assess the impact of the on-farm food safety workshops offered by Penn State Extension for participating growers.

Framework for Program Evaluation

A framework proposed by Rockwell and Bennett (2004) was used to complete the evaluation. Rockwell and Bennett elaborated seven hierarchical steps that explain how practice change among program participants is more likely to occur after they change their knowledge, attitudes, skills, and aspirations/intentions (KASA/I). Increases in participants' knowledge on specific topics, modification in their attitudes, improvement in their skills, and change in their aspirations or intentions are required to change participants' behaviors or practices (Rockwell & Bennett, 2004). These changes in program participants' KASA/I are short-term program outcomes focusing on impact evaluation (Radhakrishna & Bowen, 2010). The study focused on the KASA/I level of evidence to measure the effectiveness of the on-farm food safety workshops.

The framework for the study operates under the assumption that if participants possess GAPs knowledge and modify their attitudes about food safety and GAP-related activities, then they will have greater confidence in adopting GAP skills. If participants have knowledge and confidence, then they are more likely to adopt/implement GAP practices on their farms (Figure 1). Although Tobin et al. (2013) concluded evaluation frameworks need to incorporate variables that determine on-farm food safety outcomes beyond those included in Rockwell and Bennett's model, they also concluded that the transfer of knowledge and skills is important in encouraging growers to improve their on-farm food safety practices.

Figure 1.
Evaluation Framework (from Rockwell & Bennett, 2004)

Program Delivery

During winter 2012, Penn State Extension conducted on-farm food safety workshops statewide to train fruit and vegetable growers on Good Agricultural Practices (GAPs). A total of 10 GAP certificate training workshops were offered from January to April 2012, prior to the 2012 fruit and vegetable growing season.

Titled "Keeping Fresh Produce Safe Using Good Agricultural Practices," these workshops provided growers with technical information regarding GAP topics. Targeting growers who sold their fresh produce through auctions, cooperatives, and supermarkets, these 5-hour workshops focused on implementing GAPs on their farms. The major goals of these workshops were to educate and train growers on how to write a food safety plan, conduct food safety self-inspections, and apply for a TPC.

The purpose of the study was to assess how effective the GAP training workshops were for program participants. Specifically, the following objectives guided the study:

• Determine growers' change in GAP knowledge,
• Determine growers' change in attitudes toward GAPs,
• Determine growers' change in confidence in GAP skills, and
• Assess growers' intentions to carry out GAP-related activities.

Methodology

In order to improve Extension's GAP educational programming and assess changes in participants' knowledge, attitude, confidence, and intentions on GAPs, these workshops were evaluated using a pre- and post-test design. The pre- and post-test method has been used to evaluate various programs (Lippert, Plank, & Radhakrishna, 2000; Chapman-Novakofski et al., 2004; Fishel, 2008). This type of evaluation design is generally used to measure short-term program impact such as KASA/I. A survey was developed by faculty and educators at the Pennsylvania State University to assess the impact of on-farm food safety workshops. Questions were assessed for content validity, readability, and appropriateness for the target audience.

Growers' knowledge of GAPs was measured using 10 knowledge-based statements (True and False) covering GAP topics. Growers' attitudes on GAPs and their confidence in their GAP skills were measured on a five-point Likert scale (1 = not at all confident to 5 = very confident). Growers were asked to rate their confidence in the following GAP skills: write or update a food safety plan, conduct their own food safety inspection, and prepare for a third-party audit. Five statements related to the participants' attitudes on GAPs asked participants to rate how strongly they agreed or disagreed with each statement (1 = do not agree to 5 = very much agree). These attitude statements covered issues related to grower's produce safety responsibilities, the benefits of food safety audits to maintain farm produce sales, consumers' perceptions on the safety of growers' farm produce, and availability of adequate resources both to write a food safety plan and to prepare and pass a GAP audit. The reliability for the items measured on the Likert scales was acceptable (Cronbach's α for confidence = .83 and for attitude = .81).

Participants were asked to respond to these knowledge, confidence, and attitude statements both before and after the workshop. As part of the follow-up evaluation, participants were asked if they intended to carry out the following GAP-related activities on their farms: write or update a food safety plan, conduct their own food safety inspection, and prepare for a third-party audit.

Statistical analysis software, SPSS version 21, was used to analyze the data. Overall, 330 individuals in 10 workshops received the evaluation; 263 completed it, for a response rate of 79.7%. Paired t-tests were carried out to determine changes in knowledge, attitudes, and confidence level scores. The study was approved by the Institutional Review Board (IRB) at the Pennsylvania State University. Responses for each objective varied because only those respondents who provided answers to both pre- and post-tests questions were included. That resulted in a smaller sample for each objective than the actual number of respondents who completed surveys.

Results

Objective 1: Determine Growers' Change in GAP Knowledge

Descriptive analysis indicated workshop participants increased their GAP knowledge scores. The overall mean score for the 10 knowledge questions increased from a mean of 6.61 (before) to a mean of 8.06 (after) the workshops, an increase of 1.46 (Table 1). This increase in scores was significant at p < .001 level (SD = 1.46; mean difference = -1.46; t-value = -13.79). The largest increase in correct response (47.9%) occurred for the statement "fresh fruits and vegetables are the primary cause of foodborne illnesses" (Figure 2). However, increases in the correct responses did not occur across all of the knowledge statements. Results indicated after workshop knowledge scores increased for seven of 10 knowledge questions. Correct responses decreased from before to after the workshops for three questions: the safe application of manure-based compost (66.8% and 54.7% respectively); FDA Food Safety Modernization Act states that all produce growers submit to a farm audit (71.6% and 63.2% respectively); and the restriction of wild animals entering fields (73.2% and 66.3% respectively) (Figure 2).

Table 1.
Participant Growers' Overall Knowledge on GAP

	N	Before Workshop Mean	After Workshop Mean	SD	Mean Difference	t value
Knowledge (0-10)a	190	6.61	8.06	1.46	-1.46	-13.79*
a Knowledge score could range from 0 correct to 10 correct. * p < .001

Figure 2.
Before and After Workshop GAP Knowledge Scores of Respondents (N=190)

 

Question#	Knowledge Questions in Figure 2	Answer
Q1	USDA standards require that pond water used for irrigation be tested for microbes at least 3 times during the growing season.	True
Q2	After hand washing, hands should be dried thoroughly with a clean cloth towel.	False
Q3	The FDA Food Safety Modernization Act recently passed by Congress requires all produce growers to submit to a farm audit.	False
Q4	Produce harvested into boxes or bins should be covered when they are transported to a packing house.	True
Q5	It is possible for manure-based compost to be safely applied around produce crops.	True
Q6	Drip irrigation methods are more likely to cause crop contamination than overhead spraying.	False
Q7	USDA audit standards require produce growers to prove that wild animals are not able to enter fields.	False
Q8	Fresh fruits and vegetables are responsible for the greatest number of foodborne illnesses.	True
Q9	Hand sanitizer sprays are an acceptable substitute for hand washing.	False
Q10	USDA audit standards require packing areas to be fully enclosed.	False

Objective 2: Determine Growers' Change in Attitudes

Growers' attitudes toward GAP-related issues were measured with five statements. The overall mean score for the five attitude statements increased from a mean of 18.80 before to 21.08 after the workshops (Table 2). This increase in scores was significant at p < .001 level (SD = 2.58; mean difference = -2.27; t-value = -11.93). For the statement regarding the degree to which growers believed that they have responsibility for the safety of the produce from their farms, a majority of the respondents agreed or very much agreed both before and after the workshops (Table 3). After the workshops, the highest increase (37.2%) in growers' agreement occurred for the statement, "have adequate resources to write their own food safety plans."

Table 2.
Participant Growers' Overall Attitudes Toward Carrying Out GAPs

	N	Before Workshop Mean	After Workshop Mean	SD	Mean Difference	t value
Attitude (5-25)a	183	18.80	21.08	2.58	-2.27	-11.93*
a Measured on a scale 1 (not at all agree) to 5 (very much agree). Attitude score could range from a low of 5 to a high of 25 with a theoretical midpoint of 15. *p < .001

Table 3.
Respondents' Attitudes Toward GAP-Related Issues (N=183)

Statements	Before the Workshops			After the Workshops
	Agree/Very Much Agree	Meana	SD	Agree/Very Much Agree	Meanb	SD
Farmers have a responsibility for the safety of produce coming off their farms	90.7%	4.67	0.67	94.6%	4.77	0.58
Preparing for a food safety audit will help my farm maintain produce sales	65.6%	3.87	1.13	74.3%	4.14	1.03
How consumers feel about the safety of my farm's produce affects how much produce my farm sells	79.8%	4.29	0.99	86.4%	4.43	0.90
I have adequate resources to write my own food safety plan	31.1%	3.03	1.20	68.3%	3.94	0.96
I have adequate resources to prepare for and pass a GAP audit	28.4%	2.95	1.17	62.8%	3.80	0.95
a, b Mean score could range from 1 to 5.

Objective 3: Determine Growers' Change in Confidence

Respondents indicated their confidence both before and after the workshops in three specific skills: writing or updating a food safety plan, conducting a self-audit, and preparing for a third-party audit (Figure 3). The overall confidence mean scores increased from 8.33 before the workshops to 11.26 after the workshops (Table 4). This increase in overall mean scores was significant at p < .001 (SD = 2.58; mean difference = -2.93; t-value = -15.70). After the workshops, growers exhibited greater confidence in their GAP skills than before the workshops. Growers' confidence in writing a food safety plan increased by 44.8%, conducting a self-food safety inspection by 36.0%, and preparing for a third-party audit by 34.3%.

Figure 3.
Growers' Confidence Before and After Workshops

Table 4.
Participant Growers' Overall Confidence to Carry Out GAPs

	N	Before Workshop Mean	After Workshop Mean	SD	Mean Difference	t value
Confidence (3-15)a	192	8.33	11.26	2.58	-2.93	-15.70*
a Measured on a scale 1 (not at all confident) to 5 (very confident). Confidence score could range from a low of 3 to a high of 15 with a theoretical midpoint of 9. *p < .001

Objective 4: Assess Growers' Intentions

After the workshops, respondents indicated their intent to implement specific GAP activities (writing or updating a food safety plan, conducting a food safety inspection, and/or performing a third-party audit) for the 2012 growing season (Table 5). The majority of respondents (51.8%) indicated that they intended to write or update a food safety plan, and 63.5% indicated their intent to conduct their own food safety inspection. However, only 20.2% of the respondents indicated that they intended to have a third-party audit conducted on their farms.

Table 5.
Respondents' Intentions to Carry Out GAP-related Activities

For the 2012 growing season, will you	N	Yes	No	Unsure
Write or update a food safety plan?	220	114 51.8%	17 7.7%	89 40.5%
Conduct your own food safety inspection?	222	141 63.5%	14 6.3%	67 30.2%
Have a third-party audit done on your farm?	213	43 20.2%	75 35.2%	95 44.6%

Discussion and Implications for Extension

In order to assure a safe produce supply, all produce growers need to learn about and implement on-farm food safety practices. Penn State Extension has the responsibility to inform and train produce growers to acquire the knowledge and skills to implement updated food safety plans on their farms and to comply with food safety regulations. Evaluation of on-farm food safety programming was conducted to assess the program's impact on participating growers. The change in program participants' KASA/I as a result of program participation can be considered as evidence of program impact/outcome. Overall, on-farm food safety programming helped growers increase their GAP knowledge, attitudes, and confidence to carry out GAP-related activities. Increase in growers' GAP knowledge, modifications in their attitudes, and increase in their confidence are important with respect to their subsequent behavior change. Several studies suggest positive relationship between participants' change in knowledge and change in their practice (Edmistion & Gillet-Fisher, 2006; Jayaratne, Harrison, & Bales, 2009).

After-workshop evaluations revealed an overall increase in growers' GAP knowledge, but some GAP knowledge areas showed a decrease in correct responses after the workshops. In order to improve on-farm food safety programming and increase growers' GAPs knowledge, Penn State Extension must focus on improving its content delivery on topics related to FDA requirements on food safety, the safe use of manure-based compost, and USDA audit standards on the restriction of wild animals. These were areas that showed a decrease in correct responses after the workshops.

The majority of respondents were aware of their farm food safety responsibilities, farm produce sales, and consumers' perceptions relative to the food safety issues before the workshops. Although after the workshops, the majority of respondents indicated they have adequate resources to prepare for a TPC, a lower percentage of respondents were confident in preparing for TPC. This discontinuity in growers' attitudes and confidence might indicate that growers are unsure of what exact procedures they need to adopt to prepare for TPC because of varied, current and emerging, food safety regulations. Extension should focus its educational programming for produce growers on step-by-step instructions to prepare them for a GAP audit that meets the criteria specified by their buyers. Responding to the varied needs of produce growers, Extension should consider participatory evaluations to improve on-farm food safety programming and to assess outcomes. In participatory evaluations, program participants play a role in developing evaluations that are more relevant to their needs and practices (Lennie, 2006).

The workshops prepared growers to write food safety plans and conduct self-inspections. However, only one out of five respondents intended to become certified through a third-party audit. The growers' intentions to carry out specific farm practices might be in part driven by their buyers' food safety requirements. These findings indicate that growers are willing to take initial steps to verify GAP compliance but not the final one: obtaining an audit, which indicates that obtaining TPC might not be part of some growers' requirements. Eggers et al. (2010) suggested that growers' activities related to food safety practices are primarily influenced by customers' expectations and buyers' requirements.

In accordance with Tobin et al. (2013), we also recommend that future studies examine how growers' intentions to carry out specific GAP activities are related to their on-farm food safety requirements. It is also possible that a 5-hour workshop may not provide growers enough time to process the information provided and reach a thoughtful decision regarding their intent to carry out specific activities. The fact that the post assessment occurred immediately at the conclusion of the workshop may have contributed to a lack of workshop efficacy in affecting their intentions. A delayed follow-up study should be considered to more completely examine the relationships of growers' changes in knowledge and confidence and intended behavior.

Growers' intentions to carry out GAP-related activities as a result of program participation may not necessarily translate into practice change. Therefore, follow-up surveys need to be carried out to monitor participants' on-farm behavior change after their attendance at food safety workshops (Tobin et al., 2013). In addition, program outcomes such as practice changes are not immediate. Extension program evaluation should track changes in growers' behaviors that occur after a change in knowledge (Braverman & Engle, 2009). Penn State Extension can play a major role in reinforcing smaller key steps that need to be taken in order to ensure food safety. Smaller changes in growers' practices to ensure the safety of the produce ultimately help them to fully implement GAPs on their farms. Based on the observations of food safety specialists and educators, after attending this type of food safety workshop, growers reportedly have started using new boxes to pack produce; earlier they used the same boxes multiple times, increasing the risk of produce contamination. Future evaluation should include questions related to what practice changes the produce growers have adopted to comply with GAPs since their attendance at on-farm food safety workshops.

Evaluation frameworks, which support knowledge and skills transfer leading to behavior changes, may be less likely to address the factors that influence of growers' on-farm food safety practices (Tobin et al., 2013). Therefore, Extension must understand the complexity of on-farm food safety and examine the challenges that influence on-farm food safety practices. Future on-farm food safety programming evaluations should identify the challenges that produce growers are facing in implementing GAPs on their farms and tailor programming to address issues related to produce growers' on-farm challenges.

Acknowledgments

Partial funding for the work reported here was provided by the USDA Sustainable Agriculture Research and Education Program.

References

Braverman, M. T., & Engle, M. (2009). Theory and rigor in Extension program evaluation planning. Journal of Extension [On-line], 47(3) Article 3FEA1. Available at: http://www.joe.org/joe/2009june/a1.php

Centers for Disease Control and Prevention (CDC). (2012). Multistate outbreak of E. coli O157:H7 infections linked to romaine lettuce. Retrieved from: http://www.cdc.gov/ecoli/2011/ecolio157/romainelettuce/032312/index.html

Centers for Disease Control and Prevention (CDC). (2013). Multistate foodborne outbreak investigations. Retrieved from: http://www.cdc.gov/foodsafety/outbreaks/multistate-outbreaks/outbreaks-list.html

Chapman-Novakofski, K., DeBruine, V., Derrick, B., Karduck, J., Todd, J., & Todd, S. (2004). Using evaluation to guide program content: Diabetes education. Journal of Extension [On-line], 42(3) Article 3IAW1. Available at: http://www.joe.org/joe/2004june/iw1.php

Edmiston, K. D., & Gillett-Fisher, M. C. (2006). Financial education at the workplace, Part I: Knowledge and behavior. Community Affairs Working Paper 06-02. Retrieved from: http://www.kansascityfed.com/publicat/cap/gillettfisher_edmiston_financial_education_april_2006.pdf

Eggers, S., Ackerlund, S., Thorne, S., & Butte, G. (2010, June). Report on growers' understanding and implementation of FDA's GAPs guidance. Retrieved from: http://www.fda.gov/Food/FoodborneIllnessContaminants/BuyStoreServeSafeFood/ucm218562.htm

Fishel, F. (2008). Using a pre-and post-test to assess 1-day learning of agricultural producers participating in an educational program. Journal of Extension [On-line], 46(5) Article 5TOT3. Available at: http://www.joe.org/joe/2008october/tt3.php

Food and Drug Administration (FDA), Center for Food Safety and Applied Nutrition. (1998). Guidance for industry: Guide to minimize microbial food safety hazards for fresh fruits and vegetables. Retrieved from: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/ProducePlantProducts/ucm064574.htm

Food and Drug Administration (FDA). (2013a). Produce safety standards. Retrieved from: http://www.fda.gov/Food/GuidanceRegulation/FSMA/ucm304045.htm

Food and Drug Administration (FDA). (2013b). Standards for the growing, harvesting, packing, and holding of produce for human consumption; Proposed Rule. Retrieved from: http://www.fda.gov/downloads/Food/GuidanceRegulation/FSMA/UCM360734.pdf

Hatanaka, M., Bain, C., & Busch, L. (2005). Third-party certification in the global agrifood system. Food policy, 30(3), 354-369. Doi: 10.1016/j.foodpol.2005.05.006

Jayaratne, K. S. U., Harrison, J. A., & Bales, D. W. (2009). Impact evaluation of food safety self-study Extension programs: Do changes in knowledge relate to changes in behavior of program participants? Journal of Extension [On-line], 47(3) Article 3RIB1. Available at: http://www.joe.org/joe/2009june/rb1.php

Lennie, J. (2006). Increasing the rigour and trustworthiness of participatory evaluations: Learnings from the field. Evaluation Journal of Australasia, 6(1), 27-35. Retrieved from: http://eprints.qut.edu.au/26035/1/EJA_paper_by_Lennie_2006.pdf

Lippert, R. M., Plank, O., & Radhakrishna, R. (2000). Beyond perception: A pretest and posttest evaluation of a regional Internet Extension inservice training. Journal of Extension [On-line], 38(2) Article 2FEA2. Available at: http://www.joe.org/joe/2000april/a2.php

Lynch, M. F., Tauxe, R. V., & Hedberg, C. W. (2009). The growing burden of foodborne outbreaks due to contaminated fresh produce: Risks and opportunities. Epidemiology and Infection, 137(3), 307-315. Doi: 10.1017/S0950268808001969

Radhakrishna, R. B., & Bowen, C. F. (2010). Viewing Bennett's hierarchy from a different lens: Implications for Extension program evaluation. Journal of Extension [On-line], 48(6) Article 6TOT1. Available at: http://www.joe.org/joe/2010december/tt1.php

Radhakrishna, R. B., & Relado, R. J. (2009). A framework to link evaluation questions to program outcomes. Journal of Extension [On-line], 47(3) Article 3TOT2. Available at: http://www.joe.org/joe/2009june/tt2.php

Rockwell, K., & Bennett, C. (2004). Targeting outcomes of programs: A hierarchy for targeting outcomes and evaluating their achievement. Faculty Publications: Agricultural Leadership, Education & Communication Department. Paper 48. Retrieved from: http://digitalcommons.unl.edu/aglecfacpub/48/

Sivapalasingam, S., Friedman, C. R., Cohen, L., & Tauxe, R. V. (2004). Fresh produce: A growing cause of outbreaks of foodborne illness in the United States, 1973 through 1997. Journal of Food Protection, 67(10), 2342-2353. Retrieved from: http://www.ingentaconnect.com/content/iafp/jfp

Taylor, M. R. (2011). Will the Food Safety Modernization Act help prevent outbreaks of foodborne illness? New England Journal of Medicine, 365(9). Doi: 10.1056/NEJMp1109388

Tobin, D., Thomson, J., LaBorde, L., & Bagdonis, J. (2011). Developing GAP training for growers: Perspectives from Pennsylvania supermarkets. Journal of Extension [On-line], 49(5) Article 5RIB7. Available at: http://www.joe.org/joe/2011october/rb7.php

Tobin, D., Thomson, J., LaBorde, L., & Radhakrishna, R. (2013). Factors affecting growers' on-farm food safety practices: Evaluation findings from Penn State Extension programming. Food Control, 33(1), 73-80. Doi: 10.1016/j.foodcont.2013.02.015