Evaluation of HACCP Training Under the Grade "A" Dairy HACCP Core Curriculum

Introduction

The application of Hazard Analysis Critical Control Point (HACCP) principles developed by the National Advisory Committee on Microbiological Criteria for Foods (NACMCF, 1998) has become standard in the food industry as a means to minimize product safety risks. While adoption of HACCP principles is mostly voluntary, HACCP systems are required for certain foods, including meat (USDA, 2011), juice (FDA, 2011a), and seafood (FDA, 2011b). Grade "A" dairy plants can be regulated under a voluntary HACCP auditing system as an alternative to the traditional inspection-based system outlined in the Pasteurized Milk Ordinance (PMO; FDA, 2009). This voluntary program was developed under the National Conference on Interstate Milk Shipments (NCIMS-HACCP). Dairy plants under both programs are routinely audited and/or inspected to ensure compliance.

To develop HACCP programs, detailed training is needed. While most formal training is based on NACMCF guidelines, curriculum standardization has not been well defined (Wallace & Powell, 2005). Exceptions include training developed for regulated HACCP programs (Jackson, Harris & Cross, 1996; FDA, 2011c; FDA, 2003; FDA, 2007). Under the NCIMS, the HACCP Implementation Committee (NCIMS-HIC) developed the dairy HACCP core curriculum, which provides instruction in the application of HACCP principles and regulations for dairies enrolled in NCIMS-HACCP.

While developed for NCIMS-HACCP, NCIMS-HIC training should be applicable to all dairy operations and could serve as a model program. However, procedures evaluating training outcomes of the program have not been fully used, and there is a need to further measure success. The Kirkpatrick evaluation model provides a basic process that can be easily applied to training events (Johnson, 2011; Stup, 2003), with four levels measuring reaction, learning, behavior, and ultimate outcomes. This article describes the evaluation of two NCIMS-HIC trainings based on the first three levels. The goal was to determine the impact of the NCIMS-HIC dairy core curriculum on the participants as presented and to assess its value as an industry training program.

Methods

Training Programs and Materials

Two HACCP workshops were offered in January and June, 2011, with 29 and 30 attendees, respectively. One was offered through the Dairy Extension group at Cornell University. The other was sponsored by a local food association. Both programs were 2.5 days and had the same two instructors (NCIMS-HIC members). The course outline is shown in Table 1.

Table 1.
NCIMS-HIC Dairy HACCP Core Curriculum Program Outline

1.	Introduction: NACMCF Overview & Definitions
2.	Dairy/Food Safety Overview: Hazards & Controls
3.	Preliminary Steps to HACCP      Team Exercise: Team Documentation, Product Descriptions & Flow Diagrams
4.	Prerequisite Programs (PP) & Good Manufacturing Practices (GMPs)      Team Exercise: PP Development & Implementation
5.	HACCP Principles 1 & 2:    Conducting a Hazard Analysis/Determining Critical Control Points (CCP)      Team Exercise: Hazard Analysis & CCP determinations
6.	HACCP Principle 3 & 4: Establish Critical Limits (CL) & Monitoring Procedures
7.	HACCP Principle 5: Establish Corrective Actions for CL Deviations
8.	HACCP Principle 6: Establish Verification Procedures
9.	HACCP Principle 7: Establish Recording Keeping & Documentation Procedures
10.	Model HACCP Plans: NCIMS Pasteurization CCP (PMO) & Others Team Exercise: HACCP Plan Development
11.	NCIMS-HACCP Program and FDA Juice HACCP Overview
12.	Training Program Summary:      Class Exercise: Verification/Validation

Throughout the training, steps for HACCP development and implementation were covered along with NCIMS and Juice HACCP regulatory requirements. Juice HACCP was included because many dairies process juice. Applicable HACCP-based regulations of the new Food Safety Modernization Act (FDA, 2011d) were also included as relevant. The trainings provided in-depth coverage of PPs; hazard analyses and CCP determinations; and verification/validation. Attendees participated in team exercises, with four breakout groups in January and 5 in June. Notebooks with printouts of all lecture slides were provided. Additional materials included the NACMCF guideline, NCIMS and Juice HACCP regulations, GMPs (21CFR110), PP examples, and the NCIMS-HIC Dairy Hazards and Controls Guide (FDA, 2007). HACCP forms provided included standardized forms for product descriptions, hazard analyses, and HACCP plans; tailored forms developed for NCIMS-HIC training included forms for team documentation, PPs, CCP deviations, verification/validation activities, and program records.

Program Evaluation

Reaction

A one-page evaluation form was developed to assess participant's perceived value of topics covered and overall reaction to the program, materials, and presenters. It was distributed at the beginning and collected at the end of each course. Participants ranked the usefulness of each program topic on a 4-point scale. Overall program quality; the quality of reference materials; the value of class interactions; and instructor skills were ranked on a 5-point scale.

Learning

To measure assimilation of knowledge, a test was distributed to participants at the beginning (pre-test) and again at the end (post-test) of each course; pre- and post-tests were the same. Questions focusing on key HACCP concepts were developed by NCIMS-HIC members who administer NCIMS-HACCP program development. For January, the test included 20 short answer and multiple-choice questions with 48 total responses. The goal was for completion within 25 minutes, but it took longer. For June, the test was shortened to 38 responses to fit the desired time. January participants were allowed to use notebooks for the post-test only, emulating its use as a reference manual. June participants took both tests without the notebook because the January group appeared to spend too much time searching; they finished in 25 minutes. A paired t-test was used to determine differences between pre- and post-test scores (JMP, Version 7.0; SAS Institute, Cary, NC).

Behavior

To determine behavioral changes or how knowledge gained was applied at the plant level, a follow-up survey was sent to all participants 15 weeks after the trainings. Survey questions were designed to determine if perspectives had changed or if specific modifications were made or considered in their HACCP systems based on what was learned.

Results and Discussion

Attendee Overview

Most workshop participants were dairy employees with titles ranging from technician to management. In January there were two NCIMS-HACCP regulatory auditors, two consultants, and two supplier representatives. June attendees were primarily from local dairies. One auditor and three participants from nondairy companies also attended. Ten were from four milk plants that also processed juice. The trainings represented the first formal course for a majority (68%). Most processing attendees (86%) were from companies where HACCP programs were well established. Two participants commented that their "well established" programs were also a "work in progress," emphasizing that a HACCP system is an evolving process.

Program Evaluation Form

Participant responses to the program content are summarized in Table 2. For both courses, most lecture topics were marked as "very-useful" or "useful" by a majority, with average scores ≥3.5 on a 4.0 scale. Lower mean scores for juice regulation reflected the limited enrollment of companies processing juice. Program quality, materials, and speakers were ranked above average or better by most, with mean scores ≥ 4.4 out of 5.0. June "interactions" ranked higher, perhaps due to "local" dynamics. Several participants were involved with meeting Global Food Safety Initiative (GFSI, 2012) requirements, resulting in discussions related to GFSI-HACCP.

Table 2.
Course Evaluation Summary Scores as Assessment of Participant "Reaction"

Program Area	Average Rank January Course3	Average Rank June Course3
Lecture Topic Areas (4 point scale1)
Introduction to HACCP	3.5	3.4
Hazards in dairy products and other foods	3.7	3.3
Preliminary steps & prerequisite programs	3.6	3.5
Seven principles of HACCP	3.7	3.6
NCIMS Voluntary HACCP overview	3.8	3.5
Juice HACCP regulation overview	3.0	2.8
HACCP auditing basics	3.5	3.6
Group Exercise - preliminary steps	3.5	3.5
Group Exercise - prerequisite programs	3.6	3.7
Group Exercise - hazard analysis	3.7	3.7
Group Exercise - HACCP plan summary	3.6	3.7
Class Exercise - verification/validation	3.7	3.7
Over Program Evaluation (5 point scale2)
Overall program quality	4.5	4.5
Meeting objectives for attending	4.4	4.5
Provided program materials	4.4	4.5
Interactions with other participants	4.2	4.6
Overall knowledge/presentation skills of Speaker A	4.8	4.8
Overall knowledge/presentation skills of Speaker B	4.8	4.8
1 Rankings based on 4 point scale: "very-useful" = 4; "useful" = 3; "little-use" = 2; and "not-useful" = 1. 2 Rankings based on a five point scale: "excellent" = 5; "above-average" = 4; "average" = 3; "fair-below average" = 2; and "poor" = 1. 3 Average scores based on participants attending the January (n=27) and the June trainings (n=30)

Participants were asked to list new concepts they would take away from the course and potentially apply to their operation; nine from each course provided answers (Table 3). Verification and validation related topics were the most frequent responses, suggesting the importance of these critical HACCP concepts was better realized as a result of the training. Other take-home concepts listed by more than one respondent were related to PPs and alternative CCP monitoring. NCIMS-HIC training devotes substantial time covering PPs with emphasis on their use in hazard analyses. Alternative pasteurization monitoring was presented as a new topic to most.

Table 3.
New Concepts Learned by HACCP Course Participants

January Course1
1	Concept of "continuous" vs. periodic monitoring of a CCP [CCP monitoring]; We do not perform validation as defined in course [verification/validation]
2	Need for review of current HACCP plan [verification]
3	How to apply HACCP to small scale
4	Additional calibration requirements of all recording devices [verification]
5	HACCP regulations and hazards for dairy and juice
6	Chart record and data review [verification]
7	Incoming materials/receiving to be added to prerequisites [PP] Review of flow charts [verification]
8	Continuous monitoring change to a different frequency [CCP monitoring]; Review prerequisite programs using the chart provided in class [PP]
9	Example forms excellent
June Course2
1	Prerequisite program/HACCP relationships - idea that PP could nullify a CCP [PP]; More comfortable with concepts of verification and validation [verification]
2	Developing a HACCP program
3	Learning how to write a HACCP plan; New ideas/concepts from different people - thinking outside the box
4	Brought lots of loose ends together
5	Changing a third of the members of the HACCP team every 2 yrs
6	FSMA & HACCP - consideration of radiation as a hazard; Consumer complaints as part of verification [verification]
7	Sources of validation information helpful [validation]; Some new materials to help update existing program
8	Looking into suppliers e.g., allergens [verification]; HA update to simplify referring to PP [PP]
9	Breaking out different processing units to consider different factors; Better understanding of how SQF [GFSI scheme] & HACCP work together
1 For the January course, 9 out 27 provided answers in course "reaction" evaluation. Repeated HACCP concept areas are listed in brackets for appropriate responses. 2 For the June course, 9 out of 30 provided answers in course "reaction" evaluation. Repeated HACCP concept areas are listed in brackets for appropriate responses.

When asked for suggestions for improvement or additional comments, only two participants offered their thoughts on the January program. Several participants in June provided comments. While a number were positive, several focused on training pace, describing it as "rushed"; some related this to over-long exercises. The instructors acknowledged this perspective. More time was spent on exercises in June. Additional GFSI discussion described above likely contributed. While the January evaluation suggested little program change was warranted, June comments indicated otherwise. Because the training has a full program, keeping it at a consistent, acceptable pace can be a challenge. It is important for instructors to be able to adjust depending on class dynamics. Solutions might include condensing lectures; modifying exercises (e.g., rotate groups presenting); and maintaining focus on HACCP, limiting discussion on related topics. Expanding the course to 3 days may help, but might not be practical for many participants.

Pre-Test and Post-Test Learning Assessment

Pre- and post-test summary scores for HACCP training programs are in Table 4. Mean scores for the post-tests were significantly higher than pre-test scores for January, t(26) = -13.02, p ≤ 0.0001 and June, t(26) = -12.13, p ≤ 0.0001, indicating noteworthy levels of knowledge gain. Notebook use for the post-test in January did not result in higher scores.

Table 4.
Pre-Test & Post-Test as Assessment of Participant "Learning"

Course /Parameter	Pre-Test Scores (%)	Post-Test Scores (%)	Post- Minus Pre-Test Scores
January Course1
Mean	33.0a	72.1b	39.0
Minimum	4.2	16.7	12.5
Maximum	70.8	93.8	77.1
June Course2
Mean	42.5a	77.5b	35.1
Minimum	15.8	36.8	2.6
Maximum	78.9	97.4	68.4
1 January course had 27 participants. Pre- and post-tests were the same 20 questions with 48 possible correct answers. Scores are based on correct responses out of 48. 2 June course had 30 participants. Pre- and post-tests were the same 20 questions with 38 possible correct answers. Scores are based on correct responses out of 38. a,b Mean Scores in the same row with different letters were significantly different (p ≤ 0.0001)

Individual score improvement ranged from 12.5% to 77.1% in January and from 2.6% to 68.4% in June, demonstrating that pre- and post-course knowledge and/or testing abilities varied among participants. For example in January, the smallest increase was from one person who had the lowest pre- (4.2%) and post-test (16.7%) scores. The attendee with the greatest improvement scored well below the pre-test mean (14.6%), yet scored greater than 90% on the post-test. The highest pre-test (70.8%) and post-test (93.8%) scores were from one participant.

In addition to testing, team exercises provide participants with an opportunity to demonstrate applied knowledge and are required as part of the NCIMS-HIC curriculum. Based on product processing scenarios, participants build components of a HACCP system to present in class. While skills were not formally measured, most groups appeared to grasp HACCP tools and concepts. As with pre-/post-testing, individuals differ in previous knowledge and abilities to learn and contribute to the "team." In general, accessing student performance in training activities is challenging. Lo, Fukushima, Rippen, Gdovin, and Hahm (2004) proposed a primary trait analysis based on comprehension, transformation of information to perform the task, implementation, and outcome reflection that could be used for this purpose.

Post-Workshop Evaluation

Follow-up surveys were returned by 14 and 13 attendees from the January and June courses, respectively, for a 47% total response rate. For the January course, nine returned surveys were from dairy processing personnel; five were from the non-processing participants. For June, all 13 were from the processing sector.

All processing respondents stated that the training helped them better understand their HACCP programs, indicating that the course had potential impact on HACCP system improvement. The survey asked if they reviewed or modified existing documentation as a result of what was learned (Table 5). More than than 59% took or planned to take action on all items listed; flow diagrams, prerequisite programs, and the hazard analyses were modified the most suggesting, that new perspectives were learned in these areas.

Table 5.
Actions Taken Based on Knowledge Gained at the Trainings

	% of Respondents1
Program Area	No Action2	Reviewed, No Change2	Reviewed Modified2	Plan to Review2
Team Documentation	40.9	22.7	13.6	22.7
Product Descriptions	36.4	22.7	31.8	9.1
Flow Diagrams	27.3	18.2	40.9	13.6
Prerequisite Programs	27.3	18.2	40.9	13.6
Hazard Analysis	27.3	22.7	40.9	9.1
HACCP Plan Summary	27.3	27.3	18.2	27.3
Verification Validation	31.8	13.6	27.3	27.3
1 Respondents included 22 total with 9 from the January course and 13 from the June course. 2 Respondents marked specific action taken to review and/or modify their plant's HACCP documentation associated with listed program areas covered in the course.

Specific changes made to HACCP systems or other activities performed as a result of knowledge gained are summarized in Table 6. If no action was taken in 15 weeks post-training, the intention to do so was noted. Program areas marked most were related to the hazard analysis, including the use of the hazards guide, which was new to the participants. For PPs, >50% had or intended to change written PP summaries and monitoring practices. These responses suggest that participant's original HACCP documentation was lacking in some regard and reflect a positive impact of the training in these areas.

Table 6.
Changes in Plant HACCP Programs as a Result of Knowledge Gained

	% of Respondents1
HACCP Program Changes	Yes2	Intended3
Prerequisite Programs (PP)
Developed or modified brief written descriptions of PP	22.7	27.3
Changed how PP was monitored and/or documented	27.3	22.7
Added new specific PP to support their HACCP system	0.0	18.2
Hazard Analysis (HA)
Identified new hazards to be considered in the HA review	22.7	0.0
Used Hazards & Control Guide to review the hazard analyses	50.0	27.3
Reconsidered how identified hazards were evaluated in the HA	54.5	18.2
Critical Control Points (CCP)
Identified new CCPs	9.1	0.0
Eliminated existing CCPs based on what was learned	4.5	0.0
Reviewed / modified how pasteurization was monitored	22.7	9.1
Reviewed / modified how other identified CCPs were monitored	9.1	4.5
Verification
Reviewed / modified verification procedures for pasteurization CCP	22.7	9.1
Reviewed / modified verification procedures other identified CCPs	18.2	13.6
1 Respondents included 9 from the January course and 13 from the June course. 2 Yes - indicates that changes were made based on what was learned at the course. 3 Intended - indicates that participants intended to make changes based on what was learned at the course.

Under NCIMS-HACCP, pasteurization must be a CCP to control raw milk pathogens; other CCPs are operation dependent. Five respondents changed procedures for monitoring pasteurization by including periodic visual checks of continuous recorders. Two participants considered new CCPs, one for metal hazards with a process pump and the other for pathogen growth in yogurt with improper pH development.

A modified survey for non-processors was answered by five attendees. All five indicated that the training provided helped them better understand HACCP systems in a manner useful in their respective roles. One participant used the materials and instruction from the training to develop a HACCP guidance document for small dairy processors.

Summary

The results of the evaluation based on the Kirkpatrick model reported here showed that the two NCIMS-HIC HACCP training events provided for a participant reaction of overall satisfaction and a notable level of success in improving knowledge and improved use of HACCP concepts. Based on immediate reaction, all topic areas were considered useful overall, suggesting limited change in program focus was warranted. Constructive feedback was useful in determining that keeping program instruction at a steady pace and maintaining focus were deemed important; modification of the program and use of exercises may be helpful. Pre- and post-testing was effective in demonstrating improved understanding of key HACCP concepts, although learning varied with the individual. Use of the class notebook in answering the developed test questions did not appear necessary. The information provided resulted in changes in perspectives and actions (behavior) relative to the survey respondents and their HACCP systems. These results support the use of the NCIMS-HIC curriculum as an effective program for dairy HACCP training.

Acknowledgments

The project reported here was supported by the NY State Milk Promotion Order, dairy farmers committed to quality, safe dairy products. The author acknowledges Allen Sayler for his role as co-instructor; the NCIMS-HIC for their roles in HACCP program development; Nicole Martin, Cornell University, for statistical assistance; and Martin Wiedmann, Cornell University, for initiating the evaluation.

References

FDA (2003). Guidance for industry: standardized training curriculum for application of HACCP principles to juice processing; Final guidance. Retrieved from: http://www.fda.gov/Food/GuidanceComplianceRegulatoryInformation/GuidanceDocuments/Juice/ucm072586.htm 

FDA (2007). Hazards and control guide for dairy foods HACCP, guidance for processors. Version 1.1. 2006. Retrieved from: http://www.ncims.org/HACCP%20Documents.htm 

FDA (2009). Grade "A" pasteurized milk ordinance. U.S. Dept. of Health and Human Services, Public Health Service.

FDA (2011a). 21 CFR Part 120 - Hazard analysis and critical control point (HACCP) systems. Code of Federal Regulations, Food and Drug Administration, Department of Health and Human Services. Retrieved from: http://www.gpo.gov/fdsys 

FDA (2011b). 21 CFR Part 123 - Fish and fishery products. Code of Federal Regulations, Food and Drug Administration, Department of Health and Human Services. Retrieved from: http://www.gpo.gov/fdsys 

FDA (2011c). Fish and fishery products hazards and controls guidance. Retrieved from: http://www.fda.gov/Food/GuidanceComplianceRegulatoryInformation/GuidanceDocuments/Seafood/FishandFisheriesProductsHazardsandControlsGuide/default.htm 

FDA (2011d). Food safety modernization act. Public Law 111-353, January 4, 2011.

GFSI (2012). Global food safety initiative. Retrieved from: http://www.mygfsi.com 

Jackson, T. C., Harris, K. B., & Cross, H. R. (1996). International meat and poultry HACCP alliance: Comment. Food Control. 7:103-105.

Johnson, R. B. (2011). The Kirkpatrick model of training evaluation. University of South Alabama. Retrieved from: www.southalabama.edu/coe/bset/johnson/660lectures/Kirk1.doc 

Lo, Y. M., Fukushima, K., Rippen, T. E., Gdovin, S. L., & Hahm, T.-S. (2004). Active assessment for HACCP training: Integrating pedagogical reasoning with primary trait analysis. Journal of Extension [On-line], 42(6) Article 6IAW4. Available at: http://www.joe.org/joe/2004december/iw4.php 

NACMCF (1998). Hazard analysis and critical control point principles and application guidelines. National Advisory Committee on the Microbiological Criteria for Foods. J. Food Prot. 61:1246-1259.

Stup, R. (2003) Program evaluation: Use it to demonstrate value to potential clients. Journal of Extension [On-line] 41(4) Article 4COM1. Available at: http://www.joe.org/joe/2003august/comm1.php 

USDA (2011). 9 CFR Part 417- Hazard analysis and critical control point (HACCP) systems. Code of Federal Regulations, Food Safety & Inspection Service, U.S. Department of Agriculture. Retrieved from: http://www.gpo.gov/fdsys 

Wallace, C. A., & Powell, S. C. (2005). Post-training assessment of HACCP knowledge: its use as a predictor of effective HACCP development, implementation and maintenance in food manufacturing. British Food J. 107:743-759.