August 2013 // Volume 51 // Number 4 // Research In Brief // v51-4rb7
Evaluation of HACCP Training Under the Grade "A" Dairy HACCP Core Curriculum
Learning outcomes of two training programs using the National Conference on Interstate Milk Shipments (NCIMS) HACCP core curriculum were evaluated using a post-program questionnaire to assess participant reaction; a pre-/post-test to measure learning; and a follow-up survey to determine utilization of training information. Overall responses indicated that program materials and instruction were of value in meeting HACCP training needs; that overall participant knowledge improved; and knowledge gained was used in review and modification of dairy plant HACCP systems by survey respondents. The NCIMS core curriculum as a standardized HACCP training program meets the needs of the dairy industry.
The application of Hazard Analysis Critical Control Point (HACCP) principles developed by the National Advisory Committee on Microbiological Criteria for Foods (NACMCF, 1998) has become standard in the food industry as a means to minimize product safety risks. While adoption of HACCP principles is mostly voluntary, HACCP systems are required for certain foods, including meat (USDA, 2011), juice (FDA, 2011a), and seafood (FDA, 2011b). Grade "A" dairy plants can be regulated under a voluntary HACCP auditing system as an alternative to the traditional inspection-based system outlined in the Pasteurized Milk Ordinance (PMO; FDA, 2009). This voluntary program was developed under the National Conference on Interstate Milk Shipments (NCIMS-HACCP). Dairy plants under both programs are routinely audited and/or inspected to ensure compliance.
To develop HACCP programs, detailed training is needed. While most formal training is based on NACMCF guidelines, curriculum standardization has not been well defined (Wallace & Powell, 2005). Exceptions include training developed for regulated HACCP programs (Jackson, Harris & Cross, 1996; FDA, 2011c; FDA, 2003; FDA, 2007). Under the NCIMS, the HACCP Implementation Committee (NCIMS-HIC) developed the dairy HACCP core curriculum, which provides instruction in the application of HACCP principles and regulations for dairies enrolled in NCIMS-HACCP.
While developed for NCIMS-HACCP, NCIMS-HIC training should be applicable to all dairy operations and could serve as a model program. However, procedures evaluating training outcomes of the program have not been fully used, and there is a need to further measure success. The Kirkpatrick evaluation model provides a basic process that can be easily applied to training events (Johnson, 2011; Stup, 2003), with four levels measuring reaction, learning, behavior, and ultimate outcomes. This article describes the evaluation of two NCIMS-HIC trainings based on the first three levels. The goal was to determine the impact of the NCIMS-HIC dairy core curriculum on the participants as presented and to assess its value as an industry training program.
Training Programs and Materials
Two HACCP workshops were offered in January and June, 2011, with 29 and 30 attendees, respectively. One was offered through the Dairy Extension group at Cornell University. The other was sponsored by a local food association. Both programs were 2.5 days and had the same two instructors (NCIMS-HIC members). The course outline is shown in Table 1.
|1.||Introduction: NACMCF Overview & Definitions|
|2.||Dairy/Food Safety Overview: Hazards & Controls|
Preliminary Steps to HACCP
Team Exercise: Team Documentation, Product Descriptions & Flow Diagrams
Prerequisite Programs (PP) & Good Manufacturing Practices (GMPs)
Team Exercise: PP Development & Implementation
HACCP Principles 1 & 2: Conducting a Hazard Analysis/Determining Critical Control Points (CCP)
Team Exercise: Hazard Analysis & CCP determinations
|6.||HACCP Principle 3 & 4: Establish Critical Limits (CL) & Monitoring Procedures|
|7.||HACCP Principle 5: Establish Corrective Actions for CL Deviations|
|8.||HACCP Principle 6: Establish Verification Procedures|
|9.||HACCP Principle 7: Establish Recording Keeping & Documentation Procedures|
Model HACCP Plans: NCIMS Pasteurization CCP (PMO) & Others
Team Exercise: HACCP Plan Development
|11.||NCIMS-HACCP Program and FDA Juice HACCP Overview|
Training Program Summary:
Class Exercise: Verification/Validation
Throughout the training, steps for HACCP development and implementation were covered along with NCIMS and Juice HACCP regulatory requirements. Juice HACCP was included because many dairies process juice. Applicable HACCP-based regulations of the new Food Safety Modernization Act (FDA, 2011d) were also included as relevant. The trainings provided in-depth coverage of PPs; hazard analyses and CCP determinations; and verification/validation. Attendees participated in team exercises, with four breakout groups in January and 5 in June. Notebooks with printouts of all lecture slides were provided. Additional materials included the NACMCF guideline, NCIMS and Juice HACCP regulations, GMPs (21CFR110), PP examples, and the NCIMS-HIC Dairy Hazards and Controls Guide (FDA, 2007). HACCP forms provided included standardized forms for product descriptions, hazard analyses, and HACCP plans; tailored forms developed for NCIMS-HIC training included forms for team documentation, PPs, CCP deviations, verification/validation activities, and program records.
A one-page evaluation form was developed to assess participant's perceived value of topics covered and overall reaction to the program, materials, and presenters. It was distributed at the beginning and collected at the end of each course. Participants ranked the usefulness of each program topic on a 4-point scale. Overall program quality; the quality of reference materials; the value of class interactions; and instructor skills were ranked on a 5-point scale.
To measure assimilation of knowledge, a test was distributed to participants at the beginning (pre-test) and again at the end (post-test) of each course; pre- and post-tests were the same. Questions focusing on key HACCP concepts were developed by NCIMS-HIC members who administer NCIMS-HACCP program development. For January, the test included 20 short answer and multiple-choice questions with 48 total responses. The goal was for completion within 25 minutes, but it took longer. For June, the test was shortened to 38 responses to fit the desired time. January participants were allowed to use notebooks for the post-test only, emulating its use as a reference manual. June participants took both tests without the notebook because the January group appeared to spend too much time searching; they finished in 25 minutes. A paired t-test was used to determine differences between pre- and post-test scores (JMP, Version 7.0; SAS Institute, Cary, NC).
To determine behavioral changes or how knowledge gained was applied at the plant level, a follow-up survey was sent to all participants 15 weeks after the trainings. Survey questions were designed to determine if perspectives had changed or if specific modifications were made or considered in their HACCP systems based on what was learned.
Results and Discussion
Most workshop participants were dairy employees with titles ranging from technician to management. In January there were two NCIMS-HACCP regulatory auditors, two consultants, and two supplier representatives. June attendees were primarily from local dairies. One auditor and three participants from nondairy companies also attended. Ten were from four milk plants that also processed juice. The trainings represented the first formal course for a majority (68%). Most processing attendees (86%) were from companies where HACCP programs were well established. Two participants commented that their "well established" programs were also a "work in progress," emphasizing that a HACCP system is an evolving process.
Program Evaluation Form
Participant responses to the program content are summarized in Table 2. For both courses, most lecture topics were marked as "very-useful" or "useful" by a majority, with average scores ≥3.5 on a 4.0 scale. Lower mean scores for juice regulation reflected the limited enrollment of companies processing juice. Program quality, materials, and speakers were ranked above average or better by most, with mean scores ≥ 4.4 out of 5.0. June "interactions" ranked higher, perhaps due to "local" dynamics. Several participants were involved with meeting Global Food Safety Initiative (GFSI, 2012) requirements, resulting in discussions related to GFSI-HACCP.
|Program Area||Average Rank January Course3||Average Rank June Course3|
|Lecture Topic Areas (4 point scale1)|
|Introduction to HACCP||3.5||3.4|
|Hazards in dairy products and other foods||3.7||3.3|
|Preliminary steps & prerequisite programs||3.6||3.5|
|Seven principles of HACCP||3.7||3.6|
|NCIMS Voluntary HACCP overview||3.8||3.5|
|Juice HACCP regulation overview||3.0||2.8|
|HACCP auditing basics||3.5||3.6|
|Group Exercise - preliminary steps||3.5||3.5|
|Group Exercise - prerequisite programs||3.6||3.7|
|Group Exercise - hazard analysis||3.7||3.7|
|Group Exercise - HACCP plan summary||3.6||3.7|
|Class Exercise - verification/validation||3.7||3.7|
|Over Program Evaluation (5 point scale2)|
|Overall program quality||4.5||4.5|
|Meeting objectives for attending||4.4||4.5|
|Provided program materials||4.4||4.5|
|Interactions with other participants||4.2||4.6|
|Overall knowledge/presentation skills of Speaker A||4.8||4.8|
|Overall knowledge/presentation skills of Speaker B||4.8||4.8|
1 Rankings based on 4 point scale: "very-useful" = 4; "useful" = 3; "little-use" = 2; and "not-useful" = 1.
2 Rankings based on a five point scale: "excellent" = 5; "above-average" = 4; "average" = 3; "fair-below average" = 2; and "poor" = 1.
3 Average scores based on participants attending the January (n=27) and the June trainings (n=30)
Participants were asked to list new concepts they would take away from the course and potentially apply to their operation; nine from each course provided answers (Table 3). Verification and validation related topics were the most frequent responses, suggesting the importance of these critical HACCP concepts was better realized as a result of the training. Other take-home concepts listed by more than one respondent were related to PPs and alternative CCP monitoring. NCIMS-HIC training devotes substantial time covering PPs with emphasis on their use in hazard analyses. Alternative pasteurization monitoring was presented as a new topic to most.
|1||Concept of "continuous" vs. periodic monitoring of a CCP [CCP monitoring]; We do not perform validation as defined in course [verification/validation]|
|2||Need for review of current HACCP plan [verification]|
|3||How to apply HACCP to small scale|
|4||Additional calibration requirements of all recording devices [verification]|
|5||HACCP regulations and hazards for dairy and juice|
|6||Chart record and data review [verification]|
|7||Incoming materials/receiving to be added to prerequisites [PP] Review of flow charts [verification]|
|8||Continuous monitoring change to a different frequency [CCP monitoring]; Review prerequisite programs using the chart provided in class [PP]|
|9||Example forms excellent|
|1||Prerequisite program/HACCP relationships - idea that PP could nullify a CCP [PP]; More comfortable with concepts of verification and validation [verification]|
|2||Developing a HACCP program|
|3||Learning how to write a HACCP plan; New ideas/concepts from different people - thinking outside the box|
|4||Brought lots of loose ends together|
|5||Changing a third of the members of the HACCP team every 2 yrs|
|6||FSMA & HACCP - consideration of radiation as a hazard; Consumer complaints as part of verification [verification]|
|7||Sources of validation information helpful [validation]; Some new materials to help update existing program|
|8||Looking into suppliers e.g., allergens [verification]; HA update to simplify referring to PP [PP]|
|9||Breaking out different processing units to consider different factors; Better understanding of how SQF [GFSI scheme] & HACCP work together|
1 For the January course, 9 out 27 provided answers in course "reaction" evaluation. Repeated HACCP concept areas are listed in brackets for appropriate responses.
2 For the June course, 9 out of 30 provided answers in course "reaction" evaluation. Repeated HACCP concept areas are listed in brackets for appropriate responses.
When asked for suggestions for improvement or additional comments, only two participants offered their thoughts on the January program. Several participants in June provided comments. While a number were positive, several focused on training pace, describing it as "rushed"; some related this to over-long exercises. The instructors acknowledged this perspective. More time was spent on exercises in June. Additional GFSI discussion described above likely contributed. While the January evaluation suggested little program change was warranted, June comments indicated otherwise. Because the training has a full program, keeping it at a consistent, acceptable pace can be a challenge. It is important for instructors to be able to adjust depending on class dynamics. Solutions might include condensing lectures; modifying exercises (e.g., rotate groups presenting); and maintaining focus on HACCP, limiting discussion on related topics. Expanding the course to 3 days may help, but might not be practical for many participants.
Pre-Test and Post-Test Learning Assessment
Pre- and post-test summary scores for HACCP training programs are in Table 4. Mean scores for the post-tests were significantly higher than pre-test scores for January, t(26) = -13.02, p ≤ 0.0001 and June, t(26) = -12.13, p ≤ 0.0001, indicating noteworthy levels of knowledge gain. Notebook use for the post-test in January did not result in higher scores.
|Course /Parameter||Pre-Test Scores (%)||Post-Test Scores (%)||Post- Minus Pre-Test Scores|
1 January course had 27 participants. Pre- and post-tests were the same 20 questions with 48 possible correct answers. Scores are based on correct responses out of 48.
2 June course had 30 participants. Pre- and post-tests were the same 20 questions with 38 possible correct answers. Scores are based on correct responses out of 38.
a,b Mean Scores in the same row with different letters were significantly different (p ≤ 0.0001)
Individual score improvement ranged from 12.5% to 77.1% in January and from 2.6% to 68.4% in June, demonstrating that pre- and post-course knowledge and/or testing abilities varied among participants. For example in January, the smallest increase was from one person who had the lowest pre- (4.2%) and post-test (16.7%) scores. The attendee with the greatest improvement scored well below the pre-test mean (14.6%), yet scored greater than 90% on the post-test. The highest pre-test (70.8%) and post-test (93.8%) scores were from one participant.
In addition to testing, team exercises provide participants with an opportunity to demonstrate applied knowledge and are required as part of the NCIMS-HIC curriculum. Based on product processing scenarios, participants build components of a HACCP system to present in class. While skills were not formally measured, most groups appeared to grasp HACCP tools and concepts. As with pre-/post-testing, individuals differ in previous knowledge and abilities to learn and contribute to the "team." In general, accessing student performance in training activities is challenging. Lo, Fukushima, Rippen, Gdovin, and Hahm (2004) proposed a primary trait analysis based on comprehension, transformation of information to perform the task, implementation, and outcome reflection that could be used for this purpose.
Follow-up surveys were returned by 14 and 13 attendees from the January and June courses, respectively, for a 47% total response rate. For the January course, nine returned surveys were from dairy processing personnel; five were from the non-processing participants. For June, all 13 were from the processing sector.
All processing respondents stated that the training helped them better understand their HACCP programs, indicating that the course had potential impact on HACCP system improvement. The survey asked if they reviewed or modified existing documentation as a result of what was learned (Table 5). More than than 59% took or planned to take action on all items listed; flow diagrams, prerequisite programs, and the hazard analyses were modified the most suggesting, that new perspectives were learned in these areas.
|% of Respondents1|
|Program Area||No Action2||Reviewed, No Change2||Reviewed Modified2||Plan to Review2|
|HACCP Plan Summary||27.3||27.3||18.2||27.3|
1 Respondents included 22 total with 9 from the January course and 13 from the June course.
2 Respondents marked specific action taken to review and/or modify their plant's HACCP documentation associated with listed program areas covered in the course.
Specific changes made to HACCP systems or other activities performed as a result of knowledge gained are summarized in Table 6. If no action was taken in 15 weeks post-training, the intention to do so was noted. Program areas marked most were related to the hazard analysis, including the use of the hazards guide, which was new to the participants. For PPs, >50% had or intended to change written PP summaries and monitoring practices. These responses suggest that participant's original HACCP documentation was lacking in some regard and reflect a positive impact of the training in these areas.
|% of Respondents1|
|HACCP Program Changes||Yes2||Intended3|
|Prerequisite Programs (PP)|
|Developed or modified brief written descriptions of PP||22.7||27.3|
|Changed how PP was monitored and/or documented||27.3||22.7|
|Added new specific PP to support their HACCP system||0.0||18.2|
|Hazard Analysis (HA)|
|Identified new hazards to be considered in the HA review||22.7||0.0|
|Used Hazards & Control Guide to review the hazard analyses||50.0||27.3|
|Reconsidered how identified hazards were evaluated in the HA||54.5||18.2|
|Critical Control Points (CCP)|
|Identified new CCPs||9.1||0.0|
|Eliminated existing CCPs based on what was learned||4.5||0.0|
|Reviewed / modified how pasteurization was monitored||22.7||9.1|
|Reviewed / modified how other identified CCPs were monitored||9.1||4.5|
|Reviewed / modified verification procedures for pasteurization CCP||22.7||9.1|
|Reviewed / modified verification procedures other identified CCPs||18.2||13.6|
1 Respondents included 9 from the January course and 13 from the June course.
2 Yes - indicates that changes were made based on what was learned at the course.
3 Intended - indicates that participants intended to make changes based on what was learned at the course.
Under NCIMS-HACCP, pasteurization must be a CCP to control raw milk pathogens; other CCPs are operation dependent. Five respondents changed procedures for monitoring pasteurization by including periodic visual checks of continuous recorders. Two participants considered new CCPs, one for metal hazards with a process pump and the other for pathogen growth in yogurt with improper pH development.
A modified survey for non-processors was answered by five attendees. All five indicated that the training provided helped them better understand HACCP systems in a manner useful in their respective roles. One participant used the materials and instruction from the training to develop a HACCP guidance document for small dairy processors.
The results of the evaluation based on the Kirkpatrick model reported here showed that the two NCIMS-HIC HACCP training events provided for a participant reaction of overall satisfaction and a notable level of success in improving knowledge and improved use of HACCP concepts. Based on immediate reaction, all topic areas were considered useful overall, suggesting limited change in program focus was warranted. Constructive feedback was useful in determining that keeping program instruction at a steady pace and maintaining focus were deemed important; modification of the program and use of exercises may be helpful. Pre- and post-testing was effective in demonstrating improved understanding of key HACCP concepts, although learning varied with the individual. Use of the class notebook in answering the developed test questions did not appear necessary. The information provided resulted in changes in perspectives and actions (behavior) relative to the survey respondents and their HACCP systems. These results support the use of the NCIMS-HIC curriculum as an effective program for dairy HACCP training.
The project reported here was supported by the NY State Milk Promotion Order, dairy farmers committed to quality, safe dairy products. The author acknowledges Allen Sayler for his role as co-instructor; the NCIMS-HIC for their roles in HACCP program development; Nicole Martin, Cornell University, for statistical assistance; and Martin Wiedmann, Cornell University, for initiating the evaluation.
FDA (2003). Guidance for industry: standardized training curriculum for application of HACCP principles to juice processing; Final guidance. Retrieved from: http://www.fda.gov/Food/GuidanceComplianceRegulatoryInformation/GuidanceDocuments/Juice/ucm072586.htm
FDA (2007). Hazards and control guide for dairy foods HACCP, guidance for processors. Version 1.1. 2006. Retrieved from: http://www.ncims.org/HACCP%20Documents.htm
FDA (2009). Grade "A" pasteurized milk ordinance. U.S. Dept. of Health and Human Services, Public Health Service.
FDA (2011a). 21 CFR Part 120 - Hazard analysis and critical control point (HACCP) systems. Code of Federal Regulations, Food and Drug Administration, Department of Health and Human Services. Retrieved from: http://www.gpo.gov/fdsys
FDA (2011b). 21 CFR Part 123 - Fish and fishery products. Code of Federal Regulations, Food and Drug Administration, Department of Health and Human Services. Retrieved from: http://www.gpo.gov/fdsys
FDA (2011c). Fish and fishery products hazards and controls guidance. Retrieved from: http://www.fda.gov/Food/GuidanceComplianceRegulatoryInformation/GuidanceDocuments/Seafood/FishandFisheriesProductsHazardsandControlsGuide/default.htm
FDA (2011d). Food safety modernization act. Public Law 111-353, January 4, 2011.
GFSI (2012). Global food safety initiative. Retrieved from: http://www.mygfsi.com
Jackson, T. C., Harris, K. B., & Cross, H. R. (1996). International meat and poultry HACCP alliance: Comment. Food Control. 7:103-105.
Johnson, R. B. (2011). The Kirkpatrick model of training evaluation. University of South Alabama. Retrieved from: www.southalabama.edu/coe/bset/johnson/660lectures/Kirk1.doc
Lo, Y. M., Fukushima, K., Rippen, T. E., Gdovin, S. L., & Hahm, T.-S. (2004). Active assessment for HACCP training: Integrating pedagogical reasoning with primary trait analysis. Journal of Extension [On-line], 42(6) Article 6IAW4. Available at: http://www.joe.org/joe/2004december/iw4.php
NACMCF (1998). Hazard analysis and critical control point principles and application guidelines. National Advisory Committee on the Microbiological Criteria for Foods. J. Food Prot. 61:1246-1259.
Stup, R. (2003) Program evaluation: Use it to demonstrate value to potential clients. Journal of Extension [On-line] 41(4) Article 4COM1. Available at: http://www.joe.org/joe/2003august/comm1.php
USDA (2011). 9 CFR Part 417- Hazard analysis and critical control point (HACCP) systems. Code of Federal Regulations, Food Safety & Inspection Service, U.S. Department of Agriculture. Retrieved from: http://www.gpo.gov/fdsys
Wallace, C. A., & Powell, S. C. (2005). Post-training assessment of HACCP knowledge: its use as a predictor of effective HACCP development, implementation and maintenance in food manufacturing. British Food J. 107:743-759.