Formative Evaluation of EFNEP Curriculum: Ensuring the Eating Smart • Being Active Curriculum Is Theory Based

Introduction

Nutrition education can be defined as "any set of learning experiences designed to facilitate the voluntary adoption of eating and other nutrition-related behaviors conducive to health and well-being" (Contento et al., 1995). Nutrition education programs should include communication and educational strategies to build knowledge and motivation, behavioral change strategies to guide desired outcomes, environmental components to eliminate obstacles and enhance behavior change, and community activation (Contento et al., 1995; Higgins & Barkley, 2003). Programs designed to change behavior are most successful when rooted in appropriate behavior change theories or combination of theories (Achterberg & Miller, 2004; Baranowski, Cullen, & Baranowski, 1999; Bird & McClelland, 2010; Brownell & Cohen, 1995; Contento, Randell, & Basch, 2002; Franz, 2007; Rayner, 2003).

The mission of the Expanded Food and Nutrition Education Program (EFNEP) is to improve nutrition knowledge, skills, attitudes, and behaviors of low-income families (Burney & Haughton, 2002; USDA-NIFA, 2009). EFNEP employs experiential learning through curricula designed to help participants make healthy food and lifestyle choices (USDA-NIFA, 2009).

The lessons in EFNEP curricula are based on current Dietary Guidelines for Americans (DGA), which are updated every 5 years (US Dept. Health and Human Services, 2010). The DGA released in 2005 included many changes as compared to previous versions of the DGA. Major recommendation changes included increased emphasis on whole grain consumption, choosing fruits and vegetables in a variety of colors, the inclusion of physical activity and expanded food safety information, and shifting from using "servings" to cup and ounce equivalents (US Dept. Health and Human Services, 2010).

The release of the 2005 DGA required that nutrition-based curricula for programs such as EFNEP and the Supplemental Nutrition Assistance Program- Education (SNAP-Ed) be updated. Rather than updating existing curricula, the EFNEP in Colorado and California used the opportunity to partner and create a new curriculum for their program participants: Eating Smart • Being Active (ESBA). Developers decided this new curriculum would incorporate both the Adult Learning Theory (ALT) (Knowles, 1984) and Social Cognitive Theory (SCT) (Bandura, 1986) and be appropriate for use by Extension paraprofessionals. SCT is often used in interventions with low-income audiences and serves as the basis for the desired behavior change outcomes, whereas ALT was used to define the andragogical approach used in both the classroom setting and the written materials and instructions for the paraprofessionals.

Nationally, EFNEP uses a paraprofessional model. According to Norris and Baker, paraprofessionals are "usually hired, not for their degrees or knowledge of subject matter, but for their life experiences, cultural, social, and economic backgrounds, and their ability to relate to clients." This ability to relate to clients is termed "empathy" and has been a significant reason the paraprofessional model has been so successful in EFNEP (Norris & Baker, 1999).

During the time Colorado and California EFNEP coordinators were starting to develop the new curriculum, representatives from more than 25 states expressed interest in the curriculum; these representatives indicated that their states were not going to develop new materials but intended to use this new curriculum. Given this widespread interest, the national implications for Extension were significant. Thus, curriculum developers wanted to take the necessary steps to ensure that the content was accurate and the potential for behavior change was strong to improve the likelihood of positive participant outcomes.

Curriculum Development

The process of developing a new curriculum for a nutrition education program can be a complex, often laborious task to ensure that the curriculum is designed to achieve the intended outcomes, keeping in mind the needs of the intended audience and the method of delivery best suited to the audience (Higgins & Barkley, 2003). Research suggests that nutrition education programs with the goal of behavior change are most successful when rooted in appropriate behavior change or learning theories (Rayner, 2003; Achterberg & Miller, 2004; Brownell & Cohen, 1995; Contento, 2007; Baranowski, Cullen, & Baranowski, 1999). Although many of the most widely used theories in nutrition education involve similar constructs, no one theoretical model is used as a standard in nutrition education to impact behavior; in fact, some experts advocate the use of a polytheoretical model (Achterberg & Miller, 2004).

A limitation of nutrition education program development is that no program can provide all the necessary information an audience needs to make healthful nutrition-related choices (Contento, Randell, & Basch, 2002). Rather, curriculum developers must prioritize the key concepts to be addressed, taking into account the amount of time available, attention span of the audience, and other factors (Contento et al., 2002).

Ideally, when developing any new program or education materials, a detailed formative evaluation should be completed to improve the chances that the program/materials will be accepted and effective (Windsor, Clark, Boyd, & Goodman, 2003). A good formative evaluation has numerous steps, including determining if the materials are acceptable to the target audience and educators, gathering feedback from experts, confirming that program objectives are appropriate, and conducting a pilot study.

The process of developing Eating Smart • Being Active took several years. The steps of developing the curriculum are detailed below and include many typical formative evaluation steps (Figure 1):

Figure 1.
Evaluation of the Eating Smart • Being Active Curriculum

1. EFNEP staff in Colorado and California (heretofore referred to as the authors) started by polling EFNEP educators in the states in which the curriculum was to be piloted (California, Colorado, Iowa, and South Carolina) about their favorite recipes, activities and key topics from existing curricula.
2. Next, authors organized the activities, recipes, and key topics into content areas that could support potential lesson themes.
3. The authors, utilizing years of program experience, decided that the curriculum would be designed with the following parameters in mind:
1. The core curriculum would contain no more than eight lessons. This decision was based upon how many lessons people are typically willing to attend in EFNEP, and the national average of lessons taught in an EFNEP class series (USDA-NIFA, 2012). Authors planned on writing additional optional lessons specific to maternal, infant, toddler, and preschool nutrition topics so the number of core lessons was limited to eight to allow for flexibility in implementation.
2. The eight core lessons would be taught consecutively so that concepts could build throughout the series.
3. All lessons would be learner-centered (ALT), with hands-on activities to reinforce concepts.
4. Physical activity, food safety, and label reading would be introduced early in the series of lessons and incorporated throughout each lesson thereafter to support these important yet complicated topics while limiting the number of lessons.
5. Nutrition and physical activity would be represented as two equal parts of a healthy lifestyle, with both concepts appearing in every lesson.
6. Each lesson would include a food preparation activity.
7. Participant materials would be full-color, include photographs instead of clipart, and represent people of different ages, ethnicities, and body sizes.
8. Participant materials would be written at a 6^th to 8^th grade reading level.
4. Authors then determined lesson titles and goals and objectives of each lesson. The main messages of both the DGA and MyPyramid were used to create the lesson titles:
• Get Moving!
• Plan/Shop/$ave
• Vary Your Veggies...Focus on Fruit (The title of this lesson was subsequently changed to Fruits & Veggies: Half Your Plate with the release of the 2010 DGA and MyPlate)
• Make Half Your Grains Whole
• Build Strong Bones
• Go Lean With Protein
• Make a Change
• Celebrate! Eat Smart and Be Active
5. The authors worked together to draft eight research-based, learner-centered lessons reflecting the recommendations from the 2005 DGA (the lessons have been updated for the 2010 DGA and MyPlate).
6. Once the lesson content was close to finalization, authors worked with a graphic designer to develop lesson plans, handouts, worksheets, and visuals, and take pictures to reinforce concepts in the lessons.

Piloting the Curriculum (Implementation Testing)

Feedback from curriculum users helps drive curriculum development (Hammerschmidt, Murphy, Youatt, Sawyer, & Andrews, 1994). Four states (California, Colorado, Iowa, and South Carolina) participated in a 6-month pilot of the curriculum. Sites were chosen based on geography, participant demographics, and program delivery (group vs. individual instruction). EFNEP coordinators provided feedback on lesson content, materials, and recipes. Paraprofessionals were given a journal to provide written feedback after teaching each lesson. They were asked specifically to think about the following when giving feedback.

1. When teaching the lesson, did you have everything you needed to execute the lesson?
2. How does the lesson meet/not meet your needs?
3. What did you like/dislike about the lesson?

Authors separated the feedback from pilot state coordinators and paraprofessionals by lesson and looked for themes. The pilot feedback, along with the results from the expert review, helped guide curriculum revisions.

Theory Adherence Evaluation

Newly developed curricula should undergo a systematic review ensuring effectiveness (Coleman, Byrd-Bredbenner, Baker, & Bowen, 2011). The purpose of the formative evaluation was to assess the content validity of Eating Smart • Being Active relative to the 2005 DGA and the curriculum's appropriate application of SCT and ALT. To ensure theory tenets and content accuracy were addressed, two faculty members at Clemson University who were familiar with EFNEP, but otherwise not involved with the curriculum development, created an assessment tool to be used with each lesson.

The initial assessment tool <http://www.ext.colostate.edu/esba/curr-assess-tool.doc> was based on literature and research that used SCT or ALT as a basis for intervention design targeting the low-literate learner (Gatson & Daniels, 1988; Glanz, Lewis, & Rimer, 2002; Murphy et al., 1996, National Cancer Institute, 2011). Seven tenets of SCT (expectations, self-control, environment, overcoming emotional responses, use of reinforcements, self-efficacy, and observational learning) and five tenets of ALT (learning is geared toward participant's lifestyle and learning style, learning is self-directed, limiting the number of new concepts, learning builds on prior experience, and safe learning environment) were incorporated in the comprehensive, 96-item assessment tool, which also evaluated overall content, appearance, ease of use, and appropriateness for EFNEP.

The assessment tool was reviewed for content and face validity by three community nutrition experts at Colorado State University and the University of California, Davis. Inter-rater reliability was confirmed by two independent reviewers at Colorado State University using the assessment tool with three EFNEP lessons from another curriculum. The cover letters to the expert panel and the assessment tools were approved by the Colorado State University Human Research Committee (HRC).

Reviewing the Curriculum

EFNEP professionals and other nutrition and food safety leaders involved in Extension programs from Colorado, Indiana, Missouri, Montana, and Nevada served as a panel of experts for this review process. None of the expert panel members was involved in the curriculum development process or the pilot.

Reviewers used the assessment tool to review each of the eight lessons individually (96-statement assessment per lesson per reviewer), rated their level of agreement with each statement using a 5-point Likert scale (strongly agree [5] to strongly disagree [1]), and supported their choice with hand-written comments. A total of 40 separate assessments (five reviewers X eight lessons) were compiled, tabulated and grouped according to the applicable theory.

After curriculum revision based on the first review and the pilot, a second assessment was conducted by the same panel of experts to ensure all theory tenets remained and that those tenets, which scored poorly, had been improved. The tool in the follow-up review was shortened (38 items) to focus only on tenets of SCT and ALT (content assessment was removed for the second review). The same reviewers again used separate assessments for each of the eight lessons.

Data Analysis

Statistical analyses were conducted to determine consistency within the reviewers and that lower scoring items had improved through curriculum revisions at the two time points. Cronbach's α was computed for theory tenets with multiple questions to determine the internal consistency (reliability) of reviewer ratings. Cronbach's α was run for the ALT tenet learning is geared toward participant's lifestyle (0.85), and SCT constructs expectations (0.69) and self-control (0.76). Data were compiled and pairwise comparison of the means (t-test) from initial assessment to follow-up was computed.

Results

As seen in Figures 2 and 3 and Tables 1 and 2, the initial assessment found that the piloted version of the Eating Smart • Being Active curriculum adequately incorporated nearly all of the major tenets of the two theories. The range of reviewers' means were 2.4-4.8 (ALT) and 1.9-4.8 (SCT) for the initial review and 3.0-4.8 (ALT) and 2.8-4.8 (SCT) for the follow-up review.

As Table 1 depicts, the ALT tenets that consistently scored high (> 3.8) in both the initial assessment and follow-up were gearing learning toward the participant's lifestyle-learning style and learning is self-directed.

The tenet that scored the lowest in both reviews was the SCT concept of emotional coping responses when faced with obstacles to the desired behavior. The initial review scores mean was 2.3 (range: 1.9-3.5), and at follow-up, the mean for each lesson improved to 3.1 (range: 2.8-3.3).

Statistically significant improvements in some lessons were seen in some ALT (Table 1) and several SCT constructs (Table 2). As noted in Table 2, the SCT constructs of self-control, expectations and the use of reinforcements also scored high (> 3.9) in both assessments.

Figure 2.
Comparison of Mean for ALT "Learning Environment Is Safe" According to Lesson

Figure 3.
Comparison of Means for SCT "Self Efficacy" According to Lesson

Table 1.
Rating of Adult Learning Theory (ALT) Tenets in Curriculum

ALT Construct/Tenet	Lessons with significant improvement between initial and follow-up review	Range of means – initial review	Range of means – follow up review
Learning geared toward participant's lifestyle/learning style (scale)	-	4.4 to 4.8	4.6 to 4.7a
Learning is self-directed	-	3.8 to 4.4	4.2 to 4.6a
Only 1-2 new concepts introduced at a time	Make a Change	2.4 to 4.4	3.0 to 4.6
Learning builds on prior experience	Make Half Your Grains Whole	3.4 to 4.7	4.8
Safe learning environment	Build Strong Bones, Go Lean With Protein, Make Half Your Grains Whole, Celebrate! Eat Smart and Be Active	2.8 to 4.5	4.8
aWhile not statistically significant, improvements were observed in these theory tenets.

Table 2.
Rating of Social Cognitive Theory in Curriculum

SCT Construct/Tenet	Lessons with significant improvement between initial and follow-up review	Range of means – initial review	Range of means – follow up review
Expectations (scale)	-	3.9 to 4.3	4.3 to 4.8a
Self-control (scale)	Get Moving	4.0 to 4.5	4.6 to 4.8
Environment	-	3.0 to 4.3	3.6 to 4.4a
Emotional coping responses	Get Moving, Celebrate! Eat Smart and Be Active	1.9 to 3.5	2.8 to 3.3
Reinforcements	-	4.0 to 4.8	3.6 to 4.8a
Self-efficacy	Veggies/Fruit Make Half Your Grains Whole, Plan/Shop/Save	2.7 to 3.5	4.0 to 4.5
Observational learning	Get Moving	3.5 to 4.2	4.0 To 4.6
aWhile not statistically significant, improvements were observed in these theory tenets.

There was a trend toward improvement of the tenets in each lesson (Tables 1 and 2). Many of the tenets scored high in the initial assessment, and while improvements did occur, they were not statistically significant. The ALT tenet of creating a safe learning environment improved significantly in four lessons, and the SCT construct of promoting self-efficacy improved in three lessons.

The Final Product

Eating Smart • Being Active has eight core lessons, each with accompanying materials (visuals, worksheets, handouts and reinforcement items) to help reinforce new concepts. Worksheets provide opportunities for hands-on activities. Handouts distributed near the end of each lesson serve as a review of the major concepts. Take-home reinforcement items, e.g., water bottles, grocery list pads, produce brushes and food thermometers, are given to participants to help reinforce and practice key concepts.

Discussion

Multiple theories are often employed in an intervention (Achterberg & Miller, 2004), as was the case with Eating Smart • Being Active. When using multiple theories, curricula designers have the responsibility of ensuring that the theories are used correctly. Given the scope of each lesson, the need to keep participants adequately engaged, and the numerous behavior change theory tenets to be addressed, it was understandable that some tenets could be underrepresented. Formative evaluation processes helped ensure that theory tenets would not be inadvertently be omitted.

To paraphrase Contento (2007), no educational program, including multi-session education programs, can provide all the information necessary for making informed choices and decisions on behavior change. The context of a group setting is not as conducive to addressing issues that are largely individual and require customization, such as emotional barriers to behavior change. Rather, the lessons create a safe learning environment and stress self-efficacy, a powerful factor for overcoming barriers.

Conducting a theory assessment before finalization is a valuable exercise in curriculum development. Nutrition education experts agree that programs rooted in behavior change theory are more likely to achieve success than those in which theory is either loosely or not applied (Contento, et al., 2002; Achterberg & Miller, 2004; Bird & McClelland, 2010). However, few programs use external reviewers to confirm adherence to theory. Conducting the assessment in the development stage increases the likelihood that the curriculum will be effective and reduces the risk of costly revisions once the curriculum is in circulation.

The use of an expert panel strengthens the assessment results. The professionals serving as reviewers are experts in community nutrition. Additionally, the written feedback reviewers offered, particularly in the initial review, provided rich qualitative data that curriculum developers used to enhance the lessons.

Given that the tool was tested for validity and inter-rater reliability and that reviews were conducted by peers familiar with the intended audience of the curriculum, the researchers are confident in the quality of the assessment and feedback. The relatively narrow ranges seen in reviewers' assessments of the theory tenets suggest consistency both across and within reviewers.

Limitations

The extensiveness of the 96-item assessment tool in the first review was both a strength and a limitation. Each reviewer used the tool a total of eight times (once per lesson), risking reviewer fatigue.

It was also challenging to word statements clearly and without ambiguity, while still addressing behavior change theory tenets, content completeness, and accuracy in the assessment tool. The content validity assessment helped address this, but questions raised by reviewers' comments indicated that some statements needed further revision. In addition, some tenets were more extensively represented in the tool than others. Although some tenets are more easily addressed, standardizing the number of statements used per tenet would improve equality in assessment and analysis.

Having fellow EFNEP leaders as reviewers was helpful because of their familiarity with the program, its intended outcomes, and the audience it serves. However, including individuals unaffiliated with the program might have provided additional perspectives.

Implications for Extension

Eating Smart • Being Active is being used by far more EFNEP programs nationally as their primary adult curriculum than any other curriculum in the country. In 2011, 21 EFNEP programs reported using Eating Smart • Being Active as their sole EFNEP curriculum, and another 15 used it in combination with other curricula.

Based on the participants reached by these 21 programs, 49,579 adults received the Eating Smart • Being Active lessons, which comprises 37% of the total national EFNEP adult participants reached in FY11. This number only takes into account the programs that used Eating Smart • Being Active as their sole curriculum. The states that used Eating Smart • Being Active in combination with other curricula were not considered; therefore, the reach of the curriculum is underrepresented in these numbers.

Additionally, many Supplemental Nutrition Assistance Program- Education (SNAP-Ed) programs are using Eating Smart • Being Active. SNAP-Ed is another federally funded nutrition education program targeting a low-income audience and is the largest single-program funding stream in Extension nationally; therefore, through EFNEP and SNAP-Ed, Eating Smart • Being Active has a significant national reach in Extension.

Many nutrition education curricula exist and are rooted in theory, but the processes by which these curricula are developed have been overlooked in published research (Da Cunha, Contento, & Morin, 2000). Considering the dearth of literature on evaluating nutrition education materials in the developmental stage, the project reported here could provide guidance not only to EFNEP, but to other Extension programs as well. In a fast-paced environment, as Extension often is, it's important to remember to take time and commit resources to conduct appropriate formative evaluation when developing educational programs and materials, especially if behavior change in the intended audience is the ultimate goal.

While it does take a substantial amount of time and resources to complete a thorough formative evaluation, the end product should be greatly improved. Ultimately, Extension programs could save time and money; as curriculum developers' confidence in their materials/program increases, they may be more willing to invest larger resources in printing and distribution of materials as well as staff training. There might be fewer educational materials and curricula in our toolboxes because of the more rigorous evaluation, but those materials will be stronger and more effective—leading to larger and more consistent outcomes.

Acknowledgements

We would like to thank our Extension expert reviewers for their significant contribution to this research: Gayle Coleman, University of Wisconsin; Ellen Schuster, University of Missouri; Donna Vandergraff, Purdue University; Phyllis Dennee, formerly of Montana State University; and Mary Wilson, University of Nevada Cooperative Extension.

References

Achterberg, C., & Miller, C. (2004). Is one theory better than another in nutrition education? A viewpoint: more is better. Journal of Nutrition Education and Behavior, 36(1), 40-42.

Bandura, A. (1986). Social foundations of thought and action: a social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.

Baranowski, T., Cullen, K. W., & Baranowski, J. (1999). Psychosocial correlates of dietary intake: Advancing dietary intervention. Annual Review of Nutrition, 19, 17-40.

Bird, C., & McClelland, J. (2010). Practical application of theory-driven intervention to extension programming. Journal of Extension [On-line], 48(6) Article 6FEA4. Available at: http://www.joe.org/joe/2010december/a4.php

Brownell, K. D., & Cohen, L. R. (1995). Adherence to dietary regimens 2: Components of effective interventions. Behavioral Medicine, 20(4), 155-164.

Burney, J., & Haughton, B. (2002). EFNEP: a nutrition education program that demonstrates cost-benefit. Journal of the American Dietetic Association, 102(1), 39-45.

Coleman, G., Byrd-Bredbenner, C., Baker, S., & Bowen, E. (2011). Best practices for extension curricula review. Journal of Extension [On-line], 49(2) Article 2TOT1. Available at: http://www.joe.org/joe/2011april/tt1.php

Contento, I. R. (2007). Nutrition education: Linking research, theory and practice. Sudbury, MA: Jones and Bartlett Publishers.

Contento, I. R., Balch, G. I., Bronner, Y. L., Lytle, L. A., Maloney, S. K., Olson, C. M., & Swadener, S. S. (1995). The effectiveness of nutrition education and implications for nutrition education policy, programs, and research: A review of research. Journal of Nutrition Education, 27, 280-281, 339-346, 374-375.

Contento, I. R., Randell, J. S., & Basch, C. E. (2002). Review and analysis of evaluation measures used in nutrition education intervention research. Journal of Nutrition Education and Behavior, 34, 2-25. doi:10.1016/S1499-4046(06)60220-0

Da Cunha, Z., Contento, I. R., & Morin, K. (2000). A case-study of a curriculum development process in nutrition education using empowerment as organizational policy. Ecology of Food and Nutrition, 39(6), 417-435.

Franz, N. (2007). Adult education theories: Informing cooperative extension's transformation. Journal of Extension [On-line], 45(1) Article 1FEA1. Available at: http://www.joe.org/joe/2007february/a1.php

Gaston, N., & Daniels, P. (1988). Guidelines: Writing for adults with limited reading skills. Washington, D.C.: USDA/FNS.

Glanz, K., Lewis, F. M., & Rimer, B. K. (Eds.). (1997). Health behavior and health education: Theory, research and practice. San Francisco: Jossey-Bass.

Hammerschmidt, P., Murphy, A., Youatt, J., Sawyer, C., & Andrews, S. (1994). Evaluating curriculum effectiveness by asking the users. Journal of Extension [On-line], 32(3) Article 3IAW4. Available at: http://www.joe.org/joe/1994october/iw4.php

Higgins, M. M., & Barkley, M. C. (2003). Concepts, theories and design components for nutrition education programs aimed at older adults. Journal of Nutrition for the Elderly, 23(2), 57-75.

Knowles, M. S. (1984). The adult learner: A neglected species (3rd ed.). Houston: Gulf Publishing Co., Book Division.

Murphy, P. W., Davis, T. C., Mayeaux, E. J., Sentell, T., Arnold, C., & Rebouche, C. (1996). Teaching nutrition education in adult learning centers: Linking literacy, health care, and the community. Journal of Community Health Nursing, 13(3), 149-158.

Norris, J. A. (2003). From telling to teaching: A dialogue approach to adult learning. North Myrtle Beach, S.C.: Learning by Dialogue.

Norris JA, Baker SS. (1999). Maximizing paraprofessional potential. Malabar, Florida: Kreiger Publishing Company.

Rayner, M. J. (2003). Nutrition communication from theory to practice: Some future perspectives. Forum of Nutrition, 56, 129-131.

U.S. Department of Agriculture, National Institute of Food and Agriculture, (2009). About EFNEP (Expanded Food and Nutrition Education Program). Retrieved from: http://www.csrees.usda.gov/nea/food/efnep/about.html

U.S. Department of Agriculture, National Institute of Food and Agriculture, (2012). FY2012: NIFA – National Data: The Expanded Food and Nutrition Education Program (EFNEP). Retrieved from: http://www.nifa.usda.gov/nea/food/efnep/pdf/ntl_report_2012.pdf

U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute. (2011). Clear and Simple: Developing effective print materials for low-literate readers. Retrieved from: http://www.cancer.gov/cancerinformation/clearandsimple

U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. (2010). Dietary Guidelines for Americans, 2010. Retrieved from: http://www.health.gov/dietaryguidelines/2010.asp

Windsor, R., Clark, N., Boyd, N.R., & Goodman, R.M. (2003). Evaluation of health promotion, health education, and disease prevention programs. Boston, MA: McGraw Hill.