An Extension Application of the RE-AIM Evaluation Framework

RE-AIM is an evaluation framework that has been widely used by public health professionals for over 15 years. RE-AIM identifies five evaluation dimensions—reach, effectiveness, adoption, implementation, and maintenance—and is considered a comprehensive model of evaluation. The usefulness of RE-AIM has been well established in public health journals but has remained undocumented in journals for which Extension professionals are the target audience (Folta et al., 2015; Glasgow, Vogt, & Boles, 1999).

With the purpose of informing Extension professionals about RE-AIM, we briefly describe and present an application of the framework. We conclude with insights about the potential of applying RE-AIM for evaluation across Extension programming.

A Description of RE-AIM

RE-AIM was developed for the purpose of evaluating health behavior research but has evolved into a tool practitioners use to plan and assess programs and report program results (Gaglio, Shoup, & Glasgow, 2013). This framework allows programs to be evaluated at the individual or participant level (reach, effectiveness, and maintenance) and at the organizational level (adoption, implementation, and maintenance) (National Cancer Institute, 2012; RE-AIM, 2013).

• Reach is the number, proportion, and representativeness of intervention participants.
• Effectiveness (or efficacy) is the impact of the intervention relative to important outcomes.
• Adoption is the number, proportion, and representativeness of entities or intervention agents willing to initiate the intervention.
• Implementation is the fidelity to various elements of the intervention's protocol.
• Maintenance at the individual level is the long-term impact of the intervention in terms of program outcomes and at the organizational level is the extent to which the intervention becomes institutionalized.

Application of RE-AIM to a 4-H Health Promotion Program

The U.S. Department of Agriculture provided funding to expand 4-H Healthy Living programming to youths in selected rural counties in Mississippi and Tennessee. The Jump into Foods and Fitness curriculum (JIFF) (Baird, Branta, Mark, & Seremba, 2003), developed by Michigan State University Extension Service, was adopted for replication in these counties because evidence suggested that JIFF increases 4-H'ers frequencies of eating breakfast daily, vegetable intake levels, amounts of moderate physical activity, and enjoyment of being physically active (Downey, Peterson, Le Menestrel, Leatherman, & Lang, 2014). Also, there was evidence of states having adopted JIFF as a research-based educational curriculum (Gossett, 2012).

Prior to implementation of JIFF in the selected counties, 4-H professionals and volunteers who would be implementing the program received face-to-face training to enhance their relevant knowledge and skills. Part of the training focused on the project's evaluation protocol, which was based on the five RE-AIM dimensions.

Because JIFF had not been implemented previously in Mississippi and Tennessee, RE-AIM was used as a standardized approach for evaluating the program in the participating counties, settings that greatly differ from the one in which the program was developed. Specifically, the RE-AIM framework was used for measuring program reach, effectiveness, adoption, implementation, and maintenance as follows:

• Reach was measured by the percentage of youths who participated in JIFF, based on a valid denominator of total target population. For example, the number of third graders who participated in JIFF was compared to the total number of third graders at a participating institution. Sign-in sheets were used for obtaining numbers of youth participants, and secondary data sources (i.e., school records) were used for documenting the total number of youths eligible to participate.
• Effectiveness was assessed at two levels. First, the effectiveness of the training for 4-H professionals and volunteers in Mississippi and Tennessee was assessed through a posttest survey completed at the end of the training and through an observation checklist completed as each 4-H professional or volunteer taught a JIFF lesson at the training. Secondly, effectiveness of JIFF on youth participants was assessed through the use of a pretest/posttest based on 4-H Common Measures (National 4-H Council, 2015). The pretest was implemented prior to the start of the program and repeated as a posttest at the end of the program.
• Adoption was assessed by comparing the number of settings, for example schools or Boys and Girls Clubs, approached to participate with the number of settings eligible to participate (i.e., the number of elementary schools and out-of-school programs for children in each county).
• Implementation was assessed by documenting adherence to the JIFF protocol through a 4-H professional and volunteer readiness assessment completed after the training, an implementation checklist for 4-H professionals and volunteers to use to self-report on implementation, and an observation checklist for evaluation specialists to use on periodic site visits during program implementation.
• Maintenance was assessed at the individual level through a follow-up survey administered to youths 3 months after the posttest. It is too early to assess maintenance at the organizational level because project funding continues. Maintenance at the organizational level will be demonstrated if JIFF continues to be implemented after the grant ends.

Conclusions

Our experience with the RE-AIM framework suggests that it is a feasible tool for informing program modifications for enhanced success and sustainability. For example, examining the checklists used for the implementation dimension of the evaluation could help identify adaptations made by individual educators that would be beneficial across all sites. Evaluation based on RE-AIM is a straightforward approach for comparing public health outcomes across multiple states or sites. The framework allowed us to develop a detailed evaluation protocol, on which agents were trained so that they were prepared to properly collect evaluation data. Consistent data collected across sites made comparisons in outcomes possible. Although we used the framework to evaluate only one health promotion program, it could be used to evaluate multiple programs for the purpose of identifying those that are most effective. Use of 4-H Common Measures could make this possible. If RE-AIM were used for evaluating several programs, effectiveness could inform decisions about resource distribution toward future program implementation across one state or multiple states.

RE-AIM could serve these same evaluative purposes for other Extension programs as well. It is most intuitive to apply RE-AIM to other health-related programs in community resource and economic development and family and consumer sciences. Yet the dimensions of RE-AIM are applicable to any educational program, including those in agriculture and natural resources. It will always be important for Extension educators to know whether they have reached their target audiences and whether their programs have achieved the intended outcomes.

Acknowledgment

Funding for the 4-H Healthy Living program described in this article was provided by the U.S. Department of Agriculture through the Rural Health and Safety Education Competitive Grants Program.

References

Baird, K., Branta, C., Mark, C., & Seremba, D. (2003). Jump into foods and fitness. East Lansing, MI: Michigan State University Extension, 4-H Youth Development, Family & Consumer Sciences.

Downey, L. H., Peterson, D. J., Le Menestrel, S., Leatherman, J., & Lang, J. (2014). 4-H Healthy Living programs with impact: A national environmental scan. New Directions for Youth Development, 143, 13–24. doi:10.1002/yd.20101

Folta, S. C., Lichtenstein, A. H., Seguin, R. A., Goldberg, J. P., Corbin, M. A., Wiker, N., . . . Nelson, M. E. (2015). The Strong Women-Healthy Hearts program in Pennsylvania: RE-AIM analysis. Translational Behavioral Medicine, 5(1), 94–102. doi:10.1007/s13142-014-0286-8

Gaglio, B., Shoup, J. A., & Glasgow, R. E. (2013). The RE-AIM framework: A systematic review of use over time. American Journal of Public Health, 103(6), E38–E46. doi:10.2105/AJPH.2013.301299

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89(9), 1322–1327. doi:10.2105/AJPH.89.9.1322

Gossett, L. S. (2012). EFNEP reaches refugee youth using a mobile van. Journal of Extension, 50(1), Article 1IAW2. Available at: http://www.joe.org/joe/2012february/iw2.php

National 4-H Council. (2015). Common Measures. Retrieved from http://4-h.org/professionals/common-measures/

National Cancer Institute. (2012). Measuring the use of the RE-AIM model dimension items-checklist. Retrieved from http://www.re-aim.org/forms/re-aim_framework_constructs_checklist.pdf

RE-AIM. (2013). What is RE-AIM? Retrieved from http://www.re-aim.hnfe.vt.edu/about_re-aim/what_is_re-aim/index.html