 |
April 2019
|
April 2019 // Volume 57 // Number 2 // Tools of the Trade // v57-2tt1
Beyond Evaluation: Using the RE-AIM Framework for Program Planning in Extension
Abstract
Extension professionals need to apply sound frameworks to program planning and evaluation in order to demonstrate strong population health impact and value. Pragmatic application of the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework addresses the "who, what, when, where, how, and why" of a program's implementation. This article suggests pragmatic questions and example applications for each of the RE-AIM dimensions specifically for Extension professionals. This adapted RE-AIM tool can help Extension practitioners in all disciplines better plan and evaluate their programs and demonstrate the public value of Extension.
The Cooperative Extension System has a long history of building community trust and responding to community needs (North Central Cooperative Extension Association, 2015; Rasmussen, 2002). In spite of its community ties and impacts on health behaviors, the system continues to function as one of the nation's "best kept secrets." This lack of recognition could be detrimental, as public perception of Extension and the measurable impact the system has on population health are tied to funding streams and decision making regarding its value and structure (Franz, 2014; Warner, Christenson, Dillman, & Salant, 1996). To better disseminate results and increase public perception and health value, stronger Extension program planning and evaluation are needed. For the purposes of this article, program encompasses everything from individual-level direct education to policy, systems, and environment changes.
Fortunately, frameworks exist that can help Extension professionals with both program planning and program evaluation. RE-AIM is one such framework; it was designed to enhance the quality, speed, and public health impact of health interventions (Glasgow, Vogt, & Boles, 1999). RE-AIM goes beyond measuring reach (number of participants) and efficacy (primary program outcomes, such as whether a nutrition program increased consumption of fruits and vegetables)—factors on which Extension typically reports. The framework also includes adoption, implementation, and maintenance (individual and system-level) to assess system-level factors and determine whether a program has real-world validity (Glasgow et al., 1999). In 2017, Downey, Peterson, Donaldson, and Hardman introduced RE-AIM as an evaluation framework for Extension. Our purpose with this article is to build on that introduction by providing considerations for applying RE-AIM during program planning, including considerations for health equity.
Critical Questions to Guide Extension Program Planning
Using the RE-AIM framework when planning can help Extension professionals ensure that a program reaches the target audience, effectively achieves and maintains the intended outcomes, is adopted by delivery organizations and agents (e.g., educators/agents or volunteers), is implemented with fidelity, and is maintained within the system (Glasgow & Estabrooks, 2018; Harden et al., 2018). RE-AIM can be used to prioritize the focus of activities toward intended audiences and outcomes, determine how each dimension will be assessed, and develop data collection, management, evaluation, and reporting guidelines. Ideally, this planning process is participatory (Nichols, 2002) and includes future participants, adopter agents and staff, and evaluators to ensure that the program is a good fit for delivery in the field (Harden, Johnson, Almeida, & Estabrooks, 2017; Israel, Eng, Schulz, & Parker, 2005) and is equitable (i.e., avoidable inequities are eliminated) (Glasgow et al., 2013). Although the benefits of including all RE-AIM dimensions to balance internal and external validity are recognized, fully employing RE-AIM is not always practical (Estabrooks & Allen, 2012), especially in underfunded community settings that value pragmatic approaches (Harden et al., 2018). For example, when translating an evidence-based program to a new setting, Extension professionals may decide to focus on adoption, implementation, and system-level maintenance to determine how the program operates in the new context.
From a pragmatic perspective, RE-AIM dimensions can be translated into "who, what, when, where, how, and why" questions (Glasgow & Estabrooks, 2018). To provide a new "tool of the trade" for Extension professionals, we have adapted this series of questions with a focus on planning for evaluation and suggested pragmatic questions for each RE-AIM dimension, including health equity considerations (Glasgow et al., 2013) (see Table 1).
| RE-AIM dimension | Suggested planning questions | Extension examples |
| Reach |
Who is the target audience for the program? |
Define the priority audience or subgroups who would benefit most from exposure to the program. Target the program to those who need it rather than those who want it. |
| Health equity: How will program access be supported and participation obstacles removed? |
Develop strategies to specifically recruit those who are most underserved or at risk and enable their participation. Consider the time the program is offered and how participants will have transportation to attend. Plan how you advertise, promote, and locate the program to reach these participants. Engage community partners who serve the audience to help recruit and include the most underserved within your target audience. |
|
| Effectiveness |
What key changes or outcomes do you expect to see? |
Determine the individual or environmental-level changes you are targeting. |
|
How will you collect data to measure these outcomes? |
Consider data collection that is realistic for those who will deliver the program. For individual or interpersonal level programs, food frequency questionnaires, behavior logs, or physical activity trackers could be used. For environmental changes, meeting minutes, grant activity, readiness assessments, and asset mapping may be used. |
|
|
Health equity: How will the intervention be delivered to those most in need? |
Consider using multiple delivery channels for accessing your program (e.g., direct, Internet, and/or local media–delivered programs). Ensure that materials are culturally appropriate and designed for diverse literacy levels. |
|
| Adoption |
Who will deliver the program? |
Determine who is responsible for training, technical assistance, and support. For example, state-level specialists may train Extension educators/agents, or Extension educators/agents may be training/assisting volunteers or school staff members. |
|
How many of these delivery agents will use the program? |
Determine ways you will capture and track adoption rates, representativeness of the staff and settings associated with delivery of the program, and what resources are available where to make the work feasible and sustainable. |
|
| Health equity: How will you enhance participation in low-resource settings? |
Include delivery agents throughout the planning process to improve buy-in. Choose a feasible program that places low demands on staff and resources. |
|
| Implementation |
How will the initiative, including adjustments and adaptations, be delivered? |
Determine how you will measure fidelity. If implementation checklists are used, consider the degree to which they will seem supportive or punitive. Consider using checklists as a way to address how Extension practitioners and volunteers can improve their performance rather than to determine whether they are delivering with fidelity. |
|
What costs (including time and burden, not just money) need to be considered? |
Consider whether implementation costs are feasible for the organization. Include costs of recruiting or tailoring of materials, training of delivery personnel, and start-up (e.g., equipment and incentives) vs. continuing costs (e.g., educator/agent time, training new staff). Decide whether it is appropriate to run a cost-effectiveness analysis to determine cost of achieving the program outcomes. |
|
|
Health equity: How will you document adaptations to the original program? |
Implementation checklists should also capture population- or systems-specific adaptations that may be improving the fit of the intervention rather than deviating from its initial protocol (Chambers & Norton, 2016). |
|
| Maintenance: Individual level |
How likely is your initiative to produce lasting effects for individual participants? |
Consider the duration and evidence base of your program to determine whether long-term change is likely. Direct resources toward implementing programs with high population reach and evidence of long-term behavior change rather than single classes or informational seminars. |
|
Health equity: How will you assess long-term results? |
Engage participants in deciding how you will stay in touch to track outcomes after the program ends. For example, participants may want to engage in a follow-up event 6 months after the program or to keep in touch through newsletters, a website, or social media. Consider equitable and inclusive access to resources needed for participants to sustain program results, such as social media or a website. |
|
| Maintenance: Organization level |
How can the organization sustain the initiative over time, and what plans are there for leaving resources or trained staff in place? |
Consider what your state system values and supports, including program capacity and resources provided by managers (e.g., directors and district directors), multisector stakeholders and partners, community members, and volunteers. |
|
Health equity: How will you prepare delivery settings and systems to sustain the program? |
Assess access and sustainability barriers and facilitators within and among delivery settings. Provide tools and resources to enable long-term program monitoring and adaptation. |
RE-AIM also can be used during program delivery to monitor inputs, activities, and outcomes; track the quality and costs of implementation; assess field adaptations; and consider the intended (and unintended) effects of the intervention (Harden et al., 2018). After completion of a program, in addition to program evaluation, RE-AIM can be used for structuring reporting impacts and diffusion to new audiences and adopters. For example, program infographics and community forums may be deemed the best approaches for communicating outcomes to lay audiences, whereas scientific papers may be necessary for informing scholarly audiences and stakeholders (Brownson, Jacobs, Tabak, Hoehner, & Stamatakis, 2012).
Conclusion
RE-AIM is a useful tool for Extension program planning and evaluation. Applying its dimensions during planning can be accomplished by considering "who, what, when, where, how, and why" questions. Answering these questions can ensure equity, a good fit within the delivery system, and unbiased impacts—both in health and in public value. Although developed for health behavior interventions, RE-AIM may be useful for other Extension program areas. As noted by Downey et al. (2017), those working in all areas of Extension (e.g., agriculture, 4-H, human development) should invest in overall impacts of their programming. Better planning and program evaluation can lead to better dissemination of Extension impacts. As public value and funding are tied to demonstrating robust impacts, this is a crucial strategy for continued support and growth of 21st-century Extension.
Acknowledgments
We would like to thank the members of the Evaluation and Public Value Subcommittee of the National Institute of Food and Agriculture's Nutrition and Health Committee for Planning and Guidance (Lisa Franzen-Castle, Soghra Jarvandi, Novella Ruffin, and Michele Walsh) for their support of this work.
References
Brownson, R. C., Jacobs, J. A., Tabak, R. G., Hoehner, C. M., & Stamatakis, K. A. (2012). Designing for dissemination among public health researchers: Findings from a national survey in the United States. American Journal of Public Health, 103(9), 1693–1699. doi:10.2105/AJPH.2012.301165
Chambers, D., & Norton, W. (2016). The adaptome: Advancing the science of intervention adaptation. American Journal of Preventive Medicine, 51(4), S124–S131.
Downey, L. H., Peterson, D. J., Donaldson, J., & Hardman, A. (2017). An Extension application of the RE-AIM evaluation framework. Journal of Extension, 55(3), Article 3IAW1. Available at: https://www.joe.org/joe/2017june/iw1.php
Estabrooks, P., & Allen, K. (2012). Updating, employing, and adapting: A commentary on what does it mean to "employ" the RE-AIM model. Evaluation & The Health Professions, 26(1), 67–72.
Franz, N. (2014). The role of evaluation in determining the public value of Extension. Journal of Extension, 52(4), 4COM3. Available at: https://www.joe.org/joe/2014august/comm3.php
Glasgow, R. E., Askew, S., Purcell, P., Levine, E., Warner, E., Stange, K., . . . Bennett, G. (2013). Use of RE-AIM to address health inequities: Application in a low-income community health center–based weight loss and hypertension self-management program. Translational Behavioral Medicine, 3, 200–210. doi:10.1007/s13142-013-0201-8
Glasgow, R., & Estabrooks, P. (2018). Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Preventing Chronic Disease, 15(E02), 1–7. doi:10.5888/pcd15.170271
Glasgow, R., Vogt, T., & Boles, S. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 9, 1322–1327.
Harden, S., Johnson, S. B., Almeida, F., & Estabrooks, P. (2017). Improving physical activity program adoption using integrated research-practice partnerships: An effectiveness-implementation trial. Translational Behavioral Medicine, 7, 28–38. doi:10.1007/s13142-015-0380-6
Harden, S., Smith, M., Smith-Ray, R., Ory, M., Estabrooks, P., & Glasgow, R. (2018). RE-AIM in clinical, community, and corporate settings: Perspectives, strategies, and recommendations to enhance public health impact. Frontiers in Public Health, 6(71). doi:10.3389/fpubh.2018.00071
Israel, B., Eng, E., Schulz, A., & Parker, E. (2005). Methods in community-based participatory research for health. San Francisco, CA: Jossey-Bass.
Nichols, L. (2002). Participatory program planning: Including program participants and evaluators. Evaluation and Program Planning, 25(1), 1–14.
North Central Cooperative Extension Association. (2015). 2015 Battelle study: Analysis of the value of family & consumer sciences Extension in the North Central Region. Retrieved from http://www.nccea.org/multistate-activities/fcs-battelle-report-2015/
Rasmussen, W. D. (2002). Taking the university to the people: Seventy-five years of Cooperative Extension. West Lafayette, IN: Purdue University Press.
Warner, P. D., Christenson, J. A., Dillman, D. A., & Salant, P. (1996). Public perception of Extension. Journal of Extension, 34(4), 4FEA1. Available at: https://www.joe.org/joe/1996august/a1.php