The Importance of Evaluating Long-Term Impacts: Utah Master Naturalist Program as a Case Study

Introduction

Cooperative Extension educators conduct evaluation to measure the fulfillment of program goals and to understand whether a program resulted in positive impacts. The Utah Master Naturalist Program (UMNP) has the intentional mission of promoting stewardship of Utah's natural world through the transfer of knowledge and skills necessary for participation in applicable outreach, education, and service. Long-term evaluation of programs with clearly defined missions and goals, such as UMNP, demonstrates their relevance and value to administrators, funders, and the public.

Measuring short-term, or immediate, impacts of an Extension program can help an educator understand how different audiences learn (Larese-Casanova, 2011) or where to make and how to validate specific program improvements (Jayaratne, 2016; Larese-Casanova, 2015). However, many Extension educators focus only on evaluating short-term impacts without also addressing medium- and long-term impacts (Franz & Townson, 2008; Lamm, Israel, & Deal, 2013). In general, long-term impacts of Extension programs are rarely evaluated (Workman & Scheer, 2012) despite the need to understand and demonstrate the impacts of a program over an extended time frame. As a result, Extension educators develop an understanding of only participant satisfaction or intention to adopt practices, factors that are not necessarily indicators of future action or positive behavior change (Lohse, Wall, & Gromis, 2011). Long-term evaluation conducted months, or even years, after a program has been implemented is essential for knowing whether intentions were retained, practices were adopted, or behavior change occurred. A multiyear effort of administering unique follow-up surveys may be needed to truly understand long-term impacts (Higginbotham, Henderson, & Adler-Baeder, 2007).

The measures included in a follow-up survey should focus on program goals, which define the anticipated long-term impacts outlined in the program's logic model (Arnold, 2002). For example, implementers of a farm finance program later evaluated the degree to which farmers' confidence in managing finances had improved, whether they had enrolled in more finance programs or used other resources, and even whether their farms had been more profitable in the years since their participation in the program (Balliet, Douglass, & Hanson, 2010).

I offer evaluation of the UMNP over the past decade as an example of how to concurrently evaluate program impacts across multiple time scales. The UMNP, a series of three week-long field courses taught several times each year, has demonstrated positive short-term impacts related to appreciation, understanding, and stewardship of Utah's natural world (Larese-Casanova, 2011, 2015). However, I determined that it was essential to also understand the long-term impacts of the program to know whether program goals are being fulfilled over a longer time scale. As well, I recognized that identifying positive long-term impacts of the UMNP would help in justifying the extensive investment of time, funding, and other resources required to deliver the program.

Methods

Using an online survey, I evaluated whether the UMNP fulfilled its mission of training volunteers and professionals who promote stewardship of natural resources throughout Utah. After receiving training through the UMNP, naturalists go on to provide education, outreach, and service in their communities. Long-term evaluation of the UMNP focused on the following program goals:

1. Foster a deeper connection to nature.
2. Encourage lifelong learning.
3. Create connections to nature organizations.
4. Promote participation in stewardship activities.
5. Support professional development in education and natural resources.
6. Train educators to provide nature-based education to their students.

I developed the online evaluation survey questions according to the program goals and anticipated long-term impacts listed in the UMNP logic model (Figure 1). The survey addressed nature relatedness (Nisbet, Zelenski, & Murphy, 2009), the influence of the UMNP courses on participants' personal and professional lives, and the degree to which educators gained the knowledge and skills necessary to teach their students about nature. I developed 16 survey items that served as stewardship measures related to the first four program goals. For example, "desire to spend time in nature" served as a measure related to the program goal of fostering a deeper connection to nature. For each item, respondents stated whether the measure had increased, stayed about the same (i.e., did not change noticeably), or decreased for them since their participation in a UMNP course. In addition, I developed nine survey items that focused on how the UMNP had affected participants' professional development and ability to offer nature-based education. For these items, respondents stated the levels at which they agreed or disagreed with relevant statements, using a Likert scale of 1 (strongly disagree) to 5 (strongly agree). The survey also included opportunities for participants to provide open-ended responses to clarify their thoughts.

Figure 1.
Utah Master Naturalist Program Logic Model

Inputs	Outputs	Impacts
		Short-term	Medium-term	Long-term
1. Program Director 2. Interns 3. Funding 4. Materials 5. Equipment	1. Course curricula and materials 2. Field courses across Utah 3. Partnerships with other organizations 4. Program website 5. Facebook page	1. Increased knowledge 2. Greater awareness of Utah's ecosystems and issues 3. Inspiration to explore, learn, and volunteer more 4. Greater interest in stewardship activity	1. Increased time spent in nature 2. Increased participation in other educational programs 3. Awareness of personal impacts on natural environments 4. Sharing knowledge with others	1. Life-long learning about nature 2. Enhanced connection to nature 3. Increased sense of stewardship 4. More connections to nature organizations 5. Greater participation in stewardship 6. Highly trained educators and naturalists, leading to increased public awareness

I used several approaches to validate the survey. Prior to dissemination, I conducted pilot testing with and obtained reviews from other Extension and environmental education professionals to elicit feedback for improvement (Dillman, Smyth, & Christian, 2009). Additionally, I included on the survey itself an item asking respondents to indicate the number of years since they had completed their last UMNP course to validate that the survey was measuring long-term impacts. Lastly, internal consistency was verified through a Cronbach's alpha of 0.87, which indicated that the survey was an instrument of good reliability (Cronbach, 1951).

The survey was administered online in 2014 and 2017 to everyone who had completed at least one UMNP field course since 2007. It was disseminated to all certified Utah Master Naturalists through a special issue of the UMNP email newsletter that was opened by 43% of recipients. I included a personalized greeting in the electronic newsletter, sent periodic reminders, and offered the incentive of a drawing for prizes in an effort to promote survey completion (Dillman et al., 2009). The open and click-through rates of the email newsletter were approximately three times the industry average (Constant Contact, 2018), and overall response rates for the two surveys, 69% and 58%, were similar to those for other evaluation surveys (Archer, 2008; Monroe & Adams, 2012). Response rates were calculated as the ratio of completed surveys to the number of people who opened the survey emails (i.e., the number of eligible respondents) (Wiseman, 2003). It was not possible to determine whether nonresponse bias was present because personalized survey links were not used. Survey completion rate was 100% for both years, implying that the evaluation survey was well-designed and an appropriate length.

Results

The results of the two long-term evaluation surveys indicated that the UMNP has lasting positive impacts on participants' personal and professional lives. On average, 4.3 years had elapsed since respondents had last participated in a UMNP course, confirming that the survey was measuring impacts multiple years after course participation.

Stewardship

All stewardship measures related to desires and emotional connections increased for the majority of survey respondents. However, stewardship measures related to action or behavior change increased for less than half of the respondents. Relevant findings are detailed in Table 1 and described in the paragraphs following the table.

Table 1.
Proportions of Respondents (n = 108) Reporting Change Regarding Sixteen Stewardship Measures

Stewardship goals and related measures	Increased	Stayed about the same
1. Foster a deeper connection to nature
Desire to spend time in nature	62%	37%
Enjoyment of time spent in nature	69%	31%
Feeling of interconnectedness with nature	69%	31%
Time spent in nature	28%	70%
2. Encourage lifelong learning
Interest in experiencing or observing nature	72%	28%
Desire to learn more about nature in Utah	80%	20%
Amount of nature education programming attended	42%	57%
Desire to share knowledge with others	74%	26%
Time spent teaching others	56%	42%
3. Create connections to nature organizations
Involvement in nature organizations	30%	69%
4. Promote participation in stewardship activities
Concern for Utah's natural world	72%	27%
Caring about environmental issues	67%	31%
Desire to improve Utah's natural world	70%	31%
Belief in ability to help solve environmental problems	50%	50%
Time spent volunteering	31%	65%
Participation in stewardship projects	38%	51%

The UMNP fostered a deeper connection to nature through increasing the interest in and enjoyment of spending time in nature as well as the feeling of interconnectedness with nature. Although actual time spent in nature increased to a lesser extent, open-ended responses indicated that many UMNP participants already spent a considerable amount of time in nature.

The UMNP successfully encouraged lifelong learning through increasing participants' interest in experiencing or learning more about nature in Utah. Participation in nature education programs increased for slightly less than half of the participants after having taken a UMNP course. Sharing knowledge with others is a complement to lifelong learning, and both the desire to teach others and time spent teaching increased for majorities of UMNP participants.

For approximately one third of respondents, involvement in nature organizations increased. UMNP courses usually involve cohosting and coteaching by Extension and other nature organizations, a structure expressly intended to forge interrelationships. Indeed, many of the participants surveyed had learned about a UMNP course through existing connections with cohosting nature organizations.

The UMNP had a strong impact on participation in stewardship activities among UMNP participants, especially related to their concern for Utah's natural world and desire to have a positive impact. Approximately one third of respondents stated that actions including volunteering and participation in stewardship activities had increased.

Open-ended responses further clarified impacts:

• "The course increased my understanding of water resources in desert landscapes. This has helped me to be more aware of my water usage on a daily basis."
• "I am much more aware of things around me and understand even more the need to be stewards and educators of others."
• "I spent more time in the local rivers, and mountains, in the summer time, and I spent the summer teaching my own children what I learned."

Professional Development

Survey results also conveyed that the UMNP had positive long-term impacts on participants' professional development. Over three quarters of respondents stated that the knowledge and skills gained from the UMNP were useful in their jobs. Participating in the UMNP also helped 43% advance in their jobs and 20% obtain new jobs. Open-ended responses revealed specific long-term impacts on individuals' professional lives:

• "I now teach a naturalist class to high school students because of my participation in UMNP."
• "It helped me to get my Biology endorsement and be a better teacher. I have taught several lessons based off information and activities I learned in the program."
• "UMNP knowledge qualified me to work on a major water quality contract, and it also enabled me to make significant comments on local water quality planning projects as a volunteer."

Additionally, the findings were clear that the UMNP is particularly effective in training educators to provide nature-based education to their students. Nearly all (90%) of responding educators agreed or strongly agreed that the UMNP gave them both the skills and the resources necessary to teach their students about Utah's natural ecosystems. Three quarters of educators stated that the UMNP helped them maintain their teaching certificate.

Conclusions

Long-term evaluation of the UMNP revealed lasting impacts on program participants who had attended their last UMNP course, on average, over 4 years earlier. For many respondents, stewardship measures, including emotional connections, intentions, and actions, increased as a result of attending a UMNP course. Although more people experienced increases in emotional connections and intentions than in actions or behaviors, many survey respondents who reported that actions stayed the same also clarified that they previously participated in a high level of stewardship behavior (e.g., "These [activities] were high before and remain high"; "I have always had a high level of interest and enjoyment gained from the natural environment"). Essentially, participation in a UMNP course contributed to raising or maintaining the stewardship-related emotional connections and activities of already highly engaged individual. Despite attracting people who already had a high affinity for nature, the UMNP achieved its major goals in support of its mission to foster appreciation for and stewardship of Utah's natural world.

Implications

Ten years of UMNP evaluation were necessary to comprehending the full impacts of the Extension program. The process used to evaluate the UMNP led to an understanding of how emotions, desires, and intentions changed as a result of the program, with a lesser focus on actions. Evaluation of the long-term impacts of the UMNP will continue, with a greater focus on quantifying behavior change. Short-term evaluation surveys at the end of each field course will include items for collecting information such as number of hours spent volunteering for a nature organization, number of nature education programs attended, and number and types of personal stewardship activities conducted during a previous year. A follow-up survey will be delivered to each field course cohort exactly 1 year later to standardize and quantify the same measures. Similar surveys for quantifying past-year activities have been found to have sufficient reliability and validity as compared to direct measurement (Friedenreich et al., 2006).

Long-term evaluation is a lengthy but essential process for Extension professionals who wish to understand the lasting impacts of a program. Although we are often accustomed to evaluating short-term impacts with a survey at the end of a program, short-term impacts do not necessarily reflect whether intentions to adopt practices or change behavior are actually carried to action. Furthermore, combining quantitative data with qualitative, open-ended responses, as was done with the UMNP evaluation, can reveal the full story of a program's impacts. Collecting responses from participants several years after a program can be challenging, even with a well-designed survey, but doing so is worth the effort. The methodology and findings reported here serve as an example that evaluation is an ongoing process, often with one evaluation leading to more evaluations, that reveals information critical to gaining a deeper understanding of Extension programming.

References

Archer, T. M. (2008). Response rates to expect from web-based surveys and what to do about it. Journal of Extension, 46(3), Article 3RIB3. Available at: https://joe.org/joe/2008june/rb3.php

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension, 40(3), Article 3FEA4. Available at: https://www.joe.org/joe/2002june/a4.php

Balliet, K. L., Douglass, M. B., & Hanson, G. (2010). Long-term impact of the farm financial analysis training curriculum on FSA borrowers in Pennsylvania. Journal of Extension, 48(1), Article 1FEA6. Available at: https://www.joe.org/joe/2010february/a6.php

Constant Contact. (2018). Average open, click-through, and bounce rates of Constant Contact customers by industry. Retrieved from https://knowledgebase.constantcontact.com/articles/KnowledgeBase/5409-average-industry-rates

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Mail and Internet surveys: The tailored design method (3rd ed.). New York, NY: John Wiley and Sons.

Franz, N. K., & Townson, L. (2008). The nature of complex organizations: The case of Cooperative Extension. New Directions for Evaluation, 120, 5–14.

Friedenreich, C. M., Courneya, K. S., Neilson, H. K., Matthews, C. E., Willis, G., Irwin, M., . . . Ballard-Barbash, R. (2006). Reliability and validity of the past year total physical activity questionnaire. American Journal of Epidemiology, 163(10), 959–970.

Higginbotham, B., Henderson, K., & Adler-Baeder, F. (2007). Using research in marriage and relationship education programming. Retrieved from http://ncsu.edu/ffci/publications/2007/v12-n1-2007-spring/higginbotham/fa-4-higginbotham.php

Jayaratne, K. S. U. (2016). Tools for formative evaluation: Gathering the information necessary for program improvement. Journal of Extension, 54(1), Article 1TOT2. Available at: https://www.joe.org/joe/2016february/tt2.php

Lamm, A. J., Israel, G. D., & Diehl, D. (2013). A national perspective on the current evaluation activities in Extension. Journal of Extension, 51(1), Article 1FEA1. Available at: https://www.joe.org/joe/2013february/a1.php

Larese-Casanova, M. (2011). Assessment and evaluation of the Utah Master Naturalist Program: Implications for targeting audiences. Journal of Extension, 49(5), Article 5RIB2. Available at: http://www.joe.org/joe/2011october/rb2.php

Larese-Casanova, M. (2015). Using evaluation to guide and validate improvements to the Utah Master Naturalist Program. Journal of Extension, 53(3), Article 3IAW3. Available at: https://www.joe.org/joe/2015june/iw3.php

Lohse, B., Wall, D., & Gromis, J. (2011). Intention to consume fruits and vegetables is not a proxy for intake in low-income women from Pennsylvania. Journal of Extension, 49(5), Article 5FEA5. Available at: https://www.joe.org/joe/2011october/a5.php

Monroe, M. C., & Adams, D. C. (2012). Increasing response rates to web-based surveys. Journal of Extension, 50(6), Article 6TOT7. Available at: https://www.joe.org/joe/2012december/tt7.php

Nisbet, E. K., Zelenski, J. M., & Murphy, S. A. (2009). The Nature Relatedness Scale: Linking individuals' connection with nature to environmental concern and behavior. Environment and Behavior, 41, 715–740.

Wiseman, F. (2003). On the reporting of response rates in Extension research. Journal of Extension, 41(3), Article 3COM1. Available at: https://joe.org/joe/2003june/comm1.php

Workman, J. D., & Scheer, S. D. (2012). Evidence of impact: Examination of evaluation studies published in the Journal of Extension. Journal of Extension, 50(2), Article 2FEA1. Available at: https://joe.org/joe/2012april/a1.php