Assessment and Evaluation of the Utah Master Naturalist Program: Implications for Targeting Audiences

Introduction

In the United States, citizens have become progressively more disconnected from their surrounding environment, resulting in a need for greater environmental literacy (Coyle, 2005; Louv, 2005). Improving environmental literacy, skills, and responsibility rely upon effective environmental education (Hungerford & Volk, 1990). Greater awareness, knowledge, and skills are necessary not only to connect to one's environment but also to address environmental issues and create solutions (United Nations Educational Scientific and Cultural Organization-United Nations Environment Programme, 1976; North American Association for Environmental Education, 2004).

In order to meet this need, environmental education programs are developed and taught by a variety of agencies, universities, schools, and organizations. Within Utah, at least 266 providers of environmental education programs exist, and at least 58% of the programs focus on educating children (Utah Society for Environmental Education, 2008). Many of these providers have little funding for educational staffing and rely heavily upon partnerships with volunteer naturalists to build the capacity of their programs. In most cases, this requires organizations to train their volunteers. In response to the demand for trained volunteers, state Cooperative Extension Units, in many cases in cooperation with their respective wildlife agencies, have developed educational programs that focus on training volunteer naturalists and citizen scientists, including Master/Volunteer Naturalist Programs, Master Conservation Stewards, Master Watershed Stewards, and Coverts Projects. At least 29 programs exist or are under development in 24 states, and the broad reach of these programs throughout Extension increases each year.

A needs assessment survey was conducted in Utah to examine the amount of training available to volunteer naturalists, as well as the need for additional standardized training across the state. Ninety-one percent of the responding organizations use volunteers to deliver programs, but only 55% provided training to their volunteers. In addition, the majority of volunteer training programs lasted 1-5 hours. Regardless, 95% of the respondents thought the Utah Master Naturalist Program (UMNP) would be valuable training for and of interest to their volunteers (Larese-Casanova, 2007). This seems consistent with the results reported by Savanick and Blair (2005), in which they reported a clear need for fewer volunteers with more training from a consistent, science-based program.

Utah State University Cooperative Extension initially developed the UMNP to address this need for training volunteer naturalists. However, there was also interest in the program from professional naturalists and members of the general public. As such, the UMNP initially targeted professional and amateur naturalists equally, with minimal recruitment of the general public.

The overarching goal of the UMNP is to "develop a corps of well-informed Master Naturalists who provide education, outreach, and service to promote citizen stewardship of natural resources within their communities." In order to achieve this goal, the UMNP consists of three 40-hour classroom- and field-based training modules focusing on Utah's Watersheds, Deserts, and Mountains. Each module employs a standardized statewide curriculum to ensure consistency across all locations and instructors. This article discusses the Watersheds module that was taught during 2007-2008. As related to Utah watersheds, this module focused on the geology, hydrology, ecology, and management of aquatic and wetland systems in Utah.

In order to determine the success of any environmental education program, it is necessary to quantify the following:

• Participant Demographics—to ensure target audience is reached


• Participant Knowledge—to assess knowledge gained as a result of the program


• Curriculum and Instructor Effectiveness—to evaluate program success in promoting environmental literacy.

This could provide for greater accuracy in teaching and marketing effort, ultimately resulting in increased program success.

Methods

The UMNP Watersheds module was taught at four locations by eight instructors in 2007-2008. Each instructor was provided with copies of the standardized curriculum, which included student handbooks and electronic presentations. The Watersheds module was taught for 40 hours, including classroom presentations and field exploration, at each location.

Demographic data, including age, gender, ethnicity, highest degree earned, work status, as well as whether the participant worked at (including their position), volunteered for, or was a member of an organization that teaches about nature, were collected for each person who participated in the UMNP. The data were used to analyze differences in knowledge and program satisfaction between groups.

Participant knowledge was assessed using identical pre- and post-surveys at the beginning and end, respectively, of the Watersheds module. Both surveys contained 25 short-answer questions that were selected to be representative of the curriculum. Pre- and post-survey scores were considered to be participant knowledge prior to and after participating in the UMNP, respectively. The difference between the two scores represented knowledge gained while participating in the UMNP. Cronbach's alpha was used to determine the reliability of the pre- and post-surveys (Cronbach, 1951).

The effectiveness of the UMNP Watersheds module curriculum and instructors was determined at the end of the program using an evaluation form. Each participant was asked to quantify the level at which they agree with 18 statements about their experience (Table 1). Participants responded to each statement on a scale of 2 (strongly agree) to -2 (strongly disagree, with 0 being neutral.

For the purposes of the study reported here, all participants who worked as a naturalist or educator for an organization that teaches people about nature were considered to be "professional naturalists" (n=26), and all other participants, including non-professional volunteer naturalists and members of the general public, were considered "amateur naturalists" (n=15). In the analysis of percent improvement, two outliers that were more than nine standard deviations away from the mean were removed. Data were analyzed using the SPSS Statistical Package (SPSS 2005). Statistical significance was determined at p<0.05. The following data analyses were performed:

• Pre- and post-survey scores were compared using an independent samples t-test between professional and amateur naturalists to determine the differences in knowledge before and after UMNP participation.


• Pre- and post-survey scores were compared using a paired t-test between all participants to determine the knowledge gained during UMNP participation.


• Percent improvement between the pre- and post-surveys was calculated for each participant, and was compared between professional and amateur naturalists.


• Rankings of all 18 evaluation questions were compared using an independent samples t-test between professional and amateur naturalists to determine the difference at which the two groups evaluated the UMNP.

Table 1.
UMNP Evaluation Statements

Statement	Evaluation Statement
1	The UMNP... was worth my time
2	...was enjoyable
3	...gave me a greater understanding of Utah's aquatic and wetland systems
4	...met my expectations (curriculum and sessions)
5	...met my personal goals (written on the first day)
6	...has inspired me to explore Utah's natural world more
7	...has inspired me to learn more
8	...has inspired me to volunteer more
9	...has inspired me to think more about my own use of natural resources
10	...has inspired me to want to teach others
11	...has helped me acquire knowledge and skills that would help me be a better environmental educator
12	...introduced me to more environmental education and resource management organizations in Utah
13	...field trips helped me apply knowledge gained in class
14	...instructors demonstrated a broad understanding of the material
15	...instructors were effective in teaching the materials
16	I would recommend the Utah Master Naturalist to friends/colleagues
17	I am interested in attending other Utah Master Naturalist modules
18	I am interested in becoming a Utah Master Naturalist instructor

Results

A total of 46 people participated in the UMNP Watersheds module during the study period. A complete set of demographic, assessment, and evaluation data was available for 41 participants. The pool of participants was 76% female and 24% male, with an average age of 46 (range=22-72). In general, the participants were well educated, with 84% having at least a Bachelor's Degree. All participants were Caucasian.

The pre-survey assessment scores ranged widely, between 8-81%, and were significantly higher for professional naturalists than for amateur naturalists (p<0.05) (Table 2). In contrast, post-survey scores had a narrower range of 62-98%, and every participant had a higher post-survey score. Cronbach's alphas for both surveys were similar (i.e., .731 and .746 for pre-and post-surveys, respectively), which indicated that the surveys were reliable measures of participant knowledge. As a whole, the UMNP participants' scores were significantly higher on the post-survey than on the pre-survey (p<0.05). However, post-survey scores were not significantly different between professional and amateur naturalists.

Table 2.
UMNP Assessment Survey Scores (%)

	Pre-Survey	Post-Survey	Difference
All Participants	48±18	85±10	37*
Professionals	54±13	85±10	31*
Amateurs	42±19	83±11	41*
Difference	12	2	-
*p<0.05

Percent improvement between pre- and post-survey scores varied widely, averaging 94% (st. dev.=93%). A comparison was made between percent improvement and pre-survey score to determine the relationship between prior knowledge and knowledge gained (Figure 1). In general, the participants that scored lower on the pre-survey improved the most. In fact, as pre-survey scores decreased, percent improvement increased exponentially. Although percent improvement was, on average, higher for amateur naturalists (mean=119%, st. dev.=105%) compared to professional naturalists (mean=79%, st. dev.=84%), there was no significant difference, due to a high degree of variability.

Figure 1.
UMNP Pre-Assessment Score Compared to Percent Improvement

Mean responses to program and instructor evaluation statements were calculated to determine the overall effectiveness of the UMNP. On average, responses to all of the 18 evaluation statements were between 1 (i.e., agree) and 2 (i.e., strongly agree) (Table 3). While all statements received positive responses on average, statements 1, 2, 3, 6, 7, 12, 14, 15, 16, and 17 received the highest responses. The lowest mean responses were given to statements 4, 5, 8, and 18.

Evaluation responses were compared between professional and amateur naturalists to determine if there were any significant differences between the two groups (Table 3). For more than half of the evaluation statements (i.e., 1, 2, 3, 8, 9, 10, 13, 15, 16, and 17), amateur naturalists responded significantly more positively than professional naturalists.

Table 3.
Mean Responses to UMNP Evaluation Statements

	Evaluation Statements
	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
All Participants	1.8	1.9	1.8	1.3	1.4	1.9	2.0	1.2	1.6	1.6	1.6	1.7	1.6	1.9	1.7	1.7	1.9	1.0
Professionals	1.7	1.8	1.7	1.0	1.2	1.8	2.0	0.9	1.4	1.5	1.5	1.8	1.5	1.9	1.5	1.5	1.8	0.8
Amateurs	2.0	2.0	2.0	1.7	1.8	1.9	2.0	1.9	1.9	1.8	1.7	1.5	1.8	1.8	2.0	1.9	2.0	1.3
Significance	***	***	***					**	***	**		***	*		***	***	***
*p<0.05, **p<0.01, ***p<0.001

Discussion

UMNP participants were primarily well-educated Caucasians, and most were female. Although the demographics of the UMNP were consistent with those of similar programs from other states, they did not match the demographics of Utah. The UMNP underserved minorities (i.e., Latinos in particular), males, and individuals with less than a college degree (United States Census Bureau, 2006). Additionally, while the UMNP needs assessment initially focused only on volunteer naturalists, the majority of the participants were professional naturalists. The demographics of the UMNP were similar to those of other Master Naturalist Programs (Bonneau, Legg, Darville, Haggerty, & Wilkins, 2003; Broun, Nilon, & Pierce, 2009; Rager, Guiney, & Blair, 2008; Minnesota Master Naturalist Program, 2008), some of which also have had difficulty in attracting males and minorities.

The wide range of scores on the pre-survey assessment (i.e., 8-81%) implied that the UMNP participants as a group entered the program with a wide range of knowledge related to watersheds in Utah. As expected, professional naturalists had a significantly higher incoming level of knowledge likely due to their educational and/or work experience.

Post-survey scores increased for each UMNP participant, which implied that the knowledge level of all participants increased. Higher post-survey scores suggested that the entire group's knowledge level increased significantly while participating in the UMNP, which is consistent with other similar programs (Broun et al., 2009; Bonneau et al., 2003; Minnesota Master Naturalist Program, 2008; Van Den Berg, 2006; Van Den Berg & Dann, 2008).

UMNP post-survey scores were not significantly different between professional and amateur naturalists. This implied that, by the end of participation in the UMNP, the amateur naturalists' knowledge level increased a greater amount than that of the professional naturalists. In addition, the standard deviation of the post-survey scores was 45% lower than that of the pre-survey scores, which suggested that the knowledge level of the entire group also became more consistent.

While each participant's knowledge increased, the amount by which knowledge increased varied widely between 7-400%. While mean percent improvement was not significantly different between amateur and professional naturalists, differences between pre- and post-survey scores suggested that amateur naturalists learned more than the professional naturalists. However, regardless of status as a professional or amateur naturalist, the knowledge of participants who knew the least at the beginning of the UMNP increased the most (Fig. 1).

On average, participants evaluated the UMNP well by choosing to either "agree" or "strongly agree" to all of the evaluation statements. By this measure, the program was successful. UMNP participants enjoyed the program and believed their knowledge increased and were inspired to learn more. The instructors taught effectively, and participants wanted to continue with other UMNP modules as well as encourage others to participate. Evaluation results also differed between amateur and professional naturalists. On average, amateur naturalists consistently evaluated the UMNP as equally positive or, more often, significantly more positively than professional naturalists.

Conclusions

Demographic, assessment, and evaluation data were each in their own way particularly useful in determining program success. However, by using these three data sets together, a greater understanding of the effectiveness of the program was achieved. For the purposes of the study reported here, program success was measured by (1) the amount of participant knowledge gained during the program and (2) participant satisfaction with the program. Overall, participant knowledge increased significantly and participant satisfaction was relatively high.

Although the UMNP was successful by these two measures, some discrepancies still existed. Despite the UMNP being advertised through the organizations that originally participated in the needs assessment survey, the program did not attract its original intended audience, amateur naturalists, as much as it did professional naturalists. Assessment and evaluation data showed that amateur naturalists may have learned more and certainly evaluated the UMNP more positively. If a greater program success is to be achieved (i.e., even higher amounts of knowledge gained and greater participant satisfaction), the target audience for the UMNP must be adjusted to attract more amateur naturalists and members of the general public. Refining the target audience of the UMNP will result in more effective marketing not just in terms of better attracting the target audience, but also in terms of effort and funding spent.

The UMNP was also advertised through many agencies and organizations that helped develop the program, some of which did not train volunteers. Some of these organizations' staff members benefitted from participating in the UMNP. While these professionals will not necessarily be excluded from future offerings, UMNP advertisement will focus on venues that have a greater likelihood of attracting amateur naturalists, such as Audubon Society chapters and other grassroots nature organizations, outdoor sporting clubs, Master Gardener chapters, public radio stations, and local newspapers.

The research described in this article has a great potential to benefit Master Naturalist Programs and other volunteer training and citizen science programs throughout Extension. In a time when operational budgets are continually reduced, it is increasingly important to maximize the impacts of Extension programming as well as volunteer training and service. While a single Master Naturalist Program may appear to be limited in scope, each program creates broad, far-reaching impacts within its state. For an example, the Minnesota Master Naturalist Program trained over 500 volunteers who contributed over 56,000 volunteer hours worth over one million dollars in the first 3 years since it was established. Additionally, each successful Master Naturalist program serves as a model for those under development in other states. Through conducting specific assessment, evaluation, and targeting of program audiences, Extension program coordinators may further increase the reach and effectiveness of their programs.

Acknowledgments

I wish to thank Utah State University Extension and the Utah Botanical Center for their support of the UMNP since the beginning. Program development would not have been possible without the assistance of seasonal interns Anna Wilson and David Luecke. Thanks are due to Michael Dietz for help with the data analysis. I would also like to thank all of the agency and organization staff who provided their input during the initial development of the UMNP.

References

Bonneau, L. A., Legg, M. H., Darville, R. L., Haggerty, M. H., & Wilkins, N. (2003). Texas Master Naturalist Program assessment: Final summary of results. Texas Parks & Wildlife and Texas Cooperative Extension.

Broun, C. N., Nilon, C. H., & Pierce II, R. A. (2009). An evaluation of the Missouri Master Naturalist Program and implications for program expansion. Journal of Extension [On-line], 47(3) Article 3FEA5. Available at: http://www.joe.org/joe/2009june/a5.php

Coyle, K. (2005). Environmental literacy in America. The National Environmental Education and Training Foundation. Retrieved from: http://www.neefusa.org/pdf/ELR2005.pdf

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Hungerford, H. R., & Volk, T. (1990). Changing learner behavior through environmental education. Journal of Environmental Education, 21(3), 8-16.

Larese-Casanova, M. (2007). Utah Master Naturalist Program annual report. Utah State University.

Larese-Casanova, M. (2008). Utah Master Naturalist Program annual report. Utah State University.

Louv, R. (2005). The last child in the woods: Saving our children from nature-deficit disorder. New York: Algonquin Books.

Minnesota Master Naturalist Program. (2008). Minnesota Master Naturalist Program formative evaluation. University of Minnesota.

North American Association for Environmental Education. (2004). Environmental education materials: Guidelines for excellence. Washington, DC: North American Association for Environmental Education.

Rager, A., Guiney, P., & Blair, R. (2008). They said what?: Using participant evaluation to mold your program. Presentation at the Alliance for Natural Resource Outreach and Stewardship Program Annual Conference. Sept. 9, 2008.

Savanick, M. A., & Blair, R. B. (2005). Assessing the need for Master Naturalist programs. Journal of Extension [On-line], 43(3), Article 3FEA7. Available at: http://www.joe.org/joe/2005june/a7.php

SPSS, Inc. (2005). SPSS Statistical Package (Version 14.0.1). Chicago.

United Nations Educational Scientific and Cultural Organization-United Nations Environment Programme. (1976). The Belgrade Charter. Connect: UNESCO-UNEP Environmental Education Newsletter, 1(1), 1-2.

United States Census Bureau. (2006). Census data for Utah. Retrieved from: http://quickfacts.census.gov/qfd/states/49000.html

Utah Society for Environmental Education. (2008). Utah Project for Excellence in Environmental Education—K-12/Formal education component: Utah environmental education organization survey: Analysis and report. Salt Lake City.

Van Den Berg, H. A. (2006). Impacts of the Michigan Conservation Stewards Program on participants' knowledge, attitudes, and skills regarding ecology and resource management. Unpublished master's thesis, Michigan State University, East Lansing.

Van Den Berg, H. A., & Dann, S. L. (2008). Evaluation of an adult extension education initiative: The Michigan Conservation Stewards Program. Journal of Extension [On-line], 46(2), Article 2RIB1. Available at: http://www.joe.org/joe/2008april/rb1.php