A Multiyear Evaluation of the NaturePalooza Science Festival

Introduction

Encouraging citizens' environmentally responsible behaviors, such as recycling, composting, using water wisely, supporting local agriculture, and reducing energy consumption, is important for ensuring a more sustainable future. Effecting environmentally responsible behavior is facilitated through increases in connection to nature, science literacy, and agricultural literacy (Heck, Carlos, Barnett, & Smith, 2012; Louv, 2005) as disconnect and lack of literacy are barriers to environmental sustainability behaviors (Kollmuss & Agyeman, 2002).

Extension works to connect citizens to the environment, improve science literacy, and increase sustainability behaviors through public programming (Comer, Campbell, Edwards, & Hillison, 2006; Warner, Rumble, Martin, Lamm, & Cantrell, 2015). Though Extension professionals have measured the impact of specific programs on sustainability behavior (Diehl, Swenson, & Wente, 2012), most find that task time-consuming or difficult, especially with regard to multiyear programs (Chazdon, Horntvedt, & Templin, 2016). This circumstance leaves unanswered questions about the effect of public programming on sustainability behavior. Yet it is increasingly important for those of us working in Extension to provide evidence of engagement and impact related to our work.

Because of the importance of measuring the impact of programming on sustainability behavior, we undertook a multiyear evaluation of NaturePalooza, a University of Nebraska–Lincoln (UNL) Extension program designed to address connection to nature, science literacy, and sustainability behaviors. Our twofold purpose was both to improve the event and to measure its impact on local citizens' science literacy and sustainability behaviors. Our results offer encouragement that a thoughtfully designed annual Extension event can positively impact participants' science literacy and relevant behaviors.

Event Description

Our team in UNL Extension initiated the annual NaturePalooza event in 2011 with the purpose of showcasing the work of professionals in the School of Natural Resources. In 2013, the focus shifted to increasing science literacy, educating locals about sustainability, and encouraging environmental sustainability behaviors. In 2014, we began a process evaluation aimed at improving future NaturePaloozas. This process evaluation has involved our conducting annual visitor surveys, scrutinizing the results to identify opportunities for improvement, and making changes intended to increase impact.

One modification intended to improve science learning among attendees was organizing the event thematically, as thematic learning increases comprehension for event learners (Ham, 2013). Five themes served as organizational anchors: habitat for native species, water-use reduction and water quality, waste management and recycling, air quality and climate, and soil management. An additional theme, outdoor skills, was added in 2015. At the event, thematic science content is presented in a fun, family-friendly way through face-to-face interaction with scientists, hands-on activities, educational games, science experiments, and wildlife demonstrations.

Additionally, in 2014, we added the Know-it-All activity to improve and evaluate visitor learning. Know-it-All is based on tested educational and interpretation theory advocating thematic programming (Ham, 2013) and critical thinking (Ainsworth & Eaton, 2010). For the activity, visitors are given a punch card as they enter the event. As visitors stop at booths related to each of the five event themes, they can be quizzed by volunteer Know-it-Alls. The quiz questions test visitor knowledge of category content. Upon answering a question correctly, a visitor receives a punch on his or her punch card signifying a successful learning activity. When the punch card is filled, the visitor can claim a prize.

One measure of success for NaturePalooza is visitor numbers. Overall, the event has steadily grown since its inception. For example, it drew 605 visitors in 2011, 423 in 2012, 830 in 2013, 1,100 in 2014, and 1,350 in 2015.

However, the true success of the event must be measured in impact on visitors, including visitor learning and behavior change. Therefore, in 2014, we began addressing visitors' learning, behavioral intentions, and implemented behavior changes, in addition to gathering information about visitor satisfaction, needed improvements, and event marketing effectiveness. The survey instruments were specifically designed to measure knowledge and behavioral intentions (survey conducted at the event) and actual behavior (survey conducted 6 months after the event).

Methods

At the time of this writing, we had completed three surveys: the 2014 NaturePalooza survey, the 2015 NaturePalooza survey, and the 2015 NaturePalooza 6-month follow-up survey. All surveys were created with Qualtrics software.

2014 NaturePalooza Survey

At the 2014 NaturePalooza, volunteers using iPads asked every fourth exiting visitor to complete an online survey as visitors left the event. This approach provided a small random sample of adult event visitors (older than age 19 as per institutional review board approval). Survey questions addressed the aforementioned concepts.

• Amount of new information learned was measured with the question "How much information at NaturePalooza was new?" Answer choices were as follows: (a) everything was new, (b) I learned many new things, (c) I learned many new things but knew some already, (d) I learned 1–3 new things, and (e) I already knew everything, but it helped to reinforce things.
• Intention to implement new sustainability behaviors was assessed with the question "How many of the behaviors that you learned about at NaturePalooza (like recycling and saving water) will you try?" Answer choices were as follows: (a) My family and I will try more than 5 new behaviors, (b) My family and I will try 2–4 new behaviors, (c) My family and I will try 1 new behavior, and (d) My family and I will not try any new behaviors.
• Sustainability behaviors implemented as a result of information learned at previous NaturePaloozas was assessed by one multiple-choice question having answer choices similar to those listed above.
• Prior NaturePalooza attendance was addressed with yes/no items.
• Demographic data were collected through two multiple-choice questions and one open-ended question.
• Survey questions not used in the study reported here included qualitative items (three items) and items addressing attitudes toward science (nine items), environmental attitudes (14 items), outdoor recreation attitudes (10 items), specific sustainability behaviors (13 items), and event marketing (one item).

2015 NaturePalooza Survey

A similar survey was conducted at the 2015 NaturePalooza. All questions described above were included, with some edits. Participants also were asked for an email address for the 6-month follow-up survey. Surveys were distributed and collected through the same means as at the 2014 event; that is, volunteers using iPads asked every fourth exiting visitor to complete the online survey as visitors left the event.

2015 NaturePalooza 6-Month Follow-Up Survey

Six months after the 2015 event (March 2016), an online follow-up survey was distributed to participants who had submitted email addresses during the 2015 on-site survey. The survey was distributed according to a modified version of Dillman's survey protocol (Dillman, 2000; Dillman, Smyth, & Christian, 2014). The survey addressed how much information participants had learned at the 2015 event, how many new sustainability behaviors they had implemented, and demographics.

Data Analysis

We used SPSS software version 22.0 to analyze the survey data. We evaluated the data from each event independently, assessing demographics and measuring visitors' amounts learned, intentions regarding sustainability behavior changes, and actual behavior changes. In addition, we combined the data from both years' events to measure changes in visitor demographics, behavioral intention, and behavior change from year to year.

Results

We conducted 92 surveys at the 2014 event and 77 at the 2015 event. The 2015 6-month follow-up survey emailed to respondents from the 2015 event yielded 27 responses (35% response rate). Demographic characteristics of each survey population are displayed in Table 1. If respondents did not answer some or all of the demographic questions at the end of the survey but answered the other questions, we included their responses in our analysis; a number of respondents left demographic questions unanswered.

Table 1.
Demographic Characteristics of NaturePalooza Survey Respondents

Demographic characteristic	2014 on-site survey respondents (n = 92)	2015 on-site survey respondents (n = 77)	2015 follow-up respondents (n = 27)
Gender
Male	18 (20%)	24 (31%)	5 (19%)
Female	60 (65%)	46 (60%)	20 (74%)
Race/ethnicity
Caucasian	65	62	23
African American	1	2	0
Latino/Hispanic	1	2	1
Other	10	7	1
Age
19–29	13	10	0
30–39	34	38	16
40–49	10	17	10
50–59	6	4	1
60–69	9	3	0
70+	0	2	0
Note. Values may not sum to the total number of respondents, as surveys with missing demographic information and completed questions were accepted for analysis.

In 2014, 61 respondents (73%) were first-time NaturePalooza attendees. At the 2015 event, 54 respondents (71%) were first-time visitors, 13 (17%) had attended one previous NaturePalooza, and 9 (12%) had attended two previous NaturePaloozas.

New Information Learned

The majority of respondents from each survey reported learning new information at NaturePalooza; 71 of the 2014 survey respondents (77%), 71 of the 2015 survey respondents (92%), and all 27 of the 2015 follow-up survey respondents (100%) reported learning new information. More specifically, participants were asked how much new information they learned; results are presented in Table 2.

Table 2.
Amounts of New Information Learned

Amount learned	2014 on-site survey respondents (n = 92)	2015 on-site survey respondents (n = 77)	2015 follow-up respondents (n = 27)
All information at the event was new	7 (8%)	16 (21%)	2 (3%)
Learned 4 or more new things	35 (38%)	24 (31%)	4 (15%)
Learned 1 to 3 new things	29 (32%)	31 (40%)	21 (78%)
Already knew all the information presented	6 (7%)	2 (3%)	0

Behavioral Intentions and Behavior Changes

Survey participants at the 2014 and 2015 events were asked about their intentions to implement sustainability behaviors, such as recycling, composting, and saving water. Overall, 78 visitors in 2014 (85%) and 78 in 2015 (68%) planned to implement at least one new behavior. Table 3 shows the specific numbers of new conservation behaviors respondents intended to try.

Table 3.
Intention to Implement New Sustainability Behaviors

Intention level	2014 on-site survey respondents (n = 78)	2015 on-site survey respondents (n = 69)
Intend to try 5 or more new behaviors	16 (17%)	4 (6%)
Intend to try 2 to 4 new behaviors	46 (50%)	23 (33%)
Intend to try 1 new behavior	11 (12%)	25 (36%)
Do not intend to try a new behavior	5 (5%)	2 (3%)
Already practicing all behaviors presented	0	15 (22%)

We used two methods to collect self-reports of actual behavior change. First, to determine whether 2015 NaturePalooza attendees who were returning visitors had followed through with implementing new sustainability behaviors, we asked 2015 event survey respondents whether they had tried any behaviors learned at previous NaturePaloozas. Second, we asked 2015 6-month follow-up survey respondents whether they had followed through with their intentions to try new conservation behaviors. Results for both groups are shown in Table 4, at the end of this section.

With regard to 2015 returning visitors (n = 42), 28 of the 2015 survey respondents (66%) reported implementing new behaviors, and only four (10%) reported not trying any new behaviors. The difference between the total means (responses were combined to form total indexes) of those who intended to try a behavior in 2014 (M = 2.72, SD = .73) and those returning visitors who reported implementing new behaviors in 2015 (M = 2.06, SD = .78) proved significant through use of an independent t-test, t(108) = -4.078, p < .001. Significantly fewer behaviors were implemented than intended. However, the means analyzed were means from two representative samples rather than means from the same group of people.

In our 2015 6-month follow-up survey, 15 respondents (62%) reported actually implementing a new behavior. We conducted an independent t-test to test for differences between the total behavioral intentions visitors reported (we combined responses into total indexes) at the 2015 Naturepalooza (M = 2.64, SD = .73) and the total new behaviors implemented by respondents who completed the 6-month follow-up survey (M = 2.54, SD = 7.2). The t-test was not significant, t(91) = .558, p = .578, indicating that 2015 visitors' intentions did not differ significantly from the number of behaviors implemented; therefore, visitors actually implemented roughly (not significantly different) the number of new behaviors they intended to implement.

Table 4.
Implementation of New Sustainability Behaviors

Implementation level	2015 returning visitor respondents (n = 42)	2015 follow-up respondents (n = 24)
Tried 5 or more new behaviors	1 (2%)	2 (8%)
Tried 2 to 4 new behaviors	11 (26%)	5 (21%)
Tried 1 new behavior	16 (38%)	8 (33%)
Did not try a new behavior	4 (10%)	1 (4%)
Was already practicing all behaviors presented	10 (24%)	8 (33%)

Conclusion and Implications

The results of our three surveys are encouraging, showing that the annual NaturePalooza event increased self-reported science literacy and sustainability behaviors for attendees. The majority of visitors surveyed reported learning new information (77% in 2014, 92% in 2015, and 100% for the 2015 follow-up survey). Sixty-six percent of returning visitors surveyed in 2015 and 62% of 2015 follow-up survey respondents reported actually implementing at least one new sustainability behavior as a result of attending NaturePalooza. This finding is important as most research measures intention and not actual behavior change (Webb & Sheeran, 2006). In this case, we acknowledge the possible errors associated with using self-reports of behavior change instead of more objective measures of change (e.g., monitoring electric and water meters or weighing recycling). However, reporting objective measures is both difficult and beyond the scope of a simple process evaluation addressing the outcomes of an event. We also recognize the limitations our small sample sizes pose. Although random exit surveys were conducted, the samples were small percentages of the total numbers of adults at the events.

Analysis of data from the surveys conducted at the 2014 and 2015 events indicated that attendees had implemented significantly fewer sustainability behaviors than they intended to implement. This finding is unsurprising, as intent to change behavior and behavioral impact have long been recognized as different phenomena (Stern, 2000). However, even though fewer behaviors were implemented, the majority of return visitor respondents at the 2015 event did report having implemented at least one new behavior (f = 28, 66%), indicating that the 2014 event did have a positive, measurable influence on behavior.

Results of the 2015 6-month follow-up survey showed a different relationship. There was no statistically significant difference between visitors' behavioral intent measured at the 2015 event and implementation of behaviors 6 months later, meaning that an equivalent number of behaviors were intended as those implemented 6 months later. This finding is positive news, as it suggests that the event is achieving the intended goal of getting visitors to implement behavior change. It also implies that the 2015 event sustained more behavior change than the 2014 event, at least 6 months past the event. The 2015 event may have achieved more impact due to effective implementation of thematic learning stations, as most visitors surveyed (77%, 92%, and 100%) learned new information. It could also be that the annual nature of and increasing crowds at the event indicate to visitors that the sustainability behaviors and attitudes advocated are favorable to many local citizens, thereby becoming the norm. Indeed, norms are a powerful motivator of behavior (Stern, 2000).

On the basis of our study, we conclude that enjoyable, thematic, hands-on, and inclusive Extension events can positively affect visitors' science literacy and sustainability behaviors. In this age of accountability, our findings show that short-term presence at a family festival not only results in good public relations for Extension but also can produce meaningful impacts. Therefore, hosting these annual events and monitoring impacts, especially through the use of follow-up questions and surveys, and making associated adjustments to improve learning are important endeavors.

References

Ainsworth, H. L., & Eaton, S. E. (2010). Formal, non-formal and informal learning in the sciences. Calgary, Canada: Onate.

Chazdon, S., Horntvedt, J., & Templin, E. (2016). From knowledge to action: Tips for encouraging and measuring program-related behavior change. Journal of Extension, 54(2), Article 2TOT1. Available at: http://www.joe.org/joe/2016april/tt1.php

Comer, M., Campbell, T., Edwards, K., & Hillison, J. (2006). Cooperative Extension and the 1890 land-grant institution: The real story. Journal of Extension, 44(3), Article 3FEA4. Available at: http://www.joe.org/joe/2006june/a4.php

Diehl, C. D., Swenson, S. E., & Wente, J. N. (2012). Evaluation of a sustainable green living expo event: Attendees' reports of satisfaction, leaning, and behavior change. Journal of Extension, 50(3), Article 3FEA8. Available at: http://www.joe.org/joe/2012june/a8.php

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York, NY: John Wiley and Sons.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). New York, NY: John Wiley and Sons.

Ham, S. (2013). Interpretation: Making a difference on purpose. Golden, CO: Fulcrum.

Heck, K., Carlos, R., Barnett, C., & Smith, M. (2012). 4-H participation and science interest in youth. Journal of Extension, 50(2), Article 2FEA5. Available at: http://www.joe.org/joe/2012april/a5.php

Kollmuss, A., & Agyeman, J. (2002). Mind the gap: Why do people act environmentally and what are the barriers to pro-environmental behavior? Environmental Education Research, 8(3), 239–260.

Louv, R. (2005). Last child in the woods: Saving our children from nature-deficit disorder. Chapel Hill, NC: Algonquin Books.

Stern, P. C. (2000). New environmental theories: Toward a coherent theory of environmentally significant behavior. Journal of Social Issues, 56(3), 407–424.

Warner, L. A., Rumble, J. N., Martin, E., Lamm, A. J., & Cantrell, R. A. (2015). The effect of strategic message selection on residents' intent to conserve water in the landscape. Journal of Agricultural Education, 56(4), 59–74.

Webb, T. L., & Sheeran, P. (2006). Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychological Bulletin, 132(2), 249–268.