Evaluating the Effectiveness of Public Television as a Method for Watershed Education

Scientific evidence about environmental problems has increased public concern about these issues and underscores the importance of effective environmental education. There needs to be a better interface between what citizens know and how best to use that knowledge, a concept that joins environmental education with environmental policy and management. This idea, echoed by the National Science Foundation (NSF) (2000), emphasizes a need for increased citizen capacity in making good use of scientific data for sustainable environmental practices: "Scientific understanding of the environment, together with an informed, scientifically literate citizenry, are requisite to improved quality of life for generations to come" (NSF, 2000:51).

Educational programming on environmental issues is a key component of this translational effort. As with all educational efforts, it is not enough to evaluate a program's implementation--the impact of that program must also be evaluated (Seacrest & Herpel, 1997). The assessment must go beyond a mere count of how many stakeholders received the information, typical in Extension programming, toward an understanding of what participants learned from the experience. Effective programming can be a catalyst for behavior change, and, in terms of environmental education, one must think about how the educational program actually protects the environment (Seacrest & Herpel, 1997).

A recent national poll elucidated a disconnection between what the public knows about environmental issues and what they think they know. Results highlighted that 70% of those surveyed claimed to know a moderate amount about environmental issues (National Environmental Education and Training Foundation, 2001), yet the same research reveals that in 1997 fewer than 1/3 of Americans actually had simple knowledge about the environment and that by the year 2000 there had been "virtually no progress" made. This disparity is emphasized in the report, which notes that although public concerns about watershed issues may be justified, they often stem from incomplete knowledge.

Thus, there seems to be consensus on the importance of communicating scientific information about environmental issues to the public, and several authors have suggested mechanisms for achieving the goal. Bruner (1996) feels that the knowledge deficit is better addressed through locally oriented environmental education programs, which establish the basis for the development of a broader understanding of the problem. A common area of local environmental policy relates to water resources, which Freeman (2000) addresses by describing two kinds of knowledge in the water policy arena, generalized scientific knowledge and site-specific knowledge.

Integrating the two realms requires bringing the best science to bear on local problems (Freeman, 2000; NSF, 2000). To do this, there must be an interactive mechanism that assures a two-way flow of information (Randhir, 1999):

• Scientists must be aware of public needs and concerns; and

• Scientific knowledge must be transferred to the public.

The discussion of effective environmental educational programming becomes circular in nature:

1. Scientific information must be translated for the public in order to create an informed citizenry.

2. It is assumed, then, that this informed citizenry will address environmental problems at the local level, prompting "buy-in" and behavior change to protect the environment.

3. However, this protection is fueled by scientific research and monitoring data, bringing us back to the beginning where research is translated for the general public.

The program we describe here developed from a felt need for environmental education by local Cooperative Extension educators wanting to translate and disseminate technical environmental information in a nontraditional manner.

Upper Susquehanna Environmental Education Program

To address the issue of knowledge transfer within a watershed context, we developed a multimedia program about environmental issues in the Upper Susquehanna River watershed in New York State. Collaboration among Cornell University researchers, Extension educators within Cornell Cooperative Extension, and media consultants from WSKG Public Television Station resulted in:

• Six 30-minute television programs broadcast on consecutive Saturday mornings in early Spring, 2000;

• Three 15-minute radio programs aired on public radio stations;

• World Wide Web-based materials; and

• Information in print form placed in public libraries and mailed to homes.

The television programs used live, call-in and pretaped formats discussing:

• Private wells
• Household wastes
• Household chemicals
• Onsite wastewater treatment systems
• Runoff
• Conservation landscaping

For this discussion, we will focus on two of our research objectives:

• Increasing environmental knowledge among participants.

• Increasing commitment by householders to environmental protection.

Methods

An initial survey sample of 871 residents was randomly selected from the tax rolls in two counties in the WSKG viewing area. Summary statistics are given in Table 1. The control group was asked to complete the pre- and post-broadcast surveys but not watch the television programs or utilize auxiliary materials. Following some general questions, the remaining survey items related to the topics noted above. We derived a knowledge scale from each set of survey items for a specific topic area. In this way, we could assess the impact of the individual programs since we also had information about which program(s) the participants viewed.

Our data analysis compared knowledge scale scores pre- and post-broadcast for each topic area. The survey groups were divided into participants who watched a particular program (watcher), participants who did not watch the program in question although they were part of the overall educational project (nonwatcher), and a control group.

In order to know that there was a relationship between the variables that was not a chance occurrence, we looked for a chi-square that was significant at the 0.05 level (Table 2). Chi-square analysis does not provide information about the strength of a relationship. A large chi-square and strong significance level does not necessarily represent a stronger relationship between variables than a smaller chi-square that is moderately significant (Bryman & Cramer, 1990). Rather, a better indicator of the strength of a relationship is the correlation coefficient.

Because the results of our chi-square statistic provided insight as to the existence of a relationship, we examined the data further to determine what factors predicted a particular relationship. We used Ordinary Least Squares (OLS) regression analysis to look for the predictors of knowledge change, such as age, income, educational level, how many shows were watched, and individual knowledge level prior to the broadcast.

To do this, we developed a general knowledge scale across the range of survey items, rather than for each individual topic area. We selected certain items that seemed to be related and measured the reliability of this scale (standardized item alpha). In this case, alpha for our scale indicated internal reliability (>0.60). Our comparison group was low-income, middle-aged males with a high school education who watched only one program.

Another aspect of knowledge change that we measured was how the individual ranked his or her own level of general environmental knowledge pre- and post-broadcast. A survey item asked the participant to rate how knowledgeable he or she was about environmental issues. Choices were "not knowledgeable," "somewhat knowledgeable," and "very knowledgeable." As with the analysis of scores on the knowledge index pre- and post-broadcast, we also used OLS regression to determine what the predictors of change were in terms of how an individual assesses his or her knowledge of environmental issues.

Similar to this self-assessment of knowledge levels, we asked the participants to describe their commitment to environmental protection. We utilized OLS regression to analyze what predicted post-broadcast environmental commitment. Our independent variables were age, gender, education, income, the number of shows that were watched, and environmental commitment prior to the broadcasts. The dependent variable was post-broadcast self-assessed commitment to environmental issues. Again, our comparison group was low income, middle-aged males with a high school education who watched only one program.

Finally, we mailed a brief follow-up survey to our participant group in May 2001, and the response rate was 77%. One of the indicators we sought was the individual's motivation for taking part in our educational program. Respondents were provided 4 choices for their motivation to participate and could circle as many as they wished:

• I wanted to find out more about environmental issues in my area.

• I participated because you offered $10.00.

• I participated because you offered the Home*A*Syst booklet.

• I wanted to help Cornell University and Cornell Cooperative Extension find out more about environmental issues in my area.

Results

We attributed the low response rate of the survey group (19%) to the fact that we asked participants to watch several television programs and complete two surveys, whereas the control group had only to complete the surveys.

One objective of this research was to determine if watching the television programs increased knowledge among participants. Our analysis initially compared the sociodemographic characteristics of the survey group with the control group (Figure 1). We found that our sample group tended to be slightly more educated than the control group, but the two groups were still very similar.

Figure 1.
Sociodemographic Characteristics of Sample and Control Groups*

*median age of both groups was 57 years

Chi-Square

Table 2 provides chi-square results for each group and each individual topic area along with significance levels (p) and correlation coefficients (r). As noted in the methods section, we looked for a significance level of p<0.05 to tell us the existence of a relationship between variables that was not a chance occurrence. The values for significance levels in Table 2 with an asterisk meet this criterion. Newton and Rudestam (1999) suggest that a correlation coefficient of 0.80 represents a strong positive relationship, r=0.50 is a moderate positive relationship, and r-0.20 is a weak positive relationship. The data presented in Table 2 are inconsistent in terms of the effectiveness of our educational program, which led us to examine the data using OLS.

Ordinary Least Squares Regression

By using ordinary least squares regression, we hoped to determine if a particular variable predicted knowledge change. Our data indicated that a significant predictor of knowledge change was pre-broadcast knowledge. The adjusted R² was 0.39, indicating that approximately 1/3 of the variance was explained by pre-broadcast knowledge. Variables that were not significant in our analysis were age, gender, education, income, and the number of shows that were watched. In terms of our participants' self-assessment of knowledge levels pre- and post-broadcast, only participants' pre-broadcast ranking of knowledge was significant to their post-broadcast ranking. The relationship was not strong since the adjusted R² was a very low 0.13. Other variables, which were not significant, were age, gender, education, income, and the number of shows that were watched. The best predictor of environmental commitment following the educational program was how committed the individual was prior to the program (adjusted R² = 0.34).

Follow-Up

Eighty-three percent of the respondents wanted to find out more about environmental issues, and 61% wanted to help Cornell University and Cornell Cooperative Extension. Equal numbers (26% and 27%, respectively) participated because of the financial incentive or the Home*A*Syst booklet.

Discussion

The similarities between the control and survey groups in terms of sociodemographic characteristics allowed us to make conclusions that reflected the general population of the area and to examine the data for trends in environmental knowledge and behaviors. Sociodemographic characteristics such as age, income, education, and gender were not significant in any of our analyses. This is contrary to the findings of the National Environmental Education and Training Foundation (2001), which found that education was the most significant factor in environmental knowledge.

Pre- and Post-Broadcast Relationships--Chi Square

Our overriding goal in this program was to evaluate whether participating in the project increased individual knowledge of environmental issues and heightened levels of commitment to environmental stewardship. The chi-square statistics in Table 2 did not provide clear evidence of trends in the data that support positive impacts of the educational programming. Only the program on septic systems appeared to follow our expectations--that there was a strong relationship between pre- and post-broadcast knowledge scales for the watcher group and no significant relationship between responses pre- and post-broadcast for the nonwatcher or control groups. In terms of the topic area on wells, both the watcher and the control groups demonstrated significant relationships. For the waste topic area, all three groups demonstrated a significant relationship between pre- and post-broadcast surveys.

Our data demonstrate that, in most cases, there is a significant, positive relationship between individual topic area scores pre- and post-broadcast for all watcher levels. Initially, we hypothesized that the relationship between pre- and post-broadcast scores for nonwatchers and control would be less strong than for the watcher group, yet our data seem to diminish the impact of our television programs in terms of knowledge change.

Pre- and Post-Broadcast Relationships--OLS

A more detailed analysis of the data that utilized OLS regression did provide more insight as to the effectiveness of the educational package. Pre-broadcast knowledge level was the strongest predictor of post-broadcast knowledge. This variable was negatively related to knowledge change, indicating that high levels of pre-broadcast knowledge resulted in low levels of knowledge change. This relationship explained more than 1/3 of the variance. Again, the data suggested that our television broadcasts did not achieve the impact we desired in terms of knowledge transfer.

One's self-assessed environmental knowledge level prior to the broadcasts was the strongest predictor of how much knowledge an individual thought he or she had following the educational programming; however, this was not a strong relationship. Our television programs did not appear to influence this self-assessment because the number of programs watched was not a significant predictor of how an individual described his or her knowledge of environmental issues.

Self-assessment was also applied to individual commitment to environmental protection. Continuing the pattern seen in previous regressions, the strongest predictor of post-broadcast commitment was pre-broadcast commitment. Therefore, the data did not indicate that our educational program had a significant impact on how one felt about his or her environmental knowledge or commitment. This finding was important because we postulated that increased knowledge and commitment would increase an individual's desire to behave in a way that promotes environmental sustainability.

Conclusions

These results indicated that little measurable change resulted from our educational program. We concluded that this was due to several factors.

1. Participant group. Although our randomly selected study group and control group were similar in terms of sociodemographic characteristics, they were skewed toward middle-aged, middle-income, moderately to highly educated individuals. This was not surprising because we utilized public television as our conveyance, and this cohort is typical of a public television audience (WSKG, 1999, personal communication). The ideal sample group would have little environmental knowledge or concern prior to our educational program. However, targeting such a group would have required multiple survey waves, which budgetary constraints did not allow.

2. Program time slot. The programs were viewed on six consecutive Saturday mornings, a time slot that required a fairly dedicated population and thus one that was invested in environmental protection from the beginning. In fact, the data show that only 1% of our survey group described themselves as not committed to environmental protection. Given the time investment we expected from our survey group, it was not surprising that those who decided to participate were already committed to environmental protection.

3. Difficulty of survey items/time frame of educational program. We postulated that the educational programs were directed at a level too low to result in significant changes in environmental knowledge, and the time frame was not long enough to see behavioral change. Given the educational levels of our participant group and the fact that many outreach materials are directed to the 8th grade level, we might have underestimated the background of our participants. In terms of behavioral change, the initial survey was in January, and the post-broadcast survey was in May, undoubtedly not enough time for significant behavioral changes to occur.

Our results caused us to consider the utility of public television as an environmental education tool. Cooperative Extension educators had noticed a declining attendance at face-to-face educational events (Kevin Mathers, 1999, personal communication) and felt that video-based materials might be more effective. There are relatively few studies that examine the knowledge gained from educational programming (Shrestha, 1997). Rather, most research relates to how individuals feel about the experience. We have evidence that knowledge is gained if the individuals utilize the materials fully (Wagenet, Pfeffer, Sutphin, & Stycos, 1999), but assuring full utilization is problematic.

The work described here, combined with past research, examined various mechanisms for transferring environmental knowledge. Which mechanism of transfer is the most effective, be it written materials, Web-based materials, or video, is not the central point. Rather, we need to focus on how to generate interest in fully utilizing the materials, especially among those citizens who do not have the initial interest or who might feel that they do not possess the educational background to participate in such programming. Determining the make-up of this audience, however, is a significant challenge due to difficulties in identifying individual audience members.

By assuming that all environmental education opportunities are effective, we can misjudge the needs of the population. Given that a substantial number of our survey group participated to increase their awareness of the environmental issues in their watershed, localizing the environmental education seems to be a necessary first step (Boogerd, Groenewegen, & Hisschemoller, 1997).

As we noted early in this article, Seacrest and Herpel (1997) assume that effective programming can be a catalyst for behavior change. We need specifics on what the required elements are for making this catalysis occur. Our study highlights that the educational methods and program structure that we chose for our particular survey group did not result in substantial knowledge or commitment change.

The National Environmental Education and Training Foundation (2001) report recommends using television in a broader sense than this project undertook, i.e., changing regular television weather reporting into environmental reporting. In addition, the report suggests that environmental education should be integrated more fully into school programs, environmental news coverage should be better supported, and a comprehensive gateway on the World Wide Web should be developed. However, the report also concludes that continued measurement and reporting "on the extent and impact of the lack of adult environmental knowledge" (p. 33) should be conveyed to decision makers.

We agree with these conclusions and strongly feel that more study is needed in the realm of environmental education programming impacts and the linkages to behavior changes that result in environmental protection. The assumption that environmental education "just works" is not necessarily sufficient. Rigorous evaluation provides valuable information for focusing environmental education opportunities and targeting the audiences that would most benefit from the effort.

Acknowledgment

This work was supported in part by United States Department of Agriculture Cooperative State Research, Education and Extension Service Water Quality Initiative funds, under project number 94-EWQI-109059; Cornell Cooperative Extension; Upper Susquehanna Coalition.

References

Boogerd, A., Groenewegen, P.,  & Hisschemoller, M. (1997). Knowledge utilization in water management in the Netherlands related to desiccation. Journal of the American Water Resources Association, 33(4).

Bruner, J. (1996). The culture of education. Cambridge, MA: Harvard University Press.

Bryman, A., & Cramer, D. (1990). Quantitative data analysis for social scientists. London: Routledge.

Freeman, D. M. (2000). Wicked water problems: Sociology and local water organizations in addressing water resources policy. Journal of the American Water Resources Association, 36(3).

National Environmental Education and Training Foundation. (2001). National report card on environmental knowledge, attitudes and behaviors. Washington, D.C.: National Environmental Education and Training Foundation.

National Research Council. (1999). New strategies for America's watersheds. Washington, D.C.: National Academy Press,

National Science Foundation. (2000). Environmental science and engineering for the 21^st century. Washington, D.C.: National Science Foundation NSB00-22.

Newton, R. R., & Rudestam, K. E. (1999). Your Statistical Consultant. Thousand Oaks, CA: Sage.

Randhir, T. (1999). Interactive community decision modeling: Public involvement in watershed policy research. In Eloise Kendy (Ed.), Science into policy: Water in the public realm. Herndon, VA: American Water Resources Association.

Seacrest, S. S., & Herpel, R. (1997). Developing a results-oriented approach for water education programs. Journal of the American Water Resources Association, 33(2).

Shrestha, G. M. (1997). Factors influencing the effectiveness of satellite videconferencing within Cornell Cooperative Extension. Doctoral dissertation: Cornell University, Ithaca, NY.

Wagenet, L. P., Pfeffer, M. J., Sutphin, H. D., & Stycos J. M.. (1999). Adult education and watershed knowledge in upstate New York. Journal of the American Water Resources Association, 35(3).