Effecting Land-Use Changes Through Education and Implementation: Assessing the Effectiveness of the Watershed Stewards Program

Introduction

Maine is a water-rich state. With over 5,000 scenic lakes attracting thousands of tourists to the state each year, it is essential to protect this resource. In 1996, the University of Maine Cooperative Extension (UMCE) initiated the Watershed Stewards Program to educate people about threats to Maine lakes and actions to prevent damage to lake water quality. Designed in a manner similar to the Master Gardener Program (Simonson & Pals, 1990), participants attended 20 hours of education and in return provided 20 hours of service to their lake watershed.

Participants attended seven (once per week) 2.5-hour training sessions. The program relied on the assistance of our state partners, including the Maine Department of Environmental Protection, Maine Department of Agriculture, Food and Rural Resources, and Natural Resources Conservation Service personnel, in delivery of these courses. Non-point source pollution is such a threat to Maine lakes (Chesters & Schierow, 1985), and course topics included:

• Non-point source pollution;
• Lake biology and chemistry;
• Septic systems and their impacts on lake water quality;
• Development methods to reduce impacts to water quality;
• Road maintenance methods;
• Water quality testing methods and trends in lake water quality;
• Regulatory issues;
• Land-use management: buffers, lawn care, and gardening methods; and
• NEMO (non-point education for municipal officials).

One additional hands-on field training was conducted to teach people how to recognize sites that have the potential to increase soil (and phosphorus (P)) loading to the lake. A map showing the lake watersheds where the program has been conducted is found in Figure 1.

Figure 1.
Watershed Stewards Program Locations, 1997 - 2002

Purpose of the Study

Having conducted the course for a 7-year period, it was appropriate to assess impacts and to determine areas to improve or redirect our program. Program evaluation is increasingly a fundamental part of being an effective Extension educator (Scarborough, Killough, Johnson, & Farrington, 1997). Given tight budgetary times, solid evidence of program impact is also increasingly important (Van den Ban & Hawkins, 1996; Shepard, 2002). Documenting solid impacts and program successes would increase our opportunities to continue program funding. Therefore, the goals of this project were to:

1. Conduct a statistically valid sampling of participants to assess program impacts and

2. Compare knowledge levels, awareness of, and participation in lake management activities with people who live in the same watersheds but who have not participated in our programs.

Specific Research Questions

1. Did the UMCE water quality program significantly improve the knowledge level of program participants over non-participants?;

2. Did participant knowledge remain strong over time?; and

3. Were program participants more knowledgeable and actively involved in lake protection efforts than non-program participants living in those same lake watersheds?

Methods: Quantitative Assessment

To answer research question number one, a short 15-question test was developed. This test was first piloted on a small group of lakefront landowners. The test was sent to 161 program participants (stewards) and to approximately twice that number (328) of people on the same lakes who did not participate (non-stewards). Anticipating a better response from stewards, an attempt was made to increase project participation from non-stewards by doubling the number of non-stewards who received the test and by offering an incentive (framed wildlife photo) for participation. We obtained non-stewards addresses from lake association mailing lists.

This test was designed to test knowledge of general lake biology, land-use impacts on water quality, and similar issues that have been covered both in our programs and lake-management awareness campaigns of our partner agencies. Our hope was to determine if program participants had a significantly higher knowledge level than other lake landowners. The top score possible was 65 points, with each question worth five points. Most questions were multiple choice, but others were fill in the blank or yes/no in nature. Some questions had only one correct response possible, while others had multiple answers. Partial credit was possible. The entire test can be found at www.umaine.edu/waterquality.

Typical exam questions included the following:

• Typical sources of phosphorus to Maine lakes include (circle all that apply):
  a) atmospheric deposition, b) aquatic plants, c) septic systems, d) bottom sediments, e) urban runoff, and f) fertilizers.

• The tool used to measure lake clarity is called a:
  a) piezometer, b) secchi disk, c) yard stick, d) GIS (geographic information system) e) I don't know

• Is nitrogen a common limiting nutrient in Maine's lakes?
  ___ yes  ___ no ___ don't know.

There was 22% participation (70 out of 328) with non-program participants (non-stewards) and 39% response rate (64 out of 161) with program participants (stewards). The test was graded, and the scores were subjected to a T-test analysis to assess whether the populations were different (SAS, 2001).

Qualitative Assessment

Standard testing is an effective means to assess knowledge differences, but it is difficult to determine if one group is playing a more active role in lake protection efforts or working more closely with neighbors on lake issues. It can also be argued that standard objective tests do not allow one to further tease out differences in knowledge levels.

To more completely assess group differences (stewards from non-stewards), a standard scripted telephone survey was designed and conducted. This was an eight-question survey with multiple parts. Fifty people from each group were randomly identified as potential interviewees. Phone numbers of non-stewards were obtained from lake association contact lists. The first 25 people from each group who agreed to participate were interviewed. An individual not involved in the educational program effort conducted the surveys.

Each interview took approximately 15 minutes, and responses were entered into individual Microsoft Word files. Because most of the responses were subjective in nature, data were analyzed using qualitative data analysis software called NUDIST. This software allows one to search through multiple documents and identify trends, patterns, and similarity of responses.

Typical interview questions included the following:

• Who do you think is responsible for protecting your lake?

• Have you personally taken action to protect the lake?

• If so, what have you done

• If you haven't, do you intend to?

• How many hours per year do you spend on lake education/protection activities?

• What is the greatest threat to your lake?

Major Findings--Objective Testing

Results indicate that stewards had significantly higher understanding of lake biology, knowledge of threats to the lake, and appropriate steps to protect lake water quality. The mean score for stewards was 47, and for non-stewards, 33. Overall, stewards scored 23% higher on the objective test than those that did not been involved in the program. This difference was statistically significant (p<0.001) (Figure 2).

Figure 2.
Bar Chart of Test Scores

The Watershed Stewards Program is offered to all individuals, regardless of socioeconomic and demographic factors and educational background. In order to test whether stewards enter the program with a higher knowledge level than non-stewards and to ensure that apparent knowledge differences were gained in the program, the test was administered to the newest Watershed Stewards class (Pemaquid) during their opening session. Their class average (sample size of 12) was 33, the same average obtained by non-stewards in the mailed test. Although direct pre- and post-test measurements were not performed, the new group's pre-test scores indicate that people taking our program are not entering with a higher initial knowledge level. This indicates that the higher test scores by stewards reflected knowledge gained in the program.

To determine if knowledge gained in our program was stable over time, a regression analysis was performed on program data. The slope of the regression line was not significantly different from zero (pr>(t) = 0.754), indicating no loss of participant knowledge over time. In fact, the average scores from the first program delivered in 1996 were among the highest (see Figure 3).

Figure 3.
Average Test Scores Over Time

Program participants scored higher than non-stewards did on each question. Some interesting differences between stewards responses compared to their counterparts were found. For instance, 84% of stewards recognized that additives do not increase the life of a septic system while only 46% of non-stewards provided the correct response. Another interesting difference was that 98% of stewards knew that camp roads were a major source of P to Maine lakes, a fact that only 63% of non-stewards recognized. Thus, the quantitative test was instructive in determining differences between trained and non-trained lake residents.

Major Findings--Phone Interview

The interview process provided detail on program participants' activities and allowed some redirection of programming efforts, because many non-stewards were more involved or more knowledgeable about specific areas than had been initially considered. Some of the more interesting responses are presented later in this article.

Both groups were asked if they knew about efforts made in their lake to control polluted runoff in the lake. Sixty-eight percent of non-stewards were aware of efforts to control polluted runoff in their lake. In contrast, 98% of stewards interviewed were aware of activities to protect the lake from polluted runoff. The quality of the responses that stewards gave was much more complete. Of those non-stewards who gave detailed responses, only three actually talked about activities that would actually reduce the amount of polluted runoff reaching their lake. Below a few of the types of responses stewards gave are compared to those of non-stewards (Table 1):

Table 1.
Group Responses to Efforts Made to Control Polluted Runoff to Lakes

Non-Stewards	Stewards
Planting a barrier to prevent road runoff	Abated long-standing road bridge combination runoff
Working on erosion control committee	Implementing shoreland buffers and improving roads
Contributing money to install culverts	Working on an on-going 319 project improving culverts, ditching, and planting buffers
	Working with construction companies who are increasing runoff from construction
	Trying to get rid of runoff problems
	Participated in field studies to identify sources of polluted runoff
	Yes -- road rip-rapping and we planted new vegetation
	Worked on demonstrations for erosion control

In another interesting question, both groups were asked how many hours per year they regularly spend on lake education and protection activities. Stewards are clearly more active in the lake (Table 2).

Table 2 provides the best evidence that the program has motivated people to take action in their lake watershed. For the most part, the stewards know what the problem is, why it is important, and how to correct it. Most important, they are doing it.

Table 2.
Group Responses to Hours per Year Spent on Lake Education and Protection Activities

Hours	# of Non-Stewards	# of Stewards
Less than 10	16	3
10-20	1>	6
20-30	1	2
30-40	1	3
40-50	--	2
50-100	--	3
More than 100	--	2
Other (too few, not necessary)	2	2

When both groups were asked what were the greatest threats to their lake/Maine lakes, the responses were reasonably similar (Table 3).

Table 3.
Greatest Threat to Your Lake/Maine Lakes (Number of Respondents)

Non-Stewards	Stewards
Jet skis (2)	Invasive aquatic plants (10)
Bass	Road runoff « camp roads (4)
Growth, development, clear-cutting, lack of government intervention, and runoff.	Phosphorus (2)
Invasive plants/milfoil (7)	Nonpoint source pollution (3)
Runoff, fertilizing lawns (5)	Apathy
Acid rain	Development (3)
Chicken farms in area	Ignorance (2)
Man	Lack of clarity
Fertilizing lawns	Too many people (2)
Septic systems	Not enough shoreland zoning
Ducks and flooding
Pollution
Not sure
Don't know
Phosphorus

This strong degree of similarity is likely due to the amount of information about aquatic invasive plants that has been in the Maine news and due to the new state legislation passed last year against bringing boats into the state carrying any vegetation. The major difference between the responses of the two groups appears to be that the stewards viewed the whole issue of development as the major problem. Non-stewards had many different types of responses.

The question that really differentiated the groups was, "Do you now, or have you ever served on your lake association board, town environmental planning board, or other community group that deals with environmental issues?" In summary, less than 25% of non-stewards had participated, while 92% of stewards responded favorably. In these cases, the predominant participation was in lake association roles, town selectmen, and other related positions.

The qualitative assessment was useful to better gain an understanding of what average lake landowners know and do about lake water quality. With just a single test approach, the involvement of the stewards compared to other lake residents would not have been appreciated. Another discovery that would have been missed is that the other landowners appear to have an interest in lake protection, even though they don't have the knowledge and skills to participate fully.

Conclusions

The Watershed Stewards Program has been successful. By almost all indications, the knowledge base of stewards has improved, they know which actions are required to correct problem areas, and they are taking leadership roles in their local lake associations and town boards to more effectively improve lake water quality. State partners (Maine Department of Environmental Protection, Conservation District staff, and many others) were essential to help make this program successful. Given the apparent effectiveness of this program, educational efforts in other Maine lake watersheds will occur. Based on our experiences, this two-tiered approach to program evaluation can help other Extension personnel assess their programmatic impacts.

Acknowledgments

We would like to thank Hilarie Wilson, who conducted our phone survey, and Dr. Chris Reberg-Horton, for his statistical assistance.

References

Chesters, G., & Schierow, L. (1985). A primer on non-point pollution. Journal of Soil and Water Conservation , 40(1), pp. 9 - 13.

QSR NUD*IST (1997). Qualitative solutions and research PTY ltd.

SAS Institute Inc. (2001). The SAS system for windows (Release 8.0). Cary, NC.

Scarborough, V., Killough, S., Johnson, D., & Farrington, J. (Eds.). (1997). Farmer-led extension. London: Intermediate Technology Publications, Ltd.

Shepard, R. (2002). Evaluating Extension-based water resource outreach programs: are we meeting the challenge? Journal of Extension [On-line] 40(1). Available at: http://joe.org/joe/2002february/a3.html

Simonson, D. L., & Pals, D. A. (1990). Master Gardeners: Views from the Cabbage Patch. Journal of Extension [On-line] 28(2). Available at: http://www.joe.org/joe/1990summer/rb3.html

Van den Ban, A. W., & Hawkins, H. S. (1996). Agricultural Extension (2nd ed.). Cambridge, Massachusetts: Blackwell Science Ltd.