Evaluating the Effectiveness of a Grant-Funded Educational Program Aimed at Increasing Native Seed Collections in Nevada

Introduction

Wildfires in the United States are growing in number and size. In the period between 2000-2005, the average number of acres burned annually was 6,551,749, higher than any decade since the 1950's (National Interagency Fire Center, 2006). This massive wildfire trend is expanding and may accelerate due to global warming (Westerling, Hidalgo, Cayan, & Swetnam, 2006). In 2006, the United States Government Accountability Office reported that "the number of acres burned by wildfires annually from 2000 to 2005 was 70% greater than the average burned annually during the 1990's" (Nazzaro, 2006). According to the National Year-to-Date Report on Fires and Acres Burned, over 6 million acres were burned by wildfire in the 11 western states in 2006 (National Interagency Fire Center, 2007).

Revegetation of burned lands is critical. Responsible agencies are increasingly developing policies that encourage the use of native plant species in these efforts (Richards, Chambers, & Ross, 1998). This trend is echoed by state agencies. Because federal agencies control much of our nation's public lands, ranging from a high of 86% in Nevada to a low of 30% in Montana (General Services Administration, 2004), this revegetation policy is crucial.

Unfortunately, the increased demand for native seeds has not been matched by supply. Between 1996 and 2000, the price paid by the Bureau of Land Management (BLM) for native seeds tripled (General Services Administration, 2002). Congress responded, and, in 2002, the Interior Appropriations House directed "the Secretaries of Interior and Agriculture to report jointly to the Congress by December 31, 2001 with specific plans and recommendations to supply native plant materials for emergency stabilization and longer-term rehabilitation efforts." In 2002, the BLM and Forest Service appropriated approximately $15 million for the development of programs to increase the supply of native plant seeds.

In late 2002, personnel from the BLM state office in Nevada contacted Cooperative Extension with a request to submit a proposal aimed at the development of an educational program to increase native seed collections from lands in the state.

Because formal evaluations of grant-funded programs is becoming increasing more important to federal granting agencies as one tool to address the accountability standards demanded by Congress, the authors created and administered an evaluation survey. This article provides an example of a formal evaluation that was completed as a required activity in the grant aimed at increasing native seed collection activity on federal lands in Nevada.

While the evaluation tool and resulting data are not unique examples of program evaluation, it is interesting to see how perceived usefulness of the educational tool can be correlated to knowledge gain. Extension professionals are increasingly being asked to report these program impacts. The process used in evaluating this educational program may prove useful to other Extension professionals contemplating use of surveys as an evaluation tool.

Educational Program Development and Delivery

The proposal outlined four main objectives that were agreed to by the BLM and Cooperative Extension. The objectives addressed in the project included:

• Writing, publishing, and distributing a native seed collection manual in Nevada and other western states, as appropriate.


• Presenting the manual at conferences, technical sessions, and scientific meetings during 2003 and 2004.


• Completing a series of workshops in Nevada aimed at educating potential seed collectors about the seed collecting process using the manual as curriculum.


• Conducting a formal evaluation survey to determine the utility of the manual and to estimate the change in seed collecting activity from the responses.

A 135-page manual titled Field Guide for Collecting Native Seeds in Nevada was published in 2003. The manual was distributed to 458 individuals in 13 states. Following publication, it was presented at three scientific conferences in Idaho, Utah, and West Virginia, and two technical sessions in California and Colorado. It is available on the University of Nevada Cooperative Extension Web site: <http://www.unce.unr.edu/publications/files/nr/2003/EB0303.pdf>.

An educational team consisting of representatives from Nevada Cooperative Extension, the Bureau of Land Management, Nevada State Department of Agriculture, and two representatives from the seed collection industry was assembled. The team developed workshop materials based on the manual and taught five workshops throughout Nevada to nearly 200 individuals. Workshop attendees included agency staff, Native American tribal members, and the general public. Approximately 600 individuals were exposed to the manual during 2003 to 2005 via direct and indirect contact.

Evaluation of the first three objectives consisted of documenting manuals distributed, recording presentation sites, and recording the number of workshop attendees. The fourth objective was met using a mail-out evaluation survey that was given to manual recipients to ascertain the impact of the educational information presented in the manual. The survey purpose was to measure knowledge gained, manual utility, the sharing of information beyond workshop attendees, and change in seed collecting activities.

Methods

The evaluation consisted of a 23-question survey that was prepared by the authors and mailed to 186 individuals who had received the manual and had provided an address. A follow-up postcard was mailed 3 weeks after the first mailing reminding recipients to fill out the survey. Fifteen of the surveys were returned as undeliverable, resulting in a survey population of 171 people. A total of 53 valid responses were received from individuals in 10 states, giving a 30% response rate. Other mailed surveys targeting this agriculture and natural resource audience are showing similar rates of response, (Singletary & Smith, 2006; Smyth & Dillman, 2007).

The survey was used to describe the population of individuals interested in collecting native seed for Nevada and to evaluate the manual effectiveness. Respondents were queried about the type of organization they worked for, whether seed collecting was a primary source of their income, and their state of residence. Respondents answered two different Likert-type scales measuring knowledge gain and usefulness of the publication.

The first was a five-point Likert-type scale administered in the survey with 1) being strongly disagree and 5) being strongly agree. This scale was used to measure knowledge gained as a result of using the manual, whether respondents had begun or had increased their collection activity as a result of using the manual, and whether the manual was considered user friendly.

The second scale asked respondents to rate the "usefulness" of each of the nine chapter topics contained in the publication. Respondents completed a Likert-type scale (Dillman 2007) using a five-point response structure with 1 indicating "not useful" to 5 indicating "extremely useful." The nine topics respondents rated were:

1. Plant collecting priority list;


2. Native plant identification guides;


3. Potential native seed collection sites in Nevada;


4. Federal permits, fees, and regulations related to seed collecting;


5. Nevada state rules and regulations related to seed collecting;


6. Collecting and preserving voucher plant specimens;


7. Native seed collection, handling, and storage;


8. Marketing native seed; and


9. Appendices.

Descriptive statistics software (SPSS 14.0 Software, 2006) was used to analyze survey results. Regression analysis measured the influence of perceived usefulness of the topics in the publication on the perceived knowledge gain of reviewers related to native seed collection. This analysis was conducted to provide information to the developers of the publication regarding predictors of knowledge gain with regard to the nine topics surveyed.

Results

The respondents were asked to provide information on their employment status. Approximately 41% of those who responded to the survey were federal, state, county, or city employees. Private businesses, such as consulting, sales, seed collector and/or seed cleaning, and farming and ranching, comprised approximately 40% of the respondents. In addition, 19% of the respondents did not provide an answer to this question.

Cronbach's coefficient alpha was used to estimate internal consistency of the nine Likert-type scale items representing the ratings of usefulness of the manual. The Cronbach score was high (r = .84) and indicates that there was high internal consistency between the skill items (Carmines & Zeller, 1979).

Usefulness of Guide and Knowledge Gain

The respondents were asked to evaluate whether the manual was easy to use, improved their knowledge, and increased their activities relative to native seed collection. The responses were positive and are displayed as ranked mean scores in Table 1 below.

For the purposes of the study, mean scores were calculated for each of the nine chapter topic areas. Table 2 illustrates the ranked mean scores for the nine chapter topic areas. As shown in Table 2, the most useful topics included the native plant identification guides. Potential native seed collection sites rated last in the ranking of mean scores for the nine topic areas.

When respondents were asked whether the information in the manual had improved their knowledge of native seed collection, 73% agreed, while 20% reported neutral feelings, and 7% disagreed.

Increased Seed Collection/Sales

An important question on the survey asked the respondents if the manual had helped them increase their seed collecting activities. The results indicated that 31% of the respondents had increased their seed collecting activities as a result of receiving the manual, 55% indicated the manual did not help increase collection activity, and approximately 14% did not answer. The respondents who indicated the manual was helpful to them in increasing their seed collecting activity were asked to estimate how many pounds of seed were collected. Fifteen individuals indicated they collected 1-500 pounds, while two respondents indicated they had collected from 501-1,000 pounds, and one other individual indicated collecting in excess of 1,000 pounds of native seed.

The majority (73%) of respondents did not indicate where they had sold the collected seed. Of those responding, 2% sold to the federal government or at the retail level, 8% to a private vendor, and approximately 4% used all of these outlets to sell their seed.

Distribution of the Manual

The respondents were asked if they had shared the manual with friends, family, or colleagues in an attempt to determine how many additional people may have been exposed to this information. Approximately 67% of the respondents shared the manual with others. These respondents had shared the information with an additional 385 people. The employment sector did not affect the likelihood of sharing manual information with others.

Perceived Usefulness of the Educational Tool Correlated to Knowledge Gain

The knowledge gain variable was regressed as a block (stepwise procedure) against the nine seed collection publication chapter topics. In addition, the variable "overall usefulness of the information provided by the publication" was held constant to reduce any multicollinearity issues. Significant and positive relationships were revealed between knowledge gain and two of the publication topics: native plant identification guides (B=.594, t = 3.710, p<.01) and collecting and preserving voucher plant specimens (B=.259, t=2.159, p<.05). The adjusted R (.647) indicates that the model explains 65% of the variance. Although some variables were intercorrelated, there was no statistically significant problem with multicollinearity.

These results reveal, as shown in Table 3, that higher scores of publication usefulness of two topics of the publication: 1) native plant identification guides, and 2) collecting and preserving voucher plant specimen are predictive of increased knowledge gain reported by the respondents.

Conclusions

A grant-funded educational program and publication sponsored by the United States Department of Interior Bureau of Land Management (BLM) included the publication and distribution of a native seed collection manual and presentations at numerous workshops and conferences. The program was formally evaluated using a mailed survey to determine the utility of the manual and if seed collection activity by program participants increased.

The data provided here indicate that the Field Guide for Collecting Native Seeds in Nevada manual was effective in creating knowledge gain and behavior change. This program evaluation report may be helpful to other natural resource professionals working on this issue as they contemplate (1) program approaches that are working to create change, (2) program funding sources, and (3) program evaluation processes. This formative and summative evaluation data will be used by the authors and the granting agency to focus future educational programs aimed at increasing native seed collection activity.

Cooperative Extension professionals not working in this programming area may be interested in the process used in evaluating this educational program. It is interesting to see how perceived usefulness of the educational tool can be correlated to knowledge gain. Extension professionals are increasingly being asked to report these program impacts.

Cooperative Extension is one of the few organizations that routinely and formally evaluates its educational programs. The experience with formal evaluation activities provides Cooperative Extension organizations an advantage when competing for targeted funds with an educational component. Cooperative Extension administrative and field level personnel should market this advantage when seeking extramural funds.

References

Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Sage Publications: Beverly Hills, CA.

Dillman, D. (2007). Mail and Internet surveys: The tailored design method (2nd ed., updated). New Jersey: John Wiley & Sons, Inc.

General Services Administration. (2004). Overview of the United States Government's owned and leased real property: Federal real property profile as of September 30, 2004. Retrieved August 15, 2007 from: http://www.gsa.gov/gsa/cm_attachments/GSA_DOCUMENT/ Annual%20Report%20%20FY2004%20Final_R2M-n11_0Z5RDZ-i34K-pR.pdf

National Interagency Fire Center. (2006). Wildland fire statistics. Retrieved August 15, 2007 from: http://www.nifc.gov/stats/ytd_st.htm

Nazzaro, R. N. (2006) Wildland fire management: Update on federal agency efforts to develop a cohesive strategy to address wildland fire threats. Letter to House Committee on Appropriations May 1, 2006. United States Government Accountability Office. GAO-06-671R Cohesive Wildland Fire Strategy.

Richards, R. T., Chambers, J. C., & Ross, C. (1998). Use of native plants on Federal lands: Policy and practice. Journal of Range Management. 51(6), 625-632.

Salant, P., & Dillman, D. (1994). How to conduct your own survey. New York: John Wiley & Sons, Inc.

Singletary, L., & Smith, M. (2006). Nevada agriculture producer research and education needs: Results of 2006 statewide needs assessment. University of Nevada Cooperative Extension, EB-06-02. 118 pages.

Smyth, J., & Dillman, D. (2007). Family farming and ranching in Washington: Research results. Washington State University: CRS Information Series No. 1-07. Retrieved August 15, 2007 from: http://www.crs.wsu.edu/1-07-farmranchwa.pdf

Statistical Package for Social Sciences for Windows (Version 14.0) [Computer Software]. Chicago: Author, 2006.

Westerling ,A. L., Hidalgo, H. G., Cayan, D. R., & Swetnam. T. W. (2006). Warming and earlier spring increases western U.S. forest wildfire activity. Science, July 6, 2006. (10.1126/science.1128834).