An On-Line Survey Process for Assessing Impact of an Email-Delivered Pest Advisory

Introduction

Documenting impact is a necessary component of current Extension programming. This article describes a new email-delivered pest advisory and a survey process for evaluating its usefulness and impact. The process we used was relatively easy, required only a few hours of concentrated effort and expertise, was inexpensive, and generated very useful and reportable impact data.

The Internet has become an essential and cost-effective tool for Extension (Mayadas 1997; Muske, Goetting, & Vukonich, 2001). Tennessen, PonTell, Romine, and Motheral (1997) emphasize that Extension should maximize the use of the Internet to improve the efficiency of its operations. Since the early 1990's, integrated pest management (IPM) specialists in Virginia have been using email to convey information to Extension personnel and to their own circulation lists of growers and clientele. The information in these emails was further disseminated by Extension personnel forwarding the message, or by cutting and pasting the information into newsletters or Internet homepages. Although this system of information dissemination represented a vast improvement in efficiency over that of decades ago, it was not without its flaws. On occasion the forwarded content from a specialist would be taken out of context, or the author would not be credited for his or her work. Moreover, clientele and other Extension personnel would receive a jumble of information, sometimes from several different sources.

In July 2004 we launched the Virginia Ag Pest Advisory <http://www.sripmc.org/virginia/> in cooperation with the Southern Region IPM Center in Raleigh, NC. The advisory is a Web site that compiles pest updates from multiple IPM specialists. Specialists enter their update(s) at a time that is convenient, and each entry is categorized by commodity and pest group. The compiled advisory is automatically emailed once a week to the recipient list. Individual entries can be viewed or the entire newsletter. The advantage of this system to the recipient is that it is a single-source provider of updated pest information. Everything is in one location, and users become accustomed to having it delivered at the same time each week.

Documenting impact of Extension programs can be challenging. Mail, telephone, hand-out, and face-to-face surveys are useful but can be very time consuming, expensive, and hard to sustain year after year. The on-line survey process described here reached a large number of people easily, anonymously, and worked well to evaluate the impact of the new pest advisory.

Procedures

In order to assess the usefulness and impact of the Virginia Ag Pest Advisory, we conducted an on-line survey of all 346 email recipients at the end of the field season in 2004. The recipient list included 129 Virginia Cooperative Extension Agriculture and Natural Resource Agents linked by a listserv and 217 agricultural industry personnel and farmers. We used "survey.vt.edu" to design the questions and collect survey data. "Survey.vt.edu" is an Internet-based tool that is available to all Virginia Tech faculty, staff, and students. The survey Web site (http://survey.vt.edu) is designed for ease-of-use and can be accessed with any Internet browser. The Web site automatically compiles and summarizes the responses to the survey and an up-to-date summary report of the survey can be obtained at any time by the authorized administrators.

There were a total of nine survey questions concerning the advisory (five multiple choice and four short answer) concerning usage statistics, usefulness of the information and how it affected IPM practices, and suggestions for improvement. Figure 1 shows the first three survey questions as a respondent's Internet browser would display them. An email that hyperlinked the survey was sent on September 16, and the survey remained open for data collection for approximately 1 month. In addition, a reminder email was sent on October 13.

Figure 1.
Partial On-Line Survey

Results and Discussion

The survey measured kinds of information described by Taylor-Powell and Renner (2000), including participant reactions (e.g., usefulness and influence of the advisory), teaching and facilitation (e.g., suggestions for improving the advisory), outcomes (e.g., perceived changes in knowledge and IPM practices), and future programming (e.g., asking what information would be useful in next year's advisory). This information may be useful for evaluating programs, especially for assessing some short- and medium-term outcomes as described by the University of Wisconsin Extension's "Logic Model" (2004).

Short-term outcome assessments included measuring the participants' awareness and opinions of the advisory and their perceived knowledge gained (e.g., improving their understanding of agricultural pest problems). Medium-term outcomes included decision-making assessment (e.g., how the advisory aided in choice of pesticide products, rates, and timing). This process may serve as a simple, cost-effective model for documenting impact of many kinds of Extension programs.

Results of the Survey

There were 119 responses to our survey (34.4% response rate). Eighty-seven percent of respondents had accessed the Virginia Ag Pest Advisory during the season. Sixty-eight percent of respondents who accessed the advisory did so once a week. Twenty percent viewed it less than once a week, and 8% accessed it more than once a week.

A majority of the respondents found the advisory to be useful (Table 1). On a Likert scale of 1 (not useful) to 5 (very useful), the mean was 4.0 (SD = 0.88). In addition, most respondents reported that the advisory influenced their (or their clients') pest management practices (Table 2). On a Likert scale of 1 = did not influence and 5 = greatly influenced, the mean response was 3.3 (SD = 0.93).

Based on responses to a list of other questions in the survey (Table 3), the Virginia Ag Pest Advisory clearly had a strong impact on educating clientele on agricultural pest problems and pesticide use. For instance, comments by respondents included: "[the advisory was] excellent for alerting farmers for possible insect problems" and "[the advisory] aided me in providing Extension updates about insect monitoring and treatment recommendations."

Suggestions from the Survey

Suggestions from the survey respondents for improving the information contained in the advisory included:

• Including more information on other crops (wheat; grape) and home gardening

• Expanding the advisory coverage area (include information from: northeast and southeast United States; South Carolina and North Carolina; northern Virginia; southwest Virginia)

• Provide digital photographs, audio, and links to other Web sites for more information on pests

• Provide a quick way to identify insects

• Provide color maps showing movement and prevalence of pests and make pest forecasts

• Increase the content of the advisory (provide more information on: daily blacklight trap catches; soybean aphid thresholds and how soybean aphids affect the plant; corn borer damage in corn and cotton; weeds and herbicides; seed treatments; vegetable pests; ornamental pests)

Suggestions from the survey respondents for improving the delivery of the advisory included:

• Send out emails more often during peak times of the growing season
• Distribute pest information via cell phone messages or by calling a toll-free number
• Include the full text of each advisory in the email in case someone has a slow Internet connection
• Make accessing the information easier
• Deliver the advisory on Fridays

Our Thoughts on the Survey Suggestions

Additional authors would help to improve the advisory by increasing crop and pest expertise and would also increase the coverage area. Photographs, audio, and image files would be a welcome addition and would help to keep the advisory graphically appealing, but large file sizes will slow down Internet connection speed and make it more difficult for people to print the advisory. A solution may be to include "extras" in an attachment--authors currently may attach one file to each of their posts--thus keeping the main screen simple and uncluttered.

Forty-eight of 58 respondents thought that email delivery of the advisory was an adequate notification system. Monday email deliveries were satisfactory for most people; however, four respondents said that they would prefer Friday email deliveries because they read more articles over the weekend or that it made it easier to schedule their work plans for the following week.

Emails provide fast, practical, convenient delivery; however, problems do arise. Some people had emails blocked by anti-spam software--in one case, this required the Web site administrator to go to the client's email host and certify that the message was not spam. Also, one may have to adjust his or her email filters so that messages from the sender are not blocked.

Having no file attachments, emails should take a minimum amount of time to receive and open. However, users with slow Internet service may become frustrated by waiting for pages to load and may miss some useful articles. For example, one user commented that "when I have to click on something and wait I often decide to just delete and go." Therefore, some would benefit from having a hard-copy printout, phone message, or other method of delivery, and Extension should work with these people to achieve desired results.

Conclusions

The Virginia Ag Pest Advisory is a useful information dissemination tool that simplifies the process of collecting input from IPM specialists and distributing the information in a timely manner. Growers, Extension and agricultural industry personnel, and others will benefit from the advisory's rapid dissemination of information; it's "anytime, anyplace" availability; and knowing that the information is reliable because it comes directly from specialists.

The on-line survey was user-friendly, easy to create, and inexpensive. It served as a model for providing useful numerical program impact data. Additionally, it provided written responses by survey participants that will help improve the system and would easily fit into a program assessment report. We hope to expand, enhance, and optimize the use of the advisory to support our clientele and to promote on-line surveys for improving Extension programs and reporting impact data to Extension administrators.

Acknowledgments

Ron Stinner, Steve Toth, and the staff of the Southern Region IPM Center in Raleigh, NC are thanked for their assistance in developing the Virginia Ag Pest Advisory. We also acknowledge Lex Bruce (Virginia Cooperative Extension, Blacksburg, VA) for his assistance with the survey.

References

Logic Model. University of Wisconsin Extension, Madison, WI. Retrieved November 29, 2004 from: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Mayadas, F. (1997). Asynchronous learning networks: a Sloan Foundation perspective. Journal of Asynchronous Learning Networks, 1 (1). Available at:

http://www.sloan-c.org/publications/jaln/v1n1/v1n1_mayadas.asp

Muske, G., Goetting, M., & Vukonich, M. (2001). The World Wide Web: A training tool for family resource management educators. Journal of Extension [On-line], 39(4). Available at: http://www.joe.org/joe/2001august/a3.html

Tennessen, D.J., PonTell, S., Romine, V., & Motheral, S.W. (1997). Opportunities for Cooperative Extension and local communities in the information age. Journal of Extension [On-line], 35(5). Available at: http://www.joe.org/joe/1997october/comm1.html

Taylor-Powell, E., & Renner, M. (2000). Collecting evaluation data: End-of-session questionnaires. University of Wisconsin Extension, Madison, WI. Available at: http://cecommerce.uwex.edu/pdfs/G3658_11.PDF