Assessing Stakeholder Needs: Delphi Meets the Internet

Introduction

Cooperative Extension has a rich history of developing outreach programs that have a direct and relevant impact on stakeholders' lives. In order to develop relevant programming, clientele are asked for input during the development stages for many programs. Often, requesting and incorporating timely and relevant input to program curricula can be a time-consuming, expensive process. Decreasing state and federal resources are forcing Extension personnel to seek alternative methods to continue their rich tradition of stakeholder input in the program curricula development processes.

Conceptual Framework

Alternative methods for collecting stakeholder input to Extension program curricula provide Extension personnel with timely, relevant feedback during the curricula development process. One inexpensive alternative to holding several face-to-face or traditional postal mail surveys is achieved through the Delphi technique, using a Web-based medium.

The Delphi technique was developed by the Rand Corporation in the late 1950's as a forecasting methodology. Unlike the nominal group process, the Delphi does not require face-to face participation. It is a "systematic solicitation and collation of judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed with summarized information and feedback of opinions derived from earlier responses" (Debecq, Van de Ven, & Gustafson, 1975, p. 10). The Delphi technique affords researchers an opportunity to collect large amounts of input over a wide geographic area. Delphi techniques incorporate expert panel members' opinions, value judgments, and agreement in the consensus-building process (Somers, Baker, & Isbell, 1984).

Decisions about which participants to invite to a Delphi should be considered carefully. Ludwig (1997) recommended:

Randomly selecting participants is NOT acceptable. Instead, characteristics and qualifications of desirable respondents should be identified and a nomination process used to select participants. Because the group number will be small (12-15), the researcher needs to locate and target individuals who are "expert," have knowledge and experience to base their futuring activities upon, and are self-motivated. Delphi should not be used with groups that have difficulty in reading or expressing themselves in written communication. (p. 2)

Ladner, Wingenbach, and Raven (2002) found Web-based and traditional paper-based survey methodologies were equally valid and reliable for social science research. A significant difference occurred in the response rates between two equal groups of agricultural educators; however, no differences were found between the groups' opinions on computer usage in an educational setting. The Web-based group's response rate exceeded the traditional group, 72 to 7, in the first week of data collection. These results provided strong evidence for using Web-based data collection methods in social science research when time and/or financial constraints pose barriers to relevant, timely, effective program development processes.

Before developing new programs, it is important to be mindful of gathering and using Extension stakeholder input in developing the program materials. One of the first steps in designing adult education curricula is to conduct a needs assessment (Sork & Caffarella, 1989). Knowles, Holton, and Swanson (1998) provided two assumptions about adult learning that are critical in the needs assessment phase. These assumptions are the need to know and the learner's self-concept. Essentially, adults need to know why they need to learn something new. Also, adults will resist and resent (learner's self-concept) situations in which they feel others are imposing their will on them (Knowles, Holton, & Swanson). Such assumptions about adult learning provide important reasons for using stakeholder input during curriculum development.

Decreasing state funds for Extension programming have forced many states to seek alternative methods to continue providing quality educational programs for their clientele at the county level. Extension clientele input for developing instructional modules in the Turf for Texans Master Gardener Program was sought using innovative, cost-effective data collection methods.

Purpose and Objectives

The purpose of the study described here was to gather stakeholder input for the most Frequently Asked Questions (FAQs) in turfgrass management curricula for the Turf for Texans Master Gardener Program. The following objectives guided this study.

1. Identify FAQs for nine turfgrass instructional modules in the Turf for Texans Master Gardener Program.

2. Rank the importance of the identified FAQs.

3. Rank participants' agreement levels of the identified FAQs for inclusion in the turfgrass instructional modules.

Methods and Procedures

Descriptive survey methodology, with a Delphi technique, was used in this study. Web-based survey data collection methods (Ladner, Wingenbach, & Raven, 2002) were used after obtaining approval to conduct the study through the Texas A&M University Institutional Review Board (#2002-0276).

The target population (N = 339) consisted of all Texas County Extension Agents, program coordinators, and volunteers who participated in a Texas Master Gardener Program during 2003. A proportional stratified sample from 11 Texas Cooperative Extension Service districts was obtained by contacting two agents from each district, who in turn, chose one coordinator and one volunteer with at least 1 year of experience from their respective Master Gardener Programs. All participants were sent formal letters requesting their participation in the study. The sample consisted of 22 agents, 22 program coordinators, and 22 volunteers (n = 66).

The first instrument, posted on a secure Internet site, consisted of nine open-ended questions designed to obtain a wide range of responses. Using their own Master Gardener experiences, respondents were asked to identify the top five FAQs for turfgrass management in each of nine Turf for Texans instructional modules. The identified FAQs were used to develop content for the modules. Electronic mail reminders were sent to non-respondents to complete round one; all data were collected in 3 weeks. A total of 20 agents, 4 coordinators, and 12 volunteers (n = 36) from 33 counties in the 11 districts responded to round one, resulting in a 55% response rate. Findings from this study should not be generalized beyond the limited number of respondents.

A team of Extension turfgrass specialists, graduate students, and agricultural education faculty members condensed and combined initial responses into statements without altering their original meanings. A panel of experts from the Departments of Soil and Crop Science and Agricultural Education reviewed the instrument for face validity. The statements were posted on a secure Internet site for use in round two.

In the second round of data collection, respondents were instructed to read each FAQ for each module and rate the level of importance (Likert-type scale: 1 = Not Important - 4 = Very Important) for including the FAQ in its respective turfgrass instructional module. Electronic mail notices requesting participation in round two were sent to all 66 participants. All 66 participants were asked to complete all three rounds because of their vested interest in the consensus-building process. A total of 16 agents, 7 coordinators, and 12 volunteers (n = 35) responded, resulting in a 53% response rate. All data were collected in 2 weeks.

Upon conclusion of data collection in the second round, all statements were ranked according to their grand mean scores, sorted by level of importance, and posted in a third instrument on a secure Internet site. The third instrument allowed respondents to rate their agreement levels (Likert-type scale: 1 = Strongly Disagree - 4 = Strongly Agree) with the importance levels for each FAQ in each turfgrass instructional module. Electronic mail notices requesting participation in round three were sent to all 66 participants. A total of 15 agents, 5 coordinators, and 10 volunteers (n = 30) responded, resulting in a 46% response rate. All data were collected in 10 days.

Descriptive statistics were derived for each instructional module. ANOVA tests were used to determine significant differences among subgroups in this consensus-building process. Instrument reliability was assessed using Cronbach's alpha coefficients in rounds two and three. Results are presented in Table 1.

Findings

Due to space limitations, only grand means from the third (final) round of the Delphi are presented. For more detailed information from the first and second rounds of this study, readers should contact the authors at c-mayfield@tamu.edu or g-wingenbach@tamu.edu; information also is available regarding responses by agents, volunteers, and coordinators.

Thirty-six respondents with Texas Master Gardener Program experiences ranging from less than 1 to over 20 years (M = 4.73), identified the top five FAQs for turfgrass management in their Texas Master Gardener Programs. Overall, a total of 115 FAQs were identified, ranked, and prioritized by stakeholders. The top three FAQs for each module are depicted in Table 2. Results are sorted by descending grand means.

The consensus-building process of this Delphi technique was useful in helping respondents prioritize the most important FAQs for each of the nine modules. No significant differences were found among respondents' agreement levels of the FAQs in eight of the nine modules. Only in Module 5 (Mowing) was there a significant difference between rankings. Program coordinators agreed with the FAQ "how often should mower blades be sharpened," more than did agents in round three.

Conclusions/Recommendations

From the findings, it can be concluded that lawn health, differences between warm and cool season grasses, turfgrass selection factors, soil tests, effects of improper mowing, grass clippings, when to fertilize, frequency of irrigation, and lawn diseases were deemed the most important FAQs for inclusion in the turfgrass curricula.

Although the identified and ranked FAQs for the instructional modules proved useful in developing curricula for the Turf for Texans Master Gardener Program, the authors believe the most important finding was derived from the methodology used to gather stakeholder input. The Delphi technique, administered through online data collection techniques, provided effective means to determine stakeholders' needs in designing turfgrass management curricula. Participants were able to incorporate their opinions (round one), value judgments (round two), and agreement levels (round three) in a consensus-building process for the FAQs used in the turfgrass management instructional modules.

A practitioner's checklist for using this data collection process includes:

1. Asking respondents to provide information relevant to the programming objective. (Round I)

2. Condensing responses into statements, being careful not to change the original meanings of the responses.

3. Gathering respondents' importance level ratings for each identified statement (importance of its inclusion in the programming objective). (Round II)

4. Rank-ordering statements according to their indicated levels of importance.

5. Collecting respondents' agreement levels on the rank-ordered importance of each statement for its inclusion in the programming objective. (Round III)

Additionally, stakeholder input was gathered in an economical, shortened frame (6.5 weeks), confirming the Web-based surveying methods proposed by Ladner, Wingenbach, and Raven (2002). The Delphi technique used in the study provided consistency in the data collection procedures, as proposed by Somers, Baker, and Isbell (1984).

By including stakeholders' input to build consensus on relevant topics for Extension programs, Extension personnel address the need to know and learner's self-concept assumptions raised by Knowles, Holton, and Swanson (1998). This could allow for more rapid adoption of Extension programs. It can also allow Extension personnel to focus greater attention on developing relevant educational materials for their clientele.

We recommend these methodologies (Delphi technique and Web-based data collection methods) be used by Extension personnel when seeking stakeholder input for instructional materials development. Using these methodologies, Extension personnel can gather stakeholder input in a shortened time frame with minimal cost and inconvenience resulting in a high quality Extension program.

References

Debecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, Illinois: Scott, Foresman and Company.

Knowles, M., Holton, E., & Swanson, R. (1998). The adult learner: The definitive classic in adult education and human resource development (5th ed.). Houston, TX: Gulf Publishing Co.

Ladner, D., Wingenbach, G., & Raven, M. (2002). Internet and paper-based data collection methods in agricultural education research. Journal of Southern Agricultural Education Research, 52(1), 40-51.

Ludwig, B. (1997). Predicting the future: Have you considered using the Delphi methodology? Journal of Extension [On-line], 35(5). Available at: http://www.joe.org/joe/1997october/tt2.html

Somers, K., Baker, G., & Isbell, C. (1984). How to use the Delphi technique to forecast training needs. Performance and Instruction Journal, 23(4), 26-28.

Sork, T. J., & Caffarella, R. (1989). Planning programs for adults. In S. Merriam, and P. Cunningham, (Eds.), Handbook of adult and continuing education (pp. 233-245). San Francisco: Jossey-Bass Publishers.