Using Planning and Evaluation Tools to Target Extension Outputs & Outcomes: The New England Private Well Symposium Example

Introduction

The past decade has witnessed a revolution in the expectations for Extension programming. The success of outputs, like workshops and factsheets are viewed through a lens that focuses on measureable objectives and achieved outcomes (Hoffman & Grabowski, 2004; NOAA, 2003). In this article we illustrate the use of an instructional systems design model, the ADDIE model (NOAA), to design and evaluate a specific output, the New England Private Well Water Symposium (Arnold, 2002; Peterson, 2003).

The goal of the symposium is to improve information sharing, networking, and collaboration among professionals working in the field of private well water protection by communicating current research, educational approaches, and materials, and providing opportunities for interaction. The work emerged from a CSREES competitive grant that funded the New England Regional Water Program <www.usawaterquality.org/nesci>. Extension-led interagency teams formed to address important regional water resource issues.

Based on input from Extension water quality staff and communications with the Environmental Protection Agency (EPA) Region 1, the New England Private Well Initiative (Initiative) formed in 2001. Extension has a history of private well programming (Lemley, 1993; Swistock, 2001; Clemens, 2007). Although 2.3 million New Englanders rely on private wells as their source of potable water, regular testing is not regulated (US EPA).

The ADDIE Model

We used the ADDIE model as a framework to develop and evaluate all the Initiative's efforts as well as a specific Initiative output - the New England Private Well Water Symposium. The ADDIE model's five-step approach includes Assessment, Design, Development, Implementation and Evaluation. With its clearly defined steps, ADDIE allows for effective implementation of activities that are learner-centered and outcome based (Peterson, 2003).

Each of the model's five steps contains sub-steps, summarized in Table 1.

Table 1.
NOAA's Project Design and Evaluation Training Manual Outlines ADDIE's Steps in Detail

Assessment Identify the problem Characterize the audience Identify the issues the project will impact Identify what knowledge, skills, attitudes, or behaviors need to be taught/changed Identify how to best accomplish this

Design Outline the project Define goals and objectives Develop a timeline Develop a logic model Develop an evaluation plan

Development Select specific content How will content be structured Identify appropriate delivery methods Sequence the use of various learning methods

Implementation Pilot the project Implement in accordance with previous steps

Evaluation Use appropriate type(s) of evaluation Evaluate at the correct level(s) of impact Evaluate to determine merit and worth Make decisions about the project based on the evaluation results

Below, we discuss each of ADDIE's five components in reference to the symposium.

Assessment

In 2003, we conducted a regional needs assessment meeting with Extension faculty and staff, the EPA, state drinking water agencies, and nonprofit organizations to address the following questions.

1. What are the most serious risks to well water quality and human health?
2. What support and educational programs are in place to assist private well owners to address these risks?
3. What are the critical gaps in educational programming between what is already provided and what is needed?

Prior to this meeting, Extension representatives distributed a pre-planning questionnaire to stakeholders within their states. Eighty-nine questionnaires were returned and represented the responses of public health officials, educators, well owners, scientists/researchers, environmental regulators, analytical lab professionals, and well drillers. The responses served as a cornerstone for discussion during the assessment meeting.

Following the needs assessment, the initiative developed a logic model for regional coordination and collaboration efforts. The symposium was identified as one activity within this larger effort that would enable us to achieve several outcomes as listed in Table 2.

Table 2.
Selected Outcomes from the New England Private Well Initiative Logic Model That Relate to the Symposium

Short-Term Outcomes	Mid-Term Outcomes	Long-Term Outcomes
Increased knowledge within the region of research, education and Extension programs. Increased ability of the regional team to provide integrated research, education and Extension programming. Increased ability to strengthen state-based and regional program efforts as a result of coordination. Professional sector groups will have an increased understanding of the programs and resources available to private well owners.	Increased collaboration and joint programming among Initiative to coordinate and work regionally. Established framework for regular communication and sharing of resources among Initiative members. New partner agencies and organizations engaged in Initiative efforts.	Increased integration of research, education and Extension within Initiative. Reduced health risk associated with private well water users. High quality and sufficient quantity of groundwater resources for drinking water supplies.

The needs assessment established that key partners shared many goals but were not coordinating and communicating well, resulting in program gaps, redundancies, and missed opportunities for delivering relevant research and information to private well owners. An interagency planning committee, coordinated by Extension, was formed to develop the New England Private Well Water Symposium to address this identified gap. We selected private and public sector professionals as an audience for this effort. We believed that building capacity and collaboration within this group of professionals would eventually improve educational content and delivery to private well owners.

Design

Via conference calls and e-mails, we first used the symposium logic model to define outcomes (Bennett, 1975; UWEC, 2002; NOAA). Following the ADDIE approach, the outcomes were then restated as SMART objectives (Specific, Measurable, Audience-focused, Realistic, Time-bound) (NOAA). SMART objectives provide a basis for conducting a meaningful evaluation and documenting impacts. For example, one mid-term outcome was the integration of symposium information by our audience. Several objectives related to this outcome, including the integration of knowledge gained from the event into participants' work efforts (Objective #4, Table 3).

We then developed a project timeline and evaluation plan, linking the outcomes with the project inputs and outputs. Only outputs that contribute to achieving an outcome were pursued (NOAA). Objectives were shared with all planning team members and repeatedly referenced in discussions.

Table 3.
Selected Elements from the New England Private Well Water Symposium's Logic Model

Inputs	Activities	Audience	SMART Objectives	Short-Term Outcomes	Mid-Term Outcomes	Long-Term Outcomes
1.New England water quality Extension staff 2.Initiative Partners 3.Equipment & supplies 4.Conference Location 5.Marketing materials 6.Funding	Selected Activities: -Work with conference organizer to distribute RFP to area chamber of commerce -Solicit presentations and workshops that meet identified educational needs. -Hotel site visit and selection -Draft budget -Develop symposium website -Develop and distribute save the date and call for abstracts cards -Develop and continually update mail and email lists for marketing. -Solicit symposium sponsors & vendors -Develop final agenda -Host event	-Groundwater and/or well water related organizations. -University researcher or educator -Extension professional -Private sector - well drillers, treatment professionals	Within one year: 1. 85% will agree that the symposium was a great platform to exchange ideas. 2. 70% will report a significant increase in knowledge. 3. 30% will contact one colleague/expert they met at the symposium. 4. 60% will integrate knowledge gained from the symposium into their policy, science or educational efforts. 5. 90% will express a positive opinion about the benefit of attending the event in the future.	A greater exchange of ideas among scientists, regulators, technical professionals, and educators about private well water concerns. Increased knowledge of: -Contaminant analysis and occurrence -Water treatment systems -Best protection practices -Epidemiology -Effective risk communication strategies -Policy High degree of satisfaction with Symposium as a regular event.	Participants will apply something they learned into their own work. Increased collaboration among symposium participants.	Reduction of health risks associated with ground water use to private well users. High quality and sufficient quantity of groundwater resources for drinking water supplies.

Development

We decided to pilot a one-day symposium. We would use participants' feedback to evaluate the structure, content, and satisfaction level and also help us improve a future event if results warranted. Based on our SMART objectives, we created an agenda that allowed participants to:

• Learn about current research relating to groundwater and well water quality and availability through invited talks by research scientists.
• Learn about techniques and approaches to effectively communicate this research to well owners, such as social marketing.
• Interact with each other.
• Take advantage of different educational formats (e.g. presentations, small and large group discussions).

Implementation

The 1-day pilot was held in Portsmouth, New Hampshire, in November 2005 for 95 people. Questionnaire results indicated that respondents found the event useful and overwhelmingly desired a follow-up symposium. Responses also indicated that the event should be held biennially, that a longer format would be preferred, and that additional topics be included.

As a result, a second symposium was held in 2007 in Newport, Rhode Island. This expanded 2-day event attracted 123 participants. For the 2007 symposium, we broadened the range of topics by including a mix of keynote speakers and presenters who addressed the science, management, and legal issues surrounding groundwater wells. We selected presenters based on abstract submissions. Based on participants' comments from 2005, the second symposium included several structured opportunities for breakout sessions that enabled networking.

Evaluation

Participants completed a questionnaire at the end of the event to help us evaluate outcomes (Table 4). The questions reflected the SMART objectives. An end-of-session questionnaire is a useful way to obtain immediate feedback about the event and to assess progress towards achieving short-term outcomes (Arnold).

Table 4.
Measurable Objectives and Evaluation Results from 2005 and 2007 Symposia

Objectives	Percent of Respondents Agreeing or Strongly Agreeing
	2005	2007
At least 85% of respondents will agree or agree strongly that the symposium provided an effective avenue for exchanging ideas about private well water issues.	91%	100%
At least 70% of respondents will report a significant increase in knowledge in at least one of the symposium topic areas.	95%	100%
At least 30% of respondents will report that they anticipate contacting at least one expert/colleague they identified through the event within the year.	84%	89%
At least 60% of the respondents will report the integration of some knowledge gained from the symposium into their educational efforts within 1 year.	98%	91%
At least 90% of the respondents will express a positive opinion about the utility of this event as a regular event in the future.	97%	98%
Note: Thirty-eight% of the evaluations were returned in 2005 and 46% in 2007.

In 2008, 6 months after the 2007 event, we posted a Web-based questionnaire for participants who attended the 2005 and 2007 events. The purpose of conducting this inquiry was to begin to assess the impact of the symposium on achieving mid-term outcomes. We focused on behavior changes that may have resulted from the learning or contacts that occurred at the symposia.

We received 48 responses, a 37% return rate (Table 5). Virtually all respondents wanted to participate in any future symposium. Attendee responses included a number of our logic model outcomes, for example:

• Using information and approaches to inform the process of developing state policies for private well testing.
• Enhancing collaboration with partners.
• Developing new content and communicating water quality and safety techniques to private well owners.
• Increasing networking opportunities that are resulting in collaborative grant proposals and projects for both research and Extension projects.

Table 5.
Summary of Web-Based Questionnaire Conducted in 2008

Question	Response
1. Please select the best description of your position. Researcher Educator/Extension professional Federal/State/Local official Private sector Private, non-profit organization	18% 8% 60% 8% 6%
2. After attending the symposium, I contacted a fellow attendee for information, programmatic resources, research results, or other materials pertaining to topics discussed at the symposium.	Yes: 50%
3. After attending the symposium, I provided information to another attendee on a topic discussed at the symposium.	Yes: 48%
4. I used what I learned at the symposium to improve my work.	Yes: 92%
5. After attending the symposium, I coordinated/collaborated with a fellow attendee to enhance an existing program or develop a new program/project focused on private well water protection, education or research.	Yes: 39%

Summary

NOAA's ADDIE model has provided us with a systematic way of developing the symposia and documenting impacts. We have evaluated the program against the defined measurable objectives and have been able to point to the success of this effort. As a result, we have raised sponsorship funds to help support these efforts, offset costs to our participants, and coordinated a strong and committed network of Extension personnel and partners who want to continue to carry out extensive programming for private well water protection.

Planning for the 2009 Symposium is currently underway.

Acknowledgements

We would like to thank the Symposium Planning Committee for their dedication to this event <www.usawaterquality.org/nesci/2009Symposium>. USDA CSREES National Integrated Water Quality Program provides funding for the New England Regional Water Program and the Initiative. Additional funding for the symposia has been provided by EPA - New England, Water Systems Council, University of Rhode Island, College of the Environment and Life Sciences, University of Maine Cooperative Extension, University of Massachusetts Extension, Barnstable County Cooperative Extension, University of New Hampshire Cooperative Extension, Senator George J. Mitchell Center for Environmental and Watershed Research, University of Maine, and USDA Healthy Homes and Dartmouth College Toxic Metals Research Program<http://www.dartmouth.edu/%7Etoxmetal/>. This is contribution number 21361 of the College of the Environment and Life Sciences, University of Rhode Island.

References

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line], 40(3) Article 3FEA4. Available at: http://www.joe.org/joe/2002june/a4.php

Bennett, C. F. (1975). Up the hierarchy. Journal of Extension [On-line], 13(2). Available at: http://www.joe.org/joe/1975march/index.php

Clemens, S., Swistock, B., & Sharpe, W. (2007). The Master Well Owner Network: Volunteers educating Pennsylvania well owners. Journal of Extension [On-line], 45(4). Article 4RIB7. Available at: http://www.joe.org/joe/2007august/rb7.shtml.

Hoffman, B., & Grabowski, B. (2004). Smith Lever 3(d) Extension evaluation and outcome reporting—A scorecard to assist federal program leaders. Journal of Extension [On-line], 42(6) Article 6FEA1. Available at: http://www.joe.org/joe/2004december/a1.php

Lemley, A., & Wagenet, L. (1993). Rural water quality database. Journal of Extension [On-line], 31(3) Article 3FEA2. Available at: http://www.joe.org/joe/1993fall/a2.php

National Oceanic and Atmospheric Administration (NOAA) Coastal Services Center. (2003). Product design and evaluation. Course Training Manual. Charleston, SC.. Updated periodically. Course information retrieved June 18, 2009 from: www.csc.noaa.gov/training

Peterson, C. (2003). Bringing ADDIE to life: Instructional design at its best. Journal of Educational Multimedia and Hypermedia, 12(3) pp. 227-241.

Swistock, B. R., Sharpe, W. E., & Dickison, J. (2001). Educating rural private water system owners in Pennsylvania using satellite versus traditional programs. Journal of Extension [On-line], 39(3) Article 3FEA7. Available at: http://www.joe.org/joe/2001june/a7.php

U.S. EPA Private Well Owners. Retrieved June 18, 2009 from: http://www.epa.gov/region1/eco/drinkwater/private_well_owners.html

University of Wisconsin Cooperative Extension. Evaluation logic model (2002). Retrieved June 18, 2009 from: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html