June 2009 // Volume 47 // Number 3
 |
 |
 |  |
Editor's Page
Share JOE
v47-3ed1
"Share JOE" explains how easy it is to share JOE articles or issues with your colleagues via email thanks to our new and improved site. "June JOE" describes this issue, which focuses on evaluation and needs assessment—certainly topics worth focusing on and sharing.
Commentary
What Progress, Program Evaluation? Reflections on a Quarter-Century of Extension Evaluation Practice
v47-3comm1
The September 1983 issue of the Journal of Extension was devoted entirely to the topic of program evaluation, marking the beginning of a new emphasis in Extension programming. This "call to action" was based largely on the need for program accountability; Extension educators could no longer afford to assume their programs worked or that their worth was self-evident. In the years since, evaluation in Extension has developed considerably. This Commentary explores a new call to action for evaluation in Extension, with a focus on more "logical" logic models, organizational evaluation capacity and support, and a greater emphasis on evaluation use.
Participate in the JOE Discussion Forum on “What Progress, Program Evaluation? Reflections on a Quarter-Century of Extension Evaluation Practice”
Ideas at Work
Photos Can Inspire a Thousand Words: Photolanguage as a Qualitative Evaluation Method
v47-3iw1
Finding ways to encourage expression of individuals who are young, shy, or have limited verbal abilities can be challenging for evaluators. Photolanguage can be used to aid personal expression and small group interaction and as a tool to enhance qualitative evaluation activities. This method offers an interesting evaluation process that uses black-and-white photographic images specifically chosen for their aesthetic qualities; their ability to stimulate emotions, memory, and imagination; and their capacity to stimulate reflection in the viewer. Evaluators may find Photolanguage provides a valuable tool for use with evaluation participants who experience barriers to involvement, actual or perceived.
Priority Setting Tool for The Western Region IR-4 Center
v47-3iw2
The IR-4 Project must select a limited number of pesticide registration projects each year from among many potential projects. This complex, national decision-making process is facilitated with the Priority Setting Tool (PST). The PST collects input from Extension and industry stakeholders and displays relevant projects with the result of improved project selection supporting Western specialty crop growers. This innovative process and tool could be adapted to any Extension program that has specific project goals that are prioritized by stakeholder input.
Bringing the Environment into the English-as-a-Second-Language Classroom
v47-3iw3
Environment and Community: Caring for Our Natural Resources is a curriculum developed by UC Cooperative Extension to teach adult immigrants about the environment in English-as-a-second-language (ESL) classrooms. It was developed with participation from ESL instructors and students. Surveys were conducted to determine knowledge of and interest in the environment among ESL students and to assess changes after participation in the program. Results indicate a high level of interest in the environment. After using the Environment and Community curriculum, students experienced a significant gain in knowledge about the environment, and 63% indicated that they had implemented pro-environment behaviors.
Farm and Forest Fair Educates Fifth Graders about Natural Resource Issues
v47-3iw4
The Farm and Forest Fair educational program was developed to provide an objective view of the importance and impact of natural resource industries and promote an understanding of the issues regarding natural resource use. The targeted audience is fifth grade youth, teachers, and parents. Participants rotate through 10 stations in 9-minute intervals. Stations are staffed by agriculture- or forest-based industry personnel who demonstrate their area of natural resource involvement utilizing visual aids and hands-on learning techniques. Teachers, presenters, and youth are evaluated each year. Student pre- and post-test results indicate knowledge level increases of 21%.
Distance Education: Taking the First Steps
v47-3iw5
The project described here allowed the author to investigate the creative skills and technological knowledge required to produce an effective and engaging distance education program. The module simulates a "traditional" classroom setting by synchronizing a videotaped lecture with accompanying PowerPoint slides. The product was evaluated with a test audience, which indicated a strong and positive acceptance of this format for Extension programming.
Tools of the Trade
Ready-Made Resources for Extension Evaluation Competencies
v47-3tt1
Evaluation capacity building has been part of professional development in many Extension organizations for a number of years. Many resources are currently available for Extension organizations to create or refine the evaluation competencies they want to build in their agents, specialists, administrators, and collaborators. This article describes some current and relevant resources. Materials and networks are listed for ready access by Extension personnel.
A Framework to Link Evaluation Questions to Program Outcomes
v47-3tt2
Asking the right evaluation questions is very important to documenting program outcomes. This article provides a roadmap to link evaluation questions to program outcomes. Understanding the program, the purpose of the evaluation, and its utility are critical steps in developing focused evaluation questions. Further, grouping evaluation questions into process and outcome questions will help answer both program implementation efforts (activities and resources) and program effects (KASA change) on participants. Developing a program outcome chart also helps in writing focused evaluation questions. An important strategy in developing evaluation questions is to integrate program evaluation into the program development process.
Impact Evaluation of Integrated Extension Programs: Lessons Learned from the Community Gardening Program
v47-3tt3
Integrated programming is a coordinated Extension approach to address multi-faceted community issues. An integrated Extension approach is needed to address complex community issues in a meaningful way. Planning, implementation, and evaluation of an integrated program should be considered as a joint effort by the partnering Extension agents. This article describes how to document the impacts of integrated Extension program to reflect the coordinated effort of the Extension team. Documentation as well as sharing impacts with the partners is necessary to strengthen the collaboration and sustain the integrated Extension programming effort.
Using Planning and Evaluation Tools to Target Extension Outputs & Outcomes: The New England Private Well Symposium Example
v47-3tt4
Increasingly, the success of Extension programming is evaluated based on achieved outcomes. Here, we report on the use of the ADDIE model as a tool to plan, implement, and evaluate a specific activity within the New England Private Well Initiative's regional efforts. Using this tool, we have successfully identified outcomes and objectives for the New England Private Well Water Symposium.
Enhancing Accountability: ServSafe™ Impact Template Delivers
v47-3tt5
Generating statewide impact data from Extension programs can be challenging. To streamline reporting, Extension specialists, with the help of county agents and administrators, generated a statewide impact statement for the ServSafe™ program in Virginia. This template includes knowledge gain, behavior change, and economic impact from participants generated from current standardized evaluation methods used by all Extension educators across the state. Providing this template for agents resulted in easier, more consistent yearly reporting for those agents administering the program.
Creative Approach to Evaluating: The Tri-Fold Display Example
v47-3tt6
One benefit in working for Extension is the educator's ability to be "creative." However, creativity and evaluation typically are not two words an Extension educator uses in the same sentence. This article highlights one creative evaluation strategy used at a youth wildlife camp. The evaluation strategy utilizes a tri-fold display allowing participants the ability to "showcase" what they learned. From their "showcase," the Extension educator can use simple evaluation techniques to determine the most significant item learned. An extra bonus is that these participants use these tri-folds in communities to tell others about their experiences building critical life skills.
360-Degree Evaluations: A New Tool in Extension Programming
v47-3tt7
Evaluation is an important component of Extension programs. To elicit the most effective data, 360-degree evaluations have emerged, where multiple individuals are surveyed to evaluate the performance of volunteers and their programs. These tools should be used to assist the volunteer to perform closer to potential and to help Extension accomplish its goals and more effectively involve volunteers. Extension professionals can utilize 360-degree evaluations to evaluate camp staff, campers, agents, and volunteers. 4-H club leaders can be evaluated in the same manner. These evaluations allow agents to improve the program and to perfect volunteer position descriptions.
Feature
Theory and Rigor in Extension Program Evaluation Planning
v47-3a1
This article examines two aspects of evaluation planning for Extension programs: the use of program theory and logic models and the decision process that affects the evaluation's methodological rigor. First, Extension program planners should move beyond standard applications of logic modeling to incorporate a broader, more flexible use of program theory. Second, a highly rigorous evaluation will provide numerous benefits, but considering the costs that are typically required, an evaluation's degree of rigor should be carefully determined with reference to how the findings will be used. The article makes recommendations with the aim of promoting effective and economical Extension evaluations.
A Methodology for Evaluating Transdisciplinary Collaborations with Diversity in Mind: An Example from the Green Community Development in Indian Country Initiative
v47-3a2
Extension professionals are increasingly asked to overcome barriers formed by culture, class, ethnicity, race, and/or language differences as they facilitate transdisciplinary action-research partnerships in response to increasingly complex community issues. The many challenges involved in these complex programs include the challenge of program evaluation. This article articulates a methodological foundation for program development and evaluation that responds to these demands. This methodology draws on education, social science, health science, and insights from transdisciplinary action-research practitioners. An analysis of an ongoing transdisciplinary action-research initiative is presented to illustrate the methodology in practice.
Qualitative Tools to Examine EFNEP Curriculum Delivery
v47-3a3
Use of qualitative research methods to evaluate nutrition education programs is limited. Structured observations and focus groups, qualitative research methods, were conducted to examine use of theory-based learning strategies and participant experiences in a nutrition education program. Theory-based learning strategies included use of open-ended questions, visual aids, and experiential learning activities. Open-ended questioning and some experiential activities were used less often than desired. Language of lesson delivery and instructional setting appear to influence the use of learning strategies. Lesson delivery and instructional setting may be relevant for preferred learning styles of different cultures.
The Efficacy of KidQuest: A Nutrition and Physical Activity Curriculum for 5th and 6th Grade Youth
v47-3a4
KidQuest is a nutrition and physical activity curriculum for 5th to 6th grade youth engaging participants in goal setting, self-monitoring, and reinforcement. Evaluation of the program over the 2005-2006 school year involved a nonrandom sample of 98 intervention and 38 control group participants in rural South Dakota using baseline and ending surveys. Self-reported improvements in breakfast frequency, dairy intake, increased frequency of looking at the food label, and increased food label knowledge were observed in the intervention group, with no significant change in the control group.
An Evaluation of the Missouri Master Naturalist Program and Implications for Program Expansion
v47-3a5
The study reported here evaluated outcomes of the Missouri Master Naturalist Program. We developed a pre-test/post-test/follow-up format to determine if, after the program training, volunteers increased knowledge of key ecological, natural resources, and conservation concepts and issues. We also identified volunteers' motivations for participation in the program and compared their motivation scores with changes in their knowledge scores. Volunteers' knowledge scores increased (P ≤ 0.05) following the training and were maintained 6 months after training. Volunteers ranked values/altruism as important motivations for participating. There was no relationship between volunteers' motivation scores and changes in knowledge scores.
Assessing Public Opinion Directly to Keep Current with Changing Community Needs
v47-3a6
In communities that are changing, Cooperative Extension's programs should change to meet the needs of community members. Directly assessing community members' needs and concerns is valuable in developing and offering educational programs that will be responsive to community needs and attract audiences. We describe a simple survey method that can efficiently assess community members' concerns. The method is useful in developing broad programs that can address several related issues. A specific example of the process is presented using a county facing well-recognized rural/urban issues.
Use of the PRKC Tool in Assessment of Staff Development Needs: Experiences from California
v47-3a7
This article reviews the experiences of the authors in using the Professional Research, Knowledge, and Competency (PRKC) tool in assessing staff development needs among 4-H staff members in California. The PRKC is useful for evaluating self-perceptions of knowledge and competency levels among youth development staff. We encountered a number of challenges in working with the PRKC tool that are identified here, such as determining the best way of analyzing and interpreting the responses. The conclusions described here may assist researchers in other states in using the PRKC for similar work.
Meeting the Extension Needs of Women Farmers: A Perspective from Pennsylvania
v47-3a8
According to the 2002 Census of Agriculture, women comprised 11% of principal farm operators and 27% of all farm operators. Here we report findings from a needs assessment conducted to understand the educational needs of women farmers in Pennsylvania. We describe the characteristics of the women who responded to the needs assessment, the problems they face in making their farm operation successful, and the program topics and formats they prefer. Finally, we provide recommendations to increase Extension engagement with this growing clientele.
Extension Educators' Perceptions of the Educational Needs of Women Farmers in Pennsylvania
v47-3a9
The number of women farm operators has steadily increased in the United States, comprising 30% of all farm operators (2007 Census of Agriculture). The increasing diversity of farmers presents new audiences for Extension and for whom programs should be developed according to their educational needs. We report results from a survey of Extension educators in Pennsylvania. We identify how Extension educators perceive women farmers, the factors that influence these perceptions, and how these perceptions influence educational programming. We recommend ways to reach women farmers with programs that are appropriate in both content and delivery.
Research in Brief
Impact Evaluation of Food Safety Self-Study Extension Programs: Do Changes in Knowledge Relate to Changes in Behavior of Program Participants?
v47-3rb1
There are only limited research findings to determine whether the changes in Extension participants' knowledge lead to changes in their subsequent behavior. The research described here studied the relationship between changes in knowledge and behavior of family childcare providers in a self-study course on food safety. There was a moderately positive correlation between changes in participants' food safety knowledge and changes in their safe food handling practices. The implication of the study is that Extension professionals may use knowledge change as a reliable impact indicator in determining potential behavior changes of in-home childcare providers.
How Master Gardeners View and Apply Their Training: A Preliminary Study
v47-3rb2
Eight months after the 2006 Maine Master Gardener training, a statewide survey was sent to all 240 participants to determine to what degree they had adopted new gardening practices and what aspects of the program beyond horticultural instruction were most valued. Results showed a greater than 50% adoption rate for nine out of 16 horticultural practices as well as substantial personal growth and community enrichment. Master Gardeners expressed pride in having used their new knowledge and skills to assist others in need, a renewed faith in volunteerism, and a stronger commitment to environmental stewardship.
The Impact of the 4-H Program on Nevada Public School Youth
v47-3rb3
A 4-H impact evaluation conducted in Montana, Idaho, Colorado, and Utah was replicated in the Nevada public schools. The purpose was to measure the impact of the 4-H experience on the lives of Nevada youth. ANOVA for constructs by independent variables, age groups, gender, 4-H participation, and population density revealed that 4-H participation significantly contributed to the variance in extracurricular activity involvement (p ≤ .001), school leadership positions held (p = .025), caring for others (p ≤ .001), and self-confidence, character, and empowerment (p = .004).
Post-Course Evaluation of a Grape Management Short Course
v47-3rb4
A short course program designed to prepare potential grape growers for advanced topics in viticulture has been taught by Oklahoma State University Cooperative Extension since 2001. A post-course evaluation revealed the course was effective in meeting the educational needs of students, with more than 80% recommending the course without hesitation. The evaluation also revealed that the course positively affected growers financially, estimated between $120,032 and $490,000. Even though the course was well reviewed, many growers are still not aware of other grape-related Cooperative Extension programming from Oklahoma State University, and therefore, more effort must be expended to correct this program deficiency.
Extension Teams Collecting Industry-Specific Stakeholder Input
v47-3rb5
Extension educators have explored different methods for collecting stakeholder input, but a suitable methodology has not been agreed upon. The Michigan State University Extension dairy team works with an advisory board, but last collected broader formal stakeholder input in 1997. In 2007, the team decided to seek additional broad-based and inclusive stakeholder input. The research team developed an issue identification procedure through group meetings at different locations throughout the state. This article reports on the procedure, its advantages and disadvantages compared to other group methods, and its results.
Food Industry Needs Assessment Survey: A Case Study
v47-3rb6
The study reported here assessed the needs of the food processing industry in South Carolina in order to develop strategic plans for effective assistance. Results of the online survey indicated that developing new products and markets, solving technical problems, and the training of employees were important needs. Due to the fundamental similarities of food processing establishments, it can be assumed that food businesses in other states may have similar needs. Future plans include contacting land-grant universities in the U.S. to explore the use of the online survey in other states.
Online Graduate Education Needs of Selected Iowa Extension Professionals
v47-3rb7
The study reported here assessed selected Extension professionals' needs related to online graduate education. A census survey of 403 Iowa Extension professionals was conducted. Twenty Extension professionals indicated they were likely to apply for admission to a proposed online Master of Science degree program in agricultural education. Extension professionals have the incentives, computer resources, and computer skills needed to pursue online graduate courses or professional development workshops. This work suggests new ways, including online learning and institutional collaboration on a regional level, to meet educational needs of Extension professionals.