Tips and Tools: The Art of Virtual Program Evaluation—Measuring What We Do with Pizzazz

Background

As with most great programs we do in Extension, there is a need to evaluate and share results. Shared successes generate ideas and present opportunities for replication. Most times, however, as professionals and practitioners, we find ourselves overwhelmed with the customers, and we either do not want to take the time to evaluate or forget it because of time and understanding the process required to evaluate.

In 2008, Clark County, Nevada was faced with the challenge of evaluating a brand new program to measure its goals and outcomes. The population was more than 600 youth participants, and the results were critical. Faced with this challenge, we researched a quick and simple way to collect and analyze the data. We were impressed with what we discovered and the results we received by using the InterWrite Personal Response System (PRS) and Microsoft Excel software.

The InterWrite _TM PRS system is one of the most powerful and flexible electronic classroom performance systems, and it can be used in almost any group setting. It engages youth to learn while having fun and is interactive and simple to master. It facilitates effective response by encouraging audience participation and providing the evaluator with immediate indication of audience comprehension (Massey, Jaskolski, & Sweets, 2005, Williams, 2006). It combines a PowerPoint-developed questionnaire and provides a most rapid collection and analysis. Most of all, it is simple to use by youth, adds great results, and maintains the interest of participants as they complete the answers to their responses.

Purpose

The uses of the InterWrite PRS and Microsoft Excel program to measure outcomes in 4-H programs are to:

• Introduce a new system of data collection and analysis of 4-H programs.


• Increase the simplicity and accuracy of data collection.


• Observe participants reaction to the system.


• Provide yearlong approaches to mastery and life skill development.

Program Context

The PRS system was to be used for data collection in 4-H afterschool programs delivered to youth ages 8 through 15 years old at 15 sites. The sites included community clubs, after-school sites, school buildings, community centers, and faith-based organizations. The program activities were conducted Tuesdays and Thursdays from 3:30 to 5:00 P.M.

Methods

In 2008, a study was conducted on three 12-week sessions of a new 4-H program, New Faces, New Places—An Introduction to 4-H for Urban Audiences, (Barker, Leas, & Sanders, 2008) to 600 youth ages 8 through 15 in Clark County, Nevada. The program was developed in support of the National 4-H Initiative of Science, Engineering and Technology (4-H S.E.T.). The program is used as a promotional and recruiting tool for youth with no prior 4-H experience.

Each program activity lasted for 1 ½ hours and included: What is 4-H?; Dr. Germ, a hand-washing activity and healthy eating habits, fruit smoothie; plant science, hard and soft wood; animal science, create your pet; dairy science, butter making; space science, including rocketry, robotics, and GIS/GPS; and my first 4-H project, completing a key holder. All program activities were delivered in an afterschool setting using the experiential learning model of "DO," "REFLECT," and "APPLY."

The InterWrite TM PRS (Figure. 1), an electronic classroom performance was used to encourage and enhance responses and to collect data for a pre/post test evaluation. The data was exported to Microsoft Excel for analysis. Microsoft PowerPoint © software was used in conjunction with the "Clicker," to develop the instrument questionnaire to evaluate the program's outcomes. A 31-question questionnaire (Figures 2, 3A, and 3B) guided the evaluation.

Figure 1.
"Clicker" of the InterWrite PRS

Figure 2.
Example of the Pre/Post Evaluation

Figure. 3A.
Example of a Pre-Test Question

Figure. 3B.
Example of a Post-Test Question

Results

The PRS system did increase the accuracy and response rates of data collection. Of the 600 youth participants, 75% completed the 12-session program. Ninety percent were satisfied with the use of the PRS system and had shared their experiences with another youth outside of the group. Overall, participants were very excited about using this system and expressed a desire to use it if they were allowed in their classrooms. "This is cool" expressed the participants who used the system.

There were a few disappointments for those who missed the pre-test. They were not included in the post-test and as a result were not allowed to use the "clicker." The data from the PRS was exported into a Microsoft Excel program, where the analysis was done. The results of the data analysis and data management were simple to create (Figures 4A and 4B).

Figure 4A.
Results of the Overall Program Activities Pre- and Post-Test

Figure 4B.
Graph of Program Responses

Observations

Positive Aspects of Using the System

• Very high interest and participation in the program. Participants were always anxious to use it. They were not intimidated by it, and most asked if they would receive a unit to take home


• Rapid and quick response


• A user-friendly and convenient method of data collection. More than 90% of the questions were answered, illuminating missing data


• Increased accuracy of data collection, management and analysis


• Ease in exporting data to Microsoft Excel© and SPSS© for intensive analysis if needed


• Easy to learn and use the system


• Diversity in the different responses to the questions


• Very portable. Set up can be loaded on a laptop computer

Challenges with Using the System

• All participants' information should be entered and clicker assigned prior to use. This causes a challenge if the entered participant does not show up for the pre-test (opportunity for missing data).


• Test run should be made prior to its use to eliminate problems and frustration to participants at program activities.


• The method can only be carried out by individuals who have prior training using the system.


• The system does not allow for any qualitative responses.

Implications and Recommendation

The InterWrite PRS system is a wonderful tool to use in program evaluation. It is very simple to set up and creates great interest with participants. Te system can be used extensively in most 4-H youth development programs. The system is unique and can provide data analysis at any level.

References

Barker. W., Leas. L., & Sanders, S. (2008). New faces, new places: An introduction to 4-H for urban audiences (CM-08-05). University of Nevada Cooperative Extension. Curriculum.

Massey, R., Jaskolski, N., & Sweets, L. (2005). The use of personal response transmitters in Extension Settings. Journal of Extension [On-line], 27(1) Article 6TOT4. Available at: http://www.joe.org/joe/2005december/tt4.php

Williams, V. S. (2006). Teaching and learning with technology-personal response system evaluation report [On-Line]. Retrieved February 16, 2010 from: http://clc.its.psu.edu/Classrooms/prs/PRS-Report-02122007.pdf