Using Non-reactive Methods to Study and Improve 4-H Programs

Are you tired of handing out evaluations at every meeting, but still want data to use to plan and legitimize programs? Why not try non-reactive methods?

Non-reactive or unobtrusive methods, as they are sometimes called, are used to collect data without asking for it and overcome response problems, such as selective participation. Using existing census data is an example of a non-reactive method.

The research literature describes non-reactive approaches as those methods that do not require a response from a participant. This can certainly be observation, but using existing data is also a possibility. In fact, these methods are most everything except questionnaires, tests, and surveys (Webb, Campbell, Schwartz, Sechrest & Grove, 1981).

Generally, Extension 4-H programs have rich sources of non-reactive data. This paper will identify several of these existing data sources.

Artis Koester (1989) used non-reactive methods when she studied fair exhibits to improve 4-H clothing and textiles exhibits and subsequent programs. Inspired by this study, Pennsylvania State University faculty members tried this and other non-reactive measures to evaluate state 4-H programs. They found that fairs are an excellent way of obtaining data about programs. Exhibits can be counted and compared to previous years' entries; a team of people can view exhibits and list new ideas that may be used in curriculum development and marketing; and judge's comments can be gathered and summarized and winning entries studied to help other members improve their efforts. In addition, the difficulty and workmanship of the exhibits may be noted and observations made to discover what exhibits are of greatest interest to the public.

Sometimes record books are exhibited at fairs, but more often they are reviewed as part of a record book judging event or contest. Record books can be a source of project and program ideas and are generally good indicators of how the members "connect" with the purpose of the project. For example, you might want to note the sections of the record that show a lot of activity and those that are scarcely filled out. Try to get an idea of how well the answers correspond to the questions asked. Do youth of a specific age seem to take this project more often? Information of this kind is highly valued by curriculum specialists and curriculum committees, particularly ones rewriting curricula.

In some cases, it may be useful to follow-up with phone calls to find out more about a completed activity or project. Having most of the study completed before making the calls will help you extract the few questions that will assist you the most. Also, don't try to do a record book analysis mid-year. Most record books are tucked away or have been thrown away, so accessing them at this time may be difficult. (We found this out the hard way!)

Because registration forms for events are so much a part of county and state 4-H programs, they are much less reactive than questionnaires. A few well chosen questions not only lead to a smooth running activity, but provide information that will legitimize program efforts and enhance curriculum development. For example, information about the cost of fashion revue garments is not only helpful in providing insurance for an event, but winning entries can be compared to determine if they resulted from monetary inputs or from learning.

Registration forms can also provide input to help prepare future curriculum materials. For example, when 4-H fashion revue participants were asked what they sewed outside of their textile and clothing projects, the response was largely "making gifts and mending for other people." This information was also used to add interest to the members' narrations at the time of the contest.

Judging events are good, informal indicators of how well your 4-H members are doing. Set up score cards or judging sheets so that they not only can be used to score members and establish winners, but also to analyze the data for areas where youth seem to have the most difficulty. (This analysis can also identify problems with cheating and improve the quality of contest classes the following year.) The information you find may be given as part of training meetings, used within the curriculum, or as a single "tips" fact sheet sent out shortly after the event to improve learning. One year, a potential catastrophe was averted when large numbers of youth were notified after a contest that eating rhubarb leaves wasn't more healthful than eating lettuce.

Many types of non-reactive methods may be used. In this article, we have included ones that we have tried and found successful in our 4-H program. Note that non-reactive measures do not require pages and pages of additional information gathered on forms or at events. Usually one or two questions that can be tracked periodically are sufficient. Think carefully about what you need to know and what measures would be the best way to collect the data. Be very concerned about asking personal and unethical questions, and be sure to disclose data about specific individuals only with their permission. Registration forms should not be too lengthy. We generally use a one-page form and do not ask for information that the participant must calculate or look up from another source.

Group evaluation information is made available to all the Extension agents via our electronic database, PENpages. In fact, PENpages automatically tabulates access data, so this is another non-reactive method that can be used to discover what topics and information are the most popular with agents and clientele.

Information gleaned from non-reactive measures can help you develop or re-design programs and allow 4-H members to learn more from their activities and events. Non-reactive methods should not be the only way you evaluate your programs, but they may be one way to get beyond the "after the meeting" evaluation syndrome.

Koester, A. W. (1989). Learning from state fair exhibits. Journal of Extension, 23(Spring), 32.

Webb, E. J., Campbell, D. T., Schwartz, R. D. D., Sechrest, L., & Grove, J. B. (1981). Nonreactive measures in the social sciences (2nd ed.). Boston, MA: Houghton Mifflin.