Collaboratively Evaluating Cooperative Extension Educational Interventions

Background

Cooperative Extension has been challenged to formally evaluate state and county programs. As Cooperative Extension responds to this challenge, one important question is, "How can university research faculty and county agents better work together to formally evaluate educational programs?"

Several additional questions underlie this basic question. "Can university researchers maintain a specific, rigorous research protocol when implementation of the protocol rests largely with field educators?" "Are agents able to devote adequate time to experimental programs as a part of their routine workload?" And, "Is an agent able to realize significant county-level benefits from participating in a formal research project where the impetus for the research is external to any expressed need of the county?"

This article answers these questions, using the experiences of the Pennsylvania Central Region Farm Safety Pilot Project (PACRFSPP). We begin with a description of the interventions and then examine the role of the agent in the recruitment process and interventions.

The PACRFSPP was an agricultural safety and health project testing three different intervention models to reduce hazards and risks of farm work. The Central Region encompassed 17 of Pennsylvania's 67 counties in 1994. Counties chosen for the study were predominantly rural and typical of Pennsylvania agriculture (Murphy, Kiernan, Hard, & Landsittel, 1998). Other criteria for participation in the project included a sufficient number of farms from which to draw a sample, a minimum number of tractors, PTO-driven machines, buildings and structures, and county offices staffed by experienced Extension professionals with educational responsibilities that matched the intervention models.

The three interventions had 38 common educational objectives from which to choose to help ensure that similar educational programming existed across the three intervention counties (See Appendix). However, agents were neither precluded from using information or conducting educational programs involving non-intervention objectives, nor prohibited from combining intervention and non-intervention educational objectives within the same educational program or activity. The interventions were conducted during 1996 and 1997, with baseline data collected in 1995 and post-intervention data collected early in 1998. A total of 256 farm operations participated in this project.

The primary target of the study was the farm operator, and four areas of farm safety were selected: tractors, power take-off (PTO) driven machines, buildings and structures, and emergency preparation. Farm operators were randomly selected based on screening criteria: the farm operator had to have at least two tractors and three PTO machines, and expect to be farming in 5 years. Baseline data and post-intervention data were collected by a combination of active and recently retired county agents and agricultural educators in the participating counties. Details regarding the project rationale, methods, and baseline data analyses are reported elsewhere (Murphy et al., 1998; Landsittel, Hard, Murphy, & Kiernan, 1998). Reports on post-intervention findings are in progress.

The Three Interventions

Three counties were used for different interventions: 73 farm operations in a self-audit intervention, 30 farm operations in a youth intervention, and 41 farm operations in a community coalition intervention. Two other counties were used as controls, with 72 farm operations and 40 farm operations participating, respectively. In one of the control counties, only post-intervention data was collected to control for bias that may have resulted from participating in the baseline audit.

Intervention A: Farm Hazard Audit Program (FHAP)

The objective of the self-audit intervention was to determine the effectiveness of using hazard self-audits and systematic follow-up by a county agent with farm operators to reduce risk of injury from farm work. Two self-audits by the farm operator were conducted. The first self-audit was conducted between 8 and 9 months after the intervention began, with the second self-audit conducted 8 to 9 months after the first self-audit. The self-audits assessed the farms for the presence and condition of safety protective devices in the four targeted areas. For example, farm operators were asked to assess, among other things:

• The safety condition of rollover protection and master shields on tractors,
• PTO driveline guarding and use of warning signs on machinery,
• Fire extinguisher protection,
• Electrical service covers and housekeeping in buildings, and
• Written directions for emergency service providers for use by family members in an emergency.

The farm operator then mailed the self-audit to the county agent for scoring. After scoring, each farm operator received the results and was encouraged to seek additional assistance from the agent in correcting specific hazards. Hazard scores were marked at three different levels, with a different color indicating a different level of risk. A hazard score marked in red indicated a high risk of injury, a score marked in yellow indicated a moderate level of risk, and a score marked in green indicated a low risk of injury. Farm operators were encouraged to correct high-hazard risks quickly, to develop a management plan for correcting moderate risks in the near future, and to correct low risks whenever possible.

Intervention B: Youth Safety and Health Program (YSHP)

The objective of youth intervention was to target youth ages 7 to 17 as change agents within their own families to help reduce risk of injury from farm work. Minimal requirements for implementing this intervention were that the agent, each year of the intervention, communicate with participants at least quarterly and have two safety and health educational activities.

The communications and activity requirements were far exceeded. Examples of communication contacts include a quarterly newsletter and safety fact sheets distributed through schools. Examples of educational activity programs include farm safety day camps, safe tractor and machinery operation training, and a fire extinguisher give-away program. Thirty farm operations with a total of 64 children participated in this intervention. Several of the contacts and activities included the total farm family.

Intervention C: Community Action for Agricultural Safety and Health (CAASH).

The objective for community coalition intervention was to determine the effectiveness of having Cooperative Extension organize community leaders, organizations, and businesses for the purpose of creating a proactive forum for farm safety and health within the farm community to reduce risk of injury from farm work. The county agent sought to bring together traditional agricultural groups (e.g., farm service organizations and vocational agriculture) and non-traditional community groups, organizations, and key individuals (e.g., local doctors and nurses, insurance agents and bankers) to form an identifiable, self-sustaining community coalition around the single issue of farm safety and health.

The coalition that developed consisted of 17 members: two private citizens, one conservation district manager, four members of farm organizations, four fire co./ emergency medical rescue team members, two Cooperative Extension and outreach representatives, two FFA members, one vocational agricultural teacher, and one insurance company representative. Examples of coalition activities include a cost share program to help farmers purchase rollover protection for tractors and guards for machinery, a youth farm safety day, and distribution of safety fact sheets to all farms in the county.

Intervention Effort

All three interventions involved significant educational effort by county Extension agents that can be categorized as either information only (e.g., newsletters to participants, newspaper articles, radio messages, literature handouts at county fairs) or active (e.g., hands-on demonstrations, fund raising activities, ROPS retrofit campaign), and as targeted toward intervention objectives or non-intervention objectives. (A farm safety program or activity that was conducted by cooperative Extension in the county but that did not include intervention objectives was categorized as involving non-intervention objectives.)

Agents were encouraged to develop objectives, similar to the intervention objectives, for these other programs and activities. An example of a non-intervention objective related to tractors was, "Participants will learn that on all ROPS equipped tractors, the operator should use the seat belt when operating the tractor." An example of a non-intervention objective related to buildings was, "Participants will learn that it is best to store any flammable fuels, oils, lubricants, etc., in separately constructed noncombustible sheds or buildings."

In the self-audit and youth interventions, the level of active programs involving intervention objectives significantly increased during the intervention period. In the community coalition intervention, a significant information program involving the intervention objectives began, and the active programs, while not increasing in total number, did involve many more of the intervention objectives. This was because in the pre-intervention period, the same two programs (with the same objectives) on tractor and PTO safety were presented repeatedly to different school groups, but during the intervention period, a wider range of intervention objectives were involved in the various active programs. The breakdown of the increased effort in all three intervention counties is shown in Table 1.

Table 1
Comparison of Pre-Intervention and Intervention Educational Effort by Interventions and Controls

Interventions	Time Period	Information			Active			Grand Total
		IO3	Non-IO4	Total	IO	Non-IO	Total
Self-Audit	Pre-intervention1	17	15	32	1	0	1	33
	Intervention2	6	3	9	9	1	10	19
Youth	Pre-intervention	5	9	14	4	1	5	19
	Intervention	19	0	19	19	0	19	38
Community Coalition	Pre-intervention	0	0	0	16	0	16	16
	Intervention	12	0	12	16	0	16	28
Pre/Post Control	Pre-intervention	6	10	16	3	1	4	20
	Intervention	6	3	9	2	1	3	12
Post Only Control	Intervention	3	0	3	4	3	7	10
1 Pre-Intervention: 1994-95 2 Intervention: 1996-97 3 Targeted toward Intervention Objectives 4 Targeted toward Non-Intervention Objectives

Although the self-audit intervention shows more overall effort in the 2 years before the intervention (33 efforts vs. 19 efforts), active efforts involving intervention objectives increased during the intervention period (9 efforts vs. 1 effort). Most important, the nature of this intervention means that all nine active efforts targeted to educational objectives included participation by the same group of farm operators. This type of effort runs counter to most safety and health educational programs because most are conducted once-a-year, if even that often.

The nature of interventions in the youth and community coalition interventions do not lend themselves to making such strong and clear statements about safety and health education efforts for those farm operators. Anecdotal records in these counties, however, do indicate that all participating farm operators had much greater exposure, through information and active programs, to the 38 common educational objectives than they had during the preceding 2 years.

Project Problems and Successes

Several problems and successes common to the three interventions and control counties are identified. Both the researchers' perspective and the agents' perspective are given. These problems and successes are best understood by examining issues relating to the recruitment protocol and the ownership agents developed by conducting the interventions.

Recruitment Protocol

Using agents to recruit participants by following a precise protocol was largely successful from the perspective of the researchers but not from the perspective of the agents. The recruitment protocol involved agents telephoning a list of farm operators in their county to ask if they would participate in the project. The researchers generated a list of all farm operators in the county and listed them in random order. The agents were to call farmers in the order they appeared on the list, and they were to speak to the farm operator only (a requirement for informed consent) concerning their potential participation in the project.

Agents were also required to use structured questions to ensure that potential participants were eligible to participate in the project and to standardize the recruitment process. The recruitment for each county was to be completed before any of the baseline data could be collected. All of these techniques are standard research protocols to maximize generalizability of results.

Although the recruitment process took approximately three times longer than originally planned (9 mos. rather than 3 mos.), the researchers' objective of recruitment of a sufficient number of participants that represented farm operators in the participating counties was achieved. An average response rate of 82.7% was achieved in the intervention counties (Murphy et al., 1998)

Agents, however, had a great deal of trouble with this protocol. These problems can be summarized as follows.

1. The structured telephone recruitment process was not a comfortable fit for the way agents normally interact with farm operators. They strongly preferred unstructured, naturally occurring face-to-face contacts with farmers whom they knew to be supportive of Cooperative Extension within their county.

2. Participant farm operators had to be contacted twice, once for the initial recruitment and then a second time to schedule baseline data collection. This proved to be a very time-consuming process. Agents preferred to make only one contact.

3. Because of the difficulty of making telephone contact with eligible farm operators and the need to make a second contact to schedule baseline data collection, agents felt they were not sufficiently informed of the amount of time this part of the project might take from their regular Extension work. Additionally, because the recruitment and baseline data collection took much more timethan expected, the activities stretched across busy farming periods for the farmers. Agents did not appreciate having to contact farmers during some of the farmers' busiest times of the year.

Intervention Ownership

The researchers felt that intervention ownership was best achieved by–in addition to recruiting participants–having agents collect baseline data, conduct the interventions, select intervention objectives, and keep good records of their intervention efforts. Agents were able to identify three positive results of their participation in collecting the baseline data.

• First, collecting the baseline data gave agents an opportunity to either meet new farmers in their county or visit those they already knew, at their farms.
• Second, face to face contact with their clients helped gain support for other programming efforts and activities.
• Third, the hazard audit portion of the baseline data helped the agents define common problem areas throughout their county and assisted in program planning.

The audits also increased the agents' understanding of safety issues and increased their knowledge of the dangers in their county that are not receiving sufficient attention.

From the researchers' perspective, the baseline data collection process was successful. All agents participated in a program where researchers explained the data collection process and provided training for conducting the hazard audit. Agents then demonstrated their competency in conducting hazard audits (a Kendall's reliability score of .94 was achieved). A review by the researchers of data forms as they were received showed that data was collected and returned to the researchers according to protocol.

Agents did identify the time required to keep the detailed records for the project as a problem. Agents also felt that they did not receive a timely benefit from the record-keeping activity. As is often the case in research projects, the researchers were content to wait until the end of the project to compile and analyze the data. Agents, on the other hand, are required to show results and impacts of major program commitments to county constituents and administrators at the end of each program year.

Agents identified several positive side effects of their involvement with the interventions. For instance, agents observed that groups other than county Extension began to sponsor or conduct safety programs that they had not sponsored or conducted before the implementation of the interventions. Examples are Young Farmers' groups working with machinery dealers to do safety demonstrations, schools requesting agents to do safety demonstrations, and groups initiating First on the Scene for Farm Families training. While these additional programs and activities cannot be directly linked to the implementation of the interventions, it is reasonable to suggest that the interventions heightened the awareness of, and concern for, safety issues in the counties, and that these non-Extension groups were responding to this heightened awareness and concern

Agents in the intervention counties independently came to the conclusion, based on their experiences conducting the interventions, that farm safety activities should be targeted to the whole family for maximum participation. They found much greater participation in farm safety programs and activities when the parents and children attended safety days together, participated in hands-on activities at home together, or were involved at county fairs or 4-H activities as a family. This finding was not expected.

Summary and Conclusions

The Pennsylvania Central Region Farm Safety Pilot Program (PACRFSPP) provided an opportunity to explore several questions regarding university researchers and county agents working together to design, implement, and evaluate a multifaceted education evaluation project. All agents felt that their county's intervention was successful, and that, perhaps most important, they would continue to pursue using that intervention in future programming. They plan on expanding their safety programming in the future, using the information obtained from the intervention activities. Thus the question "Can county Extension benefit from participation in a formal university research project?" is answered in a very positive way.

On the other hand, none of the agents would willingly become involved in another research project if the same recruitment process were used. Although the researchers and agents agree that farmers are generally more receptive to the request of a familiar county agent than to that of an unknown university researcher, the experience from this project suggests that recruitment of participants should remain the responsibility of researchers.

Closely related to this is the issue of time commitment to the research project. Although all agents did spend the time necessary to successfully complete the project, it was not achieved without considerable sacrifice of other work responsibilities and without the researchers allowing considerable more time for activity completion. Thus the questions "Can university researchers maintain a specific, rigorous research protocol when implementation of the protocol rests largely with field educators?" and "Are agents able to devote adequate time to experimental programs as a part of their routine work load" were affirmed, but in a much less satisfactory manner.

In the final analysis, despite some difficulties, researchers and agents did accomplish the primary goal of the research project, namely, to scientifically evaluate models of safety education. This result suggests that the Cooperative Extension System can successfully meet the challenge of formal program evaluation when university researchers and county agents work together.

References

Murphy, D. J., Kiernan, N. E., Hard, D. L., & Landsittel, D. (1998). The Pennsylvania central region farm safety pilot project: Part I-rationale and baseline results. Journal of Agricultural Safety and Health, 4(1), 25-41.

Landsittel, D., Hard, D. L., Murphy, D. J., & Kiernan, N. E. (1998) The Pennsylvania central region farm safety pilot project: Part II-Baseline data associations between approach-to-safety and hazard conditions. Journal of Agricultural Safety and Health, Special Issue No.1, 21-28.

Acknowledgments

The authors would like to acknowledge the creativity and commitment to the scientific demands required in such a study demonstrated by the agents who delivered the three interventions over the 2-year period: Robert J. Brown, William R. Hosler, and Walter E. Whitmer. The authors would also like to acknowledge the effort made by other agents in implementing the evaluation process in both the intervention and control counties.

Appendix: Common Educational Objectives

Intervention Objectives

Participants will learn that:

1. It is best if all tractors are equipped with, in order of preference, a full cab ROPS, a 4-post ROPS, or a 2-post ROPS.
2. All tractors should be equipped with a PTO master shield.
3. It is best if all tractors are equipped with two working headlights and warning flashers, and an SMV emblem in good condition.
4. All tractors should have two rear wheel fenders.
5. All tractors should have a bypass starter cover and warning decal on all easily accessible starter motors.
6. All tractors should have a seat that is in good condition, and has a back rest, arm rests, and a good suspension system.
7. All tractors should have clean and cleared mounting steps and operator station platforms.
8. All machinery PTO driveline shafts should be guarded.
9. There should be a machine master shield on all PTO driven machinery.
10. A PTO driveline guard should be rotated to test its effectiveness after the PTO is hooked to the tractor.
11. All PTO driven machinery should have PTO hazard warning decals placed on or in the immediate vicinity of PTO driveline guards.
12. A fully charged, mounted, ABC type fire extinguisher should be located within 50 feet of all farm shops.
13. Storage of flammable fuels, oils, lubricants, etc., in farm shops should be in U.L. approved fire safety containers.
14. Good housekeeping practices in farm shops will minimize tripping, falling and fire hazards.
15. Is best to store any flammable fuels, oils, lubricants, etc., in separately constructed noncombustible sheds or buildings.
16. A fully charged, mounted, ABC type fire extinguisher should be located within 50 feet of all silo unloading rooms/areas.
17. Good housekeeping practices in silo unloading rooms/areas will minimize tripping and falling hazards.
18. Good housekeeping practices in silo unloading rooms/areas will minimize electrical shock and fire hazards.
19. Hazard warning signs/decals should be placed at all entrances to silos.
20. A fully charged, mounted, ABC type fire extinguisher should be located on each level of a two-story bank barn.
21. Good housekeeping practices in two-story bank barns will minimize electrical shock and fire hazards.
22. Hay/feed drop openings in two-story bank barns should be provided-with fixed ladders, access handholds, and a cover over the opening.
23. If pesticides are stored on the farm, the pesticides should be in a separate, lockable structure in original containers with labels intact, decontamination equipment and supplies should be stationed in the immediate area, and hazard warnings should be posted at the entrance to the storage structure.
24. Hazard warning signs/decals should be placed at all entrances to grain bins.
25. Access to a manure storage pond or structure should be restricted and that hazard warning sign should be visible from all accessible sides.
26. Hazard warning signs/decals should be placed at all entrances of manure storage structures.
27. Directions to the farm and guidelines for how to report farm emergencies should be written down and posted by a telephone.
28. The rescuer's safety takes precedence over the accident victim's safety.

Non-Intervention Objectives

Participants will learn that:

29. On all ROPS equipped tractors, the operator should use the seatbelt when operating the tractor.
30. Centrifugal force, rear axle torque, improper hitching and center of gravity cause tractor instability and rollover.
31. Teenagers are injured more often as extra riders on tractors than they are as tractor operators.
32. There is no safe place to ride on a tractor other than in the operator's seat.
33. Tractor headlights and warning flashers should always be on when operating on a public road.
34. You should never jump-start a tractor from the ground.
35. Operators should face the tractor when mounting and dismounting tractors.
36. Tractor accidents are the most common and serious type of farm accident.
37. A person should never step over a rotating PTO machine driveline.
38. Most PTO accidents involve: an operator becoming entangled by an unshielded PTO driveline; the initial contacts with the machine's first universal joint; occur while the tractor and machine are stationary; and are not usually fatal to the victim.