Integrating Development of Extension Materials and Formative Informal Evaluation: Land Application of Sewage Sludge as a Case Example

Background

In New Jersey, 66% of the 252,926 metric dry tons per year (N.J., 1999) of sewage sludge (biosolids) are land applied (42% out-of state, 58% in-state) (T. Pilawski, personal communication, 1999). This follows an increasing trend to land apply sewage sludge as a primary disposal option in the U.S. Despite this, land application of sewage sludge remains controversial in the agricultural community of New Jersey, the most densely populated state in the U.S. There are a number of issues associated with land application in New Jersey:

• The population density of New Jersey ensures that new housing developments are often adjacent to agricultural areas, and controversies about land use and farming practices often erupt at the local level.
• Southern New Jersey soils are shallow, acidic, and sandy. Shallow groundwater tables in much of the state make constituents of fertilizers and soil amendments more prone to leaching into groundwater.
• In southern New Jersey, vegetables and fruits are major commodities, and consumers may be sensitive about produce that has been in contact with sewage sludge because produce is often eaten raw.

Controversy over sewage sludge application can be due to numerous factors, including local management problems such as odors, vectors, stockpiling, and truck traffic. Other reasons include uncertainties and disagreements concerning the safety and appropriateness of sewage sludge land application among researchers, regulatory authorities, environmental groups, and the general public.

Due to the controversial nature of this topic, the development of an Extension program and Extension materials to provide Rutgers Cooperative Extension (RCE) agents and farmers with information on this topic was stalled in its initial stages for several years in New Jersey. As a result, a new approach was taken to develop an RCE program and Extension materials in a manner intended to minimize dissent and meet the pressing needs of Extension agents to provide materials on the subject to farmers.

This article discusses how simultaneous Extension material development and formative informal evaluation efforts can be applied to develop more responsive Extension programs for controversial issues, with land application of sewage sludge in New Jersey as a case example.

It summarizes one informal approach to gaining feedback (formative evaluation) on process and materials as they are developed. The purpose in this specific case was to gain feedback from Extension agents, farmers, Rutgers faculty, and external experts (using methods of semi-structured interviews, meeting evaluations, peer- reviews and pre-testing). The feedback was used to guide and improve the start of an Extension program on the controversial topic of land application of sewage sludge and specifically the development of related Extension materials. This approach was designed to better understand stakeholder concerns and to meet the needs of Extension agents and farmers.

Approach

If formative informal evaluations are an integral part of the process, evaluation results can provide feedback to guide and improve the process of developing Extension educational materials for a new Extension program. The data that evaluations can provide are disparate, including objective data concerning Extension agents' and agricultural community's needs, reinforcement to improve decision making regarding allocation of resources, and data to buttress decision making about policies that affect farmers.

In this case, the Extension materials included RCE's technical guidelines for land application of sewage sludge (RCE, 2001) and fact sheets for agricultural Extension agents and, eventually, farmers as target audiences (RCE, 2000). While RCE's guidelines provide technical information regarding land application, the fact sheets are for a lay audience, dealing with social, legal procedural, and technical issues. The process steps and the methods of gaining feedback to get a program started and to develop Extension materials on land application of sewage sludge are summarized in Table 1.

An informal evaluation approach was used because solving a problem rather than conducting rigorous qualitative research was the priority. Furthermore, funding was limited. A qualitative evaluation approach was used to guide and improve the process in iterative steps, which would have been more difficult to accomplish with less flexible, standardized quantitative evaluation methods.

However, we emphasize that this is not rigorous social science research because our evaluation approach was limited not only by the sample size of the population (selected Extension agents in New Jersey) but also by the questions that were asked. Questions were focused on program development or on the clarity of materials that were developed. They did not test the targeted audience's understanding of the materials (e.g., questions about facts) or ask questions that dealt with their perceptions of risk. Also, this program did not employ quantitative methodologies. Information on more rigorous social science approaches to evaluation can be found in Patton (1997) or Morgan, Fischhoff, Bostrom, Lave, & Atman (1992).

Results of This Process

Workshop Preparation and Telephone Interviews

Five agricultural agents in field, fruit, and vegetable crop areas were interviewed using a semi-structured interview method. These interviews were not meant to be generalized to all agricultural Extension agents, but rather to develop a better in-depth understanding of agents' feelings and concerns.

The interviewees were selected by so-called purposeful sampling of information rich cases (Patton, 2001). The interviewer chose the interviewees from a list of agricultural agents who were known to be involved with the topic and who specialized in different commodity areas. Because only five interviews were conducted, these results were used to develop a workshop agenda, but general conclusions could not be drawn. The interviewer was not part of RCE, was previously trained as a professional interviewer, and was not involved with this issue before. Despite differences in answers, a substantial overlap was found in these interviews (Table 2).

Previous program development efforts in this area at RCE had focused solely on technical issues. However, these initial interviews showed that social and legal issues were just as important. The interviews also showed that the interviewed agents had concerns regarding land application that they wanted to articulate and that they felt that technical guidelines were needed to enable agents to respond appropriately to farmers' inquiries. Using data results from the interviews, the following workshop agenda was developed:

• Listing and ranking of concerns about land application of sewage sludge.
• Discussion of concerns with experts from the Cornell Waste Management Institute. This was requested by agricultural agents who had heard about the publication "Case for Caution," which raises concerns about land application of sewage sludge.
• Status of RCE's technical guidelines for land application and Rutgers' official position on land application.

Workshop and Meeting Evaluations

Twenty-one agricultural agents and specialists volunteered to participate in this workshop. This was considered a large number of participants because it was the beginning of the growing season, the busiest time of the year for agents. Because all stakeholders were so anxious to resolve the issues involved, the workshop could not be scheduled at a more convenient time.

At the beginning, all participants described their top three concerns regarding the use of sewage sludge, and a list of concerns was created. Extension agents' and specialists' concerns, in order of priority were:

• Liability,
• Quality of sewage sludge,
• Limitation of future land use,
• The need for RCE's technical guidelines, and
• Unknowns.

During the discussion about the need for RCE's technical guidelines, a representative of the RCE administration emphasized how important it is that all information RCE provides is supported by sound scientific studies. During the discussion about "where to go from here," the suggestion to form an RCE Biosolids Working Group was made. The tasks of the group were 1) to develop fact sheets, including a fact sheet entitled "Questions to Ask Before Considering Application on Farmland" for farmers and Extension agents, and 2) to review and revise RCE's technical guidelines for land application of sewage sludge on agricultural land.

Thirteen of the 21 participants filled out the evaluation forms after the meeting. Only 13 responded, because, due to the inconvenient time, some individuals left early. All of the evaluations indicated that the meeting was well organized, productive, and useful for their work. When asked about the usefulness of various agenda items, respondents considered the initial discussion about their concerns and the discussion with the Cornell Waste Management Institute most useful. When asked what topics they would like to have covered at future meetings, respondents mentioned:

• RCE's technical guidelines,
• Health issues,
• New crops,
• Effect of heavy metals on horticultural crops,
• Outreach materials used in surrounding states, and
• Growers' perceptions.

In addition, several requested the chance to review RCE's technical guidelines as they were revised. When asked for additional comments, one respondent said, "We have waited long enough. Make this a high priority, and make a stand on what we want to say to growers and the public."

The evaluation form included a list of potential fact sheet topics, which was created based on the telephone interviews. Two issues received top ranking, liability and technical guidelines for sewage sludge use. These issues were followed in importance by sewage sludge quality and variability, heavy metals in sewage sludge, regulations, and human health issues.

Biosolids Working Group and Meeting Evaluations

The Biosolids Working Group consisted of 11 members, mostly Extension agents and specialists. Areas of expertise included:

• Field and forage crops,
• Vegetable crops,
• Dairy livestock,
• Soil fertility,
• Solid waste management, and
• Environmental communication.

The diversity of the group ensured that various aspects of sewage sludge land application were covered. In order to include more Extension personnel in the discussion, members of the working group discussed relevant issues in their commodity groups. When necessary, other experts and commodity groups were contacted for factual advice. For example, an environmental law professor was contacted about legal issues.

During the nine subsequent working group meetings, members presented information about certain topics, which were identified at each previous meeting, and RCE's technical guidelines and fact sheets were reviewed. After RCE's technical guidelines were revised and the fact sheets were drafted based on working group discussions, the outreach materials were discussed with the RCE administration, which would be ultimately responsible for the release of the developed materials. A lot of time was spent preparing for each meeting to ensure they would be productive and to reduce the number of meetings necessary to accomplish the task.

All meetings were anonymously evaluated. The responses indicated that the meetings were considered very productive, well organized, and useful for the audience. Therefore, not many improvements from meeting to meeting were needed. There were only a few comments that needed to be addressed, such as "Keep closer control over the agenda items. Although the discussion was good, we moved around a lot."

Peer-Review and Pre-Testing

Based on feedback from interviews, evaluations, and meetings, the working group revised RCE's technical guidelines for land application of sewage sludge and developed eight new fact sheets. Fact sheet topics are:

• Questions to Ask Before Considering Application on Farmland
• Regulations and Guidelines
• Different Types of Sewage Sludge
• Guidelines for Land Application in Agriculture
• Heavy Metals
• Soil Amendments and Heavy Metals
• Organic Contaminants
• Pathogens

RCE's guidelines and fact sheets were then peer-reviewed. In addition, the fact sheets were pre-tested with the intended audiences. Other issues that were identified at the workshop (e.g., liability, perception, sewage sludge composition) were addressed in journal articles because some of these topics required more research and others were not appropriate for a fact sheet (Goldfarb, Krogmann, & Hopkins, 1999; Krogmann, Gibson, & Chess, 2001; Krogmann & Chiang, 2002).

The peer-review of RCE's technical guidelines focussed on technical accuracy. Peer-reviewers were the Biosolids Working Group, Rutgers faculty, and external experts. The peer-review of the fact sheets was more elaborate and included three steps:

• Rutgers' faculty and the Biosolids Working Group reviewed the fact sheets with regard to providing accurate and appropriate information.
• The Biosolids Working Group reviewed each fact sheet to ensure agents and farmers could easily understand the fact sheets and that they would answer their questions and concerns.
• The Center of Environmental Communication reviewed the fact sheets with the following questions in mind:
  1. The key audience,
  2. If language, topic, and information included were appropriate for the intended audience, and
  3. If the fact sheets addressed the audience's needs and concerns.

Besides specific comments addressing individual fact sheets, the following general suggestions were provided by the Center of Environmental Communication:

• Address the uncertainty associated with sewage sludge use, including to the fullest extent possible.
• Use a consistent format for all fact sheets to make them more user friendly and more identifiable.
• Include the following in every fact sheet:
  1. Define purpose and the audience of the fact sheet,
  2. Explain the fact sheet issues,
  3. Discuss land application focusing on a New Jersey audience,
  4. Offer recommendations,
  5. Include references to other fact sheets in the series and contact information,
  6. List where to obtain further information.
• Package the fact sheets together in a folder.
• Develop different fact sheets for two target audiences: agricultural Extension agents and farmers. If not possible, use lay language, and include a glossary where needed.

Comments from the various peer-reviews were incorporated in RCE's technical guidelines and in the fact sheets. The next step was to pre-test the peer-reviewed fact sheets with the targeted audience. The fact sheets were pre-tested at grower meetings and County Board of Agriculture meetings. These meetings were chosen to pre-test the fact sheets, because the target audience attended these meetings and because the organizers of these meetings supported the pre-testing. As an incentive for participating in the pre-testing, homemade cookies were distributed to the participants.

For materials intended to be widely disseminated, pre-testing with a larger survey encompassing a greater cross-section of people is recommended. Because we assumed limited distribution for our materials, a less rigorous, informal pre-testing was conducted. In addition, a focus group with representatives of the intended audience may provide additional insights.

The main results of pre-testing of one fact sheet at a County Board of Agriculture meeting are provided below (Table 3). Each fact sheet was pre-tested to find out if it was clear and understandable and if it met the audience's needs. Filling out the pre-testing questionnaires took about 10-15 minutes.

The results showed the fact sheet was well designed and clear, although there was still work to be done. Main conclusion based on the pre-tests was to cross-reference other fact sheets because questions about issues such as liability and regulations were raised. There were many comments about the tone of the fact sheet (both positive and negative), pointing to the need to use value- and jargon-free language. To address this, the fact sheet benefited from reviewers who were sensitized to this controversial issue at the local level and could provide more neutral language.

Implications and Conclusions

The integration of the development of Extension materials and formative informal evaluations (semi-structured interviews, meeting evaluations, peer-reviews, pre-testing) addressed the controversial issue of land application of sewage sludge and successfully helped in the development of RCE's technical guidelines (RCE, 2001) and outreach materials (RCE, 2000). After not having a program for several years, these outreach materials were published. The novel aspect of our materials is that they also attempt to address social and legal issues.

Key for the success of this process was the use of informal evaluative methods on an ongoing basis and the ongoing participation of the intended audience. In addition, the expertise in environmental communication was essential for the evaluation portion of this process.

Without this process, the materials would still not exist. However, the process is very time consuming, requiring many hours of preparation and revisions of the materials and background information.

The integration of Extension material development and formative informal evaluations tries to address the needs of the target audience. In our case, the outreach materials were designed to address RCE agents' and farmers' concerns such as long-term soil productivity, plant growth, and liability. They do not address the concerns of other groups, such as sewage sludge generators, and may be objectionable to them. Our overall goal was to provide balanced information to RCE agents so that they can help farmers make more informed decisions.

References

Goldfarb, W., Krogmann, U., & Hopkins, C. (1999). Unsafe sewage sludge or beneficial biosolids: Liability, planning, and management issues regarding the land application of sewage treatment residuals. Boston College Environmental Affairs Law Review 26, 687-768.

Krogmann, U., & Chiang, H.-N. (2002). Selected nutrients and heavy metals in sewage sludge from New Jersey POTWs. JAWRA 38, 681-692.

Krogmann, U., Gibson, V. & Chess, C. (2001). Land application of sewage sludge: Perceptions of New Jersey vegetable farmers. Waste Management & Research 19, 115-125.

Morgan, M.G., Fischhoff, B., Bostrom, A., Lave, L., & Atman, C.J. (1992). Communicating risk to the public. Environmental Science and Technology, 26, 2049-2056.

N.J. (1999). Sludge Quality Assurance. 31 New Jersey Register 200 from February 1, 1999.

Patton, M.Q. (1997). Utilization focused evaluation: The new century text. Thousands Oaks, CA: Sage Publications.

Patton, M.Q. (2001). Qualitative research & evaluation methods. Thousands Oaks, CA: Sage Publications.

Rutgers Cooperative Extension (2001). Guidelines for the land application of sewage sludge for New Jersey agriculture. Edited by Krogmann, U., Boyles, L.S., & Heckman, J.R. Rutgers Cooperative Extension Bulletin #E228. New Brunswick, NJ. Available at: http://www.rce.rutgers.edu/pubs/subcategory.asp?cat=4&sub=31

Rutgers Cooperative Extension (2000). Land application of sewage sludge (biosolids). Series of 8 fact sheets. Rutgers Cooperative Extension Fact Sheet #FS951, #FS952, #FS953, #FS954, #FS955, #FS956, #FS957, and #FS958. New Brunswick, NJ. Available at: http://www.rce.rutgers.edu/pubs/subcategory.asp?cat=4&sub=31