A User Evaluation of a Decision-Support System: The Community Assessment Model for Odor Dispersion (CAM)

Introduction

Finding balance between the economic viability of swine production and protecting rural quality of life mainly free from objectionable odors has become one of the 21st Century's great agricultural challenges (Donham et al., 2007). A main U.S. policy tool designed to minimize conflicts regarding odor is a legally determined separation distance between production sites and potential odor receptors (Bazen & Fleming, 2004). This strategy, however well meaning, has resulted in social conflict within rural communities and negative impacts to the animal industry anyway. Problems largely manifest themselves as costly nuisance litigation (Heber & Bogen, 2006; Huang & Miller, 2006). Separation distance alone does not to account for ambient effects of existing odor sources in a community (e.g., all livestock sites in the landscape), or the effect of local weather patterns on odor dispersal (Guo, Jacobson, Schmidt, & Nicolai, 2003).

A better approach would be to provide the industry and community a procedure for making prudent decisions regarding appropriate facility location. Decisions could therefore be made not only on separation distance, but also historical weather patterns, scale of production, odor control measures already in place, and additional odor loads in a community (Johnson, 2006). To this end, Hoff, Bundy, and Harmon (2008a) developed a producer decision-support system (DSS) called the Community Assessment Model for Odor Dispersion (CAM) in which key parameters predict odor strength levels downwind. CAM helped to locate new hog production facilities in the state of Iowa for over 150 specific cases since June 2005. CAM is a voluntary process initiated by the farmer and facilitated through a joint effort between the Coalition to Support Iowa's Farmers (a non-governmental organization) (CSIF; <www.supportiowasfarmers.org>), the Extension-linked Iowa Pork Industry Center (Pork Center <http://www.ipic.iastate.edu/>), and Iowa State University.

The use of CAM is now encouraged by the state of Iowa via Iowa House File 2688 Livestock Applied Research and Evaluation. The 2008 bill authorized three state institutions (Iowa State University, Iowa Department of Natural Resources, and the Iowa Department of Agriculture and Land Stewardship) in an Extension-facilitated, producer-centered, voluntary site evaluation process that culminates with CAM modeling (IPPA, 2008). HF 2688 remains unfunded at this time, but the bill sets the stage for future appropriations aimed at expanding CAM programming.

Because CAM can help minimize the risk of costly odor conflict, it is vital that CAM is used appropriately and efficiently with a user friendliness that encourages: 1) the use of the model in the first place and 2) understanding and trust in the model so that informed siting decisions are made that benefit individual producers, the industry, and rural communities. Therefore, as part of CAM's overall programming, CAM user feedback is collected periodically to evaluate the overall Extension (outreach) process of CAM as well as assess the relative impact of CAM on neighbor relations. We conducted our first follow-up survey of CAM users in 2008.

The purpose of this article is two-fold: 1) to introduce to the agricultural Extension community (particularly those associated with livestock production and community development) a DSS that is helping swine producers in Iowa minimize potential odor conflict, and 2) to share our rationale and approach to evaluating CAM and its outreach approach.

The Community Assessment Model

From an operations stand point, CAM can currently model up to 20 swine-related sources and up to 100 receptors in a land area of any size. Key parameters that CAM accounts for are: overall size of the production system, type of swine production system, local historical weather conditions, and various odor control technologies that have documented effectiveness ratings. In terms of model output, CAM predicts the number of hours of exposure to various levels of odor (based on dilution factors), by month, for each receptor in a given community.

Thus, CAM views odor dispersion from a receptor's point-of-view, determining the odor impact on each receptor from all sources in the community. This is in stark contrast to source-based models, where radius of odor influences around the source are predicted, with no direct link to the receptors in the community (e.g., Guo, Jacobson, Schmidt, Nicolai, & Janni, 2004). For a more technical explanation of how CAM works, see Hoff et al., (2008a) and Hoff, Bundy, Harmon, and Johnson (2008); for a review of how CAM has been field validated see Hoff et al., 2006).

Currently, the use of CAM is voluntary and is put into action at the request of a producer. A producer contacts a designated ISU Extension livestock specialist and requests that CAM models a prospective site (or sites). The specialist interviews the producer (usually in person at the prospective site) to gather input data in order to map potential sites relative to community receptors and existing animal-related odor sources. ISU Department of Agricultural Biosystems Engineering faculty members then implement CAM, analyze results, and produce a one-page report. During a follow-up visit with the producer, results are thoroughly explained.

The cost of CAM is roughly $1,000 per site (mainly Extension and faculty time) and is currently borne by ISU and CSIF. Once the results are explained to producers, if a site is deemed acceptable and the producer plans to continue with site development, producers are encouraged to share model results with neighbors to explain their planning process and the steps being taken to proactively minimize risk of odor exposure to surrounding areas.

Evaluating Decision-Support System Tools in Agriculture

Decision-support systems such as CAM that link producers to sound, science-based information can facilitate the implementation of effective environmental stewardship (Harrison, Kanade, & Toney, 2004). Yet the most powerful decision-support tools have no impact if they are not used. As noted by Newman, Lynch, and Plummer (2000), despite the various benefits of using technical decision-support systems, these systems are often not widely used.

Some of the reasons behind the low adoption of DSS within agriculture have been listed by Newman et al. (2000) and Walker (2002) and are as follows.

• Farmers' perception of limited reason for changing current management methods,


• Limited opportunity and support for learning about and troubleshooting tool usage,


• Complexity of decision implications as guided by tools,


• Distrust for the output of such tools because producers do not understand the underlying theories of the models,


• Tendency to treat tools as "black-boxes" that effectively remove users from responsibility.

The use of CAM as a DSS for odor risk management, however, appears to counter many of these listed factors in that: 1) the legal and economic impetus for odor risk management as facilitated by CAM is paramount to the viability of any production facility; 2) while voluntary, the use of CAM is facilitated by three of the primary livestock Extension/support entities in Iowa and will soon be hard-lined into facility siting protocols at the state level; 3) while the CAM modeling process is complex and environmental risk in general is complex, the bottom-line results of CAM are relatively straight forward in presentation; 4) distrust of model output should be minimal as model facilitators are explicitly in place to support a socially and economically healthy livestock industry in Iowa.

All of this points to expanded use of CAM. The issue for CAM outreach is to continually seek ways to improve efficiency and clarity of process particularly in light of expanding Extension programming needs. In order for hog producers to use the information encompassed within the modeling process, they need to understand what the model is doing and ultimately concluding (Anderson & Feder, 2004; Douglah, 1998).

In order to evaluate CAM we have undertaken the collection of key stakeholder information for the purpose of making decisions regarding program approach and to improve effectiveness of the outreach process. Ultimately while the engineering side of CAM is judged in terms of accuracy and precision, from the standpoint of applied science, usefulness to producers in making better decisions becomes the primary test (Carberry et al., 2002). As such, program evaluation should be integral to program accountability (Petheram, 1998; Koelsch, Howard, Pritchard, & Hay, 2000). Specifically, Harrison and Toney (2004) noted that if Extension programming is to efficiently deliver information then it is necessary to listen to the "voice" of the customer.

Therefore, to evaluate our outreach programming and assess the impact that CAM has on producer decision-making and outcomes, we employed a straightforward CAM user survey. The survey was designed to provide information from two different, but critical user perspectives (Radhakrishna & Relado, 2009): 1) formative information that will be used to make improvements in programming efficiency and usability and 2) summative feedback in the form of a general assessment of the broader impact that CAM has had on reducing odor related conflict.

Out of the 150 producers who have had production sites modeled in Iowa between 2005 and 2008, 75 producers were identified as having had enough time since modeling to provide hindsight observations about the modeling process and discuss their opinions regarding post-modeling outcomes. The survey was conducted via telephone interviews and followed Dillman, Smyth, and Christian (2008) tailored design telephone survey protocols. Professional interviewers with the Center for Survey Statistics and Methodology (Iowa State University Department of Statistics) collected data in the Fall of 2008. Interviews were completed with contact people for 62 of the facilities, for a response rate of 83%. All data were compiled and analyzed using SPSS 17. Descriptive frequencies are presented below.

The survey questions addressed: 1) details about the producers, 2) producer impressions regarding the modeling and outreach process, 3) the understandability of the model and its results, 4) overall effect of model results on producer decisions, 5) producer impressions on the ability of the model to predict odor exposure, and finally 6) characterization of neighbor and community relations before and after the modeling process.

CAM Evaluation Findings

The following provides a descriptive summary of our CAM user survey. The results presented first cover the degree to which producers felt involved in the process and the timeliness of results. Then we report the degree to which producers understand CAM and its results. Following this is an assessment of how useful CAM has been in siting decisions and its potential impact on neighbor relations. Finally, CAM users provide their impressions of the accuracy of CAM results.

Producer Involvement and Timeliness of Results

Direct participation in the modeling process is often considered key to farmers taking ownership in decision-support tool results, thus improving the likelihood of results being used to guide behavior (Goodhue, 1995). Therefore, one of the key factors in the effectiveness of any decision-support process is the degree to which users are able to participate in the process.

• 98% of CAM users felt they were given significant opportunity to provide input into the modeling process.


• 82% felt as though the time between their request for modeling and the delivery of the report was "acceptable."


• 42% felt that there were data inputs they could have completed prior to initiating the CAM process including: mapping of planned site(s) using downloadable aerial photos; platt maps and/or manure management plan documents; estimating separation distance between site and homes, business, and wells; logging critical numbers (e.g., production capacity, building orientation, building/ventilation type, etc.) perhaps via a standardized on-line template.


• 77% stated that Web-based examples of model results would be useful.

Additionally, because producers are making capital-intensive decisions that may be time-sensitive (e.g., obtaining financing, making land bids, and pursuing appropriate permits), evaluating the time effectiveness of the process is critical. With regard to these issues, 74% of the respondents felt that the current process for executing the model from initial contact to receiving final results served their needs "well" to "very well"; 15% "moderately well."

Understanding CAM Results

Users must understand the output of any decision-support tool in order to act on the information appropriately. Because of this, a key component of our outreach process is a face-to-face review of model results allowing time and discussion so producers can feel comfortable with CAM. Therefore, we asked a series of questions to assess producer comprehension of bottom-line model output and understandability of key parameters of the model's conclusions. Moreover, to consider an application of user understanding, we explored the degree to which users can explain results to neighbors.

• 92% of CAM users feel as though the model results were explained to them by Extension personnel "well" to "very well."


• 95% of users understood the results of the model "well" to "very well."


• 88% understood "well" the critical odor risk parameter of hours of exposure a particular neighbor would be expected to experience.


• 56% understood "well" to "very well" odor levels (based on dilution factors); 32% stated they understand this factor "moderately well."


• 60% claim they could explain model results to concerned neighbors "well/very well"; 36% "moderately well."

Usefulness of CAM in Decision Support

CAM is ultimately a decision-support tool, so it was important to gauge the impact of the model on producer decisions. We asked producers to state the degree to which CAM influenced decisions to build or not build facilities.

• 68% of the respondents opted to build the sites that were modeled with CAM:
  
  
  • 48% stated the model results directly affected their decision on whether to build or not.
  
  
  • 26% said they built largely because it was located close enough to their cropping operations that they could utilize the manure (or that they had manure buyers in mind). CAM confirmed their original opinions that the location was a good one.


• 32% opted not to build facilities at the location modeled:
  
  
  • 50% of these producers who opted not to build indicated that the model contributed directly to this decision.
  
  
  • 15% opted out strictly for financial reasons (due mostly to a mix of low hog prices and high feed costs).
  
  
  • 10% did not build largely due to pre-existing social pressures regarding odor. CAM results confirmed their original concerns.

Potential Impacts to Neighbor Relations

As an indirect measure of CAM's potential broader impact on helping to mediate social conflict caused by odor, we asked several questions designed to gauge how CAM acted as a community-oriented tool.

• 81% of CAM users stated that overall concerns regarding the potential impacts to neighbors factored heavily into the decision to build or not.


• 58% of CAM users subsequently communicated the modeling process and results of the model to neighbors:


• 33%, neighbors responded to the modeling results "positively to very positively" (25% and 8% respectively).


• 31% of the neighbors reacted in a neutral way, and


• 30% reacted "poorly to very poorly" (22% and 8%, respectively).

The producers were also asked to rate their relationship with proximal neighbors at the time of modeling and if construction took place at that location afterwards. Prior to modeling, half of the CAM users were on at least "good" terms with neighbors. For those who built facilities, 20% experienced an improvement in neighbor relations, 8% experienced a worsening of relations, and 72% stayed the same. For those producers whose relations improved, the leading factor was that "things (e.g., odor, noise, flies) didn't turn out to be as bad as they (the neighbors) originally assumed they would."

Producer Opinion Regarding Accuracy of CAM

While CAM has been (and continues to be) validated empirically, user feedback is also used as a measure of model success and for diagnostic purpose. To this end, we asked producers who opted to build facilities to what degree they believe CAM accurately predicted odor dynamics.

• 67% stated the model predicted what actually happened in terms of odor exposure "well to very well"; 14% stated the model predicted "moderately well"; 14% stated that it is still too early to tell.


• Only 5% thought that the model's predictions underestimated the impact the facilities would have on neighbors.

Conclusions

This article presents to the agricultural Extension community CAM, a new DSS that has been used in Iowa to help swine producers minimize their odor related impacts on surrounding communities. We also present our simple, yet effective process for gathering key DSS user data and the rationale for doing so. With this user-generated data we will seek to improve the delivery of our expanding CAM outreach program.

Based on the findings of our producer survey, current programming appears to be effective in terms of process and understandability. Most important, the CAM model was found to be very useful in farmers' planning processes. Sixty-seven percent of the CAM users believed that all evidence (from their point of view) suggests that CAM "got it right," and 14% believed that it was moderately close. The CAM process was communicated to neighbors roughly 60% of the time (by the producer). Neighbor reactions to CAM were mixed, but CAM served as a critical platform for communication—a backbone of civic cooperation. For many producers who went on to build at sites that were CAM modeled, there appeared to be an improvement in neighbor relations. CAM appeared to have been an important contributor. From the point of-view of CAM users, in the final analysis, 75% of the producers rate the modeling process and results as "useful" to "very useful" (23% to 52% respectively) with regard to making (or justifying) risk-management decisions; 15% viewed the CAM process as "moderately useful."

Users identified ways to improve programming. A Web-based information hub with data-entry capacity (specifically for the producer to provide model parameters) will be explored by Extension personnel as a way to improve the front-end modeling initiation process. Extension specialists are working on expanding the materials available to help users explain the model and some of its key parameters to neighbors, specifically targeting improvements in data presentation and explanation (using more user-friendly graphics and lay-audience language).

All in all, the results of our survey suggests that CAM has passed the primary test of applied science as measured by usefulness to producers in making better decisions that have far reaching, private and public implications. Nevertheless, it is critical that CAM continues to undergo user evaluation as described here.

Future development of CAM will be strongly guided by the results this survey as well as our evaluatory approach. In addition to this user evaluation of CAM, an examination of the social effectiveness of CAM is currently underway. We consider it highly instructive to assess the opinions of potential receptors regarding use of modeling to aid producers in assessing odor risk. Therefore, in order to better socially gauge the physical parameters of CAM, a public survey has been conducted and is currently being analyzed to determine how willing the public is to accept modeling and modeling results in locating hog production sites. As they become available, the results of current and future evaluation surveys (user and general public) will be fully incorporated into CAM promotional materials and Extension activities.

References

Anderson, J. R., & Feder. G. (2004). Agricultural Extension: Good intentions and hard realities. World Bank Research Observer, (19)1, 41-60.

Bazen, E. F., & Fleming R.A. (2004). An economic evaluation of livestock odor regulation distances. Journal of Environmental Quality, 33, 1997-2006.

Carberry, P. S., Hochman Z., McCown R. L., Dalgliesh N. P., Foale M. A., Poulton P. L., et al. (2002). The FARMSCAPE approach to decision support: Farmers', advisers', researchers' monitoring, simulation, communication and performance evaluation. Agricultural Systems 74, 141-177.

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2008). Internet, mail and mixed mode surveys: The tailored design method. 3rd Edition. Hoboken, N.J.: Wiley. 512p.

Donham, K. J., Wing S., Osterberg, D., Flora, J. L., Hodne, C., Thu, K. M., et al. (2007). Community health and socioeconomic issues surrounding concentrated animal feeding operations. Environmental Health Perspectives, 115, 317-320.

Douglah, M. (1998). Developing a concept of Extension program evaluation. (Bulletin I-4-98-.5M-200 and No.G3658-7). Madison, WI: University of Wisconsin Cooperative Extension, Program Development and Evaluation.

Goodhue, D. L. (1995). Understanding user evaluations of information systems. Management Science, 14(21), 1827-1844.

Guo, H, Jacobson, L. D., Schmidt, D. R., & Nicolai, R. E. (2003). Evaluation of the influence of atmospheric conditions on odor dispersion from animal production sites. Transactions of the ASAE, 46(2), 461-466.

Guo, H., Jacobson, L. D., Schmidt, D. R., Nicolai, R. E., & Janni, K. A. (2004). Comparison of five models for setback distance determination from livestock sites. Canadian Biosystems Engineering 46, 6.17-6.25.

Harrison, J. D., Kanade, S. S., & Toney, A. H. (2004) Agricultural Environmental Management Information System: An online decision support tool. Journal of Extension [On-line], 42(1) Article 1TOT4. Available at: http://www.joe.org/joe/2004february/tt4.php

Harrison, J. D., & Toney, A. H. (2004) A "tracking system" to assure quality and user satisfaction. Journal of Extension [On-line], 42(2) Article 2T0T7. Available at: http://www.joe.org/joe/2004april/tt7.php

Heber, A. J., & Bogan, B. W. (2006). The study and regulation of agricultural air quality in the US. In V. P Aneja, et al. (Eds.), Proceedings: Workshop on agricultural air quality: State of the science (pp. 32-35). Potomac, MD: Published by the Ecological Society of America.

Hoff, S .J., Bundy, D. S., & Harmon, J. D. (2008a). Modeling receptor odor exposure from swine production sources using CAM. Applied Engineering in Agriculture, 24(6), 821-837.

Hoff, S. J., Bundy, D. S., Harmon, J.D., Jacobson, L., Schulte, D., Schmidt, D., & Henry C (2006). Validation of odor dispersion measurements and modeling. Proceedings: Workshop on agricultural air quality. Bolger Conference Center, Potomac, MD June 5-8, 2006. pp. 270-282.

Hoff, S. J., Bundy, D. S., Harmon, J.D., & Johnson, C. D. (2008). A receptor-based siting strategy for swine production systems. Proceedings: Mitigating Air Emissions from Animal Feeding Operations. May 19-21 Des Moines, IA. Retrieved from http://www.ag.iastate.edu/wastemgmt/Mitigation_Conference_proceedings/CD_proceedings/Siting_and_Environmental_Barrriers/Hoff-Receptor-Based%20Siting%20Strategy.pdf

Huang, H., & Miller, G. Y. (2006). Citizen complaints, regulatory violations, and their implications for swine operations. Illinois Review of Agricultural Economics 28(1), 89-110.

Iowa Pork Producer Association (IPPA). (2008, July 28). Iowa Pork Producer Association legislative summary 5(6-7), July 2008, pp 10-11. Retrieved from http://www.iowapork.org/Portals/iowapork/July%20Mag%20Web.pdf

Johnson, C. (2006, Summer). Assessing where to build. Iowa Manure Matters: Odor and Nutrient Management (EDC-129-32), 9(2), 5-6. Ames, IA: Iowa State University Extension.

Koelsh, R. L., Howard, L., Pritchard, S., & Hay, P. (2000). Implementation of a livestock systems environmental assessment tool. Journal of Extension [On-line], 38(1) Article 1FEA3. Available at: http://www.joe.org/joe/2000february/a3.php

Newman, S., Lynch, T. & Plummer A. A. (2000). Success and failure of decision-support systems: Learning as we go. Journal of Animal Science, 77, 1-12.

Petheram, R. J. (1998). Review of evaluation in agricultural Extension. In Human capital, communications and information systems research and development. (RIRDC Publication No. 98/136).

Radhakrishna, R. B., & Relando, R. Z. (2009). A framework to link evaluation questions to program outcomes. Journal of Extension [On-line], 47(3) Article 3TOT2. Available at: http://www.joe.org/joe/2009june/tt2.php

Walker, D. H. (2002). Decision support, learning and rural resource management. Agricultural Systems, 73(1), 113-127.