A Framework for the Evaluation of Large-Scale Regional Conservation and Management Strategies

Regional approaches to conservation and natural resources management are becoming commonplace throughout the United States (McKinney, Scarlett, & Kemis, 2010). Strategies are being developed and implemented across large landscapes, requiring action from multiple stakeholder groups. For example, widespread efforts are being made to increase prescribed fire use in the Southeast to support restoration and enhancement of fire-dependent ecosystems, reduce wildfire risk, and promote active land management (Burke, Steelman, & Gharis, 2012).

In this article, we describe a framework for evaluating such large-scale strategies. This framework was used to evaluate prescribed fire efforts across the Southeast in relation to the goals and priorities outlined in two strategic plans: the Southeast Cohesive Wildland Fire Management Strategy (Cohesive Strategy) (Southeast Regional Cohesive Strategy Committee, 2012) and the Comprehensive Strategy for Prescribed Fire to Restore Longleaf Pine in the Southeast United States: A Vision for 2025 (Comprehensive Prescribed Fire Strategy) (Burke et al., 2012). This approach is informing both policy and practice, ensuring that the goals and priorities of each strategy are achieved.

Step-by-Step Process for Evaluating Regional Strategies and Identifying Performance Gaps

North Carolina State University (NCSU) Extension Forestry developed and used the eight-step framework presented here for evaluating both aforementioned fire strategies. The framework can be applied during any phase of regional strategy implementation and may be adapted as needed.

1. Form an Advisory Committee and Establish a Collaborative Network

Select key stakeholders to participate on an advisory committee. This committee's role is to regulate bias, assist with data collection efforts, and provide guidance based on expertise (Berg, 2004). Include diverse interests and stakeholder groups to promote robust evaluation recommendations (Cole, 1980). Also, develop a network of collaborators to provide support in reviewing data quality and ensuring that recommendations are substantiated. Figure 1 lists the agencies and organizations included in the evaluation of the prescribed fire strategies.

Figure 1.
Prescribed Fire Evaluation Advisory Committee and Collaborators

Advisory Committee Members	U.S. Department of Defense, National Park Service, North Carolina State University, Southern Group of State Foresters, Southeast Regional Extension Forestry, U.S. Forest Service, Natural Resources Conservation Service, U.S. Environmental Protection Agency, U.S. Fish and Wildlife Service, U.S. Geological Survey
Collaborators	Southeast Regional Cohesive Strategy Committee, Southeast Regional Partnership for Planning and Sustainability, Southern Fire Exchange, Southern State Fire Chiefs

2. Select an Appropriate Frame for the Evaluation Effort

Determining an evaluation frame is paramount when planning assessments of complex regional strategies. In developing an evaluation frame, consider applicable components of the strategy for evaluation as well as complementary strategies that exhibit similar priorities and goals.

3. Develop an Evaluation Worksheet

An evaluation worksheet helps individuals organize an evaluation and connect results to the implementation of recommendations (Chhetri, 2016). The worksheet should be based on the function of the evaluation performed. Figure 2 shows the structure of such an evaluation worksheet.

4. Identify Existing Efforts Through an Environmental Scan

An environmental scan allows for the identification of current efforts across a region. Data collection involves online research and input from the advisory committee and collaborative network to ensure increased data validity (Guion, 2010). To organize data, use a spreadsheet that includes a description of the effort, classification of functionality, designation of a lead agency or organization, identification of collaborators, and relevant contact information.

5. Tie the Environmental Scan to an Associated Strategy

Connect the data collected during the environmental scan to the associated strategy or strategies for evaluation by assigning each effort a code that identifies the corresponding strategy and action item(s) (Chhetri, 2016). This approach allows for quantitative analysis of the current state of efforts in relation to goals, priorities, and actions of the strategy of interest. Figure 3 provides an example of a coded item from the environmental scan for the prescribed fire strategies evaluation.

6. Interview Key Informants to Qualify On-the-Ground Experiences

Follow the environmental scan with key informant interviews to gain insight into on-the-ground experiences from stakeholders in the field. Extension can leverage its connections and those of the advisory committee to create an initial sampling frame of informants working in priority areas. To achieve a rich data set, use snowball sampling (Berg, 2004) for identifying additional informants and, thereby, gaining new insights and perspectives.

7. Analyze Results and Integrate Findings into the Evaluation Worksheet

To analyze the collected key informant interview data, NCSU Extension Forestry used content analysis (Figure 4) identifying major themes from the data that were also triangulated with the results of the environmental scan. Using this approach increases the credibility of assertions and recommendations coming from an evaluation (Berg, 2004).

Adapted from "The Ten Steps of Content Analysis," by University of Surrey, 2016, Module 9: Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm

8. Facilitate Discussion to Develop Consensus on Gaps and Contributing Factors

Leverage the expertise of the advisory committee through facilitated discussion to develop consensus of the evaluation results. This endeavor allows for multiple interpretations of the data and more robust recommendations based on the breadth of experience and expertise leveraged by the consensus-driven process.

Challenges and Lessons Learned

One challenge to this approach is ensuring effective participation among advisory committee members. This issue is common in collaborative program evaluation and requires effort in familiarizing the committee with the process (Cole, 1980; Layman, Doll, & Peter, 2013). Committee members have limited time to devote to complex evaluation efforts, so it is important to quickly develop the knowledge and skills necessary to facilitate effective involvement. Focusing on their specific role(s) may prevent advisory committee members from feeling overwhelmed by the extent of the process (Cole, 1980).

Implications

The ability to achieve large-landscape conservation guided by regional strategies is closely linked to the capability of those in charge of oversight to stay in tune with its implementation and overall impact (Prokopy et al., 2009; Wondolleck & Yaffee, 2000). For promoting a strategy that is in tune with the on-the-ground reality, it is important to develop a mechanism that allows for adaptive management (Layman et al., 2013; Muñoz-Erickson, Aguilar-González, Loeser, & Sisk, 2010). Using a practical approach for evaluating complex strategies is critical in gaining a holistic view of a region's progress toward specific conservation goals.

Inclusive strategies involving multiple key stakeholders allow evaluations to result in robust recommendations that have the potential to create meaningful change (Layman et al., 2013). NCSU Extension Forestry demonstrated this by developing a framework for promoting a collaborative approach by multiple entities in identifying programmatic and resource gaps across a region to inform the potential development of a collaborative prescribed fire initiative. Others may be able to use the framework for similar endeavors.

References

Berg, B. L. (2004). Qualitative research methods (5th ed.). Boston, MA: Pearson Education, Inc.

Burke, C., Steelman, T., & Gharis, L. (2012). Comprehensive strategy for prescribed fire to restore longleaf pine in the southeast United States: A vision for 2025. Retrieved from https://serppas.org/media/1939/comprehensive-strategy-for-prescribed-fire.pdf

Chhetri, S. (2016). How to perform gap analysis and create gap documents. Retrieved from http://professional-businessanalysis.blogspot.com/2012/03/how-to-perform-gap-analysis-and-create.html

Cole, J. (1980). Developing effective advisory councils. Journal of Extension, 18(4). Available at: http://www.joe.org/joe/1980july/80-4-a1.pdf

Guion, L. (2010). A 10-step process for environmental scanning. Journal of Extension, 48(4), Article 4IAW2. Available at: http://www.joe.org/joe/2010august/iw2.php

Layman, C., Doll, J., & Peter, C. (2013). Using stakeholder needs assessments and deliberative dialogue to inform climate change outreach efforts. Journal of Extension, 51(3), Article 3FEA3. Available at: http://www.joe.org/joe/2013june/a3.php

McKinney, M., Scarlett, L., & Kemis, D. (2010). Large landscape conservation: A strategic framework for policy and action. Cambridge, MA: Lincoln Institute of Land Policy.

Muñoz-Erickson, T. A., Aguilar-González, B., Loeser, M. R. R., & Sisk, T. D. (2010). A framework to evaluate ecological and social outcomes of collaborative management: Lessons from implementation with a northern Arizona collaborative group. Environmental Management, 45, 132–144. doi:10.1007/s00267-009-9400-y

Prokopy, L. S., Genskow, K., Asher, J., Baumgart-Getz, A., Bonnell, J. E., Broussard, S., . . . Wood, D. (2009). Designing a regional system of social indicators to evaluate nonpoint source water projects. Journal of Extension, 47(2), Article 2FEA1. Available at: http://www.joe.org/joe/2009april/a1.php

Southeast Regional Cohesive Strategy Committee. (2012). The national cohesive wildland fire management strategy: Phase III. Retrieved from https://www.forestsandrangelands.gov/strategy/documents/rsc/southeast/SERAP_Final20130405.pdf

University of Surrey. (2016). Module 9 : Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm

Wondolleck, J. M., & Yaffee, S. L. (2000). Making collaboration work: Lessons from innovation in natural resource management. Washington, DC: Island Press.