Participatory Evaluation and Learning: A Case Example Involving Ripple Effects Mapping of a Tourism Assessment Program

Introduction

Extension professionals involved in community tourism development, as with many development efforts, are challenged to demonstrate positive value and sustainable impacts for a community. As community development educators and the evaluation team for the project that is the topic of this article, we believe that the core principles of good practice put forth by the Community Development Society (Community Development Society, n.d.) are a great resource for all types of Extension programming. Although the initial motivation triggering a community development effort may be economic, research has suggested that a holistic approach to development that considers sociocultural, environmental, and economic impacts is needed for communities to experience the greatest benefits (Bathelt & Glückler, 2011; Coccossis, 1996; Papatheodorou, 2004; Ritchie & Crouch, 2005; Sanso, 2009). As with other Extension areas of interest, a body of expert knowledge underlies the field of community tourism development. Relevant scholarly literature addresses a variety of approaches to local tourism development and analysis strategies that draw from disciplines such as business, marketing, customer service management, and community development (Alberty & Mikalik, 1989; Haywood, 1986; Martilla & James, 1977; Matzler, Bailom, Hinterhuber, Renzl, & Pichler, 2004; Song, Dwyer, Li, & Cao, 2012). These approaches and strategies highlight tourism development tactics, including planning, identifying community tourism assets, understanding existing and potential markets, and intentionally designing tourism efforts to match supply with demand (Butler, 1980; Messer, 2010; Ritchie & Crouch, 2005).

Yet for a truly engaged approach to tourism development, or any other type of Extension programming, those involved must recognize that a two-way exchange of local, community-based expertise and academic expertise is required for effective delivery of that programming. Hassel (2005) explained this concept from an Extension standpoint:

Two-way exchange implies building communicative space perceived by all who participate as "a level playing field" for authentic exchange of perspectives. . . . Given our history of a more one-sided educator-delivery model, creating and maintaining what everyone perceives as a level playing field can be a challenge, but it is a prerequisite for further progress. ("CCE as a Two-Way Process")

The Community Development Society convenes professionals in the community development field and includes among its core principles of good practice the directives that professionals "engage community members in learning and understanding community issues," "incorporate the diverse interests and cultures of the community in the community development process," and "work actively to enhance the leadership capacity of community members, leaders, and groups within the community" (Community Development Society, n.d., "Principles of Good Practice").

In this article, we discuss an emerging program evaluation and learning approach and its implications for educators, community leaders, and program evaluators. After framing the contextual approach we use to engage communities, we explain the University of Minnesota Extension Tourism Assessment Program strategy and then describe our 2013–2014 evaluation of the program. We also highlight implications for community tourism development practitioners and others providing highly engaged programs in communities. These implications include the importance of applying an engaged approach to community tourism development programming and evaluation.

Framing the Context and Strategy for Community Tourism Development

Community tourism development engages all aspects of a community and requires awareness of community context and a clear program strategy to be successful. Our review of tourism development and general development literature pointed to key concepts related to the contexts in which community tourism development takes place and identified program strategies that work best for engaging communities.

Community Context: The Community Capitals Framework

The role of community involvement in sustainable tourism development has been widely discussed in tourism development literature. Murphy's (1985) seminal work on community tourism places community squarely at the heart of tourism development. He suggested a method "by which the industry's contribution can be directed toward community goals, and thereby warrant public support" (p. 37). This idea was reinforced in later sustainable tourism development literature (Hunter, 1997; Messer, 2010; Ritchie & Crouch, 2005; Trudeau-Poskas & Messer, 2015). The success of any tourism development strategy, however, is dependent on community context. Richards and Hall (2000) explained the role of community in sustainable tourism from both theoretical and applied perspectives, noting that "place-based communities have become central to a holistic concept of sustainability, which embraces and integrates environmental, economic, political, cultural and social considerations" (p. 5).

Emery, Fey, and Flora (2006) advanced the idea that supporting community development through a community capitals logic model helps development practitioners not only identify community assets but also map relationships among asset categories. The asset categories are described in Table 1. These asset categories, or community capitals, can be monitored for change or development opportunities. Several tourism researchers have considered this logic framework when looking at tourism impacts on a destination. According to McGehee, Lee, O'Bannon, and Perdue (2010) and Moscardo (2009), identifying the different forms of capital available to communities and how they change over time is a better way to understand tourism impacts. Using the community capitals framework in tourism development enables a community to view each of its asset categories as another bank in the community.

Table 1.
Community Capitals

Community capitals category	Description
Natural capital	Assets that abide in a location, including resources, amenities, and natural beauty. Natural capital might include parks, farmland, and features of the landscape or of nature.
Cultural capital	The way people "know the world" and how to act within it. Cultural capital might include ethnic festivals, multilingual populations, or a strong work ethic.
Human capital	The skills and abilities of people and the ability to access outside resources and bodies of knowledge in order to increase understanding and identification of promising practices. Human capital might include a local leadership development program, a high level of educational achievement within a population, or a cluster of skilled craftspeople.
Social capital	The connections between people and organizations or the social glue that makes things happen. "Glue" refers to the strength or levels of trust and connections among people, groups, and organizations within a community.
Political capital	Ability of organizations or communities to engage the public or influence local policy decisions made by elected officials and leaders of area businesses.
Financial capital	The fiscal resources available to invest in community capacity building, underwrite business development, support civic and social entrepreneurship, and accumulate wealth for future community development. Many times, financial capital becomes the focus of community efforts.
Built capital	The infrastructure that supports a community, such as telecommunications, industrial parks, main streets, water and sewer systems, roads, etc. Built capital is often a focus of community development efforts.
Note. Adapted from "Using Community Capitals to Develop Assets for Positive Community Change—CDPractice: Promoting Principles of Good Practice," by M. Emery, S. Fey, and C. Flora, 2006, Community Development Society, 13, and "The Role of Social Capital in Strengthening Community Capitals," by J. Horntvedt, May 20–23, 2012, presentation at the Annual NACDEP Conference (West), Park City, Utah.

This holistic framework for understanding community development is of significance to University of Minnesota Extension's Tourism Assessment Program because of its focus on increasing the capacity (e.g., knowledge and skills) of a community to undertake sustainable tourism development. By adding a human capital category, communities (at large) can value and see the impact of their time and effort while engaged in development.

Program Strategy: Participatory Evaluation and Learning

Participatory evaluation and learning were key strategic components in the design of the Tourism Assessment Program. We drew on experience with Horizons, a Northwest Area Foundation–funded community leadership program that aims to reduce poverty in small rural communities. In reviewing that program's successes, Herrera and Hoelting (2010) noted that relinquishing some ownership of a program provides educators and researchers with a learning experience as well: "Stepping back to let peer facilitators lead the process and implement change was a challenge. Moving to a position of coaching and support required Extension to check some of their positional authority at the door" (p. 46).

Participatory evaluation strategies (Cousins & Chouinard, 2012; Cousins & Whitmore, 1998), another strategic component of our work, move evaluators beyond thinking of program stakeholders as recipients of evaluative information to integrating them in the evaluation process. This participatory approach dramatically increases the usefulness of evaluation results for program stakeholders, such as community-based organizations (Patton, 2012). In an analysis of what constitutes active participation in evaluation, Cousins and Whitmore (1998) explained that practical participatory evaluation supports program refinement and problem solving with decision making balanced between the researcher and community members. Yet Lawton and Weaver (2015) drew on Arnstein's (1969) ladder of participation to explain that although ideal engagement with residents in tourism development is on a voluntary or citizen-driven basis, use of a consultative approach is more common. To help explain the role of practical participation in evaluation at land-grant universities in general, and in university extension programs in particular, Torock (2009) made the following point:

Quality educational programs require participants to recall prior knowledge, introduce new knowledge, and help participants make connections between prior and new information for individual internalization. Therefore it is the responsibility of Extension educators to ensure opportunities for reflection—not just program evaluation—are a part of the learning process. ("Experiential Learning and Cooperative Extension: Making the Connection")

Teaching Communities Through Participatory Evaluation

Implementers of University of Minnesota Extension's Tourism Assessment Program use tools from the Community Tourism Development manual developed by the University of Minnesota Tourism Center. In applying the tools and strategies outlined in the manual, Extension educators and tourism specialists work with communities to analyze their tourism potential and determine next steps in tourism development. The purpose of the program, created by University of Minnesota Extension in 2007, is to engage community members in the tourism development process. The program includes the following components:

• an identification of tourism assets by community teams;
• a community visit and assessment by a team of tourism specialists;
• a community meeting to identify residents' perceptions of the strengths and weaknesses of, opportunities for, and threats to tourism in the community; and
• a written report of findings and recommendations for local action.

An early application of the program occurred in 2007–2008 in several communities that had participated in the Horizons community leadership program in 2007, when community leaders identified tourism as a development opportunity. Then, during 2013–2014, we reviewed results of an assessment of the 2007–2008 endeavor, conducted an evaluation of program outcomes in three of the communities, and engaged in follow-up evaluation reporting sessions (report-back sessions) with those three communities. The three communities were selected for the following reasons:

• They were early participants in the program, and several years had passed since their involvement, allowing for implementation of relevant projects.
• The members of the program team were aware of community action following the tourism assessment program visits.
• The community leaders involved in the program were still engaged in their communities.

Our research team, which comprised two Extension educators, one Extension tourism specialist, and one Extension program evaluation specialist, was assisted by a graduate student during the data collection and coding portions of our project. Prior to our evaluation effort, the Tourism Assessment Program team had conducted no formal program evaluation of multiple communities to understand and compare the tourism development effects resulting from communities' engagement with the Tourism Assessment Program.

Findings from Implementation of the Program in 2007–2008

We reviewed the results of the original Tourism Assessment Program report given to each community between 2007 and 2008. Communities A and C possessed strong cultural resources, whereas Community B had strong nature-based amenities and hosted its county's fair. The reports indicated that all three communities had very limited lodging opportunities and that two of the three competed with neighboring towns for dining, overnight, and retail customers. Table 2 provides a summary of the assets and challenges identified in the reports.

Table 2.
Results of Initial Tourism Assessment Program Efforts in Three Target Communities

Information category	Community A	Community B	Community C
Assets	Regional community cultural center	Many nearby nature-based recreation amenities	Significant cultural heritage amenities
	Culturally specific museum in community	Regionally popular county fair	Popular area for lake-based recreational opportunities
	Strong regional connections
Challenges	Limited lodging	Limited lodging	Limited lodging
	Weak community theme	Perceived competition for customers with neighboring communities	Lack of regional and community coordination of events
Sociodemographic information	Population of 3,325 (2012), located in west central Minnesota	Population of 835 (2012), located in northwest Minnesota	Population 4,408 (2012), located in east metro Minnesota
Initial program participant engagement details	20 participants in 2007 Tourism Assessment Program	18 participants in 2008 Tourism Assessment Program	55 participants in 2008 Tourism Assessment Program

Ripple Effects Mapping

By using ripple effects mapping (REM) as our data collection method, we were able to document significant outcomes of participation in the Tourism Assessment Program. REM is a group participatory evaluation strategy that allows program participants and stakeholders to retrospectively and visually "map" the complex chain of effects resulting from a program (Chazdon & Paine, 2014; Hansen-Kollock, Flage, Chazdon, Paine, & Higgins, 2012; Jordan, Chazdon, & Alviz, 2016).

We conducted REM sessions with the three communities in May 2013. Participants in each session included Tourism Assessment Program participants and nonparticipants connected to tourism. The numbers of participants in the REM sessions were as follows: 14 in Community A, 22 in Community B, and 18 in Community C.

REM sessions used for program evaluation include 12 to 20 participants who are paired but organized approximately equally between "participant" and "other stakeholder" categories. In an initial appreciative inquiry interview process, paired participants interviewed each other (Cooperrider & Whitney, 2007) and then all participants reported on program outcomes (i.e., effects) they discussed during the interviews. Each pair shared its interview answers with the group while the evaluation team recorded responses on a mind map visible to the entire group. In reviewing the mind map, participants then informed evaluators of supplemental postprogram actions taken by community members. The facilitator (who was a member of our team) then worked with the group to organize the disparate effects into common themes. The facilitator then explored the reported effects (via follow-up calls with REM participants) to create visual causal chains displaying how the effects came about and additional effects that may have arisen.

In reviewing the results from the REM sessions, we identified three common outcome themes. These themes aligned with the learning objectives of the Tourism Assessment Program, which were

• to realize increased relationships developed either within the community or with new partners from outside or neighboring communities,
• to increase awareness of the community's visitor market demographics, and
• to expand awareness and development of tourism infrastructure within the community.

After the themes were identified, teams of two evaluators coded each effect recorded on a community's map relative to the community capitals framework to determine whether any patterns existed in how the communities had allocated their resources to achieve the outcomes. For example, in the case of one community, the recorded effect "trail development" was coded as corresponding with mobilization of both political assets and built assets in the community. To relate this coding back to the learning objectives of the program, subsets of community capitals were grouped according to each program theme as follows:

• realized relationship building—social capital and political capital;
• recognized awareness of visitor markets—financial capital and human capital; and
• achieved awareness and development of infrastructure—cultural capital, built capital, natural capital, and health capital (see below for information about this capital type).

Each coded ripple effects map aligned with the unique interests of the applicable community (Bhattacharyya et al., 2013). During the coding process, an eighth community capital, heath capital, was isolated to align with the coding system already developed by University of Minnesota Extension (Chazdon et al., 2007). The REM session for Community C yielded 90 effects, the largest number of distinct effects; Community A's session yielded 71 effects; and Community B's session yielded 80 effects. Although they were recorded as individual effects in each community, some ripple effects mobilized more than one type of community capital. In these circumstances, the effects were coded as corresponding with more than one category in the community capitals framework. Through recording of these postprogram effects within a community, the larger impact of the initial event could be captured and measured. As seen in Table 3, Community A focused on mobilizing its cultural assets, Community B focused on mobilizing its financial assets, and Community C focused on mobilizing its social assets.

Table 3.
Outcomes by Community Capitals Category

Community capitals category	Community A		Community B		Community C
	# of outcomes	% of reported outcomes	# of outcomes	% of reported outcomes	# of outcomes	% of reported outcomes
Natural	0	0.0%	4	5.0%	7	7.8%
Cultural	22	31.0%	9	11.3%	11	12.2%
Human	13	18.3%	18	22.5%	8	8.9%
Social	18	25.4%	22	27.5%	27	30.0%
Political	16	22.5%	9	11.3%	11	12.2%
Financial	21	29.6%	27	33.8%	26	28.9%
Built	15	21.1%	14	17.5%	17	18.9%
Health	6	8.5%	6	7.5%	7	7.8%
Note. Some effects mobilized more than one type of community capital; in these circumstances, an effect was coded as corresponding with more than one community capitals category. Boldface denotes the most significant community capitals category mobilized in a community.

Report-Back Sessions

The last component of the 2013–2014 evaluation process was facilitation of a final report-back session for each of the participant communities. To provide participants an opportunity to reflect on the program evaluation results presented to them, we chose to apply a focused discussion strategy known as the objective, reflective, interpretive, and decisional (ORID) method to the evaluation reporting sessions (Stanfield, 2000). This application of the ORID method facilitated the communities' reengagement with the evaluation data and contextualized it within a community capitals framework perspective on their tourism development efforts. Through the use of the ORID method, participants involved in the report-back sessions moved from rote listening and response to a dialogue of discovery (Stanfield, 2000). The ORID method focuses a participant's summative thinking process through four lenses of perception when considering how to apply new knowledge, moving the participant from passive acceptance of new information to a cognitive state that converts knowledge into action (Lapp, 2010).

Two of the three communities participated in in-depth report-back sessions on the mapped results (one had limited time for in-depth discussion due to other agenda items). In Communities B and C, the ORID method was used to give participants a way to consider the report's content in relation to their day-to-day activities and ongoing tourism projects. Participant feedback on the report-back sessions showed that this activity helped the community members contextualize their recent tourism development efforts within the community capitals framework. The significance of these results, however, is likely to have been skewed by the low numbers of attendees at the report-back events (five in both Community B and Community C). Members of Communities A and B were familiar with the community capitals framework from previous participation in the Horizons program, and, consequently, they were more receptive to the concept of the results of the maps being coded using the abbreviated community capitals definitions. Members of Community C, however, first heard of the community capitals framework during the report-back session, so for them, the connection was not as clear regarding how the community capitals could be immediately applicable to current concerns. They did, however, value the ripple effects map.

Testing the use of REM, the ORID method, and the community capitals framework in this program evaluation provided significant insights into both the learning capacities and the challenges associated with using participatory evaluation and other educational tools when working with communities. REM proved to be an effective participatory evaluation tool valued by all three communities. The ORID method was shown to be a tool that can support reflective connection making between new and old information for adult audiences. Additional work is needed, however, to determine whether the use of development models such as the community capitals framework is effective in helping communities (a) identify previously overlooked assets and (b) build capacity to undertake tourism development.

Implications

Tourism development in communities offers a range of engagement opportunities for Extension professionals and others. With regard to the methods and tools described in this article, potential implications specific to audiences engaged in community-based tourism development initiatives are varied. Some of these implications are as follows:

• When considering implementing tourism projects as a development strategy for communities, destination managers and leaders should consider projects that engage stakeholders across a community. Use of this approach ensures that tourism opportunities that align with a community's vision are facilitated.
• Within communities actively developing community tourism products and services via Extension programming, local action groups can use REM to gain a better understanding of tourism development and its impact.
• REM is most effective within 1 to 2 years of completing a program. Community interest in the results of REM may wane if too much time elapses between the community's REM session and the report-back session when the finalized map is shared with the community. It is more effective to complete the mapping session and the report-back session within 3 months of each other.
• Finishing the evaluation does not mean the evaluative work is complete. Continued use of engagement methods, such as the ORID method, is crucial for ensuring that evaluation findings are used by program participants and stakeholders.

Conclusions

Engaging communities in research-based participatory evaluation and learning methods takes time but can be rewarding for both community development professionals and Extension professionals. For such engaging programs to have long-standing meaning within communities, creating participatory space in the program evaluation process is essential. Creation of such space results from formally established partnerships among community leaders, development professionals, and educators prior to project implementation. It is also important to remember that to sustain these spaces, Extension educators must make a concerted effort to include communities in participatory decision making about evaluation and the potential use of both initial program findings and program evaluation results in future development. The evaluation project described here provides just one example of participatory evaluation and learning in community development. Additional application of these processes is recommended to gain deeper insight into how communities learn and apply participatory evaluation to enact sustainable change.

References

Alberty, S., & Mihalik, B. (1989). The use of importance-performance analysis as an evaluative technique in adult education. Evaluation Review, 13(1), 33–44.

Arnstein, S. (1969). A ladder of citizen participation. Journal of American Institute of Planners, 35(4), 216–224.

Bathelt, H., & Glückler, J. (2011). The relational economy: Geographies of knowing and learning. Oxford, UK: Oxford University Press.

Bhattacharyya, R., Templin, E., Messer, C., Chazdon, S., & Hermanson, L. (2013, October). Ripple effect mapping of the Tourism Assessment Program. Poster session presented at the meeting of the Annual Fall Program Conference, Duluth, MN.

Butler, R. W. (1980). The concept of a tourist area cycle of evolution: Implications for management of resources. Canadian Geographer, 24(1), 5–12.

Chazdon, S., Bartholomay, T., Marczak, M., & Lochner, A. (2007, November). Using the community capitals framework for Extension impact evaluation. Paper presented at the Fourth Annual Community Capitals Framework Institute, Ames, IA.

Chazdon, S., & Paine, N. (2014). Evaluating for public value: Clarifying the relationship between public value and program evaluation. Journal of Human Sciences and Extension, 2(2), 100–118.

Coccossis, H. (1996). Tourism and sustainability: Perspectives and implications. In G. K. Priestley, J. A. Edwards, & H. Coccossis (Eds.), Sustainable tourism? European experiences (pp. 1–21). Wallingford, UK: CAB International.

Community Development Society. (n.d.). Principles of good practice. Retrieved from http://www.comm-dev.org/about-us/item/86-principles-of-good-practice

Cooperrider, D. L., & Whitney, D. (2007). Appreciative inquiry: A positive revolution in change. In P. Holman & T. Devane (Eds.), The change handbook (2nd ed.) (pp. 73–88). San Francisco, CA: Berrett Koehler Publishers, Inc.

Cousins, J. B., & Chouinard, J. A. (2012). Participatory evaluation up close: An integration of research-based knowledge (Evaluation and Society). Charlotte, NC: Information Age Publishing.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23.

Emery, M., Fey, S., & Flora, C. (2006). Using community capitals to develop assets for positive community change. CDPractice: Promoting principles of good practice. Community Development Society, 13. Retrieved from http://www.comm-dev.org/publications/cd-practice

Hansen-Kollock, D., Flage, L., Chazdon, S., Paine, N., & Higgins, L. (2012). Ripple effect mapping: A "radiant" way to capture program impacts. Journal of Extension, 50(5), Article 5TOT6. Available at: https://www.joe.org/joe/2012october/tt6.php

Hassel, C. A. (2005). The craft of cross-cultural engagement. Journal of Extension, 43(6), Article 6FEA1. Available at: https://www.joe.org/joe/2005december/a1.php

Haywood, K. M. (1986). Can the tourist-area life cycle be made operational? Tourism Management, 7(3), 154–167.

Herrera, M., & Hoelting, J. (2010). To establish an effective community spirit: A land grant Extension and deliberative dialogue. New Directions for Higher Education, 2010, 43–50.

Hunter, C. (1997). Sustainable tourism as an adaptive paradigm. Annals of Tourism Research, 24(4), 850–867.

Jordan, C., Chazdon S., & Alviz, K. (2016). Evaluating beyond the Ivory Tower: Lessons learned from the scholars program. Journal of Community Engagement in Higher Education, 8(3).

Lapp, C. (2010, December). Focused Conversation ORID Method. Technology of Participation®. Presentation at University of Minnesota, Minneapolis, MN.

Lawton, L. J., & Weaver, D. B. (2015). Using residents' perceptions research to inform planning and management for sustainable tourism: A study of the Gold Coast Schoolies Week, a contentious tourism event. Journal of Sustainable Tourism, 23(5), 660–682.

Martilla, J. A., & James, J. C. (1977). Importance-performance analysis. Journal of Marketing, 41(1), 77–79.

Matzler, K., Bailom, F., Hinterhuber, H. H., Renzl, B., & Pichler, J. (2004). The asymmetric relationship between attribute-level performance and overall customer satisfaction: A reconsideration of the importance-performance analysis. Industrial Marketing Management, 33(4), 271–277.

McGehee, N. G., Lee, S., O'Bannon, T. L., & Perdue, R. R. (2010). Tourism-related social capital and its relationship with other forms of capital: An exploratory study. Journal of Travel Research, 49(4), 486–500.

Messer, C. C. (2010). Community tourism development (3rd ed.). Minneapolis, MN: University of Minnesota Extension.

Moscardo, G. (2009). Tourism and quality of life: Towards a more critical approach. Tourism and Hospitality Research, 9(2), 159–170.

Murphy, P. E. (1985). Tourism: A community approach. New York, NY: Methuen.

Papatheodorou, A. (2004). Exploring the evolution of tourism resorts. Annals of Tourism Research, 31(1), 219–237.

Patton, M. Q. (2012). Essentials of utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

Richards, G., & Hall, D. (Eds). (2000). Tourism and sustainable community development. London, UK, and New York, NY: Routledge.

Ritchie, J. R. B., & Crouch, G. I. (2005). The competitive destination: A sustainable tourism perspective. Wallingford, UK: CABI Publishing.

Sanso, A. (2009).The tourist area lifecycle and the unit roots test. A new economic perspective for a classic paradigm in tourism. DEA Working Papers. Universitat de les Illes Balears, Departament d'Economía Aplicada.

Song, H., Dwyer, L., Li, G., & Cao, Z. (2012). Tourism economics research : A review and assessment. Annals of Tourism Research, 39(3), 1653–1682.

Stanfield, B. (2000). The art of focused conversation. Gabriola Island, Canada: New Society Publishers.

Torock, J. L. (2009). Experiential learning and Cooperative Extension: Partners in non-formal education for a century and beyond. Journal of Extension, 47(6), Article 6TOT2. Available at: https://www.joe.org/joe/2009december/tt2.php

Trudeau-Poskas, D., & Messer, C. C. (2015). Investigating leadership applications in tourism: A case study of leadership in community tourism. Journal of Teaching on Travel and Tourism, 15(2), 186–198.