A Methodology for Evaluating Transdisciplinary Collaborations with Diversity in Mind: An Example from the Green Community Development in Indian Country Initiative

Extension professionals are increasingly asked to facilitate collaborations across the spectrum of professional disciplines to address complex community issues (Blewett, Keim, Lesser, & Jones, 2008). When these collaborations iteratively identify issues, develop strategies, then implement and evaluate the effectiveness of these strategies, this approach fits the description of "action-research" (Strand, Marullo, Cutforth, Stoeker, & Donohue, 2003; Stringer, 1999). When these collaborations transcend not only disciplinary boundaries, but civilian and cultural boundaries as well, to include multiple agencies and a diversity of community members, this approach fits the definition of "trans-disciplinary action-research" (Stokols, 2006; CDE, 2009; Brown, 2005).

These transdisciplinary action-research partnerships are often founded in response to conditions that jeopardize public health, safety, and welfare (e.g., lead poisoning, food security, homelessness, joblessness) in communities that are underserved by Extension professionals (O'Connor, 1995; Stringer, 1999). Thus, Extension professionals are required to identify and overcome barriers formed by culture, class, ethnicity, race, and/or language differences as they facilitate transdisciplinary action-research partnerships. The many challenges involved in these complex programs include the challenge of program evaluation. While documenting tangible program outcomes is relatively straightforward (e.g., number of children treated, quantity of fresh vegetables produced, number of affordable housing units constructed, number of jobs created), practitioners of transdisciplinary action-research find it particularly difficult to substantiate their claims of educational, partnership building, and community capacity building outcomes (Goodman et al., 1998).

Documenting outcomes that are by definition "intangible" (e.g., "transformative learning," "effective partnerships," "community capacity") is always a challenge to Extension professionals (O'Connor, 1995; Rockwell, Jah, & Krumbach, 2003). However, quantitative approaches to program evaluation that assume classroom or conference room venues and high levels of literacy (e.g., pre and post program evaluations, community surveys) offer little guidance when the site of the action is a clinic, a community garden, or a construction site. Conversely, qualitative approaches are subject to persistent criticisms about lack of rigor (Braverman & Arnold, 2008). Of particular importance when keeping diversity in mind are the challenges of program development and evaluation when working with communities where residents are wary of outsiders, where literacy rates are low, or where cultural sensitivities are unknown (Thering, 2007).

Thus, if Extension professionals are to engage transdisciplinary action-research approaches in response to increasingly complex community issues, they require a program development and evaluation methodology that:

1. Offers cogent guidance for identifying and overcoming barriers to partnerships with underserved communities;
2. Draws on relevant theory to articulate links among activities and intended educational and community capacity building outcomes; and
3. Is flexible enough to allow iterative exploration with multiple and mixed methods as the transdisciplinary action-research program unfolds, i.e., evaluating with diversity in mind. This article offers a methodological response to these challenges.

Identifying Barriers with Diversity in Mind

A number of recent Journal of Extension authors focus on identifying barriers to partnerships with groups of people who are historically underserved by university Extension programs. Hassel (2004) offers a critique of conventional Extension programming that reveals structural barriers to the inclusion of "local expertise" in program development and evaluation processes. Hassel suggests that "expert" or "scientific" approaches to Extension programming are often perceived as patronizing, while successful approaches are perceived as respectful and inclusive of "other ways of knowing." Klemme, Hausafus, and Shirer (2005) conducted focus groups to investigate institutional barriers facing Extension professionals who program with "at-risk" groups. Among the barriers they list are the time commitments required to develop respectful partnerships and sustain programming with "at-risk" groups and a lack of clarity about the meaning of the term "at-risk." Thering (2007) suggests "Survivor Community" may be a useful term and an effective heuristic for identifying barriers to successful programming with a specific type of at-risk groups: communities that survive for generations in the wake of disaster. Thering suggests these barriers include a general sense of despair, internal factionalism, and wariness of "Outsiders."

Collectively, these authors begin to illuminate a spectrum of barriers that may be unfamiliar to Extension professionals. Further, these authors suggest that program development and evaluation processes that include identifying and overcoming barriers are essential for producing the intended educational and community capacity building outcomes. However, none of these authors offer guidance for operationalizing "community capacity" or "transformational learning" in the interests of overcoming these barriers or for program development and evaluation.

A review of literature from the fields of community health and global education found two sources that offer some guidance. These reports, and their potential contribution to program development and evaluation with diversity in mind, are reviewed below.

Community Capacity & Cross-Cultural Transformative Learning

Community Capacity

In 1995, the US Centers for Disease Control and prevention (CDC) invited community health researchers to join CDC community specialists in a symposium focusing on "Identifying and Defining the Dimensions of Community Capacity to Provide a Basis for Measurement" (Goodman et al., 1998). The CDC recognized the importance of "community capacity" while acknowledging the lack of clarity of the concept.

The result of the symposium was a report that identified and described 10 dimensions and dozens of characteristics of community capacity. These characteristics are helpful guides when conceptualizing desirable outcomes and implementing any Extension program. However, recalling the observations made by Hassel, Klemme, Hausafus, and Shirer, and Thering, a few characteristics are particularly relevant when keeping diversity in mind. These characteristics include:

• Receptivity to prudent innovations;
• Ability to access external resources;
• Frequent cooperative decision making processes amongst local leaders, agencies, and organizations;
• Ability to reflect on the assumptions underlying ideas and actions (adapted from Goodman et al., 1998).

The characteristics "receptivity to prudent innovations" and "ability to access external resources" are important because they suggest the program has overcome barriers to trusting relationships with outsiders; the characteristic "frequent cooperative decision making processes amongst local leaders, agencies, and organizations" is important because it suggests the program has overcome barriers of local factionalism.

The "ability to reflect on the assumptions underlying ideas and actions" is a particularly important characteristic because that ability is also a defining characteristic of "transformative learning" (Habermas, 1979; Mezirow, 1997). Thus, recent literature on "cross-cultural transformative learning" that contrasts the assumptions underlying the "expert/client" approach to Extension programming with a more inclusive approach will be particularly helpful for operationalizing these characteristics for program development and evaluation with diversity in mind.

Cross-Cultural Transformative Learning

Subedi (2004) and Merryfield and Subedi (2006) built on the ideas of "Communicative Learning" and "Transformative Learning" developed by Habermas (1979) and Freire (1970), and later by Mezirow (1997) by focusing on learning that engages critical reflection on assumptions about unfamiliar cultures. Subedi and Merryfield compare and contrast how knowledge, culture, and language are conceptualized in a "Deficit Model" relative to a "Transformative Model" of education.

• In the Deficit Model, legitimate knowledge, human history, and truth are assumed to originate in European/Western societies; other sources and viewpoints are understood as inferior. Conversely, the Transformative Model emphasizes the value of multiple perspectives, avoids hierarchical frameworks for truth and legitimacy, and acknowledges the relationship between legitimizing knowledge and legitimizing power.
• The Deficit Model reinforces stereotypes that imagine non-mainstream cultures as homogeneous communities of exotic, bizarre, or primitive people, thus a "problem" to be studied or solved. Conversely, the Transformative Model respects differences between civilizations and culture groups, while recognizing economic, ethnic, religious, linguistic, and/or political differences exist within any civilization or culture group.
• The Deficit Model is characterized by disregard, ignorance, or unwillingness to recognize the fact that some terminologies are preferred, proper, and/or respectful when speaking with or about individuals or culture groups. Conversely, the Transformative Model recognizes the importance of language and terminology, and recognizes that language, including choosing to recognize or not recognize preferences in terminology, is a political choice and an exercise of power.

These definitions of "community capacity" and "cross-cultural transformative learning," when combined with the insights offered by Habermas, Hassel, Klemme, Hausafus, and Shirer, and Thering, offer the cogent, theory-based guidance required to identify and overcome barriers to partnerships and articulate links among activities and intended educational and community capacity building outcomes of transdisciplinary action-research partnerships. However, as noted in the introduction, if Extension professionals are to engage transdisciplinary action-research approaches in response to increasingly complex community issues, they require a program development and evaluation methodology that is flexible enough to allow iterative explorations with multiple and mixed methods.

Relatively recent responses from the specialized discipline of program evaluation research have focused attention on investigating the challenges of evaluating complex community initiatives. The section below describes two recent developments that offer a program development and evaluation framework laying the groundwork for the methodology proposed in the subsequent section.

The Logic Model and Successful Outcome Markers

In 1995, the Aspen Institute sponsored a Roundtable on "New Approaches to Evaluating Community Initiatives" (Connell, Kubisch, Schorr, & Weiss, 1995). The "Theories of Change Approach" articulated in that report (Weiss 1995) has become the foundation of the "Logic Model" approach to program evaluation, which is now a standard tool for program evaluation in Extension around the country and, recently, a required component of proposals to many state, federal, and non-profit programs (e.g., US Department of Housing, 2009; Kellogg Foundation, 2001).

Weiss noted that social programs are based on explicit or implicit assumptions (theories) about how and why they will work. Thus, the evaluation of any program should identify the underlying assumptions and then develop methods for data collection and analysis to track the "unfolding of the assumptions" (Weiss, 1995, p. 67).

At the most basic level, the Logic Model approach to program development and evaluation asks program planners to articulate: 1. The issues the program is to address; 2. The activities they plan to undertake to address the issues; 3. The intended outcomes of the program; 4. How they intend to document the outcomes; and 5. The assumptions (theories of change) that explain the relationships between the other four items (i.e., the logic of the program) (Kellogg Foundation 2001; UWEX 2009).

The Logic Model is a flexible framework that allows Extension professionals to explore multiple and mixed program development and evaluation methods. However, it offers no cogent guidance for identifying and overcoming barriers to partnerships with underserved communities, or relevant theory to articulate links among activities and intended educational and community capacity building outcomes with diversity in mind.

The Success Outcome Markers in Extension (SOME) approach (Rockwell, Jah, & Krumbach, 2003) makes a contribution by refining the logic model into an evaluation flowchart grounded in education theory. This approach asks program planners to: 1. Create a vision for broad long-term outcomes; 2. List the "WHOs" (all the individuals or groups involved); 3. Write an outcome challenge for each WHO; 4. List the Success Outcome Markers (SOMs) (i.e., behavioral indicators of transformative learning) for each WHO; and 5. Decide how to monitor and report on each SOM. However, while the SOME framework articulates causal links between activities and intended outcomes, its focus on transformational learning limits its flexibility, and, like the Logic Model, it offers no cogent guidance for identifying and overcoming barriers to partnerships with underserved communities.

Thus, as helpful as both these frameworks are, neither fulfills all three characteristics of a program development and evaluation methodology required for transdisciplinary action-research as articulated in the introduction. What is needed is a model that:

• Combines the flexibility of the Logic Model with SOME's attention to transformative education theory;
• Has built-in processes that allow for iterative critical reflection on assumptions; and
• Of particular importance, offers guidance for identifying and overcoming barriers to partnerships with diversity in mind, i.e., informed by cogent insights from social science, education, and human health research.

The section below introduces a methodology that fulfills all three characteristics. This methodology emerged in response to iterative critical reflection and program evaluation research conducted during the early years of an ongoing transdisciplinary action-research partnership with First Nations in the Upper Midwest.

A Program Evaluation Methodology with Diversity in Mind

Table 1 is an excerpt from a recent program evaluation report for the ongoing Green Community Development and Green Affordable Housing in Indian Country Initiative. That initiative began in 2002, when the Director of Housing from a neighboring First Nations community contacted the Extension faculty from the Department of Landscape Architecture at the University of Wisconsin-Madison (UW Team) for technical assistance with community development. Over the subsequent years, upon the recommendations of our professional colleagues and local leaders in that community, and with support from the state, federal, and non-profit organizations with whom they work, the UW Team has developed partnerships with planning professionals and local leaders in several First Nations across the state.

These transdisciplinary partnerships have engaged a broad scope of projects and programs. Participants have included local professionals, elected officials, non-profit staff, community members, faculty and students from nearby Tribal Colleges, local artisans and tradespeople, and faculty and students from several professional programs at the university. Venues have included community centers, construction sites, studios, classrooms, and conference rooms.

While this methodology continues to reveal barriers and document outcomes for all participants as the program evolves, the process began with noticing barriers attributable to characteristics of each participant or group. The example focuses on "Outsiders" to illustrate the usefulness of the methodology.

Table 1.
Research Program: Identifying Barriers to Transdisciplinary Partnerships and Documenting Transformative Learning Outcomes

Phase One		Phase Two	Phase Three
Characteristics of Academics, Professionals, Agency Staff ("Outsiders")	Barriers to transdisciplinary partnerships	Intended Outcomes of Program	Documenting Outcomes
			Short-term (Communicative)	Longer-term (Behavioral)
Research findings and/or organizational protocol dictates what is correct	Outsiders are unaware or uninterested in local expertise and/or protocol.	Outsiders are respectful of ideas from local staff.	Outsiders suggest solutions that reflect new understanding of local expertise and/or protocol.	Collaborative decision making informing action in real time.
Often from dominant culture.	Unaware or uninterested in preferred terminologies.	Outsiders are aware that the terms they choose to use and/or not use are indicators of attitudes.	Outsiders respectfully inquire about preferred terms, names, and titles	Outsiders routinely include study of preferred terms when preparing for engagement with non-mainstream cultures.
Interaction with locals based on stereotypes or broad impressions gleaned from a few brief interactions.	Sustains local perceptions of impersonal, detached bureaucrat/expert outsider.	Outsiders recognize that a spectrum of values, beliefs, skills, behaviors exist in the partner community.	Outsiders remember names, titles, and unique roles and responsibilities of locals.	Locals and outsiders anticipate/respect/forgiving of each others' individual concerns/preferences/abilities, professional and personal.

Phase One of this methodology asks the Extension professional (EP) to engage in critical reflection on program assumptions to: 1. Note signs of frustration, confusion, and/or conflict between various individuals and/or groups; 2. Identify specific characteristics of the various partner groups that may be contributing factors; 3. Describe the behaviors in terms that illustrate how they are barriers to transdisciplinary partnerships and common goals; and 4. Substantiate their observations with relevant theory.

In the example, the author suggested: "Research findings and/or organizational protocol dictates what is correct;" "Often from dominant culture;" and "Interaction with locals based on stereotypes or broad impressions gleaned from a few brief interactions" are characteristic of many Outsiders from university, government, and professional organizations. The barriers attributed to these characteristics were described as: "Unaware or uninterested in local expertise and/or protocol;" "Unaware or uninterested in preferred terminologies;" and "Sustains local perceptions of impersonal, detached bureaucrat/expert outsider," respectively. The report cited the articles by Hassel, Subedi and Merryfield, Subedi, and Thering to substantiate these observations.

Phase Two of this methodology asks the EP to respond to the barriers identified in Phase One by: 1. Describing the intended outcomes of the initiative, for individuals, groups, and of particular importance, the partnership, and 2. Substantiating the list of intended outcomes with relevant theory.

In the example, the author described: "Outsiders are respectful of ideas from local staff;" "Outsiders are aware that the terms they choose to use and/or not use are indicators of attitudes;" and "Outsiders recognize that a spectrum of values, beliefs, skills, and behaviors exist in the partner community" as intended outcomes. The report again cited the articles by Hassel, Subedi and Merryfield, and Subedi.

Phase Three of this methodology asks the EP to: 1. Explore multiple and mixed methods to identify and document learning outcomes that respond to the results of Phase Two and 2. Substantiate the list of outcomes with relevant theory and data. (It is important to reiterate that this approach is cyclic, thus the objectives of Phase One and Phase Three are often addressed simultaneously).

In the example, the author listed: "Outsiders suggest solutions that reflect new understanding of local expertise and/or protocol;" "Outsiders respectfully inquire about preferred terms, names, and titles;" and "Outsiders remember names, titles, and unique roles and responsibilities of locals" as short-term outcomes. Long-term outcomes included: "Collaborative decision making informing action in real time;" "Outsiders routinely include study of preferred terms when preparing for engagement with non-mainstream cultures;" and "Locals and outsiders anticipate/respect/forgive each others' individual concerns/preferences/abilities, professional and personal." In addition to the articles cited in Phase One and Two, this section of the evaluation cited the CDC report on community capacity and the Habermas and Mezirow articles on education theory. These references were included to explain how verbalizations and actions indicate communicative and transformative learning. Thus, these outcomes substantiated claims that the program initiated critical reflection on assumptions, i.e., made the Outsiders aware of the barriers to partnership and inspired them to adjust assumptions.

Summary and Recommendations for Future Research

Extension professionals are increasingly asked to overcome barriers formed by culture, class, ethnicity, race, and/or language differences as they facilitate transdisciplinary action-research partnerships in response to increasingly complex community issues. The many challenges involved in these complex programs include the challenge of program evaluation. This article articulates a methodological foundation for program development and evaluation that responds to these demands. This methodology draws on transformative education theory, cross-cultural transformative learning theory, health science research, and insight from transdisciplinary action-research practitioners. An excerpt from a report on the Green Communities and Green Affordable Housing in Indian Country Initiative is analyzed to illustrate the methodology in practice.

It is important to note that while the example focuses solely on overcoming obstacles to partnerships attributed to the characteristics of academics, professionals, and agency staff ("Outsiders"), the ongoing program evaluation research investigates characteristics, barriers, and outcomes for all participants. Ongoing research includes development and testing of competing theories, new heuristics, and new methods that further inform the practice of transdisciplinary action-research.

References

Blewett, T., Keim, A., Lesser, J., & Jones, L., (2008). Defining a transformational education model for the engaged university. Journal of Extension [On-line], 46(3) Article 3COM1. Available at: http://www.joe.org/joe/2008june/comm1.php

Braverman, M., & Arnold, M. (2008). An evaluator's balancing act: Making decisions about methodological rigor. Program evaluation in a complex organizational system: Lessons from Cooperative Extension. M. Braverman, M. Engle, M. Arnold, & R. Rennekamp (Eds.), New Directions for Evaluation, 120, 71-86.

Brown, V. (2005). Linking knowledge cultures of sustainability and health. Sustainability and Health: Supporting Global Integrity in Public Health, Brown V., Grootjans, J., Ritchie, J., Townsend, M. Verrinder, G., Eds. Canberra, Australia: Allen & Unwin, and Earthscan.

CDE (2009). Social learning processes and sustainable development. Center for Development and Environment. Retrieved 16 March 2009 from: http://www.cde.unibe.ch/Themes/LP_Th.asp

Connell, J., Kubisch, A., Schorr, L., & Weiss, C., Eds. (1995). New approaches to evaluating community initiatives: Concepts, methods, and contexts. Washington, DC: Aspen Institute.

Freire, P. (2005). Pedagogy of the Oppressed 30th Anniversary Edition. NY. NY.: Continuum International.

Goodman R., Speers, M., Mcleroy, K., Fawcett, S., Kegler, M., Parker, E., Smith, S., Sterling, T., & Wallerstein, N. (1998). Identifying and defining the dimensions of community capacity to provide a basis for measurement. Journal of Health Education & Behavior. 25 (3), 257-278.

Habermas, J. (1979). Communication and the evolution of society. Boston, MA: Beacon Press.

Hassel, C. (2004). Can diversity extend to ways of knowing? Engaging cross-cultural paradigms. Journal of Extension [On-line], 42(2) Article 2FEA7. Available at: http://www.joe.org/joe/2004april/a7.php

Kellogg Foundation. (2001). Logic model development guide: Logic models to bring together planning, evaluation & action. Battle Creek, MI: W.K. Kellogg Foundation.

Klemme, D., Hausafus, C., & Shirer, K. (2005). Extension staff response to increased programming for at-risk audiences. Journal of Extension [On-line], 43(1) Article 1FEA6. Available at: http://www.joe.org/joe/2005february/a6.php

Merryfield, M., & Subedi, B., (2006). Decolonizing the mind for world centered global education. The Social Studies Curriculum: Purposes, Problems, and Possibilities. Ross, E., ed. Albany NY: SUNY Press.

Mezirow, Jack (1997). Transformative learning theory to practice. New Directions for Adult and Continuing Education. 74 (Summer): 5-12.

O'Connor, A. (1995). Evaluating comprehensive community initiatives: a view from history. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 1. Connell, J., Kubisch, A., Schorr, L., & Weiss, C., eds. Washington, DC: Aspen Institute.

Rockwell, S., Jha, L., & Krumbach, E. (2003). Success outcome markers in Extension (SOME): Evaluating the effectiveness of transformational learning programs. Journal of Extension [On-line], 41(5) Article 5FEA4. Available at: http://www.joe.org/joe/2003october/a4.php

Stokols, D. (2006). Toward a science of transdisciplinary action research. American journal of community psychology. 38 (1-2), 63-77.

Strand, K., Marullo, S., Cutforth, N., Stoecker, R., & Donohue, P., (2003). Community-based research and higher education. San Francisco, CA: Jossey-Bass.

Stringer, E. (1999). Action research: second edition.. Thousand Oaks, CA: Sage Publications, Inc.

Subedi, B. (2004). An issue-oriented transformational model on global curriculum/teaching. Keynote address. Internationalizing University of Wisconsin-Extension Conference. Wisconsin Dells, WI.

Thering, S. (2007). A practical theory-based approach to action-research in survivor communities. Journal of Extension [On-line], 45(2) Article 2FEA3. Available at: http://www.joe.org/joe/2007april/a3.php

US Department of Housing and Urban Development (2009). Using the HUD eLogic Model: an accountability tool. Retrieved 16 March 2009 from http://www.hud.gov/offices/adm/grants/training/odgmotraining.cfm

UWEX. (2009). University of Wisconsin-Extension. Program Evaluation; Logic Model. Retrieved 16 March 2009 from http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, 1. Connell, J., Kubisch, A., Schorr, L., & Weiss, C., eds. Washington, DC: Aspen Institute.