Evaluation of Capacity-Building Programs: A Learning Organization Approach

Many Extension programs are designed to enhance the capacity of organizations, including small businesses, local community development organizations, and youth groups, to define and reach their goals. Improving capacity in organizations usually involves changing the process by which members of an organization work together and make decisions.

Evaluating the effectiveness of Extension programs has emerged as a critical issue. For programming that entails immediate behavioral changes and/or improvements in individual skill levels, appropriate evaluation techniques are widespread (e.g., Gentry-Van Laanen & Nies, 1995; Earnest, 1999; Stevens & Lodl, 1999). However, capacity-building involves collective behavior, not simply the individual behavior of participants. Changes in organizational behavior may not occur for several years. Furthermore, measuring changes in organizational process and decision-making are problematic. There is a need among Extension educators for a new set of simple and systematic evaluation tools that capture the impact that their programming has in producing organizational change.

This article identifies questions that Extension educators can ask in evaluating the impact of their interventions on a specific organization, whether a non-profit enterprise (e.g., a local development organization or chamber of commerce) or a business. The article is a first step in the design of a new "toolbox" to evaluating organizational change based on the learning organization model developed by Peter Senge and colleagues (1990, 1994, 1999). Under this model, an organization's capacity is defined by its ability to learn, to share that learning throughout the organization, and to modify its behavior to reflect new knowledge and insights.

Senge (1990) defines a learning organization as one "where people continually expand their capacity to create the results they truly desire, where new and expansive patterns of thinking are nurtured, where collective aspiration is set free and where people are continually learning how to learn together." Such an organization has tremendous capacity to reach its goals. Any type of organization can be a learning organization, including businesses, educational institutions, nonprofits, and community groups.

In this article, we begin by briefly describing the five practices that form the framework for a learning organization. Next we provide several examples of the approach's success in empowering organizations. Finally, we identify broad questions for Extension educators to ask in evaluating organizational change.

The Learning Organization Approach

Becoming an organization that engages all members in active learning and provides mechanisms for the transfer and application of that knowledge requires a collective mind shift at all levels. Such mammoth change is a complex, long-term undertaking. Therefore, a Learning Organization is best viewed as an ideal, a vision of what organizations might become. Organizations or parts of organizations achieve this ideal to varying degrees.

Senge (1990) identifies five disciplines that enable organizations to move toward the ideal of a Learning Organization. (In this article, the term "practice" is used instead of "discipline" to avoid confusion with the familiar term "academic discipline.") All five practices are concerned with a mind shift from seeing parts to seeing wholes, from seeing people as helpless reactors to seeing them as active participants in shaping their reality, from reacting to the present to creating the future.

The five practices are:

• Systems thinking
• Personal mastery
• Mental models
• Shared vision
• Team learning

System Thinking

Senge places system thinking first on his list, because it is the conceptual cornerstone that underlies all of the other practices. Systems thinking is a body of knowledge and tools developed over the last 50 years that serve to make clearer the full patterns of the problems, issues, and situations that confront us. The tools of systems thinking allow us to talk about interrelationships more easily because they are based on feedback processes.

The channels by which elements of a system "feed" influence and information to each other over time are the key to understanding the behavior of a system. It is about interdependencies within a system and between systems. It is not about organizational charts or functions. Farm children do systems thinking when they see links among the milk that a cow gives, the grass that she eats, and the droppings that fertilize the field. Systems thinking is useful as a problem-solving tool, but also as a language that changes the ordinary ways we think and talk about complex issues.

Personal Mastery

Personal mastery is the practice that people are drawn to the most (Senge, 1994). Personal mastery is the practice of continually clarifying and deepening our personal vision, of focusing our energies, of developing patience, and of seeing reality objectively. People in organizations often not only want to increase their own capabilities, but also to improve the capabilities of those around them. Yet, while a supportive environment for learning can be set up within the organization's infrastructure, it is the responsibility of individuals to ensure that their own learning and development continue.

Mental Models

The practice of mental models examines deeply ingrained assumptions, generalizations, or even pictures or images that influence our behavior and understanding of the world. Mental models can explain why two people can observe the same event yet have different descriptions or reactions to it. They simply pay attention to different details. Mental models are shaped in a social context. We learn deep-seated values and develop our views and understandings of the world around us through the social groups and networks of which we are a part.

Because of the tacit nature of mental models, they are generally invisible to us (Senge, Roberts, Ross, Smith, & Kleiner, 1994, p. 236). This practice seeks to bring these mental models to the surface, so that we can discuss them. Senge identifies reflection and inquiry as the two types of skills that are central to this work.

Shared Vision

Through the practice of shared vision, people are bound together around a common identity and sense of destiny whereby they excel and learn. Building a shared vision includes a vision or image of an organization's desired future and a set of governing values by which organization members define how they behave with each other, how they regard their stakeholders and the lines that they will and will not cross. When people know and understand these agreed-upon values, they are able to speak more easily, to speak honestly, and to reveal information. This fosters a supportive environment in which knowledge sharing can flourish.

Team Learning

Team learning is a practice of group interaction. Teams transform their collective thinking. They learn to mobilize their energies and actions to achieve common goals and thereby draw forth an intelligence and ability greater than the sum of the individual members' talents.

Team learning uses skillful discussion and dialogue to enable team members to move beyond the more superficial requirements of team building. People can then start to move into coordinated patterns of action, and the tedious process of planning and decision-making becomes unnecessary. They are able to act in a coordinated way, each knowing what is best to do, just as a flock of birds does when it takes flight.

Examples of Success

There are many organizations that have are applied these practices to enhance their effectiveness. Senge (1990) cites business examples, such as AT&T Corp, Intel Corp, Harley Davidson, Hewlett-Packard, Toyota, Ford Motor Co, and FedEx. At Chaparral Steel, 80% of the work force is in some form of educational enhancement at any time. They now produce a ton of steel in 1.5 employee hours compared to the national average of 6 hours (Kerka, 1995). The Electrical and Fuel Handling Division of Ford Motor Company has created 30 active team learning projects involving 1,200 employees. Sales and profits have demonstrated unprecedented growth and turn-around for the company (Bierema, 1997).

However, organizations other than businesses have also benefited from the learning organization approach. For example, the Sullivan elementary school in Tallahassee applied shared vision and core values to transform itself. Evidence of its success: teacher approval ratings are up 20%, and parents are more involved (Kerka, 1995). In the United Kingdom, many community groups have adopted the learning organization principles and declared themselves "learning towns and cities." The goals of these community groups are to encourage lifelong learning and promote social and economic regeneration. For more information, visit <http://www.lifelonglearning.co.uk/learningcities/>.

There are many examples of Extension programming that has utilized these practices. For example, the Cooperative Extension Service in Florida, Kentucky, and North Carolina practice systems thinking in their Natural Resource Leadership Institute. Each Institute involves approximately 30 participants who represent various sectors in natural resource issues, including environmentalists, developers, industrialists, and regulators. They spend 2 days every month studying issues from each other's perspectives. They are also taught skills of systems thinking, public conflict resolution, and deliberation.

In Kentucky, more than 100 people have participated in the program. They are changing the typical culture surrounding natural resource issues from an adversarial one in which people shout at each other to one where a critical mass of natural resource advocates, developers, and government regulators can reach a better understanding of each other and begin to explore options (Hustedde, 2002).

Evaluating Organizational Learning

Although learning organizations are still concerned with tangible results, i.e., market share, productivity, profitability, and growth, they understand that learning is the key to acquiring greater results. Therefore, the orientation of the learning organization is simply learning. Under this model, the critical question in evaluating Extension programming is: To what degree has our intervention changed the structure or practice of the organization so as to facilitate learning?

Here are some evaluation questions that Extension educators might ask based on the five practices.

Some Systems Thinking Evaluation Questions

• Has the interaction between units of the organization increased?
• How have we increased our understanding of how units within the organization interrelate?
• How have we expanded our understanding of the external systems that impact us? Are we more aware of the options for responding to these external forces?
• Do organization members now interact in wider networks both inside and outside of the organization?

Some Personal Mastery Evaluation Questions

• What are our personal values and how do they relate to this organization?
• What new skills and knowledge have we learned? What do we need to know that we haven't learned?
• Does the organization now provide more learning opportunities for its members?
• Does the organization help its members achieve what they really want?
• How do your goals complement goals of the others in the team or organization?
• Are there new mechanisms within the organization to share and reward learning?

Some Shared Vision Evaluation Questions

• What are the basic values undergirding our organization?
• If we understand our values, what is our vision of where we want to go as an organization?
• Has the vision of the organization become more widely shared and supported?

Some Team Learning Evaluation Questions

• How would you describe your team? Has the functioning of your team changed as a result of the Extension programming?
• Are tools such as inquiry and dialogue more widely used in the organization as a result of the programming?
• Have attitudes within the organization changed so that unexpected surprises and even failures are viewed as opportunities to learn?

The questions posed under each of the five practices are intended as useful tools for evaluating major Extension programming efforts. Obviously, there are more questions that can be asked, and some can be phrased differently. There is the potential to incorporate some questions into Likert-type scales, followed by open-ended questions, to elicit both quantitative and qualitative responses. There are many methodologies that can be used to address these questions, including facilitated discussions, focus groups, surveys, and informal feedback.

Final Comments

We believe that Extension programming, whether in community development, nutrition, youth development, small business, or other fields, strengthens groups by enhancing the capacity of members to work together effectively. Yet evaluating these impacts is difficult and rarely done in practice. The model of a learning organization may provide a framework to better evaluate these interventions.

This article is the first step in designing a learning-based approach to program evaluation. Clearly, more research needs to be done in designing evaluation tools. However, the learning organization approach offers the promise of providing Extension educators with mechanisms to demonstrate the value of the work that they are doing in improving the long-term stability and effectiveness of organizations.

References

Bierema, L. L. (1997). Research as development: A learning organization implementation. Annual Conference Proceedings of the Academy of Human Resource Development: 390-397.

Earnest, G. (1996). Evaluating community leadership programs. Journal of Extension [On-line], 34(1). Available at: http://www.joe.org/joe/1996february/rb1.html

Gentry-Van Laanen, P., & Nies, J. I. (1995). Evaluating extension program effectiveness: Food safety in Texas. Journal of Extension [On-line], 33(5). Available at: http://www.joe.org/joe/1995october/a4.html

Hustedde, R. J. (2002). The role of the Cooperative Extension Service and innovations in community development. Unpublished manuscript.

Kerka, S. (1995). The learning organization. Myths and realities. (ERIC/ACVE MR 00004).

Senge, P. (1990). The fifth discipline. New York: Doubleday.

Senge, P, Roberts, C., Ross, R., Smith, B., & Kleiner, A. (1994). The fifth discipline fieldbook: Strategies and tools for building a learning organization. New York: Doubleday.

Senge, P., Kleiner, A., Roberts, C., Ross, R., Roth, G., & Smith, B. (1999). The dance of change: The challenges to sustaining momentum in learning organizations. New York: Doubleday.

Stevens, G., & Lodl, K. A. (1999). Community coalitions: Identifying changes in coalition members as a result of training. Journal of Extension [On-line], 37(2). Available at: http://www.joe.org/joe/1999april/rb2.html