Can We Define and Measure Excellence in Extension?

Are We Interested in Our Egos or Excellence?

Chester Fehlis in his 2004 Ruby Lecture, "A Call for Visionary Leadership," stated:

In Extension, every institution and its faculty have their own personally defined metrics. . . . I have discovered that every Extension Director believes that his or her state's Extension program is in the top five or ten in the country. That may be good for our egos, but it is not good for Extension. . . . If we are truly to have visionary Extension leaders and administrators . . . then we should consider establishing accepted criteria and metrics that clearly define excellence in Extension (Fehlis, 2004; Fehlis, 2005).

In a follow-up presentation to the Extension Committee On Policy (ECOP) in February 2005 (Fehlis, 2005), Fehlis stated that because there were no established criteria that define "Excellence in Extension," we do not capitalize on the synergy of our national Extension system, but rather have 70 Extension programs going in many different directions, each believing theirs is the most effective and best Extension program in the nation.

Fehlis challenged the Extension system to take the initiative to define excellence and to identify ways to measure it. In the summer of 2005, ECOP responded by appointing a task force representing all five regions to focus on measuring Excellence in Extension. The group was charged with:

• Identifying the criteria that defines Excellence in Extension


• Determining the measures of those criteria


• Delineating the necessary collection methods


• Determining how to gain acceptance of the criteria and measures by the Extension system

What Do Extension Administrators Do Best? They Delegate!

The task force, along with a work group of Extension evaluators, set about defining the criteria of excellence and identifying appropriate measures. They:

• Brainstormed what constitutes Excellence in Extension


• Developed a matrix to display the criteria of excellence


• Obtained feedback on criteria from directors and administrators at the regional level


• Engaged directors and administrators in determining priorities.

Who Do We Serve? A Matrix Evolves

The work group struggled in early months with how to identify criteria of excellence that are meaningful to all stakeholders, given the diversity of institutions and Extension systems around the country. The product was a 4" by 5" matrix based on two conceptual frameworks. One axis represents the traditional academic perspective of universities. The four columns in the matrix are: Teaching and learning; Discovery and scholarship; Engagement; and Management. This approach reflects and values the language of research and teaching faculty and university administrators.

The second axis represents a perspective that is more familiar to Extension personnel and supporters. The five rows in the matrix are: University commitment to Extension; Relevance of Extension programs; Quality of Extension programs; Outcomes and impacts; and Funding and external support.

These two dimensions illustrate some of the variability among Extension's many partners and stakeholders as to how they perceive the value of Extension. The primary audiences for this information are persons making funding decisions, university administrators, and Extension decision makers.

Identifying criteria of overall Excellence in Cooperative Extension is a valuable exercise for the Extension system as well as for individual institutions. These criteria will enable Extension to describe its strengths and to differentiate itself from other agencies and organizations. (See "Excellence in Extension: Two Products for Definition and Measurement" in this issue for a fuller description of the Matrix of Criteria of Excellence in Cooperative Extension.)

Can a System to Measure Excellence in Extension System Actually Be Established?

The task force and work group encountered and resolved some significant problems: agreeing on the criteria for excellence, developing a widely accepted definition for each criterion that was deemed highly important, and identifying measures for each criterion. Several challenges remain: gaining acceptance of the measures by the system, collecting the data, and developing, maintaining, and updating a national database.

Through a yearlong interaction of task force members, work group members, and periodic interaction with Extension administrators at the regional and national levels, 61 criteria were identified. Based on the priorities expressed by Extension directors and administrators, the number of criteria was reduced to 19, and seven were identified as most important. The original 61 were retained because some may be useful to individual institutions.

For the seven most important criteria, uniform definitions were developed. This was vital because individual institutions currently use unique definitions that are rooted in history and tradition. Without uniform definitions, aggregating measures and comparing across institutions is problematic.

Measures were also identified for the seven highest priority criteria. The focus was on measures that are reasonable to collect without placing an undue burden on institutions; however, to implement some of the measures, there is a need to conduct more in-depth studies.

A national database will be established and patterned after one developed by Texas Cooperative Extension for use by the Association of Southern Region Extension Directors. It will focus on institutional-level indicators for internal decision-making. This does not duplicate the Cooperative States Research, Education, and Extension Service (CSREES) USDA reporting system, which focuses on program results of individual programs. It capitalizes on the work of individual states, regional and national subject matter work groups developing measures of program impact. Institutions will self report data through password-protected access. The database will require maintenance, and a mechanism will be needed to make decisions about changes in indicators that can be expected to evolve over time.

It is important to understand the distinction between two separate elements of this product. One element is the seven criteria for excellence and associated measures and definitions. The other element is the set of measures to be included in the national database. These are related, but they are not the same.

The seven criteria are better employed as an internal mechanism for individual Extension systems to determine how well they are doing on the things that would make their own systems excellent. Only a portion of the criteria measures will be included in the national database. Measures from the criteria that are included in the national database have the greatest chance of being collected in an easily quantifiable and uniform manner. The national database measures will be used more for comparison and aggregation purposes across the entire Extension system.

Who Should Use the Products? All Extension

A national effort will give credibility to the criteria of excellence and will encourage uniformity in measurement throughout Extension. The criteria and measures identified will assist institutions as they identify indicators; consistent criteria will provide a basis for standardized comparisons with other institutions; and measures reported by all institutions can be aggregated to represent the entire Cooperative Extension System

This information can be used to describe the overall scope of Extension and to market it as a national system. Such information could also be used to document Extension's strengths or to focus on areas that could be improved.

The matrix provides a two-dimensional perspective on criteria for Excellence in Cooperative Extension. It describes Excellence in Extension in a concise, yet comprehensive way.

A "pocket card" provides a list of five attributes that make Extension unique, six summary points of qualities for Excellence in Extension, and the seven criteria for Excellence in Cooperative Extension. This card summarizes what can become every Extension worker's or administrator's talking points about Extension. (See "Excellence in Extension: Two Products for Definition and Measurement".)

The database provides the mechanism for entering, aggregating, sharing, and archiving the identified measures, as well as other data that will help standardize comparisons between different Extension institutions. How well this works will depend greatly on how individual institutions of the Extension System embrace the need.

Are There Flaws? You Decide

This has been an internal effort. Extension directors and administrators established the priorities, and Extension evaluation specialists outlined the detail. Would the matrix, criteria, definitions, and measures be different if rated by alternative audiences, e.g., legislators, other government officials, university administrators or clients?

We think not much. Over a period of 18 months, Extension has taken a thoughtful and critical look at itself and proposed a process to measure its internal progress, make comparisons among institutions, and at the same time provide a succinct description of Excellence in Extension to the world.

What Now? Gain Acceptance

The final charge of the ECOP Task Force on Measuring Excellence in Extension is to determine how to gain acceptance of the criteria and measures by the Extension System. Extension administrators and directors have been updated several times during the 18 months of development. The products have been, or soon will be, published in a variety of venues to share with the Extension System.

To be effective, this process and these products for understanding and measuring Excellence in Extension must be widely embraced. This may be the greatest challenge of all. Institutions may need to develop or modify data collecting and reporting routines to fit with the uniform measures that are proposed.

But the potential payoff can be huge. If this approach gains acceptance throughout the Extension System, then perhaps we will have responded adequately to Dr. Fehlis' call to "establish accepted criteria and methods that define Excellence in Extension."

References

Fehlis, C. P. (2004). A call for visionary leadership, Ruby Lecture, National Epsilon Sigma Phi Annual Meeting, September 17, 2004. Available at: http://espnational.org/historybyyear/2004/ruby2004.htm

Fehlis, C. P. (2005). A call for visionary leadership. Journal of Extension [On-line], 43(1) Article 1COM1. Available at: http://www.joe.org/joe/2005february/comm1.shtml

Fehlis, C. P. (2005) Defining excellence in Extension. Presentation to Extension Committee on Policy (ECOP), February 15, 2005.