Leveraging Utilization-Focused Evaluations to Engage Elected Officials and County Personnel

Introduction

It was universally agreed near the end of the 20th century that elected officials are an audience essential to the existence, relevance, and funding of Extension (Hahn, 1990). It has been argued, however, that Extension does not always know enough about its customers and that many audiences do not know enough about Extension and its offerings (King, 1993). County-level elected officials and personnel comprise one such group. For Extension programming to continue to be successful, elected officials must be aware of Extension's effectiveness (Kabes, 1991; Kalambokidis, 2004). Moreover, suggesting that awareness alone is inadequate, Frederick (1998) noted that "it's not enough [for Extension] to provide elected officials with an overview of programs offered to the general public" but that instead Extension must "offer programs that directly benefit [elected officials] in their role as policymakers" ("Policymakers as Customers for Public Policy Education" section, para. 1). Extension can support elected officials in solving local problems, building consensus, and improving strategic planning. More specifically, one opportunity Extension professionals have to better serve county commissioners is to assist them in implementing an engaged evaluation process to achieve organizational change.

Vines (2018) noted a dichotomy between expert and engagement models. This dichotomy can be important with regard to supporting and serving elected officials. The difference is one-way communication (expert model) versus two-way communication (engagement model). Using their expertise, Extension professionals can engage commissioners in a meaningful exchange to achieve success for both a county and Extension within that county. We are part of a Washington State University (WSU) Extension team that has supported Ferry County, Washington, in such a way. In our case, Extension and county officials and personnel, working together, implemented an evaluation model featuring two-way communication and consensus building to establish continuous process improvement that has led to sustainable programs. In other words, county officials wanted to develop an engaging annual evaluation process that would inform strategic planning, budgeting, and governance efforts, which in turn would support local Extension programs deemed important to the county, and we are achieving that goal.

There are often financial challenges associated with the ability to gather and use data in rural areas. County leaders are faced with expensive options when they need to conduct an evaluation or otherwise collect critical information. Simply put, research and evaluation can be expensive or cost prohibitive. One way for Extension to help counties achieve effective evaluation is by assisting in the application of utilization-focused evaluation (UFE). Providing such assistance generates the opportunity for organizational development and organizational change within the county and leads to a close, engaged relationship between Extension and the stakeholders.

Developed by Michael Quinn Patton, UFE is based on the premise that evaluation should be judged on its usefulness to the intended users (Patton, 1978). Therefore, evaluators should design an evaluation and facilitate the process with careful consideration of how everything that is done, from beginning to end, will affect use of the evaluation (Patton, 1997). UFE is particularly appealing to Extension practitioners for a couple of reasons. First, UFE allows for customization of the evaluation process. Second, UFE is likely to foster buy-in as it engages stakeholders in developing the evaluation methodology and instrument.

Herein, we describe our example of applying the engagement model and UFE to assist a county in creating effective two-way communication between county commissioners and county employees. In the case we present, WSU Extension collaborated with Ferry County to leverage UFE after the county experienced a series of economic setbacks that culminated in the closure of an important gold mine and the state's highest unemployment and poverty rates. Ferry County needed to assess its organizational climate on a regular basis to better understand changes in infrastructure, operational needs, economic development issues, and budget challenges that were affecting resources, funding, and morale. According to Patton (1997), an engaged approach has been found to improve the probability that both the evaluation process itself and an evaluation's findings will be used for informing decisions and improving performance. Beyond benefits to the county, WSU Extension's efforts to support the commissioners also demonstrated that collaborating with counties to develop effective evaluations is an important role that Extension leaders are highly capable of filling. Following the stark change in county economic conditions and disappearance of critical industries, the integrated and engaging support Extension provided was of great value to Ferry County and served as a reminder of Extension's importance.

Survey Approach and Framework

When gold mine officials announced the mine's closure and the company's exit strategy in early 2014, county commissioners asked Extension to evaluate the situation. Then in late 2014, commissioners again turned to Extension for assistance with ongoing economic development programming and planning. As a result of ensuing work by Extension and the county, a long-term sustainable annual evaluation process, achieved through a county climate survey, is now in place. A high level of annual collaboration stemming from a model of engagement between Extension and county commissioners has led to a valid UFE process for informing community and economic development. Since the point at which consensus was reached on the evaluation framework and approach, Extension faculty have continued using the engagement model to facilitate and contribute to two-way communication involving department heads and county employees, key intended stakeholders in the UFE process. Collective efforts have been made to distribute the evaluation findings to employees each summer. Monthly meetings with county department heads are used for reviewing the findings, analyzing the data, and disseminating results to the commissioners for use in strategic planning and budget hearings. The evaluation framework and approach are explained herein.

The original UFE framework introduced by Patton in 1978 has five steps for coordinating useful surveys. Each step contributes to maintaining focus on the usefulness of both the process and findings of the evaluation. The latest version of UFE involves 17 steps; however, for our collaboration with Ferry County, we applied the original five steps. Using this approach shortened the process and streamlined input, allowing us to implement an online survey, collect data, and begin analysis in a few short weeks. UFE focuses on specific users of an evaluation, such as commissioners and county employees, and specific uses of the evaluation, such as for strategic planning, rather than on general or abstract users and uses. Along with five critical steps, UFE as described in Patton's original work involves two essential elements (Table 1).

Table 1.
Essential Elements of Utilization-Focused Evaluation

Element	Description
Users	Primary users of the evaluation must be clearly identified and personally engaged at the beginning of the process to ensure that their primary intended uses of the evaluation can be identified.
Evaluators	Evaluators must ensure that the intended uses guide all decisions made about the evaluation process.

An examination of how we initially applied the five steps and two essential elements of UFE in an engaged effort in Ferry County can illustrate how others in Extension may directly engage and benefit elected officials in a meaningful and valuable way.

Step 1. Identify primary intended users. In the case of Ferry County, the intended users of the evaluation process and findings are all county employees, elected or not. Originally, the commissioners needed input from county employees due to perceived changes in service and morale coupled with a drastic downturn in the local economy.

Step 2. Gain commitment to UFE, and focus the evaluation effort. The commissioners were intentional in having Extension faculty use their social capital and resources to engage in UFE because they realized the value of anonymously collecting useful information to identify needs, challenges, and opportunities correlating to governance and public service offerings. In this case, the UFE element of inclusion led to commitment to a sustainable evaluation process that would involve distribution of an annual survey.

The stakeholders focused the evaluation effort by further agreeing that the sustainable evaluation process would include the following procedures:

• approving survey questions on the basis of input from the intended users to ensure that questions would be useful and pertinent to relevant issues;
• distributing the survey, collecting anonymous data, and analyzing the results with intended users; and
• using the evaluation to inform economic development, strategic planning, and budgeting.

Step 3. Make decisions about and develop the evaluation methodology and survey instrument. Extension faculty worked with the county to accomplish the following goals as part of the commitment to annual evaluation:

• Create a secure, confidential, and anonymous survey.
• Create and maintain an unbiased instrument designed to establish a baseline data set contributing to an annual evaluation.
• Incorporate peer review and feedback loops in the survey design or evaluation process and in the dissemination of the findings.

In this case, Extension's role as the evaluator was inclusive, rather than one performed in isolation as an expert developing an evaluation methodology. WSU Extension professionals engaged the intended users, county employees, in developing and implementing the survey. They worked with these intended users to connect with other faculty, researchers, and subject matter experts. The subject matter experts communicated with both commissioners and county employees (engagement model, Vines, 2018) and contributed to developing a useful and effective survey (UFE, Patton, 1997). Using input from employees gained through monthly department head meetings, WSU Extension faculty established the three-pronged approach that would be used to develop the survey instrument and conduct the survey:

1. Use an online survey platform, such as the academic Qualtrics research tool easily accessible to faculty.
2. Amend or edit existing validated questions from the online system to fit local needs.
3. Implement the survey online to reduce costs, improve efficiency, and establish the initial baseline for a longitudinal study.

This three-pronged approach was approved by the institutional review board at WSU and agreed on through consensus building among the intended users.

Step 4. Analyze and interpret findings and reach conclusions. Analysis and interpretation of the findings were achieved through engagement with departments and individuals, which took place at department head meetings and in individual follow-up sessions. Specifically, our team of collaborating Extension professionals and Ferry County officials and personnel quantified the data and devised displays of the findings using standard visuals (e.g., bar graphs, line graphs, pie charts, and tables with aggregated data), data presentation options available via most online survey products.

An example of a conclusion based on our analysis was that questions that were not equally relevant to all county employees may have contributed to a high number of "neutral" responses by survey takers. This observation in the analysis phase created open and ongoing dialogue between commissioners and frontline staff, who seldom have direct input in the analysis and interpretation aspects of evaluation processes. Other conclusions related to addressing concerns identified in the survey responses.

Step 5. Disseminate evaluation findings and demonstrate utility. The intended users of the UFE process also were engaged in disseminating findings, including in the annual report. Upon completion of data analysis, department heads shared the findings with county employees before sharing them with the commissioners. Extension faculty followed up with commissioners to further disseminate the findings and to help them prepare for use of the findings in budget hearings, strategic planning, and future decision making. A final meeting during budget hearings was also part of the process to further disseminate the findings (and to demonstrate utility of the survey as well).

Utility was demonstrated with regard to both the evaluation process and the evaluation findings. For example, one illustration of the utility of the process is that the survey itself became a motivator for participation, as demonstrated through increased response rates over time. Additionally, analyzing the data through an engagement model, rather than an expert model, established not only a baseline data set but also a process for continual improvement. With regard to the evaluation findings, feedback loops throughout the process helped commissioners and Extension faculty make decisions to address concerns identified through the survey responses. UFE aligns decision making with the results of a survey of intended users, and the UFE framework aids in facilitating decision making. In the case of our initial application of UFE in Ferry County, findings informed decision making related to both county governance and collaboration with Extension on countywide programming. For example, the process led to Extension's offering grant writing workshops and obtaining funding for 4-H newsletters. In other words, an engagement model coupled with UFE coupled helped align the county's annual strategic planning and budgeting processes with Extension's needs and funding.

Implications

Although other evaluation methods were options for Extension professionals and commissioners, UFE provided a useful framework for exploring the needs, challenges, and opportunities of an economically distressed community. Through the engagement model we applied, a quasi-advisory board and an environment of skillful experts were created, and feedback loops informed decisions. Kern (2018) emphasized that advisory boards and subject-matter expertise are essential to "economic development, good governance, and other concepts" ("Background" section, para. 1). The literature on strengthening the connection between Extension professionals and core audiences is as relevant today as it has been in the past. Flourishing technology and better access to information suggest that Extension's relevance may be in question. Moreover, Houser, Denny, Reimer, Marquart-Pyatt, and Stuart (2018) revealed that declines in public funding and cutting-edge private sector advancements have contributed to low use of Extension for support with agriculture and crops. Obviously, demonstrating Extension's relevance and value, especially to funding stakeholders, is important work. Bull, Cote, Warner, and McKinnie (2004) noted that agriculture and rural programming should align with the needs of the public and those elected to serve them because "Extension is a living, evolving, market-driven organization that responds to society's changing needs" ("Conclusion" section, para. 1). Thus, UFE, applied via an engagement model, is a great tool for achieving collaboration that helps leaders use data for evidence-based decisions and organizational change and further demonstrates Extension's value.

Using UFE in Ferry County helped commissioners bring county employees into the decision-making process and created an informed environment leading to useful exchanges that resulted in organizational change over time. Underscoring the success of the endeavor, one county employee said, "The County Climate Survey [UFE] helped me see the need for changing or improving Ferry County services. I also realized that changes to our county policies needed to happen."

In describing a similar collaboration with the public that involved use of a different evaluation tool, Robinson and Shepard (2011) revealed that conducting a comprehensive needs assessment can be a valuable method for identifying priorities on which to focus programs or services, just as the collaborators in our case realized. For instance, once the needs of Ferry County were understood, the departments implemented new technology, partnered on projects, strategically shared resources, reorganized processes, and created new services. Additionally, the UFE process improved morale and services, according to the data. As important, by employing the engagement model, Extension benefited from the process and became a valued resource in countywide economic development, strategic planning, and budgeting. These outcomes reflect the suggestion by Robinson and Shepard (2011) that the intersection of needs assessment and programming ensures that collaboration supports cognizant decision making and that diverse voices (not just the loudest voices) are incorporated.

In the case of Ferry County, Extension professionals achieved success in shaping change and creating stronger collaborations while building consensus that revealed opportunities to evaluation participants. Use of the UFE framework resulted in an engaged, evidence-based approach to addressing limited budgets and resources. The overarching implications for Extension professionals are twofold. One, this kind of UFE work is an effective way for Extension to demonstrate its value and relevance to elected officials and counties. Two, the UFE framework can provide an engaging opportunity to offer relevant programming, which could assist with maintaining or improving Extension funding and resources.

Conclusion

From the Morrill Act to the Smith-Lever Act, it is clear that land-grant universities are charged with engaging communities to provide the best available research and science. According to Gavazzi and Gee (2018), it has been said that in pursuit of increased research, engagement, and teaching, land-grant universities should be fierce in leveraging the abilities of Extension to interface with communities. In keeping with the land-grant mission, university and Extension faculty are well equipped to engage resources throughout the university for achieving public benefit and teaching elected officials basic tenets of research and evaluation. As demonstrated in the case presented here, personnel in county Extension offices can work with university experts to engage leaders in their communities with valuable evaluation or research support. Ferry County Commissioner Nathan Davis explained Extension's usefulness in this way:

Because WSU is a tier one research institution, Extension personnel are known for surveys, evaluations, and research. We wanted all of our leadership and public servants to benefit from this collective institutional knowledge as we deal with the fact that our gold mine is closed. Better understanding the attitudes and perceptions of our employees is important as we move forward in fighting rural or economic decay and shrinking budgets, as well as the stresses that come with those issues at a County level.

Through the engagement model and UFE lens, an additional benefit the university brings to the table for local government is the ability to serve as a neutral, nonpartisan provider with significant experience working with multiple partners. University involvement brings academic and scientific rigor to bear on complex problems communities are facing. UFE has evolved from education evaluation into a powerful tool that can be used for change and leadership. For example, the literature revealed that the progression of basic evaluation to the UFE model has affected public–private partnerships and created change related to conservation education programming (Flowers, 2010), water quality assessment (Severtson, Baumann, & Shepard, 2004), quality of life or health services (Williams, 2010), medical orientation programs (Meyer & Meyer, 2000), court reform or balance of power (Henry, 2001), and Extension program evaluation (Shepard, 2002). What this means to community leaders and to local government officials is that by working with local Extension professionals, who are already familiar with the people, players, and politics in their communities, they can, as in the case described here, gain access to expertise from across the university. This access to university resources will assist leaders in better understanding the needs of their communities and efficiently obtaining scientific or evidence-based recommendations on how to meet those needs that affect organizational change.

References

Bull, N., Cote, L., Warner, P., & McKinnie, M. (2004). Is Extension relevant for the 21st century? Journal of Extension, 42(6), Article 6COM2. Available at: https://www.joe.org/joe/2004december/comm2.php

Flowers, A. B. (2010). Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program. Evaluation and Program Planning, 33(2), 165–171.

Frederick, A. L. (1998). Extension education opportunities with policymakers. Journal of Extension, 36(2), Article 2COM1. Available at: http://www.joe.org/joe/1998april/comm1.php

Gavazzi, S. M., & Gee, E. G. (2018). Land-grant universities for the future: Higher education for the public good. Baltimore, MD: Johns Hopkins University Press.

Hahn, A. J. (1990). Issues-oriented public policy education: A framework for integrating the process. Journal of Extension, 28(1), Article 1FEA3. Available at: http://www.joe.org/joe/1990spring/a3.php

Henry, G. (2001). How modern democracies are shaping evaluation and the emerging challenges for evaluation. American Journal of Evaluation, 22(3), 419–429.

Houser, M., Denny, R., Reimer, A., Marquart-Pyatt, S., & Stuart, D. (2018). Strategies for enhancing university Extension's role as an agricultural information source. Journal of Extension, 56(6), Article 6FEA5. Available at: https://joe.org/joe/2018october/a5.php

Kabes, D. E. (1991). Legislators' criteria for Extension funding. Journal of Extension, 29(4), Article 4RIB1. Available at: https://www.joe.org/joe/1991winter/rb1.php

Kalambokidis, L. (2004). Identifying the public value in Extension programs. Journal of Extension, 42(2), Article 2FEA1. Available at: https://www.joe.org/joe/2004april/a1.php

Kern, M. (2018). Developing and managing an advisory board to support Extension-based centers and other programs. Journal of Extension, 56(1), Article 1IAW1. Available at: https://www.joe.org/joe/2018february/iw1.php

King, D. (1993). Facing the image deficit. Journal of Extension, 31(3), Article 3TP1. Available at: http://www.joe.org/joe/1993fall/tp1.php

Meyer, R. M., & Meyer, M. C. (2000). Utilization-focused evaluation: Evaluating the effectiveness of a hospital nursing orientation program. Journal for Nurses in Professional Development, 16(5), 202–208.

Patton, M. Q. (1978). Utilization-focused evaluation. Beverly Hills, CA: Sage Publications.

Patton, M. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications.

Robinson, P., & Shepard, R. (2011). Outreach, applied research, and management needs for Wisconsin's Great Lakes freshwater estuaries: A Cooperative Extension needs assessment model. Journal of Extension, 49(1), Article 1FEA3. Available at: https://www.joe.org/joe/2011february/a3.php

Severtson, D., Baumann, L., & Shepard, R. (2004). A utilization-focused and theory-based evaluation of an arsenic well testing program. In Proceedings of the Best Education Practices (BEPs) Symposium for Water Outreach Professionals: Defining BEPs, Refining New Resources and Recommending Future Actions (pp. 2–4).

Shepard, R. (2002). Evaluating Extension-based water resource outreach programs: Are we meeting the challenge? Journal of Extension, 40(1), Article 1FEA3. Available at: https://www.joe.org/joe/2002february/a3.php

Vines, K. (2018). Exploration of engaged practice in Cooperative Extension and implications for higher education. Journal of Extension, 56(4), Article 4FEA1. Available at: https://joe.org/joe/2018august/a1.php

Williams, A. M. (2010). Evaluating Canada's Compassionate Care Benefit using a utilization-focused evaluation framework: Successful strategies and prerequisite conditions. Evaluation and Program Planning, 33(2), 91–97.