Assessing Program Implementation: What It Is, Why It's Important, and How to Do It

Introduction

Extension educators have long known the importance and necessity of evaluating the impact of their programs and services (Rennekamp & Arnold, 2009). Accordingly, the steps and components of outcome evaluations are well documented in the Extension literature (e.g., Roucan-Kane, 2008). While outcome evaluations have become the norm across most Extension programs (e.g., McCann, Peterson, & Gold, 2009), improvement can be made in the realm of implementation evaluations. The topic of program implementation, though alluded to by some authors (Duttweiler & Dayton, 2009), has not received a full treatment, as defined in the evaluation literature, in Extension outlets. Therefore, the purpose of this article is to provide an overview of implementation evaluations and some application examples.

Although most organizations develop plans detailing how various services and programs should be conducted, the level of actual adherence to these plans varies greatly (Durlak & Wells, 1997). Without understanding the degree to which a program was implemented as originally planned, often referred to as "program integrity," it becomes difficult to suggest linkages between outcomes and programs. Evaluating program integrity provides important information to multiple stakeholders, including evaluators, funders, and program staff and administrators (Rossi, Lipsey, & Freeman, 2004). Obtaining a clear picture of how well a program was implemented allows programmers to more confidently link programs to observed outcomes (Dobson & Cook, 1980). Additionally, implementation findings provide Extension educators insights into how their programs are being conducted and how they can be improved (Rossi et al., 2004).

Implementation evaluations, when combined with outcome evaluations, can also facilitate the identification of effective programs and practices. This information promotes the dissemination of evidence based programs as well as providing insights regarding how programs should be designed and implemented in order to produce observed positive results. Extension educators and general youth practitioners could greatly benefit from an increased focus on integrated evaluations that address both implementation and outcomes.

Review of the Literature

At the core of implementation is the concept of program integrity, defined as the degree to which a program is implemented as originally planned. Program integrity consists of five main dimensions: adherence, dosage, quality of delivery, participant responsiveness, and program differentiation (Dane & Schneider, 1998):

• Adherence refers to how closely program implementation matches operational expectations


• Dosage represents the amount of a provided service received by a participant


• Quality of Delivery deals with the manner in which the service was provided


• Participant Responsiveness measures individuals' engagement and involvement in the program (Domitrovich & Greenberg, 2000)


• Program Differentiation identifies program components in order to ascertain their unique contributions to the outcomes (Dusenbury, Brannigan, Falco, & Hansen, 2003)

Reasons for Studying Implementation

Evaluations too often focus solely on program outcomes without considering how the program and its components actually produced the observed results. In other words, understanding is gained regarding what happened as a result of the program without a clear picture of how it happened. This approach can lead to inaccurate claims about how program actually produced its observed outcomes (Chen, 1998).

Understanding whether or not a program was implemented correctly allows researchers to more accurately interpret the relationship between the program and observed outcomes (Durlak, 1998; Gresham & Gansle, 1993; Moncher & Prinz, 1991). Implementation research also helps researchers more accurately describe program components and their associated degree of program integrity, thus fostering more accurate replication of the intervention. Without a clear understanding of these issues difficulties can arise when replicating previously successful programs because practitioners will lack information regarding how best to implement the program and the degree of integrity needed to produce observed outcomes (Backer, Liberman, & Kuehnel, 1986).

To illustrate this point, consider the hypothetical example of a youth program designed to promote team building. The original developers of the program have experienced significant success at realizing targeted outcomes and have solid evaluation data to support their claims. Unfortunately, they did not collect implementation data and thus do not realize that a large portion of the program success is tied to the quality of youth and staff interactions, which are due to the organization's extensive, ongoing staff training program. Without understanding the role of staff training in the program's success, other organizations that attempt to replicate the program may not realize the same outcomes.

When implementation data is collected it allows researchers to more accurately determine the components of the program responsible for observed changes. Assessing implementation increases the quality of evaluation findings because it provides insights into how programs work and why they succeed or fail, rather than just focusing on outcomes (Chen, 1998; Domitrovich & Greenberg, 2000).

Existing research supports the case for assessing implementation when evaluating programs and interventions. Findings suggest that implementation influences program outcomes (Dane & Schneider, 1998; Dusenbury, et al., 2003). Although implementation is often overlooked in outcome focused studies, assessing program integrity offers important insights into why outcomes do or do not occur. Research has also shown that implementation varies widely across sites and change agents, meaning that a program implemented in multiple sites may experience varying degrees of success due to different degrees of program integrity (Durlak, 1998).

In addition to the aforementioned benefits of implementation research, gathering program integrity data can increase statistical power and promote dissemination. Moncher and Prinz (1991) suggest that since higher levels of program integrity may increase a program's probability of producing targeted outcomes, they also improve an evaluator's ability to detect change. Additionally, when researchers can accurately describe both program processes and outcomes, which they can if implementation data is gathered, it makes it easier to identify and disseminate information about programs that work (Domitrovich & Greenberg, 2000; Dusenbury et al., 2003).

Current State of Implementation Research

As noted, the study of implementation is essential to truly understanding program outcomes. Unfortunately, reviews of research from a number of different disciplines suggest that issues pertaining to implementation are often ignored. In one of the first reviews to address implementation, Peterson, Homer, and Wonderlich (1982) found that on average only 16% of the experimental studies published in the Journal of Applied Behavior Analysis between 1968 and 1980 that provided an operationalized definition of independent variables actually measured the degree of implementation of these variables. In another review of 181 studies of behavior interventions for young children, only 14.4% included assessments of treatment integrity (Gresham & Gansle, 1993). Similarly, only 18.5% of 479 learning disability intervention studies published from 1995 to 1999 measured implementation.

Those studies that do include measures of implementation integrity often report that this information contributes to the understanding of program performance and outcomes. For example, findings from studies of adolescent drug abuse prevention programs suggest that higher levels of implementation are related to increased program effectiveness and participant outcomes (Botvin, Baker, Dusenbury, Tortu, & Botvin, 1990; Dusenbury et al., 2003; Pentz et al., 1990). The findings from these studies highlight the need to evaluate program implementation in order to more fully understand program functioning and impacts.

Evaluating Program Implementation

In order to evaluate program implementation Extension educators need to develop an in-depth understanding of their programs. Potter Watts, and Preslar (2002) suggest that in order to accurately measure implementation, programmers need to focus on three key areas: program foundations, the implementation system, and program monitoring.

Many programs are based upon a theoretical foundation. Program theory explicates the processes whereby program components interact to produce desired outcomes (Scheirer, 1987). Although administrators and staff may possess an understanding of the theory underpinning their program, Extension educators need to work with program personnel to make sure everyone understands the reasons why a program is believed to work (Rossi et al., 2004). Quality programs, even those not based on specific theories, should be intentional enough that practitioners can explain what the program does and why they think it works. Once a clear understanding of program theory has been established it can be used to develop an implementation process theory, which outlines the mechanisms whereby program services are delivered to participants (Scheirer, 1987).

The implementation system simply represents the internal workings of a program, what they key components are and how they relate to each other (Potter et al., 2002). The implementation system and the intervention interact to produce program outcomes (Chen, 1998). The evaluation of implementation systems requires an understanding of factors both internal and external to the program (Dusenbury et al., 2003). These domains include characteristics of the participant, implementer, delivery mode, implementing organization, interorganizational relationships (e.g., coordination between multiple agencies), participants' external social contexts, and broader societal factors (Chen, 1998). Factors that may influence implementation system performance include implementer training, program characteristics (e.g., program complexity, availability of training manuals, etc.), implementer characteristics, and organizational characteristics (Dusenbury et al.). While the degree to which the implementation system is evaluated must be tempered by practical considerations such as time and money, the explanatory power of the evaluation will be increased by collecting data regarding as many of the implementation factors as possible (Potter et al., 2002).

After developing a clear understanding of a program's foundation (i.e., program theory) and implementation system, a strategy can be developed to measure integrity. This process is commonly known as program monitoring and is defined by Rossi, Lipsey, and Freeman (2004) as "the systematic documentation of aspects of program performance that are indicative of whether the program is functioning as intended" (p. 64). The success of any program monitoring plan is largely contingent upon the clear description of what data is being collected, how it is being collected, and by whom (Potter et al., 2002).

Keys to Effective Implementation Evaluations

There are a number of key steps Extension educators should consider when conducting implementation evaluations. While the general framework of implementation evaluations has been reviewed in preceding sections, the following serve as more specific recommendations. First, uniform operationalized definitions of the components of program integrity (e.g., adherence, dosage, etc.) need to be employed when studying implementation (Dusenbury et al., 2003). In other words, a quality implementation evaluation needs to collect data from as many of the core integrity domains as possible. This process allows for comparison of implementation research findings across programs and disciplines. Furthermore, all pertinent components of the program and implementation system need to be fully and clearly described in order to facilitate accurate effective measurement (Gresham & Gansle, 1993; Peterson et al., 1982).

One of the most important aspects of implementation research is the methods and measures used to actually collect the necessary data. Although multiple forms of data collection (e.g., observational, self-report, participant report) should be employed, consensus exists that observational data represent the most reliable method for assessing implementation (Durlak, 1998; Dusenbury, Brannigan, Hansen, Walsh, & Falco, 2005; Moncher & Prinz, 1991). Financial and other considerations may limit researchers' ability to conduct extensive site observations regarding implementation, and some experience suggests that telephone interviews with implementers may be an appropriate compromise (Scheirer, 1987). When both self-report and observation data are collected, they can be compared against each other to determine reliability (Dusenbury et al., 2003). Aside from the measurement methods and types of data collected it is also important to gather information from as many sources and regarding as many aspects of implementation integrity as possible (Dane & Schneider, 1998). To make full use of implementation data researchers need to always link these findings to program outcome data (Durlak, 1998). As noted earlier, research findings suggest that implementation impacts program outcomes in a variety of ways and investigating this relationship provides greater insights into program efficacy.

Implementation Evaluations: An Example

Consider a youth leadership development program that involves assigning youth to work with mentors from the community. In describing their program, staff mention that the program's underlying belief is that youth benefit from developing positive relationships with adult mentors. Therefore, the program's curriculum is focused on providing opportunities for youth and their mentors to interact in meaningful ways. The curriculum is provided to adult mentors who are expected to meet every week with their mentee and participate in the prescribed activity.

While leadership is the main outcome, staff descriptions of the program reveal a number of key implementation components that are summarized in Table 1. The table also identifies which implementation domains would be addressed by evaluating each component along with a potential observation method. The collection of the data identified in this brief implementation outline would provide program leaders a much clearer picture of both the operation and impact of their program.

Table 1.
An Example Implementation Evaluation Strategy

Evaluation Components	Observation Method	Implementation Domain
Use of curriculum during mentoring sessions	Observations of mentoring sessions and session logs completed by adult mentors	Adherence
Number of mentoring sessions	Sessions logs	Dosage
Quality of curriculum delivery and youth adult relationships	Observations of mentoring sessions and survey questions about relationship quality completed by youth and adults	Quality Of Delivery
Youth satisfaction and engagement in mentoring process	Interviews with youth and surveys with satisfaction and engagement questions	Participant Responsiveness

As is often the case with impact evaluations, the thought of developing an implementation evaluation strategy can seem, at least initially, quite imposing. Fortunately, when broken down, the steps are straightforward. The main requirement is that Extension educators have a comprehensive enough understanding of the program in question to be able to clearly describe key program components and processes. With this knowledge in place and using the steps outlined in this article, a strategy can be developed to gather data to assess key implementation domains such as adherence, dosage, quality of delivery, and participant satisfaction.

Summary

Implementation research is one of the most important, and at the same time most neglected, aspects of evaluation research. This is unfortunate due to the benefits related to quality implementation evaluations such as:

• Increased validity of outcome findings


• Better picture of how programs are implemented


• Greater understanding of program outcomes


• Improved dissemination of best practices


• More opportunities for making informed program improvements

Effective implementation evaluation requires Extension educators to clearly understand what a program is supposed to accomplish and how it should be put into practice. At its core, it simply is checking to make sure your program is running the way it was supposed to run. Together, program outcome and implementation data allow Extension educators to understand both what happened during their program (implementation) and the resulting effects (outcomes). This knowledge will empower educators to be able to better understand their current program offerings, improve future services, and ultimately better serve their target audiences.

References

Backer, T. E., Liberman, R. P., & Kuehnel, T. G. (1986). Dissemination and adoption of innovative psychosocial interventions. Journal of Consulting and Clinical Psychology, 54(1), 111-118.

Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E. M. (1990). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: results of a 3-year study. Journal of Consulting and Clinical Psychology, 58(4), 437-446.

Chen, C. (1998). Theory-driven evaluations. Advances in Educational Productivity, 7, 15-34.

Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18(1), 23-45.

Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3, 269-276.

Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11(2), 193-221.

Durlak, J. A. (1998). Why program implementation is important. Journal of Prevention and Intervention in the Community, 17(2), 5-18.

Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. American Journal of Community Psychology, 25(2), 115-152.

Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237-256.

Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20(3), 308-313. doi: 10.1093/her/cyg134

Duttweiler, M. W., & Dayton, S. F. (2009). Program integrity: A powerful organizing contstruct or just more jargon? Journal of Extension [On-line], 47(5) Article 5COM1. Available at: http://www.joe.org/joe/2009october/comm1.php

Gresham, F. M., & Gansle, K. A. (1993). Treatment integrity of school-based behavioral intervention studies: 1980-1990. School Psychology Review, 22(2), 254.

McCann, A., Peterson, J., & Gold, A. J. (2009). Using planning and evaluation tools to target Extension outputs & outcomes: The New England private well symposium example. Journal of Extension [On-line], 47(3) Article 3TOT4. Available at: http://www.joe.org/joe/2009june/tt4.php

Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11(3), 247-266.

Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. P., Dwyer, J. H., Johnson, C. A., . . . Cormack, C. (1990). Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review, 14(3), 264-289.

Peterson, L., Homer, A. L., & Wonderlich, S. A. (1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15, 477-492.

Potter, L. R., Watts, C. E., & Preslar, H. L. (2002). A conceptual framework for examining implementation issues. Unpublished manuscript.

Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation practice. Journal of Extension [On-line], 47(3) Article 3COM1. Available at: http://www.joe.org/joe/2009june/comm1.php

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage Publications, Inc.

Roucan-Kane, M. (2008). Key facts and key resources for program evaluation. Journal of Extension [On-line], 46(1) Article 1TOT2. Available at: http://www.joe.org/joe/2008february/tt2.php

Scheirer, M. A. (1987). Program theory and implementation theory: Implications for evaluators. New Directions for Program Evaluation, 1987(33), 59-76.