Evaluation: An Afterthought or an Integral Part of Program Development

During the past decade program evaluation has developed as a process distinct from educational research and has become a force for educational improvement (Worthen & Sanders, 1991). To be as effective as possible in the development and implementation of successful programs, administrators and other professionals must be aware that evaluation procedures start at the beginning of a program's development. It is no longer enough to know that certain objectives were met. It is further suggested that experienced evaluators be involved in any given program's development from its very inception. This article describes an emerging framework for Extension program evaluation called program theory.

The underlying components and relationships upon which a program is based has been termed "program theory." Lipsey (1987) defines program theory as "a set of propositions regarding what goes on in the black box during the transformation of input into output" (p. 7). Black box evaluations are those which consider what goes into a program and what comes out of a program without considering what goes on inside a program. For example, when developing a program to enhance reading skills in children age 8- 10 years old, doing pre-test and post-test on their level of reading only demonstrates that something happened, not how or why it happened. Identifying processes occurring between input and output can be facilitated during the initial development of the program by asking questions such as: is the program developmentally appropriate for the targeted age-groups, or will a particular theory of learning be used to develop techniques? Making these and similar questions a part of the planning process can support replication of successful programs as well as assessing why a program did or did not work as expected.

Evaluation in educational programs, such as those found in 4-H and youth development, has long been a process that occurs after completion of a project. Given this context, its primary purpose is to assess whether or not specific objectives have been met. Several studies reported in the Journal of Extension exemplify this type of evaluation. For example, one recent study reported on a teacher training program. The authors clearly defined problems to be addressed in this program, the objectives of the program, and how these objectives would be measured (Turner & Travnichek, 1992). Another study evaluating stress and coping programs, reported its findings in much the same way (Fetsch, 1990). Both programs were reported as successful in meeting their objectives. What is not known is why they were successful. For example, what factors were leading contributors to making the program work? Was there a particular theory used as a basis for developing the program? If so, did the results support the theory?

Over the past decade there has been a move away from the "black box" approach to evaluation. Although the use of program theory in evaluation has steadily increased over the past decade, it is not in wide use in evaluation practice (Bickman, 1987).

For most program developers and implementors, the term "theory" has meant a few assumptions about why the program should work. The transition to theory-driven evaluations means an emphasis on the development and utilization of a more intricate framework that describes the basis of the program. This does not presuppose a single correct program theory, but suggests that different theories may be applied, depending on the discipline of the evaluator. For example, in a program designed to improve a 4-H program area, one evaluator may focus on the motivations of the program leaders and volunteers, whereas another may focus on the interrelations of the system as a whole, (i.e., visibility in the community, increased competencies in youth, or approval of advisory committees). The important issue is that the evaluation be theory-driven, thus allowing examination of the causative elements underlying the program itself.

The restructuring of evaluation offers an opportunity to assess what is "right" with a program and allows these findings to be generalized to other programs. This restructuring also shifts the focus from finding out "what went wrong" to "how did this work or not work and why." Measures must be developed to assess not only what goes into a program and what comes out, but what happened in the process. A program is not complete nor fully implemented until the process has been evaluated from beginning to end.

From this perspective, evaluators' activities take on a much broader focus than analyzing program outcomes. Scheirer (1987) explored the relation between program evaluation theory and implementation process theory. She concludes that program evaluation theory suggests what parts of the program should be measured to report the extent of implementation. That is the "what" of program delivery. In contrast, implementation process theory suggests the social system parts that should be explored to explain and manage the extent of implementation. That is the "how" and "why" of program delivery.

This shift in evaluation perspective also brings to light explicit and implicit objectives of program practitioners termed "theories-of-action" and "theories-of-use." Theories-of-action are explicit descriptions of how strategies and techniques produce outcomes. Theories-of-use are what practitioners actually do in the field. For example, a United States Department of Agriculture funded project in rural North Florida was developed, in part, to strengthen literacy skills. A major component of the program was a summer computer camp focused on second through fifth graders. The population that actually attended the camp ranged in age from four to 17 years. This occurred for a variety of reasons. Identifying these reasons is a vital part of assessing the viability and success of the program. The significant factor here is the dichotomy that exists between the espoused theories-of-action and what actually goes on in the program. Identifying both the theories-of-action and the theories-of-use is necessary in determining how a program works and helps bring together theory and practice (Kolb, 1992). One major implication illuminated by this dichotomy is the importance of discovering the implicit theories held by developers and implementors of the program being evaluated. Their expectations can effect their cooperation as well as their reactions to evaluation findings that do not fit these expectations.

How can programs be made better? Program theory serves several important functions that would enable practitioners to generalize from particular evaluations and achieve consensus in evaluation planning. Among those functions are: contributing to social science knowledge, assisting policymakers, discriminating between a program's success or failure and theory failure, identifying the problem and target group, finding unintended outcomes, specifying intervening variables, improving formative use of evaluation, clarifying measurement issues, and improving consensus formation (Bickman, 1987).

Citing Dunst et al., Miller (1991) stresses the importance of a solid theoretically derived program foundation. From this perspective, the implication is that a given theory will set the basis for program goals, implementation design, and expected outcomes. For instance, theories emphasizing social and emotional development will differ operationally from those emphasizing learning of specific skills or from those stressing intellectual development.

Some concerns and problems are identified with the program theory approach. First, there are barriers of added expense. Second, evaluators may be perceived as moving outside role boundaries and as threatening to other program providers. Third, if resources are to be used on a program, it is necessary to assess whether the goals and objectives were achieved and the resources "spent" appropriately thus making the providers "accountable" for the outcome of the program. Fourth, this approach may not provide a clear yes or no answer about the success of the program.

Conceptually, the questions answered during program evaluation must be incorporated into the way our programs are structured. For example, an objective may be to enhance self- esteem of youth. Changes in self-esteem can be measured without really knowing what process occurred. Practitioners need to know what it was that impacted the changes measured. Were activities based on specific developmental theory? Were the activities appropriate to the ages of the program participants? Was the measure of self-esteem used appropriate to the ages of the participants? What other factors may have impacted these changes? Answers to these and other questions provide insight into how programs can be developed, enhanced, and successfully replicated.

Bickman, L. (1987). The functions of program theory. In L. Bickman (Ed.), New directions for program evaluation, No. 33 (pp. 5-17). San Francisco: Jossey-Bass.

Fetsch, R. J. (1990). Prevention program impacts. Journal of Extension, Spring, 34-35.

Kolb, D. G. (1992). The practicality of theory. The Journal of Experiential Education, 15(2), 24-27.

Lipsey, M. W. (1987). Theory as a method: Small theories of treatment. Paper presented at the National Center for Health Services Research Conference, Tucson, AZ.

Miller, P. S. (1991). Linking theory to intervention practices with preschoolers and their families: Building program integrity. Journal of Early Intervention, 15(4), 315-325.

Scheirer, M. A. (1987). Program theory and implementation theory: Implications for evaluators. In L. Bickman (Ed.), New directions for program evaluation (pp. 59-75). San Francisco: Jossey-Bass.

Turner, J. & Travnichek, R. J. (1992). Measuring the success of teacher training. Journal of Extension, Winter, 38.

Worthen, B. R. & Sanders, J. R. (1991). The changing face of educational evaluation. Theory Into Practice, 30(1), 3-12.