Is There a Solution?

Extension agents frequently observe desirable changes in clientele during programming, but report them as "generally positive outcomes," while decision makers who fund programs need documented evidence of specific program results. Documenting impact "is establishing with as much certainty as possible whether or not an intervention is producing its intended results."1

When a program's goals are social rather than economic, how can its impact be assessed? Do valid and reliable instruments exist for measuring variables in the social domain? What additional methods can be used to measure social outcomes?

One answer to these questions is combining case study and clinical assessment methods for program evaluation in areas such as basic living skills and nutrition education. Using both assessment methods can provide stronger evidence of program results than would either alone.

The complexity of managing a household can produce frustration and stress, creating an environment for potential child abuse or neglect. Parents also tend to rear their children as they were parented, without recognizing that these methods may be harmful.2 They lack the basic skills to support family members' development.

In 1984, a county Children's Services levy created nearly $2 million for programming to prevent child abuse and neglect in Butler County (southwestern Ohio). The home economics Extension agent secured funding for a Development of Living Skills (DLS) program to target parents that Department of Human Services caseworkers identified as "at risk," the goal being "to reduce or modify the cited family's dysfunction, which often leads to child abuse and neglect."3

Serving only parents referred by caseworkers, the program involved 67 clients in the first year. Following referral, the client, caseworker, and DLS instructor identified the needs of the family which were addressed through lessons taught in the home. Lessons included information on self-esteem, nutrition, child development, parenting, clothing, home management, decision making, and pre- and post-natal care. Of all topics, program instructors identified low self-esteem as the primary problem among clients referred to this program and thus emphasized it in the lessons.

Both a clinical and case-study approach were used to report the impact.

Clinical: The Rosenberg Self-Esteem Scale was administered at the second and last home visit of the DLS instructor. Some items on the scale are positive ("I feel I have a number of good qualities.")4 while other statements are negative ("I feel I do not have much to be proud of.").5 Clients respond by indicating whether they strongly agree, agree, disagree, or strongly disagree with the statements.

In the initial assessment, 73% of client responses to positive statements about self were "agree" or "strongly agree." In the exit assessment, the rate increased to 92%.

Case Study: Clients completed a 60-item behavioral checklist developed for this program. Examples of items in the checklist are shown in Table 1.

Using these data, the DLS instructor developed case-study profiles to identify client perceived needs. The following are examples of actual case-study profiles:

1. After completing DLS lessons in parenting, the client showed progress in listening to the children's needs and using play as an educational activity. On the first visit, this client was very negative because Children's Services had just taken custody of her child. She was sixteen years old with an eight-month-old child. Her husband was nineteen and didn't work with DLS services.
2. This client needed information to help her cope with a runaway teenager plus two other teenage daughters. After the lesson "Helping the Teen Accept Responsibility," the client reported a positive feeling about her ability to be a good parent. She realized her daughters were responsible for their behavior. Each lesson includes a series of questions or suggested activities for the client's follow-up. These have always been completed.
3. This client left her alcoholic husband who abused her and her children. She took the two children who were under the age of three to a crisis center. Initially, they only had the clothes on their backs and food from the Salvation Army. With help from the caseworker, the client now has an apartment and a limited income to provide for the family's needs. DLS services helped the client budget her money and plan for the nutritional needs of the family.
4. This client was a complete stranger to the kitchen. She said she used only packaged foods and ran out of food before the end of the month. After several nutrition lessons on shopping skills and basic cooking, the client was able to use her resources to provide the family with food without needing additional outside resources.

Results of this program were reported to the Department of Human Services in terms of scores on the Rosenberg Self-Esteem Scale and the case-study profiles. Clients provided Human Services caseworkers with positive feedback on their participation. Because of the documented accomplishments, funding was continued for another year.

A similar strategy for documenting program impact could be used in programs such as nutrition education.

Helping people learn food consumption practices that lead to optimal health is the overall goal of nutrition education. Recent research indicates that 5% to 20% of college-age women engage in bulimic activities (purging the body of food after overindulging).6 An appropriate program goal for targeting youth with such problems might be, "Participants will develop healthy patterns of food consumption." Those who are bulimic have negative self-esteem.7 Thus, a second appropriate goal could be, "Increasing self-esteem of participants."

Both a clinical and case-study approach can be used to assess outcomes of this programming.

Clinical: An instrument such as the California Test of Personality, Ego Development Scale, or The Offer Self-Image Questionnaire for Adolescents could be used to assess self-esteem. The instrument should be compatible with the program's objectives. The specific variables assessed by each of these instruments are listed in Table 2.

The clinical assessment would be conducted before programming and at some point afterwards to determine any change in self-esteem.

Case Study: Semi-structured interviews with questions on food consumption patterns would be conducted before programming and again at some point after to see if healthy patterns are now being used. A parent or roommate of participants who observe food consumption patterns could also be interviewed to verify participant responses.

Each interview would be analyzed for indicators of healthy patterns of food consumption established before the programming. These indicators would be used to develop profiles of behavior at entry and exit.

If indicators of both healthy food consumption and self-esteem increased, evidence of program impact would be stronger than either indicator alone would provide.

Using both a case-study and a clinical approach to identify program results can provide substantial evidence of program impact. Published instruments with established validity and reliability can be used to assess some program results, while interview data can be analyzed for indicators of program impact. Case-study profiles can be developed by using behavioral checklist data. These provide a rich description of participant behavior and may indicate program results beyond the original goals.

Goals in the social domain are the focus of a substantial amount of Extension programming, and decision makers need evidence of program impacts to justify funding.

1. Peter Rossi, Howard Freeman, and Sonia Wright, Evaluation: A Systematic Approach (Beverly Hills, California: Sage Publications, 1979).

2. Jean Giles-Sims, "A Longitudinal Study of Battered Children of Battered Wives," Family Relations, XXXIV (April 1985), 205-10.

3. Peggy H. Simmons, Development of Living Skills: 1985 Research Findings (Hamilton, Ohio: Butler County Cooperative Extension Service, 1985).

4. Morris Rosenberg, Concerning the Self (New York: Basic Books, Inc., 1979).

5. Ibid.

6. Betty G. Kirkley, "Bulimia: Clinical Characteristics, Development and Etiology," Journal of the American Dietetic Association, LXXXVI (April 1986), 468-72.

7. R. C. Hawkins and P. F. Clements, "Binge Eating: Measurement of Problems and Conceptual Model," in R. C. Hawkins, N. S. Fremouw and P. F. Clements, The Binge-Purge Syndrome: Diagnosis, Treatment, and Research (New York: Springer Publishing Co., 1984).

8. Oscar K. Buros, ed., The Eighth Mental Measurement Yearbook, Vol. 2 (Highland Park, New Jersey: Gryphon Press, 1978).