Measuring Impacts with Young Audiences: Adapting a Life-Skills Instrument for Use with Third- to Fifth-Grade Youth

Capturing impacts of Extension programming on younger school-age audiences is often difficult and time consuming, yet staff are increasingly asked to document program effectiveness by Extension administrators, granting agencies, and policy makers. The limited literacy skills and concrete reasoning of young school-age children (Piaget, 2002) make the use of written evaluation instruments challenging. Observations and interviews, however, can be very time consuming and costly for programs. The project discussed here sought to develop a means to evaluate youth programming for third to fifth graders that was accurate, useful, ethical, and feasible (Joint Committee on Standards for Educational Evaluation, 1994).

The Washington State University (WSU) Life Skills Evaluation System (http://ext.wsu.edu/lifeskills/) is a Web-based system designed to allow Extension staff to create custom evaluation forms for their programs and generate summary reports on a local, county, and statewide level (Bailey & Deen, 2002). This system is useful in measuring short-term gains in life skills taught in many Extension youth and family programs. The system was developed and successfully tested for validity and reliability for youth and adults ages sixth grade and older.

Many Extension programs, however, are serving youth younger than sixth grade and need easy-to-use methods to measure program outcomes. This article outlines how we are working to adapt an older youth and adult version of the Life Skills System for younger audiences. Implications for others who are designing evaluation tools for young audiences are provided.

The WSU Life Skills Evaluation System was designed by and for Extension staff in the WSU Extension system. From the beginning of the project, a utilization approach to program evaluation was taken asking for participation and feedback from the users of the system, (Patton, 1997). Extension staff selected eight life skills from the Targeting Life Skills model (Hendricks, 1998) that they believed they were teaching through their program. These life skills were:

1. Decision Making,
2. Wise Use of Resources,
3. Communication,
4. Accepting Differences,
5. Leadership,
6. Useful/ Marketable Skills,
7. Healthy Lifestyle Choices, and
8. Self-Responsibility.

Although staff was excited about the new system, we quickly realized the need for a version that could be used with youth under the age of 11 years. We also realized that the version for younger children needed to assess the same skills as the current version because many Extension programs include multiple-age groups.

In meeting this challenge, we encountered several issues related to feasibility and accuracy in evaluation (Joint Committee on Standards for Educational Evaluation, 1994). The current system uses a retrospective pretest/posttest design (Pratt, McGuigan, & Katzev, 2000) to assist in making the evaluation process easier for busy Extension staff. However, because younger children rely more on concrete operations (Piaget, 2002) and may not readily conceptualize their former experience, a retrospective pretest/posttest was deemed to be an unreliable method for age group. Instead, a regular pretest/posttest design was used in which pretests need to be matched with posttests--a more time consuming method.

The second issue relates to accuracy. The instrument is a pencil and paper measurement, requiring reading and comprehension skills. For younger audiences, wording of questions and responses need to be adapted to the cognitive and literacy skills of this age group. Youth need to be able to make a link between program content and life skills taught. This is more difficult for youth in the concrete operational stage (Berk, 1999; Piaget, 2002).

Preparation and Implementation of Pilot

A 4-H Youth Camp, held in the northwest corner of Montana, provided the opportunity to develop and pilot test a Life Skills Evaluation System for younger audiences between the ages of eight and eleven. The study was deemed exempt by our university's Institutional Review Board Human Subjects' Committee. Passive parental consent was obtained through a notice sent to campers' parents.

The 31 indicators used to measure the eight life skills for the current Life Skills Evaluation System were adapted for a younger audience with a lower literacy level (see Appendix). This list was given to the camp director who met with the camp youth leaders. The contributions of youth leaders were instrumental to the planning and implementation of the program.

At the pre-camp retreat, the camp director introduced the process of teaching and evaluating life skills to counselors and worked with them in areas of leadership, communication, and child development. The current version of the Life Skills Evaluation System was used to measure the youth leaders' acquisition of life skills, while also increasing their familiarity with the test process. Following the pre-camp retreat, youth leaders planned camp activities that would help youth learn these skills.

The 4-H Youth Camp, held for 4 days during the summer of 2002, provided recreational activities that incorporated the learning of life skills. For example, in craft activities, campers were introduced to using resources wisely and the importance of recycling. Cabin and mealtime stressed self-responsibility.

Unlike the current Life Skills version, this version used a standard pre- and posttest that were administered by the camp counselors during group cabin time. Sixty-seven youth ages 9 to 13 attended the camp. There were 26 campers ages 9-10; 33 campers ages 11-12; and three campers age 13 years (Figure 1). The response rate for matched pretests and posttests was 97% or 65 of the 67 youth attending the camp. Nearly twice as many girls (n=40) attended the camp than boys (n=22).

Figure 1.
Age and Gender of Youth Campers

The 4-H camp is located in a small community with limited ethnic diversity; therefore, most of the respondents were White (n=39). However, small numbers of other ethnic groups were represented, including American Indian (n=9), Asian American/Pacific Islander (n=5), and those who are Racially Mixed or marked more than one response (n=17). Campers came from homes on farms or ranches, rural non-farm/ranch locations, towns under 50,000, and cities over 50,000 (Figure 2.)

Figure 2.
Ethnicity and Residence of Youth Campers

The Instrument and Testing Process

The Life Skills Evaluation System Version for Youth Ages Eight to Eleven was used to assess the life skills learned during the youth camp. The instrument was composed of 31 indicators, measuring the following eight life skills: Decision Making, Wise Use of Resources, Communication, Accepting Differences, Healthy Lifestyle Choices, and Self-Responsibility. A three-point scale of 1 to 3, with 1 being (never), 2 (sometimes), and 3 (usually), was used, because we determined that a four- or five-point scale might be too complex for younger youth.

Content and construct validity were assessed through a review of literature and expert feedback from Extension staff and specialists. The SAS statistical software package was used in analyzing these data. Using Cronbach's alpha, the instrument was tested for internal reliability, resulting in an alpha of 0.81. We were unable to conduct a factor analysis to determine subscales because a sample size of 100-200 is needed for this statistical test (Tabachnick & Fidell, 1989).

Although the goal of this pilot was to test the instrument reliability and validity, paired t-tests were conducted to assess gains from pretest to posttest. Statistically significant overall gains (p<.05) were made by campers in the life skills presented at camp. As measured by the pretest and posttest self reports, youth made gains in areas of Decision Making, Wise Use of Resources, Communication, Healthy Lifestyle, and Self-Responsibility. Although caution must be used in drawing conclusions given that the intervention was 4 days and there was no comparison group, the use of single-group pretest/posttests design can be supported if situational factors are taken into account (Eckert, 2000). In this case, the youth took the pretest upon arriving at camp, and the youth remained at the camp until the posttest was given.

Overall Patterns

Through this process a number of interesting results were obtained. While changes from pretest to posttest consistently noted an increase in life skills from never to sometimes and usually, this range of possible answers appeared to produce some confusion for this age group. One camper in the 9-to-10-age group replaced the word never with the word no on one of the choice headings, indicating a need for simplification and explanation of terms.

More concrete wording with more distinct choices and fewer nuances may alleviate possible sources of confusion. Possibilities could include a two-part response system of true or false; agree or disagree; a three-part system of no, maybe, yes; or a wider range of choices, such as not at all like me, somewhat like me, like me, and very much like me.

Also, it is important to note the interaction of indicators and possible responses. The use of double negatives, produced by the negatively worded indicator "I do not pick on kids who are different," along with the possible choice of never, was bewildering to these young readers. This pointed to a need for revision of this indicator to a clearly positive statement.

Mixed Results

On some indicators, the change from pretest to posttest did not demonstrate a uniform increase in life skills to the highest response of usually, but toward the center choice of sometimes, with a decrease in the two extreme responses. For instance, on the Decision Making indicator, "I am happy with choices I make," campers moved their preference from usually to the more self-reflective sometimes.

Similarly, there were interesting mixed results in the area of Accepting Differences. The ethnic composition of 63 responding camp members, according to self-reports on the pretest, included 16% Native American, 5% Asian American/ Pacific Islander, 19% Racially Mixed, and 60% White Caucasian backgrounds. On the three indicators regarding relations with children of a different skin color, while three quarters of the campers continued to indicate a preference for the usually option, numbers shifted slightly into the sometimes category in two instances.

First, on the indicator "I play with kids who have a different skin color than I do," campers' preferences decreased in the usually response and increased in the sometimes response. Second, on the indicator "I have friends who have a different skin color than I do," campers' preferences shifted from never to sometimes, while usually remained a high, but unchanged, choice.

While no children with handicaps attended the camp, the four indicators regarding relating to children with handicaps demonstrated small to moderate gains in the usually response. On the indicator, "I play with kids who are handicapped," while the highest number of campers continued to choose the sometimes response, there was a gain from never to usually. On the indicator "I have friends who are handicapped," while the highest number continued to choose never, there was a similar upward shift from this response to usually.

Areas of Greatest Gains 

It appears that the greatest overall gains were made in life skills that could be practiced in camp. For instance, in the life skill Wise Use of Resources, campers showed increases on all indicators. The greatest gains were made on the indicator "I pick up litter when I see it lying around," which may reflect the immediacy of teaching such values in the camp setting. Similarly, in Decision Making, high gains were made on the indicator "I think about my choices before making a decision." In the area of Communication, campers generally made strong gains on all indicators, with the greatest gain being on the indicator "I apologize when I am wrong," closely followed by other communication indicators. This demonstrates the effectiveness of learning communication skills in the camp setting.

Demographics

Indicators pertaining to demographics, which were filled out on both tests, produced some variation from pretest to posttest, reflecting a need for refinement in the process. Counselors perceived that the younger campers, especially the 9-year olds, had difficulty with the demographic questions and needed assistance reading and responding to them. There were inconsistencies from pretest to posttest on ethnicity, current home, and even gender choices.

The need for further explanation of the demographic questions is demonstrated by the campers' choices on the questions regarding their current home and ethnicity. A number of respondents changed current home categories between tests, and four campers chose several categories on either one or both of the tests to creatively describe their place of residence. One person added a written comment that she presently had two homes, a logical response for a youth from divorced parents. Similarly, in addition to campers who changed ethnicity from pretest to posttest, some described themselves using two ethnicity categories. Such inconsistencies can be alleviated by double-checking that all possible choices are listed and given adequate explanation, surveying these variables on the pretest alone, and assistance to campers at the time of testing.

Open-Ended Questions

Campers also responded to open-ended questions with statements that reflected their learning of life skills through camp activities. Responses to the question, "The most important thing I learned from attending Multi-County 4-H Camp is . . . " included comments that demonstrated skill building in the areas of Wise Use of Resources, Self-Responsibility, Communication, and Accepting Differences. One camper wrote, "I learned how to be clean and tidy" and to "respect the camp ground," reflecting increased awareness of Wise Use of Resources. Another camper noted "I learned how to take care of my things," indicating gains in the area Self-Responsibility. Other campers stated, "I learned to think before I talk" and to "get along and be careful about what you say," demonstrating gains in Communication skills.

There were many comments on the importance of interpersonal relations and making new friends at camp. In the area of Accepting Differences, one camper wrote that she learned "to be yourself, get along with others and just have fun." One camper wrote, "I learned that sometimes in life there are things that you have to do and people that you have to get along with that you don't want to." Another stated, "I learned responsibility and how to be kind to people with differences." These voluntary comments demonstrate the success of imparting life skills in this particular setting.

Implications

There are several implications from this pilot study for Extension staff who are planning evaluations with younger audiences. An essential part of planning is the coordination of life skills to be taught with the indicators to be used in the evaluation process. Discussing the life skill indicators and the program content during program planning sets the stage for an increased linkage between the two. A three-step process has been an effective sequence to meet this goal:

• Introduction of life skills to be used to program staff
• Collaborative planning of corresponding activities
• Refining of indicators based on the planned program

Designing an evaluation instrument for this age group that is feasible to administer given the busy schedules of Extension staff is challenging. Our experience found that it is useful to:

• Use close-ended questions to reduce the time needed to respond to questions and for ease in calculating results;
• Use a few open-ended questions to capture anecdotal information;
• Formulate indicators and responses adapted to participants' cognitive level using age-appropriate language;
• Use indicators that describe actual experiences, which youth have had the opportunity to practice in the program (e.g., communication with peers, decision making during activities); and
• Avoid using double negatives.

The limited abstract reasoning skills of the younger age group, grades three to five (ages 8 to 11) (Piaget, 2002), require that separate pretests and posttests be administered before and after the program. Prior to the program, staff needs to be oriented to the method of administering the evaluations to participants. A mechanism for matching pretests and posttests for each participant can include:

• Numbers on youth participants' nametags,
• A master list of names and numbers to match pre and posttests,
• Color-coding of pretests and posttests into two separate groups to avoid confusion, and
• Designating a specific time for staff to administer pre- and posttests.

During the process of administering the tests, it is important to have leaders available to clarify any points of confusion that participants may have, minimizing their frustration and enhancing their ability to respond to indicators:

• Train staff on how to administer the evaluation;
• Orient the youth to the evaluation process (e.g., explaining that they should choose the answer that best describes the youth's situation, thoughts, or feelings.); and
• Use consistent probes to clarify questions youth have in answering questions.

Conclusion

Developing effective ways to evaluate Extension programming for youth under age 11 is difficult. Extension staff often do not have the time or evaluation skills to use complex data collection methods such as observations. This project sought to adapt an evaluation instrument designed for youth and adults sixth grade and older for use with younger audiences. Through the process we have learned the limitations in this data collection method, but have found that it can be reliable and useful for busy Extension staff.

We are continuing to refine the instrument and the data collection process through further testing. With careful planning and administration of a life skills evaluation system that is closely matched to program activities and cognitive skill level of youth, a feasible and reliable system, which measures program outcomes, can be developed for young audiences.

References

Bailey, S. J., & Deen, M. Y. (2002). Development of a web-based evaluation system: a tool for measuring life skills in youth and family programs. Family Relations, 51, 138-147.

Berk, L.  E. (1999). Infants, children, and adolescents. (3rd ed.). Needham Heights, MA: Allyn and Bacon.

Eckert, W.  A. (2000). Situational enhancement of design validity: The case of training evaluation at the World Bank Institute. American Journal of Evaluation, 21, 185-193.

Hendricks, P. (1998). Developing youth curriculum using the targeting life skills model: Incorporating developmentally appropriate learning opportunities to assess impact of life skill development (Tech. Rep. No. 4H-137A). Ames: Iowa State University Extension.

Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.

Piaget, J. (2002). The language and thought of the child. (Reprint of 3^rd ed., c1959). New York: Routledge.

Powell, M. F. (1985). A program of life-life skills training through interdisciplinary group process. Journal of Group Psychotherapy, Psychodrama, & Sociometry, 38, 23-34.

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.

Tabachnick, B. G., & Fidell, L. S. (1989). Using multivariate statistics. New York: HarperCollins.

Appendix: Life Skills Instrument
Third to Fifth Grade

DECISION MAKING	NEVER	SOMETIMES	USUALLY
I think about my choices before making a decision.	1	2	3
I think about what might happen because of my choice.	1	2	3
I am happy with the choices that I make.	1	2	3
When I need to, I ask an adult for help in making a decision.	1	2	3
WISE USE OF RESOURCES	NEVER	SOMETIMES	USUALLY
I recycle at home or at school.	1	2	3
When I get money, I save some of it.	1	2	3
I make time for doing my homework.	1	2	3
I clean my room without my parents reminding me.	1	2	3
I pick up litter when I see it lying around.	1	2	3
When making a project I only take the things I need.	1	2	3
COMMUNICATION	NEVER	SOMETIMES	USUALLY
I do not talk when others are talking.	1	2	3
I listen when someone is talking to me.	1	2	3
I tell people how I feel when they hurt my feelings.	1	2	3
I apologize when I am wrong.	1	2	3
I get along with people.	1	2	3
If I do not understand something I ask for help.	1	2	3
ACCEPTING DIFFERENCES	NEVER	SOMETIMES	USUALLY
I play with kids who have a different skin color than I do.	1	2	3
I have friends who have a different skin color than I do.	1	2	3
I would invite a kid who has a different skin color than I do over to my house to play.	1	2	3
I play with kids who are handicapped.	1	2	3
I have friends who are handicapped.	1	2	3
I would invite a kid with a handicap to my house to play.	1	2	3
I do not pick on kids who are different.	1	2	3
I would have a sleepover with a handicapped kid.	1	2	3
I let others play with me even if they aren't very good at the game.	1	2	3
HEALTHY LIFESTYLE CHOICES	NEVER	SOMETIMES	USUALLY
I choose healthy snacks like vegetables and fruits.	1	2	3
I wear a safety helmet when riding a bike, using a skateboard, or using roller blades.	1	2	3
I talk to someone when I am upset or scared.	1	2	3
I say "no" and don't do what others are doing if it looks dangerous to me.	1	2	3
I like to play outside everyday	1	2	3
SELF RESPONSIBILITY	NEVER	SOMETIMES	USUALLY
I follow a schedule for doing homework.	1	2	3
I watch less than three television programs every day.	1	2	3
I do what I say I am going to do.	1	2	3
I go to an adult and ask for help solving a problem.	1	2	3
I take care of my things.	1	2	3
I ask for permission before I use other people's things.	1	2	3
I wait for my turn when doing an activity.	1	2	3
THANK YOU!!!!!!!!!