Evaluating a Youth Leadership Life Skills Development Program

Introduction

The goal of leadership education programs is to teach leadership life skills using an experiential model. The participants learn valuable skills that are implemented within the program, moving the participant toward an active learning position (Kleinfeld & Shinkwin, 1983). However, experiential education is often more expensive than in-class educational activities, which puts pressure on programs to demonstrate effectiveness.

Boyd (1991) called for empirical research to identify effectiveness of programs and more specifically leadership training programs. Although some programs have demonstrated positive outcomes related to skill attainment, most evaluations are strictly pre-post samples plagued with an inability to address inflated pre-assessment scores (Hensel, 1991; Karnes, Merriweather, & D'Lilio, 1987; Seevers & Dormody, 1994).

Purpose and Objectives

The purpose of the study described here was to evaluate the development of leadership life skills in the participants in the Appalachian Regional Commission Youth Leadership Incubator Program, also known at the Youth Empowerment Program (YEP). This program seeks to develop the leadership life skills of each participant while also implementing a program, designed by the youth, to foster economic development within their home counties.

The goal is to effectively evaluate the program using a design that takes into account the standard shift in self evaluation that occurs following training. A common feature of leadership training is that many people come to the training with a high self concept of their abilities and thus report inflated self evaluations. The high ratings at pre-test artificially deflate post-test scores. However, providing hindsight scoring at post tests and follow-up allows for the participants to re-evaluate pre-test and even post-test knowledge given post-test and follow-up awareness.

It was hypothesized that the participants will have an overall rating increase from pre-test to post-test using the hindsight method. These leadership qualities will be maintained at 9-month follow-up.

Training Program

Adolescent participants were identified by school officials from seven economically distressed Alabama counties. Final participant selection was made by the YEP county steering committee members. The steering committee was also charged with overseeing the process of mentor selection, planning the retreats and the summit, and acting as county facilitators.

University students who were residents of the seven identified counties applied to work as mentors to the adolescent participants. These mentors primarily served as a bridge between the steering committee and the teens.

Adolescent participants attended an orientation and six monthly meetings. These meetings required each team to assess county needs and subsequently plan and implement a project aimed at addressing the identified problem. Projects included teen pregnancy prevention, park renovation, water treatment and improvement, and development of recreational facilities.

Two leadership life skills training retreats and a summit were also held over the span of the program. The two retreats provided 20 hours of instruction in leadership life skills, while the week long summit provided 30 hours of leadership training (Mecsko, 1996). At program completion, each teen had received 60 hours of leadership training.

Method

Measures

The level of leadership life skills possessed by the participants was measured with the 30-item self report, "Youth Leadership Life Skills Development Scale" (YLLSDS) (Seevers, Dormody, & Clason, 1995). Subjects rated their ability on each of the 30 items along a four-point Likert scale ranging from 0 (no ability) to 3 (a lot of ability). Example statements on the YLLSDS are: Can listen to others, can set goals, and consider the needs of others. Seevers et al. (1995) report a Cronbach's alpha reliability coefficient of .98; however, for this study the alpha was .89.

Participants were also requested to rate their level of involvement in 18 different types of school, community, and/or religious clubs or organizations. Involvement level was measured on a four-point Community Involvement Scale (CIS). Choices included non-member (18 points), some meeting attendance (36 points), attendance of most meetings, or committee membership or officer and attendance of most meetings (72 points). Although the CIS is non-standardized, the Cronbach's reliability coefficient demonstrates internal consistency of .93.

Analyses

A Repeated Measure General Linear Model, designed for analyzing multiple scores for the same subject, was used. The rationale behind using this particular model is that repeated measure analyses reduce the number of statistics, compared with ANOVA or correlation analyses. Self rating of leadership ability scores was assessed at pre-test, hindsight of pre-test, post-test, hindsight of post-test, and follow-up, on the YLLSDS, and on the CIS.

Providing hindsight scoring at post-test and follow-up allows the evaluation team to control for change in the participants shift in perception as to their personal ratings (Rohs & Langone, 1996).

Procedures

The YLLSDS was administered to participants at the first retreat prior to any leadership training. The same instrument was administered again at the close of the summit, asking participants to respond twice to each item. First they were asked to report how they perceived themselves to be at the time of administration (post-test), and second they were asked to report how they perceived themselves on the same item at the beginning of the program (hindsight of pre-test).

The hindsight method was carried out to control for change in the participant's response shift as described by Rohs and Langone (1996). The response shift is a common evaluative shift occurring when participants learn new skills and at post-test realize the limit of pre-test knowledge or ability before training. Follow-up with the YLLSDS was also conducted at 9-month post treatment, with participants providing 9-month follow-up data and a hindsight evaluation of post-test knowledge.

Results

Participants

Seven participants from each county were selected for a total of 49 youth. The ages of the participants at the beginning of the program ranged from 12 to 16 years of age, with a mean age between 14.4 years of age. There were 37 females and 12 males. The majority of participants were Caucasian (n= 35), while the remaining participants were African-American (n= 11) and Native American (n= 3).

There were 29 females and 10 males at post-test. The majority of the sample was Caucasian (n= 30), followed by African-American (n=7) and Native American (n= 2) participants. At follow-up, there 25 females and 8 males. The number of Caucasian participants fell to 27, while African-Americans dropped to four participants, and Native Americans remained at two.

Chi-square analysis was used to test for differences across demographic variables between those remaining in the study and those who dropped out at post-test and 9-month follow-up. There were no differences between drop-outs and non-dropouts across demographic variables; however, fewer than expected African American respondents completed follow-up 0² (2, 49) = 6.33, p = 0.04. T-tests demonstrated no significant differences between premature terminators and completers of the study on either of the YLLSDS or the CIS, even when taking into account racial identity.

Analyses

A repeated measures analysis was conducted with the dependent variable being leadership ability scores at pre-test, hindsight of pre-test, post-test, hindsight of post-test, and follow-up, on the YLLSDS . Wilks' lambda was used as the multivariate test of significance (Green, Salkind, & Akey, 1997). The results of the ANOVA indicated a significant time effect, Wilks' Λ= .47, F(4,28) = 7.879, p <.001, multivariate η²= .53. Paired samples t-tests indicated significant differences between five of the ten possible pairings.

Analyses of community involvement at pre-test, post-test, and follow-up, revealed no significant differences in the overall level of program involvement or overall number of leadership positions held by the participants. Examination of the data indicated a mean level of involvement of 36.47 at pre-test, 37.78 at post-test, and 36.72 at follow-up. Although personal rating of leadership changed over the course of the program, the level of community involvement remained the same.

Discussion

Without the hindsight approach, the program outcomes would not have been detected. This procedure should be used in future studies that require a self-evaluation of skills and ability. The initial hypothesis of this study, that the participants scores would be significantly higher at post-test than at pre-test, and that the change would be stable at 9-month follow-up, was supported using the hindsight approach. Response shifts of the participants, measured with a hindsight test at post-test and follow-up, indicate statistically significant differences in the participants' perception of change in their leadership ability.

Outcomes demonstrated that participants were confident of their leadership abilities but became cognizant of newly learned leadership qualities. These new qualities helped participants re-evaluate actual ability. It was interesting that the re-evaluation not only occurred between pre- and post-test, but also occurred between post-test and follow-up. It seems that new mental constructs of leadership occur over time and are followed by a new standard of evaluation.

The above finding is consistent with the results of Rohs and Langone's (1997) study, indicating a change in the participants' standard of measurement for level of leadership skill. Participants in this program not only perceive their ability to perform leadership functions in real life as being improved during the program, they also have a higher expectation as a leader. Additionally, the lack of a significant difference between the hindsight of post-test and the follow-up scores indicates that the participants see their abilities as relatively consistent at the 9-month follow-up period. The stability of this measure provides support for the hypothesis that leadership abilities would remain consistent the nine months following program completion.

Limitations

The methodological limitations include lack of a control group and relatively small sample size. High community involvement by participants prior to beginning the program made it difficult to assess any changes in level of activity. Including a group of adolescents who were minimally involved in the community prior to program participation would have provided more information concerning program effectiveness in changing future participant activity.

Future Research

Using the pre-test/post-test/hindsight format with a self report measure should be continued. Future evaluations including both quantitative and qualitative measures would further clarify the accuracy of the shift. Having an independent evaluation of the leadership qualities of each student at different stages of the project would provide supporting data.

Conclusion

Participants improved their leadership life skill ability and maintained these skills at follow-up. The participants also maintained previous levels of involvement in community activities. The hindsight tests allowed for evaluation of a change that would not have been detected in the traditional pre-test/post-test/follow-up evaluation format. However, more needs to be done to evaluate the accuracy of the response shift.

Acknowledgments

This program and evaluation was supported by the Appalachian Regional Commission, Contract, AL-12306-96. The project was a joint effort of the Auburn University Department of Human Development and Family Studies, the Alabama Cooperative Extension System, and the Auburn University Economic Development Institute.

References

Boyd, B. L. (1991). Analysis of 4-H participation and leadership life skill development in Texas 4-H club members. Unpublished doctoral dissertation, Texas A&M University, College Station.

Green, S. B., Salkind, N. J., & Akey, T. M. (1997). Using SPSS for windows. Upper Saddle River, NJ: Prentice Hall.

Hensel, N. H. (1991). Social leadership skills in young children. Roeper Review, 14(1), 4-6.

Karnes, F. A., Meriweather, S., & D'Llio, V. (1987). The effectiveness of the leadership studies program. Roeper Review, 9(4), 238-241.

Kleinfeld, J., & Shinkwin, A. (1983). Youth organizations as a third educational environment: Particularly for minority group youth. Final report to the National Institute of Education. Washington, D.C. NIE. September. (ERIC Document Reproduction Service No. ED 240 194).

Mecsko, L. (1996). Youth empowerment program: Resource handbook. (Available from Thomas A. Smith, Glanton House, Auburn University, Auburn, Al., 36849-5604)

Rohs, F. R., & Langone, C. A. (1996). Measuring leadership skills development: A comparison of methods. Paper presented at the Association of Leadership Educators, Burlington, Vermont.

Rohs, F. R., & Langone, C. A. (1997). Increased accuracy in measuring leadership impacts. The Journal of Leadership Studies, 4(1), 23-31.

Seevers, B., & Dormody, T. J. (1994). Predicting youth leadership skills development among senior 4-H members: A tri-state study. Journal of Agricultural Education, 35(3), 64-69.

Seevers, B., Dormody, T. J., & Clason D. L. (1995). Developing a scale to research and evaluate youth leadership life skills development. Journal of Agricultural Education, 36(2), 28-35.