Opening Doors: A Qualitative Evaluation of the Waterbury Youth Leadership Project

Parents, students, and government funding sources are beginning to recognize the growing need for effective, high-quality after school programming. Although the benefits of these programs are generally perceived by parents, students, facilitators, and school systems as being positive, they are often unevaluated, and many evaluations yield questionable results (Anderson & Sabatelli, 2002; Gottfredson, Gerstenblith, Soule, Womer, & Lu, 2004; Riggs & Greenberg, 2004)

This article describes the initial efforts to evaluate an inner city after-school youth program, after completing 3 years of operation. The program is designed to offer the following key components: (1) a safe setting, (2) supportive relationships with adults and peers, (3) a variety of challenging and stimulating activities, (4) skills training, and (5) a sense of meaningful involvement. These components have been found to characterize effective out-of-school or after-school, youth programs (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002; Eccles, 2005; Eccles, & Gootman, 2002; Gambone & Arbreton, 1997; Roth, Brooks-Gunn, Murray, & Foster, 1998).

Program Components

Background

In May 2001, The University of Connecticut Cooperative Extension System's 4-H Program, collaborating with Waterbury Youth Services, Inc., designed and implemented The Waterbury Youth Leadership Project (WYLP): Opening Doors. One of three local community projects of Connecticut's New Communities Project, funded through the USDA Children, Youth and Families at Risk (CYFAR) Initiative, this project is located in the city of Waterbury, one of Connecticut's poorest cities. The WYLP addresses all five of the key components outlined above and evolved as a natural extension of prior programs conducted by the New Haven County 4-H educator with this youth agency.

Safe Setting

Youth were provided a safe and supportive environment as they met in the Waterbury Youth Services building and on the adjacent Waterbury Campus of the University of Connecticut. During the summer program, meetings took place 4 days a week for 5 hours each day, while meetings were held 2 days a week for 1.5 hours each day during the school year. Research has shown that children and adolescents who have a safe, supportive environment to go to after school are less likely to experience legal problems, drug and alcohol problems, and social problems than peers who are alone after school, especially in low-income areas (Richardson et al., 1989; Schinke, Orlandi, & Cole, 1992; Vandell & Shumow, 1999).

Supportive Relationships

Program activities included academic components such as tutoring and homework assistance. In fact, satisfactory academic performance was one of the criteria for remaining in the program. A primary feature of the WYLP is the openness of relationships with participants and staff, and the encouragement of appropriate peer interaction. Supportive interactions with peers and adults were enhanced through opportunities to facilitate peer mediation activities or participate in individual and group counseling. Such relationships have been shown to encourage motivation, high expectations, standards for acceptable behavior, use of resources to help with problems, academic progress, and positive social and emotional development (Diversi & Mecham, 2005; Gottfredson et al., 2004; Posner & Vandell, 1994; Ross, Saavedra, Shur, Winters, & Felner, 1992).

Variety of Activities, Skills Training, and Meaningful Involvement

The programming also targeted specific areas such as leadership, citizenship, employability training, and computer skills. Participants' meaningful involvement was fostered through community service and citizenship projects. Many community service projects were aimed at helping abused and neglected children. This was done directly--through having holiday parties, running after-school activities, mentoring and providing day care--and indirectly--through assisting in benefit events, making food baskets for families, and participating in a variety of other fund-raisers.

Participants learned about local, state, and national government through 4-H programs. They interacted with legislators and other government representatives in Waterbury, Hartford, Connecticut, and Washington, DC. In discussion groups, the participants tackled issues such as discrimination, sexual orientation, race and gender, drugs, violence, death, prostitution, diversity, sexual and verbal abuse, peer pressure, stereotyping among the elderly, poverty, and careers. Youth were exposed to many cultures because of the diverse participant group. All of these activities have been shown to encourage educational attainment, identity development, social integration into adult society, and future civic involvement (Larson, 1994; Mahoney, Larson, & Eccles, 2005; Verba, Schlozman, & Brady, 1995).

Methods

Participants

For year 3 of the 5-year New Communities project, the participants consisted of 10 youth in grades 10 through 12. The mean age of participants was 17.2 years. Six of the participants were male, and four were female. Five identified themselves as Black/African American, four as Hispanic/Latino, and one identified as both Black and Hispanic. Eight participants reported receiving a free or reduced price lunch at school, indicating that they came from a low-income family. Nine of these participants had been in the program for at least 1 year, and one participant was a new member. Of these 10, the range for participation was 1 through 3 years.

Evaluation Strategies

For the first 3 years of the New Communities Project, the evaluation team attempted to use traditional quantitative measurements to assess the impact of the WYLP. Participants completed evaluation instruments including: Achievement Motivation, School Self Esteem, Empathy, Conflict Resolution (subscales of Self-Control and Cooperation), Peer Self-Esteem, and Self-Efficacy. With few exceptions, these instruments consistently failed to show significant improvements in any area between the fall pretest and spring posttest.

One exception was a significant one-way analysis of variance (ANOVA) performed on the self-control posttest measures, using the number of years in the program as a between-subjects factor. This analysis indicated that the longer participants had been in the program, the higher their self-control scores.

The evaluators hypothesized that the failure to find statistically significant changes in program participants was likely due to a very small sample size. This hypothesis was supported by positive anecdotal responses of participants and program staff about the program and by the portion of the posttest survey that assessed participants' satisfaction with the program. All 10 participants stated that the program had helped them, and eight participants indicated an interest in continuing their participation next year. The two participants who did not report interest in continuing were graduating from high school and therefore ineligible to participate.

Because of the positive anecdotal and survey responses of participants, a qualitative focus group was included as an additional evaluation component. The objective was to better understand the experiences of participants and the factors that had led to their positive assessments of the program. Such qualitative strategies are begi(Ansay, Perkins, & Nelson, 2004).

A member of the evaluation team who had been trained in qualitative methodology arranged a date to meet with the participants. The day of the focus group, only six participants (three male, three female) attended. The interview was an hour long and semi-structured: the interviewer created a plan to ask participants about their practical and emotional experiences in the program and their ideas about how the program has affected them personally, at school, and at home. The interview was audiotaped and transcribed for analysis.

Results

Content analysis of the transcription resulted in the identification of several themes. Participants indicated that the program helped them to develop new skills in the areas of public speaking, computer use, and schoolwork. Participants also reported increased self-confidence and interest in community involvement/citizenship, greater class participation, and greater clarity of career goals

When asked about their lives before entering the program, only one participant expressed an expectation that she would attend college. Others stated an expectation to drop out of school. Some participants stated that they had been living "day to day" or were "living on the street" with no expectation for positive outcomes. When asked about their current lives, two of the participants reported that they were graduating.

One planned to enter college in the fall of 2004, and the other planned to enlist in the Navy. The remaining participants also expressed ambitious career goals. Two wanted to pursue careers in medicine, one wished to be a veterinarian, and the final participant expressed plans to be a businessman. Several participants also reported that their relationships with parents and siblings had improved during the time they had been in the program.

Specifically relating to school, most participants stated that their grades had improved. One stated that she "used to have all Ds and Fs . . . now I get mostly As." Another theme respondents emphasized as a result of participating in the program was being more motivated to do well in school. Youth reported that they felt rewarded in the program, gaining increased privileges and approval of staff for trying hard at school. Additionally, several participants stated that they viewed themselves as role models for their peers both in school and in the community agency where their program met.

Support was another aspect of the program that was identified as being very important. Participants stated that they got a lot of support from their peers, which filled the role of the family at the outset of the program. They also identified positive relationships with staff members as important. One participant stated, "Coming here is like coming home." This sense of home has been found to characterize other successful urban, after-school programs (Hirsch, 2005).

Each individual was asked what aspect of the program was most important to him or her, what had been the biggest change, and what, if anything, he or she would change about the program. In regards to the most important aspect, participants identified leadership skills, becoming a better person, increased responsibility, social skills, problem solving, and increased confidence as the most important aspects. Group members seemed to resonate with the reports of their peers, especially in terms of self-confidence. One group member stated, "There's nothing we can't do."

In identifying their biggest personal change, members reported becoming more independent, more responsible, having a better attitude (defined as no longer being rude to others), learning to communicate clearly and confidently in English (native Spanish speaker), and gaining focus in life. Participants also emphasized the positive impact of various presentations they had made to professional and governmental organizations regarding their group. They reported having traveled to Washington, DC, Minneapolis, Minnesota, and other locations to advocate for youth and to share the benefits of their program with other youth-development organizations.

Finally, there was a divergence of opinion among the group regarding what participants would change about their program. Some stated that they thought the group should be bigger and therefore "more productive," while others stated that they liked the small group because it was intimate.

Discussion

The results of the focus group interview revealed that participants have been strongly and deeply affected positively by their participation in the Waterbury Youth Leadership Project. The participants stated that they call their program "Opening Doors" because they feel that it has opened many doors for them. Analyzing the themes that resulted from the interview reveals that many aspects of positive youth development are addressed (leadership, self-control, responsibility, etc). This seems to fit the lone significant quantitative result of self-control being higher for those participants who had been in the program for more years.

When relying on self-report data for use in evaluation, the risk of a social desirability bias is ever-present. One advantage to the focus group format is that other members of the group can confirm or refute the statements of others. Indeed, throughout the focus group interview, participants continually affirmed each other and told specific stories that exemplified the issues they were discussing. Despite the apparent strength of the participants' testimonials, one must take caution in generalizing the results of this evaluation.

Is there something unique about these particular participants that contributed to the success of their program? Will the entry of new participants and the loss of graduating seniors have a negative affect on the group process? These are issues to be addressed in the next round of evaluation.

Implications

Beyond providing a positive evaluation of the efficacy of the WYLP, this study also contributes to the growing body of research supporting qualitative methods of evaluation research. In the face of poor quantitative results, the focus group allowed the evaluators to access benefits that participants derived from their program that may be immeasurable through quantitative research. Ansay and colleagues (2004) recommend extending focus groups to include other community members who may be involved with the group, such as program staff and parents of participants. This strategy was not used in the current study largely due to time and availability constraints, but is recommended for future evaluation efforts.

Qualitative techniques such as the focus group interview described in this study can be effectively used to evaluate Extension programs in many areas. Any program with a relatively small participant group or with a participant group that does not respond well to pencil and paper surveying (due to reading level, language barriers, etc.) may be more responsive to qualitative inquiries. Extension professionals who are successfully using quantitative methods to evaluate their programs may also wish to include a qualitative aspect to their evaluation plan in order to capture the richness of participants' experiences.

Finally, Extension professionals without a research background may find qualitative results more meaningful. Indeed, it meant more to the professionals of the WYLP to hear that participants felt that "coming here is like coming home" than to hear that the p-value of the ANOVA for self-control by years experience was significant. By using both quantitative and qualitative methods of evaluation, Extension professionals can ensure that all stakeholders in their programs have a way of seeing the effects of the program in an accessible manner.

Acknowledgements

This material is based upon work supported by the Cooperative State Research, Education, and Extension Service, U.S. Department of Agriculture, under the Connecticut "New Communities Project" (2001- 2006), Award #2001-41520-01137. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the view of the U.S. Department of Agriculture.

The Center for Applied Research is a joint venture of the School of Family Studies and the Department of Extension at the University of Connecticut.

References

Anderson, S. A., & Sabatelli, R. M. (2002). Effectiveness of after school programs: An overview of relevant literature. Unpublished manuscript, University of Connecticut, School of Family Studies, Center for Applied Research.

Ansay, S. J., Perkins, D. F., & Nelson, J. (2004). Interpreting outcomes: Using focus groups in evaluation research. Family Relations, 53(3), 310-316.

Catalano, R. F., Berglund, M. L.; Ryan, J., Lonczak, H. S., & Hawkins, D. (2002). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. Prevention and Treatment, 5, 15, 1-106.

Diversi, M., & Mecham, C. (2005). Latino(a) students and Caucasian mentors in a rural after-school program: Towards empowering adult-youth relationships. Journal of Community Psychology, 33(1), 31-40.

Eccles, J. (2005). The present and future of research on activity settings as developmental contexts (pp. 353-371). In J. L. Mahoney, R.W. Larson, & J.S. Eccles (Eds.), Organized activities as contexts of development. Majwah. NJ: Lawrence Erlbaum Associates.

Eccles, J., & Gootman, J. A. (2002). Community programs to promote youth development. Washington, D.C.: National Academy Press.

Gambone, M. A., & Arbreton, A. J. (1997). Safe havens: The contributions of youth organizations to healthy adolescent development. Philadelphia: Public/Private Ventures.

Gottfredson, D. C., Gerstenblith, S. A., Soule, D. A., Womer, S. C., & Lu, S. (2004). Do after school programs reduce delinquency? Prevention Science, 5(4), 253-266.

Hirsch, B. J. (2005). A place to call home: After-school programs for urban youth. Washington, DC: American Psychological Association.

Larson, R. (1994). Youth organizations, hobbies, and sports as developmental contexts. In R. K. Silbereisen & E. Todt (Eds.), Adolescence in context: The interplay of family, school, peers, and work adjustment. (pp. 45-65). New York/Berlin: Springer-Verlag.

Mahoney, J. L., Larson, R. W., & Eccles, J. S. (2005). Organized activities as contexts of development. Majwah. NJ: Lawrence Erlbaum Associates.

Posner, J. K., & Vandell, D. (1994). Low-income children's after-school care: Are there beneficial effects of after-school programs? Child Development, 65, 440-456.

Richardson, J. L., Dwyer, K., McGuigan, K., Hansen, W. B., Dent, C., Johnson, C. A., et al. (1989). Substance abuse among eighth-grade students who take care of themselves after school. Pediatrics, 84, 556-566.

Riggs, N. R., & Greenberg, M. T. (2004). After-school youth development programs: A developmental-ecological model of current research. Clinical Child and Family Psychology Review, 7(3), 177-190.

Ross, J. G., Saavedra, P. J., Shur, G. H., Winters, F., & Felner, R. D. (1992). The effectiveness of an after-school program for primary grade latchkey students on precursors of substance abuse. Journal of Community Psychology (OSAP Special Issue), 22-38.

Roth, J., Brooks-Gunn, J., Murray, L., & Foster, W. (1998). Promoting healthy adolescents: A synthesis of youth development program evaluations. Journal of Research on Adolescence, 8, 423-459.

Schinke, S. P., Orlandi, M. A., & Cole, K. C. (1992). Boys and girls clubs in public housing developments: Prevention services for youth at risk. Journal of Community Psychology (OSAP Special Issue), 118-128.

Vandell, D., & Shumow, L. (1999). After-school child care programs. The Future of Children, 9(2), 64-80.

Verba, S., Schlozman, K. L., & Brady, H. E. (1995). Voice and equality: Civic volunteerism in American politics. Cambridge, MA: Harvard University Press.