Using Multi-Site Methodology to Evaluate 4-H Youth Leadership Retreats

Introduction

Helping youth develop leadership and other life skills is one of the main emphases of the 4-H Youth Development program (Hendricks, 1996). Furthermore, there is evidence supporting the relationship between participation in 4-H and a young person's perceived leadership skill development (Boyd, Herring, & Biers, 1992; U.S. Department of Agriculture, 2001). These skills, which include critical thinking, problem solving, responsibility, public speaking, teamwork, accepting and valuing others, conflict resolution, cooperation, teaching, communication and, of course, leading, are skills that are essential to ensuring the success of young people, both now and in the years to come.

The 4-H Youth Development program believes in providing settings where youth can learn experientially through hands-on interaction with the subject matter. One of the ways in which youth have been invited to develop leadership and other life skills experientially is through youth leadership retreats. These retreats typically take place over 2 or 3 days, in a residential camp-like setting. The design of the retreats allows plenty of opportunity for hands-on learning about leadership skills. In many cases, the retreats are planned and led by older youth who gain real experience in planning, teamwork, responsibility, communicating, and teaching others.

Over the years the 4-H Youth Development program has struggled with the question of how to evaluate the success of its programs. One of the main factors involved in this struggle is that programs vary from county to county and state to state (Meyers, 1980). While individual local programs are often very successful, documenting that success in a rigorous and systemic way can be difficult due in part to the fact that local programs rarely have the number of participants needed to make any generalized statement about the program. One way to address this issue is to conduct multi-site evaluations, which is the process of conducting evaluations of the same program that is taking place at different geographical locations (Straw & Herrell, 2002).

Research Objectives

Local 4-H programs vary tremendously from location to location, but there are many programs that are similar in intent and design. The county fair, judging contests, and youth leadership training are just a few examples. The study discussed here was undertaken both to test the feasibility of conducting a large-scale, multi-site evaluation of a program that has common outcomes from site to site, as well as to gather aggregate program evaluation data to be used as a statewide program evaluation.

Methods and Procedures

In the fall of 2001, county 4-H faculty in Oregon who conduct youth leadership retreats were invited to participate in the multi-site study. As Straw and Herrell (2002) point out, there are several factors that must be considered when designing a multi-site evaluation. Of these factors, there were two key considerations for this study. First, given the variance in the educational program from site to site, how could we ensure that the same outcomes were intended? Second, what sort of coordination would be necessary to ensure that data collection was conducted the same way across sites?

The first factor was addressed through the use of logic modeling during a group training session prior to the evaluation ("Logic Model," 2002). At the training session, county faculty interested in participating in the evaluation were presented with a logic model for youth leadership retreats. The learning (short-term) outcomes were reviewed, discussed, and agreed upon. In this manner all participants left knowing what the outcomes for their retreat were to be and what modifications might be necessary in order to achieve the outcomes. County faculty were also given a worksheet to work through the inputs for their program to help determine the "logic" and plan for obtaining the multi-site outcomes in their local programs.

One benefit of presenting the logic model and short-term outcomes in the training session was that county faculty began to think more critically about the design of their programs. The exercise of completing a logic model in order to ascertain whether their program could meet the outcomes led faculty to see the places where their programs were missing important links between inputs, outputs, and outcomes. As a result, local programs were modified or enhanced in order to ensure that targeted outcomes could be met.

The second factor, consistency in data collection, was also addressed in the group training. A handbook with data-collection and data-entry procedures was presented and carefully reviewed with the group. In addition, because the data collection involved only a survey to be given at the end of the retreat, variance with data collection procedures was somewhat controlled.

Participants

Participants in the evaluation were 283 4-H youth in grades 7-12, from 20 of Oregon's 36 counties, including the Warm Springs Indian Reservation. There were 199 female and 82 male participants (with 2 missing cases). The youth participated in one of five different regional leadership events across the state (Table 1).

Table 1.
Number of Participants by Retreat Site

Retreat Site	Frequency	Percent
Eastern Oregon Leadership Retreat	96	33.92
High Desert Leadership Retreat	56	19.79
Mid-Columbia Leadership Retreat	58	20.49
Southwestern Oregon Junior Leadership Retreat	39	13.78
Douglas County Older Youth Retreat	34	12.01
Total	283	100.00

Program Outcomes

The following short-term learning outcomes were identified.

As a result of participating in the leadership retreat, youth would report an increase in knowledge and abilities in the following areas:

• Understand the responsibilities of being a leader
• Be prepared to take a leadership role at home, school, or in the community
• Know how to work as a team to achieve goals
• Learn personal responsibility for actions
• Learn how to involve others in shared leadership
• Have an opportunity to practice leadership skills
• Learn that there are important leadership roles to take right now
• Learn that being a leader is an important part of being an adult
• Understand that leadership is a skill that can develop over time
• Think about alternatives before making a decision
• Consider the consequences of making a decision
• Understand that leadership skills lead to success in life
• Feel more prepared for the future
• Feel good about self
• Value the contributions of others
• Understand the value of being friends with those different from one's self

Data Collection and Analysis

A standard survey instrument was designed for use with the retreats. At the end of each retreat, participants were asked to fill out the survey. Data were entered into spreadsheets at the county level and then sent with the hard copies of the surveys to the state 4-H office for analysis.

The survey employed a retrospective pre-test procedure for gathering self-reported levels of knowledge from participants (Pratt, McGuigan, & Katzev, 2000). Participants were asked to complete the survey questions regarding their level of knowledge on a scale of 1-5, both before attending the leadership retreat and after attending the leadership retreat. A paired t-test was used to test the significance of the difference in the group mean scores from before the retreat to after the retreat for each item.

In addition to the learning assessment questions, the survey also contained questions regarding the overall impact of the retreat and the 4-H program on the participant's leadership development.

Results

Mean scores revealed that participants reported a higher score after the retreat than before the retreat for all learning outcomes (Tables 2 & 3).

Table 2.
Mean Scores for Before and After

Short-Term Learning Outcome	N	Mean Before	Std. Dev.	Mean After	Std. Dev.
Understands the responsibilities of being a leader	283	3.80	0.98	4.51	0.67
Prepared to take a leadership role	283	3.69	1.07	4.39	0.79
Knows how to work as a team	282	4.05	0.78	4.65	0.58
Recognizes responsibility for own actions	281	4.36	0.82	4.74	0.59
Knows how to involve others	281	3.69	0.98	4.45	0.66
Has been able to practice leadership skills	281	3.63	1.07	4.43	0.78
Understands being a leader is important role now	282	3.92	1.01	4.61	0.67
Understands being a leader is important role as adult	282	4.05	1.00	4.67	0.64
Understands that leadership skills develop over time	282	3.97	0.94	4.67	0.60
Important to think about alternatives before making decisions	282	3.96	0.98	4.56	0.64
Important to consider consequences of decisions made	276	4.12	0.92	4.64	0.64
Developing leadership skills aids success	276	4.05	0.95	4.66	0.67
Feels prepared for the future	275	3.70	0.96	4.39	0.76
Feels challenged to do his or her best	273	3.84	0.97	4.46	0.75
Feels good about his or her self	275	4.07	0.96	4.53	0.77
Sees the value of others' contributions	274	3.96	0.84	4.53	0.66
Understands the value of having diverse friends	275	4.18	0.88	4.66	0.64

Table 3.
Difference in Mean Scores Before to After

Short-Term Learning Outcome	N	Mean Difference	Std. Dev.
Understands the responsibilities of being a leader	283	0.71	0.95
Prepared to take a leadership role	283	0.71	0.97
Knows how to work as a team	282	0.60	0.82
Recognizes responsibility for own actions	281	0.38	0.79
Knows how to involve others	281	0.76	0.95
Has been able to practice leadership skills	281	0.80	1.07
Understands being a leader is important role now	282	0.69	0.97
Understands being a leader is important role as adult	282	0.63	0.94
Understands that leadership skills develop over time	282	0.70	0.94
Important to think about alternatives before making decisions	282	0.60	0.99
Important to consider consequences of decisions made	276	0.53	0.91
Developing leadership skills aids success	276	0.62	0.96
Feels prepared for the future	275	0.69	0.93
Feels challenged to do his or her best	273	0.63	0.97
Feels good about his or her self	275	0.46	0.88
Sees the value of others' contributions	274	0.58	0.80
Understands the value of having diverse friends	275	0.48	0.87

In addition, 81.63% of participants reported a 4 or 5 on a 1-5 scale, indicating that the leadership training helped develop leadership skills. Similarly, 85.16% of respondents reported a 4 or 5 on a 1-5 scale, indicating that the 4-H experience has helped develop leadership skills.

Conclusions

In all, the results of the multi-site evaluation of 4-H youth leadership retreats indicate that participants are achieving the learning outcomes for the event. This evaluation focused solely on the assessment of learning outcomes and made no attempt to measure long-term impacts directly. There are a few reasons for this.

Reasons for Evaluation Focus

First, the evaluation was designed in line with the program logic model, meaning that medium- and long-term outcomes could not realistically be claimed at the end of a 2-day retreat. This isn't to say that medium- and long-term outcomes aren't possible, but rather, an effort was made to evaluate those things that could be attributed directly to the retreat experience itself.

Second, because so little evaluation has taken place on the youth leadership events in the past, it made sense to begin with the assessment of learning (Arnold, 2001). Once it is established that participants are indeed learning at the retreats, more sophisticated methods and designs can be used to understand better the different aspects of the impact of youth leadership retreats, including long-term impacts.

Finally, because this was a first statewide evaluation, the design and methodology was purposely kept simple in order to test the process and help ensure county participation. The relative simplicity of the evaluation should not discount the important confirmations revealed by the results.

Key Factors

Overall, the use of a multi-site evaluation methodology worked well. There appeared to be several key factors that made the methodology work in this setting. First, the commonality of goals and methods across the programs was key to obtaining results with any validity. Using logic modeling to accomplish this commonality was very useful and, as pointed out before, had beneficial side effects that ultimately helped improve the programs.

The second factor was the buy-in from the county faculty who participated. At the end of the evaluation, each county received a detailed evaluation report about its own leadership event. This information is highly valuable to county programs and thus contributed to the willingness to participate.

Finally, having a training session for all people involved in the evaluation prior to embarking on the project was quite useful. The training session allowed for talking about the philosophy of the evaluation, its goals, and its methods, and allowed time for a detailed question and answer period. Thus, all participants left with a clear understanding of what was to take place.

Implications for Future Research

The results of this multi-site evaluation lead us to believe with some confidence that participants in 4-H leadership retreats are learning about leadership and seeing its relevance and importance both now and in the future in continuing the development of important life skills. The understanding that the short-term outcomes for the programs are being met now invites a more rigorous and sophisticated assessment of the impact of the more nuanced and long-term outcomes of providing youth leadership development.

In addition, the potential for using multi-site evaluation methods in situations where it is appropriate to do so invites us to ponder other ways in which program evaluations might be carried out in situations that are inherently complex.

Acknowledgements

This project was made possible through the efforts of the following 4-H faculty members in Oregon: Janice Cowan, 4-H Faculty, Baker County; Dana Martin, 4-H Faculty, Crook County; Doug Hart, 4-H Faculty, Curry County; Robin Van Winkle, 4-H Faculty, Douglas County; JoAnn Mast, 4-H Faculty, Coos County; David White, 4-H Faculty, Deschutes County; Cindy Osterland, 4-H Faculty, Gilliam County, Elaine Husted, 4-H Faculty, Grant County, Shana Withee, 4-H Faculty, Harney County; Billie Stevens, 4-H Faculty, Hood River County; Anne Manlove, 4-H Faculty, Jackson County; Amy Davis, 4-H Faculty, Jefferson County; Erica Zollman, 4-H Faculty, Klamath County; Bill Broderick, 4-H Faculty, Morrow County; Carole Smith, 4-H Faculty, Union County; Deb Schreiber, 4-H Faculty, Wallowa County; Tonya Aitkin, 4-H Faculty, Wasco County; and Arlene Boileau, 4-H Faculty, Warm Springs Indian Reservation.

Thank you to all the county staff who assisted with data entry and to Blythe Kneedler, graduate research assistant, for her assistance in coordinating this statewide project.

References

Arnold, M. E. (2001). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line]. 40(3). Available at: http://www.joe.org/joe/2002june/a4.html

Boyd, B. L., Herring, D. R., & Briers, G. E. (1992). Developing life skills in youth. Journal of Extension [On-line]. 30(4). Available at: http://www.joe.org/joe/1992winter/a4.html

Henricks, P. A. (1996). Targeting life skills model. Ames, IA: Iowa State University Extension Service.

Logic Model. Madison, WI: University of Wisconsin Extension. Available at: http://www.uwex.edu/ces/pdande/index.html

Meyers, J. M. (1980). Integrity versus pragmatism: Planning the national 4-H evaluation. Education and Urban Society, 13(1) 13-36.

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective methodology. The American Journal of Evaluation, 21(3), 341-349.

Straw, R. B., & Herrell, J. M. (2002). A framework for understanding and improving multi-site evaluations. New Directions for Evaluation, 94, 5-15.

U. S. Department of Agriculture. (2001). Prepared and engaged youth: National 4-H Impact assessment Project. Washington, D. C.: Author.