From What to How: Targeting Specific Factors That Influence Outcomes

Background

Camp is an important delivery mode for Extension education and a powerful experience for children. Many studies have found that camp experiences produce positive developmental outcomes in youth (Henderson, Bialeschki, & James, 2007; Mishna, Michalski, & Cummings, 2001; Readdick & Schaller, 2005; Thurber, Scanlin, Scheuler, & Henderson, 2007; Yuen, Pedlar, & Mannell, 2005). Camp research conducted by 4-H/Extension (Arnold, Bourdeau, & Nagele, 2005; Baughman, Garst, & Fuhrman, 2009; Ferrari, & McNeely, 2007; Forsythe, Matysik, & Nelson, 2004; Garst & Bruce, 2003; Garton, Miltenberger, & Pruett, 2007) with further validation by the American Camp Association's National Outcomes Study (American Camp Association, 2005) provides ample empirical evidence that the camp experience contributes to healthy child development.

Benefits of Outcomes Evaluation for Extension Programming

Historically, many Extension professionals interested in measuring youth outcomes that result from camp experiences have been forced to use outcomes measures that were either untested or not specific to the camp setting. For example, camps have used self-esteem measures designed for school settings to measure how camp influences a child's sense of self. New resources have been needed to meet the increased expectations from funding agencies and university administrators who want systematic outcome evaluation documentation and even logic models that reflect intentional programming and evidence-based practices. To support this need, camp-specific resources such as the National 4-H Camping Research Consortium's National 4-H Camp Toolkit for Program Planning and Evaluation (2007) and the American Camp Association's Youth Outcomes Battery (2009) have been developed to support an outcomes-focused approach. State-level instruments have also emerged in conjunction with outcomes-focused camp evaluation projects.

The emphasis on outcomes has been important and productive. As a community of Extension and camp professionals, we have learned much about why camp is important for positive youth development by exploring youth outcomes. Outcomes-focused evaluation approaches, some guided by logic modeling, have improved the capacity to conduct evaluation in many states, with camp directors and Extension agents becoming much more aware of and interested in the outcomes of specific programs (Arnold, 2006). This outcomes-focused evaluation has allowed a progression of thinking from happenstance to intentionality.

Moving from What to How

But the time has come to go deeper—to ask not only "what" questions (i.e., "What changes in youth result from camp participation?), but also "how" questions (i.e., How are developmental impacts in campers facilitated?). Answering "how" questions require us to focus our attention on antecedents in addition to results. Questions appropriate for such an inquiry might include: "How do different staff behaviors influence the likely development of positive youth outcomes?"; "How do camp programs and activities lead to positive changes in youth?"; and "How do a camp's organizational structures and policies make positive youth outcomes more likely?"

To truly be intentional in our program planning, implementation, and evaluation, we must explore these antecedents. The impact of this shift could be profound. If implementation of curriculum X leads to specific short-term outcomes in children, shouldn't curriculum X be more widely used? If staff behaviors Y and Z make children more likely to make friends, shouldn't all staff be trained in those friend-making behaviors?

For camp administrators, program directors, and Extension educators, focusing on "how" questions is particularly relevant for the assessment of program quality and the development of promising practices, because these questions often hinge on point of service (such as the performance of camp staff) that can be controlled or influenced. Basing programmatic and administrative decisions based solely on "what" questions, such as those found on a camper self-report measure, can limit the information on which program design changes may be based.

A movement is taking place within youth development and experiential education to move from primarily documenting program outcomes toward more of a focus on the participant and program characteristics that contribute to those outcomes. This approach has been used in studies emerging in adventure program research based on a program theory model (Baldwin, Persing, & Magnuson, 2004) that includes the program activities, the intended outcomes, and the mechanism by which program activities are understood to lead to intended outcomes (Rogers, 2000). Sibthorp, Paisley, and Gookin (2007) used the theory-program-outcome model to study the relationship between participant characteristics, program components, and program outcomes for courses conducted by the National Outdoor Leadership School (NOLS). Program theory was also used by Hough and Brown (2009) to explore camper outcomes for youth with disabilities.

Implications for Methodology and Design

As we ask more rigorous questions about antecedents of camp outcomes, the methods of our inquiry will also have to evolve, which will provide new opportunities for both quantitative and qualitative studies. Along with descriptive statistics and analyses of variance, we may need to consider approaches such as structural equation modeling (SEM) and hierarchical linear modeling (HLM) as well as constant comparison and other text analyses for qualitative data to uncover camp components that influence youth outcomes.

Examples of this type of research can be found in Hickerson's (2009) study of factors that correlated with physical activity in camps. Qualitative research, specifically longitudinal case studies that evaluate camper outcomes over time in comparison with specific camp practices, can also help illuminate how change occurs in camps. For example, Bialeschki, Lyons, and Ewing (2005) conducted a non-experimental, cohort study that followed one group of 5th graders through 4 years of participation in the camp-school program Project Morry.

Emerging resources designed for camp professionals (and informed by camp research) may offer a guide for the use of different methodological approaches in camp research. Roark and Evens (2010) explore how specific experiences can be designed to elicit specific youth outcomes in Play it, Measure It: Experiences Designed to Elicit Specific Youth Outcomes. Observation-based tools that integrate observations of specific programs with director interviews, such as the Camp Program Quality Assessment (C-PQA), developed by the Weikart Center for Youth Program Quality and the American Camp Association (2010), may be helpful for camps seeking to do more than outcome measurement. Observation research may be useful in identifying unique program components or staff approaches and behaviors that are particularly influential on youths' experiences in camp.

Future Research and Evaluation

As opportunities arise to investigate the impacts and outcomes of the 4-H camp experience, we need to challenge ourselves to explore "how" questions and not just "what" questions. Undergraduate and graduate students have an important role to play in this exploration. Encourage students interested in the camp experience to consider the antecedents of change. Introduce students to research questions that explore organizational practices, program components, and staff/volunteer behaviors to better understand cause-and-effect relationships. A thorough study of the antecedents of positive youth outcomes will help us improve practice and may also help us better understand our return on investment of resources to the camp experience.

References

American Camping Association. (2005). Directions: Youth development outcomes of the camp experience. Retrieved April 20, 2010 from: http//www.acacamps.org/research/enhance/directions.php

American Camping Association. (2009). Camp youth outcomes battery—Measuring developmental outcomes in youth programs. Retrieved April 22, 2010 from: http//www.acacamps.org/members/outcomes/index.php/

American Camping Association and Weikart Center for Youth Program Quality (2010). Camp program quality assessment. Unpublished assessment.

Arnold, M. E. (2006). Developing evaluation capacity in Extension 4-H field faculty: A framework for success. American Journal of Evaluation, 27, 257-269.

Arnold, M., Bourdeau, V. D., & Nagele, J. (2005). Fun and friendship in the natural world: The impact of Oregon 4-H residential camping programs on girl and boy campers. Journal of Extension [On-line], 46(6) Article 6RIB1. Available at: http://www.joe.org/joe/2006december/rb1.php

Baldwin, C., Persing, J., & Magnuson, D. (2004). Role of theory, research, and evaluation in adventure education. Journal of Experiential Education, 26(3) 167-183.

Bialeschki, M. D., Lyons, K., & Ewing, D. (May, 2005). Youths' perspectives on camp outcomes: Intentionality and positive youth development. Paper presented at the Twelfth Canadian Conference on Leisure Research, Nanaimo, Vancouver, Canada.

Baughman, S. Garst, B. A., & Fuhrman, N. E. (2009). Consistency of developmental outcomes of 4-H camp experiences over time and across sites. Journal of Youth Development, 4(2).

Ferrari, T. M., & McNeely, N. N. (2007). Positive youth development: What's camp counseling got to do with it? Findings from a study of Ohio 4-H camp counselors. Journal of Extension [On-line], 45(2) Article 2RIB7. Available at: http://www.joe.org/joe/2007april/rb7.php

Forsythe, K., Matysik, R., & Nelson, K. (2004). Impact of the 4-H camp counseling experience. Madison: University of Wisconsin-Extension, Department of Youth Development.

Garst, B. A., & Bruce, F. A. (2003). Identifying 4-H camping outcomes using a standardized evaluation process across multiple 4-H educational centers. Journal of Extension [On-line], 41(3) Article 3RIB2. Available at: http://www.joe.org/joe/2003june/rb2.php

Garton, M. S., Miltenberger, M., & Pruett, B. (2007). Does 4-H camp influence life skill and leadership development? Journal of Extension [On-line], 45(4) Article 4FEA4. Available at: http://www.joe.org/joe/2007august/a4.php

Henderson, K. A., Bialeschki, M. D., & James, P. A. (2007). Overview of camp research. Child and Adolescent Clinics of North America. 16, 755-767.

Hickerson, B. D. (2009). Individual, social, physical environmental, and organizational correlates of children's summer camp-based physical activity. ETD etd-08112009-142955. Retrieved May 13, 2010 from: http//www.lib.ncsu.edu/theses/available/etd-08112009-142955/unrestricted/etd.pdf

Hough, M., & Browne, L. (2009, February). Connecting camp mechanisms to camper outcomes: A case for program theory. Paper presented at the 2009 American Camp Association National Conference, Orlando, Florida.

Mishna, F., Michalski, J., & Cummings, R. (2001). Camps as social work interventions: Returning to our roots. Social Work with Groups, 24(3/4), 153-171.

National 4-H Camping Research Consortium (2007). National 4-H camp tolkit for program planning and evaluation. Unpublished report.

Roark, M.F, & Evans, F. (2010). Play it, measure it: Experiences designed to elicit specific youth outcomes. Healthy Learning, Monterey, California.

Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In D.L. Stufflebeam, G.F. Madaus, & T. Kellaghan (Eds.), Evaluation models (pp. 209-232). Boston: Kluwer Academic.

Sibthorp, J., Paisley, K., & Gookin, J. (2007). Exploring participant development through adventure based programming: A model from the National Outdoor Leadership School. Leisure Sciences, 29(1), 1-18.

Thurber, C. A., Scanlin, M. M, Scheuler, L., & Henderson, K. A. (2007). Youth development outcomes of the camp experience: Evidence for multidimensional growth. Journal of Youth and Adolescence, 36, 241-254.

Yuen, F., Pedlar, A., & Mannell, R. C. (2005). Building community and social capital through children's leisure in the context of an international camp. Journal of Leisure Research, 37(4), 494-518.