Identifying 4-H Camping Outcomes Using a Standardized Evaluation Process Across Multiple 4-H Educational Centers

Camping professionals and researchers have long since recognized that camp is more than a location or a program; it includes what happens to youth during and after the camping experience. Camp participation has been found to affect youth in multiple ways, enhancing many forms of growth, including:

• Affective (self-esteem and self-concept),
• Cognitive (knowledge, skills, abilities, and attitudes),
• Behavioral (self-reported behaviors and behavioral intentions),
• Physical,
• Social, and
• Spiritual (Shepard & Speelman, 1986; Gillett, Thomas, Skok, & McLaughlin, 1991; Hopkins & Putnam, 1993; Chenery, 1994; Brannan & Fullerton, 1999; Henderson, 1999).

While many youth service organizations provide residential camping, 4-H is one of the largest providers, with approximately 400,000 youth served annually.

In Virginia, camping is recognized as a primary 4-H delivery mode and as a proven and effective way of reaching boys and girls. The primary purpose of Virginia 4-H camping is to develop youths' life skills through exciting, hands-on programming as the instrument of positive youth development. Virginia has six regionally based 4-H educational centers that provide year-round 4-H camping programs and support local and regional 4-H efforts. While 4-H camping occurs at the local level, a majority of camping programs are conducted at the 4-H Centers.

With increasing emphasis on responsible youth program management, Extension educators and 4-H camp directors are expected to document the benefits of their 4-H youth development programs. Program evaluation is a way to:

• Establish baseline measures,
• Assess goals and objectives,
• Ascertain outcomes and impacts,
• Identify and document benefits,
• Explain success or failure, and
• Rationalize future action (Henderson & Bialeschki, 1995).

Unfortunately, the majority of evaluations conducted in camp settings have been internal process evaluations aimed at program improvement and identifying levels of participant satisfaction (Dworken, 2001). Based on logic modeling, Barkman and McKee (2001) defined these types of assessments as output evaluation, which focuses on describing activities and participation, including participant characteristics and their reactions and responses to a program. In contrast, outcome evaluation includes identifying the immediate learning, short-term action, and long-term impacts resulting from program participation.

Identifying the outcomes of statewide 4-H camping can be challenging. Many 4-H camping programs are conducted at sites over a wide geographical area and far away from their land-grant university. Barkman (2001) suggests that this

result[s] in miscommunication, leading to the utilization of a variety of methodologies and instruments to evaluate the same program. Because of this lack of consistency in methodologies and instruments, the data gathered from scattered sites are not comparable and cannot be summarized across sites" (p. 5).

One of the most effective ways to address the problems and inconsistencies of 4-H camp evaluation across multiple sites is through collaboration. Collaboration is characterized by the sharing of information and resources by specific agencies to achieve common goals (Meek, 1992). Collaboration allows agencies to achieve more together than they could accomplish individually.

Developing a Standardized Evaluation Process

Purpose

Although the Virginia 4-H program conducted a standardized camping outcome evaluation using the perceptions of parents/guardians of 4-H youth campers in 1995 and 1998, a standardized participant evaluation was not developed. Each 4-H Center used different methods, instruments, and procedures for collecting information from camping participants regarding the programs' outputs and outcomes. Over a period of months in the spring of 2001, State 4-H office faculty collaborated with the six 4-H Center program directors to address the inconsistencies in participant evaluations.

There were two major purposes for the collaboration. The first purpose was to develop a standardized evaluation process by:

• Generating buy-in through an open discussion of the benefits and limitations of a standardized process,
• Identifying the desired life skills targeted by the Virginia 4-H camping program,
• Creating standardized survey instruments that would provide consistency yet also allow program directors to add site-specific questions or items, and
• Clarifying the roles and responsibilities of the State 4-H Office and the 4-H Centers in the evaluation process.

The second purpose was to evaluate camping participants' outcomes using standardized instruments in order to:

• Identify 4-H camping benefits as perceived by youth campers and
• Identify 4-H camping benefits as perceived by youth campers' parents/guardians.

Collaboration

Through the collaboration of 4-H Center and university faculty and staff, two instruments were created, one for youth campers and another for counselors-in-training, teen counselors, adult volunteers, and participating Extension Agents. Both instruments included demographics questions, program ratings, and open-ended questions pertaining to participants' reactions (i.e., outputs).

The youth camper instrument also included a 10-item life-skill measure based on the "Targeting Life Skills Model" (TLS) (Hendricks, 1998). Life skills in five domains of the TLS Model were targeted by the developed instrument:

• Being,
• Relating,
• Caring,
• Thinking,
• Giving, and
• Working.

The measure identified youth camping participants' immediate learning changes (i.e., outcomes).

Finally, to address each 4-H Center's individual site concerns, a portion of the survey instrument was left blank, allowing each 4-H Center to include site-specific questions.

Evaluation Process

In implementing the evaluation process, each 4-H Center agreed to administer the two surveys at the end of each of their 5-day camps to all participating youth 4-H campers, counselors-in-training (CITs), teen counselors, adult leaders, and Extension agents (i.e., population sample). In addition, each 4-H Center agreed to enter data from the surveys on-site into a database. The State 4-H Office agreed to provide training to program directors regarding this evaluation process (i.e., collecting and compiling data; descriptive data analysis; reporting findings to the State 4-H Office), and to provide a summarized report to each 4-H Center at the end of the summer.

Each 4-H Center also agreed to continue a pre-established process of mailing a survey to the parents/guardians of randomly selected 4-H youth campers approximately 2 weeks after the completion of camp. The survey contained 24 questions based on the TLS Model (Hendricks, 1998) that assessed life skill behavior change by comparing behaviors before and after the 4-H camping experience across eight domains of the TLS Model:

• Being,
• Giving,
• Caring,
• Relating,
• Thinking,
• Managing, and
• Working.

The evaluation team assessed the validity of these life skill questions by considering the degree to which they reflected the life skills targeted by the Virginia 4-H camping experience.

Results

Youth Campers' Perceived 4-H Camping Benefits

While output and outcome data were collected, the results presented here focus on the outcome evaluation results of the youth camper and parent/guardian surveys. A total of 9296 surveys were distributed directly to 4-H youth campers at the six 4-H Centers. A total of 8118 surveys were returned and entered into the respective databases, for a response rate of 87%. Youth campers' ages ranged from 9 to 13 years old, with a mean age of 11 years old (S.D. = 1.36). The majority of youth campers were female (61%), with the most prominent ethnic groups represented being White (78.5%) and African-American (15%).

Ten questions on the survey asked campers to identify whether or not participating in 4-H camp had helped them acquire life skills on a scale of 1-4, where 1= "helped me very little" and 4= "helped me very much." The campers indicated that 4-H camp participation helped them most in making new friends (3.34), developing new skills (3.25), and becoming more independent and able to take care of themselves (3.06) (Table 1).

Table 1.
Perceived Benefits of Junior 4-H Camp Participation at a 4-H Center by Youth ages 9-13 Who Completed at Post-Camp Survey¹ (n= 8,118)

How has attending Junior 4-H Camp helped you? 2	Mean	S.D.
Make new friends	3.34	.923
Develop new skills in an area that I enjoy	3.25	.978
Be more independent and take care of myself	3.06	1.048
Learn more about different subjects	3.05	1.003
Develop closer friendships with people I already know	3.01	1.083
Improve my self-confidence	2.97	1.049
Improve my communication skills	2.83	1.064
Improve my leaderships skills	2.81	1.094
Become more responsible	2.70	1.072
See the world beyond my hometown	2.69	1.154
1 Cronbach alpha = .88 2 Scale of 1-4, where 1= helped me very little and 4= helped me very much.

Parent/Guardians' Perceived 4-H Camping Benefits

A total of 791 surveys were mailed to the parents/guardians of the 4-H youth campers. Of these, 363 were returned, for a response rate of 46%. Parents/guardians were asked to rate their child both before and after camp on 24 different life skill items on a scale of 1-5, where 1 = "almost never" and 5 = "almost always." Comparisons were made between each of the "before camp" and "after camp" means using paired t-tests. While significant differences were found (p<.05) between all of the "before camp" and "after camp" means, the most gains were found with the following items:

• "Takes care of his/her own things" (.37),
• "Shares work responsibilities" (.26),
• "Takes initiative and is a self-starter" (.24),
• "Takes responsibility for his/her own actions" (.23),
• "Handles success and failures" (.23),
• "Has a good mental attitude" (.23), and
• "Adapts to change" (Table 2).

Table 2.
Parents/Guardians' Life Skill Rankings of 4-H Youth Campers Before and After the 4-H Camp Experience¹ (n=363)

Life Skill Item	N size	Before2 Camp Mean	SD	After2 Camp Mean	SD	Gain3
Takes care of his/her own things	317	3.61	1.08	3.98	.95	.37
Shares work responsibilities	320	3.63	2.06	3.89	.88	.26
Takes initiative and is a self-starter	315	3.58	1.01	3.82	.93	.24
Takes responsibility for his/her own actions	319	3.72	.96	3.95	.83	.23
Handles success and failures	318	3.73	.87	3.96	.79	.23
Has a good mental attitude	318	4.10	.87	4.33	.75	.23
Adapts to change	317	3.89	.89	4.12	.80	.23
1 Cronbach alpha for pretest = .87; posttest = .94. 2 Parents were asked to rate their child's life skill behavior before and after the Junior 4-H Camp experience on a scale of 1-4, where 1 = almost never and 5 = almost always. 3 Paired t-test analysis found significant differences (p<.05) between all of the before and after camp mean comparisons.

Discussion

Benefits of 4-H Camping Participation

These results suggest that Virginia 4-H camping participation positively affects the development of immediate and short-term life skill behavior in youth campers. Specifically, youth campers self-reported that they benefited from 4-H camp participation by making new friends, developing new skills, and becoming more independent and capable of self-care. Parents/guardians thought their children had benefited from 4-H camp in taking care of their own things, sharing work responsibilities, and taking initiative on their own.

The primary benefit of 4-H camping participation supported by both youth and parent/guardian perceptions was related to responsibility. Participation in 4-H camp helped youth to become more responsible for themselves, more independent, and better able to take care of their personal belongings. Considering the nature of the camping experience, these results are not surprising. When youth are placed in a novel environment where they are expected to be responsible for themselves and have the support of caring teen counselors and adult leaders, such behavior change is a natural result.

While these results are encouraging and support the expectation that 4-H camp participation positively affects youth in many ways, the most important finding of this 4-H study is that a standardized evaluation process could be successfully implemented.

Benefits and Limitations of a Collaborative Standardized Evaluation Process

Collaboration was a powerful tool for Virginia Cooperative Extension programming. The value and utility of this cooperative effort involving 4-H Center/4-H camp staff and state-level faculty in developing an on-going evaluation process suggest that other Extension departments can do likewise. Lessons learned include the following.

• Reaching a consensus can take time. Therefore, collaborative planning for 4-H camp evaluation should begin well before the implementation of the program. This supports the Extension programming model whereby purposeful planning for program outcomes occurs prior to start of camp.

• In order for evaluation to be a mutually beneficial process, representatives from both the university and 4-H centers/4-H camps need to have the opportunity to provide shared input into the process and methods of evaluation. Communication and trust are critical at this stage.

• Survey design should allow each 4-H camp facility to add questions based upon individual needs. This flexibility encourages 4-H camp staff to care about the results and makes evaluation both relevant and responsive.

• When possible, other stakeholders, such as teen counselors, adult leaders, summer camp staff members, parents, etc., should be allowed to provide input into the evaluation process. Again, this helps to generate buy-in. Evaluation can be extremely time consuming and energy intensive. Support from these stakeholders is important, and recommended changes should be thoughtfully considered in during the subsequent planning for 4-H camp evaluation.

• Data should be collected from multiple sources (i.e., youth, leaders, and parents) to strengthen results and to explore different aspects of 4-H camping outcomes (i.e., immediate, short-term, and long-term impacts).

• Resources may limit what data collection can occur. Faculty and staff need to recognize these limitations and structure research designs that are practical with given resources.

Acknowledgements

This research would not have been possible without the combined efforts of Program Directors from each of Virginia's 4-H educational centers. The authors thank the following individuals for their contributions: Marvin Heimbach, Airfield 4-H Educational Center; Lance Johnson, Northern Virginia 4-H Educational Conference Center; Bryan Branch, Holiday Lake 4-H Educational Center; Becky Gilles, Smith Mountain Lake 4-H Educational Center; Chrystle Gates, Jamestown 4-H Educational Center; and Ruben Brown, Interim Program Director, Jamestown 4-H Educational Center.

References

Barkman, S. (2001, Fall). Four-fold: A research-based model designing and evaluting the impact of youth development programs. News & Views , 54(4), National Association of Extension 4-H Agents.

Barkman, S., & McKee, R. (2001, October). Measuring the impact of your youth and parent/families programs. Workshop presented at the National Association of Extension 4-H Agents Conference, Bismarck, ND.

Brannan, S., & Fullerton, A. (1999). Case studies reveal camper growth. Camping Magazine, January-February, 22-25.

Chenery, M. F. (1994). Explaining the value of camp. Camping Magazine, May-June, 20-25.

Dworken, B. (2001). Research reveals the assets of camp. Camping Magazine, September/October, 40-43.

Meek, J. (1992). How to build coalitions. Iowa State University Extension.

Gillett, D. P., Thomas, G. P., Skok, R. L., McLaughlin, T. F. (1991). The effects of wilderness camping and hiking on the self-concept and the environmental attitudes of knowledge of twelfth graders. Journal of Environmental Education, 21, 33-44.

Henderson, K., & Bialeschki, M. D. (1995). Evaluating leisure services: Making enlightened decisions. State College, PA: Venture Publishing, Inc.

Hendricks, P. A. (1998). Targeting Life Skills Model. Ames, IA: Iowa State University Extension.

Hopkins, D., & Putnum, R. (1993). Personal growth through adventure. London: David Fulton Publishing.

Meadows, Robert R. (1997) History of Virginia's 4-H Camping Program: A case study on events leading to the development of the 4-H Educational Centers. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University.

Shepard, C. L., & Speelman, L. R. (1986). Affecting environmental attitudes through outdoor education. Journal of Environmental Education, 17, 20-23.