Youth Participatory Evaluation: Matching 4-H Youth Experience to Program Theory

Introduction

A new theoretical model for the 4-H program was recently proposed (Arnold, 2018). The 4-H thriving model predicts that participation in high-quality 4-H programs increases youth thriving, and thriving youth, in turn, achieve key developmental outcomes. The model was pilot tested, with results that support its structure (Arnold & Gagnon, in press), and is undergoing refinement and additional testing (Arnold, 2019). Although the 4-H thriving model is grounded in current youth development literature, and initial model testing has been promising, youth voice had not been included in the refinement of the model. This article describes a youth participatory evaluation process through which we gathered data to support the validity of the 4-H thriving model.

Review of the Literature

The 4-H Thriving Model

The 4-H thriving model proposed by Arnold (2018) has three structures that describe and predict the effect of 4-H programs on youth development (Figure 1).

Figure 1.
The 4-H Thriving Model

The first structure is the 4-H developmental context, which is the setting and experiences provided by 4-H for youths. The developmental context is made up of three elements: (a) facilitating youths' sparks, (b) fostering developmental relationships, and (c) following principles for high-quality youth development programs. High-quality developmental contexts lead to youth thriving, which is the second structure of the model. Six indicators (e.g., possessing a growth mind-set and having a hopeful purpose) describe youth thriving (Search Institute, 2014). Thriving youth, in turn, achieve positive youth development outcomes (e.g., academic motivation and success), which constitute the third structure of the model (Arnold & Gagnon, in press). Statistical testing involving structural equation modeling (Kline, 2015) supported a full mediational model; youths who participate in 4-H programs that provide a high-quality developmental context thrive, and thriving youths achieve positive developmental outcomes (Arnold & Gagnon, in press). The 4-H thriving model articulates the way in which 4-H contributes to youth development, thus elucidating the process of youth development (Roth & Brooks-Gunn, 2016) and establishing a way to evaluate that process (Lerner, 2016). An important aspect of refining the model is the inclusion of youth voice in order to understand how the model matches youths' experience in the 4-H program.

Youth Participatory Evaluation

Youth participatory evaluation evolved from similar evaluation approaches used with adults and has gained in popularity, resulting in the development of best practices for engaging youths (Checkoway & Richards-Schuster, 2003; Flores, 2008; White, Shoffner, Johnson, Knowles, & Mills, 2012). The pragmatic purpose of participatory evaluation focuses on evaluation use through which programmers define and improve programs by engaging participants in program evaluation (Whitmore, 1998). For example, Fox and Cater (2011) described how engaging youths in participatory evaluation can help organizational development. Duke, Sollie, and Silva (2016) described the benefits to curriculum development that result from engaging youths in a participatory evaluation of a curriculum to identify the activities most salient to youths. In engaging youths in this way, the authors were able to reduce a curriculum's length, focus the curriculum on the most meaningful activities for youths, and enhance activities through the inclusion of active learning and youth-accessible language (Duke et al., 2016). Despite the potential power of engaging youths to shape the 4-H programs that serve them, there is a relative paucity of literature describing engagement of youths in 4-H program development.

Study Purpose

To begin this stage of the model refinement, we engaged 4-H youths in a participatory evaluation process to examine the impact of 4-H on their lives and asked them to reflect on how well their experience aligns with the model's six thriving indicators. Herein we describe the participatory evaluation process and its results. We sought to answer two evaluation questions:

1. How do 4-H youths describe the impact of 4-H on their lives?
2. How well do youths' descriptions of the impact align with the six thriving indicators of the 4-H thriving model?

Methods

Participants

We purposefully selected 24 adolescent 4-H members on the basis of their extended experience in the 4-H program to participate in the study. Extended experience was broadly defined as having been actively engaged in 4-H for at least 3 years and having had sufficient program engagement to be able to reflect on and articulate the program's impact on one's life. Purposeful selection is recommended in studies where a certain level of participant expertise is required (Creswell & Plano Clark, 2017). To broaden the scope of youth perspective beyond one county 4-H program, we involved youths from three contiguous, rural counties. Eight youths each from the three targeted counties participated. Participants were 19 females and 5 males and ranged in age from 13 to 17 years old. The ratio of females to males reflects the gender balance of the 4-H program in Oregon. We collected data in one 3-hr evening session. We provided dinner before the session and gave each youth a $10 Starbucks gift card for participating.

Participatory Procedure

We organized the youths in four groups of six to facilitate a tabletop graffiti activity (Arnold & Gifford, 2015) and gave each group a piece of poster paper with a question written on it. Providing square sticky notes, we instructed youths to write their answers to the question, one answer to a sticky note. The questions were broad enough that multiple answers per youth were expected to be generated. We instructed youths to work quickly and write as many answers as they could generate within 2 min. At the end of 2 min, the posters were rotated to the next table, and youths answered the second question. This process was repeated until each group had answered all the questions. The four questions were as follows:

1. How is 4-H making a difference in your life?
2. What has 4-H given you?
3. What things do you do (actions, decisions, etc.) as a direct result of being in 4-H?
4. What skills have you learned through 4-H?

The four questions were not intended to be mutually exclusive, but rather to present four different ways of helping youths think about the ways in which 4-H had affected their lives, thus potentially leading to multiple descriptions from the youths.

After completing the activity, each group took one of the posters and conducted a content analysis of the sticky notes, summarizing, categorizing, and labeling the responses. Each group then created a poster of the results and presented it to the other groups.

The second phase of the participatory procedure engaged youths in learning about the six indicators of youth thriving and assessing the alignment of their experience with the indicators. To facilitate this activity, we organized youths in six groups of four and provided each group a one-page summary of one of the thriving indicators. We asked the youths to spend a few moments reading the summary to learn about the indicator and then to engage in a small-group discussion of how they saw their experience in 4-H aligning with the indicator, supported by examples. The six groups then came together and shared their reflections on their assigned thriving indicators.

Analysis and Results

Following the data collection session, we gathered the posters with sticky notes to perform further analysis as a way to corroborate the youths' analysis. To do this, we first conducted our own content analysis of the original sticky notes for each question and categorized the data in largely the same categories as the youths had, making some minor modifications to labeling in order to enhance clarity and concision. For example, we combined several different specific skills the youths had articulated related to their 4-H projects into one category titled "4-H project and knowledge skills." In cases where a skill was identified prominently but was not specific to a particular 4-H project area, we gave that skill its own category (e.g., public speaking, record keeping). Doing this secondary coding after the youths' coding helped us both confirm the accuracy of the youths' analysis and further combine similar categories for usefulness of data presentation and interpretation.

The design of the study was qualitative, and exploratory in nature. Therefore, the appropriate statistical presentations are limited to descriptive rather than inferential statistics. By design, this approach limits the types of statistical analysis that occur. After categorizing the responses, for each question we calculated the frequency of responses associated with each category and expressed the result as a percentage of the total number of answers to the question. For example, for Question 1, of the 73 answers generated, 18% identified "providing new opportunities" as a way 4-H had made a difference in participants' lives. The top six categories for each question are presented in the Content Analysis subsection below. Although, admittedly, the percentages reported here are not large, they do align with the purpose of the study and serve to answer the research questions.

Content Analysis

As shown in Figure 2, youths reported that 4-H had made a difference in their lives by providing them new opportunities, especially for travel, facilitating friendships, supporting general life and social skill development, improving self-confidence, and promoting self-improvement.

Figure 2.
How 4-H Makes a Difference in Youths' Lives

As indicated by Figure 3, responses to the question of what 4-H had given the youths were similar to those for the question of how 4-H had made a difference in their lives. Through participation in 4-H, youths had gained friends and developed leadership and general life skills, self-confidence, grit, and social skills.

Figure 3.
What 4-H Gives Youths

In terms of actions resulting from participation in 4-H, youths reported volunteering, using knowledge gained in their 4-H projects, applying social skills, helping others, having a positive work ethic, leading others, and engaging in public speaking (Figure 4).

Figure 4.
Youth Actions as a Result of 4-H Participation

In the area of skill development, youths reported developing leadership skills, skills related to their project areas, social skills, public speaking skills, personal responsibility, and record keeping skills (Figure 5).

Figure 5.
Skills Learned Through 4-H Participation

Alignment of Thriving Indicators with Youth Impact Categories

The questions posed to the youths were not intended to be mutually exclusive; therefore, it was not surprising that similar categories emerged in the responses across the four questions. In all, 14 unique categories emerged. In the second step of the analysis, we aligned those categories with the six thriving indicators on the basis of face validity, which is the subjective judgment of how well a construct matches reality (Drost, 2011). Table 1 shows the alignment of the 14 categories with the thriving indicators.

Table 1.
Alignment of 4-H Impact Categories with Thriving Indicators

Thriving indicator	Indicator description	4-H impact category
Openness to challenge and discovery	Youths have the desire and ability to explore and try new things and challenges. Youths also possess a growth mind-set that supports effort in learning over innate ability.	4-H project knowledge and skills Grit New opportunities Public speaking
Hopeful purpose	Youths have a sense of hope and purpose and see themselves as being on the way to a happy and successful future.	Life skill development
Pro-social orientation	Youths see helping others as a personal responsibility and live up to the values of respect, responsibility, honesty, kindness, and generosity. Youths care about and give back to their communities.	Helping others Leading others Making friends Social skills Volunteering
Transcendent awareness	Youths are aware of a reality bigger than themselves from which meaning and purpose is derived. This awareness shapes everyday thoughts and actions.	Self-improvement
Emotional regulation	Youths are positive and optimistic and are able to manage emotions in ways that lead to health and well-being.	Self-confidence Social skills
Intentional self-regulation	Youths set goals and persevere in achieving their goals. They also make self-regulatory decisions that lead to better short- and long-term success.	Personal responsibility Positive work ethic Record keeping

Alignment of Youths' Lived Experience with the Thriving Indicators

The last part of the participatory session was a whole-group discussion in which we asked each of the six groups to share their assigned thriving indicator and assess whether they thought the indicator was a match with their 4-H experience. Each of the groups affirmed the match of the indicator with their experience, thus confirming the face validity of the thriving indicators. In addition, the groups provided illustrations of the thriving indicators in their lives. For example, one girl stated that 4-H had helped her "become a contented loser" (emotional regulation). Another described how she saw the opportunities for creativity that 4-H provides as facilitating transcendent awareness.

Conclusion and Implications

The study reported in this article was the first step in engaging youth voice to determine the validity of the 4-H thriving model for describing the impact of 4-H on youths. By using a youth participatory evaluation approach, we first established the impact 4-H has on youths from a youth perspective. The tabletop graffiti activity primed youths for the subsequent exploration of the thriving indicators and set the stage for assessing the match of the indicators with the youths' lived experience. The participatory evaluation method we used allowed us to accomplish the goal of orienting youths to the purpose of the study and eliciting their lived experience. This approach ensured that the assessment of the thriving indicator alignment was grounded in the lived experience of youths. Participatory evaluation methods such as the one we used in our study have potential across Extension for ensuring that program theory, development, and implementation match participants' needs and experience.

Establishing face validity is an important first step in determining the validity of constructs in the social sciences (Drost, 2011). The results of the study described in this article confirmed the face validity of the thriving indicators through the eyes of youth participants. The illustrations the youths shared of how an indicator was represented in their experience provided a new descriptive richness to the program theory that will be used in further refinement of the 4-H thriving model. The research reported here was a first step and sets the stage for additional research. Future research should include more rigorous and inferential designs to establish the general validity of the 4-H thriving model with youths from a diversity of backgrounds and program settings.

References

Arnold, M. E. (2018). From context to outcomes: A thriving model for 4-H youth development programs. Journal of Human Sciences and Extension, 6(1), 141–160.

Arnold, M. E. (2019). A revised model of youth thriving in 4-H youth development programs. Manuscript in preparation.

Arnold, M. E., & Gagnon, R. J. (in press). Illuminating the process of youth development: The mediating effect of thriving on youth development program outcomes. Journal of Human Sciences and Extension.

Arnold, M. E., & Gifford, L. (2015). YA4-H! Youth Advocates for Health—Youth participatory action research. Corvallis, OR: Oregon State University Public Health Extension.

Checkoway, B., & Richards-Schuster, K. (2003). Youth participation in community evaluation research. American Journal of Evaluation, 23(1), 21–33.

Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: Sage.

Drost, E. A. (2011). Validity and reliability in social science research. International Perspectives on Higher Education Research, 38(1), 105–124.

Duke, A. M., Sollie, D. L., & Silva, K. (2016). Using youth participatory evaluation to improve a bullying prevention program. Journal of Extension, 54(4), Article 4FEA4. Available at: https://www.joe.org/joe/2016august/a4.php

Flores, K. S. (2008). Youth participatory evaluation. San Francisco, CA: Jossey-Bass.

Fox, J., & Cater, M. (2011). Participatory evaluation: Factors to consider when involving youth. Journal of Extension, 46(2), Article 2TOT2. Available at: https://www.joe.org/joe/2011april/tt2.php

Kline, R. B. (2015). Principles and practices of structural equation modeling (4th ed.). New York, NY: Guilford Press.

Lerner, R. M. (2016). Evaluating programs aimed at promoting positive youth development: A relational development systems–based view. Applied Developmental Science, 20(3), 175–187.

Roth, J. L., & Brooks-Gunn, J. (2016). Evaluating youth development programs: Progress and promise. Applied Developmental Science, 20(3), 188–202.

Search Institute. (2014). The Search Institute model of thriving orientation: Overview of its evolution through recent funding from the Thrive Foundation for Youth. Unpublished paper. Minneapolis, MN: Author.

White, D. J., Shoffner, A., Johnson, K., Knowles, N., & Mills, M. (2012). Advancing positive youth development: Perspectives of youth as researchers and evaluators. Journal of Extension, 50(4), Article 4FEA4. Available at: https://www.joe.org/joe/2012august/a4.php

Whitmore, E. (1998). Understanding and practicing participatory evaluation (Editor's notes). New Directions for Evaluation, 80, 1–3.