Evaluating Adolescent Satisfaction of a 4-H Leadership Development Conference

Background

One of the hallmarks of the 4-H Youth Development Program has been an emphasis on youth leadership development (e.g., Radhakrishna & Doamekpor, 2009; Seevers & Dormody, 1995). The 4-H program has strived to provide youth opportunities to practice leadership skills through leadership conferences and meaningful youth leadership roles. Leadership development is embedded in a broader framework of positive youth development (PYD). Research in PYD suggests that engaging teenagers in leadership activities that promote increased agency, a sense of belonging, and development of competence promotes physical, intellectual, psychological, and social development (National Research Council and Institute of Medicine [NRCIM], 2002). Known as the ABC's of student experience, agency, belonging, and competence are outcomes of experiential education, a pedagogy employed by 4-H (see Carver, 1996). The ABC's have also been studied in the field of intrinsic motivation. Ryan and Deci (2000) identify autonomy, competence, and relatedness as important factors contributing to increased intrinsic motivation and promoting psychological well-being.

• Agency and autonomy are related constructs and refer to internal perceived locus of control (Ryan & Deci, 2000), support for efficacy and mattering (NRCIM, 2002), and being treated with respect (Schirmer, Weidenstedt, & Reich, 2013). The last point, the relationship between autonomy and respect, is of particular interest. Schirmer et al. (2013) argued that the ascribed agency of a person is the respect accorded to him/her by others. In this sense, the perceived respect accorded by another is an indicator of one's ascribed agency by that other.
• Belonging and relatedness are related to a sense of support, acceptance, and membership to a program or group (Anderson-Butcher & Conroy, 2002).
• Competence is typically defined as learning and mastery of skills and knowledge.

Satisfaction is another important consideration for youth programming, in addition to attending to the ABC's. When asked to rate their satisfaction, people rate and evaluate their experience based on a personal subjective standard, including expectations and perception of the benefits, individual characteristics, positivity bias, and cultural differences (Crowe et al., 2002). Youth who feel satisfied with their participation in youth leadership development activities may be more likely to continue to engage, hence learn, and be more apt to participate again in the future (e.g., Cassel, 1968). Even though satisfaction may be a predictor of participation and retention in learning activities, it is not a direct measure of youth leadership development or program effectiveness. In the study reported here, satisfaction was relevant because of its relationship with program retention and the desire to retain adolescence in programs that promote leadership development.

The study examined perceived agency, sense of belonging, and development of competence in relation to youth satisfaction from participation at California 4-H State Leadership Conferences 2005 through 2012. The conference brings together high school youth from across California in a 4-day leadership learning experience grounded in an atmosphere of positive youth development. The bulk of the planning is conducted by the 4-H State Ambassador team. Each year, eight-18 teenage Ambassadors and two-four adult 4-H volunteers are appointed to serve for 14 months planning the conference, including: emceeing plenary sessions; recruiting presenters; and ensuring the appropriate behavior of youth participants. The opportunity to plan a large conference is a tremendous developmental experience for the youth 4-H State Ambassadors. The 4-H State Ambassadors, peers-in-power, have an important role in being role models and demonstrating respect of their fellow conference youth delegates, peers-as-participants.

Evaluating the 4-H State Leadership Conference

With seven years of evaluation data, this study's goals were to identify trends in the following research questions:

1. How accurately can a linear combination of feeling respected, sense of belonging, and perceived competence predict youth satisfaction with a leadership conference?
2. What is the strength of the relationship between youth satisfaction and feelings of respect by youth, adults, and Ambassadors controlling for sense of belonging and perceived competence?

Each year, from 2005 through 2012 (excluding 2009), a scantron evaluation form and pencil were provided to participants at the last plenary session. Youth response rates ranged from 63% (lowest) to 83% (highest) (Table 1). No evaluation was collected in 2009 due to staff transition challenges. Data were analyzed using SPSS version 21. Cases with missing data were omitted from respective analyses.

The evaluation survey was created in 2003 by a team from the California State 4-H Office. Those involved in survey development had experience in assessing adolescent development or expertise in youth leadership development, which contributed to face validity. To ensure high response rates, it was decided to keep the survey to one page containing Likert-items. The survey was piloted in 2003 and 2004, modified, and then implemented in 2005. To compare results over time, the basic survey remained the same from 2005 through 2012, though a few questions were added in 2011 and the wording of three questions were slightly modified. Each survey contained 20-22 items; three-four demographic items, three-four open-ended questions, and 12-14 Likert-type items (using response options of Strongly Agree–Agree–Neutral–Disagree–Strongly Disagree).

The survey contained one composite dependent variable measuring overall satisfaction and three composite independent variables: a) feeling respected (an indicator of agency); b) sense of belonging; and c) perceived competence. These variables were created by summing the respective related Likert-items of each variable as outlined by Boone and Boone (2012), though with only two to three variables each instead of their recommended four. Reliability of the instrument was assessed with internal consistency using Cronbach Alpha. These values range from 0.59 (lowest) to 0.88 (highest) (Table 2).

• The satisfaction variable was composed of the three questions: "I am satisfied with the [year] 4-H State Leadership Conference"; "I want to attend the 4-H State Leadership Conference again"; and "I will recommend the State Leadership Conference to other 4-H members."
• The feelings of respect variable contained three items: "I was treated with respect by the youth delegates"; "I was treated with respect by the adult chaperones"; "I was treated with respect by the State Ambassadors and conference staff." In 2005-2007, the last question did not contain the words "State Ambassadors": it was added in 2008.
• The sense of belonging variable relied on one item in 2005-2010 and two items in 2011-2012. Reliability was low in 2011 but improved in 2012. The items included: "I met one or more 4-H members outside my county with whom I will keep in contact"; "I feel more connected to 4-H teens from across the state" (Added in 2011). Prior to 2011, the first question began with "I met at least three 4-H members …" but was changed to "one or more" in 2011.
• The perceived competence variable was created by summing two items in years 2005-2010 and three items in 2011-2012. These questions included: "I learned new facts and/or skills" (Added in 2011); "I gained information that I will be able to use to develop my personal leadership skills"; "I obtained new resources and/or materials to aid in my personal growth." Prior to 2011, the last question began with "I have new…" and not "I obtained new."

For all years, there existed a statistically significant correlation (p<0.01) between the dependent variable and each of the independent variables, ranging from a low of 0.217 with sense of belonging in 2010 and a high of 0.663 with competence in 2005. Correlations between the independent variables ranged from 0.157 (between respect and belonging in 2007) and 0.629 (between learning and belonging in 2012).

Findings

1. How accurately can a linear combination of sense of belonging, feeling respected, and perceived competence predict youth satisfaction with a leadership conference?

To address the first research question, ordinary least squares regression was employed to determine the relationship between satisfaction and the three dependent variables (Table 3). The assumptions of OLS regression were tested prior to analysis. Though there was a noticeable "ceiling effect" and excessive kurtosis all years, analysis proceeded because regression is robust to the violation of kurtosis and data from multiple years was available to identify consistencies in models. A sensitivity analysis was conducted, and 10 outliers were removed, total across years, which reduced the standard errors of the coefficients within each model, respectively.

In each year, the three independent variables were able to account for 38% (lowest in 2011) to 62% (highest in 2012) variance of participant's satisfaction. In all years, there were at least two statistically significant predictors of satisfaction: competence and feelings of respect. In 2011 and 2012, the third predictor, sense of belonging, was also statistically significant. The competence variable often saw the strongest effect on satisfaction prior to 2011. A one-point difference in competence was able to predict, on average, a 0.25 (lowest) to 0.73 (highest) point increase in satisfaction, holding constant feelings of respect and belonging. A one-point increase in satisfaction corresponded with a participant rating one of the satisfaction items an entire response higher (e.g., from "Neutral" to "Agree").

2. What is the strength of the relationship between youth satisfaction and feelings of respect by youth, adults, and Ambassadors controlling for sense of belonging and perceived competence?

To address the second research question, analysis of covariance (ANCOVA) was employed to determine the strength of the relationship between the dependent variable of youth satisfaction and three independent variables feelings of respect by 1) youth; 2) adults; and 3) Ambassadors using two covariates: sense of belonging and perceived competence. Due to unequal sample sizes in each of the five levels of ordinal data in the independent variables, each side of the response options were combined (e.g., Strongly Agree was merged with Agree), resulting in variables that each contained three levels (Agree–Neutral–Disagree).

The ANCOVA analysis demonstrated that in some years there was a relationship between satisfaction and one or more of the feeling respected variables (Table 4): by adults (in 2006 and 2007), youth peers (in 2006, 2008, and 2012), and youth 4-H State Ambassadors (in all years except 2005 and 2010). In years with statistically significant relationships, effect sizes (measured by eta squared) ranged from 0.035 (lowest) to 0.116 (highest). The State Ambassador respect variable was the most consistent in being significant, with an effect size ranging from 0.046 to 0.116. In addition, the covariate of competence was significantly related to satisfaction in all years, while sense of belonging was only significant in 2011 and 2012.

Discussion

The findings of the study reported here are consistent with the literature on the importance of agency/autonomy, belonging/relatedness, and competence for youth development, with respect to leadership development experiences (Carver, 1996; NRCIM, 2002; Ryan & Deci, 2000). Satisfaction of the 4-H State Leadership Conference was consistently predicted each year by youth who developed their competence and had been ascribed agency by being treated with respect. A sense of belonging was also a significant predictor of satisfaction in 2011 and 2012.

The findings in 2011 and 2012 may be more representative of the relationship among the four constructs, given the survey modifications in 2011, such as adding a question to the sense of belonging scale to improve its reliability. The findings demonstrated that satisfaction of the 2011 and 2012 4-H State Leadership Conference was predicted by youth who (in order of influence): 1) had a sense of belonging by feeling more connected to other 4-H teens and who had met one or more 4-H members outside their group; 2) had been ascribed agency by peers, adults, and Ambassadors by perceiving they had been treated with respect; and 3) perceived an increase in their competence through gaining information, resources, or skills.

A second important consideration is how youth in leadership roles interact with their peers. This 4-H leadership conference involved youth in governance and leadership, including instances where they often exerted authority over their peer participants. The 4-H State Ambassador team was charged with organizing a large multi-day conference that was often a new experience for these young people. As adult mentors, we must remember to prepare youth to not only plan a conference, but also to interact positively with their peer participants. As the results in the study revealed, the respect accorded by the youth 4-H State Ambassador team was significantly related to how youth conference participants rated their satisfaction of the event. The interactions between peers-in-power and peers-as-participants were often of even more significance in predicting satisfaction than the perceived respect of adult chaperones or fellow peer participants.

Several limitations constrain the findings of the study. First, the data presented here are of a correlational nature and do not imply causation. Second, there was a ceiling effect observed in the data evidenced by the negative skewness where data bunched at the high-end of the scale. Third, while survey space was limited, the reliability of the scales may have been increased by adding more items. Fourth, only 38-62% of the variance in satisfaction was explained, which means there were other factors influencing satisfaction not measured by this instrument. Finally, the use of satisfaction as the outcome variable was not intended to evaluate program effectiveness. Learning, leadership development, or youth development may be a better measure of a successful program. While there may be a relationship between satisfaction and these other constructs, these relationships may be mediated by other factors.

Recommendations

Youth who feel satisfied with their participation in an activity may be more likely to engage in that activity in the future. Ensuring 4-H leadership development programs are satisfying and provide opportunities for adolescents to strengthen their ABC's may help youth strengthen their leadership skills resulting in benefits to 4-H programs and the communities in which young people live.

As 4-H continues to provide leadership development opportunities, program planners need to:

• Ensure a welcoming atmosphere where youth feel a sense of belonging by ensuring that youth are treated with respect and that activities promote individually valued learning outcomes.
• Prepare youth leaders to treat their peers with respect at events where they serve in governance roles. Due to the potentials for power imbalance, these teen leaders must pay special care to treating their participant peers with respect.

References

Anderson-Butcher, D., & Conroy D.E. (2002). Factorial and criterion validity of scores of a measure of belonging in youth development programs. Educational and Psychological Measurement, 62(5), 857-876

Boone, H. N. & Boone, D. A. (2012). Analyzing Likert data. Journal of Extension [On-line], 50(2) Article 2TOT2. Available at: http://www.joe.org/joe/2012april/tt2.php 

Carver, R. (1996). Theory for practice: A framework for thinking about experiential education. Journal of Experiential Education, 19(1), 8-13.

Cassel, R. N. (1968). Facilitating student satisfaction in learning. Peabody Journal of Education, 45(4), 225-228.

Crowe, R., Gage, H., Hampson, S., Hart, J., Kimber, A., Storey, L., & Thomas, H. (2002). The measurement of satisfaction with healthcare: Implications for practice from a systematic review of the literature. Health Technology Assessment, 6(32), 1-244.

Huck, S. W. (2008). Reading statistics and research (Fifth Ed.). San Francisco: Pearson Education, Inc.

Gable R. K.,& Wolfe M. B. (1993) Instrument development in the affective domain: Measuring attitudes and values in corporate and school settings. Boston, MA: Kluwer Academic Publishers.

National Research Council and Institute of Medicine. (2002). Community programs to promote youth development. Washington, DC: National Academy Press.

Radhakrishna, R., & Doamekpor, P. (2009). Teaching leadership and communication skills and responsibilities: A comparison of 4-H and other youth organizations. Journal of Extension [On-line], 47(2) Article 2FEA6. Available at: http://www.joe.org/joe/2009april/a6.php

Ryan, R. & Deci, E. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68-78.

Schirmer, W., Weidenstedt, L., & Reich, W. (2013). Respect and agency: An empirical exploration. Current Sociology, 61(1), 57-75

Seevers, B. S., & Dormody, T. J. (1995). Leadership life skills development: Perceptions of senior 4-H youth. Journal of Extension [On-line], 33(4) Article 4RIB1. Available at: http://www.joe.org/joe/1995august/rb1.php