Impact of a 4-H Youth Development Program on At-Risk Urban Teenagers

Introduction

Science literacy, as defined by the National Research Council (1996), is ''the knowledge and understanding of scientific concepts and processes required for personal decision making, participation in civic and cultural affairs, and economic productivity'' or understanding basic science skills. Science literacy in this context is more than just traditional science because technology and gardening programs can be important facets within the educational environments of schools. As noted by Goodman (2003), integration of technology has made a mark on school based education.

Further, there is a technology divide among low- and higher-income families (Wilhelm, Carmen, & Megan, 2002), implying the need to teach and increase access of technology to at-risk youth. Gardening, on the other hand, has been shown to provide opportunities to, among many others, learn science literacy (Fusco, 2001; Blair, 2013; McFarland, Glover, Waliczek, & Zajicek, 2013), decrease sedentary behavior (Phelps, Hermann, Parker, & Denney, 2010) and serve as a protective factor against youth delinquent and risk behavior while at the same time enhancing youth educational achievement and civic engagement (McFarland et al., 2013; Adedokun & Balschweid, 2009). In addition, research has found that more than 25% of youth do not have the skills to be prepared for work and that 21% do not have the skills to be workforce-ready (Fields, Brown, Piechocinski, & Wells, 2012; Cochran, Catchpole, Arnett, & Ferrari, 2010) and that 4-H programs should help close the workforce skills gap.

However, researchers have not yet systematically evaluated knowledge gains when at-risk urban teens are exposed to a combination of science literacy topics such as gardening, technology, and workforce readiness. This article serves to fill this gap in the literature while also providing a positive educational experience for at-risk urban teens. Further, we discuss the implications for gardening and science programs on youth development.

A 4-H Teen Urban Gardening Project was implemented in two urban cities: Bridgeport and Windham, Connecticut. The project consisted of three components: gardening, technology, and workforce readiness. Pre- and post-tests were administered to measure the before and after knowledge of teens who participated in gardening and technology programs. Further, a workforce-readiness component included an employer-employee simulation, where participating teens earned a stipend (money) based on points accumulated throughout the semester for work ethic (observed punctuality), communication (positive participation), and team work. Work ethic, communication, and team work have been identified as key workforce readiness elements (Cochran, et. al., 2010; O'Neil, Allred, & Baker, 2014). These soft skills, punctuality and attitude are commonly noted by employers as being lacking of high school and common graduates (Olsen, 2007; Palmer, 2012).

Purpose and Objectives

This article focuses on four main questions:

1. How much knowledge do at-risk urban teens gain when exposed to science literacy and technology programs?
2. Does past program participation (attending a program over subsequent years) influence knowledge gain among urban teens?
3. What differences in knowledge gains can be found when urban teens learn gardening and technology under the same project?
4. What is the impact of workforce readiness skills (punctuality and positive participation) have on knowledge gains?

Methods

The outreach program was conducted with two 5th to 8th grade (10-13 years old) at-risk urban teen groups participating in after-school programs devoted to urban gardening (such as gardening and composting) and technology. The program was conducted at two sites, one located at Barnum Elementary School in Bridgeport and another at Windham Middle School in Willimantic, Connecticut. Children within these schools meet many of the characteristics associated with at-risk, notably increased levels of poverty and household characteristics such as non-English speaking household and residential mobility (Robbins, Stagman, & Smith, 2012). Data were collected from students enrolled from summer 2012 through fall 2013. All participating teens qualified for school free lunch, which has been used as a poverty indicator or risk factor. Enrollment was on a first-come first-serve basis, and students needed to complete a registration package signed by their parents or legal guardian. Both sites taught the same content and used the same data collection instruments. Students met from 3:00 to 5:00 p.m. twice per week.

Program Design

The gardening and composting program were taught using the Experiential Learning Model by Certified Master Gardeners who used the Junior Master Gardener Curriculum. The gardening session within the program was offered in the summer (July-August) of 2012 and 2013. This program consisted of approximately 40 hours; 10 1-hour classroom lessons and about 30 hours of field activities. The program focused on identifying plants based on types of seeds and types of leaves, identifying soil particles (clay, silt and sand), naming edible parts of plants (flowers, roots, seeds, fruits, etc.), and classifying insects and non-insects.

The composting program, about 5-6 hours, took place during the fall of 2012. Composting was taught similarly to the summer gardening program, with both classroom and hands-on instruction. However, it took 5-6 hours to teach course content and build a composter. This program was based on the Rodale Book of Composting by Martin and Gershuny (1992). Students learned about identifying compost from non-compost soils, materials that can be used to make compost, recommended carbon – nitrogen ratios to make compost, introduction to vermiculture, and recommended containers to make compost.

Technology programs were also offered as a means to teach "job" market skills to students. They focused on digital video and videography and were taught during the spring 2013 and fall 2013, respectively. The technology programs were 10 hours in spring and 16 hours in the fall. These programs were adapted from the 4-H Filmmaker program developed by UConn Extension faculty. The content of this program focused on digital video production. Teens learned to write video scripts, record digital video, use green screens, and then edit digital video. Students worked in small groups to produce 3-5 minute videos. See number of students per program in Table 1.

Table 1.
Number of Students in Each Program

	Bridgeport	Windham
Program
Gardening 2012	10	18
Gardening 2013	13	16
Composting 2012	16	10
Digital Video 2013	6	14
Videography 2013	5	10
Total	50	68

Data Analysis Methods

In analyzing the data a combination of descriptive and statistical tests was used to gain more insights and better understand knowledge gains among urban teens. The first step was to examine pre- and post-tests to identify any differences (e.g., knowledge gains) associated with the gardening and technology programs. However, we also wanted to examine pre- and post-test scores for students that participated in multiple sessions. Looking only at statistical means potentially leaves valuable information out of the analysis. For instance, Cutz and Theuri (2011) found that gender and age are linked to different knowledge gains. So simply relying on mean program scores could potentially give biased implications about whether the program is working. Using a simple ordinary least squares regression to parse out the various effects is not appropriate given that some students attended more than one program and thereby their performance in one program could be correlated with their performance in another program. To account for students participating in multiple programs we can incorporate either fixed- or random-effects into the regression model. Using a Hausman test we found that the random-effects regression model was preferred (Hausman, 1978).

For model analyses we used a generalized least squares regression with random effects (GLS). (See Greene [2003] for more details.) The GLS with random effects was used because some students had participated in multiple programs over the course of their tenure at the school. Thereby, our data was structured as an unbalanced panel dataset. By using the GLS with random effects, we can control for any variation across students while examining several key variables of interest. Our dependent variable was the gain in knowledge (post- minus pre-test score) with our independent variables of interest being type of program (technology, composting, and gardening), number of past programs attended, and percent of points gained for being on time and behaving in class (total points gained for being on-time and behaving/total points offered during session for being on-time and behaving). As control variables we used gender, age, and a student's pre-test score. We also calculated robust standard errors to reduce bias from model misspecification. Based on the results from the above analysis, an interesting question arose, whether past program participation leads to gains on future pre-test scores. To answer this question we modeled pre-test scores as a dependent variable with the above mentioned explanatory variables.

Results and Discussion

Results from the mean comparisons indicated that students scored the highest pre-test score on the gardening program. Technology was close behind, with composting pre-test scores falling below the other program scores (Table 2). However, technology had the highest post-test score, with students answering 63% of questions correctly. Composting post-test scores were not as high as the technology scores, but the knowledge gain (as defined by the difference between post- and pre-test scores) was equivalent to the technology gain.

Table 2.
Knowledge Gains by Type of Program and Past Participation

	Pre-test score (% Correct)	Post-test score (% Correct)	Difference (post-pre) (% Correct)
Session
Gardening	32.9	49.7	16.8
Technology	25.2	63.0	37.8
Composting	13.9	49.0	35.1
# of Past Sessions
0	25.0	42.2	17.1
1	28.3	70.0	41.7
2	23.8	66.2	42.4
3	27.6	41.2	13.6
4	33.3	70.5	37.1

Examining the role of number of past programs attended, pre-test scores were equivalent across the number of past programs. However, the knowledge gain scores seem to indicate that students attending an increasing number of sessions had increased knowledge gains overall (Table 2). These results seem to provide support for our question that participating in past programs will enhance future knowledge gain.

Using only the mean comparisons presented in Table 2 could lead to misleading recommendations given that such factors as age, gender, etc., could be playing a role in knowledge gain. Using the GLS with random effects we were able to determine which factors impact knowledge gain. The next couple of result sections discuss how various demographic factors (descriptive statistics found in Table 3) affect knowledge gain. We first discuss the effect of demographics, program type, and pre-test scores. Subsequently, we delve into our two variables of interest, behavior/punctuality scores and past program participation. Based on the findings from the first model, we examine the impact of past program participation on pre-test scores in order to better gauge the effectiveness of the youth development program.

Table 3.
Descriptive Statistics for the Explanatory Variables

	Mean	Std. Dev.
Student Characteristics
Gender (% male)	38	0.49
Age	11.3	0.93
Type of Program
Gardening (%)	41	0.49
Technology (%)	32	0.47
Composting (%)	27	0.44
Other Factors
Percent of points for behavior /punctuality	66.3	26.27
# of past sessions	1.8	1.14
Pre-test score (%)	26	21.39

Role of Demographics, Program Type, and Pre-Test Scores

Examining the results of the GLS model (Table 4), we found that location (Bridgeport versus Windham), as well as age and gender did not affect overall knowledge gain. We also saw that students scoring highly on the pre-test did not experience the knowledge gains of students scoring lower on the pre-test. For instance, a 1% increase in the pre-test score resulted in a 1.3% decrease in the overall knowledge gain. This seems to indicate that students who had a better understanding of the material pre-program did not gain as much knowledge as students scoring lower on the pre-test. Several factors may have influenced these results, notably (1) students with more knowledge at the beginning did not work as hard on the material as those that were less familiar or (2) the curriculum was not advanced enough for students with increased a priori knowledge.

Table 4.
Results of the Random Effects Generalized Least Squares Regression

Variables a	Difference (post - pre) Model		Pre-Test Score Model
Constant	22.2		-13.8
Bridgeport	-6.2		-13.8	***
Gender (1=male)	6.5		2.3
Age	-0.2		4.6	*
Pre-test score	-1.3	***	--
Technology	14.1	*	-12.2	**
Composting	-10.5		-28.4	***
Percent of points for behavior/punctuality	0.5	***	0.0
# of past sessions	6.2	**	4.8	***
Model Diagnostic Statistics
R-square within group	0.66		0.55
R-square between groups	0.53		0.24
R-square overall	0.54		0.31
Wald Chi2	167.4		66.4
Prob > Chi2	0.000		0.000
Std. deviations of residuals within groups	13.8		5.4
Std. deviations or residuals of overall error term	26.2		16.9
Rho (intraclass correlation)	0.22		0.09
a The left-out categories for comparison are Windham, gardening, and female. For interpretation, the technology program had a 14.1% higher knowledge gain compared to the gardening program. *, **, *** indicate statistical significance at the 0.1, 0.05, and 0.01 levels.

In comparing the type of educational program, the technology program had the largest knowledge gain compared to gardening and composting when controlling for other variables. The technology program experienced a 14% increase in post- minus pre-test scores compared to gardening. Composting had non-significant differences in score difference (i.e., knowledge gain) compared to gardening. A possible reason for the technology gains is that students are potentially more familiar with and interested in technologies given technology is an integral part of many students' lives, thereby creating an interest to learn more about them (Roberts & Foehr, 2008).

Impact of Behavior/Punctuality and Past Program Participation

We also found that increasing the percentage of points students earned for behavior/being on time had a significantly positive impact on knowledge gain. Our result was not unexpected because students who are better behaved are more likely to pay attention in class and receive higher scores (Wentzel, 1991). For instance, for a 1% increase in behavior/punctuality points, a student experienced a 0.5% in knowledge gain. A half-percentage point change may seem like a small change, however, assuming a student received a 76% (i.e., a 10% increase in behavior/punctuality points from the mean reported in Table 3), the student's knowledge gain would have increased by 5%.

With respect to knowledge gains, students who attended a previous program received a significant bump in the knowledge gained in their current program. For instance, for each program attended prior to their current program, the student experienced a 6% increase in knowledge gain. Given that we have controlled for student characteristics, behavior, and program content, it is likely that we are capturing the true effect of past programs on knowledge gain. More research is needed to determine the exact reason, but this effect may be due to one of two reasons, (1) retention of material if the student has taken the course before, or (2) the student has learned strategies that can help the student correctly figure out answers to the material presented. In either case, the program offered is clearly having a positive impact both in the near- (increased post-test scores) and long-term (retention of material or new strategies).

Pre-Test Score Model

The knowledge gain results lead to another question, does past program participation lead to gains on pre-test scores (i.e., does previous program participation impact pre-test scores via previously learned skills or retained knowledge)? Using pre-test scores as the dependent variable in a GLM model, we found that the number of past programs significantly affects a student's pre-test score. So as the number of programs a student had attended increased, pre-test scores were higher (i.e., for every past program attended there was a 4.8% increase in pre-test score). These results provided even more evidence that students were retaining knowledge or learning strategies that could help them in the future.

Recommendations for Extension Educators

By combining gardening, technology, and workforce readiness, data collected from participants allowed us to learn that integrated Extension programming may be the appropriate approach when working with teens, especially in urban settings. Although the article focuses on results from an urban gardening/technology-based program, the findings can be applied to varying youth development, Master Gardener and technology programs. The results reported in this article provide a starting point to begin implementing integrated Extension programming. This article shows how a combination of gardening and technology resulted in positive knowledge gains in both areas. By implementing an employer-employee simulation, students learned how important some workforce readiness skills were. Indeed, knowledge gains were also influenced by students' behavior and punctuality. Students who gained more stipend points had greater knowledge gains than their counterparts. Findings reported also show that repetition of programs might help at-risk teens gain more knowledge or figure out strategies to answer pre and post-tests. This article also provides evidence that simply examining means may not be enough to pull out meaning from a program, especially if clientele participated in several programs. The value of an Extension program may be under- or over-estimated by only examining results of a single program as a part of wider scale efforts. When evaluating the impact of a project, Extension personnel should consider how preceding efforts are impacting their current efforts.

Specific recommendations are as follows.

• 4-H educators working with at-risk urban teens should systematically measure knowledge gains when exposing teens to science literacy topics such as gardening.
• When working with at-risk urban teens (or in evaluating other programs), past participation in program areas should be used to explain current performance.
• Pre-test differences associated with age and content can be offset over the course of a program.

Conclusion

This article identified factors affecting knowledge gain for a gardening and technology program for 5th-8th graders in Connecticut. Results indicate that not only did behavior, content, and pre-test scores affect knowledge gain, but so did past participation in programs. Students either retained knowledge from earlier programs or learned strategies to figure out correct answers. These results gain further traction given that past participation had a significant impact on pre-test scores.

From these results it is clear that a combination of programs such as gardening and technology, as described in this article, increased knowledge gains. Further, Extension educators should consider how past program participation is effecting the evaluation of current program content.

References

Adedokun, O. & Balschweid, M (2009). Are rural 4-hers more connected to their communities than their non-4-h counterparts?. Journal of Extension [On-line], 47(1) Article 1FEA6. Available at: http://www.joe.org/joe/2009february/a6.php

Blair, D. (2010). The child in the garden: An evaluative review of the benefits of school gardening. The Journal of Environmental Education. Routledge Publisher. Winter 2009 40(2).

Cochran, G., Catchpole, K., Arnett, N., & Ferrari, T (2010). Extension's role in preparing youth for the workforce: A challenge to Extension professionals. Journal of Extension [On-line], 48(4) Article 4COM1. Available at: http://www.joe.org/joe/2010august/comm1.php

Cutz, G., & Theuri, E. (2011). Teaching technology to hispanic youth: A report on factors affecting students' learning. JSRI Research Report #45. East Lansing, MI: The Julian Samora Research Institute, Michigan State University.

Fields, N., Brown, M., Piechocinski, A., & Wells, K. (2012) A statewide train-the-trainer model for effective entrpreneurship and workforce readiness programming. Journal of Extension [On-line], 50(5) Article 5TOT9. Available at http://www.joe.org/joe/2012october/tt9.php

Fusco, D. (2001). Creating relevant science through urban planning and garden. Journal of Research in Science Teaching 38(8): 860-877.

Goodman, S. (2003). Teaching youth media: A critical guide to literacy, video production and social change. New York: Teachers College Press.

Greene, W. H. (e). Econometric analysis (5th edition). Upper Saddle River, NJ: Prentice Hall.

Hausman, J. A. (1978). Specification tests in econometrics. Econometrica 46: 1251–1271.

McFarland, A., Glover, B., Waliczek, T., & Zajicek, J. (2013). The effectiveness of the National Wildlife Federation's schoolyard habitat program: Fourth-grade students' standardized science test scores and science grades. HortTechnology. Vol 23 No.2.

National Research Council. (1996). National science education standards. Retrieved from: http://books.nap.edu/openbook.php?record_id=4962&page=22#

Olson, L. (2007). What does 'ready' mean? Education Week 26(40):7-8, 10, 12.

O'Neil, Jr., H. F., Allred, K., & Baker, E. L. (2014). Review of workforce readiness theoretical frameworks. In Workforce readiness: competencies and assessment. Ed. H.F. O'Neil, Jr. Psychology Press, Inc. New York, NY.

Palmer, K. (2012). 7 Things Employers Want form New Grads. U.S. News and World Report. Retrieved from: http://money.usnews.com/money/blogs/alpha-consumer/2012/05/15/7-things-employers-want-from-new-grads

Phelps, J., Hermann, J. R., Parker, S., & Denney, B. (2010). Advantages of gardening as a form of physical activity in an after-school program. Journal of Extension [On-line], 48(6) Article 6RIB5. Available at: http://www.joe.org/joe/2010december/rb5.php

Robbins, T., Stagman, S. & Smith, S. (2012). Young children at risk: National and state prevalence of risk factors. Factsheet for the National Center for Children in Poverty. Retrieved from: http://www.nccp.org/publications/pdf/text_1073.pdf

Roberts, D. F., & Foehr, U. G. (2008). Trends in media use. The Future of Children 18(1):11-37.

Wilhelm, T., Carmen, D. & Reynolds, M. (2002). Connecting kids to technology: Challenges and opportunities. KIDS COUNT Snapshot. Annie E. Casey Foundation, Baltimore, MD. Retrieved from: http://eric.ed.gov/?id=ED467133

Wentzel, K. R. (1991). Relations between social competence and academic achievement in early adolescence. Child Development 62(5):1066-1078.