Challenge to Bolster the Evidence Base for 4-H Science, Technology, Engineering, and Mathematics Programming

Introduction

Since 1902, 4-H has engaged children and youths in science activities in areas including large and small animal sciences, food sciences, gardening, and horticulture (Turnbull, 2013). In 2003, 4-H launched the 4-H Science initiative with the goal of engaging children and youths in advanced science, technology, engineering, and mathematics (STEM) education and careers through 4-H STEM programs that complement and reinforce school-based learning (Locklear, 2013). This initiative has allowed the organization to broaden the scope of 4-H STEM programming to include a variety of offerings, such as robotics, computer technology, and engineering (Turnbull, 2013). 4-H STEM programs provide hands-on, interactive experiences for children and youths in diverse communities (Donaldson & Franck, 2018; Worker, 2013).

Research has shown that 4-H participation is associated with positive outcomes among children and youths. A longitudinal study conducted over 9 years indicated that 4-H children and youths have higher grades in school, demonstrate more positive behaviors, and engage in fewer negative behaviors compared to youths who do not participate in 4-H (Lerner & Lerner, 2013). Likewise, research and evaluative studies on specific 4-H STEM programs have shown that children and youths gain knowledge and positive attitudes related to STEM. Flores-Lagunes and Timko (2015) found a positive association between 4-H participation and knowledge in science and math, as evidenced in part by higher standardized test scores. In addition, participation in 4-H is related to taking more advanced science courses in school and reporting a more positive attitude about science overall (Heck, Carlos, Barnett, & Smith, 2012; Lerner & Lerner, 2013; Rice, Rugg, & Davis, 2016).

The Dilemma

The National 4-H Science Logic Model (National 4-H Council, 2010) encompasses all four components of STEM and has guided program delivery, curricula, and evaluation for the past decade. The long-term societal condition outcomes for 4-H Science as stated in the model are as follows:

• Increased number and more diverse pool of youth pursuing education and careers in science related fields.
• Increased and more diverse pool of trained teachers, educators, scientists.
• Increased science literacy in general population.
• Increased innovation addressing social problems using science. (National 4-H Council, 2010)

However, what has been accomplished and what has been learned through 4-H STEM programming? Perhaps the most comprehensive study was by Mielke and Butler (2013), who surveyed 418 youths representing 38 4-H STEM programs in eight states over 3 years. Among the findings, 4-H participants self-reported that they had gained science and life skills, and their attitudes toward having a science-related job outpaced student data from the National Assessment of Educational Progress (Mielke & Butler, 2013; see National Center for Education Statistics, 2018, for more detail about the National Assessment of Educational Progress).

The Mielke and Butler (2013) study supports the National 4-H Science Logic Model (National 4-H Council, 2010) and offers evidence of the association between 4-H STEM participation and positive outcomes (knowledge and behaviors). Yet the 4-H STEM literature is highly divergent. It is not clear the extent to which other studies have involved reliable, valid instrumentation or provided robust findings that support the National 4-H Science Logic Model. It is also not clear the extent to which the studies can be used to demonstrate (a) the contribution of 4-H to the stated long-term societal condition outcomes and (b) the collective impact of nationwide 4-H STEM programs.

The Challenge

We are challenging the Cooperative Extension System and the land-grant universities of which Extension is an essential component to marshal the intellectual and financial resources to pursue an evidence-based practice for 4-H STEM programs. Evidenced-based practice is characterized by a systematic review of all published research, a summary and analysis of the research, guidelines to inform the practice of frontline staff and others, and dissemination of the resulting guidelines, practices, and recommendations (Dunifon, Duttweiler, Pillemer, Tobias, & Trochim, 2004). The research underlying application of an evidenced-based practice would identify STEM programs and approaches that have been effective and could be replicated across 4-H programs.

To lay the groundwork for evidence-based practice, Cooperative Extension and land-grant universities need to conduct systematic, comprehensive literature reviews and meta-analyses of 4-H STEM programs. Findings of existing 4-H STEM studies must be synthesized and disseminated to improve the knowledge base. Results of systematic, comprehensive literature reviews and meta-analyses may provide a number of benefits. This work would inform 4-H and other nonformal STEM programs for children and youths, provide a stronger foundation for 4-H STEM evidence, and delineate directions for future research. Some key questions to address through systematic, comprehensive literature reviews and meta-analyses of published materials on 4-H STEM programs are as follows:

• To what extent are 4-H STEM programs addressing the fundamental needs of greater science literacy and a more highly skilled workforce?
• What are the effects of 4-H STEM programs on STEM outcomes related to knowledge, attitudes, and behaviors among children and youths?
• What best practices for 4-H STEM programs produce positive outcomes such as greater science literacy and college and career readiness among children and youths?
• What best practices for 4-H STEM programs will produce longitudinal outcomes such as more minorities in STEM fields?

Perspectives

Literature reviews and meta-analyses are valuable tools. Literature reviews summarize previous research and identify theories to explain a phenomenon (Colton & Covert, 2007). In a meta-analysis of 28 STEM studies, Becker and Park (2011) found that integration of the different STEM subjects, especially mathematics and engineering, had positive effects on learning. This synthesis has the potential to save significant time and other resources as experts (researchers, curricula writers) and practitioners (Extension 4-H agents and other educators in youth-serving organizations) can have confidence that integrating mathematics and engineering education in their programs is a research-based practice.

For robust analysis and synthesis of 4-H STEM findings, we suggest conduct of both quantitative and qualitative reviews. Quantitative meta-analysis should draw on multiple fields of study, including program evaluation and information sciences, two domains with distinct standards (for program evaluation standards, see Fitzpatrick, Sanders, & Worthen, 2004, and for information sciences standards, see Templier & Pare, 2015). Qualitative meta-analysis would allow a content analysis of the results of the various studies and would provide a context for interpreting how 4-H STEM programs work in different venues and with different youths (Pawson, Greenhalgh, Harvey, & Walshe, 2005). Literature reviews should draw on the work of Onwuegbuzie, Leech, and Collins (2012), who proposed qualitative approaches to literature reviews.

Pitfalls

The major pitfall for any meta-analysis of findings on 4-H is the vast differences in programs and dosages. 4-H STEM programs involve youths in different levels of activities, ranging from 1-day engagement projects such as the National Youth Science Day to projects that meet regularly and culminate in a competition or demonstration. 4-H STEM programs are offered in a variety of nonformal learning settings that include clubs, afterschool programs, and camps. 4-H STEM programs are often offered through other organizations and agencies, such as local housing authorities and schools (Donaldson & Franck, 2018; Worker, 2013). To understand effects, those undertaking a meta-analysis would need various types of data, such as numbers of programming hours, contexts of programming, and curricula used.

Additionally, researchers may lack sufficient experimental and quasi-experimental studies of 4-H STEM programs to conduct a meta-analysis, and a metasynthesis may be appropriate instead. In our cursory review of materials addressing 4-H STEM programs, we discovered that many evaluative findings of 4-H STEM programs were not published in peer-reviewed publications but were captured in national and state reports as well as numerous theses and dissertations. This circumstance limits the availability, accessibility, and utility of relevant information for researchers and practitioners. Such limitation is a serious pitfall and will require that much time, money, and intellectual resources be invested to capture and understand important aspects of 4-H STEM programming. Yet such investment must occur because any meaningful effort to synthesize 4-H STEM programs will require synthesizing published research, unpublished reports, theses, and dissertations.

Summary

4-H has the potential to serve as a model for STEM programs for all other organizations serving children and youths. 4-H is the largest positive youth development organization in the United States, reaching over 6 million youths aged 8 to 18 each year (National 4-H Council, 2017). 4-H is an inclusive organization—in other words, 4-H is open to all youths who want to participate, regardless of gender, race/ethnicity, or socioeconomic status. As an element of land-grant universities and the Cooperative Extension System, 4-H programs are offered throughout the country in over 3,100 counties serving youths in rural, suburban, and urban communities (National 4-H Council, 2017). Thus, the comprehensive nature of 4-H as an organization provides rich opportunities to explore the effects of STEM programs on a large number and diverse populations of youths.

We believe it is critical to review and synthesize all relevant research on 4-H STEM programs and disseminate the results to inform and expand the knowledge base. This work would identify next steps for strengthening youth STEM programs by informing specific program practices. Dissemination of systematic, comprehensive literature reviews and meta-analyses of published materials on 4-H STEM programs would be valuable to researchers and practitioners because it would build the infrastructure and capacity for both nonformal STEM education and experiential education. This work would combine the strengths of land-grant universities' research and Extension programming in dynamic ways. Finally, and most importantly, as best practices gleaned from the literature reviews and meta-analyses are implemented and the Cooperative Extension System uses the results to strengthen programming, greater, more robust outcomes would be realized for more children and youths. The Cooperative Extension System and the land-grant universities of which Extension is an essential component are challenged to coordinate Extension and research enterprises to produce evidence-based STEM programs for children and youths. Let us wholeheartedly accept this challenge.

References

Becker, K., & Park, K. (2011). Effects of integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students' learning: A preliminary meta-analysis. Journal of STEM Education: Innovations & Research, 12(5/6), 23–37. Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=a9h&AN=72320466&site=eds-live&scope=site

Colton, D., & Covert, R. W. (2007). Designing and constructing instruments for social research and evaluation. San Francisco, CA: John Wiley and Sons.

Donaldson, J. L., & Franck, K. L. (2018). 4-H Science: Building a 4-H career pathway initiative (University of Tennessee Extension publication W668). Retrieved from https://extension.tennessee.edu/publications/Documents/W668.pdf

Dunifon, R., Duttweiler, M., Pillemer, K., Tobias, D., & Trochim, W. M. K. (2004). Evidenced-based Extension. Journal of Extension, 42(2), Article 2FEA2. Available at: https://www.joe.org/joe/2004april/a2.php

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Pearson.

Flores-Lagunes, A., & Timko, T. (2015). Does participation in 4-H improve schooling outcomes? Evidence from Florida. American Journal of Agricultural Economics, 97(2), 414–434.

Heck, K. E., Carlos, R. M., Barnett, C., & Smith, M. H. (2012). 4-H participation and science interest in youth. Journal of Extension, 50(2), Article 2FEA5. Available at: https://www.joe.org/joe/2012april/a5.php

Lerner, R. M., & Lerner, J. V. (2013). The positive development of youth: Comprehensive findings from the 4-H Study of Positive Youth Development. Retrieved from https://4-h.org/wp-content/uploads/2016/02/4-H-Study-of-Positive-Youth-Development-Full-Report.pdf

Locklear, E. L. (2013). 4-H Science initiative from inception to impact: Methodologies, lessons learned, best practices and recommendations. Retrieved from https://4-h.org/wp-content/uploads/2016/02/4-H-Science-Initiative-from-Inception-to-Impact-Comprehensive-Report.pdf

Mielke, M., & Butler, A. (2013). 4-H Science initiative: Youth engagement, attitudes, and knowledge study. Retrieved from https://files.eric.ed.gov/fulltext/ED591155.pdf

National 4-H Council. (2010). National 4-H Science logic model. Retrieved from https://4-h.org/wp-content/uploads/2016/02/4-H-Science-Logic-Model.pdf

National 4-H Council. (2017). National 4-H Council 2017 annual report. Retrieved from https://4-h.org/wp-content/uploads/2018/06/AnnualReport-2017.pdf

National Center for Education Statistics. (2018). National Assessment of Educational Progress: An overview of NAEP. Washington, DC: Author. Retrieved from https://nces.ed.gov/nationsreportcard/

Onwuegbuzie, A. J., Leech, N. L., & Collins, K. M. (2012). Qualitative analysis techniques for the review of the literature. The Qualitative Report, 17(28), 1–28. Retrieved from https://nsuworks.nova.edu/tqr/vol17/iss28/2

Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005). Realist review—A new method of systematic review designed for complex policy interventions. Journal of Health Services Research Policy, 10(S1), 21–34.

Rice, J. E., Rugg, B., & Davis, S. (2016). Minnesota 4-H Science of Agriculture Challenge: Infusing agricultural science and engineering concepts into 4-H youth development. Journal of Extension, 54(3), Article 3IAW4. Available at: https://www.joe.org/joe/2016june/iw4.php

Templier, M., & Pare, G. (2015). A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems, 37, 112–137.

Turnbull, B. J. (2013). The 4-H Science initiative: Summary observations from an evaluation. Retrieved from https://4-h.org/wp-content/uploads/2016/02/4-H-Science-Initiative-PSA-Summary-Report.pdf

Worker, S. M. (2013). Embracing scientific and engineering practices in 4-H. Journal of Extension, 51(3), Article 3IAW3. Available at: https://joe.org/joe/2013june/iw3.php