Multiple-Methods Needs Assessment of California 4-H Science Education Programming

Background

The 4-H youth development program has targeted the improvement of youth scientific literacy through the 4-H Science Initiative, or 4-H Science (4-H National Headquarters, 2007). A decade has passed since the initiative was established, and as situations evolve over time—institutional needs, staff requirements, and capacities shift—the regular assessment of programmatic and organizational needs is essential to help guide ongoing efforts (Gunn & Loy, 2015). A needs assessment is a systematic approach to studying the current state and future vision of participants in a defined group that elucidates gaps, allowing program planners to make informed decisions about future efforts and directions (Altschuld & Kumar, 2010).

Needs assessments are common in Cooperative Extension because of their importance in developing effective Extension programming. These assessments often involve quantitative survey methods. Examples include assessments of the educational needs of women farmers in Pennsylvania (Barbercheck et al., 2009), Pennsylvania dairy processors (Syrko & Kaylegian, 2015), and Michigan farmers and agribusiness operators (Suvedi, Jong, & Coombs, 2010). Although not as common, other Extension needs assessments have involved qualitative methods, such as listening sessions for assessing land, farm, production, and health needs of Iowa Beef ranchers (Gunn & Loy, 2015) and key informant interviews for evaluating needs of families and children living in Waushara County, Wisconsin (Caravella, 2006). Even more uncommon are Extension needs assessments that employ multiple methods, such as surveys and phone interviews for assessing clientele satisfaction with Virginia's 4-H horticulture programming (Phibbs, Relf, & Hunnings, 2005), survey and focus group sessions for assessing types of Extension programming on renewable energy generation (Thomas & Brain, 2016), or action research involving focus groups, surveys, and feedback sessions for ascertaining Extension employee perceptions about educational programs (Havercamp, Christiansen, & Mitchell, 2003).

The use of a single data collection method can be a weakness when conducting needs assessments. Findings generated through reliance on one methodology suffer from the inherent biases of that methodology (Johnson, Onwuegbuzie, & Turner, 2007). Thus, there is benefit in employing multiple methods whereby data can be triangulated to provide persuasive evidence for claims (Mathison, 1988; Patton, 2015). This article describes a needs assessment that resulted in findings formed through the integration of three qualitative data sources.

Methods

We, the California 4-H Science Leadership Team, conducted a statewide assessment to explore the needs of 4-H academic and program staff as they related to the implementation and evaluation of 4-H science programs. The approach involved three qualitative data sources: an open-ended survey, focus group interviews, and reports from participatory working groups. We analyzed data from each source using a set of typologies (Hatch, 2002) based on six key program areas outlined in the National 4-H Science Initiative Framework—program development and design, professional development, curricula, evaluation, partnerships, and fund development (4-H National Headquarters, 2007; Locklear, 2013). The study was approved by the institutional review board of the University of California, Davis.

Online Open-Ended Survey

All California 4-H academic and program staff were invited to respond to an online survey consisting of 27 open-ended questions. Specifically, questions prompted participants to describe the current state and future vision for each of the aforementioned six key program areas relative to 4-H science programming. Seven academic and 12 program staff responded to the survey (23% response rate), representing 20 of 58 California counties.

Focus Group Interviews

We conducted four concurrent face-to-face focus group interviews with 4-H academic and program staff and other Extension personnel involved in 4-H science programming. These individuals were eight academic staff and 12 program staff representing 17 counties. Seven of them also had responded to the survey. The focus group participants were asked a set of 12 open-ended questions around the six key program areas related to 4-H science programs; questions were worded differently from those on the survey. The interviews were recorded and transcribed.

Participatory Working Groups

Participatory working groups are grounded in a process that emphasizes building on the perspectives of each participant to create a product that reflects a consensus viewpoint (see McCawley, 2009). Participants were identified through a stratified purposive sample process (Patton, 2015), resulting in 18 individuals: 15 4-H professionals (nine academic staff, six program staff) representing 11 California counties and three external stakeholders. Of the 4-H professionals, two had responded to the survey, five had participated in focus groups, and four had both responded to the survey and participated in focus groups. Participants were organized in three participatory working groups that met for 7 hr to discuss the prompt "What is needed to build staff and volunteer capacity to facilitate comprehensive 4-H science education programs?" as it pertained to the six key program areas. Groups produced written summaries of organizational strengths, challenges, opportunities, and recommendations.

Data Analysis and Data Integration

We analyzed data collaboratively through a consensus-based process such that researcher agreement was achieved, resulting in strengthened trustworthiness of the findings (Armstrong, Gosling, Weinman, & Marteau, 1997). Data analysis, conducted manually, involved data triangulation and analyst triangulation to identify emerging themes within a data source and across data sources, further helping reduce bias (Patton, 2015). Through this process, we integrated findings from the data sources, thereby applying "a level of reflexive engagement" with the process, the participants, and the data (Chamberlain, Cain, Sheridan, & Dupuis, 2011, p. 165).

Findings

Findings, along with salient examples from each data source, are reported herein and are organized according to the six key areas outlined in the National 4-H Science Initiative Framework. Findings from academic and program staff are merged, not delineated, and the two groups are collectively referred to hereafter as staff.

Program Development and Design

A wide variety of 4-H science programs were identified. Three approaches to educational programming (termed "delivery modes") arose as the most common: 4-H clubs, 4-H camps, and educational afterschool programs. Four subject areas emerged as top choices for future 4-H science programming: animal science, environmental education, gardening, and nutrition. Additionally, staff believed that California 4-H was strongly positioned and prepared to use adult volunteers and teen teachers in science programming. Data analysis also suggested that several areas related to program development were lacking. The most notable gaps were in intentional planning and implementing of 4-H science programs toward a common organizational vision and mission and in staff understanding of the definition of scientific literacy—that is, an institutionally recognized goal for youth science learning (e.g., in California, scientific literacy has been defined with four anchor points [Smith, Worker, Ambrose, & Schmitt-McQuitty, 2015]). Respondents provided ample examples of programs that targeted improved content knowledge but generally were less confident of their ability to improve young people's scientific reasoning skills, interests in and attitudes toward science, or applied community participation. Excerpts from participants' responses and discussions are presented in Table 1.

Table 1.
Participant Responses Related to Program Development and Design

Data source	Sample comment
Online open-ended survey	"[Volunteer] project leaders generally are not emphasizing that their project meetings involve science."
Focus group interviews	"When [volunteers] hear that we should be teaching STEM, to them STEM is 'let's teach them how to solder,' not approaching it from the perspective that this experiential learning . . . We're trying to get [youths] to critically think and to reason, not necessarily how to use a soldering iron."
Participatory working group reports	"Incorporate STEM into [4-H volunteer] orientation as 'business as usual' and integrated into everything we do in 4-H. Science is all around us."
Note. STEM = science, technology, engineering, and math.

Professional Development

Participants identified a gap in opportunities for professional development of staff and volunteer educators with respect to both pedagogy and science content. There was agreement that there are additional needs for staff and volunteer professional development. Some respondents discussed the reluctance of 4-H volunteers to participate in professional development and the need to help them be more receptive to participating in opportunities to advance their pedagogical abilities. Respondents reported that volunteers were drawn to professional development focused on content (e.g., woodworking, robotics) but not focused on pedagogy (e.g., experiential learning). Although a majority of respondents indicated their preference for in-person professional development, participants also noted that travel can be a limiting factor. The prevailing method for professional development involved single-episode, expert-led, group-based approaches (e.g., workshops); respondents also discussed a desire to implement ongoing professional development series that are learner-centered (e.g., communities of practice). Quotes from participants are provided in Table 2.

Table 2.
Participant Responses Related to Professional Development

Data source	Sample comment
Online open-ended survey	"I offer 12 hours of professional development per year for volunteers. Two to three hours are devoted to science. Not many take advantage of the opportunity."
Focus group interviews	"Our county would like to be able to offer volunteer trainings and would appreciate state help with that. We would like to be able to send our key leader and facilitators to workshops and/or have the workshops made available to them."
Participatory working group reports	"Consider 'leveling-up' and building mastery for professional goals and competencies. Should there be a minimum of training [required]?"

Curricula

A broad array of curricula were being used by 4-H volunteer educators in 4-H science programs. However, there was not much consistency in how the curricula were employed to address specific science content. Access to curricula was inconsistent as well; much was gathered from Internet searches, whereas a few respondents reported that they obtained curricula from the national 4-H Mall or the state 4-H online curriculum library. Accessibility of resources was an identified gap, and even more fundamentally, the mere awareness of what constitutes a high-quality curriculum was limited. Furthermore, some staff expressed the concern that volunteers were reluctant to use published curricula. Many staff stated a desire for a central clearinghouse for "4-H Science–approved" curricula. Selections from participants' responses are included in Table 3.

Table 3.
Participant Responses Related to Curricula

Data source	Sample comment
Online open-ended survey	"I think that overall we have a need for high quality science curriculum, especially in our club program. I am not sure what if any curriculum is being utilized."
Focus group interviews	"I don't think they [volunteers] use it [curriculum] because they don't understand why it's so important. And I think that's where the professional development [would assist] in really training them on what experiential learning is and what our focus is."
Participatory working group reports	"Consider implementing internal processes to ensure ongoing efforts for identifying appropriate curriculum."

Evaluation

Output data were being collected consistently, primarily through the 4-H enrollment process, but there was limited evidence of systematic evaluation of 4-H science programs. Of the evaluation efforts being undertaken, most assessed science content learning, and a few assessed improving reasoning skills or changes in attitudes. Staff had limited knowledge of the difference between formative evaluation and outcome evaluation. Also, there was misunderstanding as to who was responsible for evaluation, perhaps because some staff had institutional responsibilities for applied research that may have been confused with program-oriented evaluation. In general, there was a need to build staff capacity relative to evaluation strategies and methodologies, incorporate formative evaluation into educator professional development, and explore creative mechanisms for acquiring evaluation data. Excerpts from participants' comments are contained in Table 4.

Table 4.
Participant Responses Related to Evaluation

Data source	Sample comment
Online open-ended survey	"Making sure evaluation is planned for in the beginning is essential. I would be interested in exploring different methodologies beyond surveys, focus groups, and onsite observations."
Focus group interviews	"There needs to be a place . . . where [academics] can talk about and work through their research questions with people who have an interest in that area."
Participatory working group reports	"Explore alternative methods for evaluation—not surveys! Can [youths] apply what they've learned—presentations, photo/video stories over time; teens as evaluators."

Partnerships

There was strong evidence of partnerships with afterschool sites, companies, government agencies, and other organizations. Staff reported that developing partnerships took effort but enhanced 4-H science program effectiveness. Respondents indicated that 4-H provided education programs, curriculum resources, and professional development to other organizations, thereby supporting productive partnerships. In general, staff desired additional help in identifying successful approaches to working with external partners and tools for building effective partnerships. Quotes from participants are provided in Table 5.

Table 5.
Participant Responses Related to Partnerships

Data source	Sample comment
Online open-ended survey	"Our partners provide program leadership and delivery, funding, participants, and volunteers. We [4-H] provide high quality science and service learning experience to the partnership. We provide training, curriculum, support for afterschool program staff."
Focus group interviews	"I want us to be a resource in our community . . . a place that the adults can go to, these frontline afterschool providers can go to, and say, 'This is what we want to do. We want to do what 4-H did for X.' And . . . [we] facilitate professional development . . . so that they can incorporate experiential learning and STEM into their afterschool programs."
Participatory working group reports	"How can California 4-H be a leader in supporting local 'ecosystems' of STEM learning within California? . . . 4-H brings extensive experience recruiting, training, and managing adult volunteers, and in many cases, science expertise that other youth development organizations do not have. Explore possibilities and offer choice."
Note. STEM = science, technology, engineering, and math.

Fund Development

There was not much evidence of strategic fund development for 4-H science programming. Funding sources were largely serendipitous. Staff obtained funding by identifying diverse sources, from local agencies to grant providers, private donors, and 4-H volunteer councils. Staff would benefit from having an integrated approach to fund development wherein fewer funders might be stewarded; as it was pointed out, too many funding sources may be problematic when the situation calls attention away from one's strategic goals. Excerpts from participants' responses are shown in Table 6.

Table 6.
Participant Responses Related to Fund Development

Data source	Sample comment
Online open-ended survey	"We would be in favor of resources from the State 4-H Office (e.g., The 4-H Foundation) to assist us in our fund-raising efforts."
Focus group interviews	"I know there [are] funders in my area . . . alumni in my area . . . people out there willing to partner, but it takes time to develop the relationships. It takes time to figure out who and what."
Participatory working group reports	"Identify numerous types of funding, support, and opportunities (e.g., grants, in-kind, or museums) and share ideas with program staff and academics."

Conclusions

The needs assessment described in this article indicated support and buy-in regarding 4-H Science and revealed opportunities for more intentional and systematic 4-H science programming. Organizationally, there is a need for national and state efforts that provide practical program models and examples of programs targeting scientific literacy. Professional development is needed to improve staff members' abilities to intentionally frame programming and evaluation around scientific literacy, particularly with regard to connecting programs to the real world.

Recommendations

Our study resulted in a rich data set that we used to understand the capacities, challenges, and opportunities staff have in coordinating 4-H science programs. Although the sheer abundance of qualitative data extended the time required to sort, analyze, and interpret the data, we believe that using more than one method improved our understanding of the needs and gaps in 4-H science programming beyond what a survey, or any single data source, could have provided. Those in Extension should know, nonetheless, that the use of multiple methods, particularly those resulting in qualitative data, requires an investment in time and effort to gain optimal benefit. In our case, there were important positive trade-offs, including increased participant awareness of the breadth of science programming and community building promoted by the use of participatory working groups.

Acknowledgments

We appreciate the assistance of Ramona Carlos, Vanessa Murua, Shelbey De Genova, and Hannah McNeill. Financial support for our study was provided by the S. D. Bechtel, Jr. Foundation.

References

4-H National Headquarters. (2007). 4-H science, engineering and technology: A strategic framework for progress. Washington, DC: U.S. Department of Agriculture.

Altschuld, J. W., & Kumar, D. D. (2010). Needs assessment: An overview. Los Angeles, CA: Sage.

Armstrong, D., Gosling, A., Weinman, J., & Marteau, T. (1997). The place of inter-rater reliability in qualitative research: An empirical study. Sociology, 31(3), 597–606.

Barbercheck, M., Brasier, K. J., Kiernan, N. E., Sachs, C., Trauger, A., Findeis, J., & Stuart Moist, L. (2009). Meeting the Extension needs of women farmers: A perspective from Pennsylvania. Journal of Extension, 47(3), Article 3FEA8. Available at: https://www.joe.org/joe/2009june/a8.php

Caravella, J. (2006). A needs assessment method for Extension educators. Journal of Extension, 44(1), Article 1TOT2. Available at: https://www.joe.org/joe/2006february/tt2.php

Chamberlain, K., Cain, T., Sheridan, J., & Dupuis, A. (2011). Pluralisms in qualitative research: From multiple methods to integrated methods. Qualitative Research in Psychology, 8, 151–169.

Gunn, P., & Loy, D. (2015). Opportunities and challenges in a changing beef industry: Results of a statewide needs assessment in Iowa. Journal of Extension, 53(5), Article 5RIB2. Available at: https://www.joe.org/joe/2015october/rb2.php

Hatch, J. A. (2002). Doing qualitative research in education settings. Albany, NY: State University of New York Press.

Havercamp, M., Christiansen, E., & Mitchell, D. (2003). Assessing Extension internal organizational needs through an action research and learning process. Journal of Extension, 41(5), Article 5FEA2. Available at: https://www.joe.org/joe/2003october/a2.php

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133.

Locklear, E. L. (2013). 4-H Science Initiative from inception to impact: Methodologies, lessons learned, best practices, and recommendations. Chevy Chase, MD: National 4-H Council.

Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13–17.

McCawley, P. F. (2009). Methods for conducting an educational needs assessment: Guidelines for Cooperative Extension System professionals. Moscow, ID: University of Idaho Extension.

Patton, M. Q. (2015). Qualitative research and evaluation methods (4th ed.). Thousand Oaks, CA: Sage Publications.

Phibbs, E., Relf, D., & Hunnings, J. (2005). Implementing a needs assessment for long-term strategic planning in 4-H horticulture programming. Journal of Extension, 43(4), Article 4RIB7. Available at: https://www.joe.org/joe/2005august/rb7.php

Smith, M. H., Worker, S. M., Ambrose, A. P., & Schmitt-McQuitty, L. (2015). Scientific literacy: California 4-H defines it from citizens' perspective. California Agriculture, 69(2), 77–82.

Suvedi, M., Jong, E., & Coombs, J. (2010). Education needs of Michigan farmers. Journal of Extension, 48(3), Article 3RIB7. Available at: https://www.joe.org/joe/2010june/rb7.php

Syrko, J., & Kaylegian, K. E. (2015). Developing a contemporary dairy foods Extension program: A training and technical resource needs assessment of Pennsylvania dairy foods processors. Journal of Extension, 53(5), Article 5RIB1. Available at: https://www.joe.org/joe/2015october/rb1.php

Thomas, B. H., & Brain, R. G. (2016). Opportunities for and barriers to renewable energy outreach in Extension: A mixed-methods needs assessment. Journal of Extension, 54(5), Article 5FEA1. Available at https://joe.org/joe/2016october/a1.php