Lesson Plan Helps Volunteers Improve Learning Among 4-H Youths in Animal Projects

Introduction

Discussions with county Extension educators, state 4-H staff, county fair judges, parents, and 4-H volunteers allowed for collection of anecdotal data identifying gaps in knowledge of many animal science subjects among Idaho 4-H youths involved in market animal projects. A review of available curricula revealed other states using animal science curricula with a quality assurance focus. After a thorough investigation of these curricula, the cost for their use, and the feasibility of providing adequate training for staff and volunteers to use them, a group of 14 Idaho 4-H Extension educators, of which we were members, developed instead lessons that would require less training for volunteer leaders regarding each subject matter area.

We created a library of 60 lesson plans on a variety of animal science topics. Because Idaho 4-H had adopted the Ohio State University Extension Resource Handbooks for beef, sheep, swine, and goat projects, it was a priority to develop animal science–focused lessons from those handbooks. Lesson authors constructed the lessons using the essential components the National Science Foundation recommends for science-based curricula: establishing a relationship between content and experience and facilitating animal science education from an experiential perspective (Horton & Hutchinson, 1999). Each of the 60 animal science lesson plans includes a goal, learning objectives, and directions for implementing the lesson following the experiential learning model of Do, Reflect, Apply (Torock, 2009). The lesson authors also developed the lessons to be developmentally appropriate for the learners: Level 1 for beginners, Level 2 for those with some knowledge, and Level 3 for advanced youths. During the year prior to the study, we conducted pilot tests of the lessons and refined the lessons according to feedback from youths, facilitators, and 4-H professionals.

Multiple studies have demonstrated that youths participating in animal science programs or workshops learn animal science skills and knowledge (Cummins & Nash, 2014, Fassett, Nold, & Rockwell, 2005; Goodwin, Murphy, & Briers, 2002; Ivey, O'Rourke, Grenawalt, & Hobson, 2015; Nold, & Hanson, 2001) and enhance their communication and teamwork skills (Rusk & Machtmes, 2002). However, the teaching methods addressed in these studies required that 4-H staff trained in the subject matter conduct the lessons. That method was not feasible for Idaho, given the number of topics, number of clubs, and dispersed locations of market animal clubs in the state. Therefore, we crafted the University of Idaho 4-H Animal Science Lesson Plans (Nash et al., 2019). These peer-reviewed lessons were designed to be "grab and go," allowing them to be used as needed with no training on animal science topics.

Methods

Our goal with the study reported here was to investigate the statewide implementation of one of these animal science lessons by volunteer leaders with no prior training in the subject matter for the purpose of improving knowledge among youths in traditional 4-H market animal club programs. The animal science knowledge level of volunteers was assessed when they were interviewed and screened to become volunteers. Of the 60 lesson plans developed, we chose to examine the effectiveness of the quality assurance lesson because the topic is foundational across all market animal projects. Many 4-H educational programs have been implemented to increase youth producers' education on quality assurance and ethics topics and have been shown to improve youths' knowledge of quality assurance practices, behaviors, skills, and ethical decisions. However, there is limited research available on the effectiveness of using 4-H volunteers who have no training in the subject area to deliver programs created by Extension faculty.

Design

We used a stratified random-assignment treatment–control design in which "treatment" consisted of use of the lesson and "control" was no use of the lesson. We implemented the quality assurance lesson and measured youth knowledge of quality assurance using a test specific to the learning objectives of the lesson. We communicated information about the study to 4-H county professionals and indicated when and how to conduct the knowledge assessment tests. The study was approved by the University of Idaho Institutional Review Board (18-023). Club leaders trained by our author team administered the test to control and treatment groups at the beginning of the season (pretest) and then again at the end of the season (posttest). The time between pre- and posttesting was 4 to 5 months.

Selection of Study Sample

On the basis of (a) the goals for the study and results from the lesson pilot implementation, (b) the minimum number of youths needed in each group, (c) an estimated response rate of 50%, and (d) the feasibility of conducting the study statewide, we planned for a sample size of 64 clubs (over 600 youths) for both the control group (32 clubs) and treatment group (32 clubs). We used stratified sampling to ensure that we had clubs from all four market animal groups and all four state districts.

We identified the statewide population of 498 market animal clubs (4,713 youths) through the 4-H Online registration system. Clubs within each district and market animal group were first profiled and then matched to form pairs of clubs comparable for average club member age, average club member years in 4-H, and number of club members. We randomly selected one matched pair of clubs from the younger end and one matched pair from the older end of the age range, resulting in two matched pairs of clubs per district per market animal group. Then we randomly assigned the clubs from each matched pair to the treatment or control group. Selecting clubs and matching them based on potentially confounding factors and then randomly assigning the matched clubs to treatment or control allowed us to minimize the confounding effects of those factors and ensure that we had statewide representation of clubs from all four market animal projects.

Clubs were informed that participation in the study was optional and that there were no consequences for not participating in or not completing the study. Two clubs opted out and were replaced with a random selection of clubs with similar club profiles.

Statistical Analysis

We performed all analyses in R Version 3.5.2 (R Development Core Team, 2018). Due to nonnormality of the raw pretest data, we performed a Mann–Whitney U test and Kruskal–Wallis test to compare the pretest results for the control and treatment groups to assess whether the cohorts were different prior to lesson delivery. We further evaluated whether the cohorts were substantially equivalent by testing whether they differed by less than 0.5 correct answers for pretest results using the TOSTER package (Lakens, 2017). The response data were expressed as number of correct pretest answers subtracted from number of correct posttest answers, which we interpreted as gain in knowledge.

We analyzed the data using a linear mixed model where gain in knowledge was the dependent variable, cohort was a fixed effect, and the 4-H club was a random effect. The model residuals were examined for homoscedasticity and adherence to normality. We conducted this analysis using the packages 'lme4' (Bates, Maechler, Bolker, & Walker, 2015) and 'lmerTest' (Kuznetsova, Brockhoff, & Christensen, 2017) and extracted the least squares means using the package 'emmeans' (Lenth, Singman, Love, Buerkner, & Herve, 2019).

Results

Of the 64 clubs solicited to participate in the study, 42 clubs (65.6% response rate) completed the pretest and 22 clubs (34.4% response rate) completed the posttest. Youths from only nine clubs from the control group and 12 clubs from the treatment group (32.8% response rate) completed the pre- and posttests (Table 1).

Table 1.
Number of Youth Participants in Each Study Group and Numbers Completing Pre- and/or Posttest

Study group	No. of youths	No. completing pretest only	No. completing posttest only	No. completing both pre- and posttest
Control	310	157	55	42
Treatment	355	203	109	93
Total	665	360	164	135

We conducted the analyses using the results from the subgroup of 135 participants who complete both a pretest and a posttest.

Except for the lack of Hispanic youths within the control group and a slightly higher percentage of females, the study groups were comparable to and representative of the population (Table 2).

Table 2.
Comparison of Demographics and 4-H Experience Between Study Groups and Population of Youths Enrolled in 4-H Market Animal Projects

Study group	Average age	Percentage female	Average years in 4-H	Percentage Hispanic
Control	12.3	57.1%	4.1	0.0%
Treatment	12.1	64.5%	3.8	3.8%
Populationa	12.2	53.3%	4.2	4.5%
aRefers to the population of youths in Idaho 4-H market animal projects.

The study groups also retained balanced representation from all four state districts and all four market animal groups (Table 3).

Table 3.
Number of Records by District and Animal Project

District	Beef	Goat	Sheep	Swine	Total
Northern	3	5	4	23	35
Southern	7	11	5	5	28
Central	1	5	6	17	29
Eastern	12	1	17	13	43
Total	23	22	32	58	135

Very high pretest scores were observed for both groups, especially on Questions 1, 3, and 4 (Table 4). These high scores greatly limit the ability to observe improvement and differentiation between the pre-lesson-plan and post-lesson-plan conditions. Furthermore, the control group sample size was very small (n = 42) and thus susceptible to small perturbations in sampling variation. The treatment group showed improvement on every question, whereas the control group showed meaningful improvement on only Question 2 (Table 4).

Table 4.
Percentages of Correct Answers on Pre- and Posttests by Test Group

Assessment question	Control		Treatment
	Pre	Post	Pre	Post
1. Quality Assurance in 4-H Livestock Programs means: (Correct answer: Making sure my animal is high quality and a safe product for the consumer.)	98.7%	98.2%	95.6%	96.3%
2. Who is responsible for providing safe and wholesome livestock products to consumers? (Correct answer: Everyone who is part of the livestock industry.)	50.3%	60.0%	41.9%	50.5%
3. Quality Assurance in 4-H Livestock Programs is (Correct answer: A promise a 4-H'er makes by being in the livestock industry.)	80.9%	78.2%	79.8%	88.1%
4. Which statement is correct when you think about feeding your 4-H animal? (Correct answer: My animal needs to be fed the same feed the same time each day.)	94.3%	94.5%	88.2%	93.6%
5. Which of the following has an impact on the product quality of your 4-H animal? (Correct answer: All of the above.)	70.7%	65.5%	68.0%	78.0%
Total	80.0%	81.9%	74.0%	83.4%
Note. n = 135.

A greater proportion of participants in the treatment group improved on the posttest compared to control group (Figure 1). For the control group, 75% of the participants scored a 3.25 or greater on both the pre- and the posttests. In the treatment group, 75% of the participants scored a 3 or above on the pretest but a 4 or above on the posttest. And although the control group scored higher overall on the pretest than the treatment group, the treatment group improved to a greater degree, scoring higher on the posttest.

Figure 1.
Box Plots of Test Scores for Control and Treatment Groups

Analysis

The Mann–Whitney U test and Kruskal–Wallis test both provided mixed indications that the control and treatment groups were drawn from the same population, with test statistics of 2,304.5 and 3.04, respectively, and p-values of .082 and .081, respectively. These results do not indicate overwhelming rejection of the null hypothesis that the groups are the same, but neither do they favor the null hypothesis strongly. The results from equivalence testing are −0.967 (p = .170) for the lower equivalence bound test statistic and 3.890 (p < .001) for the upper equivalence bound test statistic. These results provide evidence that favor the null hypothesis that the samples are at least 0.5 units apart and also provide evidence that favors the alternative hypothesis that the samples are substantially equivalent. The conclusion is that these samples are neither statistically equivalent nor statistically different.

We expressed the dependent variable as gain in knowledge, the difference in posttest and pretest results, to address the possible differences between the control and treatment cohorts for the pretest results. These data also met the assumption of normality and equality of variance that the original data did not. The results from the linear mixed model indicate that cohort had a modest effect on gain in knowledge (F-value = 3.16, p = .087). The least squares mean for gain in knowledge for the control group was 0.09 ± 0.181 (standard error) and for the treatment group was 0.4814 ± 0.128. A likelihood ratio test indicated that 4-H club minimally affected gain in knowledge (Χ² = 0.48, p = .4874).

Discussion

The results of our study indicate that the youths are attaining and retaining quality assurance knowledge from the 4-H programming. Although scores on the pretest in the study were higher than expected on the basis of results from the instrument pilot test, our results show that use of the quality assurance lesson plan helped bolster those other efforts conducted throughout the state to improve youth quality assurance knowledge.

The study demonstrates that the use of a prepared lesson for clubs can help increase the number of youths improving their content knowledge for the topic. More youths who received the lesson correctly answered the quality assurance questions as compared to youths who did not receive the lesson. Indeed, youths in the control group showed little knowledge gain and, in some cases, showed a decrease in correct answers on the posttest.

Our goal for the Idaho 4-H Animal Science Lesson Plans is to provide inexpensive and feasible means to help volunteers teach a wide array of animal science knowledge and skills. The benefit for the volunteer and the youth is that the library of 60 lessons enables clubs to examine new topics for which a leader might not have a comfortable level of knowledge. The conclusion is that when youths were administered an Idaho 4-H Animal Science Lesson Plan by 4-H volunteers with no training, they learned the intended information provided by the lesson. Although more work is needed to examine the effectiveness of the other lessons, our study provides good evidence that use of the quality assurance lesson plan can improve learning via an approach that is inexpensive and feasible to implement.

References

Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects model using lme4. Journal of Statistical Software, 67(1), 1–48. doi:10.18637/jss.v067.i01

Cummins, M., & Nash, S. (2014). Urban youth develop life skills raising livestock. Journal of Extension, 52(5), Article v52-5iw7. Available at: https://www.joe.org/joe/2014october/iw7.php

Fassett, J. L., Nold, R. A., & Rockwell, S. K. (2005). Assuring youth raising livestock for food produce a quality product. Journal of Extension, 43(1), Article 1RIB7. Available at: http://www.joe.org/joe/2005february/rb7.php

Goodwin, J. L., Murphy, T., & Briers, G. (2002). Measuring the ethical cognition effects of a videotape livestock show ethics education program. Journal of Extension, 40(6), Article 6RIB2. Available at: https://www.joe.org/joe/2002december/rb2.php

Horton, R. L., & Hutchinson, S. (1999). Nurturing scientific literacy among youth through experientially based curriculum materials. National Network for Science and Technology Home Page. Retrieved from https://ohio4h.org/sites/ohio4h/files/d6/files/4H_591 Nurturing Scientific Literacy Among Youth Through Experientially Based Curriculum Materials.pdf

Ivey, D., O'Rourke, B., Grenawalt, A., & Hobson, P. (2015). Youth outcomes of the Wisconsin Meat Animal Quality Assurance Program, evaluative paper, Department of Youth Development, University of Wisconsin Extension, February 2015. Retrieved from https://fyi.extension.wisc.edu/youthlivestock/files/2015/12/MAQA-paper-3-2015-FINAL.pdf

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest Package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. http://doi.org/10.18637/jss.v082.i13

Lakens, D. (2017). Equivalence tests: A practical primer for t-tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362. doi:10.1177/1948550617697177

Nash, S. A., Baker, S. D., Dalton, N., Harrison, S. N., Jefferies, E. T., Jensen, S., . . . Wilson, J. (2019). University of Idaho 4-H Animal Science Lesson Plans, Level 1. PNW 726. Moscow, ID: University of Idaho.

Nold, R., & Hanson, D. (2001). Effectiveness of quality assurance training for youth. Journal of Extension, 39(2), Article 2FEA4. Available at: https://www.joe.org/joe/2001april/a4.php

R Development Core Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/

Rusk, C., & Machtmes, K. (2002). Teaching youth through 4-H animal science workshops. Journal of Extension, 40(5), Article 5IAW7. Available at: https://www.joe.org/joe/2002october/iw7.php

Lenth, R., Singman, H., Love, J., Buerkner, P., & Herve, M. (2019). emmeans: Estimated marginal means, aka least-squares means. R package Version 1.4.2. Retrieved from https://CRAN.R-project.org/package=emmeans

Torock, J. L. (2009). Experiential learning and Cooperative Extension: Partners in non-formal education for a century and beyond. Journal of Extension, 47(6), Article v47-6tt2. Available at: https://www.joe.org/joe/2009december/tt2.php