Youths Perceive Some Improvement in Substance Abuse Prevention Knowledge, Skills, and Assets from Participation in 4-H Health Rocks!

Introduction

The use of tobacco, alcohol, and other drugs among young people remains a national dilemma, with an estimated 23.9 million Americans 12 years old and older having used an illicit drug or abused a prescription medication (National Institute on Drug Abuse, 2014). Youth access to tobacco, alcohol, and other drugs is a considerable problem. In a 2012 study on the availability of marijuana among youths, 37% of eighth graders, 69% of 10th graders, and 82% of 12th graders reported that it was very easy to obtain marijuana (Johnston, O'Malley, Bachman, & Schulenberg, 2013). Almost all youths aged 12 to 14 are at risk for trying a substance due to peer pressure or the normalizing of substance use in their communities (Miller & Hendrie, 2008). Using tobacco, alcohol, and other drugs before the brain is fully developed increases the risk of addiction to other substances (National Council on Alcoholism and Drug Dependence Inc., 2015).

4-H Health Rocks! Program Delivery

Through the combined efforts of the National 4-H Council, its donors, and faculty of the land-grant university system, Cooperative Extension provides 4-H Health Rocks!, a program to prevent tobacco, alcohol, drug abuse among 8- to14-year-old youths.The program is delivered through 10 hr of instruction on the negative health outcomes of substance abuse, communication, decision making, relationship building, and self-esteem. The program includes a two-part curriculum, with a beginning level for youths aged 8 to 12 and an intermediate level for youths aged 12 to 14 (Donaldson & Crowe, 2013). The program is delivered in multiple settings, including residential camps, day camps, afterschool programs, and in-school club meetings. The program can be organized using various time frames; for example, an afterschool program may devote 1 hr to 4-H Health Rocks! each week for 10 weeks (Kumaran, Fogarty, Fung, & Terminello, 2015; National 4-H Council, 2009).

4-H Health Rocks! Program Evaluation

Kumaran et al. (2015) evaluated 4-H Health Rocks! through a qualitative study of Florida Extension 4-H professionals conducting the program. They identified time constraints, unsuccessful volunteer recruitment, and an inability to fulfill the program's 10 hr of implementation as major barriers to program success. An evaluation of the 4-H Health Rocks! program in Georgia indicated that the program produced positive, significant gains in youth participants' perceptions of their knowledge, beliefs/attitudes, skills, and behavioral intentions (Self, Morgan, Fuhrman, & Navarro, 2013).

The Georgia study (Self et al., 2013), as well as the research reported here, involved use of a retrospective "post-then-pre" questionnaire. The retrospective post-then-pre questionnaire is completed at the end of the program and asks a participant to evaluate his or her status in a particular realm (i.e., behavior) after participating in the program and then, in a second question, to evaluate what his or her status had been in that realm before participating in the program. Retrospective post-then-pre design helps respondents have "sufficient knowledge to answer the question validly" (Rockwell & Kohn, 1989, "Correcting Problem," para. 1), requires minimal time investment from both respondents in answering questions and researchers in analyzing data (Davis, 2003), and provides a more complete data set than traditional pretest/posttest approaches do (Raidl et al., 2004). However, the design has some important drawbacks. Retrospective post-then-pre design relies on self-reports, and respondents may provide the answers they think researchers want; respondents may not understand the concept because of cultural differences, language barriers, and/or literacy levels; and research is lacking regarding how to best administer post-then-pre questionnaires (Klatt & Taylor-Powell, 2005).

On the whole, the research reported here differed from the previous 4-H Health Rocks! evaluations. Unlike with the Florida study (Kumaran et al., 2015), a quantitative approach was used. And although a quantitative, retrospective post-then-pre design was used for both the Georgia study (Self et al., 2013) and the study reported here, different instruments were used in the two studies. The Georgia study involved a questionnaire with 36 items (Self et al., 2013) requiring two-part responses, whereas the research reported here involved a questionnaire with 13 items requiring two-part responses.

Purpose

The purpose of the study reported here was to describe results of a 4-H Health Rocks! survey of Tennessee program participants during the 2011–2012 program year. Specific objectives were

1. to describe the demographic characteristics of respondents (gender, race, ethnicity, and grade) and
2. to describe differences, if any, in the postprogram and retrospectively identified preprogram scores of respondents regarding their knowledge, skills, and assets.

Methods

During the 2010–2011 academic year, a retrospective post-then-pre survey was constructed by researchers at the University of Nebraska–Lincoln (UNL), who established face and content validity through a review by six panel members familiar with both 4-H youth development and the 4-H Health Rocks! curriculum (Yan & de Guzman, 2011). All questions were written to elicit participants' self-reports of their statuses after and before participating in the program; the instrument consisted of 13 Likert scale questions, with the scale being 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree (Yan & de Guzman, 2011). The pilot test group consisted of 63 survey respondents, and principal component analysis showed three constructs, which the researchers identified as knowledge (four items), skills (four items), and assets (five items) (Yan & de Guzman, 2011). The researchers calculated Cronbach's alpha reliability values for the instrument's constructs and reported that those values for the overall before and after scales were above .70, which is considered acceptable (Bland & Altman, 1997); see Table 1. The instrument also included nine demographics questions regarding gender, race, ethnicity, and grade in school (Yan & de Guzman, 2011).

Table 1.
Cronbach's Alpha from 2010–2011 University of Nebraska Pilot Test of 4-H Health Rocks! Evaluation Instrument

Group (number of items)	Cronbach's α: Before	Cronbach's α: After
Knowledge (4 items)	.755	.645
Skills (4 items)	.769	.640
Assets (5 items)	.864	.772
Total (13 items)	.913	.831

During the 2011–2012 academic year, the 4-H Health Rocks! curriculum was implemented in Tennessee. The study reported here was approved by the institutional review boards at the University of Tennessee, Knoxville (study number 828913) and University of Tennessee at Martin (study number 15-383-E05-4007). According to Donaldson and Crowe (2013), the statewide 4-H Health Rocks! effort in Tennessee for the 2011–2012 academic year reached 18,850 youths with at least 10 hr of instruction in 17 counties. Given the large population of participants, intact groups totaling 100 youths per county were randomly selected and invited to complete a paper copy of the survey. For example, names of 4-H classes in a given county were randomly drawn and invited as intact groups of 4-H'ers to participate in the survey. The survey had a total of 1,192 respondents, for a response rate of 70% (1,192 of 1,700). The survey data were analyzed through the use of NCSS (Version 10). The data did not represent a normal distribution, and were positively skewed. A nonparametric test, the Wilcoxon signed-rank test, was implemented because the data were skewed and the goal of the analysis was to compare categorical data (Hollingsworth, Collins, Smith, & Nelson, 2011). The Wilcoxon signed-rank test is comparable to the paired-samples t-test, and it was used to compare participants' "before," or "retrospective-pre," and "after" or "post" responses. The Wilcoxon signed-rank test has been used to evaluate Extension risk management (Nagler, Bastian, Hewlett, & Weigel, 2007), food safety (Mathiasen, Morley, Chapman, & Powell, 2012), parenting and youth development (Behnke & Kelly, 2011), and 4-H Camp (Garton, Miltenberger, & Pruett, 2007) programs. Because the Wilcoxon signed-rank test is a nonparametric test, results apply only to the survey respondents rather than the larger population. A significance level of p ≤ .05 was established a priori.

Findings

The survey involved a total of 1,192 respondents. Not every question was answered by every respondent.

Objective One: Describing the Demographic Characteristics of Respondents

Respondents were asked to indicate their gender, race, ethnicity, grade in school, and age. More than half of the respondents were girls (52.43%, 625), a majority of respondents were Caucasian/White (65.02%, 775), and most were not Hispanic/Latino (85.40%, 1,018). Table 2 provides respondent demographic data.

Table 2.
Demographics of 4-H Health Rocks! Program Survey Respondents

Characteristic	f	%
Gender
Female	625	52.43
Male	545	45.72
Not reported	22	1.85
Race
Caucasian/White	775	65.02
African American/Black	190	15.94
Native American	40	3.36
Asian	14	1.17
Multiracial	75	6.29
Not reported	98	8.22
Ethnicity
Hispanic/Latino	72	6.04
Not Hispanic/Latino	1018	85.40
Unknown/not reported	102	8.56
Grade
4th grade	165	13.84
5th grade	252	21.14
6th grade	248	20.81
7th grade	283	23.74
8th grade	113	9.48
9th grade	60	5.03
10th grade and above	51	4.28
Not reported	20	1.68
Age
8 years old and younger	24	2.01
9 years old	82	6.88
10 years old	161	13.51
11 years old	266	22.32
12 years old	261	21.90
13 years old	153	12.84
14 years old	80	6.71
15 years old	38	3.19
16 years old and older	33	2.77
Not reported	94	7.89

Objective Two: Describing Differences, if Any, in the Post and Retrospective-Pre Scores of Respondents Regarding Knowledge, Skills, and Assets

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their knowledge levels before and after participation in the 4-H Health Rocks! program and involved four knowledge statements. With the Wilcoxon signed-rank test as applied in this context,

• negative ranks indicated higher after (post) scores than before (retrospective-pre) scores,
• positive ranks indicated lower after (post) scores than before (retrospective-pre) scores, and
• ties indicated equal after (post) scores and before (retrospective-pre) scores.

Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the four knowledge items, as shown in Table 3.

Table 3.
Wilcoxon Signed-Rank Test Results for Respondent Knowledge Levels Before and After Participation in 4-H Health Rocks! Program

Item	No. of respondents	Mean rank	Z	p-value (2-tailed)
People who smoke can die of lung cancer.	1,114
Negative ranks	365	220.24	13.258	.000***
Positive ranks	70	206.33
Ties	679
Once you start smoking, it is hard to stop.	1,127
Negative ranks	432	264.68	13.829	.000***
Positive ranks	92	252.28
Ties	603
People who use drugs sometimes see or hear things that are not really there.	1,111
Negative ranks	468	291.15	14.723	.000***
Positive ranks	102	259.56
Ties	541
Using drugs can ruin my relationship with my family and friends.	1,120
Negative ranks	400	238.62	13.502	.000***
Positive ranks	76	237.89
Ties	644
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores. ***p ≤ .001.

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their skill levels before and after participation in the 4-H Health Rocks! program and involved four statements about skills. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the four skill items, as shown in Table 4.

Table 4.
Wilcoxon Signed-Rank Test Results for Respondent Skill Levels Before and After Participation in 4-H Health Rocks! Program

Item	No. of respondents	Mean rank	Z	p-value (2-tailed)
If a friend wants to try drugs, I can talk them out of it.	1,109
Negative ranks	449	293.03	13.156	.000***
Positive ranks	124	265.17
Ties	536
When I feel stressed, I am able to talk about it with people I trust.	1,112
Negative ranks	432	264.68	12.778	.000***
Positive ranks	92	252.28
Ties	603
I don't have to drink or smoke even if some other young people do it.	1,120
Negative ranks	468	291.15	12.247	.000***
Positive ranks	102	259.56
Ties	541
I am able to say "no" if others offered me drugs.	1,121
Negative ranks	400	238.62	12.368	.000***
Positive ranks	76	237.89
Ties	644
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores. ***p ≤ .001.

Wilcoxon signed-rank tests were conducted for the purpose of comparing respondents' perceptions of their assets before and after participation in the 4-H Health Rocks! program and involved four statements about assets. All post scores were significantly higher than retrospective-pre scores, as shown in Table 5. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for the five asset items, as shown in Table 5.

Table 5.
Wilcoxon Signed-Rank Test Results for Respondent Assets Before and After Participation in 4-H Health Rocks! Program

Item	No. of respondents	Mean rank	Z	p-value (2-tailed)
It is important for me to stay focused on learning at school.	1,122
Negative ranks	255	161.40	10.760	.000***
Positive ranks	59	140.63
Ties	808
I need to think about how my choices will affect my future.	1,117
Negative ranks	373	223.51	13.733	.000***
Positive ranks	69	210.62
Ties	675
I have goals for myself.	1,115
Negative ranks	299	173.79	12.744	.000***
Positive ranks	45	163.90
Ties	771
I feel good about myself.	1,115
Negative ranks	308	181.45	13.263	.000***
Positive ranks	47	155.37
Ties	760
I would help other kids like me to stay away from alcohol or other drugs.	1,123
Negative ranks	318	195.89	13.271	.000***
Positive ranks	59	151.84
Ties	746
Note. Negative ranks indicate higher "after" ("post") scores than "before" ("retrospective-pre") scores; positive ranks indicate lower after (post) scores than before (retrospective-pre) scores; and ties indicate equal after (post) scores and before (retrospective-pre) scores. ***p ≤ .001.

Conclusions, Discussion, and Recommendations

The 4-H Health Rocks! program had positive, significant impacts on many respondents' self-perceived knowledge, skills, and assets regarding substance abuse prevention, as shown by retrospective post-then-pre survey results. Although results included a large number of ties, mean negative ranks were significantly higher than mean positive ranks for all survey items. Additional data analysis should be undertaken for the purpose of understanding differences, if any exist, between and among groups of respondents, such as between female and male respondents or among respondents in different grades.

In spite of the evidence of positive, significant impacts, however, the Wilcoxon signed-rank test also showed that for all 13 items, the majority of youths reported no change from before to after participation in the program. This is an interesting finding that precipitates discussion and recommendations for both programming and evaluation. Because of youths' relatively high retrospective-pre scores, it appears that the majority of youths possessed the measured knowledge, skills, and assets before participating in the program. However, the program may be highly valuable in other ways. For example, although the majority of youths may possess the targeted skills, the program provides an important opportunity for youths to practice those skills. Also, future program evaluators should consider a different instrument for measuring program effectiveness. With the instrument used in the study described here, the forced-choice responses (1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree) may be too narrow to measure changes in youths' knowledge, skills, and assets.

The limitations of the study reported here should be noted and addressed in future research; most notable are limitations related to program delivery. We could not align specific delivery settings, such as residential camps, afterschool programs, or in-school 4-H club meetings, to respondents' perceptions; nor could we align specific lessons from the curriculum to respondents' perceptions. Also, we suggest that research related to ascertaining the appropriateness of the curriculum for different age groups is warranted. And, finally, although there is value in retrospective evaluation, the retrospective post-then-pre design was a limitation of the research described here. Future research should examine specific practices for conducting post-then-pre surveys with young people, given the limitations of retrospective post-then-pre questionnaires described by Klatt and Taylor-Powell (2005).

We recommend the formation of focus groups to provide further, more in-depth insight into the perceptions of 4-H Health Rocks! participants regarding the content of the program and ways to enhance the curriculum and instruction. This qualitative approach would complement the data already collected and provide a more comprehensive evaluative view of the 4-H Health Rocks! program. Additionally, it is recommended that Extension professionals consider random selection of participants for evaluation of other large-scale, statewide Extension programs.

Acknowledgments

We wish to express appreciation to the individuals and institutions that made our study possible: Dr. Yan Xia and Dr. Maria de Guzman of the University of Nebraska–Lincoln, who served as primary authors of the survey and conducted all validity and reliability studies; Justin Crowe, who coordinated the 4-H Health Rocks! implementation in Tennessee; and all the local Extension agents and volunteers who conducted the 4-H Health Rocks! program. 4-H Health Rocks! was funded by National 4-H Council.

References

Behnke, A. O., & Kelly, C. (2011). Creating programs to help Latino youth thrive at school: The influence of Latino parent involvement programs. Journal of Extension, 49(1), Article 1FEA7. Available at: https://www.joe.org/joe/2011february/a7.php

Bland, J., & Altman, D. (1997). Statistics notes: Cronbach's alpha. British Medical Journal, 314, 275.

Davis, G. A. (2003). Using a retrospective pre-post questionnaire to determine program impact. Journal of Extension, 41(4), Article 4TOT4. Available at: http://www.joe.org/joe/2003august/tt4.php

Donaldson, J. L., & Crowe, J. E. (2013). Healthy decisions. Retrieved from https://extension.tennessee.edu/publications/Documents/PB1811.pdf

Garton, M. S., Miltenberger, M., & Pruett, B. (2007). Does 4-H Camp influence life skill and leadership development? Journal of Extension, 45(4), Article 4FEA4. Available at: https://joe.org/joe/2007august/a4.php

Hollingsworth, R. G., Collins, T. P., Smith, V. E., & Nelson, S. C. (2011). Simple statistics for correlating survey responses. Journal of Extension, 49(5), Article 5TOT7. Available at: https://www.joe.org/joe/2011october/tt7.php

Johnston, L. D., O'Malley, P. M., Bachman, J. G., & Schulenberg, J. E. (2013). Monitoring the future national results on drug use: 2012 overview, key findings on adolescent drug use. Ann Arbor, MI: Institute for Social Research, University of Michigan.

Klatt, J., & Taylor-Powell, E. (2005). Using the retrospective post-then-pre design, quick tips #27. Retrieved from http://www.uwex.edu/ces/pdande/resources/index.html

Kumaran, M., Fogarty, K., Fung, W. M., & Terminello, A. (2015). Improving healthy living youth development program outreach in Extension: Lessons learned from the 4-H Health Rocks! program. Journal of Extension, 53(3), Article 3RIB5. Available at: http://www.joe.org/joe/2015june/rb5.php

Mathiasen, L., Morley, K., Chapman, B., & Powell, D. (2012). Using a training video to improve agricultural workers' knowledge of on-farm food safety. Journal of Extension, 50(1), Article 1FEA6. Available at: https://www.joe.org/joe/2012february/a6.php

Miller, T., & Hendrie, D. (2008). Substance abuse prevention dollars and cents: A cost-benefit analysis. Rockville, MD: U.S. Department of Health and Human Services.

Nagler, A., Bastian, C. T., Hewlett, J. P., & Weigel, R. R. (2007). Risk management for ag families: Evaluation of an integrated educational program for producers on the northern plains. Journal of Extension, 45(3), Article 3RIB3. Available at: https://www.joe.org/joe/2007june/rb3.php

National 4-H Council. (2009). 4-H Health Rocks! Intermediate level. Chevy Chase, MD: National 4-H Council.

National Council on Alcoholism and Drug Dependence Inc. (2015). Alcohol, drugs and youth. Retrieved from https://ncadd.org/for-youth/233-overview

National Institute on Drug Abuse. (2014, January). DrugFacts: Nationwide trends. Retrieved from: http://www.drugabuse.gov/publications/drugfacts/nationwide-trends

NCSS 10 Statistical Software [Computer software]. Kaysville, UT: NCSS.

Raidl, M., Johnson, S., Gardiner, K., Denham, M., Spain, K., Lanting, R., . . . Barron, K. (2004). Use retrospective surveys to obtain complete data sets and measure impact in Extension programs. Journal of Extension, 42(2), Article 2RIB2. Available at: http://www.joe.org/joe/2004april/rb2.php

Rockwell, S. K., & Kohn, H. (1989). Post-then-pre evaluation. Journal of Extension, 27(2), Article 2FEA5. Available at: http://www.joe.org/joe/1989summer/a5.php

Self, C., Morgan, A. C., Fuhrman, N. E., & Navarro, M. (2013). An evaluation of the 4-H Health Rocks! program: Implications for program improvement. Journal of Drug Education, 43(1), 49–63.

Yan, X., & de Guzman, M. (2011). [4-H health rocks survey pilot test]. Unpublished raw data.