National 4-H Common Measures: Initial Evaluation from California 4-H

Introduction

Evaluation is a key component for documenting the effectiveness of a program (Rennekamp & Arnold, 2009). A crucial element of evaluation is having well-developed, valid, and reliable measures to capture expected program outcomes (Radhakrishna, 2007). Additionally, having a set of common measures can aid comparisons across programs (Payne & McDonald, 2012). For example, comparisons of the effectiveness of 4-H across states can be made if the states are evaluating youth using the same measure. At the request of the National Institute of Food and Agriculture (NIFA), National 4-H Council led a process to develop a set of Common Measures for use across the 4-H system. These common measures assess indicators of scientific literacy, healthy living, citizenship, and youth development (e.g., positive choices, communication). A major goal for developing these measures was to have a standard instrument to assess common indicators across the 4-H Youth Development Program. Indeed, the measures are being used across the country and are being required for grants issued through 4-H National Headquarters and National 4-H Council. For more information about these measures, see http://www.4-h.org/about/youth-development-research/.

A challenge of creating any measure is developing an instrument that is valid and reliable (Van Tilburg Norland, 1990), that is, creating instruments that measure what they are intended to measure (validity) and can consistently do so (reliability; Radhakrishna, 2007). Prior to the widespread use of an instrument, information about the reliability (e.g., Cronbach's alpha, test-retest) and validity (e.g., construct, face) should be made available so that potential users of the instrument can make informed decisions about which measures to use in their evaluations (for examples, see Lackman, Neito, & Gliem, 1997; Stewart, Roberts, & Kim, 2009). To date, there is no psychometric information on the National 4-H Common Measures; our goal is to provide such information to aide in measurement refinement and use.

The study reported here had three objectives:

1. Provide descriptive statistics from data collected from California (CA) 4-H youth that can be used for comparison with other states and future time points,
2. Evaluate the measures in terms of reliability and validity, and
3. Make recommendations to the system and users of the Common Measures.

Methods

Data Collection and Sample

Common Measure data were collected from 721 CA 4-H youth in 2012 through 2014, primarily through surveys embedded in the California 4-H Online Record Book (ORB) system. Implemented in 2011 in all California counties, ORB provides an online alternative to paper 4-H Record Books. Sample demographics are presented in Table 1. The sample is predominately female and non-Hispanic White, and dispersed in terms of residence. The mean age of the sample is 14.13 years.

Table 1.
Sample Demographics (N=721)

	N	%
Gender
Female	488	67.7
Male	233	32.3
Ethnicity
Non-Hispanic or Latino	629	87.2
Hispanic or Latino	92	12.8
Race
White	612	84.9
Black or African-American	6	0.8
Asian	33	4.6
American Indian or Alaska Native	6	0.8
Native Hawaiian or other Pacific Islander	1	0.1
Undetermined	63	8.8
Residence Type
Farm	192	26.6
Town (non-farm, rural, population <10,000)	129	17.9
Town or city (population 10,000 - 50,000)	154	21.4
Suburb of city (population > 50,000)	108	15.0
Central city (population > 50,000)	138	19.9
	Mean (SD)
Age
Overall sample	14.13 (2.42)
Grades 4-7 (N=361)	12.11 (1.21)
Grades 8-12 (N=360)	16.15 (1.44)

Measures

Table 2 provides a list of the measures included in this article. For each scale, the development team for the Common Measures proposed different items for younger (Grades 4-7) and older (Grades 8-12) youth. In California, for science, youth of all grades received the measure designed for older youth. Therefore, some of the items from the National 4-H science measure for youth in grades 4-7 were not collected.

Table 2.
Scales and Number of Items for Each Age Group

Measure	Number of Items
	Grades 4-7	Grades 8-12
Science
Attitudes	5	6
Interest	4	4
Skills	3	5
Application	--	4
Total	12	19
Healthy Living
Follow Dietary Guidelines	18	18
Physical Activity	5	3
Risk Prevention Behaviors	12	12
Total	35	33
Citizenship
Awareness of Community Issues	--	7
Cultural Diversity	1	5
Community Engagement	5	9
Understanding the Democratic Process	2	3
Total	8	24
Youth Development
Make Positive Choices	9	10
Effectively Communicate	7	9
Build Connections	4	5
Contribution	5	5
Total	25	29

Analysis Steps

Prior to analyses, we (1) standardized (created z-scores for) items from subscales that varied in response options (e.g., some healthy living items had four response options and some had five response options) and (2) dropped items with extremely low variance (e.g., "When you use a firearm, how often do you follow safety rules"). Exploratory factor analysis (EFA; e.g., Santos & Clegg, 1999) was conducted for each measure to test whether the hypothesized subscales emerged (Table 2). Five steps were followed:

1. Ran an unconstrained EFA (i.e., did not specify how many subscales were expected).
2. Dropped any items that did not clearly belong on at least one subscale (i.e., items that had factor loadings lower than .30 on all factors).
3. Items that loaded onto a factor with only one other item were combined to make a new item that was the average of the two items to avoid problems with reliability of two-item subscales (cf. Widaman, Gibbs, & Geary, 1987).
4. Re-ran the EFA and repeated steps 2 and 3 as needed.
5. Final subscales were retained when a solution was found in which all remaining items worked well.

The CA recommended scales (hereby referred to as "CA scales") were compared with the proposed National scales (hereby referred to as "National scales"). We tested whether the CA scales appear to be as strong and tap the same proposed constructs as the National scales by comparing the alpha reliabilities and correlates with other constructs. In this comparison, the average (across the subscales) was also tested for each scale as this is another common option that evaluators use in analyzing data.

Appendix A shows the full list of items and indicates which items were retained. Appendix B shows correlations between National and CA scales and other outcomes collected by CA. Appendix C shows correlations between the National and CA scales.

Results

Science

• For younger youth, National proposed two subscales; the EFA revealed one factor.
• For older youth, National proposed four subscales; the EFA revealed three.
• Skills and Application subscales came out as proposed by National for older youth.
• For both age groups, Attitudes and Interest emerged as one factor. Subsequent analyses further suggest these two subscales measure the same construct. Specifically,
  • The two subscales were highly correlated (r = .72 for younger youth and .77 for older youth),
  • For both ages, Attitudes and Interest show the same pattern of correlations (see Appendix B), and
  • Multiple regression analyses revealed that for both age groups, the Interest subscale did not significantly add information over and above Attitudes.
• CA recommends using the Skills and Application subscales as proposed by National, and using only the Attitudes subscale because the Interest subscale appears to provide redundant information (younger youth) or is unrelated to outcomes when controlling for Attitudes (older youth).
• Table 3 presents the reliability and descriptive statistics for these results.

Table 3.
Reliability and Descriptive Statistics for the Two Science Measures

Subscale	Sample Size	Number of Items	Alpha	Mean (SD)	Range
Grades 4-7
Attitudes	287	5	.84	3.00 (.58)	1.20, 4.00
Interest	291	4	.84	3.24 (.58)	1.00, 4.00
Average of all items	276	9	.90	3.10 (.54)	1.44, 4.00
Grades 8-12
Attitudes	331	6	.91	2.95 (.63)	1.00, 4.00
Interest	321	4	.86	3.08 (.61)	1.00, 4.00
Skills	336	5	.92	2.92 (.83)	1.00, 4.00
Application	328	4	.60	0.56 (.34)	0.00, 1.00
Average of all itemsa	337	19	.91	0.00 (.59)	-2.19, 1.27
asubscale that requires item standardization prior to analyses for the National versions.

Citizenship

• National proposed three factors for younger youth and four factors for the older youth, but the EFA suggested one overall Citizenship Scale for both age groups.
• Because an overall Citizenship factor emerged for both older and younger youth, CA recommends using the items that overlap between the age groups to remove unnecessary items and provide a common measure across age (see Appendix A).
• The correlations patterns were similar between the National and CA scales for younger and older youth (see Appendix B).
• The CA and National subscales were highly correlated, suggesting that these measure the same construct (see Appendix C).
• Table 4 presents the reliability and descriptive statistics for these results.

Table 4.
Reliability and Descriptive Statistics for the Two Citizenship Measures

	Sample Size		Number of Items		Alpha		Mean (SD)		Range
Subscale	Nat	CA	Nat	CA	Nat	CA	Nat	CA	Nat	CA
Citizenship-Grades 4-7
Cultural Diversity	268	--	1	--	--	--	3.32 (.59)	--	1.00, 4.00	--
Community engagement	270	--	5	--	.76	--	3.45 (.47)	--	1.80, 4.00	--
Democratic process	266	--	2	--	.54	--	3.38 (.49)	--	2.00, 4.00	--
Average of items	270	--	8	8	.83	.83	3.41 (.42)	3.41 (.42)	2.25, 4.00	2.25, 4.00
Citizenship-Grades 8-12
Awareness of community issues	191	--	7	--	.84	--	3.12 (.44)	--	2.00, 4.00	--
Cultural Diversity	195	--	5	--	.77	--	3.32 (.46)	--	2.20, 4.00	--
Community engagement	196	--	9	--	.81	--	3.27 (.46)	--	1.00, 4.00	--
Understanding of democratic process	191	--	3	--	.76	--	3.36 (.48)	--	1.67, 4.00	--
Average of items	196	196	24	8	.92	.87	3.24 (.41)	3.41 (.42)	1.00, 4.00	2.00, 4.00
Note. Nat = National Scale; CA = California Scale. For scales with only two items, the correlation for the two items is presented instead of an alpha.

Healthy Living

• National proposed three subscales for each age group.
• All items for both age groups had large amounts of missing data, possibly due in part to the non-applicability of the item; e.g., "I follow safety rules when using a firearm" and in part due to the length of the scale.
• The sample size for both age groups was not large enough to conduct an EFA, as few youth completed all the questions within the measure. Therefore, we only present descriptive statistics for the National recommended subscales in Table 5.
• For all youth, the National subscales show low to moderate correlations with related CA outcomes (see Appendix B) and low to moderate correlations among themselves (see Appendix C). This is expected from scales that tap separate but related constructs.
• CA will continue to collect data on these measures. Additionally, 4-H evaluators should review the measures for item applicability before utilizing the measure. For example, the item regarding firearm usage may be applicable only to a few youth that are in a shooting sports project.

Table 5.
Descriptive Statistics for the Two Healthy Living Measures

Subscale	Sample Size	Number of Items	Alpha	Mean (SD)	Range
Healthy Living-Grades 4-7
Follow Dietary Guidelinesa	228	18	.81	-0.00 (.50)	-2.04, 2.16
Physical Activitya	227	5	.70	-0.04 (.74)	-5.71, 0.46
Risk Prevention Behaviors	277	12	.93	3.42 (.49)	1.00, 4.00
Average of all itemsa	229	35	--	0.00 (.45)	-2.33, 1.44
Healthy Living-Grades 8-12
Follow Dietary Guidelinesa	169	18	--	-0.04 (.69)	-2.63, 1.29
Physical Activitya	181	3	.49	-0.06 (.73)	-2.14, 1.88
Risk Prevention Behaviorsa	183	12	--	-0.03 (.54)	-1.77, 1.14
Average of all itemsa	185	33	--	-0.05 (.52)	-2.63, 1.24
Note. Because alpha reliabilities and EFA are conducted on the sample of youth that completed all the items within a subscale, some sample sizes were too low for analysis; therefore alpha and EFA results are not presented. We note that all subscales for all measures were computed such that a youth received a score for the subscale if they answered at least one question within the subscale. For example, if youth A answered two questions in the "Risk Prevention Subscale" and youth B answered all 12, both youth received a score based on the mean of the items to which they responded. This represents a liberal approach and other researchers may wish to calculate more conservative scores. aindicates subscales that require item standardization prior to analyses for the National versions.

Youth Development

• National proposed four subscales for each age group.
• All items for both age groups had large amounts of missing data. As with healthy living, the sample size for both age groups was not large enough to conduct an EFA. Descriptive statistics for the National recommended subscales are presented in Table 6.
• For all youth, the National measures show low to high correlations with related CA outcomes (see Appendix B).
• Both age groups showed high correlations with a measure of positive youth development, suggesting that the youth development measure is tapping into a similar construct.
• For all youth, correlations of National's subscales among themselves were high (see Appendix C).
• CA will continue to collect data on these measures for future analyses.

Table 6.
Descriptive Statistics for the Two Youth Development Measures

Subscale	Sample Size	Number of Items	Alpha	Mean (SD)	Range
Grades 4-7
Positive choices	173	9	.86	3.30 (0.45)	2.22, 4.00
Communication	173	7	.78	3.25 (0.48)	2.00, 4.00
Connections	173	4	.76	3.43 (0.48)	1.75, 4.00
Contribution	166	5	.84	3.35 (0.48)	2.00, 4.00
Average of all items	174	25	--	3.30 (0.43)	2.00, 4.00
Grades 8-12
Positive choices	86	10	--	3.38 (0.42)	2.50, 4.00
Communication	86	9	--	3.34 (0.46)	2.33, 4.00
Connections	86	5	--	3.46 (0.48)	2.40, 4.00
Contribution	82	5	--	3.27 (0.55)	1.00, 4.00
Average of all items	86	29	--	3.35 (0.42)	2.13, 4.00
Note. Because alpha reliabilities and EFA are conducted on the sample of youth that completed all the items within a subscale, some sample sizes were too low for analysis; therefore this information is not presented.

Recommendations and Best Practices

We recommend that the measures undergo further refinement before broad use. CA recommendations are the following.

1. Keep item response options consistent across all items for each scale. The current measures propose subscales that include items with varying response categories, in terms of both labels and the number of categories. Items with different response categories cannot be used to form a scale unless the items are first standardized. This requirement may result in a loss of valuable information, and it will be easy for researchers to miss this step and conduct analyses that are not meaningful.
2. Whenever possible do not use items that vary across age groups. When items vary across age groups it is not possible to study developmental trends (unless very sophisticated statistical models and analyses are used). That is, when item content changes across age it is no longer possible to disentangle differences due to age versus differences due to the change in wording or item content. Thus, long-term impacts of a program cannot be assessed. We note, however, that longitudinal analysis was not one of the original goals of National 4-H, and states may not be interested in evaluating youth outcomes over time.
3. Make item wording more general. The wording of items is specific to youth involved in 4-H. Unless the items are reworded, potentially changing the psychometrics of the items, comparisons of youth in other programs cannot be made. Again, we note that this was not one of the original goals, and states may not be interested in comparing youth in other programs.
4. Have a core set of items that are applicable for most youth, with additional "supplemental" items that may be project-specific. For example, the item "I follow safety rules when using a firearm" in not applicable to most youth, lending to the extremely low variance of the item.

For 4-H evaluators and practitioners utilizing the measures, we recommend the following.

1. Science: Use the Attitudes, Skills, and Application subscales.
2. Citizenship: Use the items present in the scale for youth ages nine to 13 years for all ages.
3. Healthy Living: Use the measure as described by National 4-H until further analyses can be conducted. Items with limited relevance to youth (e.g. "I wear a helmet when riding an all-terrain vehicle") should only be included when relevant to the 4-H project.
4. Youth Development: Use the measure as described by National 4-H until further analyses can be conducted.

Summary

A common set of tools to measure youth outcomes across programs and states is useful for program evaluation (Payne & McDonald, 2012). In the study reported here, we found support for several of the subscales as proposed by National 4-H; however, some subscales did not emerge, and missing data and extremely low variance on some items made it difficult to conduct analyses. Data were drawn from the community club program of 4-H, and that the demographics of youth who participate in this delivery mode are consistent throughout the state. However, this is a limitation of our study that may limit the generalizability of the findings to more diverse populations. Other programs that pilot these measures on more diverse samples should publish their psychometric findings for comparison. Appendices and EFA and scale creation syntax are available from http://4h.ucanr.edu/Research/4HPublications/commonmeasure/.

References

Lackman, L., Nieto, R. D., & Gliem, R. (1997). Instrument development for low literacy audiences: Assessing Extension program personnel teaching effectiveness. Journal of Extension [On-line], 35(1) Article 1RIB1. Available at: http://www.joe.org/joe/1997february/rb1.php

Payne, P. B., & McDonald, D. A. (2012). Using common evaluation instruments across multi-site community programs: A pilot study. Journal of Extension [On-line], 50(4) Article 4RIB2. Available at http://www.joe.org/joe/2012august/rb2.php

Radhakrishna, R. B. (2007). Tips for developing and testing questionnaires/instruments. Journal of Extension [On-line], 45(1) Article 1TOT2. Available at: http://www.joe.org/joe/2007february/tt2.php

Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation practice. Journal of Extension [On-line], 47(3) Article 3COM1. Available at: http://www.joe.org/joe/2009june/comm1.php

Santos, J. R. A., & Clegg, M. D. (1999). Factor analysis adds new dimension to extension surveys. Journal of Extension [On-line], 37(5) Article 5RIB6. Available at: http://www.joe.org/joe/1999october/rb6.php

Stewart, P. K., Roberts, M. C., & Kim, K. L. (2010). The psychometric properties of the Harter Self-perception Profile for Children with at-risk African American females. Journal of Child and Family Studies, 19, 326-333. doi: 10.1007/s10826-009-9302-x

Van Tilburg Norland, E. (1990). Controlling error in evaluation instruments. Journal of Extension [On-line], 28(2) Article 2TOT2. Available at: http://www.joe.org/joe/1990summer/tt2.php

Widaman, K. F., Gibbs, K. W., & Geary, D. C. (1987). Structure of adaptive behavior: I. Replication across fourteen samples of nonprofoundly mentally retarded people. American Journal of Mental Deficiency, 91, 348-360. Retrieved from: http://psycnet.apa.org/psycinfo/1987-15966-001

Appendix A

List of Common Measure Items

Science Literacy-Grades 4 to 7

		National Scale			California Scale
Item #	Item	Subscale	Factor Loading	CA recommended edits	Subscale	Factor Loading
1.	I like science	Attitudes	.82	--	Attitudes	.82
2.	I am good at science	Attitudes	.76	--	Attitudes	.71
3.	I would like to have a job related to science	Attitudes	.81	--	Attitudes	.67
4.	I do science activities that are not for school	Attitudes	.77	--	Attitudes	.70
5.	I think science is useful for solving everyday problems	Attitudes	.80	--	Attitudes	.67
6.	I like to see how things are made or invented	Interest	.85	--	Attitudes	.69
7.	I like experimenting and testing ideas	Interest	.80	--	Attitudes	.67
8.	I get excited about new discoveries	Interest	.83	--	Attitudes	.69
9.	I want to learn more about science	Interest	.80	--	Attitudes	.83
10.	I can do an experiment to answer a question	Skills	Data not collected on this item
11.	I can tell others how to do an experiment	Skills	Data not collected on this item
12.	I can explain why things happen in an experiment	Skills	Data not collected on this item
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Science Literacy-Grades 8 to 12

		National Scale			California Scale
Item #	Item	Subscale	Factor Loading	CA recommended edits	Subscale	Factor Loading
1.	I like science	Attitudes	.87	--	Attitudes	.89
2.	I am good at science	Attitudes	.85	--	Attitudes	.84
3.	I would like to have a job related to science	Attitudes	.85	--	Attitudes	.79
4.	I do science activities that are not for school	Attitudes	.81	--	Attitudes	.83
5.	I think science will be important in my future	Attitudes	.87	--	Attitudes	.82
6.	I think science is useful for solving everyday problems	Attitudes	.77	--	Attitudes	.55
7.	I like to see how things are made or invented	Interest	.83	Exclude	--	--
8.	I like experimenting and testing ideas	Interest	.87	Exclude	--	--
9.	I get excited about new discoveries	Interest	.84	Exclude	--	--
10.	I want to learn more about science	Interest	.82	Exclude	--	--
11.	I can use scientific data to form a question	Skills	.83	--	Skills	.70
12.	I can design a scientific procedure to answer a question	Skills	.89	--	Skills	.90
13.	I can use data to create a graph for presentation to others	Skills	.88	--	Skills	.86
14.	I can create a display to communicate my data and observations	Skills	.85	--	Skills	.83
15.	I can use science terms to share my results	Skills	.87	--	Skills	.85
16.	I have helped with a community service project that relates to science (for example: planted trees or garden, road or stream clean-up, recycling)	Application	.79	--	Application	.69
17.	In my 4-H program, I used science tools to help in the community (for example: mapped with GIS, tested water quality)	Application	.92	--	Application	.92
18.	I taught others about science (for example: demonstrated, gave presentation, led a project)	Application	.89	--	Application	.86
19.	I organized or led science-related events (for example: science fair, environmental festival)	Application	.92	--	Application	.90
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Citizenship-Grades 4-7

		National Scale			California Scale
Item #	Item	Subscale	Factor Loading	CA recommended edits	Subscale	Factor Loading
1.	I enjoyed learning about people who are different from me	Cultural Diversity	--	--	Citizenship	.64
2.	I can make a difference in my community through community service	Community Engagement	.78	--	Citizenship	.69
3.	I help make sure everyone gets an opportunity to say what they think	Understanding Democratic Process	.88	--	Citizenship	.69
4.	I can apply knowledge in ways that solve "real-life" problems through community service	Community Engagement	.79	--	Citizenship	.68
5.	I gained skills through serving my community that will help me in the future	Community Engagement	.79	--	Citizenship	.67
6.	I treat everyone fairly and equally when I am in charge of a group	Understanding Democratic Process	.88	--	Citizenship	.56
7.	I plan to work on projects to better my community	Community Engagement	.71	--	Citizenship	.52
8.	I am encouraged to volunteer more	Community Engagement	.91	--	Citizenship	.52
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Citizenship-Grades 8-12

		National Scale			California Scale
Item #	Item	Subscale	Factor Loading	CA recommended edits	Subscale	Factor Loading
1.	I pay attention to news events that affect my community	Awareness of Community & Community Issues	.57	Exclude	--	--
2.	I am aware of the important needs in my community	Awareness of Community & Community Issues	.75	Exclude	--	--
3.	I really care about my community	Awareness of Community & Community Issues	.68	Exclude	--	--
4.	I talk to my friends about issues affecting my community, state, or world	Awareness of Community & Community Issues	.68	Exclude	--	--
5.	I'm interested in others' opinions about public issues	Awareness of Community & Community Issues	.78	Exclude	--	--
6.	I listen to everyone's views whether I agree or not	Awareness of Community & Community Issues	.65	Exclude	--	--
7.	When I hear about an issue, I try to figure out if they are just telling one side of the story	Awareness of Community & Community Issues	.65	Exclude	--	--
8.	I explore cultural differences	Cultural Diversity	.78	Exclude	--	--
9.	I value learning about other cultures	Cultural Diversity	.81	Exclude	--	--
10.	I respect people from different cultures	Cultural Diversity	.73	Exclude	--	--
11.	I have learned about people who are different from me	Cultural Diversity	.73	--	Citizenship	.56
12.	I can make a difference in my community through community service	Community Engagement	.72	--	Citizenship	.71
13.	I help make sure everyone gets an opportunity to say what they think	Understanding Democratic Process	.81	--	Citizenship	.75
14.	I can apply knowledge in ways that solve "real-life" problems through community service	Community Engagement	.71	--	Citizenship	.73
15.	I gained skills through serving my community that will help me in the future	Community Engagement	.73	--	Citizenship	.80
16.	I treat everyone fairly and equally when I am in charge of a group	Understanding Democratic Process	.86	--	Citizenship	.73
17.	I am able to lead a group in making a decision	Understanding Democratic Process	.71	Exclude	--	--
18.	I would enjoy hosting someone from another culture	Cultural Diversity	.44	Exclude	--	--
19.	I can contact someone I've never met before to get their help with a problem	Community Engagement	.50	Exclude	--	--
20.	I plan to work on projects to better my community	Community Engagement	.78	--	Citizenship	.65
21.	I am encouraged to volunteer more	Community Engagement	.70	--	Citizenship	.51
22.	After high school I will continue to work to better my community	Community Engagement	.73	Exclude	--	--
23.	I am interested in a career that helps others	Community Engagement	.55	Exclude	--	--
24.	I am interested in working in government (such as school board, Director of parks and rec, legislator, legislative aide, intern)	Community Engagement	.38	Exclude	--	--
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Healthy Living-Grades 4-7

Item #	Item	Subscale	Factor Loading
1.	Eat fruit for a snack	Dietary Guidelines	-.10
2.	Eat vegetables for a snack	Dietary Guidelines	-.17
3.	Choose water instead of soda pop or Kool-Aid when I am thirsty.	Dietary Guidelines	-.01
4.	Drink 1% or skim milk instead of 2% or whole milk	Dietary Guidelines	-.23
5.	Choose a small instead of a large order of French fries	Dietary Guidelines	-.07
6.	Eat smaller servings of high fat foods like French fries, chips, snack cakes, cookies, or ice cream	Dietary Guidelines	-.17
7.	Eat a low-fat snack like pretzels instead of chips	Dietary Guidelines	-.04
8.	Drink less soda pop	Dietary Guidelines	-.00
9.	Drink less Kool-Aid	Dietary Guidelines	.01
10.	I do moderate physical activities like walking, helping around the house, raking leaves, or using the stairs	Physical Activity	.38
11.	I exercise 30-60 minutes every day	Physical Activity	.22
12.	Being active is fun	Physical Activity	.76
13.	Being active is good for me	Physical Activity	.89
14.	Physical activity will help me stay fit	Physical Activity	.91
15.	I learned the foods that I should eat every day	Dietary Guidelines	.81
16.	I learned what makes up a balanced diet	Dietary Guidelines	.80
17.	I learned why it is important for me to eat a healthy diet	Dietary Guidelines	.89
18.	I learned how to make healthy food choices	Dietary Guidelines	.88
19.	I eat more fruits and vegetables	Dietary Guidelines	.77
20.	I eat more whole grains	Dietary Guidelines	.75
21.	I eat less junk foods	Dietary Guidelines	.60
22.	I drink more water	Dietary Guidelines	.73
23.	I encourage my family to eat meals together	Dietary Guidelines	.71
24.	When I cook food I am safe and careful	Risk Prevention	.88
25.	If I am sick, I ask an adult before taking medicine	Risk Prevention	.84
26.	I wear a helmet when I ride a bicycle	Risk Prevention	.61
27.	I wear a helmet when I rollerblade or ride a skateboard	Risk Prevention	.62
28.	I wear a helmet when riding an All-Terrain Vehicle	Risk Prevention	.69
29.	I follow safety rules when using a firearm or bow	Risk Prevention	.86
30.	I wear reflective clothing when walking after dark	Risk Prevention	.64
31.	I use a pedestrian crossing when crossing the road	Risk Prevention	.85
32.	I tell my friends what I think when they are going to do something unsafe	Risk Prevention	.75
33.	I avoid using substances that could harm me	Risk Prevention	.86
34.	I wear a seat belt when riding in a car	Risk Prevention	.88
35.	I avoid riding in cars with unsafe drivers	Risk Prevention	.86
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Healthy Living-Grades 8-12

Item #	Item	Subscale	Factor Loading
1.	I learned about the foods that I should eat every day	Dietary Guidelines	.78
2.	I learned what makes up a balanced diet	Dietary Guidelines	.82
3.	I learned why it is important for me to eat a healthy diet	Dietary Guidelines	.73
4.	I learned how to make healthy food choices	Dietary Guidelines	.78
5.	I learned how many calories I need to eat each day	Dietary Guidelines	.60
6.	I learned the importance of fruits and vegetables in my diet	Dietary Guidelines	.74
7.	I learned the importance of whole grains in my diet	Dietary Guidelines	.66
8.	I think about what foods my body needs during the day	Dietary Guidelines	.77
9.	I make food choices based on what I know my body needs	Dietary Guidelines	.71
10.	I make healthy food choices whenever I can	Dietary Guidelines	.69
11.	I match my food intake to the number of calories I need to eat each day	Dietary Guidelines	.43
12.	I eat more fruits and vegetables	Dietary Guidelines	.55
13.	I eat more whole grains	Dietary Guidelines	.63
14.	I eat less junk foods	Dietary Guidelines	.63
15.	I drink less soda	Dietary Guidelines	.59
16.	I drink more water	Dietary Guidelines	.45
17.	I encourage my family to eat meals together	Dietary Guidelines	.41
18.	When I cook food, I am safe and careful	Risk Prevention	.68
19.	If I am sick, I ask an adult before taking medicine	Risk Prevention	.69
20.	I wear reflective clothing when walking after dark	Risk Prevention	.71
21.	I use pedestrian crossings when crossing the road	Risk Prevention	.79
22.	I tell my friends what I think when they are going to do something unsafe	Risk Prevention	.68
23.	I avoid using substances that could harm me	Risk Prevention	.63
24.	My family eats at least one meal a day together	Dietary Guidelines	-.06
25.	During the past 7 days, on how many days were you physically active for a total of at least 60 minutes per day (add up all the time you spent in any kind of physical activity that increased your heart rate and made you breathe hard some of the time)	Physical Activity	.54
26.	On an average school day, how many hours do you spend watching television?	Physical Activity	.76
27.	On an average school day, how many hours do you play video games, looking at a computer, smartphone or tablet for something that is not for school?	Physical Activity	.79
28.	When you ride a bicycle how often do you wear a helmet	Risk Prevention	.64
29.	When you rollerblade or skateboard how often do you wear a helmet?	Risk Prevention	.47
30.	When you ride an All-Terrain Vehicle (ATV) how often do you wear a helmet?	Risk Prevention	-.11
31.	When you use a firearm, how often do you follow safety rules?	Risk Prevention	.32
32.	How often do you use a seatbelt when riding in a car?	Risk Prevention	-.01
33.	Have you ever ridden in a car driven by someone who had been drinking alcohol?	Risk Prevention	.13
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Youth Development-Grades 4-7

Item #	Item	Subscale	Factor Loading
1.	I use information to make decisions	Positive choices	.59
2.	I set goals for myself	Positive choices	.62
3.	I take responsibility for my actions	Positive choices	.59
4.	I listen well to others	Communication	.68
5.	I am respectful of others	Communication	.66
6.	I have the confidence to speak in front of groups	Communication	.59
7.	I can work things out when others don't agree with me	Communication	.75
8.	I work well with other youth	Connection	.67
9.	I am comfortable making my own decisions	Positive choices	.75
10.	I have a plan for reaching my goals	Positive choices	.74
11.	I know how to deal with stress in positive ways	Positive choices	.76
12.	I can explain my decisions to others	Positive choices	.72
13.	I can change my plan when I need to	Positive choices	.77
14.	I don't let my friends talk me into doing something I don't want to do	Positive choices	.66
15.	I am comfortable sharing my thoughts and feelings with others	Communication	.74
16.	I can use technology to help me express my ideas	Communication	.67
17.	I know who I can go to if I need help with a problem	Communication	.55
18.	I can work successfully with adults	Connection	.79
19.	I have friends who care about me	Connection	.81
20.	I am connected to adults who are not my parents	Connection	.80
21.	I am someone who wants to help others	Contribution	.80
22.	I like to work with others to solve problems	Contribution	.82
23.	I have talents I can offer to others	Contribution	.79
24.	I learned things that helped me make a difference in my community	Contribution	.77
25.	I helped with a project that made a difference in my community	Contribution	.72
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor

Youth Development-Grades 8-12

Item #	Item	Subscale	Factor Loading
1.	I use information to make decisions	Positive choices	.56
2.	I set goals for myself	Positive choices	.64
3.	I take responsibility for my actions	Positive choices	.63
4.	I can explain why my decision is a good one	Positive choices	.68
5.	I consider the consequences of my choices	Positive choices	.71
6.	I can resist negative social pressures	Positive choices	.60
7.	I listen well to others	Communication	.59
8.	I am respectful of others	Communication	.67
9.	I have the confidence to speak in front of groups	Communication	.59
10.	I can resolve differences with others in positive ways	Communication	.66
11.	I work well with other youth	Connection	.43
12.	I am comfortable making my own decisions	Positive choices	.53
13.	I have a plan for reaching my goals	Positive choices	.72
14.	I know how to deal with stress in positive ways	Positive choices	.71
15.	I can make alternative plans if something doesn't work	Positive choices	.76
16.	I am comfortable sharing my thoughts and feelings with others	Communication	.80
17.	I can use technology to help me express my ideas	Communication	.67
18.	I know who I can go to if I need help with a problem	Communication	.74
19.	I am willing to consider the ideas of others even if they are different than mine	Communication	.77
20.	I can stand up for things that are important to me	Communication	.75
21.	I can work successfully with adults	Connection	.82
22.	I have friends who care about me	Connection	.83
23.	I know community leaders who support me	Connection	.91
24.	I have adults in my life who care about me and are interested in my success	Connection	.89
25.	I am someone who wants to help others	Contribution	.89
26.	I like to work with others to solve problem	Contribution	.90
27.	I have talents I can offer to others	Contribution	.82
28.	I learned things that helped me make a difference in my community	Contribution	.90
29.	I led a project that made a difference in my community	Contribution	.78
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Appendix B

Correlations of the National Scales and California Scales with other Outcomes

Science-Grades 4 to 7

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Attitudes	.29**	-.17**	-.06	.22**	.24**	.15	-.00	.35**
Interest	.25**	-.16**	-.03	.20**	.22**	.17*	.02	.44**
Average	.30**	-.18**	-.05	.23**	.25**	.19*	.01	.42**
California
Attitudes	.30**	-.18**	-.05	.23**	.25**	.19*	.01	.42**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Science-Grades 8 to 12

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Attitudes	.10	-.24**	-.09	.16**	.19**	.23**	.10	.27**
Interest	.10	-.20**	-.02	.16**	.16**	.19**	.13*	.31**
Skills	.13*	-.21**	-.08	.16**	.22**	.23**	-.06	.25**
Application	-.02	-.05	-.03	.08	-.01	.03	.30**	.10
Average	.12*	-.26**	-.08	.21**	.21**	.26**	.15**	.35**
California
Attitudes	.10	-.24**	-.09	.16**	.19**	.23**	.10	.27**
Skills	.13*	-.21**	-.08	.16**	.22**	.23**	-.06	.25**
Application	-.02	-.05	-.03	.08	-.01	.03	.30**	.10
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Citizenship-Grades 4 to 7

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Cultural Diversity	.19**	-.20**	-.07	.19*	.24**	.21*	.08	.50**
Community Engagement	.23**	-.22*	.03	.23**	.40**	.24**	.10	.61**
Democratic Process	.34**	-.26**	-.10	.38**	.17*	.36**	.14*	.59**
Average	.29**	-.25**	-.06	.29**	.36**	.30**	.13*	.67**
California
Citizenship	.29**	-.25**	-.06	.29**	.36**	.30**	.13*	.67**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Citizenship-Grades 8 to 12

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Awareness of community issues	.20**	-.24**	-.18*	.27**	.30**	.48**	.06	.58**
Cultural Diversity	.19*	-.06	.09	.19**	.31**	.18	-.02	.46**
Community engagement	.15*	-.24**	-.08	.20**	.38**	.41**	.07	.57**
Democratic Process	.24**	-.18*	-.04	.24**	.30**	.25**	.03	.66**
Average	.17*	-.25**	-.09	.25**	.39**	.45**	.04	.65**
California
Citizenship	.19**	-.18*	-.02	.23**	.40**	.42**	.06	.70**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Healthy Living-Grades 4 to 7

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Follow Dietary Guidelines	.10	-.05	.14*	.08	.10	.14	.16*	.19**
Physical Activity	.12	-.14*	-.09	.19**	.06	.28**	.02	.25**
Risk Prevention Behaviors	.20**	-.23**	-.07	.19*	.21**	.28**	.10	.42**
Average	.17*	-.14*	.10	.18**	.16*	.27**	.13	.34**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Healthy Living-Grades 8 to 12

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Follow Dietary Guidelines	.18*	-.15	-.20	.21**	.27**	.35**	.13	.47**
Physical Activity	-.05	-.12	-.10	-.09	-.08	.24*	.05	.25**
Risk Prevention Behaviors	.23**	-.16*	-.13	.06	.19*	.33**	.15*	.33**
Average	.16*	-.19*	-.07	.13	.29**	.45**	.20**	.50**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Youth Development-Grades 4 to 7

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Choices	.13	-.36**	-.14	.33**	.31**	.52**	.11	.67**
Communication	.16**	-.30**	-.16*	.36**	.35**	.47**	.05	.64**
Connections	.21**	-.36**	-.20*	.37**	.30**	.48**	.06	.69**
Contribution	.07	-.33**	-.14	.29**	.27**	.36**	.19*	.62**
Average	.17**	-.40**	-.19*	.36**	.34**	.49**	.11	.72**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Youth Development-Grades 8 to 12

	Mindset	Stress	Depression	Self Esteem	Spark	Goals	YSOC	PYD
National
Choices	.31**	-.26*	-.11	.24*	.42**	.58**	.00	.62**
Communication	.25*	-.14	-.04	.23*	.40**	.48**	-.05	.59**
Connections	.29**	-.24*	-.17	.28*	.16	.38**	-.09	.43**
Contribution	.28*	-.13	-.01	.22*	.29*	.40**	.01	.45**
Average	.31**	-.22*	.09	.28**	.39**	.53**	-.03	.63**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Appendix C

Correlations between National Scales and California Scales

Science-Grades 4 to 7

	Nat-Attitudes	Nat-Interest	Nat-Average/CA-Attitudes
Nat-Attitudes	--	.72**	.94**
Nat-Interest	.72**	--	.91**
Nat-Average/CA-Attitudes	.94**	.91**	--
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.

Science-Grades 8 to 12

	Nat/CA-Attitudes	Nat-Interest	Nat/CA-Skills	Nat/CA-Application	Nat-Average
Nat/CA-Attitudes	--	.77**	.53**	.03	.85**
Nat-Interest	.77**	--	.50**	.05	.83**
Nat/CA-Skills	.53**	.50**	--	-.01	.77**
Nat/CA-Application	.03	.05	-.01	--	.28**
Nat-Average	.85**	.83**	.77**	.28**	--
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.

Citizenship-Grades 4 to 7

	Nat-Cultural Diversity	Nat-Community Engagement	Nat-Democratic Process	Nat-Average/ CA Citizenship
Nat-Cultural Diversity	--	.52**	.50**	.67**
Nat-Community Engagement	.52**	--	.56**	.94**
Nat-Democratic Process	.50**	.56**	--	.77**
Nat-Average/CA-Citizenship	.69**	.94**	.77**	--
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.

Citizenship-Grades 8 to 12

	Nat-Awareness of community issues	Nat-Cultural Diversity	Nat-Community Engagement	Nat- Democratic Process	Nat-Average	CA-Citizenship
Nat-Awareness of community issues	--	.64**	.66**	.60**	.88**	.67**
Nat-Cultural Diversity	.64**	--	.51**	.56**	.75**	.63**
Nat-Community Engagement	.67**	.51**	--	.62**	.91**	.86**
Nat- Democratic Process	.60**	.56**	.62**	--	.77**	.80**
Nat-Average	.88**	.75**	.91**	.77**	--	.89**
CA-Citizenship	.67**	.63**	.86**	.80**	.89**	--
*p < .05, **p < .01. Nat = National Scales, CA = California Scales.

Healthy Living-Grades 4 to 7

	Nat-Follow Dietary Guidelines	Nat-Physical Activity	Nat-Risk Prevention Behaviors	Nat-Average
Nat-Follow Dietary Guidelines	--	.09	.45*	.85**
Nat-Physical Activity	.09	--	.17*	.42**
Nat-Risk Prevention Behaviors	.45**	.17*	--	.77**
Nat-Average	.85**	.42**	.77**	--
*p < .05, **p < .01. Nat = National Scale.

Healthy Living-Grades 8 to 12

	Nat-Follow Dietary Guidelines	Nat-Physical Activity	Nat-Risk Prevention Behaviors	Nat-Average
Nat-Follow Dietary Guidelines	--	.03	.49**	.85**
Nat-Physical Activity	.03	--	.03	.30**
Nat-Risk Prevention Behaviors	.49**	.03	--	.77**
Nat-Average	.85**	.30**	.77**	--
*p < .05, **p < .01. Nat = National Scale.

Youth Development- Grades 4 to 7

	Nat-Choices	Nat-Communicate	Nat-Connections	Nat-Contribution	Nat-Average
Nat-Choices	--	.78**	.74**	.74**	.93**
Nat-Communicate	.78**	--	.74**	.69**	.91**
Nat-Connections	.74**	.74**	--	.78**	.87**
Nat-Contribution	.74**	.69**	.78**	--	.85**
Nat-Average	.93**	.91**	.87**	.85**	--
*p < .05, **p < .01. Nat = National Scale.

Youth Development- Grades 8 to 12

	Nat-Choices	Nat-Communicate	Nat-Connections	Nat-Contribution	Nat-Average
Nat-Choices	--	.82**	.68**	.61**	.91**
Nat-Communicate	.82**	--	.78**	.70**	.94**
Nat-Connections	.68**	.78**	--	.72**	.84**
Nat-Contribution	.61**	.70**	.72**	--	.83**
Nat-Average	.91**	.94**	.84**	.83**	--
*p < .05, **p < .01. Nat = National Scale.