 |
June 2015
|
June 2015 // Volume 53 // Number 3 // Research In Brief // v53-3rb4
Examination of Attitude and Interest Measures for 4-H Science Evaluation
Abstract
Science education research has demonstrated the influence of affect on learning. The National 4-H Science Logic Model outlines outcomes from youth participation in 4-H science programs, which includes attitude and interest outcomes. The associated measure, the National 4-H Science Common Measure, assesses these attitude constructs and not other affective factors. Tie study reported here sought to determine whether additional affective constructs were separate from the general constructs assessed on the Common Measure. We found the additional measures have good reliability and moderate correlations among the outcomes, suggesting the new measures assess different constructs than currently assessed by the Common Measure.
Introduction
The development of positive attitudes towards, interests in, and growth mindsets for science is essential to advancing scientific literacy among youth (National Research Council [NRC], 2009). Research from science education has shown the impact of affective factors on motivation, cognitive engagement, and future aspirations (e.g., Irvin, Meltzer, & Dukes, 2007). For example, the NRC (2009) report included interest in their strands of science learning: strand 1 "addresses motivation to learn science, emotional engagement with it, curiosity, and willingness to persevere over time despite encountering challenging scientific ideas and procedures over time" (p. 43). A study of science learning in afterschool revealed two important affective outcomes: developing young people's interest and curiosity in science and helping youth become aware and value science (Krishnamurthi, Bevan, Rinehart, & Coulon, 2013). Despite the importance of affect in learning, there continues to be a paucity of published research on affect in science education (Fortus, 2014).
Advancing scientific literacy is a national priority and a core mission of the 4-H Youth Development Program (Schmitt-McQuitty, Carlos, & Smith, 2014; Worker, 2013). This mission developed in part from research showing that science achievement in the U.S. is lower compared to other countries and that science is a necessary part of our everyday lives (Heck, Carlos, Barnett, & Smith, 2012). The National 4-H Science Logic Model was developed in 2007 to help guide program planning by outlining the expected outcomes from participation in 4-H science education programs. The logic model includes general interest/attitude goals, including "improved attitudes towards science among youth," "increased awareness of science among youth," and "youth express interest/demonstrate aspirations towards science careers" (4-H Science Logic Model, 2010). In order to assess these outcomes, National 4-H developed the 4-H Science common measure (CM), which includes attitudes and interest scales. (For more information, see http://www.4-h.org/about/youth-development-research/ or Lewis, Horrillo, Widaman, Worker, & Trzesniewski, 2015.)
On closer inspection, the bulk of the interest and attitude items only inquire about affect towards science-related topics; e.g., "I like science," "I would like to have a job related to science," and "I like experimenting and testing ideas." These types of items may be useful for gauging general feelings towards a topic, but they inadequately measure other important affective factors associated with learning such as self-efficacy or identity about one's scientific ability. In particular, emerging research suggests that youths' mindset about their scientific ability and identity (e.g., believing that one's scientific abilities or views about being a scientist can grow and develop and are not innate or something they are "just born with") is an important consideration for assessment as an attitude towards science, as youth's beliefs about their development of scientific skills or identity can be influenced by 4-H Science programs.
The study reported here sought to determine whether additional affective constructs, such as confidence to learn and do science, perception of the relevance usefulness of science, perception of gender bias in science, and science mindset, were separate from the general attitudes/interest assessed on the National 4-H Science CM. Our hypothesis was that these constructs were not being adequately measured, and thus we would find them only modestly correlated to the 4-H CM attitudes/interest scale. This article presents preliminary psychometric and correlational data on the National 4-H Science CM as well as four additional affective scales.
Methods
Procedures
To examine 4-H members' attitudes towards and interests in science, four surveys (10 scales) were collected from 879 California (CA) 4-H youth in 2012-2014 through the CA 4-H Online Record Book (ORB). In addition to helping 4-H youth complete their annual record book digitally, ORB allows for data collection on a variety of outcomes. Youth received gift cards for completion of the surveys. The ORB data includes demographic information collected on the 4-H enrollment forms, such as geographic location, age, years in 4-H, and project enrollments. The sample for the study is representative of the CA club program delivery mode. The study was approved by Institutional Review Board at the University of California, Davis.
| % | |
| Gender | |
| Female | 67.8 |
| Male | 32.2 |
| Ethnicity | |
| Non-Hispanic or Latino | 86.1 |
| Hispanic or Latino | 13.9 |
| Race | |
| White | 89.0 |
| Black or African-American | 0.7 |
| Asian | 4.1 |
| American Indian or Alaska Native | 1.0 |
| Native Hawaiian or other Pacific Islander | 0.4 |
| Undetermined | 4.9 |
| Residence Type | |
| Farm | 27.1 |
| Town (non-farm, rural, population <10,000) | 20.4 |
| Town or city (population 10,000 - 50,000) | 20.5 |
| Suburb of city (population > 50,000) | 14.6 |
| Central city (population > 50,000) | 17.5 |
| Mean (SD) | |
| Age | 13.06 (2.27) |
| Years in 4-H | 4.73 (2.54) |
| Number of Projects | 4.06 (4.06) |
Instruments and Scales
The first survey was the National 4-H Science CM, with four additional interest and three additional experience items (18 items total, mixture of 5, 4, and 2-point response options). This included four scales: "Attitudes," "Interest in Science," "Skills," and "Application."
The second survey (16-items, 5-point Likert response options), "Science and Me," assessed confidence to learn and do science (Confidence in Science), perception of the relevance and usefulness of science (Usefulness of Science), and gender bias in science (Women in Science). The first two scales were adapted from two of the Fennema-Sherman Mathematics Attitudes Scales (Fennema & Sherman, 1976), specifically, "confidence in learning mathematics" and "mathematics usefulness" scales. The third scale was adapted from the "Women in Science" scale (Owen et al., 2007).
The third survey (17-items, 5-point Likert response options), "Becoming a Scientist," assessed mindset and identity about one's scientific ability (Science Mindset) adapted from Blackwell, Trzesniewski, and Dweck (2007). A fourth survey was administered, general intelligence mindset (Intelligence Mindset; Blackwell et al., 2007), for comparison to scientific mindset.
Results
Reliability
For all measures, the means were generally high (e.g., 3 or higher on scales that range from 1-4; 4 or higher for scales that range from 1-5). Additionally, alpha coefficients for all scales were sufficient (range= .69 to .92). See Table 2. (For full measures, please see http://4h.ucanr.edu/About/Research/Outcomes/Measures/.)
| N | Alpha | Mean (Std. Dev). | Range | Example Item | |
| CM: Attitudes | 750 | .89 | 2.98 (0.62) | 1-4 | I like science |
| CM: Interest in Science | 734 | .86 | 3.14 (0.62) | 1-4 | I like to see how things are made or invented |
| CM: Skills | 757 | .92 | 2.81 (0.84) | 1-4 | I can use scientific data to form a question |
| CM: Application | 748 | .69 | 0.38 (0.33) | 0-1 | In my 4-H projects, I have helped with a community service project that relates to science |
| Confidence in Science | 416 | .90 | 4.00 (0.86) | 1-5 | I'm not the type to do well in science |
| Usefulness of Science | 417 | .86 | 3.96 (0.83) | 1-5 | I will use science in many ways as an adult. |
| Women in Science | 415 | .75 | 4.43 (0.68) | 1-5 | Girls can make important scientific discoveries |
| Science Mindset | 487 | .91 | 4.10 (0.55) | 1-5 | Your scientific ability will improve the more you try. |
| Intelligence Mindset | 545 | .82 | 4.03 (0.78) | 1-5 | You can learn new things but you can't really change your basic intelligence |
| Note: Std. Dev.= Standard Deviation | |||||
Comparison of the Outcomes to One Another
There were moderate correlations among the nine scales. The additional outcomes of confidence in science, usefulness of science, gender bias in science, and science mindset correlated moderately with the existing outcomes from the National 4-H CM of attitudes (r range = .30 to .60), interest (r range = .25 to .50), and skills (r range = .27 to .42). There were no significant correlations with the application subscale. These correlations suggest that these outcomes are related, but different constructs are being measured. This supports the hypothesis that the additional outcomes of confidence in science, usefulness of science, and gender bias in science are assessing constructs that are not currently measured by the National 4-H Science CM. See Table 3.
| CM: Attitudes | CM: Interest | CM: Skills | CM: Application | |
| Confidence in Science | .56*** | .50*** | .42*** | .03 |
| Usefulness of Science | .60*** | .49*** | .39*** | .02 |
| Women in Science | .39*** | .35*** | .34*** | .05 |
| Science Mindset | .30*** | .25*** | .27*** | -.03 |
The additional outcomes correlated well with one another (r range = .36 to .62). This suggests that youth's beliefs about their skills and abilities in regards to science and identity about becoming a scientist are related to other science outcomes, such as confidence in science, attitudes and interest in science, and gender bias in science. Additionally, the outcomes of confidence in science, usefulness of science, and gender bias in science had moderate correlations with one another (r range = .48 to .62). Some of these correlations are a bit high (over .60). It is possible these measures are tapping into similar constructs. Preliminary exploratory factor analysis suggest, however, that these are three separate measures. See Table 4.
| Science Mindset | Confidence in Science | Usefulness of Science | Women in Science | Intelligence Mindset | |
| Science Mindset | -- | ||||
| Confidence in Science | .38*** | -- | |||
| Usefulness of Science | .41*** | .62*** | -- | ||
| Women in Science | .36*** | .48*** | .49*** | -- | |
| Intelligence Mindset | .51*** | .24*** | .19*** | .23*** | -- |
Comparing Intelligence Mindset to Science Mindset
One measure we wanted to examine more closely was the scientific mindset measure. This measure correlated with general intelligence mindset (r = .51). This suggests that although the science mindset measure is related to general intelligence mindset, science mindset taps into a different construct of youth's beliefs about themselves than does general mindset. See Table 4.
Discussion
The study reported here examined measures of science outcomes pertinent to the National 4-H Logic model and additional constructs not presently assessed as part of the evaluation of 4-H science programming. The additional measures have good reliability and were low to moderately correlated with related affective science outcomes. More data collection on these measures in multiple 4-H science program contexts are necessary to assess the reliability and validity of the measures and their use to evaluate 4-H Science program outcomes.
We recognize the need for short self-assessment surveys given the structure of the out-of-school time (OST) environment. However, we must also balance concise measurement against the need for instruments sensitive and broad enough to capture the benefits of 4-H science programs. The four additional scales we tested were, in our experience, relatively short (33 items total) while maintaining an adequate internal consistency reliability (lowest alpha was .69). Given the nature of OST science learning—the diverse nature of content offerings—and that previous empirical research has shown tremendous promise in OST for improving attitudes, interests, and motivations for science (e.g., NRC, 2009), the inclusion of additional targeted affective-related constructs should be given serious consideration.
More broadly, attitudes and interest are important influences on advancing scientific literacy (Fortus, 2014). The California 4-H Science, Engineering, and Technology Leadership Team developed a framework for scientific literacy in the context of 4-H with programming and evaluation anchored in the development of attitudes and interest as well as relevant science content, scientific reasoning abilities, and authentic application of knowledge and skills in real-world environments (Smith, Worker, Ambrose, & Schmitt-McQuitty, in press). The study reported here helps specify affective constructs that should be evaluated in 4-H science programs relative to interest and attitudes towards science.
Recommendations and Best Practices
Extension professional and program evaluators should intentionally select instruments (and their constructs) that are sensitive enough to measure expected program outcomes. In some cases, using a general attitudinal assessment may not be enough, and more specific affective constructs may be desirable. The National 4-H Science Logic Model and associated CM may benefit from including other affective factors involved in learning. These might include science mindset (Blackwell, Trzesniewski, & Dweck, 2007), confidence/self-efficacy (Fennema & Sherman, 1976), and other constructs, such as continuing motivation for science learning (Fortus & Vedder-Weiss, 2014).
References
4-H Science Logic Model. (2010, November). National 4-H Council. Retrieved from: http://www.4-h.org/resource-library/professional-development-learning/science-training-guides-resources/
Blackwell, L., Trzesniewski, K., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78, 246-263.
Fennema, E., & Sherman, J. (1976). Fennema-Sherman Mathematics Attitudes Scales. JSAS Catalog of Selected Documents in Psychology, 6.
Fortus, D. (2014). Attending to affect. Journal of Research in Science Teaching, 51(7), 821-835.
Fortus, D., & Vedder-Weiss, D. (2014). Measuring students' continuing motivation for science learning. Journal of Research in Science Teaching, 51(4), 497-522.
Heck, K. E., Carlos, R. M., Barnett, C., & Smith, M. (2012). 4-H participation and science interest in youth. Journal of Extension [On-line], 50(2) Article 2FEA5. Available at http://www.joe.org/joe/2012april/a5.php
Irvin, J. L., Meltzer, J., & Dukes, M. (2007). Taking action on adolescent literacy: An implementation guide for school leaders. Alexandria, VA: Association for Supervision & Curriculum Development.
Krishnamurthi, A., Bevan, B., Rinehart, J., & Coulon, V. R. (2013). What afterschool STEM does best. Afterschool Matters, 18, 42-49.
Lewis, K. M., Horrillo, S. J., Widaman, K. F., Worker, S. M., & Trzesniewski, K. (2015). National 4-H common measures: Initial evaluation from California 4-H. Journal of Extension [On-line], 53(2) Article 2RIB3. Available at: http://www.joe.org/joe/2015april/rb3.php
National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. Washington DC: The National Academies Press.
Owen, S. V., Toepperwein, M. A., Pruski, L. A., Blalock, C. L., Liu, Y., Marshall, C. E., & Lichtenstien, M. J. (2007). Psychometric Reevaluation of the Women in Science Scale (WiSS). Journal of Research in Science Teaching, 44, 1461-1478.
Schmitt-McQuitty, L., Carlos, R., & Smith, M.H. (2014). Learnings and recommendations to advance 4-H science readiness. Journal of Extension [On-line], 52(4). Article 4FEA1. Available at: http://www.joe.org/joe/2014august/a1.php
Smith, M., Worker, S., Ambrose, A., Schmitt-McQuitty, L. (in press). "Anchor points" to define youth scientific literacy within the context of California 4-H. California Agriculture.
Worker, S. M. (2013). Embracing scientific and engineering practices in 4-H. Journal of Extension [On-line], 51(3). Article 3IAW3. Avaialble at: http://www.joe.org/joe/2013june/iw3.php