 |
February 2017
|
February 2017 // Volume 55 // Number 1 // Tools of the Trade // v55-1tt9
Development of a Health Survey Instrument for 5- to 8-Year-Old Youths
Abstract
Measuring program outcomes is required for documenting effectiveness of interventions with youths participating in programs funded through the U.S. Department of Agriculture's Children, Youth, and Families at Risk (CYFAR) initiative. The California CYFAR program provided programming for youths aged 5–8, which necessitated the development of an age-appropriate survey measure. Evaluating younger youths to assess healthful living outcomes is challenging, especially with youths in kindergarten through second grade. This article addresses development and testing of the measure and resultant lessons learned. Recommendations for developing an evaluation survey for younger youths are provided.
Background
We developed, implemented, and evaluated the Food, Fitness, Farming, and Fun (4 Fs) Program to reach underserved youths in two California communities. The program provided intensive engagement for children and adolescents by involving hands-on experiential activities, positive youth development practices, place-based and service learning, innovative technology applications, and youth-adult partnerships to address childhood obesity. The expected learning outcomes were that youths would improve their knowledge and attitudes around nutrition, gardening, and cooking; gain skills needed to act on this knowledge; and improve their physical fitness. The two curricula used focused on nutrition, cooking, gardening, physical activity, and agriculture literacy to enhance youth self-efficacy and increase life skills related to healthful living.
The U.S. Department of Agriculture's Children, Youth, and Families at Risk (CYFAR) grant program (the funding entity) uses survey-based methods to evaluate desired participant outcomes. The CYFAR program requires that CYFAR Approved Common Measures be used to assess youth outcomes (Payne & McDonald, 2015); however, these common measures are not designed for youths aged 5–8. This age group composes the bulk of 4 Fs program participants; therefore, we sought an appropriate youth survey. We reviewed youth surveys available on the CYFERNet and national Expanded Food and Nutrition Education Program websites and concluded that it was necessary to develop a new youth survey.
Survey Development and Testing
Survey Development
To align our survey with program outcomes, we identified four constructs from the 4 Fs program's learning objectives: nutrition, gardening, cooking, and physical activity. We began by creating a four-by-four matrix, with one column for each construct and one row for each of four learning categories (attitude, knowledge, skill, behavior) (e.g., Friedman, 2008). This process allowed us to ensure that we covered each learning objective. We note, however, that only one survey item assessed each construct–learning category intersection and that this circumstance might decrease the validity and reliability of our instrument. We deemed this risk acceptable primarily because of contextual and structural constraints. Asking 5- to 8-year-old youths to respond to more than 16 questions may not have been developmentally appropriate and would have been out of place in the program. Research on measure lengths varies, and there are no definitive guidelines on the ideal length of a survey in terms of number of items or time spent completing the survey. Given the mixed recommendations in the field, we encourage pilot testing measures in the target population to help determine the appropriate length of the measure.
We began to develop items for each cell of the matrix by adapting from existing measures and curricula being used in the 4 Fs program—for example, the CYFAR Approved Common Measures (Regents of the University of Minnesota, 2016a, 2016b), the Eating Healthy from Farm to Fork curriculum evaluation instruments (Hazzard, Heneman, Junge, & Zidenberg-Cherr, 2009), and Expanded Food and Nutrition Education Program materials (Leavens, Townsend, Donohue, Schneider, & Santiago, 2014; U.S. Department of Agriculture National Institute of Food and Agriculture, 2015).
Through an iterative and consensus-based process, we narrowed items to one per cell, resulting in a total of 16 items (one each for every construct–learning category intersection; four items per construct and four items per learning category). The items are shown in Table 1.
- Attitudes were assessed through the use of a 5-point Likert scale. In place of the traditional response options related to levels of agreement and disagreement, we adapted the Wong-Baker Faces pain rating scale (Wong & Baker, 2000). The saddest face matched to "disagree," and the happiest face matched to "agree"; each of the middle three faces did not have a written anchor.
- Knowledge was assessed through questions with multiple-choice response options that appeared in written and graphic form. Youths were asked to circle the correct answer.
- Skills were assessed through different item types. For the gardening, cooking, and physical activity questions, youths were asked to circle "No," "Maybe," or "Yes." For the nutrition question, youths were asked to circle the correct answer from multiple-choice response options that appeared in written and graphic form.
- Behaviors also were assessed through different item types. For the nutrition question, youths were asked to circle the correct answer from multiple-choice response options that appeared in written and graphic form. For the cooking and gardening questions, youths were asked to circle "No," "Maybe," or "Yes." For the physical activity question, youths were asked to indicate how often they exercised by circling "once a month," "once a week," "2 times a week," or "every day."
| Constructs | ||||
| Learning category | Nutrition | Gardening | Cooking | Physical activity |
| Attitude | "I like vegetables" | "I like gardening" | "I like cooking" | "I like exercising" |
| Knowledge | "Circle the healthiest drink" | "Circle where food comes from" | "Circle what a recipe is used for" | "Circle what you should do when you exercise" |
| Skill | "Circle what helps you make healthy food choices" | "I can grow food to eat" | "I can make something to eat with fruits or vegetables all by myself" | "I can be active even if it is hot or cold outside" |
| Behavior | "Circle the drink you choose when you are thirsty" | "I eat food from the garden" | "I helped prepare a meal at home last week" |
"I exercise" "once a month" "once a week" "2 times a week" "every day" |
Survey Testing
The survey was administered to 406 youths at two sites over the course of 2 years. The survey was delivered to youths before the program (pretest) and after the program (posttest). Program staff read the questions aloud as youths followed along and responded to each question. The evaluation was approved by the institutional review board at University of California, Davis. Of the youths who responded to demographic questions, 52% were girls, and 69% were in the target age range for the measure (ages 5–8). The remaining 31% were over age 8, did not report their ages, or reported an age that was not feasible (e.g., 0, 3).
Lessons Learned
After soliciting feedback from program staff, including feedback about strengths and weaknesses of using the measure with youths aged 5–8, we suggest making revisions to the survey to better assess knowledge and behavior change as a result of the program. Table 2 summarizes feedback and suggested modifications to the survey. Survey questions not listed in the table did not have associated feedback.
| Survey content or question | Program staff feedback | Suggested modification(s) |
| General survey content | ||
| Survey printed in color | The colorful appeal of the survey is especially intriguing for the younger youths. The inclusion of images along with words for response options assists youths in responding. | No modification needed. |
| Use of faces to represent Likert scale | The 5-point Likert scale of faces representing "disagree" to "agree" presents too many response options to the youths. | Revise survey to use 3-point Likert scale of three faces for appropriate use with younger youths. |
| Question lengths and overall survey length | Questions are too lengthy for younger youths to complete. | Identify questions to shorten or combine to reduce the length of the questions asked to improve readability for youths. |
| Age appropriateness | Youths in kindergarten through second grade have some difficulties in responding to questions and with the survey length. | Revise the survey for readability and use across a broader age range. |
| Specific survey questions | ||
| Circle the healthiest drink: Soda, Milk, Coffee, Energy Drink | Including "Energy Drink" as an option is beneficial because it is relevant to the demographics of the target audience. | No modification needed. |
| Circle where food comes from: Grocery Store, Park, Farm, Library | Youths and program staff administering the survey were confused because both "Grocery Store" and "Farm" are viable options. | Clarify question to read "Circle where food is grown or raised" keeping the same four answer selections as options. |
| I exercise: Once a month, Once a week, 2 times a week, Every day | The format of this question differs from that of preceding questions. Youths were confused, and many circled all options. Additionally, "No exercise" is not an option, leaving some youths unable to answer the question appropriately. | Adapt question format to mirror format of preceding questions. Include "No exercise" as a response option. |
Conclusions
Evaluating healthful living outcomes is challenging, especially for youths aged 5–8. When developing an evaluation survey for younger youths, consider the following suggestions:
- Include images along with words for response options.
- Use color throughout the survey.
- Use a maximum of three face options to represent a Likert scale.
- Use short questions and limit the length of the survey to increase readability for younger youths.
- Test the survey instrument and revise questions on the basis of feedback from youths and program staff.
Acknowledgments
We appreciate the assistance of Shannon Horrillo, Roger Ingram, Debra Mason, and Aleta Barrett. Financial support for the study was provided by the U.S. Department of Agriculture (Grant Number 2011-41520-30430). The content is solely the responsibility of the authors and does not necessarily represent the official views of the U.S. Department of Agriculture.
References
Friedman, A. J. (Ed.). (2008). Framework for evaluating impacts of informal science education projects. Retrieved from http://www.informalscience.org/sites/default/files/Eval_Framework
Hazzard, E., Heneman, K., Junge, S. K., & Zidenberg-Cherr, S. (2009). An evaluation of the 2nd grade Eating Healthy from Farm to Fork curriculum. Journal of Nutrition Education and Behavior, 41(4S), P47.
Leavens, L., Townsend, M. S., Donohue, S., Schneider, C., &Santiago, O. (2014). EFNEP instruction guide for K-2 evaluation: Eat well + move. Retrieved from https://www2.ag.purdue.edu/programs/hhs/efnep/_layouts/download.aspx?SourceURL=/programs/hhs/efnep/Resource/K-2_EatWell_InstGuide_v4.pdf
Payne, P. B., & McDonald, D. A. (2015). Common evaluation tools across multi-state programs: A study of parenting education and youth engagement programs in Children, Youth, and Families At-Risk. Journal of Extension, 53(3). Article 3FEA5. Available at: http://www.joe.org/joe/2015june/a5.php
Regents of the University of Minnesota. (2016a). Nutrition: Middle school, elementary grades 3-5. [CYFAR Approved Common Measures]. Retrieved from https://cyfernetsearch.org/content/nutrition-0
Regents of the University of Minnesota. (2016b). Physical activity: Middle and high school. [CYFAR Approved Common Measures]. Retrieved from https://cyfernetsearch.org/content/physical-activity-2
U.S. Department of Agriculture National Institute of Food and Agriculture. (2015). EFNEP evaluation—Youth program. Retrieved from http://nifa.usda.gov/resource/efnep-evaluation-youth-program
Wong, D. L., & Baker, C. M. (2000). Reference manual for the Wong-Baker Faces pain rating scale. Duarte, GA: City of Hope Pain/Palliative Care Resource Center.