Assessing Educational Materials Using Cognitive Interviews Can Improve and Support Lesson Design

Introduction

Americans are not consuming the recommended number of fruit and vegetable servings daily (Stables et al., 2002). Most interventions to increase fruit and vegetable intake among low-income audiences have reported change in fruit and vegetable consumption together (Anderson et al., 2001; Havas et al., 1998; Havas et al., 2003). Interventions that do report consumption separately find the biggest contributor to the initial increase in consumption is higher fruit intake as opposed to higher vegetable intake (Buller et al., 1999; Devine, Farrell, & Hartman, 2005). Therefore, interventions focusing on increasing vegetable consumption specifically may be needed (Stables et al., 2002).

For an educational program on increasing vegetable intake to be successful, it should be appropriate for the target audience and meet their needs. One way to examine program message effectiveness, which is imperative among low-income audiences for whom customary programs may be ineffective, is through cognitive interviewing. This technique is also used in the development of surveys (Carbone, Campbell, & Honess-Morreale, 2002). Cognitive interviewers can use two approaches: concurrent (participants verbalize their thoughts while receiving a message or answering a survey question) or retrospective (participants are debriefed immediately afterwards). Either approach can help researchers identify program or message content needing clarification, simplification or greater focus.

The study reported here used cognitive interviews to assess reactions to an educational lesson designed to facilitate family acceptance of functional vegetable. Reactions to this "model lesson," derived from a series of eight functional foods lessons, were used to revise and increase the clarity and impact of the lesson series.

Methods

We created a "model" nutrition education lesson geared towards increasing consumption of deep yellow, green leafy, and cruciferous vegetables, using the format and key content derived from a set of functional foods lessons. The lesson consisted of a 20-minute PowerPoint presentation highlighting health effects, selection, storage and use in family meals (including a recipe demonstration), and four handouts (fact sheet, tip sheet, crossword puzzle, and recipes). The goal was to inspire trying of featured recipes by improving participant knowledge and perceived ability to do so, as suggested by the Theory of Planned Behavior (Azjen, 1991).

A convenience sample of four males and four females was recruited from a Central Pennsylvania food bank during the summer of 2004. Participants were low-income (<185% of poverty), married or cohabiting, and between 35 and 60 years old. During the individual interviews, each participant was asked to a) listen to the presentation and then b) review one handout so each was reviewed by a male and a female. After each step (a, b) retrospective cognitive interviewing was used to assess participant reactions.

Scripted questions addressed wording, slide titles, format, graphics, and message clarity and relevance, and attempted to identify areas needing clarification, simplification, or restructuring. Probes were used as needed to clarify and expand on participant responses. Participants received $15 upon completing the interviews, which were audiotaped, transcribed verbatim, and analyzed using principles outlined by Krueger (1994). Results are grouped as themes characterizing participant responses.

Results

Overall, participants enjoyed the lesson and felt no need to alter the general content or length. Male and female comments did not differ substantially. Findings and actions taken are presented below.

Unfamiliar Wording

Some information on the effects of various vitamins was confusing, and words like "epithelial tissue" and "gastrointestinal tract" were unfamiliar. Participants said, "That blew over my head a little bit. I'm sure if you said that to a lot of people, . . . they would be numbified with it." While most information was retained, it was described using simpler terms (i.e., "body tissue," "digestive tract").

Slide Titles

Participants gave their preference between alternative slide titles. The majority favored "we" to "I" in "How much should I/we eat?" and "Which ones should I/we eat?" They also preferred "How do I choose vegetables?" over "Choose fresh vegetables." The favored titles were put into the final lessons. (Note, "I" was changed to "we" in "How do I choose vegetables?" to reflect preference for "we.")

Format and Graphics

When shown alternative formats, they preferred less information per slide and more slides to fewer slides with condensed information. Thus, the less cluttered formats were implemented.

Some vegetable illustrations were unrecognizable. Others did not match the information being presented. Problematic graphics were replaced per participants' suggestions.

Message Clarity

Participants asked whether it was okay to eat more than the recommended number of servings per week of the vegetable groups. As one woman explained, "Well, I eat more than that. I wonder if I'm going overboard." Consequently, the words "at least" were added to the recommendations.

Message Relevance

First, participants thought linking vegetable consumption and cancer prevention was important. One woman commented, "you throw that in somebody's face that doesn't like vegetables . . . they're gonna be thinking about, 'Wow, that really does that?'" Second, the comparison between fresh, frozen, and canned vegetables plus how to choose and store different vegetables was useful and desired. Third, advice about servings of the vegetable groups to be eaten weekly was realistic, although potentially overwhelming. Thus, the lesson was modified to suggest initially adding one of these vegetables to meals once a week and slowly adding more over time.

Handouts

Participants liked the handouts. The only substantive suggestions were for the recipes: (a) give more detailed instructions, (b) clarify recipe sources, and (c) indicate serving size and preparation time. The first two ideas were implemented, but the information needed to execute the third was not available for all recipes, so this change was not made.

Summary

Cognitive interviewing provided valuable insight into low-income men and women reactions to a "model" nutrition education lesson addressing functional vegetable intake. Participant feedback informed the following revisions to a functional foods lesson series:

• Unfamiliar wording was simplified


• Preferred slide titles and less cluttered format were utilized


• Problematic graphics were replaced


• Unclear advice on recommended servings was rewritten


• Recipes instructions were simplified and sources clarified

The revised lesson series can be downloaded from The Pennsylvania State University's Cooperative Extension and Outreach Web site at <http://www.extension.psu.edu/HealthWellness/FunctionalFood.html>.

References

Anderson, J. V., Bybee, D. I., Brown, R., McLean, D. F., Garcia, E. M., Breer, M. L., et al. (2001). 5 a day fruit and vegetable intervention improves consumption in a low income population. Journal of the American Dietetic Association, 101, 195-202.

Azjen, I. (1991). The theory of planned behavior. Organizational behavior and human decision processes 50, 179-211.

Buller, D. B., Morrill, C., Taren, D., Aickin, M., Sennott-Miller, L., Buller, M.K., et al. (1999). Randomized trial testing the effect of peer education at increasing fruit and vegetable intake. Journal of the National Cancer Institute, 91(17), 1491-1500.

Carbone, E. T., Campbell, M. K., & Honess-Morreale, L. (2002). Use of cognitive interview techniques in the development of nutrition surveys and interactive nutrition messages for low-income populations. Journal of the American Dietetic Association, 102, 690-696.

Devine, C. M., Farrell, T. J., & Hartman, R. (2005). Sisters in Health: experiential program emphasizing social interaction increases fruit and vegetable intake among low-income adults. Journal of Nutrition Education and Behavior, 37(5), 265-270.

Havas, S., Anliker, J., Damron, D., Langenberg, P., Ballesteros, M., & Feldman, R. (1998). Final results of the Maryland WIC 5-A-Day promotion program. American Journal of Public Health, 88, 1161-1167.

Havas, S., Anliker, J., Greenberg, D., Block, G., Block, T., Blik, C., et al. (2003). Final results of the Maryland WIC food for life program. Preventive Medicine, 37, 406-416.

Krueger, R. A. (1994). Focus groups: A practical guide for applied research, 2nd ed. Thousand Oaks, CA: Sage Publications.

Stables, G. J., Subar, A. F., Patterson, B. H., Dodd, K., Heimendinger, J., Van Duyn, M. A., et al. (2002). Changes in vegetable and fruit consumption and awareness among US adults: results of the 1991 and 1997 5 A Day for Better Health Program surveys. Journal of the American Dietetic Association,102(6), 809-817.