Development and Formative Evaluation of a Bilingual Interactive Multimedia Dietary Assessment Tool

Introduction & Background

Dietary assessment methods, both within a general population and within population subgroups, need continual improvements in both methodology and appropriateness for diverse populations. Dietary intake information, including 24-hour recalls, food frequency questionnaires, and diet histories, are traditionally collected using self-administered written methods or collected verbally via a trained interviewer. Each method has unique disadvantages related to administration, reliability and validity (Thompson & Subar, 2001). Assessing dietary intake is significantly complicated when dealing with low-income, low-literate, and minority populations. Although low literacy rates (Kirsch, Jungebut, Jenkins, & Kolstad, 1993) and an increasing Hispanic population in the U.S. (U.S. Census Bureau, 2003) are well documented, the specialized needs of this socio-culturally diverse population excludes them in research intended to improve dietary assessment methods (Subar et al., 2001; Thompson et al., 2002).

One potentially promising method of improving dietary assessment is with the use of computerized interactive multimedia (IMM) technology. Interactive multimedia is a feasible and cost-effective option for conveying nutrition education and evaluating outcome assessments in low-income and Spanish-speaking participants (Brug, Campbell, & van Assema, 1999; Brug, Oenema, & Campbell, 2003; Campbell, Honess-Morreale, Farrell, Carbone, & Brasure, 1999; Gould & Anderson, 2000; Jantz, Anderson, & Gould, 2002). Although computerized IMM for nutrition education has been explored quite extensively, there is relatively little research utilizing IMM for comprehensive nutrient assessment. Existing IMM tools only look at specific aspects of food intake, for example fat or fruit and vegetable intake (Block et al., 2000; Campbell et al., 1999). No known IMM tools generate a comprehensive nutrient profile.

Participants enrolled in the Expanded Food and Nutrition Education Program (EFNEP) are required to complete a written entry and exit questionnaire that includes a 24-hour dietary recall. Colorado State University has utilized advancements in computer technology to develop a bilingual IMM recall for EFNEP participants. The IMM recall can be used in low-literate and Spanish-speaking populations. It can potentially save staff time spent on dietary assessment and allow focus on more nutrition education. The IMM recall also has applications to other food assistance programs such as WIC and the Food Stamp Nutrition Education Program.

Originally, a bilingual IMM recall was developed and tested to assess fat and fiber at breakfast (Lowe, 2001). The IMM recall was then expanded to assess a comprehensive nutrient profile, including 20 dietary constituents, at lunch, supper, and snacks (Zoellner, 2004). This is the first known attempt at developing a food recall into an IMM format to generate a comprehensive nutrient profile. The overall goal of the project described here was to develop and evaluate an IMM recall program with a low-literate, low-income, Hispanic, and Caucasian population. This article describes the development and formative evaluation process used to create the IMM recall. The use of touch-screen computers may be a viable means of gathering nutrition information, but nutrition educators should understand the development process and anticipate the challenges involved in utilizing technology to assess dietary intake.

Development of a Bilingual Interactive Multimedia (IMM) Dietary Recall

The research described here details the development of the lunch, supper, and snacks portion of the IMM recall. Below are the seven progressive steps of this process.

Step 1: Determining Food Choices

Food choice of the targeted audience was determined by examining the 1982-1984 Hispanics Health and Nutrition Examination Survey (HHANES) and 191 individual 24-hour recalls from Colorado EFNEP participants (Block, Norris, Mandel, & DiSogra, 1995). The EFNEP recalls accounted for seasonality of food choice and were representative of Colorado's EFNEP participant demographics for the 1998-1999 reporting period.

Foods reported by greater than 2% of the population in one or both surveys, with a few exceptions, were included in the IMM dietary recall. Exceptions were made for foods reported between 2-4% in the HNANES, yet were not reported at all in the 24-hour recalls. Examples include tongue, gizzard, game, summer squash, plantains, and Mexican pan dulce. Watermelon and cantaloupe also fell below the 2% threshold in both surveys, yet were included in the IMM recall.

It has been suggested that close-ended questionnaires may yield overestimation of fat intake because subjects are not given the option of reporting low-fat varieties of food choices (Vandenlangenberg et al., 1997). For this reason, modified fat varieties of food items were also illustrated in the IMM recall, even though they were not frequently reported in the HHANES or EFNEP recalls.

The resulting IMM dietary recall included 167 food items. The food choice for a population should represent the intake of the population and greatly minimize measurement error through exclusion of food items. Two studies by Block and colleagues indicate that successful questionnaires include foods representing the top 80% or 90% of the nutrient intake of a population (Block et al., 1986; Block, Hartman, & Naughton, 1990). For this study it was approximately 98%. Six cooking method techniques were also included: 1) fried with oil, 2) fried with lard, 3) fried with butter, 4) fried with margarine, 5) fried with pan spray, and 6) baked, broiled, grilled, boiled, or microwaved.

Step 2: Determining Portion Sizes

The next step was to determine which portion sizes should be represented in the IMM recall. The US Department of Agriculture's 1994-1996 Continuing Survey of Food Intake by Individuals (CSFII) quantities consumed per eating occasion for all individuals age 2 and over was used to help determine IMM recall portion sizes (Smiciklas-Wright, Mitchell, Mickle, Cook, & Goldman, 2002). The CSFII portion represented the first portion size in the IMM recall. Nutrition professionals, experienced in collecting dietary information, aided in determining three additional increments in portion sizes. Consistent increments were used within food groups, but not always between foods groups. When only three different portion sizes are used, Guthrie (1984) found a strong tendency to choose the middle size. Therefore, the IMM recall features four portion sizes of each food, which forces participants to differentiate between the two middle sizes.

Step 3: Food Photography

Food photographs are a useful and convenient aid in the estimation of food portion sizes (Robinson, Morritz, McGuiness, & Hackett, 1997). When comparing portion size reports for the same foods using food models and equivalent two-dimensional pictures, some respondents reported differently, but no apparent bias in the direction of reporting was evident (Posner et al., 1992). A study by Nelson, Atkinson, and Darbshire (1994) showed that using multiple photographs was more accurate than a single average portion photograph.

Based on these findings, an individual photograph for each portion size of food was taken so that participants would be able to interactively grow and shrink portion sizes. All foods and portions featured in the recall were photographed with a Canon Power Shot 5300 Digital Elph camera. PhotoshopŽ (Version 7.0) was used to format and modify food photographs and to remove identifying food brand information.

Step 4: Developing a Script/Format

A script was developed to outline the IMM program format and wording of questions regarding eating behavior. The script addressed food choice, food variety (regular, low-fat, fat-free), portion size, and cooking methods. Seven main menu screens were developed whereby similar foods were grouped together. There were four to five groups of foods featured on each main menu screen. The user first selected a group of foods, then the foods were featured individually, and the user selected the specific food eaten. A formative evaluation process was used to determine the most effective order of questions, organization of the script, and overall format of the program (described below in the Formative Evaluation Methods & Results section).

Step 5: Recording Sound Files

Once the English script was developed, recorded, and reviewed, the script was translated into Spanish. There were two Spanish translators. One translated the text into Spanish, and one recorded the Spanish sound files. This process allowed double-checking of translations. Recording audio files was an ongoing process because script changes were constantly being made as feedback was obtained through formative evaluation (described below in the Formative Evaluation Methods & Results section). All audio was recorded in a professional sound studio and recorded and edited using Sound ForgeŽ (Version 4.5).

Step 6: Authoring the IMM Recall Program

A private company was hired to program the IMM recall with ToolBookŽ (Version 8.1). The IMM recall produced a text-based logfile indicating the user's identification number and list of foods and portion sizes reported.

Step 7: Designing a Nutrient Database

A database was designed to generate a nutrient profile for each user based on foods selected in the IMM recall. The database was comprised of 20 nutrients or dietary constituents, including total calories, carbohydrates, protein, total fat, saturated fat, cholesterol, fiber, iron, calcium, vitamin A, vitamin C, vitamin B6, folic acid, alcohol, and the number of servings from each food group, including breads and cereals, fruits, vegetables, dairy, meat, and others. The EFNEP Evaluation/Reporting System (ERS) nutrient database was primarily utilized to identify nutrient composition of foods because it is the mandatory reporting system used by EFNEP agencies nationwide (EFNEP ERS User's Guide, 1998). An AccessŽ database was created to read the text-based logfile and produce a nutrient profile.

Formative Evaluation Methods & Results

During development of the IMM recall, input from nutritional professionals familiar with collecting self-reported dietary information was needed. The logistics and feasibility of using a computerized approach to assess dietary intake needed to be explored. The intended audience for this instrument was EFNEP participants; however, many methodological and practical problems of developing an IMM recall needed to be resolved prior to delivering a testable product to the target audience. Therefore Extension and EFNEP state staff and nutrition faculty, staff and graduate students at Colorado State University were recruited to provide feedback on the IMM recall through two formal phases of formative evaluations.

Methods Phase I Alpha Testing

Three different prototypes, with a subset of 35 foods, were initially developed to determine an effective script/format for the IMM recall. Prototype one queried about food choices collectively, per day, for example "Tell me which of these foods you ate for lunch, supper, and snacks yesterday by touching the pictures." Additionally, prototype one queried about specific information regarding portion sizes and cooking methods after each food item was reported.

Prototype two queried about food choices separately, per eating occasion, for example "Tell me which of these foods you ate for lunch yesterday by touching the pictures." Unlike prototype one, prototype two queried specific information about portion sizes and cooking methods after all of the food items had been reported.

The querying structure of prototype three was similar to prototype two. However, prototype three had visual and audio features, as well as food groupings and directions, which were different from prototype two.

The prototypes were tested to determine:

1. Whether dietary information should be reported collectively (per day) or separately (per eating occasion) at lunch, supper and snacks.

2. Whether specifics about portion variety, portion sizes, and cooking methods were to be queried after each separate food choice or after all food choices for the entire day had been collected.

3. The effectiveness of the introduction/directions, food grouping, graphics, audio, flow, and delivery.

During Phase I Alpha Testing, all participants completed prototype one and either prototype two or three. Participants entered a sample menu into the recall. An interviewer recorded the participants' comments and difficulties encountered while navigating through the program. Participants completed a questionnaire that included four open-ended questions and six statements on a five-point Likert scale (1= negative response; 5 = positive response) to assess the acceptability and user friendliness of each prototype completed. Likert scale item responses were averaged to create an overall mean score for each question.

Results Phase I Alpha Testing

The goal with Phase I Alpha Testing was to perform sufficient testing of the prototypes so that an appropriate format and logistical problems could be identified and addressed before further development proceeded. Interviews were analyzed for emerging themes and consistent comments among participants (Glaser & Stauss, 1967; Krueger & Casey, 2000). This occurred after testing with only six people (N=6). Participants included two EFNEP state staff, two nutrition faculty, and two nutrition graduate students.

Participants preferred to report dietary information separately (per eating occasion) at lunch, supper, and snacks (5 of 6 participants). It was easier and more logical to answer questions about each eating occasion separately as opposed to answering questions collectively regarding food choice across all eating occasions for the entire day. Participants preferred to report specifics about variety, portion sizes, and cooking methods after all food choices for the entire day had been collected (4 of 6 participants). It seemed faster and more efficient to simply select foods first and was distracting to answer specific questions about variety, portion sizes, and cooking methods after each food choice.

The mean score of each 5-point Likert scale statement received neutral to high ratings as explained in Table 1.

Two major improvements emerged as a result of this first phase of alpha testing:

1. A navigation tool bar listing foods that would appear on subsequent main menu screens was added to the left side of the screen. This allowed users to anticipate the program flow and order of food item queries.

2. A feature that summarized all food selections made and provided the participant with the opportunity to add and delete food items was also incorporated at the end of the food recall.

In addition to these two improvements, a variety of others were incorporated. These included:

• Clarification and condensation of the directions;

• Terminology changes;

• Improvements in formatting, clarity, and uniformity of the graphics;

• Displaying text beneath each food item represented; and

• Correction of programming glitches, including distracting transition between screens, delayed loading of graphics, and incorrect path files for graphic and audio files.

Methods Phase II Alpha Testing

The results from Phase I Alpha Testing contributed to further development of the IMM recall. After the entire program, including all 167 foods, was developed and reviewed several times, Phase II of Alpha Testing was conducted. Extension and EFNEP state staff and nutrition faculty, staff, graduate students, and work-study students were recruited to participate. Participants reported foods eaten during the previous day via the IMM recall and documented difficulties and errors encountered. Afterwards, they completed a questionnaire including two open-ended questions and five questions on a five-point Likert scale (1= negative response; 5 = positive response) to assess the acceptability and user friendliness of the program.

Results Phase II Alpha Testing

The goal for Phase II of Alpha Testing was to perform sufficient testing so that emerging themes and reoccurring problems could be identified and addressed before testing within the target audience. This occurred after testing with 25 participants (N=25). The mean score of each 5-point Likert scale question received high ratings as explained in Table 2.

• Three or more participants reported the following five concerns:

• Explain how long the introduction/directions were going to last,

• Explain why breakfast was not included in the IMM recall,

• Provide reminder to click the "next" button after completing each screen,

• Choppy/inconsistent volume, and

• Needs to go backwards through the program.

These concerns were addressed and programming glitches were corrected. Many additional comments were not reported consistently enough to warrant change. This testing also produced positive feedback and excitement, evidenced by these quotes.

• "Really fun, it's like a game, I want to play again."

• "Just the bugs need to be worked out---great program."

• "After the first food choice the screens became easy."

Final Description of the IMM Dietary Recall

The IMM recall begins with four minutes of introduction/directions detailing the use of the program. Every choice is made by listening to audio and touching appropriate graphics. Figure 1 details the steps that users navigate through the IMM recall.

Figure 1.
Steps to Navigate Through the Interactive Multimedia Dietary Recall

The user first selects the eating occasion at which they consumed food/beverages during the previous day. For each eating occasion identified, the user navigates through seven main menu screens and identifies foods eaten within a group of foods, as illustrated in Figure 2.

Figure 2.
Main Menu Screen Example for Side Dish #1

After a group of foods have been identified, the user identifies the specific food eaten as illustrated in Figure 3.

Figure 3.
Menu Screen Example for Identifying Specific Food Eaten

After all foods are selected, users identify food variety (regular, low-fat, fat-free). Then users choose portion size, as illustrated in figure 4. The smallest portion size is the first image displayed. Users touch an arrow on the right side of the screen to interactively grow and shrink portion sizes, stopping on the portion size that looks most like the portion size consumed before proceeding to the next food.

Figure 4.
Menu Screen Example for Identifying Portion Size

After all the foods have been selected cooking methods are identified as illustrated in Figure 5.

Figure 5.
Menu Screen Example for Identifying Cooking Method

Finally, each eating occasion is displayed with the list of foods selected. Users are able to add and delete foods. After all the foods are verified, the user exits the program. A text-based logfile is created and imported into the AccessŽ database to generate a nutrient profile.

Development Challenges of the IMM Recall

The use of technology in diet assessment has many potential benefits (Kolasa & Miller, 1996). However, there are also several challenges that nutrition educators should anticipate and account for in the development of future IMM dietary recall programs, including budget estimate, web-based programming, naming files, recording audio, and food photography. Due to the exploratory nature of the project, unanticipated changes were constantly being recommended, and the scope of the program was continuously changing. Changes initially perceived as small proved to be very time consuming and challenging for the programmer. As a result, development of the IMM recall ran over budget.

The IMM recall was developed using ToolBookŽ for CD-ROM, which limits usability of the program on networked computers. With the exponential improvements made for web-based applications, including broader use of structured query language (SQL), the developers recommend that the recall be adapted to this format to optimize reach.

A large number of files, including 960 graphics and 1125 audio files, are represented in the IMM recall. Establishing a meaningful naming convention is essential and should be well thought-out prior to programming. File names from the audio folder and graphic folders need to coincide for a successful query of the nutrient database.

Audio recording and photography occurred in many different sessions, resulting in inconsistencies. Using a professional sound studio and applying a filter during the recording process is highly recommended. Camera angle and camera distance from foods should be consistent to assure a flawless transition from one portion size to the next and to avoid labor-intensive photograph editing tasks such as cropping and removing shadows.

Discussion

The input from nutrition professionals proved to be critical in the development of the IMM recall. Researchers at Colorado State University had developed and tested an IMM recall to assess fat and fiber at breakfast; however, there was little published information available on the use of computer technology to develop an IMM recall to assess a comprehensive nutrient profile. Obtaining feedback and ideas from nutrition professionals familiar with the challenges involved with collecting self-reported dietary information, assisted in solving the logistical, methodological, and practical problems of developing an IMM recall.

After the development and formative evaluation, the IMM recall has undergone a validation study in the target audience (Zoellner, Anderson, Gould, In press). Eighty participants completed an interview-administered recall and the IMM recall. In brief, correlations of approximately 0.6, provided evidence the IMM recall was as good as the comparison interview-administered recall. Correlations in this range are considered to be satisfactory for making inferences about intakes by groups of individuals (Willett, 1998), such as the EFNEP's use of group data to monitor and evaluate program impacts.

Due in large part to the problems identified and addressed through formative evaluation, the IMM recall tool was well received in the target audience. The use of food graphics and audio, combined with the elimination of reading, writing, and keyboarding skills, makes IMM a logical and feasible option for improving the assessment of diet in multiple ethnic groups and populations with low literacy skills. The IMM recall can be of benefit to Extension educators and others and was shown to have concurrent validity with an interview-administered recall in the target audience (Zoellner, Anderson, Gould, In press).

Application

Extension programming will benefit from this research and a computerized IMM food recall for several reasons.

1. This IMM recall can save staff time and resources, allowing EFNEP and food assistance agencies to focus on nutrition education.

2. Directly linking the users' food choice, portion sizes, and cooking methods to a computer-generated database will eliminate data coding and entry errors.

3. The audio and visual cues supported by IMM technology may help stimulate food recall and promote more honest reports of food intake because food choice is disclosed privately without verbal or non-verbal feedback from a staff member.

Future Development

Lessons learned through this formative evaluation process will assist in the continued development of the IMM recall. Many steps are needed prior to a wider implementation within EFNEP, including incorporation of more population specific foods from different geographic regions and the ability to interface the IMM recall with the EFENP Reporting System (ERS). Breakfast should be included with the lunch, supper, and snacks components, resulting in a complete 24-hour time frame, and then formative evaluation and validation in the target audience should occur. Performing cognitive interviewing in the target population to identify respondent error would also improve the usability of the IMM recall.

Conclusion

Interactive multimedia computer technology can be used to enhance the efforts of nutrition educators. Computer technology provides many opportunities to link dietary intake information to nutrition education targeted to meet the clients' needs. Although the use of computer technology to provide dietary education has exploded in recent years, there is a continued need to develop and validate computerized methods to assess dietary intake. Computers can be an effective means of gathering and disseminating nutrition information, but nutrition educators should understand the steps involved, the challenges encountered, and the importance of formative evaluation in developing a quality product.

Acknowledgment

Funding for this research was provided by USDA-CSREES-NRI #00-35200-9114; USDA-Food and Nutrition Services through the Colorado Department of Human Services Food Stamp Program; and Share Our Strength.

References

Block, G., Hartman, A., Dresser, C., Carroll, M., Gannon, J., & Gardner, L. (1986). A data-based approach to diet questionnaire design and testing. American Journal of Epidemiology, 124(3), 453-469.

Block, G., Hartman, A., & Naughton, D. (1990). A reduced dietary questionnaire: development and validation. Epidemiology, 1(1), 58-64.

Block, G., Norris, J., Mandel, R., & DiSogra, C. (1995). Sources of energy and six nutrients in diets of low-income Hispanic- American women and their children: quantitative data from HHANES, 1982- 1984. Journal of the American Dietetics Association, 95(2), 195-208.

Block, G., Miller, M., Harnack, L., Kayman, S., Mandel, S., & Cristofar, S. (2000). An interactive CD-ROM for nutrition screening and counseling. American Journal of Public Health, 90(5), 781-785.

Brug, J., Campbell, M., & van Assema, P. (1999). The application and impact of computer-generated personalized nutrition education: a review of the literature. Patient Education and Counseling, 36(2), 145-156.

Brug, J., Oenema, A., & Campbell, M. (2003). Past, present, and future of computer-tailored nutrition education. American Journal of Clinical Nutrition, 77(Suppl.), 1028-1034S.

Campbell, M., Honess-Morreale, L., Farrell, D., Carbone, E., & Brasure, M. (1999). A tailored multimedia nutrition education pilot program for low-income women receiving food assistance. Health Education Research, 14(2), 257-267.

EFNEP evaluation/reporting systems user's guide. (1998). [On-line]. Available at: http://www.csrees.usda.gov/nea/food/efnep/ers4.html

Glaser, B., & Stauss, A. (1967). The discovery of grounded theory: strategies for qualitative research. Hawthorne, N.Y.: Aldine de Gruyter.

Gould, S., & Anderson, J. (2000). Using interactive multimedia nutrition education to reach low-income persons: An effectiveness evaluation. Journal of Nutrition Education, 32(4), 204-213.

Gould, S., & Anderson, J. (2002). Economic analysis of bilingual interactive multimedia nutrition education. Journal of Nutrition Education and Behavior, 34(5), 273-278.

Guthrie, H. (1984). Selection and quantification of typical food portions by young adults. Journal of the American Dietetics Association, 84(12), 1440-1444.

Jantz, C., Anderson, J., & Gould, S. (2002). Using computer-based assessments to evaluate interactive multimedia nutrition education among low-income predominantly Hispanic participants. Journal of Nutrition Education and Behavior, 34(5), 252-260.

Kirsch, I., Jungebut, A., Jenkins, L., & Kolstad, A. (1993). Adult literacy in America: A first look at the results of the national adult literacy survey. Washington D.C.: U.S. Department of Education.

Kolasa, K., & Miller, M. (1996). New developments in nutrition education using computer technology. Journal of Nutrition Education, 28, 7-14.

Krueger, R., & Casey, M. (2000). Focus groups: A practical guide for applied research (3rd ed.) Thousand Oaks, CA: Sage Publications, Inc.

Lowe, M. (2001). Development and evaluation of an interactive multimedia tool to assess food intake. Unpublished master's thesis, Colorado State University, Fort Collins.

Nelson, M., Atkinson, M., & Darbshire, S. (1994). Food photography I: The perception of food portion size from photographs. British Journal of Nutrition, 72(5), 649-663.

Posner, B., Smigelski, C., Duggal, A., Morgan, J., Cobb, J., & Cupples, L. (1992). Validation of two-dimensional models for estimation of portion size in nutrition research. Journal of the American Dietetics Association, 92(6), 738-741.

Robinson, F., Morritz, W., McGuiness, P., & Hackett, A. (1997). A study of the use of a photographic food atlas to estimate served and self-served portion sizes. Journal of Human Nutrition and Dietetics, 10(2), 117-124.

Smiciklas-Wright, H., Mitchell, D., Mickle, S., Cook, A., & Goldman, J. (2002). Foods commonly eaten in the United States: Quantities consumed per eating occasion and in a day, 1994-96. U.S. Department of Agriculture NFS Report No. 96-5. [On-line]. Available at: www.barc.usda.gov/bhnrc/foodsurvey/Products9496.html

Subar, A., Thompson, F., Kipnis, V., Midthune, D., Hurwitz, P., McNutt, S., et al. (2001). Comparative validation of the Block, Willett, and National Cancer Institute food frequency questionnaires: The Eating at America's Table Study. American Journal of Epidemiology, 154(12), 1089-1099.

Thompson, F., & Subar, A. (2001). Dietary assessment methodology. In A. M. Coulston, C. L. Rock, & E. R. Monsen (Eds.), Nutrition in the prevention and treatment of disease (pp. 3-30). San Diego, CA: Academic Press.

Thompson, F., Subar, A., Brown, C., Smith, A., Sharbaugh, C., Jobe, J., et al. (2002). Cognitive research enhances accuracy of food frequency questionnaire reports: Results of an experimental validation study. Journal of the American Dietetics Association, 102(2), 212-225.

U.S. Census Bureau. [On-line]. Available at: http://www.census.gov/

Vandenlangenberg, G., Mares-Perlman, J., Brady, W., Klein, B., Klein, R., Palta, M., et al. (1997). Incorporating fat-modified foods into a food frequency questionnaire improves classification of fat intake. Journal of the American Dietetics Association, 97, 860-866.

Willett W. (1998) Nutritional epidemiology. 2nd ed. New York: Oxford University Press.

Zoellner, J. (2004). Development and evaluation of a bilingual interactive multimedia computerized food recall. (Doctoral dissertation, Colorado State University, 2004) Dissertation Abstracts International, 65(05), 2352.

Zoellner, J., Anderson, J., & Gould, S. (In press). Comparative validation of a bilingual interactive multimedia dietary assessment tool. Journal of the American Dietetics Association.