Use Retrospective Surveys to Obtain Complete Data Sets and Measure Impact in Extension Programs

Introduction

Demonstrating impact in Extension programs is necessary to maintain funding and ensure their continuation. Traditionally, pre/post surveys have been conducted to determine changes in participants' knowledge, behavior, attitude, or skills. However, evaluation with a pre/post format has two problems. The first is incomplete data sets, where either pre- or the post- only are filled out. The second problem is referred to as "response-shift bias," where participants overestimate their behaviors on the pre-survey and underestimate their behaviors on the post-survey due to a change in frame of reference.

These problems can be circumvented by using a retrospective survey. All data sets are complete because the individual completes the post/pre survey. Also, Howard, Ralph, Gulanick, Maxwell, Nance, and Gerber (1979) and Pratt, McGuigan, and Katzev (2002) have found that using a retrospective survey decreases the "response-shift bias" that occurs in pre/post surveys. Researchers have adapted retrospective surveys to determine changes in behavior of individuals in drug prevention and leadership development (Rhodes & Jason, 1987; Rohs, 1999); changes in attitudes toward individuals with HIV/AIDS (Riley & Greene, 1993), and changes in knowledge from taking a nutrition course (Rockwell & Kohn, 1989).

In Idaho, the Food Stamp Nutrition Education (FSNE) program, called the Extension Nutrition Program (ENP), switched from having participants complete a traditional EFNEP survey that had a pre-post format, which measured self-reported changes in nutrition, food safety, and resource management behaviors to the retrospective survey. The purpose of the project reported here was to determine if a retrospective survey decreased the number of incomplete responses and provided an effective measure of self-reported changes in nutrition, food safety, and resource management behaviors.

Methodology

Step I: Changing a Pre-Post Survey to a Retrospective Survey

To change the pre-post to a retrospective survey, two questions were asked for each behavior being measured. The first question was a post-test question because the participants were asked about their behavior after the program. The second question, the pre-test question, asked them about their behavior before the program (Rockwell & Kohn, 1989).

Figure 1 is an example of a question from the Retrospective Survey that has 5 choices. The choices were converted to a 5 point scale: 1 = do not do; 2 =seldom; 3 =sometimes; 4= most of the time; 5=always.

Step II: Training on the Retrospective Survey

All of the Nutrition Advisors, paraprofessionals who conducted the ENP lessons and the surveys, were trained on how to administer the retrospective survey.

Step III: Implementing the Revised Questionnaire and Analyzing the data

The retrospective data analyzed for this study covered all FY2002 participants (N=346) who "graduated" from the program and completed at least six core lessons. The pre-post data analyzed for this study covered all FY2001 participants (N=220).

Data for analyses included: (1) the number of incomplete responses for the FY2001 pre- and post-test and the FY2002 retrospective survey; (2) Frequency of pre- and post-behaviors on the retrospective survey; and (3) Paired t-tests for the 15 retrospective survey questions, where subjects served as their own control. All t-tests were two-tailed, and df=345. A Bonferroni adjustment (Rimm, Hartz, Kalbfleisch, Anderson, & Hoffman, 1980) was made to control the experiment-wide error rate, and the level of significance was set at <0.003 (0.05/15).

Results

Number of Incomplete Responses

As indicated earlier, one of the problems associated with the pre-post survey was the number of incomplete responses that occurred when people completed this survey. Table 1 lists the percentage of incomplete responses of the pre-post survey and the retrospective survey. The percentage of incomplete responses for the pre-survey was 16% and for the post-survey was 15.6%. In contrast, there were 0.0% incomplete responses with participants who completed the retrospective survey.

Changes in Behaviors

Results from the retrospective survey in Table 2 indicate that frequency of positive behaviors increased, negative behaviors decreased, and participants significantly improved (p<0.000) in all 15 behaviors.Frequencies for positive behaviors were combined from the "most of the time" and "always" categories, and negative behavior frequencies were combined from the "do not do" and "seldom" categories.

Participant ratings showed that all resource management, nutrition, and food safety mean behaviors significantly (p<0.00) improved. After attending ENP, there was a 34 - 44.8% increase in participants who indicated that they most of the time/always planned meals, compared prices, used grocery lists, used a spending plan, and had emergency money set aside and a 32.7% decrease in the number of participants who said they did not or seldom ran out of food.

Nutrition mean behaviors increased by 38.2 - 48.2%. After attending ENP, participants reported that they most of the time or always read food labels and ate low fat items, fruits and vegetables.

Food safety mean behaviors increased by 9.5 - 13.6%. After attending ENP, participants reported that they most of the time or always washed utensils, cooked meat thoroughly and washed their hands. Also, there was a 50.6% decrease in participants who said that they seldom or did not thaw their meat on the counter.

Discussion

Using a Retrospective Survey

The results from this study indicate that a retrospective survey can be used to measure the behavior changes that occur in participants of an FSNE program. This evaluation tool was tested on a large representative sample (N=346) of individuals graduating from a FSNE program. Results showed the retrospective approach to be successful in a number of ways.

While the pre-post method resulted in approximately 16% incomplete responses in both surveys, there were no incomplete responses for the retrospective survey. Reasons why the number of incomplete responses was higher in the pre-post surveys include:

1. The participant may not have had a clear understanding of what the pre-survey was asking them and therefore chose not to answer the question (Howard, Ralph, Gulanick, Maxwell, Nancy & Gerber, 1979);

2. Post-survey questions may not have been answered because participants may have felt that they already answered these questions (in the pre-test) and;

3. Participants may have felt that the post-survey took up valuable time that could be better spent on their class (Marshak, deSilvva, & Silberstein, 1998).

In contrast, when the retrospective surveys were conducted at the end of the core classes, participants may have had a better understanding of these questions and therefore were more willing to spend the time to make sure the survey was filled out completely.

Participants completing the retrospective survey reported that they significantly changed their resource management, nutrition, and food safety behaviors. Kempson, Palmer-Keenan, Sadani, Ridlen, & Rosato (2002) found that before individuals in low socioeconomic groups can change nutrition and food safety behaviors, they must not be food insecure, i.e., they must have "a ready availability of nutritionally adequate and safe foods and an assured ability to acquire acceptable foods in socially acceptable ways" (Anderson, 1990).

It is estimated that 11% of the households in the U.S. are food insecure, and Idaho is above the national average at 13% (Nord, Andrews, & Carlson, 2002). Therefore, some of the first classes covered in the FSNE program focus on resource management skills.

The resource management behaviors are practiced by a much higher percentage of ENP participants when compared to national statistics, which show that that approximately 50% reported that they plan meals (Food Marketing Institute, 2000), 21% do cost comparison, and 12% buy only those items on their grocery list (Food Marketing Institute, 2002). However, even after the ENP classes on resource management, approximately 17% of participants indicated they ran out of food, which is similar to the national average of 16% of food insecure individuals (Nord, Andrews, & Winicki, 2002).

Seven out of 10 Americans believe eating healthfully is too complicated (Dinkins, 2000). The improvement in eating habits of ENP participants could be related to the classes that focused on teaching participants how to apply this information in order to improve their dietary quality.

Also, the most recent Continuing Survey of Food Intake by Individuals (CSFII) indicates that low-income adults are interested in consuming more fruits and vegetables and less fat (USDA, 1998). It is estimated that the average American consumes 1.5 servings of fruit and 3.3 servings of vegetables daily (USDA, 1998). The ENP nutrition classes were effective in increasing fruit and vegetable intake of ENP participants to 2 - 3 or more servings per day. Research by Marion Neuhoser and colleagues (1999) indicates that many individuals use food labels to decrease their fat intake, and this behavior may have been linked to the increased consumption of low-fat food items.

Rusin, Orosz-Coughlin, & Gerba (1998) found that the kitchen environment was more heavily contaminated with bacteria than the bathroom. The food safety behaviors covered in the ENP focus on sanitation practices in the kitchen. Proper food handling practices can prevent cases of foodborne disease (EPA, USDA, DHHS, 1997).

Schoenborn and Barnes (2002) found that 43% of adults living below the poverty level reported participating in physical activity. This is a little lower than the 50% of ENP participants that reported being physically active.

A limitation of this study was that all of the data collected was self-reported and participants may have reported what they thought we wanted to hear, instead of what really occurred. Even though it is impossible to eliminate all bias in a study, the retrospective survey ensures that individuals evaluate their pre-post behaviors using the same frame of reference.

In this study, a retrospective survey was used to measure changes in behavior after completing six core lessons. However, because many Extension programs consist of a single lesson, the post/pre format could be used to develop a questionnaire that participants complete at the end of the lesson to determine impact of the lesson.

Implications

In summary, the study reported here showed that a retrospective survey has three positive benefits. The first two benefits are that it gets rid of incomplete data sets and wasted data sets that usually occur in pre-post tests. The third benefit is that it is an effective way to measure self-reported behavior change.

References

Anderson, S. A. (1990). Core indicator of nutritional state for difficult to sample populations. Journal of Nutrition,120 (Suppl),1559-1600.

Dinkins, J. M.(2000). Beliefs and attitudes of Americans toward their diet. Family Economics and Nutrition Review, 13 (1), 98-100.

Food Marketing Institute. (2000). Four types of shoppers, Supermarket Research 2000, 2(3), 1-3.

Food Marketing Institute. (2002). Trends in the United States: Consumer attitudes & the supermarket, 2002. Food Marketing Institute, Washington, D.C.

Howard, G. S., Ralph, K. M, Gulanick, N. A., Maxwell, S. E., Nance, D., & Gerber, S. L. (1979). Internal invalidity in pretest-posttest self-report evaluations and the re-evaluation of retrospective pretests. Applied Psychological Measurements, 3, 1-23.

Kempson, K., Palmer-Keenan, D., Sadani, P. S., Ridlen, S., & Rosato, N. S. (2002). Food management practices used by people with limited resources to maintain food sufficiency as reported by nutrition educators. Journal of the American Dietetic Association,102,1795-1799.

Marshak, H. H., de Silva, P., & Silberstein, J. (1998). Evaluation of a peer-taught nutrition education program for low-income parents. Journal of Nutrition Education, 30:314-322.

Neuhouser, M. L., Kristal, A. R., & Patterson, R. E. (1999). Use of food nutrition labels is associated with lower fat intake. Journal of the American Dietetic Association,99,45-50, 53.

Nord, M., Andrews, M,, & Carlson S. (2002, October).Household food security in the United States, 2001.ERS Food Assistance and Nutrition Research Report No. FANRR29. 52 pp,

Nord, M., Andrews, M., & Winicki, J.(2002). Frequency and duration of food insecurity and hunger in US households. Journal of Nutrition Education And Behavior,34,194-201.

Pratt, C., McGuigan, W. & Katzev, A. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21,341-349.

Rhodes, J. E., & Jason, L. A. (1987). The retrospective pretest: An alternative approach in evaluating drug prevention. Journal of Drug Education, 17:345-356.

Riley, J. L., & Greene, R. R. (1993). Influence of education on self-perceived attitudes about HIV/AIDS among human services providers. Social Work, 38(4), 396-401.

Rimm, A. A., Hartz, A. J., Kalbfleisch, J. H., Anderson, A.J., & Hoffman, R. G., (1980). Basic biostatistics in medicine and epidemiology. New York, NY: Appleton-Century-Crofts.

Rockwell, S. K., & Kohn, H. (1989). Post-then pre evaluation. Journal of Extension [On-line], 27(2). Available at: http://www.joe.org/joe/1989summer/a5.html

Rohs, F. (1999). Response-shift bias: A problem in evaluating leadership development with self-report pretest-posttest measures. Journal of Agricultural Education, 40(4):28-37.

Rusin, P., Orosz-Coughlin, P., & Gerba, C. (1998). Reduction of faecal coliform and heterotrophic plate count bacteria in the household kitchen and bathroom by disinfection with hypochlorite cleaners. Journal of Applied Microbiology, 85,819-828.

Schoenborn, C. A., & Barnes, P. M. (2002). Leisure-time physical activity among adults: United States, 1977-98. Advance Data from Vital and Health Statistics; no 325. Hyattsville, Maryland, National Center for Health Statistics.

U.S. Department of Agriculture, Agricultural Research Service. (1998). 1994-96 Continuing Survey of Food Intakes by Individuals and 1994-96 Diet and Healthy Knowledge Survey and related materials (CD-ROM).

U.S. Environmental Protection Agency, U.S. Department of Agriculture, Department of Health and Human Services. (1997). Food safety from farm to table. A National food safety initiative. A report to the president. U.S. Environmental Protection Agency, Washington, D.C.