Using Evaluation to Guide Program Content: Diabetes Education

Introduction

While Extension nutrition educators have the primary responsibility to provide "normal nutrition" education, changes in society have produced consumers with a wide array of diet and health questions. Extension programming has responded to these questions with programs addressing diet and heart disease, weight maintenance, and diabetes. Even 20 years ago, 93% of agents in Florida received questions concerning modified diets (Lenicheck, Anderson, & Tichenor, 1986).

The incidence of diabetes continues to increase for all ages, both genders, and all races and educational levels (Mokdad et al., 2002). A critical element of diabetes care is patient education. Optimally, diabetes education begins in the clinical setting with a health care team that includes a physician, nurse, dietitian, and pharmacist. The patient is followed by this team as needed but at least twice per year to review glycemic control (American Diabetes Association, 2003). The health care of the person with diabetes is referred to as "diabetes self-management education" because the person with diabetes is taught how to manage their food, medication, and exercise as an ongoing process (Mensing et al., 2002).

However, the majority of people with diabetes do not receive any formal diabetes education (Coonrod, Betschart, & Harris, 1994). Because food is an integral part of diabetes management, it is logical that Extension educators receive many requests for diabetes information.

In fact, diabetes education has become widespread in Extension programming. Three years ago, Extension educators in Illinois began a program that would increase the knowledge of healthy food choices for those with diabetes.

Diabetes Education in Illinois

The "Dining with Diabetes" program originated in the West Virginia Extension Service in 1997. The program included three lessons and an optional class reunion held 6 months after the class. Each class lasted approximately 2 hours, covering desserts, main dishes, and side dishes, respectively. The program was based on the Stages of Change and the Social Cognitive Theory and included evaluation tools to demonstrate program impact. One aspect of the evaluation was food-related diabetes knowledge. Within seven knowledge questions, there were 31 items that participants could correctly or incorrectly select.

Classes were held in local community sites and taught by Extension educators. Participants were given an outline of the lesson and educational handouts at the beginning of each class. At the end of the class, recipe demonstrations and taste-testing emphasized key concepts of the three lessons. Prior to Lesson One, participants completed a pre-test. Classes were usually spaced 1 to 2 weeks apart. After the third lesson, participants completed a post-test.

Participants for the "Dining with Diabetes" Program were recruited from newspapers, physicians' offices, radio advertisements, and public bulletins, and through the local health departments. The program targeted those with diabetes as well as caregivers. Participants were not required to complete pre-tests/post-tests to participate in the educational program, although very few declined. The mean age of the participants was 62 + 12 years. Most were white (90%) females (83%), who rated their health as good or excellent (75%). Over half (59%) had diabetes themselves, and, of those, most had known they had diabetes for longer than 5 years. Most were high school graduates (44%).

Evaluation of the Diabetes Program in Illinois

The program in Illinois followed the guidelines of the Institutional Review Board and collected pre- and post-education information that was analyzed at a central location. The knowledge questions were scored as correct or incorrect and summed for a composite score. A paired t-test of pre- vs. post-education knowledge showed a significant improvement in knowledge (67 vs. 81% correct, p<.0001, Table 1).

Further analysis divided the knowledge questions into those addressing the carbohydrate content of foods and those addressing fats listed on the Nutrition Facts label. Both groups of questions were statistically improved by about 10% (~70 to 80% correct, p<.001). What was disturbing was that at post-education, more than half scored less than 80% correct in identifying foods that contain carbohydrates (Table 2).

This evaluation was not a "true experiment" in that participants were not randomly chosen and there was not "control" group. However, quasi-experimental designs such as this can provide important information about a program's impact, even if the results cannot be generalized to the whole population (Diem, 2002).

Using Evaluation to Focus Program Content

The results were reviewed at an Educator Team meeting where the consensus was to re-evaluate the lesson content and focus on two or three main ideas per lesson. The main ideas were identified at the meeting by brainstorming and listing all possible main ideas for each lesson and then voting on the two that seemed to be the most important. The main ideas for each of the original and new lessons are as follows (Table 3).

This process of using the evaluation of the diabetes education program to sharpen the focus of the program exemplifies how evaluation can be used in Extension programming. Optimally, the evaluation would demonstrate a change in behavior. However, although knowledge alone doesn't change behavior, without it, behavior change is unlikely. For instance, a goal of having a consistent number of carbohydrates eaten throughout the day cannot be achieved if the participant doesn't know which foods contain carbohydrates. It may seem to be a step backwards--measuring what is learned rather than how it is applied. However, knowledge is a vital step in the climb towards a healthy diet.

Discussion of the Process

Certainly the process doesn't stop here. Learning is essential but not enough. Stakeholders want documentation of behavior change as a result of Extension programs. To ensure that those changes occur, however, learning must occur. Once a good program demonstrates the level of learning that is desired by the Extension team, then the team can address attitudes and behaviors that should change as a result.

The process of evaluating our diabetes program was not overtly fashioned on the logic model. However, several aspects of the Logic Model (Madison, 2001) pertain to this evaluation and may guide future evaluations of the program. Clearly, we have turned our focus onto the short-term outcomes of knowledge change. Medium-term outcomes will include behavior and practice changes, while long-term outcomes may address social or economic changes (Arnold, 2002).

Currently we are testing the evaluation questions for reliability and validity with our current participants. This was omitted when we first "adopted" the "Dining with Diabetes" program because the program contained its own evaluation questions. However, as we change the program to have a different focus, the evaluation questions must also change. Having reliable and valid questions is vital to having meaningful evaluations.

Using evaluation in the process of improving a program is sometimes called "formative evaluation." By looking at the results of prior programs and discussing if that is really what the intended learning was meant to be, Educators have some time for reflection on the direction of their programming efforts. Formative evaluation provides an opportunity to be critical of the program as a group. The critique is not fashioned for personnel appraisal but for program appraisal. Developing and delivering better programs is the result.

References

American Diabetes Association. (2003). Standards of medical care for patients with diabetes. Diabetes Care, 26, S23-S50.

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with learning assessment. Journal of Extension [On-line]. 40(3). Available at: http://www.joe.org/joe/2002june/a4.html

Coonrod, B. A., Betschart, J., & Harris, M. I. (1994). Frequency and determinants of diabetes patient education among adults in the US population. Diabetes Care, 17, 852-858, 1994.

Diem, K. G. (2002). Using research methods to evaluate your Extension program. Journal of Extension [on-line]. 40(6). Available at: http://www.joe.org/joe/2002december/a1.shtml

Lenicheck, J., Anderson, C. R., & Tichenor, D. (1986). The changing face of nutrition education. Journal of Extension [On-line]. 24(2). Available at: http://www.joe.org/joe/1986summer/a5.html

Logic model development guide. (2001). W. K. Kellog Foundation [On-line]. Available at: http://www.wkkf.org/Programming/ResourceOverview.aspx?
CID=281&ID=3669

Mensing, C., Boucher, J., Cypress, M., Weinger, K., Mulcahy, K., Barta, P., Hosey, G., Kopher, W., Lasichak, A., Lamb, B., Mangan, M., Norman, J., Tanja, J., Yauk, L., Wisdom, K., & Adams, C. (2002). National standards for diabetes self-management education. Diabetes Care, 25 (suppl 1), S140-S147.

Mokdad, A. H., Ford, E. S., Bowman, B. A., Dietz, W. H., Vinicor, F., Bales, V. S., & Marks, J. S. (2003). Prevalence of obesity, diabetes, and obesity-related health risk factors, 2001. Journal of the American Medical Association, 289(1), 76-79.