Using Research Methods to Evaluate Your Extension Program

For the Extension practitioner, research is typically considered an ominous practice reserved for ivory tower academics, and evaluation is nearly as mysterious. Therefore, Extension agents often shy away from using scientific methods to evaluate educational programs. However, research is simply a methodical way of finding answers to questions, to be used to discover new information, or to prove scientific theories. And research methods can also be useful to effectively evaluate an educational program or its participants in the most objective way.

For evaluation purposes, the questions to be answered might be "Did this program meet its objectives?" or "How effective was the program in achieving desired results?" The challenge for the evaluator is to choose the most appropriate methods to systematically answer such questions. This article presents an overview of the types of research that might be used in program evaluation, organized by the prospective purposes of your study.

Start by Learning the "Lingo"

Research methods don't seem so intimidating when you're familiar with the terminology. This is important whether you're conducting evaluation or merely reading articles about other studies to incorporate in your program. To help with understanding, here are some basic definitions used.

• Variable: Characteristics by which people or things can be described. Must have more than one level; in other words, to be able to change over time for the same person/object, or from person to person, or object to object. Some variables, called attributes, cannot be manipulated by the researcher (e.g., socioeconomic status, IQ score, race, gender, etc.). Some variables can be manipulated but are not in a particular study. This occurs when subjects self-select the level of the independent variable, or the level is naturally occurring (as with ex post facto research).
  
  
• Manipulation: Random assignment of subjects to levels of the independent variable (treatment groups).
  
  
• Independent variable: The treatment, factor, or presumed cause that will produce a change in the dependent variable. This is what the experimenter tries to manipulate. It is denoted as "X" on the horizontal axis of a graph.
  
  
• Dependent variable: The presumed effect or consequence resulting from changes in the independent variable. This is the observation made and is denoted by "Y" on the vertical axis of a graph. The score of "Y" depends on the score of "X."
  
  
• Population: The complete set of subjects that can be studied: people, objects, animals, plants, etc.
  
  
• Sample: A subset of subjects that can be studied to make the research project more manageable. There are a variety of ways samples can be taken. If a large enough random sample is taken, the results can be statistically similar to taking a census of an entire population--with reduced effort and cost.

One of your first decisions to make is whether to use qualitative or quantitative research methods. Quantitative research focuses on measurement and counting, attempting to categorize and summarize using numbers and labels. Qualitative research aims more at thoroughly describing a situation or explaining reasons for a problem or circumstance. It is typically thorough and provides in-depth understanding of a situation or group of people but does not attempt to quantify results. Often, both quantitative and qualitative approaches are used in a research study or evaluation effort because they provide complementary information. This article deals primarily with quantitative methods.

To select the most appropriate methods to achieve the results you are trying to achieve, it is important to match the purpose of your study with the type of research to use.

Purpose: Explore or Describe
Type of Research to Use: Descriptive Study

If you are looking to gain insight into a problem or issue to better focus additional study or develop a clear research hypothesis, then the end sought is exploration. If you want to illustrate accurately and clearly the characteristics of a group or situation, then the purpose is description.

It is common for any type of research to include descriptive methods. Also, a descriptive method (such as a survey) is often used as the data collection technique for all kinds of research. Start with a research question or research objective.

Examples of research questions:

• How many farmers use no-till methods in the county?
• What is the household income of EFNEP participants?
• How many projects does a typical 4-H member complete each year?

Examples of research objectives:

• To determine the average number of acres of corn planted by dairy farmers.
• To determine the average number of calories consumed per person per day in the household.
• To determine the median family income of children enrolled in after-school child care programs.

Types of descriptive studies include survey research and developmental and case studies.

Survey Research

The researcher gathers data from a large group of subjects, usually via mail, telephone, or in-person interviews. Because information is gathered at one point in time, survey research is sometimes referred to as a "status" or "normative" study. Relationships between variables are not explored. Examples include public opinion surveys, needs assessments, follow-up studies, etc.

Developmental Study

A developmental study is survey research where surveys are taken at different points in time and compared. For instance, take longitudinal studies.

• Trend study: General populations are sampled at each collection point. (Example: a sample of participants from a specific Extension course is taken and studied every year. The sample differs each year.)
  
  
• Cohort study: A specific population is followed over a period of time and sampled at each data collection point. (Example: a sample of this year's participants in an Extension course is taken and studied, and a different sample taken from this year's participants is taken and studied next year and successive years.) Although the same population is studied each year, the sample from that population is different each year.
  
  
• Panel study: An identical sample taken from the initial population at the initial data collection point is used at each data collection point. (Example: a sample of this year's participants in an Extension course is taken and studied for successive years.) Although it is difficult to keep in contact with an identical group over a long period of time, this allows changes in both the group and the individuals in the group to be studied over time.

Case Study

A case study is conducted for similar purpose as the above but is usually done with a smaller sample size for more in-depth study. A case study often involves direct observation or interviews with single subjects or single small social units such as a family, club, school classroom, etc. This is typically considered qualitative research.

Purpose: Explain or Predict
Type of Research to Use: Relational Study

In a relational study you start with a research hypothesis, that is, is what you're trying to "prove."

Examples of research hypotheses for a relational study:

• The older the person, the more health problems he or she encounters.
• 4-H members attending 4-H summer camp stay enrolled in 4-H longer.
• The greater the number of money management classes attended, the greater the amount of annual savings achieved.

Types of relational studies include correlational studies and ex post facto studies.

Correlational Study

A correlational study compares two or more different characteristics from the same group of people and explains how two characteristics vary together and how well one can be predicted from knowledge of the other.

A concurrent correlational study draws a relationship between characteristics at the same point in time. For example, a student's grade point average is related to his or her class rank.

A predictive correlational study could predict a later set of data from an earlier set. For example, a student's grade point average might predict the same student's grade point average during senior year. A predictive correlational study could also use one characteristic to predict what another characteristic will be at another time. For example, a student's SAT score is designed to predict college freshman grade point average.

Ex Post Facto (After the Fact) Study

An ex post facto study is used when experimental research is not possible, such as when people have self-selected levels of an independent variable or when a treatment is naturally occurring and the researcher could not "control" the degree of its use. The researcher starts by specifying a dependent variable and then tries to identify possible reasons for its occurrence as well as alternative (rival) explanations. Such confounding (intervening, contaminating, or extraneous) variables are "controlled" using statistics.

This type of study is very common and useful when using human subjects in real-world situations and the investigator comes in "after the fact." For example, it might be observed that students from one town have higher grades than students from a different town attending the same high school. Would just "being from a certain town" explain the differences? In an ex post facto study, specific reasons for the differences would be explored, such as differences in income, ethnicity, parent support, etc.

It is important to recognize that, in a relational study, "cause and effect" cannot be claimed. All that can be claimed is that that there is a relationship between the variables.

For that matter, variables that are completely unrelated could, in fact, vary together due to nothing more than coincidence. That is why the researcher needs to establish a plausible reason (research hypothesis) for why there might be a relationship between two variables before conducting a study. For instance, it might be found that all football teams with blue uniforms won last week. There is no likely reason why the uniform color had any relationship to the games' outcomes, and it certainly was not the cause for victory. Similarly, you must be careful about claiming that your Extension program was the "cause" of possible results.

Purpose: Determine Cause and Effect
Type of Research to Use: Experimental or Quasi-Experimental Study

An experimental study start with development of a research hypothesis, that is, what you're trying to "prove." Such a research hypothesis is likely based on professional experience or review of prior studies.

Examples of research hypotheses for an experimental study:

• Youth who complete the school enrichment program will have higher math scores.
• Flossing teeth daily prevents gum disease.
• High blood pressure causes heart attacks.
• "Pesticide B" eliminates "Disease A" in soybean crops.
• Participants who complete the course will have increased household incomes.

Experimental research is a methodical way of comparing two or more groups to determine differences in the effect of different treatments received by each group. In experimental research, the researcher purposely manipulates a treatment (independent variable) to see if it causes a change in the dependent variable (effect). A treatment can be an educational program, new drug, herbicide, or procedure that is being tested for its "effect" on the dependent variable.

An example would be giving a new reading program to one group of students and using the old way of teaching reading to a different group of students to see if the new way yields higher reading scores. Extraneous variables are also controlled by the researcher so they can be ruled out as other possible" causes." Experimental research is the only type of study where true "cause and effect" can be claimed.

A true experiment requires the random assignment of subjects (such as people, animals, or plants) to a treatment group. Random assignment is the only way that groups can be considered statistically equivalent.

In a quasi-experiment, groups of subjects are constructed using a method other than random assignment. When using human subjects, it is often impossible to do random assignment. They are often part of intact groups such as school classrooms, community organizations, neighborhoods, 4-H clubs, or nursing homes. Although groups might be reasonably similar in a practical sense, using data from intact groups limits the conclusions that can be drawn regarding program effects. Still, quasi-experiments are useful in providing valuable evidence of program impacts. This is a highly under-utilized evaluation method that has great potential for determining the impact and value of educational programs.

A pre-experimental design has little control over environmental factors that could affect the outcome of a study. For example, a one-group, pretest/posttest design doesn't even use another group for comparison. But such a design does provide some evidence of program impact (with major limitations in the conclusions that can be drawn) and is commonly used when more elaborate designs are not possible. One-group designs can be strengthened as an evaluation method by simply adding a comparison group.

Conclusion: There Is No "Holy Grail" of Program Evaluation

Each research method has benefits, but no method alone is likely to solve all your problems or answer all your research questions. That is why methods are often combined. It just may not be possible to conduct a single study to give a complete and definitive result. Studies are often repeated over time. The most important recommendation is to choose methods that meet your needs and to conduct the study in a careful, thorough, and objective way. Then, you can be confident that your findings can be believed. Therefore, pay attention to the purpose of your study and match up the methods that help achieve that purpose.

The following references will provide help as you select the research methods to use.

Ary, D., Jacobs, L. C., & Razavieh, A. (1985). Introduction to research in education. New York, NY. Holt, Rinehart and Winston.

Brethower, D.M., Brinkerhoff, R. O., Hluchyj, T., & Nowakowski, J. R. (1983). Program evaluation: A practitioner's guide for trainers and educators. Boston, Massachusetts. Kluwer-Nijhoff Publishing.

Campbell, D., & Stanley, J. C. (1963). Experimental and quasi experimental designs for research. Chicago, Illinois: Rand McNally Co.

Diem, K. (1999). Choosing appropriate research methods to evaluate educational programs. Rutgers Cooperative Extension Fact Sheet #FS943. New Brunswick, NJ.

Diem, K. (1997). Measuring impact of educational programs. Rutgers Cooperative Extension Fact Sheet #869. New Brunswick, NJ

Gay, L. R. (1981). Educational research: Competencies for analysis & application. Columbus, Ohio. Bell & Howell Company. 1981.

Hagen, E. P., & Thorndike, R. L. (1977). Measurement and evaluation in psychology and education. New York, New York. John Wiley & Sons.

Wentling, T. L. (1980). Evaluating occupational education and training programs. Boston, Massachusetts. Allyn and Bacon, Inc.