Using an Immediate Feedback Tool to Improve Learning and Facilitate Program Evaluation

Surveys are commonly used to collect information about people's knowledge and behaviors (Caffarella & Daffron, 2013; Fink, 2017; Stup, 2003). University Extension personnel are asked to document program impact, and land-grant universities have a strong evaluation culture (Caffarella & Daffron, 2013; Franz, Arnold, & Baughman, 2014). Surveys are usually prepared and administered by Extension personnel, and there are many ways to formulate questions (Fink, 2017; Taylor-Powell, 1998; Taylor-Powell & Hermann, 2000). Educators tend to think closed-ended questions (e.g., multiple-choice items) are reliable and can be quickly and easily scored (Taylor-Powell, 1998); however, it can be difficult to force specific responses as occurs with multiple-choice items with certain topics that could have multiple outcomes (Epstein et al., 2002; Fink, 2017).

Regardless of format, merely asking questions without providing feedback to learners regarding their answers may not foster their learning and, ultimately, retention. Feedback is information learners receive about their learning process; positive feedback enhances learner motivation (Caffarella & Daffron, 2013). Usually people want to know whether they answered questions correctly, and not knowing can actually cause anxiety (Epstein et al., 2002), which may affect learning and retention. Often questions are related to one another, and answering incorrectly results in similar errors later in the survey. Moreover, understanding the correct answer also can help reinforce new learning. By using active involvement in the evaluation process, one's acquisition and retention of correct information increases (Epstein et al., 2002).

Research has demonstrated that adults learn differently than younger people (Knowles, 1996). Andragogy, or the science of teaching adults, consistently indicates that adults (a) need to know why they should be learning something, (b) have a greater volume and different quality of experience than youths, and (c) enter a learning experience with a problem-centered orientation (Knowles, 1996). Adults can be challenging to educate, as they often like to validate the information they receive according to their established beliefs and experiences. Many Extension programs are voluntary learning opportunities, and adults expect the topics to be immediately useful. Additionally, adults generally have less free time to dedicate to learning, so the experience must be well organized and efficient to stimulate learning (McGrath, 2009).

One efficient method of educating adults involves use of a card-based system that delivers immediate feedback for enhanced learning. The immediate feedback assessment technique (IFAT) developed by Epstein, Epstein, and Brosvic (2001) promotes active knowledge acquisition and retention of subject materials. Extension professionals interested in applying IFAT can obtain IFAT cards for about $0.20 per card from Epstein Educational Enterprises (www.epsteineducation.com/home/about/). To our knowledge, this is the only source of a low-technology evaluation tool that provides immediate feedback. There are options for the number of questions and response options per card. The cards have a thin film covering the answer options, making them nearly impossible to see through (Figure 1, Part a). Just like a lottery ticket, the participant scratches off an answer on the card. If the participant selects the correct response, a star appears within the rectangle and immediately reinforces the participant's understanding (Figure 1, Part b). If the participant selects an incorrect response, the rectangle is blank and the participant can select another response. With the IFAT cards, participants can readily confirm the correct answer to every question if they keep trying. The multiple-choice questions combined with the immediate feedback from the IFAT cards provide an organized and efficient method of information delivery to enhance learning and retention by adult program participants.

Figure 1.
Example of Immediate Feedback Assessment Technique (IFAT) Cards, Including (a) a New Card and (b) a Completed Card

Note. Comparison of the new card and completed card indicates that the participant incorrectly answered questions 2 and 4 but was able to try again and eventually select the correct responses.

Use of IFAT in Extension

Our field crop entomology Extension program is focused on promoting integrated pest management and insecticide resistance management strategies. With over 10 million ac of soybean in Iowa having over $4.85 billion in production value (U.S. Department of Agriculture National Agricultural Statistics Service, 2018), crop protection from insect pests is a critical issue. With a combination of publications, presentations, and social media, we promote use of multiple proactive tactics to protect crop yield from these persistent or occasional pests.

In the past, IFAT cards were applied in the program to raise awareness of a new management issue for the established pest soybean aphid (Hodgson, 2018); soybean aphid has been in Iowa since 2001, but recently insecticide resistance to pyrethroids was confirmed (Hanson et al., 2017), complicating future crop protection. For the study described here, we wanted to demonstrate short-term learning via Extension programming for a new pest in Iowa—soybean gall midge. Soybean gall midge is a new soybean pest in the north central region of the United States (e.g., Iowa, Nebraska, Minnesota, and South Dakota). This small fly was described as a new species, Resseliella maxima (Diptera: Cecidomyiidae), only recently (Gagné, Yukawa, Elsayed, & McMechan, 2019). Soybean gall midge was first detected in Iowa in 2015 in O'Brien County, and distribution in western Iowa counties has been slowly increasing (Hodgson & Dean, 2018). In 2018, there was an exponential increase in positive detections, with notable economic injury in soybean (McMechan, Hunt, & Wright, 2018). We wanted to focus our Extension program on raising awareness of this new pest in Iowa and evaluate our efforts with IFAT evaluation cards.

Materials and Methods

Between November 2018 and February 2019, we spoke about soybean gall midge for about 45–50 min at 12 Iowa State University Extension and Outreach events in Iowa (Author Hodgson led the program at seven locations, and Author Dean led the program at five locations [sometimes multiple sessions per location]). Three learning objectives were outlined at the beginning of each presentation: (a) describe the identification, biology, and life cycle of soybean gall midge, (b) promote scouting and recognizing plant injury, and (c) raise awareness about research plans for 2019. Participants received an IFAT card as they walked in the room. After reviewing the learning objectives, we announced the plan for a "pretest versus posttest" style quiz, comprised of the same set of five questions asked at the beginning and end of the session. Given that our presentation covered a new pest in the state, it is unlikely that the participants had knowledge of the information presented prior to the evaluation; however, it is not impossible that they might have known the answers to some questions. We designed multiple-choice questions to produce uniform responses (Taylor-Powell, 1998). Each card had 10 questions with four response options: Questions 1–5 were for the pretest, and questions 6–10 were for the posttest. In an effort to increase the response rate, we ensured that voluntary participants would remain anonymous (Fink, 2017; Taylor-Powell & Hermann, 2000). We asked people to use a pen, coin, or key to scratch the IFAT cards. Five questions were embedded into a PowerPoint presentation, with one question and four possible response options displayed per slide (Table 1).

Table 1.
Pre- and Posttest Multiple-Choice Questions Asked Using Immediate Feedback Cards

Question	Response optionsa
1. Soybean gall midge is what kind of insect?	(a) moth (b) fly (c) beetle (d) bug
2. The immature life stage of soybean gall midge is called a what?	(a) maggot (b) grub (c) caterpillar (d) nymph
3. Where is soybean gall midge injury concentrated?	(a) roots (b) flowers (c) near the soil line (d) near the top of the canopy
4. Soybean gall midge maggots (3rd instars) are what color?	(a) purple (b) yellow (c) white (d) orange
5. Adult soybean gall midge are described by which of the following body characters?	(a) slender body, long legs, short antennae (b) slender, hairy body; long antennae (c) stout body, bristle-like antennae (d) stout, hairy body; humpbacked
aCorrect response is in bold.

During the pretest, participants using the IFAT cards instantly knew whether they had answered questions correctly or could continue to guess until the correct answer was selected. We did not review the correct answers before starting the presentation, and this may have induced anxiety for some participants (Caffarella & Daffron, 2013; Epstein et al., 2002). However, we assured the groups that the questions would be addressed during the session and allowed time for discussion prior to the posttest. During the posttest at the end of the session, we asked the same five questions having the same four possible response options but rearranged the order of the response options. Again, participants could continue to guess until the correct answer was selected if they did not select it at first. We also reviewed the learning objectives at the end of the presentation as a form of repetition. We collected the cards at the end of each session and kept them separated by location.

From our observations, some participants did continue to guess if they did not initially answer correctly, but some did not attempt to get the correct answer. We assumed that if there were multiple guesses for a question, the participant had answered incorrectly initially. Some participants did not answer one or more questions on the pretest or posttest, which resulted in nonresponses (i.e., blanks) in the data collection. We did not use any cards having blanks; in other words, we used only cards that were completely filled out.

We used a logistic regression to examine the short-term learning effect of the participants. In detail, we used to denote the participant ID and to denote the quiz question ID. We set to be the binary "correct/incorrect" indicator of the answer from participant to question in the pretest, where indicates a correct answer and indicates an incorrect answer. Correspondingly, we used to denote the binary "correct/incorrect" indicator of the answer in the posttest. Therefore, and are the probabilities of giving the correct answer before the presentation and after the presentation, respectively, and are modeled as

Here, the logit link function is the most commonly used link to model binary data.

Odds of giving the correct answer is defined as the ratio of the probability of giving the right answer to the probability of giving a wrong answer, which represents the participant's ability to solve the question. In detail, before the presentation (i.e., pretest), the odds of participant giving the correct answer to question is . Here, this odds is before the presentation and after the presentation. Thus is the multiplicative change in the odds due to knowledge gained from the presentation, and represents the short-term learning effect on question . More specifically, means that the odds of giving the correct answer to question is increased after the presentation (i.e., a positive short-term learning effect), and means that the odds remains the same after the presentation (i.e., no improvement on question ). Therefore, to examine the significance of the learning effect is to test whether the value of is positive or not.

Because the parameter of interest is and s are nonrelevant nuisance parameters, the conditional likelihood is constructed, providing a reliable estimation of (Agresti, 2003). To estimate the overall learning effect of the presentation, we changed the model of into the following equation:

Here, has no subscript , meaning that it describes the overall learning effect over all five quiz questions. See the appendix for additional notes on model estimation and interpretation.

Results

In total, 931 people attended our sessions, and 584 people completed the IFAT cards, for a 62.72% response rate. More than half of attendees participated in our evaluation. Another 53 attendees (5.69%) partially filled out the cards, but their cards were not included in the overall analysis. Audience members at the events were primarily farmers and male. For each question, participants answered more correctly on the posttest than on the pretest (Table 2). For example, question 2 was answered correctly by 52.32% of participants on the pretest; this increased to 96.83% on the posttest, for an 85% increase. The greatest increase (150%) was observed for Question 5 (Table 2), with correct responses increasing from 33.79% to 84.39% between the pre- and posttests.

Table 2.
Comparisons of Percentages of Correct Answers on Pre- and Posttest Questions and Learning Effects Estimation

Question	% correct pretesta	% correct posttesta	βk + SEb,c	exp(βk)d
1	60.72	94.17	2.52+0.25*	12.34
2	52.32	96.83	3.06+0.30*	21.33
3	57.98	98.80	4.39+0.58*	80.64
4	60.03	97.94	3.64+0.41*	38.09
5	33.79	84.39	2.32+0.19*	10.18
Overall	52.97	93.83	2.89+0.12*	17.99
aTotal number of responses is 584. bβk = the short-term learning effect of the program on question k. cAn asterisk indicates that improvement in correct posttest responses is significant. dexp(βk) means that the odds of giving the right answer to question k is multiplied by exp(βk) after the presentation.

After fitting the model using our survey data of 584 complete cases, we obtained the estimation results shown in Table 2. All terms are significant and positive. This indicates that short-term learning occurred by participants for all five questions asked during the presentation. The overall short-term learning effect is estimated as 2.89, with standard error of 0.12 (Table 2). The overall odds of giving the correct answer is 18 times greater after the presentation. This result indicates a significant positive effect on short-term learning in our Extension program. By comparing the question responses before and after the presentation, we can conclude that participants benefited from the program by increasing learning overall and for each question individually.

Discussion

For Extension personnel, program evaluation can be used to determine whether an Extension program has met the desired learning objectives (Caffarella & Daffron, 2013). The outcomes from an evaluation can help formulate future programming needs (Conner, Dev, & Krause, 2018). Specifically, evaluative data can help with adding and adjusting learning objectives, justifying resource allocations, and changing delivery methods to be more effective (Caffarella & Daffron, 2013). In addition, documented program successes can be used in U.S. Department of Agriculture National Institute of Food and Agriculture annual reports and the Council of Agricultural Research, Extension, and Teaching updates required by most land-grant universities for continued financial support.

Looking forward, implementing program evaluation will continue to be an important focus for university Extension personnel. Unfortunately, many universities around the United States have declining capacities, with fewer staff and reduced financial support (Franz, 2011). We wanted to improve adult participant learning about field crop entomology in Iowa by providing immediate feedback regarding responses to multiple-choice questions but had a limited evaluation budget. Therefore, we used the IFAT technique to do both and can communicate the following information gained from doing so:

• Unlike traditional paper surveys, IFAT cards are a unique tool to use in an Extension setting (Hodgson, 2018). The responses are easy to score and quickly summarized. The tool is highly flexible and thus can be used in various venues (e.g., outdoor or low-technology venues).
• Adults were receptive to the immediate feedback technique as a tool for promoting learning, as indicated by a 62.6% response rate from voluntary participants at 12 Extension events.
• IFAT cards offered an organized and efficient learning experience for adult participants; they also were easy and quick to score.
• The use of the logistic regression model was an effective way to describe the short-term learning effect comparing pre- and posttest evaluations.
• Adults demonstrated increased short-term learning after a presentation and engagement in learning activities.
• Closed-ended evaluations provide reliable data about participant knowledge and behaviors; however, they do not generate opinions and discussion like other survey tools. Use of IFAT cards alleviates this situation.

Acknowledgments

We thank Chris DiFonzo (Michigan State University) and Matt O'Neal (Iowa State University) for suggesting use of the IFAT cards in Extension and Zhulin He for assisting with the statistical analysis. Our work was partially supported by the North Central Soybean Research Program, the Iowa Soybean Association and the soybean checkoff, and the North Central Integrated Pest Management Center.

References

Agresti, A. (2003). Conditional logistic regression for binary matched pairs. In Categorical Data Analysis, Wiley Series in Probability and Statistics (2nd ed., pp. 414–420). Hoboken, NJ: Wiley.

Caffarella, R. S., & Daffron, S. R. (2013). Planning programs for adult learners: A practical guide (3rd ed.). San Francisco, CA: Jossey-Bass.

Conner, N. W., Dev, D., & Krause, K. (2018). Needs assessment for informing Extension professional development trainings on teaching adult learners. Journal of Extension, 56(3), Article v56-3a1. Available at: https://joe.org/joe/2018june/a1.php

Epstein, M. L., Epstein, B. B., & Brosvic, G. M. (2001). Immediate feedback during academic testing. Psychological Reports, 88, 889–894.

Epstein, M. L., Lazarus, A. D., Calvano, T. B., Matthews, K. A., Hendel, R. A., Epstein, B. B., & Brosvic, G. M. (2002). Immediate feedback assessment technique promotes learning and corrects inaccurate first responses. The Psychological Record, 52, 187–201.

Fink, A. (2017). How to conduct surveys: A step-by-step guide (6th ed.). Los Angeles, CA: SAGE.

Franz, N. K. (2011). Advancing the public value movement: Sustaining Extension during tough times. Journal of Extension, 49(2), Article v49-2comm2. Available at: https://www.joe.org/joe/2011april/comm2.php

Franz, N., Arnold, M., & Baughman, S. (2014). The role of evaluation in determining the public value of Extension. Journal of Extension, 52(4), Article v52-4comm3. Available at: https://www.joe.org/joe/2014august/comm3.php

Gagné, R., Yukawa, J., Elsayed, A. K., & McMechan, A. J. (2019). A new pest species of Resseliella (Diptera: Cecidomyiidae) on soybean (Fabaceae) in North America with a description of the genus. Proceedings of the Entomological Society of Washington, 12, 168–177.

Hanson, A. A., Menger-Anderson, J., Silverstein, C., Potter, B. D., MacRae, I. V., Hodgson, E. W., & Koch, R. L. (2017). Evidence for soybean aphid (Hemiptera: Aphididae) resistance to pyrethroid insecticides in the upper midwestern United States. Journal of Economic Entomology, 110(5), 2235–2246. doi:10.1093/jee/tox235

Hodgson, E. W. (2018). Using immediate feedback to improve short-term learning in Extension. Journal of Integrated Pest Management, 9(1). doi:10.1093/jipm/pmy001

Hodgson, E. W., & Dean, A. (2018). Soybean gall midge: A new field crop pest (Iowa State University Extension and Outreach CROP 3157). Retrieved from https://store.extension.iastate.edu/Product/15512

Knowles, M. S. (1996). Adult learning. In R. L. Craig (Ed.), The ASTD training and development handbook: A guide to human resource development (4th ed., pp. 253–265). New York, NY: McGraw-Hill.

McGrath, V. (2009) Reviewing the evidence on how adult students learn: An examination of Knowles' model of andragogy. Adult Learner: The Irish Journal of Adult and Community Education, Publication EJ860562. Retrieved from https://files.eric.ed.gov/fulltext/EJ860562.pdf

McMechan, J., Hunt, T., & Wright, R. (2018). Soybean gall midge: Adult stage identified. Retrieved from University of Nebraska–Lincoln CropWatch website: https://cropwatch.unl.edu/2018/soybean-gall-midge-adult-stage-identified

Stup, R. (2003). Program evaluation: Use it to demonstrate value to potential clients. Journal of Extension, 41(4), Article 4COM1. Available at: https://www.joe.org/joe/2003august/comm1.php

Taylor-Powell, E. (1998). Questionnaire design: Asking questions with a purpose (Program Development and Evaluation G3658-2). Madison WI: University of Wisconsin–Madison, Division of Extension.

Taylor-Powell, E., & Hermann, C. (2000). Collecting evaluation data: Surveys (Program Development and Evaluation G3658-10). Madison WI: University of Wisconsin–Madison, Division of Extension.

U.S. Department of Agriculture National Agricultural Statistics Service. (2018). 2018 state agriculture overview. Retrieved from https://www.nass.usda.gov/Quick_Stats/Ag_Overview/stateOverview.php?state=IOWA

Appendix

Notes on Model Estimation and Interpretation

In our logistic regression model

,

has subscripts to account for the variation in difficulty of the questions and knowledge of the participants. The parameter of interest is (short-term learning effect), and s are nonrelevant nuisance parameters. We used the following conditional distribution by which the terms are cancelled out:

Then, the conditional likelihood is constructed, which provides a reliable estimation of (Agresti, 2003).

Note that is the multiplicative change in the odds of giving the correct answer, not the change in the probability of giving the correct answer. The probability of a participant's giving the correct answer is (pretest) versus (posttest). So, the multiplicative change in this probability is , which varies among participants, not . If a participant is already knowledgeable before a presentation, the probability of knowing the correct answer in the pretest is high and cannot be greatly improved at posttest. Therefore, the change in the probability of giving the correct answer varies among different participants, but the change in the odds are the same across participants.