Wireless Audience Response System: Does It Make a Difference?

Introduction

Many Extension programs use traditional workshops and assume they increase the audiences' understanding of specific educational objectives. Numerous studies have demonstrated that using a mixture of learning methods (hearing, seeing, discussing, etc.) increases knowledge retention in participants (Richardson, 1994). Because Extension  seminars are costly both in preparation and delivery, questions about the costs of various educational methods and their effectiveness are important to consider.

One such method is a technology-assisted workshop using a computer-facilitated, interactive testing system. The OptionFinder® (Audience Response Device [ARD]) is an innovative technique, believed to grab audiences' attention and increase retention. Similar to systems used in television game shows, it consists of individual wireless remote keypads given to audience members and a control station operated by the lecturer or coordinated with an assistant. With this technology, a speaker can ask a multiple choice or true/false question and, within seconds, assess the response of a very large audience. Lecturers can immediately present the results to the audience for discussion and reinforcement.

Extension professionals often evaluate short-term outcomes, such as knowledge retention over time, to assess the general effectiveness of their workshops (Arnold, 2002). While our study focused on short-term knowledge retention, it is important to note that long-term behavioral change is typically the desired outcome of Extension programs. As Extension professionals, we hope that knowledge retention will lead to behavioral change; however, measuring long-term effects is difficult and beyond the scope of this study.

Background and Methodology

The study described here aimed to determine the effectiveness of interactive workshop techniques in achieving knowledge retention 6 months after a workshop was conducted. In 2001, we gave a series of lectures on pocket gopher control as part of three statewide vertebrate pest workshops.

Pocket gophers are small mammals that cause extensive damage throughout California's agricultural industry and urban landscapes. Covering basic information on breeding habits, burrowing systems, and control strategies, the 45 minute PowerPoint®-based lecture focused on increasing awareness of proper control methods including the safe and legal use of pesticides. The California Pesticide Applicators Professional Association (PAPA) and the Vertebrate Pest Council co-sponsored these workshops as part of the annual continuing education training required for certified pesticide applicators (http://www.papaseminars.com).

In addition to the lecture, we developed a-20 question, multiple-choice test based on the major points of the seminar. Three additional questions were asked to gauge the participants' prior knowledge of pocket gophers. Participants took the test immediately after two of our sessions in order to reinforce key information and to help determine the impact of a post-test on knowledge retention.

Three workshops were conducted in March 2001. In workshop A (n=208), the pocket gopher lecture was given with no post-test. Workshop B's (n=280) lecture was followed by the 23-question test using a paper based scantron scoring system given. In workshop C (n=273), the lecture was followed by the 23-question test administered via the OptionFinder® system. Because the OptionFinder® system was unfamiliar to most participants, we did a short training prior to the lecture by asking five questions from the post-test.

In workshop B, the test answers were discussed with the audience with the goal of reinforcing key information from the workshop. In C, the interactive nature of OptionFinder® allowed the participants to immediately know whether they answered correctly and how others responded. We hypothesized that by stimulating the audience to validate their responses in real-time, knowledge retention would be higher. Because of the uniqueness of the OptionFinder® system, we wanted to test if it had a more powerful affect on retention.

The fourth PAPA seminar (D; n=210) was added to determine the "intuitiveness" of our questions. This group, composed of participants with similar backgrounds, did not receive the pocket gopher control lecture.

Six months after the workshops, PAPA mailed a follow-up test with the same 20 questions to workshop participants in each of the four groups. Those who returned the test (survey) were awarded 1/2 hour of continuing education credit. These follow-up tests allowed us to compare information retention rates between the different workshop methods.

Results

To assess the effectiveness of each seminar on knowledge retention, we took an average of the 6-month post-test scores from each seminar group. The return rates for the 6-month post-test averaged 36.18% ± 3.24%. The average score for groups A, B, and C was approximately 80%. As expected, the score from group D, where no pocket gopher control information was presented, was significantly lower (64.6%, α=.05).

To eliminate any bias from the five questions used to acquaint the participants with OptionFinder®, we reanalyzed the responses after removing these five questions. Groups A, B, C, and D averaged 78.70%, 77.12%, 78.97%, and 63.08% respectively. Again, the relative differences in the score between groups remained unchanged.

To further evaluate whether the OptionFinder® technique was more effective in increasing knowledge retention than the other methods, we used the five least intuitive questions (i.e., Questions that people are least likely to know without attending a seminar or having previous pocket gopher knowledge). Here, the least intuitive questions were the five questions from group D (no seminar) that received the lowest percentage of correct answers. For example, one question was: "On average how many gophers typically occupy one burrow system?" Using these five questions, we calculated the average scores. Groups A, B, C, and D averaged 55.12%, 55.88%, 62.06%, and 35.14% respectively. 

Next, we looked at the retention of the most critical information from the test using five specific questions. These five questions, defined by the Extension specialist, involved either serious health risks or legal issues about using pesticides on pocket gophers. One example is this multiple-choice question: "Name of type of control method or material that may NOT be used for controlling pocket gophers." If the audience remembered nothing else from the seminars, we wanted them to remember the information represented by the five questions pertaining to health and safety issues relating to pocket gopher control. The results were as follows: Group A 83.41%, B 83.73%, C 87.63%, and D 70.00%.

With the aim of evaluating the effectiveness of the seminar in general, three questions regarding participants' involvement in gopher control were also asked on the 6-month post-test. The most "telling" question regarding participants' prior knowledge of pocket gopher control was, "Is pocket gopher control part of your job description?" (#2). Answering "yes" to this question indicates likely prior knowledge of the subject. To determine the effectiveness of the seminar, we broke down the average test scores for those who answered "no" on #2. Again, we used the tests questions excluding the five asked prior to the OptionFinder® seminar. Groups A, B, C, and D averaged 70.00%, 67.30%, 71.11%, and 50.98% respectively.

Finally, in order to determine any difference in knowledge retention between the two groups that received an immediate post-test (groups B and C), we calculated the immediate post-test scores from these two groups. Group B averaged 86.27%, while group C averaged 85.07%. Next, we compared these scores with the average scores from the 6-month post-test. Group B's average score declined 6.67%, while group C's score declined 4.55%.

Conclusions

The results suggest that neither the immediate post-test nor the interactive format had a large impact on knowledge retention 6 months after the seminar. From the original results, groups A, B, and C had overlapping confidence intervals (α=.05), meaning there were no significant differences in test scores between A, B, and C. Similar results were shown after removing the five questions given before group C's seminar, suggesting that both the OptionFinder® and simple post-test did not increase knowledge retention compared to no test at all (Figure 1).

Figure 1.
Results from the 6-Month Post-Test

A-Escondido, B-Visalia, C-San Jose, D-Tracy (α=.05)

With the interactive OptionFinder® system, group C did show a higher average on most data sets, especially on the results using the five least intuitive questions. The higher averages, however, have overlapping confidence intervals in each data set (α=.05). On the other hand, the results do confirm that the seminar itself was effective in increasing general knowledge of key points in pocket gopher control compared to similar workshop participants who did not attend the program. Each data set shows significantly higher averages in the groups that received pocket gopher control seminars compared to the group that did not (α=.05).

Although the data show that the ARD used with group C did not drastically increase the retention of key information presented at the seminar, the results did reveal a trend of higher test scores. In addition, the difference between the immediate post-test average and 6-month post-test average was less in group C than in group B, suggesting the OptionFinder® did have a positive impact on knowledge retention (Figure 1).

Despite the relatively small differences in our results, we believe there could be additional benefits to using such a device. The system can provide instant assessment of the audience's understanding of the material and is especially valuable in obtaining instant feedback from the audience. If used on a long-term basis or with a more directed and interactive method, the results on retention would likely be greater. Additionally, group C was asked via OptionFinder® whether they thought the device was useful in a seminar setting. The audience overwhelmingly recommended using the ARD in future workshops (199 recommended out of 204 participants). The value of this shouldn't be underestimated. Advantages and disadvantages of the system are listed in Table 1.

Whether or not this system will be useful for an Extension  workshop depends on the size of the audience, availability of the ARD, workshop structure, and objectives. For instance, a workshop with 300+ participants is logistically very difficult to accomplish using an ARD (as we learned the hard way). Each participant requires a wireless keypad that has to be distributed and recovered.

An additional concern is cost. According to the OptionFinder® Web site, a one-time rental of a 20-keypad OptionFinder System is approximately $2,400. Purchasing the 20-keypad OptionFinder system costs $16,530 (http://www.optionfinder.com). Obviously, large meetings will be very expensive; however, smaller class-like workshops that meet on a regular basis might use the system in a cost-effective manner. They would benefit from the system because the instructor could frequently poll the class to determine the comfort level with the material and adjust "on the fly."

As stated before, studies maintain that involving a variety of learning methods leads to a higher degree of information retention (Richardson, 1994). In our study, we found that neither the immediate post-test nor the interactive post-test significantly increased the participants' knowledge retention. This is not to say that these methods never increase knowledge retention, but simply that the value of the techniques, with respect to knowledge retention and cost effectiveness was low in our workshops.

The relative benefits and disadvantages of an ARD depend on the structure and goal of each workshop. However, a simple post-test is inexpensive (virtually free if you give a paper test), easy to develop, and requires the Extension  provider to determine the most important concepts of the workshop before developing the seminar. This helps the provider prioritize and deliver the information in a logical and meaningful way. Additionally, a post-test is an excellent method to evaluate the effectiveness of your program. While we didn't observe a large positive effect when using OptionFinder®, we believe Extension  professionals who consider and explore the methods used in this study will ultimately develop more effective Extension  programs.

References

Arnold, M. E. (2002). Be "logical" about program evaluation: Begin with leaning assessment. Journal of Extension [On-line], 40(2). Available at: http://www.joe.org/joe/2002june/a4.html

Richardson, J.G. (1994). Learning best through experience. Journal of Extension [On-line], 32(2). Available at: http://www.joe.org/joe/1994august/a6.html