December 2015 // Volume 53 // Number 6 // Tools of the Trade // v53-6tt5
Use of Interactive Electronic Audience Response Tools (Clickers) to Evaluate Knowledge Gained in Extension Programming
Effectively measuring short-term impact, particularly a change in knowledge resulting from Extension programming, can prove to be challenging. Clicker-based technology, when used properly, is one alternative that may allow educators to better evaluate this aspect of the logic model. While the potential interface between clicker technology and Extension programming has been a topic of discussion, the successful use and stakeholder attitude towards such technology in an Extension setting is not well known. This article addresses this gap in understanding and provides an assessment of clicker use in Extension programming.
Extension programming is built on the principle of providing education that impacts knowledge, behavior, or condition (Figure 1). Program evaluation, and in particular, measuring impact can be challenging as evaluation response rate and ease of data analysis may serve as barriers to educators engaging in program evaluation (Lekies & Bennett, 2011). Moreover, as changes in behavior and condition are often the focus of program evaluation, short-term outcomes such as changes in knowledge can be overlooked in the evaluation process. To truly measure knowledge change, baseline knowledge of program participants must be established. The burden associated with collecting baseline data often results in educators substituting Likert scale questions into end-of-meeting evaluations to gauge how much knowledge the participant feels that they gained as a result of the program. While this perceived change in knowledge can be useful to educators, it still does not effectively measure change in knowledge.
|Inputs||Outputs||Outcomes – Impact|
|Activities||Participation||Knowledge (Short Term)||Behavior (Medium Term)||Condition (Long Term)|
One potential and perhaps less invasive method for collecting pre- and post-program participant knowledge is through the use of interactive electronic audience response tools (clickers). In an attempt to broaden the means of data collection, Parmer, Parmer, and Struempler (2012) reported on the potential relevance of clicker use in Extension programming. Likewise, Bird and McClelland (2010) identified that clicker portability was well suited for Extension and that such technology allows educators to better engage participants as well as permits broader participation in educational settings. Furthermore, when collecting potentially sensitive data, clickers may better facilitate anonymity and increase response rate in a group environment (Ginter, Maring, Paleg, & Valluri, 2013; Carlson, 2014).
Clickers present an opportunity for educators to quickly collect demographic data as well as change in knowledge resulting from a program without the need for end-of-session, written surveys that often result in suboptimal response rates. Moreover, the accompanying clicker software (Turning Point 5; Turning Technologies, Youngstown, Ohio) may be easily evaluated by individual session or aggregated across many programs.
The use of clickers in a formal educational setting such as college classrooms has become commonplace. However, the potential benefits of clickers when used in Extension programming to document knowledge change as well as participant attitude towards use of this technology is limited. This article aims to address these gaps in understanding.
Clicker Data Collection
A statewide educational series on beef heifer development was conducted at 12 locations across Iowa. Prior to the beginning of each program, clickers were activated and synchronized with the Turning Point 5 software. Clickers were used by all participants, but clicker identification was not affiliated with individual participants.
At initiation of the meeting, as part of the Microsoft® PowerPoint® presentation, a series of six questions, including participant age, size of farming operation, and operational challenges important to heifer development, were posed via Turning Point software. Depending on the question length and whether or not multiple responses were allowed, participants were allowed between 15 and 60 seconds to answer each question with their clicker. The subsequent educational program intermittently asked three questions both before and after key concepts related to heifer development were discussed. Recorded participant answers were later compiled to determine changes in knowledge of those key concepts.
At the end of the presentation, participants used their clickers to respond to the comment "I like providing input using clickers as a partial substitute to written evaluations" on a Likert scale with options of strongly agree, agree, neutral, disagree, or strongly disagree.
In total, there were 309 attendees of the state-wide program, 245 of which were unique participants not affiliated with implementation of the program. The use of the clicker technology was successful in tracking a change in knowledge. Specifically, the correct response rate to baseline questions was 47.1%, and the correct response rate to identical follow-up questions was 87.2% (see Table 1).
Of the 245 participants at the meetings, 100% answered at least 1 clicker question through the Turning Point software and 90.6% (222/245) answered all 16 Turning Point questions using the clicker. In addition, 94.7% (231/244) of participants had a positive attitude towards use of the clicker technology, 4.5% of participants were indifferent to the technology, and only 0.8% did not like the use of clickers during the program.
|Question||Pre-presentation Correct Responses||Post-Presentation Correct Responses||Change in Knowledge|
|What is the goal for body condition score for heifers at time of calving?||50.5% (112/222)||71.9% (167/232)||+21.4%|
|What is the target weight for heifers at time of calving?||53.0% (124/234)||94.6% (227/240)||+41.6%|
|How much colostrum should a calf consume within the first 24 hrs after birth?||38.5% (94/244)||94.3% (231/245)||+55.8%|
|Average||47.1% (330/700)||87.2% (625/717)||+40.1%|
Data collected during this educational series validated clicker technology as a useful tool for collecting anonymous demographic information and measuring knowledge change of attendees. Furthermore, clicker-based technology yielded was viewed in an overwhelmingly positive manner by participants. Thus, clicker technology is well-suited as an evaluation tool in Extension programming and in particular an effective means by which to track changes in knowledge (short-term outcomes).
Bird, C., & McClelland, J. (2010). Have you used clickers in programming? Journal of Extension (On-line), 48(5), Article 5TOT9. Available at: http://www.joe.org/joe/2010october/tt9.php
Carlson, B. (2014). Using Turning Point to conduct an Extension needs assessment. Journal of Extension (On-line), 52(1), Article 1TOT3. Available at: http://www.joe.org/joe/2014february/tt3.php
Ginter, A. C., Maring, E. F., Paleg, B., & Valluri, S. (2013). Using clicker technology with rural, low-income mothers: Collecting sensitive data anonymously. Journal of Extension (On-line), 51(4), Article 4FEA3. Available at: http://www.joe.org/joe/2013august/a3.php
Lekies, K. S., & Bennett, A. M. (2011). The evaluation attitudes and practices of 4-H educators. Journal of Extension (On-line), 49(1), Article 1RIB2. Available at: http://www.joe.org/joe/2011february/rb2.php
Parmer, S. M., Parmer, G., & Struempler, B. (2012). Testing a new generation: Implementing clickers as an extension data collection tool. Journal of Extension (On-line), 50(5), Article 5TOT5. Available at: http://www.joe.org/joe/2012october/tt5.php
Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide. Benefits, 3, 22.