Reach Versus Impact: Comparing Webinars and Online Short Courses for Educational Outcomes

Introduction

Online Extension program delivery has become a common mechanism for disseminating educational materials as personnel and federal funding have remained level or decreased over the last decade (Agnew, 1991; Green, 2012; Taylor & Anderson, 2008). Extension agents and specialists are expected to reach more clientele with fewer resources. Online options, such as webinars and online courses, are tools Extension professionals can use to expand audiences and provide clientele with needed information (Rich et al., 2011). However, questions often arise regarding the reach and impact of virtual interactions. Previous research has indicated that webinars and other online educational opportunities are effective (Allred & Smallidge, 2010; Greene et al., 2010), especially when interactive components are included (McCann, 2007). There is a knowledge gap, though, regarding which online tools are the most successful in expanding program reach and having positive impacts on participants (Rich et al., 2011). As members of the eXtension Horses Community of Practice (CoP), a multistate collaborative group of university state Extension horse specialists, we undertook a study to address this gap with regard to equine-related content by comparing the reach and impact of associated webinars and online courses. Our study methods and findings may serve as reference points for others in Extension interested in exploring the comparative effectiveness of various online tools.

Background/Program Description

In 2014 and 2015, the eXtension Horses CoP developed a series of six webinars that were broadcast live and recorded and then made available for subsequent public viewing. The webinars were developed by invited speakers, and each was formatted as a voiced-over presentation with a concluding voluntary survey. Recorded webinars became available to the public following each live broadcast (at eXtension.org/horses and myhorseuniversity.com).

The CoP also developed an online short course based on each webinar. Each short course consists of the recorded webinar (30–45 min) as well as related content organized in a detailed outline format with definitions, images, and linked resources, including video(s) and/or interactive activities; an online quiz; and a brief voluntary survey. Short course participants have the opportunity to earn a printable certificate upon successful completion of the course quiz.

USA Equestrian Trust (USAET) funded the project to facilitate delivery of educational materials covering topics of ongoing USAET research. Topics include pasture-associated laminitis, equine herpesvirus-1 outbreaks, disaster preparedness, sales fraud, rescues and rehoming options, and manure management. We used e-newsletters, social media, and the eXtension Horses and My Horse University websites to promote the live webinars, recorded webinars, and online short courses to the national horse community, including owners, volunteers, professionals, students, and horse enthusiasts. All materials remain available for online use.

Methods

Evaluating Reach and Impact

To measure project reach, we collected participation numbers for the live and recorded webinars and the short courses from March 25, 2014, through December 31, 2016. We also evaluated our reach with particular groups through demographic data we collected via the voluntary surveys included with each educational experience.

We hypothesized that active engagement in a short course would result in a more impactful educational experience as compared to live webinar participation. To evaluate and compare impacts, we examined responses from the voluntary online surveys for the live webinars and the short courses. At the end of each live webinar, participants were provided a link to the webinar survey (17 questions), and they received an email reminder within the subsequent 48 hr. The short course surveys (14 questions) were located at the end of the quiz for each course. We designed the survey items, including the demographic questions (which differ slightly between webinars and short courses), to obtain data for examining increases in topic knowledge, webinar or course usefulness for making horse management decisions, and likelihood of recommending the webinar or course to others. Additionally, both surveys contained questions regarding potential financial savings related to horse ownership expected due to knowledge gained from watching the webinar or completing the short course. We compared three questions common to the surveys for the live webinars and the short courses (referred to in this article as questions one, two, and three): 1. As a result of participating in this webinar/course, to what extent did your knowledge change on the topic? (response options ranged from 1, no extent, to 5, very great extent, and included a response of no basis to judge); 2. Overall, how useful was this webinar/course in helping you make informed horse management decisions? (response options were very useful/useful, somewhat useful, and hardly useful/not very useful); and 3. What is your potential to save money? (response options were $0–$100, $100–$1,000, and $1,000+) Additionally, the short course survey included a question on the importance of and/or incentive tied to earning a certificate upon completion of the educational materials.

Statistical Analysis

The survey response rates for participants attending the live webinar or completing the short course were 21.3% and 23.9%, respectively. We used analysis of variance to test the responses to all live webinar surveys and to test the responses to all short course surveys to determine whether responses from live webinar participants differed significantly across the webinar topics and whether responses from course participants differed significantly across the course topics. There were no significant differences among topics for either webinars or courses. This circumstance allowed us to group all the webinar responses and to group all the short course responses for comparison.

We used chi-square testing to compare live webinar responses (49 responses for question one and 48 for question two) and short course responses (48 responses for question one and 47 for question two). Due to lower numbers of responses to question three for both the live webinar (6 responses) and short course surveys (19 responses), we used a Fischer chi-square test to compare those responses.

Results and Discussion

Reach

As of December 31, 2016, there were 236 live webinar views and 4,605 recorded webinar views (Table 1). Of 490 participants who registered for the short courses, 201 participants completed the courses (Table 1). These data demonstrate the reach of the two delivery methods. Easy access and short-term commitment requirements may be factors contributing to the much larger reach of the webinars as compared to the short courses. The short courses contain the associated recorded webinar, but also involve readings and additional materials beyond the webinar content, thus requiring a greater time commitment for completion.

Table 1.
Webinar and Short Course Participation (Through December 31, 2016)

Webinar/short course title	Webinars			Short courses
	Date of live webinar	# live webinar views	# recorded webinar views	Date short course published	# short course participants	# completing short course	Avg. quiz score	# certificates awarded
Pasture-Associated Laminitis	3/25/14	30	1,136	11/4/14	100	39	91.03	33
EHV-1	4/29/14	53	1,554	11/4/14	147	65	92	55
Disasters	9/16/14	52	374	7/2/15	55	22	94	18
Sales Fraud	10/14/14	32	374	7/2/15	42	18	93.89	16
Second Chances for Horses	11/25/14	21	316	7/2/15	64	26	97.69	24
Manure Management	11/25/14	48	851	7/2/15	82	31	92.58	24
Total		236	4,605		490	201	93.53	170
Note. Equine herpesvirus-1 has been abbreviated to EHV-1.

Although we did not conduct a statistical comparison of participant demographics, on the basis of their selections of all options that applied, we discerned that the majority of webinar and short course participants identified themselves as horse owners (Figure 1). Participants also categorized themselves under a broad distribution of other choices, including industry members and educators (Figure 1). These data demonstrate that our efforts to reach the equine industry's largest group of stakeholders (horse owners) was successful for both webinars and short courses.

Figure 1.
Self-Identification of Webinar and Short Course Participant Types

Note: Participants were asked to check all that apply.

Impact

For the three questions on each survey that we used to assess impact, some respondents chose not to answer every question. The knowledge question (question 1) was answered by 49 live webinar participants and 48 short course participants. Both educational experiences resulted in perceived knowledge gain, with 82% of the live webinar participants and 75% of the short course participants reporting moderate to great changes in knowledge. There was no significant difference (p = .443) in perceived knowledge gained between the two groups, indicating that both platforms were equally effective in providing knowledge to participants. However, because both groups watched the webinar (the live version or recorded version used in the course), it makes sense that the knowledge gained would be similar.

Question two, about program usefulness, was answered by 48 live webinar participants and 47 short course participants. Of those respondents, 67% of webinar participants and 92% of short course participants rated the educational program as useful/very useful (p = .003) (Figure 2). The difference between the groups may be because the short courses were built from and expanded on the original webinar content and participants could take their time to reflect on the application of the new information to their current husbandry practices as they worked through the short courses.

Figure 2.
Perceived Usefulness: Webinars Versus Short Courses

The 19 short course participants who answered question three (Figure 3) reported an overall higher potential to save money (p = .042) when compared to six live webinar viewers who answered the question, with 42% of short course participants reporting that they expected to save more than $1,000, compared to 0 webinar viewers. As with the usefulness responses, participants expressed the expectation of experiencing more savings from the expanded short course content versus the webinar content. The prospect of long-term monetary savings can incentivize owners to adopt better management strategies and practices, suggesting that participation in the short course may be more likely to lead to changes in practices.

Figure 3.
Potential to Save Money: Webinars Versus Short Courses

Live webinars served as a stand-alone learning opportunity with no additional assessment incorporated and no certificate offered. For participants wishing to enhance their learning experience, short courses were available with additional content, including activities, videos, articles, and assessments. Due to the additional content and assessments offered in the short courses, certificates were awarded after successful completion of the short courses.

With regard to receiving a certificate for course completion, 69% of respondents to the short course survey indicated that receiving a certificate was moderately to very important. Certificates may serve to motivate participants to complete the learning materials and thus could be an asset for marketing courses and achieving higher completion rates. Other research has shown that online noncredit course participant completion rates increased from 3% to 20% when participants were taking the course with the specific goal of receiving a certificate (Chuang & Ho, 2016). The courses in our study had an overall relatively high completion rate of 41%, which may be related in part to the certificate incentive. Digital certificates can be created with little or no financial outlay (participants print their own certificates) and can incentivize follow-through to completion of the courses.

Limitations

Program impact results may be influenced by the survey response rates of 21.3% (live webinars) and 21.9% (short courses). Furthermore, a preprogram assessment of the knowledge level of participants and was not conducted. Similarly, the long-term use and retention of information was not assessed. According to work by Rossi, Lipsey, and Freeman (2004), there may have been a positive impact bias due to absence of a preprogram control or retention of knowledge survey. Finally, the results generated from a group of horse owners may not reflect possible outcomes from a more expansive Extension audience.

Implications

Webinar production took approximately 15 hr per webinar, and an additional 30 hr of time was required for production of the associated online course. When recorded views are included, webinars had a broader reach when compared to short course participation (4,841 versus 490) while still imparting similar levels of knowledge. However, the added investment in developing the short courses led to a greater reported impact regarding both usefulness for management decisions and expected potential money savings for horse owners and equine businesses. Data from our study suggest that recorded webinars can be used as the core content for developing more impactful short courses. Recorded webinars can be used as an introductory overview of content delivered in short courses that involve a variety of media, such as videos, interactive activities, and assessments. Offering a certificate for successfully completing a course may provide incentive for participants to complete the course. Additional work involving pre- and postprogram assessments of knowledge gained is needed to examine the effectiveness of webinars and short courses in different topic areas.

There are many benefits of creating and delivering online content to Extension clientele, including extending the reach to a broader audience (Pulec, Skelly, Brady, Greene, & Anderson, 2016). Online continuously accessible content allows participants to view webinars and participate in short courses at their own pace. Extension educators can extend their program reach by offering online educational programs such as webinars. Enhancing the information from webinars by creating an online course with additional learning tools, such as videos, interactive activities, and assessments, can further increase the impact on participant ability to use the knowledge gained to make important management decisions. Our study serves as a basis for recommendations for developing online educational tools for effective Extension programming.

Acknowledgments

We would like to acknowledge additional contributions to our project by Extension professionals including Gwyn Shelle, MS, and Karen Waite, PhD, of Michigan State University; Bridget McIntosh, PhD, of Virginia Tech; and the content design leader of eXtension, Ashley Griffin. Funding was provided by the USA Equestrian Trust.

References

Agnew, D. M. (1991). Extension program delivery trends. Journal of Extension, 29(2), Article 2RIB2. Available at: https://www.joe.org/joe/1991summer/rb2.php

Allred, S. B., & Smallidge, P. J. (2010). An educational evaluation of web-based forestry education. Journal of Extension, 48(6), Article 6FEA2. Available at: https://www.joe.org/joe/2010december/a2.php

Chuang, I., & Ho, A. (2016). HarvardX and MITx: Four years of open online courses—Fall 2012–summer 2016. Retrieved from SSRN: https://ssrn.com/abstract=2889436

Green, S. (2012). Extension online: Utilizing technology to enhance educational outreach. Journal of Extension, 50(5), Article 5TOT4. Available at: https://joe.org/joe/2012october/tt4.php

Greene, E. A., Griffin, A. S., Whittle, J. L., Williams, C. A., Howard, A. B., & Anderson, K. P. (2010). Development and usage of eXtension's HorseQuest: An online resource. Journal of Animal Science, 88(8). 2829–2837. https://doi.org/10.2527/jas.2010-2810

McCann, B. (2007). The effectiveness of Extension in-service training by distance: Perception versus reality. Journal of Extension, 45(1), Article 1FEA4. Available at: https://joe.org/joe/2007february/a4.php

Pulec, K. E., Skelly, C. D., Brady, C. M., Greene, E. A., & Anderson, K. P. (2016). Effectiveness of webinars as educational tools to address horse industry issues. Journal of Extension, 54(3), Article 3TOT8. Available at: https://www.joe.org/joe/2016june/tt8.php

Rich, S. R., Komar, S., Schilling, B., Tomas, S. R., Carleo, J., & Colucci S. J. (2011). Meeting Extension programming needs with technology: A case study of agritourism webinars. Journal of Extension, 49(6), Article 6FEA4. Available at: https://joe.org/joe/2011december/a4.php

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach. Thousand Oaks, CA: Sage Publications.

Taylor, G. M., & Anderson, D. M. (2008). Hard times ahead: Creating alternative revenue streams for Extension. Journal of Extension, 46(4), Article 4FEA3. Available at: https://www.joe.org/joe/2008august/a3.php