Evaluating the Impact of Cooperative Extension Outreach via Twitter

Introduction

Twitter is increasingly being used by Extension educators as a teaching and program-marketing tool (Cornelisse et al., 2011; O'Neill, Zumwalt, & Bechman, 2011; Santiago, Franz, Christoffel, Cooper, & Schmitt, 2013). This is not surprising given its vast outreach capabilities. In January 2014, Twitter had 645,750,000 active registered users (Twitter Statistics, 2014). While constrained to short messages (tweets) of 140 characters or less, users can expand their content with links to blogs, Web sites, videos, online surveys, and other social networking sites (e.g., Facebook). Further, a Twitter user's messages can be re-tweeted by his or her followers, further extending initial outreach. In a manner of minutes, it is possible to reach hundreds, even thousands, of people with research-based information.

The key to being followed and/or re-tweeted is creating messages that provide value to users and/or create a sense of "community" through shared discussion of topics of interest. Like any program outreach method, Twitter use should be evaluated for evidence of impact upon message recipients. This article describes strategies for Extension educators to evaluate the impact of outreach efforts via Twitter. Although examples of personal finance program evaluation methods are provided, these strategies can be adapted for any subject matter topic or Extension program area.

Social Media Campaign Evaluation Metrics

For the past 4 years, the eXtension Financial Security for All Community of Practice has conducted a social media campaign to promote saving for future financial goals and enrollment in the America Saves social marketing program. As described in O'Neill, Zumwalt, Gutter, and Bechman (2011), "cut and paste" tweets were sent to professionals who participated in the project to distribute content to their followers. The program used a triangulated (multiple methods) evaluation methodology with the following methods used to measure impact: (1) a unique Twitter hashtag (#eXasw) embedded into tweets, (2) a follow-up online consumer (follower) survey, (3) a follow-up online professional participant survey, (4) bit.ly analytics to determine the number of clicks on links embedded in tweets, (5) pre- and post-program Twitter influence metrics (Klout scores) of professionals who distributed financial education messages, and (6) TweetReach impact evaluation metrics.

Each year, impact data is compared to previous years as a measure of program outreach. In 2014, 1,091 tweets were recorded with hashtag #eXasw. This compares to 1,503 tweets in 2013, 1,437 in 2012, and 1,190 in 2011. A total of 1,277 clicks from tweets (and Facebook) messages were recorded (versus 2,814, 1,550, and 602, respectively, in 2013/2012/2011).

An evaluation survey embedded in tweets during the final days of the project assessed participant behavior change. In 2014, 67% of respondents visited one or more America Saves or eXtension Web site links that were contained in the messages and 17% said they planned to. With regard to joining America Saves, 22% of respondents reported that they did, 33% said that they planned to, 17% were already American Savers, and 28% said they had not joined. Positive open-ended comments were also received about the usefulness of the information provided.

Of 113 project participants in 2014, 13 (11.5%) completed a follow-up evaluation survey. The number of social media followers/friends reported by these collaborators ranged from 35 to 885 and totaled 3,056 versus 4,045 in 2013, 5,966 in 2012, and 8,163 in 2011). When the outreach numbers reported by all of the respondents were combined, there was an average of 254 message recipients apiece (versus 192 in 2013, 199 in 2012, and 226 in 2011). As another way to measure the outreach of the social media project, in addition to participant self-reports, a Tweet Reach report (see http://tweetreach.com) for #eXasw was pulled at the end of the month-long project. This report indicated estimated outreach of 3,751 tweets.

The Klout scores of 13 respondents who reported them in 2014 increased from before to after the project, and respondents' total average Klout scores increased from 29.76 to 38.84 (versus 32.6 to 39.6 in 2013, 20.3 to 29.3 in 2012, and 11.22 to 19.68 in 2011). This progression indicates project participants' growing influence on Twitter as those who participated in the 2011 Twitter savings campaign continued to tweet year-round. For information about Klout scores, see http://klout.com/.

Twitter Chat Evaluation

Four, hour-long Twitter chats were held during February 2014 with a unifying hash tag of #eXASchat. Each chat consisted of five questions that participants responded to. Tweets were also prepared to respond to question topics in between participant tweets. Weekly prizes (gift cards) were awarded for the best posts. The Web sites http://tweetchat.com/ and http://www.tchat.io/ were used to manage the flow of incoming tweets. See Table 1 for post-chat tweet metrics.

Estimated reach is the estimated overall reach of tweets in a chat and exposure impressions is the total number of times tweets are delivered to timelines (including repeats). A TweetReach report also indicates the number of impressions generated by individual contributors. Those with a large following and frequent tweets boost the outreach of a Twitter chat. Twitter accounts that have 1,000+ followers are in the 96th percentile of active Twitter users (Zipkin, 2013).

In the above example, almost a quarter million impressions were reported during the month-long project, about half by contributors with more than 1,000 followers. Thus, how many people participate in Twitter chats and who participates are key factors. TweetReach also provides a chat transcript so content can be easily shared (as a PDF file), summarized, or "recycled" (e.g., newspaper articles).

Summary

It is not enough for Extension educators to simply use Twitter as a program outreach method. Steps must be taken to evaluate program impact with quantitative and qualitative data. This article described the following Twitter evaluation metrics: unique hashtags, clicks on unique links, online surveys of professionals and their followers, pre- and post-project Klout scores, and TweetReach reports. A triangulated evaluation with several metrics can provide substantial evidence of program effectiveness.

Acknowledgments

Appreciation is expressed to Andrew Zumwalt (University of Missouri) for assistance with data collection for the Twitter hashtag #eXasw and clicks on links embedded in the America Saves savings tweets. In addition, Susan Shockey (NIFA-USDA), Erik Anderson (University of Idaho), and Elizabeth Kiss (Kansas State University) provided assistance with the management and delivery of the America Saves Twitter chats.

References

Cornelisse, S., Hyde, J., Raines, C, Kelley, K., Ollendyke, D., & Remcheck, J. (2011). Entrepreneurial Extension conducted via social media. Journal of Extension [On-line], 49(6). Article 6TOT1. Available at: http://www.joe.org/joe/2011december/tt1.php

O'Neill, B., Zumwalt, A., & Bechman, J. (2011). Social media use of Cooperative Extension family economics educators: Online survey results and implications. Journal of Extension [On-Line], 49(6), Article 6RIB2. Available at: http://www.joe.org/joe/2011december/rb2.php

O'Neill, B., Zumwalt, A., Gutter, M., & Bechman, J. (2011). Financial education through social media: Can you evaluate its impact? The Forum for Family and Consumer Issues. Retrieved from: http://ncsu.edu/ffci/publications/2011/v16-n1-2011-spring/oneil-zumwalt-gutter-bechman.php

Santiago, A., Franz, N., Christoffel, R., Cooper, K., & Schmitt, B. (2013). The family-environment connection: Filling a nationwide program gap. Journal of Extension [On-Line], 51(5), Article 5IAW3. Available at: http://www.joe.org/joe/2013october/iw3.php

Twitter Statistics. (2014). Statistic Brain. Retrieved from: http://www.statisticbrain.com/twitter-statistics/

Zipkin, N. (2013, December 19). Have 1,000 followers? You're in the 96^th percentile of Twitter users. Entrepreneur. Retrieved from: http://www.entrepreneur.com/article/230487