UserTesting.com: A Tool for Usability Testing of Online Resources

Introduction

Extension educators are increasingly using online resources in their program design and delivery. Online educational programming can be as effective as face-to-face instruction (Campbell, Koszewski, & Behrends, 2013), serves a large number of audiences at lower costs (Case, Cluskey, & Hino, 2011; Sutherin, Lombard, & St. Hilaire, 2015), and can be useful when personnel cuts have occurred (Francis, Martin, & Taylor, 2011). There is an increasing demand on Extension educators to be cost-effective in their educational programming (Jayaratne, 2015), and one way to meet this demand is through online programming.

However, online products must still meet learners' needs. Lambur (2013) asserted that developers of online projects should ensure that usability testing is performed to prevent a mismatch between what program designers intend and what users experience. Usability encompasses aspects such as learnability, memorability, errors, and satisfaction (Nielsen, 2012). In this article, we describe our experience with a specific usability testing tool that we implemented for our project Useful to Usable (U2U): Transforming Climate Variability and Change Information for Cereal Crop Producers (referred to hereafter as U2U).

Project Context

U2U is a research and Extension project that was funded by the U.S. Department of Agriculture and designed for the purpose of improving the resilience and profitability of U.S. farms in the Corn Belt in the face of a variable and changing climate. Our team works with members of the agricultural community to develop online decision support tools (DSTs) and educational materials that can lead to effective decision making and the adoption of climate-resilient practices. Using an iterative usability testing approach, we developed five DSTs: AgClimate View, Corn Growing Degree Days, Climate Patterns Viewer, Corn Split N, and Irrigation Investment Tool. We have promoted the DSTs (www.agclimate4u.org/tools) at outreach events and implemented a marketing campaign to supplement the in-person outreach.

Usability Testing

Several private companies now offer fee-based online usability testing services that connect their clients with video and audio feedback from representative users. For our project, we used the online usability testing service UserTesting (www.usertesting.com) to iteratively develop the U2U DSTs and associated marketing materials. We chose UserTesting because the pricing and scope of the available tests were most aligned with our needs.

UserTesting requires application of the following steps in planning and implementing its testing service:

1. Develop a mock or draft version of the product to be tested.
2. Draft and thoroughly review tasks and/or questions for testers.
3. Decide on the number of people who will complete the test.
4. Develop screener questions to ensure that testers will be representative of actual users. (Figure 1 shows screener questions for our project.)
5. Deploy the test.
6. Click URLs received via email from UserTesting to access feedback videos of testers performing the assigned tasks and talking aloud.
7. Watch and respond to each feedback video. (Figure 2 shows a still from a feedback video.) Enter your comments on the "Annotations" tab (saving comments at particular points in the video); save individual clips from the video by accessing the "Clips" tab; watch any particular task from the "Tasks" tab; and view answers to any open-ended questions on the "Answers" tab.
8. Rate each test to provide feedback to UserTesting. This step includes rating bad tests poorly so that UserTesting can contact you for additional information and then replace any bad test with a new one.

We negotiated a contract and bought 90 user credits for $4,500 that we could use over a 2-year period. One user test of up to 15 min equals one credit. UserTesting emailed feedback videos shortly after we deployed the test.

Figure 1.
Screener Demographic Questions Used for Testing Irrigation Investment Tool

Screener Question #1 Are you a corn/soybean grower OR do you advise farmers related to any aspect of corn/soybean production? Multiple Choice Options: Yes (Include users who select this answer) No None of the above Screener Question #2 In what state do you primarily work? Multiple Choice Options: Illinois (Include users who select this answer) Indiana (Include users who select this answer) Iowa (Include users who select this answer) Kansas (Include users who select this answer) Michigan (Include users who select this answer) Minnesota (Include users who select this answer) Missouri (Include users who select this answer) Nebraska (Include users who select this answer) North Dakota (Include users who select this answer) Ohio (Include users who select this answer) South Dakota (Include users who select this answer) Wisconsin (Include users who select this answer) None of the above

Figure 2.
Tester Feedback from Irrigation Investment Tool Development

Recommendations

On the basis of our experience, we are able to provide recommendations to others interested in performing user testing of online resources.

• Carefully select representative testers by developing appropriate screener questions. The more specific you are, the better feedback you will get. However, if you are too specific, testers may be unavailable. Also consider wording screener questions in a nonleading way because testers are paid and, therefore, might try to qualify for a test for which they are not suitable. For example, if you want testers who work in the agriculture field, instead of asking a question such as Screener Question #1 shown in Figure 1, you could ask "For what industry do you work?" and include agriculture as one of several response options. Then only those who select agriculture in response to the question would qualify as testers.
• Start by administering tests to four or five users. Nielsen (2000) suggested that best results come from running tests with no more than five users as saturation is typically reached by the fifth user.
• Have testers respond to no more than six or seven tasks/questions as there is not enough time for too many tasks/questions within the 15-min user-test window.
• Frame the tasks/questions to elicit descriptive answers. Do not ask yes-no or agree-disagree questions.
• If the online usability testing service you are using allows you to do so, be sure to rate each test. A testing service uses these ratings to improve its user pool, and if you have a poorly conducted test, the company likely will offer a free replacement test.

Implications

We learned about online usability testing services, including UserTesting, during an Extension Education Evaluation Topical Interest Group presentation at the 2011 American Evaluation Association conference. We adopted this usability testing method for the U2U project and have found it to be cost-effective, convenient, and effective in improving our online Extension programming. However, in presenting the approach at Extension-related events, we have found that most participants are unfamiliar with it. Therefore, it is our hope that this article will create greater awareness of online usability testing as an effective tool for Extensionists. Many of the lessons we learned from our experience with UserTesting should be transferable to usability testing conducted through other companies.

UserTesting works well for any online product. We used it for testing the U2U DSTs and marketing materials. On the basis of the testers' feedback, we implemented changes in the layout and design of the DSTs and clarified phrasing across the marketing materials. Our experience led us to conclude that usability testing has implications for Extension educators, administrators, and program evaluators who design and evaluate educational programming that fully or partially involves web-based or online resources. Usability testing saves resource development costs. It prevents unsatisfactory learning experiences that can result from irrelevant, out-of-context, or badly designed educational materials.

Usability testing has other benefits as well. Extension educators can enhance client learning by making educational resources more usable and, thereby, have a greater likelihood of achieving desired program outcomes. Evaluators can gather important data during formative evaluation and better explain outcomes and impact during summative evaluation (Paulsen, 2011).

Acknowledgments

The research reported here is part of Useful to Usable (U2U): Transforming Climate Variability and Change Information for Cereal Crop Producers and is supported by Agriculture and Food Research Initiative Competitive Grant no. 2011-68002-30220 from the U.S. Department of Agriculture National Institute of Food and Agriculture. Special thanks to Will Cushman of the University of Wisconsin Environmental Resources Center's Marketing and Communications Unit for editing the article.

References

Campbell, C., Koszewski, W. M., & Behrends, D. (2013). The effectiveness of distance education, using blended method of delivery for limited-resource audiences in the nutrition education program. Journal of Extension, 51(4), Article 4FEA4. Available at: http://www.joe.org/joe/2013august/a4.php

Case, P., Cluskey, M., & Hino, J. (2011). Online nutrition education: Enhancing opportunities for limited-resource learners. Journal of Extension, 49(6), Article 6RIB5. Available at: http://www.joe.org/joe/2011december/rb5.php

Francis, S. L., Martin, P., & Taylor, K. (2011). Revising an Extension education website for limited resource audiences using social marketing theory. Journal of Extension, 49(6), Article 6RIB5. Available at: https://www.joe.org/joe/2011december/a7.php

Jayaratne, K. S. U. (2015). Cost effectiveness ratio: Evaluation tool for comparing the effectiveness of similar Extension programs. Journal of Extension, 53(6), Article 6TOT3. Available at: http://www.joe.org/joe/2015december/tt3.php

Lambur, M. (2013). EEE week: Michael Lambur on affordable website online usability testing [Web log post]. Retrieved from http://aea365.org/blog/eee-week-michael-lambur-on-affordable-website-online-usability-testing/

Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved from http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Nielsen, J. (2012). Usability 101: Introduction to usability. Retrieved from http://www.nngroup.com/articles/usability-101-introduction-to-usability/

Paulsen, C. (2011, October 20). Christine Paulsen on usability testing [Web log post]. Retrieved from http://aea365.org/blog/christine-paulsen-on-usability-testing/

Sutherin, S., Lombard, K. A., & St. Hilaire, R. (2015). Website? Video? Facebook? Mobile app? One group's experience developing and comparing urban landscape water conservation digital outreach resources. Journal of Extension, 53(1), Article 1FEA1. Available at: http://www.joe.org/joe/2015february/a1.php