MTurk 101: An Introduction to Amazon Mechanical Turk for Extension Professionals

Internet-based surveys are a useful and low-cost method Extension professionals can use to recruit participants for research. Archer (2003) lists advantages and disadvantages of conducting surveys on the web and recommends the use of this environment for Extension surveys. Amazon Mechanical Turk (MTurk) (www.mturk.com) is an online marketplace for labor recruitment that has become a popular platform for collecting data online for social science research. Individuals or organizations looking for "workers" can register as "requesters" and post jobs or human intelligence tasks (HITs) for registered workers to complete. Common tasks include completing online surveys, categorizing photographs or other data, and providing feedback on websites or documents. Workers are paid upon successful task completion, with compensation ranging from pennies to several dollars. Research has shown MTurk to be cost effective for obtaining large, diverse samples for questionnaire research (Buhrmester, Kwang, & Gosling, 2011; Dworkin, Hessel, Rudi, & Gliske, 2015).

Studies have found that MTurk samples are more diverse in age, geographic location, and race/ethnicity than general Internet samples (Buhrmester et al., 2011; Goodman, Cryder, & Cheema, 2013). Amazon.com (2016) reports employing more than 500,000 MTurk workers in more than 190 countries. Tasks can be completed in 13 languages quickly and accurately (Pavlick, Post, Irvine, Kachaev, & Callison-Burch, 2014). Amazon typically pays international workers with gift cards but in 2009 began offering workers in India the option of being paid in Indian rupees, greatly increasing the percentage of Indian workers (Ipeirotis, 2010).

Data Quality

Some researchers are concerned about the quality of data from MTurk workers. However, a number of researchers have replicated studies to compare online and offline respondents and found that data quality was not significantly different between MTurk and offline and laboratory experiments (Horton, Rand, & Zeckhauser, 2011; Paolacci, Chandler, & Ipeirotis, 2010; Suri & Watts, 2011). MTurk workers provided high-quality data that met or exceeded psychometric standards associated with research conducted in face-to-face settings (Buhrmester et al., 2011).

Uses of MTurk by Extension Professionals

Possible applications of MTurk within Extension are wide ranging. Specifically, MTurk can be used for research in the following ways:

• Researchers can collect data from national and international samples about individual behaviors and perceptions. A HIT can link workers to a website hosting an online survey (e.g., SurveyMonkey). MTurk also provides its own online survey tool.
• The platform allows researchers to collect unique data in ways that previously could be done only in face-to-face settings, such as by simulating group interactions or synchronous tasks.
• Researchers can create virtual "waiting rooms" to hold workers while others accept a HIT and then proceed with the experiment by having all workers act simultaneously (Mason & Suri, 2012).
• Researchers can conduct in-depth qualitative interviews with geographically diverse participants, homebound participants, participants without college degrees, and minority groups (Williamson, 2014).
• Participants can be recruited online for in-person data collection.

MTurk workers can provide a variety of services, including

• writing reviews, descriptions, and blog entries for websites;
• editing, transcribing, or rating the accuracy of search engine results;
• giving feedback about videos and photos, advertisements, recruitment flyers, and program descriptions;
• providing feedback about whether people or materials in photos are relatable and culturally appropriate; and
• tagging photos or videos with keywords that can be used by practitioners to recruit a particular audience.

MTurk has many benefits over other methods of data collection:

• It provides access to a worldwide workforce of people who can perform tasks in multiple languages.
• The platform has workers who are available on-demand 24 hours a day, 7 days a week. The cost of developing an online recruitment space and recruiting participants is much lower than hiring staff or working with a consulting firm.
• It provides templates for designing tasks. Using the MTurk HTML editor, requesters can incorporate images, figures, and videos into tasks.
• It provides access to a more diverse sample than may be available in person or through other online recruitment strategies.
• It can be used to obtain feedback from a particular group (e.g., parents of young children) about a website's format, design, or content. For example, workers could complete a usability task by exploring a website and answering a short survey that provides feedback about their experience.

Suggestions and Recommendations

Below are some guidelines to consider when using MTurk. See Table 1 for suggested resources and the appendix for step-by-step guidance for using MTurk.

• Prior to setting up a task, consider the type of participant you want to recruit (e.g., single parents, non-college-going youth).
• Become an MTurk worker yourself to become familiar with the worker experience, including with how to get paid and how to peruse the various HITs that are available.
• Test a HIT in the MTurk Sandbox before publishing it to workers; it is a good idea to ensure that surveys work well with a target population and are thoroughly pilot tested before recruiting begins (Monroe & Adams, 2012). The MTurk Sandbox simulates the MTurk environment without any payment involved; it is solely for testing HITs. You can enter your task into the sandbox (just as you would in the actual MTurk website) and choose parameters for how many workers can pilot test the survey to check for errors and identify difficulties with the survey experience (Johnson & Borden, 2012).
• Take advantage of MTurk's tools for sample selection by specifying worker requirements and inclusion and exclusion criteria for who can see HITs. For example, you can choose to show HITs to only experienced MTurk workers (those who have completed a certain number of HITs successfully), MTurk workers with high reputations (those with high completion ratings), or MTurk workers in particular countries. You can also exclude workers who have already completed the HIT, thereby avoiding duplicate responses.
• Incorporate attention checks into your survey. Attention checks are questions that are not related to your study but allow you to determine whether respondents are paying attention (e.g., "Select 'Strongly Disagree' for this question").

MTurk is useful to Extension professionals who want to learn more about the communities they work with. By providing access to individuals from diverse backgrounds who are able to provide quality data, MTurk supports both research (e.g., data collection) and outreach (e.g., receiving feedback on documents).

References

Amazon.com. (2016). Service highlights. Retrieved from https://requester.mturk.com/tour

Archer, T. M. (2003). Web-based surveys. Journal of Extension, 41(2) Article 4TOT6. Available at: http://www.joe.org/joe/2003august/tt6.php

Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.

Dworkin, J., Hessel, H., Rudi, J., & Gliske, K. (2015). A comparison of three online recruitment strategies for engaging parents. Manuscript submitted for publication.

Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.

Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425.

Ipeirotis, P. G. (2010, March). Demographics of Mechanical Turk. NYU Working Paper No. CEDER-10-01. Retrieved from http://ssrn.com/abstract=1585030

Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon's Mechanical Turk to increase study-faculty collaborative research. Teaching of Psychology, 39(4), 245–251.

Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44(1), 1–23.

Monroe, M. C., & Adams, D. C. (2012). Increasing response rates to web-based surveys. Journal of Extension, 50(6) Article 6TOT7. Available at: http://www.joe.org/joe/2012december/tt7.php

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.

Pavlick, E., Post, M., Irvine, A., Kachaev, D., & Callison-Burch, C. (2014). The language demographics of Amazon Mechanical Turk. Transactions of the Association for Computational Linguistics, 2, 79–92.

Suri, S., & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS One, 6(3), e16836.

Williamson, V. (2014, August 10). On the ethics of crowd-sourced research. Retrieved from http://scholar.harvard.edu/files/williamson/files/mturk_ps_081014.pdf

Appendix

Guidance for Using MTurk