CloudResearch's MTurk Toolkit was specifically designed to simplify the process of setting up and launching studies on Amazon's Mechanical Turk. To setup your study, follow the steps below.
- Select Panel Options
- Fill out Basic Info
- Describe your HIT
- Setup HIT and Payment
- Choose How Workers are Approved
- Select Worker Requirements
- Data Quality Solutions
- Select Pro Features
- Save, Review, and then Launch your Study
To begin creating your study, click Create a Study in the header and then choose MTurk Toolkit.
1. Select Panel Options
The first step in setting up your study is selecting the demographic features of the sample you intend to target. On the first page, you can:
- Enter the number of participants you wish to sample
- Choose from a variety of panel options.
Panel options include basic demographics such as age, ethnicity, and gender, as well as more sophisticated categories like the industry people are employed in, the nature of their political views, whether they own a pet, and much more.
To understand the cost of your panel and whether we have enough people profiled to meet your needs, you can use the Feasibility Calculator. Once you begin selecting the demographic characteristics for your sample and enter the number of participants you'd like, the calculator will estimate the cost. Panel costs range from .15 to .75 cents per participant, depending on how difficult it is to reach the sample you select. Panel costs are in addition to the worker compensation, MTurk fees, and CloudResearch Pro Feature fees associated with your study (view pricing information).
Whenever we believe recruiting your desired sample may be difficult, you will receive a message that your study may not reach the number of participants you desire. When you receive this message you can: a) proceed anyway, launching your study to see if you can collect enough participants, or b) contact our customer service to receive a quote for what it would cost to guarantee you reach your desired sample.
2. Fill Out Basic Info
After selecting your Panel options, the next step is to fill out basic information for your study. On this page, you can give your project a name for internal use (i.e., not shown to workers) and provide up to five email addresses to associate with the study. We will send a message to the email addresses you list when your study launches and when it completes.
3. Describe your HIT
After giving basic information, the next step is to describe how you want your study to appear on MTurk. First, you can give your study a public title that will be visible to workers. Next, you can provide a description of the HIT, any custom instructions, and keywords you'd like associated with the HIT. The description and custom instructions boxes should be used to inform workers of anything they may be asked to do that is beyond the normal scope of MTurk studies (e.g., download an app, participate in a dyadic study). Finally, you can choose whether to warn workers about explicit or potentially offensive content within your study.
4. Setup HIT and Payment
On the Setup HIT and Payment tab, you can specify how much each worker should be paid and provide an estimate of how long the study should take. In addition, you can specify what the maximum amount of time workers should have to complete your study is with the "Time allotted per assignment" box. The maximum time per assignment influences not only how long workers have to work on your study, but also several important aspects of restarting, pausing, and editing your HIT.
Also on the Setup HIT and Payment tab is a place for you to enter your Survey Hyperlink (i.e., the link to your Qualtrics, Alchemer, QuestionPro, or other survey software survey).
Further options on the Setup HIT and Payment tab include scheduling a HIT launch time, setting up an automatic worker bonus, and deciding to whom and how your HIT gets displayed.
5. Choose How Workers are Approved
After setting up your HIT and payment, you must decide how workers who complete your study will be approved. There are three different completion code options—fixed, custom, or dynamic—and two approval methods: automatic or manual. If you use an automatic approval method, you must also decide how long to wait before workers are automatically approved.
6. Worker Requirements
After setting up your HIT and deciding how workers will be approved, there are a number of options for specifying which workers are eligible or ineligible for your study. For example, if you want to Exclude workers who took one of your previous studies or Include workers who took a Time 1 study so that they can participate in Time 2, both options are available on this page. On this page you can also specify worker qualifications such as approval rating and number of HITs completed. Finally, from this page you can choose whether to Block Workers from Suspicious Geolocations and whether to apply a Universal Exclude List.
7. Data Quality Solutions
To manage data quality, you can choose which slice of the MTurk population your study is open to. Opening your study to All MTurk Participants means everyone on MTurk may potentially take your study, including people who are inattentive, or are outside of the US and using VPN to fraudulently make it appear that they are within the US.
If you choose to Block Low Quality Participants, people who have failed CloudResearch's ongoing screening will be unable to access your study. This setting is the default option.
Finally, if you choose the CloudResearch Approved Participants option, your study will only be open to people who have been vetted by CloudResearch. Participants on this list have shown prior evidence of attention and engagement and represent various levels of experience on MTurk, different ages, races, incomes, and are generally demographically similar to other MTurk workers. The Approved List is our recommended option for the highest level of data quality. However, the Approved List may not be appropriate for studies with multiple demographic requirements or that target sample sizes in the thousands of participants.
See our blog for more details about our screening process and each of these options.
8. Pro Features
On the Pro Features page, you can choose from a number of options to enhance your study. For example, you can choose to MicroBatch or HyperBatch your study to save 50% of MTurk fees. You can also choose to Anonymize Worker IDs, target participants in various US regions, or to employ a Survey Group.
9. Save, Review, and then Launch your Study
The final step in the study setup sequence is to Save your study. Once the study is saved, it will appear on your Dashboard where you can Review and then Launch it. You should always review your HIT, preview your survey, and when possible pilot your study before launching the full version.