How to Setup an MTurk Toolkit Study

CloudResearch's MTurk Toolkit was specifically designed to simplify the process of setting up and launching studies on Amazon's Mechanical Turk. To setup your study, follow the steps below.

  1. Select Panel Options
  2. Fill out Basic Info
  3. Describe your HIT
  4. Setup HIT and Payment
  5. Choose How Workers are Approved
  6. Select Worker Requirements
  7. Decide whether to Use MicroBatch
  8. Select Pro Features
  9. Save, Review, and then Launch your Study

To begin creating your study, click Create a Study in the header and then choose MTurk Toolkit.

1. Select Panel Options

The first step in setting up your study is selecting the demographic features of the sample you intend to target. On the first page, you can:

  • Enter the number of participants you wish to sample
  • Choose from a variety of panel options.

Panel options include basic demographics such as age, ethnicity, and gender, as well as more sophisticated categories like the industry people are employed in, the nature of their political views, whether they own a pet, and much more.

TurkPrime MTurk Toolkit Study Setup: Select Panel Options

 

To understand the cost of your panel and whether we have enough people profiled to meet your needs, you can use the Feasibility Calculator. Once you begin selecting the demographic characteristics for your sample and enter the number of participants you'd like, the calculator will estimate the cost. Panel costs range from .15 to .75 cents per participant, depending on how difficult it is to reach the sample you select. Panel costs are in addition to the worker compensation, MTurk fees, and CloudResearch Pro Feature fees associated with your study (view pricing information).

 

CloudResearch MTurk Toolkit Study Setup: Feasibility Calculator

 

Whenever we believe recruiting your desired sample may be difficult, you will receive a message that your study may not reach the number of participants you desire. When you receive this message you can: a) proceed anyway, launching your study to see if you can collect enough participants, or b) contact our customer service to receive a quote for what it would cost to guarantee you reach your desired sample.

TurkPrime MTurk Toolkit Study Setup: Feasibility Calculator

2. Fill Out Basic Info

After selecting your Panel options, the next step is to fill out basic information for your study. On this page, you can give your project a name for internal use (i.e., not shown to workers) and provide up to five email addresses to associate with the study. We will send a message to the email addresses you list when your study launches and when it completes.

TurkPrime MTurk Toolkit Study Setup: Fill Out Basic Info

3. Describe your HIT

After giving basic information, the next step is to describe how you want your study to appear on MTurk. First, you can give your study a public title that will be visible to workers. Next, you can provide a description of the HIT, any custom instructions, and keywords you'd like associated with the HIT. The description and custom instructions boxes should be used to inform workers of anything they may be asked to do that is beyond the normal scope of MTurk studies (e.g., download an app, participate in a dyadic study). Finally, you can choose whether to warn workers about explicit or potentially offensive content within your study.

4. Setup HIT and Payment

On the Setup HIT and Payment tab, you can specify how much each worker should be paid and provide an estimate of how long the study should take. In addition, you can specify what the maximum amount of time workers should have to complete your study is with the "Time allotted per assignment" box. The maximum time per assignment influences not only how long workers have to work on your study, but also several important aspects of restarting, pausing, and editing your HIT.

Also on the Setup HIT and Payment tab is a place for you to enter your Survey Hyperlink (i.e., the link to your Qualtrics, SurveyGizmo, or other survey software survey).

TurkPrime MTurk Toolkit Study Setup: Setup Hit and Payment

 

Further options on the Setup HIT and Payment tab include scheduling a HIT launch time, setting up an automatic worker bonus, and deciding to whom and how your HIT gets displayed.

 

TurkPrime MTurk Toolkit Study Setup: Setup Hit and Payment

5. Choose How Workers are Approved

After setting up your HIT and payment, you must decide how workers who complete your study will be approved. There are three different completion code options—fixed, custom, or dynamic—and two approval methods: automatic or manual. If you use an automatic approval method, you must also decide how long to wait before workers are automatically approved.

6. Worker Requirements

After setting up your HIT and deciding how workers will be approved, there are a number of options for specifying which workers are eligible or ineligible for your study. For example, if you want to Exclude workers who took one of your previous studies or Include workers who took a Time 1 study so that they can participate in Time 2, both options are available on this page. On this page you can also specify worker qualifications such as approval rating and number of HITs completed. Finally, from this page you can choose whether to Block Workers from Suspicious Geolocations and whether to apply a Universal Exclude List.

7. MicroBatch

On the MicroBatch tab, you can choose whether you would like to break your study up into smaller batches that reduce MTurk's fee by 50% and help you spread out your data collection.

8. Pro Features

On the Pro Features page, you can choose from a number of options to enhance your study. For example, you can choose to Block Responses from Duplicate IP Addresses, Block Responses from Repeated Geolocations, whether to use HyperBatch or Anonymize Worker IDs, and whether to Verify Worker Country and State Location among others.

TurkPrime MTurk Toolkit Study Setup: Pro Features

 

TurkPrime MTurk Toolkit Study Setup: Pro Features

9. Save, Review, and then Launch your Study

The final step in the study setup sequence is to Save your study. Once the study is saved, it will appear on your Dashboard where you can Review and then Launch it. You should always review your HIT, preview your survey, and when possible pilot your study before launching the full version.