You can easily select various worker requirements to determine who is eligible and ineligible for your study.
Determining Eligibility with Include and Exclude Lists
You can determine which participants are eligible or ineligible for your study using Include and Exclude lists. An Include list specifies a group of participants to be Included in your study, while an Exclude list specifies a group of participants to be Excluded.
You might use an Include list when running a longitudinal, tracking, or follow-up study and you might want to use an exclude list to make sure people who took the first study in a series of projects do not also take a subsequent study. You can set include and exclude criteria at the survey level or at the individual level with Worker IDs.
Including Participants at the Survey Level
To Include participants from a previously completed study, enter the Worker Requirements page, select the Include checkbox, and then select which studies you would like to include and choose whether workers must have completed All or Any of the studies selected.
Excluding Participants at the Survey Level
To Exclude participants who completed a previous study, select the Exclude checkbox on the Worker Requirements page. Then, simply select the study or studies you want to exclude using the dropdown box.
Including Workers by Worker IDs
Workers can also be Included for a study using their Worker IDs. All you need to do is paste a list of MTurk Worker ID’s (or CloudResearch ID’s) into the Include workers box. No more than 2,000 workers can be Included using this method.
Excluding Workers by Worker ID
Just as you can Include workers by Worker ID, you can Exclude workers using their Worker IDs. Paste each Worker ID on a separate line and they will be excluded from your study. Only 2,000 workers can be excluded via this method.
TIP! In order for our Include and Exclude features to work, workers must have submitted and been approved for your previous study or studies.
Determining Eligibility with Worker Qualifications
In addition to Including and Excluding workers, you can set worker qualifications to ensure that only workers with a certain reputation can take your study. This is also done from the Worker Requirements page.
Master Workers
Perhaps the most basic way to divide workers based on reputation is between those who have MTurk's Masters qualification and those who do not. MTurk does not explicitly state how workers receive the Masters qualification except to say that Master "Workers have consistently demonstrated a high degree of success in performing a wide range of HITs across a large number of Requesters. We leverage statistical models that analyze all Workers based on several Requester-provided and marketplace data points to make that determination."
For basic survey studies, there is no evidence to suggest Masters workers are worth the extra 5% fee MTurk charges to recruit them. In addition, most workers are not Masters, so selecting the Master qualification will slow your data collection.
MTurk Reputation Metrics
Mechanical Turk uses two pieces of information to quantify a worker’s reputation: 1) the worker’s approval rating, and 2) the number of HITs a worker has had approved.
1. Worker Approval Rating
Approval rating refers to how often a workers' work is approved by requesters. Approval rating is equal to the number of times a workers’ submission was accepted by a requester divided by the total number of HITs a worker has submitted. Approval ratings are expressed as a percentage (e.g., 95% approval rating).
NOTE: MTurk assigns all workers a 100% approval rating until the worker completes at least 100 HITs. After a worker completes 100 HITs, their approval rating is based on their performance in past HITs.
2. Number of HITs Approved
The number of HITs approved refers to the number of HITs the worker has had approved by requesters in the past. When setting up a study, you can specify that workers must have a completed a certain number of past HITs, or in other words, have a certain level of experience. The number of past HITs can range from 0 to 1,000,000.