The data quality tab allows you to decide which slice of the MTurk population your study is open to and whether to apply features such as Blocking Duplicate IP Addresses.
Data Quality Solutions
To manage data quality, you can decide which slice of the MTurk population you want to open your study to.
- CloudResearch Approved Participants (recommended)
If you want to be sure everyone who participates in your study has shown prior evidence of attention and engagement, you can use our CloudResearch-Approved Participants feature. This feature allows people who have passed CloudResearch's data quality screening to participate in your study. We created this feature to combat data quality issues that emerged on MTurk in 2018.
When you use the Approved Participants feature, people who fail simple attention checks, provide “bot-like” responses, or who have otherwise shown evidence that are unable or unwilling to follow study instructions will be excluded from the pool of people able to take your study.
The Approved group consists of more than 50,000 people in the U.S. whose demographics mirror that of the MTurk population. - All MTurk Participants
When you open your study to all MTurk participants, anyone can participate in your study. This includes participants who may be Using this option
Why Are These Options Necessary?
Mechanical Turk is an international platform. People from all over the world can sign up to complete HITs and earn money. Unfortunately, over the last several years there has been an increasing number of people outside of the US who use VPN and other tools to hide the location of their web traffic and complete HITs meant for people within the US. This creates an obvious problem because people outside the US are often not the population researchers seek to sample. There are also often language comprehension issues that lead to low data quality.
To deal with this issue, CloudResearch screens all participants on Mechanical Turk on an ongoing basis. People who pass through our screening measures and provide consistently good data are placed onto the CloudResearch-Approved group. People who fail the measures are blocked from CloudResearch studies. Choosing to leave your study open to everyone on MTurk means it is also open to people we have screened and blocked due to data quality. Learn more about our screening procedure or data quality best practices.
Advanced Data Quality Options
In addition to using CloudResearch-Approved participants, there are several other things you can do to protect data quality.
- Use the “My Universal Exclude List” Feature
If you gather data from a participant you would prefer to never have in one of your studies again, you can place them onto your Universal Exclude List. The Universal Exclude List is a personal list of participants that are excluded from all of your studies. Checking the box to turn on your Universal Exclude List makes participants you have added over time ineligible for your study. - Block Duplicate IP Addresses
Participants on MTurk may share the same IP address for a variety of reasons (e.g., they are in the same public space, there are multiple people in the same house who both use MTurk, etc). Most researchers do not want to run the risk of collecting responses from multiple people who might share information about a study or from one person who somehow submits multiple responses to the same study. For this reason, we created the Block Duplicate IP Addresses feature. When this feature is on, participants from the same IP address will be unable to accept your HIT.
NOTE: Blocking Duplicate IP Addresses prevents multiple workers from accepting or submitting the HIT from the same IP address. You may still see multiple responses from the same IP address in your data file if the participant who accepted the HIT enters your survey more than once.
Participants can enter a survey more than once even if you are using Qualtrics's "Prevent Ballot Box Stuffing" feature or something similar. The reason why is that the Ballot Box feature works by placing a cookie into the participant's browser. If the participant blocks cookies, uses an Incognito browser, or switches from one browser to another (from Chrome to Firefox), they can re-enter your survey simply by copying the survey link into a new browser window. - Block Suspicious Geocode Locations
The Block Suspicious Geocode Locations tool prevents participants from taking your HIT if their web traffic originates at one of several VPN or server farm locations we have identified as suspicious. The locations on our suspicious list have been associated with low quality data in the past. - Verify Worker Country and State Location
When you use the feature to "Verify Worker Country and State Location", we will prevent any participant whose IP address does not match the country and location settings you designate for your study. This feature prevents people from taking your study who are not part of your intended sample.
NOTE: You may find people in your study sample who do not match your targeting criteria if participants use a VPN or similar tool to hide the location of their web traffic.
