Enhancing Data Quality

The data quality tab allows you to decide which slice of the MTurk population your study is open to and whether to apply features such as Blocking Duplicate IP Addresses.

Data Quality Solutions

To manage data quality, you can decide which slice of the MTurk population you want to open your study to.

  1. Block Low Quality Participants (default)
    When you use this feature, people who fail simple attention checks, provide “bot-like” responses, or who have otherwise shown evidence that they are disengaged or unable to follow study instructions will be excluded from the pool of people able to take your study.

    The list of people who are excluded from your study when you use this feature is based on CloudResearch's ongoing screening of the MTurk population. People who we identify as providing low quality responses are added to the Block List. However, because new accounts are created all the time and some people may never complete our screening measures, this feature cannot guarantee that no one in your sample will provide poor data. 
  2. CloudResearch Approved Participants (recommended)

    If you want to be sure everyone who participates in your study has shown prior evidence of attention and engagement, you can use our CloudResearch Approved Participants feature. This feature allows only people who have passed CloudResearch's data quality screening to participate in your study.

    Because the Approved Participants list operates by including a large list of specific workers, it may not be ideal for studies that target participants based on multiple demographic criteria or that seek to gather thousands of participants. In these instances, you may be better off using the Block Low Quality Participants option.
  3. All MTurk Participants
    When you choose to leave your study open to all MTurk participants, anyone who meets the other eligibility requirements of your study will be able to access your HIT.


Why Are These Options Necessary?

Mechanical Turk is an international platform. People from all over the world can sign up to complete HITs and earn money. Unfortunately, over the last several years there has been an increasing number of people outside of the US who use VPN and other tools to hide the location of their web traffic and complete HITs and studies meant for people within the US. This creates an obvious problem because people outside the US are often not the population researchers seek to sample and there are often language comprehension issues that lead to low data quality.

To deal with this issue, CloudResearch has decided to screen all participants on Mechanical Turk on an ongoing basis. People who pass through our screening measures and provide consistently good data are placed onto the Approved Workers List. People who fail the measures are placed on a Block List. The open list leaves your study open to everyone on MTurk. Learn more about our screening procedure in this blog and learn more about data quality best practices in this blog.

Advanced Data Quality Options

In addition to deciding which slice of the MTurk population your study is open to, there are several other selections you can make to protect data quality.

  • Apply My Universal Exclude List
    If you gather bad data from a worker who you would prefer to never have in one of your studies again, you can place them onto your own Universal Exclude List. Checking this box applies your Universal Exclude List, making sure workers you have added overtime are unable to take your study.
  • Block Duplicate IP Addresses

    Workers on MTurk may share the same IP address for a variety of reasons (e.g., they are in the same public space, there are multiple people in the same house who both use MTurk, etc). Most researchers do not want to run the risk of collecting responses from multiple people who might share information about a study or one person who somehow submits multiple responses to the same study. For this reason, we created the Block Duplicate IP Addresses feature. When this feature is on, workers from the same IP address will be unable to accept your HIT.

    NOTE: Blocking Duplicate IP Addresses prevents multiple workers from accepting or submitting the HIT from the same IP address. You may still see multiple responses from the same IP address in your data file if the worker who accepted the HIT enters your survey more than once.

    Workers can enter a survey more than once even if you are using Qualtrics's "Prevent Ballot Box Stuffing" feature or something similar. The reason why is that the Ballot Box feature works by placing a cookie into the worker's browser. If the worker blocks cookies, uses an Incognito browser, or switches from one browser to another (from Chrome to Firefox), they can reenter your survey simply by copying the survey link into a new browser window.

  • Block Suspicious Geocode Locations
    The Block Suspicious Geocode Locations tool prevents workers from taking your HIT if their web traffic originates at one of a large variety of VPN or server farm locations we have identified as suspicious. The locations we on our suspicious list have been associated with low quality data in the past.
  • Verify Worker Country and State Location
    When you use the "Verify Worker Country and State Location" feature, we will prevent any worker whose IP address does not match the country and location settings you setup for your study. This feature prevents people from taking your study who are not part of your intended sample.
    NOTE: You may find people in your study sample who do not match your targeting criteria if workers use a VPN or similar tool to hide the location of their web traffic.