Mechanical Turk allows requesters to reject submissions that do not meet standards for quality. To reject a submission, follow the steps below.
Rejecting a Worker's Submission
If you chose to manually approve workers, each worker will enter a "pending" state after submitting their HIT. If you chose automatic approval, workers will only enter a pending state if they submit the incorrect approval code. Thus, if you want the option to review each worker before they are approved you should setup manual approval.
On the lower banner of your study on the dashboard, you will see a “Manage Pending” button. When you click on this button, you will be taken to a page where you can either approve or reject each worker who completed your HIT. Once approved, a worker cannot be rejected. Hence, we recommend checking the data before approving or rejecting workers.
To reject a worker, select the "Reject" option. If you want, you can enter a message to the worker about why you are rejecting their work. Then, click "Save" at the bottom and your rejection(s) will be issued. If you later decide to reverse a rejection you have issued, you can do so easily within the CloudResearch system. Rejection reversals must be done within 30 days of issuing the rejection.
NOTE: You will not be charged by Mechanical Turk for participants you reject. However, the worker's submission will still be counted toward the overall progress of your study. Rejected workers are not automatically replaced. If you need to increase your sample size after rejecting some workers, there are two options. If your study is still Live, you can choose to edit the study and increase your sample size. If your study is Completed, you copy your study and open a new HIT.
TIP! Workers sometimes submit their Worker ID instead of the approval code they were given. Anytime this occurs you should check the worker's data before issuing a rejection.
What Rejection Means for the Worker
When workers receive a rejection, they are not paid for the time spent on the HIT and their reputation suffers. Mechanical Turk keeps track of each worker's approval rating. Rejections hurt the approval rating and can jeopardize the worker's potential to qualify for future studies. For this reason, rejections should not be given indiscriminately or for trivial reasons.
What Rejection May Mean for the Requester
Although requesters have full discretion to approve or reject worker submissions, there are some potential negative consequences for requesters who reject lots of work. In particular, there are several websites workers use to share information about requesters (e.g., TurkOpticon). Rejecting workers without providing a reason or rejecting too many workers may harm a requester's reputation on review sites.