On Mechanical Turk, workers submit approval codes to demonstrate they have completed a study. CloudResearch allows you to choose between automatically or manually approving workers once they have submitted a code.
Approval Codes on Mechanical Turk
When your study is housed on an external platform like Qualtrics or QuestionPro, Mechanical Turk cannot verify that workers actually completed your study. To deal with this issue, researchers commonly place a "secret code" at the end of their study (this code is a secret in that the researcher knows it and the worker does not). Once workers reach the end of a study, they must copy the secret code from the survey into the HIT submission box on Mechanical Turk to demonstrate to the researcher that they went all the way through the study. CloudResearch offers three different forms of secret codes: fixed, custom, and dynamic.
1. Fixed secret codes
Fixed completion codes are static, alphanumeric codes that are the same for each worker who takes your study. You can generate a fixed completion code however you want, but the code cannot contain the numbers 1 and 0 or the letters I or O.
The fixed code you choose should be prominently displayed on the final page of your survey with instructions for workers to submit the code in order to have their HIT approved. Fixed completion codes are compatible with both automatic and manual approval.
TIP! A drawback of fixed completion codes is that they are the same for each worker, making it possible for workers to share codes. Sharing would allow a worker to submit your HIT with the correct approval code without actually taking your survey. While this behavior does not appear particularly widespread, we recommend custom or dynamic codes to remove the potential for fraud.
2. Custom secret codes
Custom completion codes eliminate concern about workers sharing codes by presenting each worker with a unique or custom code. If your survey platform has the capability, you can generate a random completion code. Alternatively, you could use JavaScript to embed a random code at the end of your study. Custom completion codes are only compatible with manual approval because the code is generated inside your survey and CloudResearch does not have access to the data inside your survey (i.e., we don’t know the code each worker was supposed to submit).
TIP! Because custom codes are provided within the survey and not on a page after the survey has ended, workers sometimes forget to actually submit your survey. In such cases, your survey platform may mark the responses as incomplete or "in progress." To make sure you download all your data, we recommend closing any “Responses in Progress” before downloading your data. Doing so will ensure you have all the responses associated with your study.
3. Dynamic secret codes
Dynamic completion codes are similar to custom codes except that dynamic codes are integrated with CloudResearch, which makes automatic approval possible. When you setup a dynamic approval code, each worker will be given a unique, randomly generated code. Dynamic completion codes are compatible with both automatic and manual approval.
Approval Methods
1. Automatic Approval
The fastest way to approve workers is with automatic approval. When you choose automatic approval, all workers who enter the correct approval code will be automatically approved as soon as they submit their approval code. If workers enter the wrong code, they will go into a “pending” state, where you can manually decide whether to approve or reject them. Rejections are never automatically issued.
2. Manual Approval
When you choose manual approval, all workers enter a “pending” state after submitting your study. You can select "Manage pending" from the study Dashboard to see a list of workers and their submissions. If a worker entered the correct code, you will see a green check mark next to their Worker ID. If they entered the wrong code, you will see a red X. From this page, you can manually approve or reject each worker.
TIP! To ensure you don't forget to pay workers when using manual approval, you can set a deadline by which any workers still pending will be automatically approved. You can set the auto-approval deadline in minutes, hours, or days. Once Workers are approved, the approval cannot be reversed.
