You can approve MTurk workers either manually or automatically depending on the type of approval code you setup for your study.
Approval Methods
1. Automatic Approval
The fastest way to approve workers is with automatic approval. When you choose automatic approval, all workers who enter the correct approval code will be automatically approved as soon as they submit their work. If workers enter the wrong code, they will go into a “pending” state where you can manually decide whether to approve or reject them. Rejections are never automatically issued.
Automatic approval is compatible with both static and dynamic secret codes. Once a worker is approved, the approval cannot be undone.
2. Manual Approval
When you choose manual approval, all workers enter a “pending” state after submitting your study. You can select "Manage pending" from the study Dashboard to see a list of workers and the approval code each submitted. If a worker entered the correct code, you will see a green check mark next to their Worker ID. If they entered the wrong code, you will see a red X. From this page, you can manually approve or reject each worker.
Manual approval is compatible with static, unique, and dynamic custom codes.
Types of Approval Codes
When you create your study with an external survey platform like Qualtrics or Alchemer, Mechanical Turk cannot verify that workers actually completed your study. To deal with this issue, researchers commonly place a "secret code" at the end of their study. Once workers reach the end of a study, they copy the secret code from the survey into the HIT submission box on Mechanical Turk to demonstrate to the researcher they have completed the study. CloudResearch offers three types of secret codes: fixed, custom, and dynamic.
- Fixed secret codes are static, alphanumeric codes that are the same for each worker who takes your study. You can generate a fixed completion code however you want, but the code cannot contain the numbers 1 and 0 or the letters I or O.
- Custom secret codes allow you to present each worker with a unique or custom code. You can do this using your survey platform, JavaScript, or any other method you choose. Because custom codes are housed within your survey, CloudResearch has no access to the code. This means when using custom codes you must manually approve workers.
- Dynamic secret codes are similar to custom codes in that they allow you to give each worker a unique code. Unlike custom codes, however, dynamic codes are integrated with CloudResearch, which allows you to use automatic approval if you wish.
Things to Consider
When choosing among different approval methods and types of approval codes there are a few things to keep in mind.
- Once a worker is approved, the approval cannot be reversed. If you would like to check each worker's submission before it is approved, you should use manual approval.
- Although it doesn't appear widespread, static codes create the possibility for fraud by allowing workers to share codes. If workers shared the completion code to a survey then other workers could, theoretically, submit the survey with doing little or no work and try to collect the reward. Dynamic codes eliminate this possibility by generating a unique code for each worker.
- If you reject a worker and later decide you would like to reverse the rejection, you can do so. You cannot, however, reverse an approval once it has been issued.