Studies are sometimes marked as complete even though the progress bar shows the desired sample size has not been reached. This may occur for several reasons, but it often means the worker was able to submit a HIT to CloudResearch but not to MTurk.
Whenever your study is marked as complete but the progress bar on CloudResearch shows fewer participants than the desired amount, it is because workers were able to submit their HIT to CloudResearch but not to MTurk.
The progress bar on CloudResearch logs participants as they are approved on Mechanical Turk. When a worker submits a HIT, the submission is recorded through CloudResearch, and once CloudResearch confirms the completion, the information is sent to MTurk. If a worker is on a slow internet connection, experiences an interruption in service while submitting the HIT, or runs out of time to complete the HIT, the information may be recorded on CloudResearch but never sent to MTurk. When any of these issues occur, CloudResearch will show that the total number of requested workers have taken your survey and mark the study as Complete. However, since the worker’s HIT was never recorded on MTurk the participant will not be counted on the progress bar.
Despite this error, the full number of requested participants should appear in your data file (from Qualtrics, SurveyGizmo, or whatever survey platform you use). In addition, the number of participants in your data file should match the number you requested. Please check your data file and contact us if the number of participants in your file does not match your requested total.
TIP! In the scenarios described above workers are able to submit their HIT to CloudResearch but not to MTurk. This means the workers' submission is never approved on MTurk and the worker never gets compensated. In such cases, the worker will usually reach out to you and ask for compensation. To pay this worker, you can setup a Compensation HIT. For a comprehensive article on compensating workers, please see our Compensating Mechanical Turk Workers blog.