CloudResearch intermittently asks demographic information from MTurk workers and awards demographic qualifications based on response consistency.
How CloudResearch Gathers Demographic Data
We determine people’s demographic characteristics by adding a few questions to each HIT submission page. Researchers are given the option to allow or disallow these questions on HITs associated with their studies and workers are free to answer these questions or ignore them. This means all the demographic data CloudResearch collects from workers on MTurk is done through volunteer submission.
When researchers grant permission, we add three questions to each HIT. The questions are randomly selected from a list and are asked in the order of priority that CloudResearch assigns to them. We ask workers demographic questions at various points in time and unassociated with any one HIT. Workers are incentivized to answer these questions because doing so creates more opportunities for them to participate in well-paying studies. An example of how these questions appear to workers is below.
The Accuracy of Panel Data
Because we ask worker’s the same questions periodically, we can determine their response consistency. People who consistently respond the same way are granted a qualification that makes them eligible when a specific Panel is selected. When the Panel is selected, workers are targeted directly but the study appears like any other study keeping workers from knowing what criteria made them eligible.
While our process of determining response consistency for demographic data is fairly accurate it is not perfect. Sometimes people slip through who do not actually possess the qualities we’re looking for. If you find participants in your dataset who do not match the demographic criteria of a panel feature you selected, let us know and we will replace the participant.