Inter-Coder Reliability and Mechanical Turk

After the discussion yesterday regarding inter-coder reliability I had an interesting idea. There is a service that Amazon offers called "Mechanical Turk". This service allows businesses and developers to programmatically create what they call human intelligence tasks (HITs). An HIT usually consists of a simple task, such as identifying the object in an image to more complex tasks, such as language translation.

In the case of inter-coder reliability, a researcher could develop an HIT that requires a user to complete short coding examples, and then compare those results across (potentially) thousands of users. The only downside is that people expect to be paid for completing an HIT - but so do coders. Even more interesting is the fact that one could use Mechanical Turk to do other types of experiments as well. You can also choose your sample relatively granularly. In any case examples like this show some pretty interesting potential.