Language Log

« previous post | next post »

I can find no better description of Amazon's Mechanical Turk than in the "description" tag at the site itself:

The online market place for work. We give businesses and developers access to an on-demand scalable workforce. Workers can work at home and make money by choosing from thousands of tasks and jobs.

This is followed by a "keywords" meta tag:

make money, make money at home, make money from home, make money on the internet, make extra money, make money …

This makes the site sound a bit like the next stop on Dave Chapelle's tour of his imagined Internet as physical place, and indeed it does have its seamy side. But I come to defend Mechanical Turk as a useful tool for linguistic research — a quick and inexpensive way to gather data and conduct simple experiments.

This post is long, so here's some help with navigation:

1. An experiment
2. Results
3. Assessment of Mechanical Turk

1. An experiment

My fellow Language Logger Dan Jurafsky was the first to tell me about the service. He emphasized that it worked best for quick, simple tasks. Recently, I had just the thing: I wanted to learn more about the pragmatic effects of attaching an intensive like absolutely, totally, or damn to positive and negative adjectives of various strengths. It is clear that superb and horrible are emotively positive and negative, respectively. What about totally superb and totally horrible? We have a lot of evidence from online product reviews about what the intensive signals, but the reactions to this evidence are sometimes skeptical. People worry about the specialized form of the reviews, the reliability of the meta-data, and so forth. Fair enough. This looks like a job for the Mechanical Turk.

Step 1: Data prep I extracted from the UMass Amherst Linguistics Sentiment Corpora all the words with particular distributions across the five-star rating categories. Intensives appear mainly in the one-star and five-star reviews, distributed equally in those extreme categories. Emotively positive adjectives typically correlate with the rating categories, rising steadily in frequency from one to five stars. Emotively negative adjectives like horrible correlate inversely with the rating categories, falling off steadily from one to five stars. And words like but, quite, and somewhat are largely confined to the middle of the rating scale.

Step 2: Data format I combined a subset of the intensives with a subset of the positive and negative adjectives to form phrases like totally superb and totally horrible. I put these in a list with the unmodified adjectives, threw in some fillers like possibly superb and quite, shuffled them up, and put them in a file, one phrase per line:

phrase
wow
mostly
fairly superb
absolutely helpful
...

The first row gives the name of the column. (There can be multiple, comma-separated columns, each with its own name.) I had 43 phases in all.

Step 3: HIT design I used the Mechanical Turk Web interface to create my basic Human Intelligence Task (HIT) design.* Here's what it looked like:

Assessing emotivity

Please read the following word or phrase and assess its emotivity on the scale provided:

${phrase}

+3	Highly emotively positive
+2	Emotively positive
+1	Somewhat emotively positive
0	Not emotive
-1	Somewhat emotively negative
-2	Emotively negative
-3	Highly emotively negative

The ${phrase} piece is a placeholder for the phrases in my list. When I clicked "Publish", Mechanical Turk created 43 HITS of this form, one for each value of ${phrase} drawn from my input data file.

I requested 50 assignments for each of my 43 HITs, for a total of 2150 jobs. I was paying $0.01 for each HIT, and Amazon tacks on a surcharge, so the total cost for this was $32.25 ($21.50 + surcharge). I paid this amount to Amazon (it's just like buying a book from them). When I published my HITs, the $32.25 disappeared from my account. Amazon pays the workers.

Step 4: Wait a few hours I sat back and waited for my results. The first time I requested a batch of HITs like this, I thought I would have all the results in a week or so. That time, I had them in two hours. (It was a Friday afternoon.) I had to wait about five hours for all of the above to come in. When I have asked workers to do more intricate things like provide text, the results have come at a slower pace, but they've always finished in a few days.

* The command-line tools work well, but the styling options seem to be more limited than those provided by the Web interface, which is also easier to deal with.

2. Results

The results suggest that, true to their label, intensives intensify. Here's an example involving the strongly positive modifier awesome:

This is a box-and-whisker plot, but with the individual data points displayed (jittered left and right so that they don't sit atop each other). The thick black lines are the median values. The box picks out the core of the response data. For the intensive versions, the box has collapsed and the median line is almost hidden; essentially all the responses were at the very top of the positivity scale.

With negative adjectives, the effect is largely the same, but on the negative end of the scale. Here is a look at disappointing, which has less variation than awesome when unmodified but which an intensive can push all the way to the extremes.

For adjectives like superb and horrible, already extreme in their own right, the picture is slightly different, but still very much in line with the notion that intensives push scalar predicates further along their scale:

Here, the median values of both are at the highest point on the scale. But the intensified version is less likely to stray below this point.

All these results are completely in line with the U-shape for intensives seen above: sometimes it combines with positive adjectives, sometimes with negative ones. In both cases, it pushes the sentiment out to the extremes.

3. Assessment of Mechanical Turk

One might say that this is not an especially surprising result. I agree. If the results had been too surprising, I'd have lost faith in Mechanical Turk. The role these results play in my research is to fill a small empirical hole. It's important to have it filled, and it would have been difficult and/or costly to fill it with a normal experimental set-up at my university.

I've found the Mechanical Turk workers to be conscientious on the whole. I've gotten a few messages from them. One was having trouble with the buttons (I promptly blamed Amazon), another had misentered his intended choice, and a third had a question about what I was after.

Overall, the workers are incentivized to do well. It is possible for requesters to approve/disapprove their work, and one can limit one's HITs to workers at or above certain approval thresholds. Thus, doing well leads to higher approval ratings, which opens up more and better jobs to them.

As a result of this, workers are highly likely to try to model the requester's expectations. If you display an image, they are likely to try to describe the whole image if given the chance, because they are likely to assume that you are doing image annotation. If your instructions reveal that you are after a particular effect, they are likely to strive for it. Similar factors threaten regular psycholinguistic experiments too.

If you look back over the above plots, you see outliers. It happens that some of these are the same people over and over again; each data point has a worker ID attached to it, so it is easy to identify these outliers. These might be non-native speakers, trouble-makers, people with unreliable browsers, etc. They can be dealt with separately. Again, one has to make similar moves in the lab, e.g., for people who claim native competence in language X and then turn out to be struggling with their first classes in X, for people who go on auto-pilot two-thirds of the way through the experiment, etc.

So my considered view is that Mechanical Turk is a useful tool for language researchers, and that it can only become more useful if we exchange notes on our experiences with it.

March 15, 2009 @ 10:17 am · Filed by Chris Potts under Computational linguistics, Pragmatics, Research tools, Resources, Words words words

Permalink