Language Log

« previous post | next post »

That's apparently the commonest 5-word sequence in English, barely beating out "property of their respective owners". At least, those are the commonest five-word sequences on the web.

Last week, in commenting on Geoff Pullum's "Familiar six-word phrase or saying" post, I observed that

For five-word phrases, a version of the question "What five-word phrase occurs most often on Google?" can definitively be answered by reference to the Web 1T 5-gram corpus, created by researchers at Google, which contains English n-gram counts from about one trillion words of web text.

Several readers asked me what the answer actually is.  The answer turned out to be not entirely trivial to get, and it may not be as interesting as you'd expect. Or maybe it's more interesting, I don't know. Anyhow, I live to serve, if not always at internet speeds, and the details are below.

The Web 1T 5-Gram Corpus contains all the length-five sequences of tokens occuring more than 40 times, from a sample of the web containing 1,024,908,267,229 tokens. I'll pass over in silence various minor technical difficulties associated with the fact that the corpus is 54 GB of text, stored in compressed form on five DVDs, and that it contains 1,176,470,663 5-grams, divided in alphabetical order into 118 files. It's a tribute to modern technology that I was nevertheless able to get to an answer, using only a laptop computer and a few minutes spared now and then from other activities.  (Of course, it's a much more significant tribute to modern technology that Thorsten Brants and Alex Franz were able to compile the corpus in the first place.)

The tokenization used in creating this corpus follows the tradition established by the Wall Street Journal portion of the Penn Treebank, which represents sentence boundaries as special tokens, and splits punctuation out as separate tokens as well.

As a result, the most-frequent 5-grams tend to be things like

– – – – –

which occurs 88,974,253 times.

If we ignore sentence-boundary markers and punctuation (other than the apostrophes in things like "what's hot" and "you're looking for"), then the commonest 5-gram in the corpus is

Send a private message to

which occurs 26,672,131 times. (Free tip: as far as amazon.com knows, there is no published book whose title includes the string "send a private message"…)

The rest of the top-20 commonest sequences, with their counts, are:

Use of this Web site 19678811
of this Web site constitutes 19703371
this Web site constitutes acceptance 19723554
Web site constitutes acceptance of 19724386
eBay User Agreement and Privacy 19807811
the eBay User Agreement and 19808132
acceptance of the eBay User 19850253
of the eBay User Agreement 19850627
constitutes acceptance of the eBay 19850700
Designated trademarks and brands are 20820815
User Agreement and Privacy Policy 20917050
trademarks and brands are the 20975334
and brands are the property 21113548
brands are the property of 21139112
site constitutes acceptance of the 21556427
this result in new window 24059811
Open this result in new 24059963
the property of their respective 24891265
are the property of their 24938581
property of their respective owners 25640531

See what I mean?  This is not really telling us anything about the state of the English language, and  what it's telling us about web culture is pretty limited.  (Though because of the overlapping 5-grams, we can also conclude that "are the property of their respective owners" is the commonest 7-word sequence on the web. If we care.)

Even looking down the list to strings that occur only about a million times, nearly everything is either boilerplate from commercial websites, or fragments of random lists:

have not been able to 1000387
Philippines Pitcairn Poland Portugal Puerto 1000489
Islands Faroe Islands Fiji Finland 1001073
Shopping help About this site 1001307
part of The New York 1001557
NM NY NC ND OH 1003534
is the responsibility of the 1003710
your web hosting at eNom 1005283
Get your web hosting at 1005294
MD ME MI MN MO 1005327
See more of the brand 1006237
d d d d d 1006434
great domain name at eNom 1006715
in TripAdvisor 's popularity index 1007567
a great domain name at 1007579
poster 's website AIM Address 1007641
Get a great domain name 1007806
Visit poster 's website AIM 1007875
browser does not support script 1010343
Ivoire Croatia Cuba Cyprus Czech 1011221
NY NC ND OH OK 1011482
State University of New York 1011879
ND OH OK OR PA 1012023

Obviously the 5-gram resource as a whole is not like this — the lower-frequency end of its n-gram lists is an invaluable aid to language modeling. But it's inevitable, if you think about it, for things like eBay boilerplate to dominate the high-frequency end of the higher-order n-grams in a web snapshot.

I'm happy that "Send a private message to" nosed out "property of their respective owners". (Another free tip — there's also no book entitled "Property of their respective owners". Nor any album or band …) But neither one of these phrases strikes me as a plausible candidate for the real "commonest five word sequence in English".  And because it's harder to filter out commercial and legalistic boilerplate than to filter out punctuation, it's not going to be easy to get a useful answer from a web snapshot of this kind.

It would be more meaningful to take counts from a balanced corpus like the BNC — though at a mere 100 million words, that's 10,000 times smaller. This creates a different problem. Since a count of 1,000,000 in the 1T corpus translates to a count of 100 in the BNC,  it's likely that the commonest real 5- and 6-word sequences will have low enough counts that the ordering is statistically unreliable.

[I'm not sure why the 1T corpus slightly different counts for overlapping 5-grams that must be pretty much coextensive on the web, e.g.

the property of their respective 24891265
are the property of their 24938581
property of their respective owners 25640531

The differences are fairly small — in this case, around 3% — but it seems surpising to see even that much divergence associated with such near-zero-entropy string continuations. Perhaps this does reflect real differences in the frequency counts in the sampled corpus, but perhaps there is a technical reason having to do with approximations in the counting algorithm. I'll see if I can find out. ]

[Meanwhile, for those interested in the earlier discussion of that all-too-common six-word phrase "before turning the gun on himself", Fev at Headsup: The Blog has something to add, in the form of a reference to Gaye Tuchman, "Objectivity as strategic ritual", American Journal of Sociology 77:660-679, 1972. Fev supports rootlesscosmo's suggestion that

recourse to boilerplate may actually help persuade editors and referees that the work meets professional standards; if a paper says "the animals were sacrificed" it's science, if it says "I killed the mice" it's macabre, though the event is the same.

by quoting Tuchman:

Attacked for a controversial presentation of 'facts,' newspapermen invoke their objectivity almost the way a Mediterranean peasant might wear a clove of garlic around his neck to ward off evil spirits.

For more on journalistic objectivity rituals, see e.g. "Ritual questions, ritual answers", 6/25/2005; "Down with journalists!", 6/27/2005; "Ritual interviews", 9/18/2005.  Perhaps it's time for the anthropologists to devote more time to documenting the culture of journalists, who seem to be at least as endangered as Mediterranean peasants are. ]

March 15, 2009 @ 7:03 am · Filed by Mark Liberman under Syntax

Permalink