News flash: Congresscritters using slightly shorter words and sentences
« previous post | next post »
And this is apparently a Bad Thing. Tamara Keith, "Sophomoric? Members Of Congress Talk Like 10th-Graders, Analysis Shows", NPR Morning Edition, 5/21/2012:
Every word members of Congress say on the floor of the House or Senate is documented in the Congressional Record. The Sunlight Foundation took the entire Congressional Record dating back to the 1990s and plugged it into a searchable database.
Lee Drutman, a political scientist at Sunlight, took all those speeches and ran them through an algorithm to determine the grade level of congressional discourse.
"We just kind of did it for fun, and I was kind of shocked when I plotted that data and I saw that, oh my God, there's been a real drop-off in the last several years," he says.
In 2005, Congress spoke at an 11.5 grade level on the Flesch-Kincaid scale. Now, it's 10.6. In other words, Congress dropped from talking like juniors to talking like sophomores.
No. Really, no. David Haglund, "No, Members of Congress Don't "Talk Like 10th Graders", Slate 5/22/2012:
Yesterday, NPR's Morning Edition included a story which, online at least, ran under the headline, "Sophomoric? Members of Congress Talk Like 10th-Graders, Analysis Shows." "OMG," you probably said to yourself on seeing that headline, "they totes do that, because Congress is, like, such an idiot." Or, rather, you didn't say that, because you don't talk like a 10th-grader, and neither do members of the U.S. Senate and House of Representatives. The"analysis" cited in the NPR headline proves no such thing. It is merely the latest attention-grabbing use of the Flesch-Kincaid test, an enormously reductive little tool that measures two things: how long one's sentences are, and how big the words are in those sentences. The results of that test are then rather brilliantly assigned a "grade-level," giving headline-writers everywhere a faux-scientific excuse to call politicians stupid.
Chad Nilep, "Sophomoric application of readability tests", Society for Linguistic Anthropology, 5/21/2012:
According to NPR’s Tamara Keith, “In other words, Congress dropped from talking like juniors to talking like sophomores.”
Nonsense. The Flesch-Kinkaid Readability Test purports to be a test of readability, not speech style. Furthermore, although the grade-level score is intended to map reading-ease scores to US grade levels, both scores need to be taken with a proverbial grain of salt.
As Gabe Doyle at Motivated Grammar argued in 2008, Flesch-Kinkaid scores do not correlate significantly with listening comprehension, even for edited prose. Transcription of extemporaneous speech is likely even less well correlated, since things like corrections, false starts, and fillers do not appear in most written texts, and there is often no obvious and uncontroversial way to segment a spoken utterance into sentences. (Flesch-Kinkaid essentially measures the number of words per sentence, and the number of syllables per word. See below for an example of how arbitrary such measurements can be.)
The analysis behind all of this: Lee Drutman, "The changing complexity of congressional speech", Sunlight Foundation 5/21/2012.
Just to show that you can't win, and generally can't even get out of the game, please think back to the press coverage in June 2010 of an even sillier analysis of one of Barack Obama's speeches (covered by LL as "Language guru runs with the journalistic pack", 6/17/2010). Under the headline "Language guru: Obama speech too 'professorial' for his target audience", CNN told us that
President Obama's speech on the gulf oil disaster may have gone over the heads of many in his audience, according to an analysis of the 18-minute talk released Wednesday.
Tuesday night's speech from the Oval Office of the White House was written to a 9.8 grade level, said Paul J.J. Payack, president of Global Language Monitor. The Austin, Texas-based company analyzes and catalogues trends in word usage and word choice and their impact on culture.
The pattern is clear. Just take whatever preconception you've decided to reinforce (Congress is stupid and ignorant; President Obama is aloof and elite), and cite some Flesch-Kincaid numbers to make it scientific-y.
As for the Flesch-Kincaid test, I'll just note that not only doesn't it pay any attention to the structure or content of sentences, it doesn't even take any account of word frequency. It's just
0.39 * (Words/Sentences) + 11.8 * (Syllables/Words) – 15.59
That's it. The difference between a Flesch-Kincaid score of 11.5 and one of 10.6 might be 0.9/0.39 = 2.3 fewer average words per sentence, or 0.9/11.8 = 0.076 fewer average syllables per word, or a mixture of the two, say 1.15 fewer average words per sentence combined with 0.038 fewer average syllables per word.
As we've previously noted, there are some genuine long-term trends towards decreasing word length and (especially) decreasing sentence length. But it's not plausible that this represents a gradual infantilization of American prose style.
Shanth said,
May 23, 2012 @ 3:10 am
Clearly we need more run on sentences from senators droning on about polysyllabic subjects.
Theo Vosse said,
May 23, 2012 @ 4:04 am
Just to give a simple concrete example: if the topic changes, say from "Afghanistan" to "banks", and a speech consists for 2.6% of this topic word, then –even when the rest of the speech stays the same– that change alone can make the average syllable per word drop 0.076, and thus a whole grade.
There is so much wrong with this way of analyzing politics…
Luke said,
May 23, 2012 @ 4:13 am
I read at the weekend that French keyboards make you shift for a full stop, apparently because their sentences are so long that it is rarely used. Is either that factoid or the explanation true?
Andrew said,
May 23, 2012 @ 4:32 am
Luke – apparently it is true that the full stop requires a shift: see http://www.fentek-ind.com/images/kbfs.jpg (or maybe they use the decimal point from the numeric keypad). The explanation sounds like nonsense though.
AlexB said,
May 23, 2012 @ 5:02 am
I'm typing this from an azerty, and I can tell you that the . and ? require a shift. As you normally start a sentence with an uppercase letter, this has always made sense to me, you simply keep pressing the shift key.
Andy Averill said,
May 23, 2012 @ 5:18 am
@Luke, I'm sure somebody's done a study comparing average sentence length in French and English, but I'm having trouble finding it. The closest I found was a paper claiming that English sentences were slightly longer, but it's on JSTOR so I can't read more than the abstract.
Edith Maxwell said,
May 23, 2012 @ 5:42 am
(Clapping her hands with delight) I knew you'd be all over this story when I heard it!
D.O. said,
May 23, 2012 @ 6:25 am
@Theo Vosse: of course, "banks" is to simple a word. An aspiring polititian should talk about "banking institutions" which increases both sentence length and syllable rate.
languagehat said,
May 23, 2012 @ 8:03 am
But it's not plausible that this represents a gradual infantilization of American prose style.
I'm not sure whether by that you mean it's not plausible that this limited result can be taken to demonstrate an (otherwise plausible) infantilization or whether you think the infantilization itself is implausible; if the latter, I suggest reading some Congressional speeches from a century or two ago and comparing some from today. "Infantilization" is of course a loaded word, but I don't think one can plausibly deny that the level of complexity of both sentences and thought has gone way down. There are obvious reasons for this, some good and some bad, but the fact itself is (it seems to me) pretty plain.
[(myl) Well, to start with, I'm skeptical that there's been a ~10% change since 2005. And taking a longer view, there are plenty of thoughtful and elegant speeches in a modern style, with shorter sentences and plainer words, as well as plenty of awkward or vapid old-fashioned bombast from the 19th century. For a carefully-constructed example of the latter, consider the start of Grant's 1873 Inaugural Address:
Under Providence I have been called a second time to act as Executive over this great nation. It has been my endeavor in the past to maintain all the laws, and, so far as lay in my power, to act for the best interests of the whole people. My best efforts will be given in the same direction in the future, aided, I trust, by my four years' experience in the office.
When my first term of the office of Chief Executive began, the country had not recovered from the effects of a great internal revolution, and three of the former States of the Union had not been restored to their Federal relations.
It seemed to me wise that no new questions should be raised so long as that condition of affairs existed. Therefore the past four years, so far as I could control events, have been consumed in the effort to restore harmony, public credit, commerce, and all the arts of peace and progress. It is my firm conviction that the civilized world is tending toward republicanism, or government by the people through their chosen representatives, and that our own great Republic is destined to be the guiding star to all others.
Under our Republic we support an army less than that of any European power of any standing and a navy less than that of either of at least five of them. There could be no extension of territory on the continent which would call for an increase of this force, but rather might such extension enable us to diminish it.
The theory of government changes with general progress. Now that the telegraph is made available for communicating thought, together with rapid transit by steam, all parts of a continent are made contiguous for all purposes of government, and communication between the extreme limits of the country made easier than it was throughout the old thirteen States at the beginning of our national existence.
That address as a whole has a Flesch-Kincaid grade-level index of 15.39, compared to 9.32 for Barack Obama's 2009 Inaugural. There's certainly a difference in style, but I don't think it's at all obvious that this is a process of decay.]
Zubon said,
May 23, 2012 @ 8:33 am
@languagehat
You would want to read a representative sample of things said in Congress two hundred years ago, not the ones that history has deemed worth remembering or preserving. If you have all the words, the older group surely looks more vapid than anyone remembers from the "best of" speeches. Note also that members may "revise and extend" speeches in the Congressional Record, and only recently was it noted when this happened, so the Record will include things never said and exclude some things said.
But yes, a trend of losing 0.9 grade levels per year is not plausible unless the Congressional Record was mostly "goo goo ga ga" during the 1980s, and I'm not sure what it means to have a -170 grade level 200 years ago.
[(myl) The claim is 0.9 grade levels in 6 years, so a linear extrapolation over 200 years would take us to a grade level of 10.6 + 200*0.9/6 = 40.6 back in 1812, corresponding I suppose to 3 PhDs plus an MD; and extrapolating forwards, we should reach a grade level of 0 in 10.6/(0.9/6) = 70 2/3 years, i.e. sometime in the late fall of 2083.]
Tom said,
May 23, 2012 @ 9:23 am
I have to admit, those of us at Sunlight were wondering if LL would pick up this story — and we weren't expecting a warm response, given past coverage here of the media's fascination with F-K. This is much friendlier than I'd feared!
At any rate, while I agree with all of you that the "speaks like high schooler" formulation adopted by most reports on our work is unfortunate, I would like to emphasize the results from our analysis that I think are genuinely interesting, and which have been ignored by most media accounts (and, less understandably, by writeups like the Slate article that you link to).
Specifically: F-K scores have been dropping *recently* (not at a constant rate, as Zubon implies); and they are negatively correlated with ideological extremism (as measured by NOMINATE), particularly on the right. There are some weaker but still interesting relationships between F-K and seniority and geography as well.
None of us at Sunlight think F-K has much to do with high schoolers (for one thing, we know the measure was never calibrated for extemporaneous speech). But–and I am admittedly speaking as a non-expert on these matters–it's a handy scalar, and it seems reasonable to say that it measures something about the nature of how speech is constructed, whether you want to call it complexity or ornateness or whatever. And that attribute has been changing recently. To us, at least, this is interesting and worth noting.
Finally, let me speak to the primary reason we did this analysis in the first place: to draw attention to our CapitolWords.org project, which debuted last fall. That project parses the Congressional Record and exposes it through an interface similar to Google Books' N-Gram Viewer. To my knowledge this is the most complete parsing of the CR that's available (and it's freely available through our API), and I think there are many more interesting analyses that the data could enable. We're already working with scholars at Duke, LSE and other institutions to facilitate their analysis of the data. If anyone else is interested in the project, I'd encourage you to reach out — tlee (at) sunlightfoundation (dot) com.
[(myl) In fairness to you folks, I should have noted the well-documented care with which you did the analysis, and the caveats about its interpretation that you expressed in your online presentation of the results. And the "Capitol Words" project looks interesting — more on it later. As for Congress "talking like tenth graders", I guess the PR temptation was just to great to resist :-).]
Rod Johnson said,
May 23, 2012 @ 9:40 am
When I was in 15th grade, that Grant speech would have been totally unexceptional. We all talked that way.
ENKI-2 said,
May 23, 2012 @ 9:46 am
@D.O.
I was thinking about that, and it occurred to me that words considered to be 'intellectual' are not necessarily longer than 'simple' words (and do not necessarily have more syllables). In the context of banking discussion, using the word 'usury' would be perceived as indicating an upward tick in the information content of the conversation, yet it has fewer syllables than 'economy' and is the same number of letters as 'banks'. Perhaps this test would be improved by adding in scrabble scores as a factor…
KathrynM said,
May 23, 2012 @ 11:21 am
No doubt it's just me, but if I were NPR trying to make this sound legitimate, I would not have quoted Mr. Drutman's gushy run-on sentence as an explanation. Of course, that sentence earns a Fleisch-Kinkaid 14; gotta be sentence length, not word length.
John said,
May 23, 2012 @ 11:31 am
I think it indisputable that Congress is stupid, but sentence complexity is not the reason why.
Congress may be getting dumber, but grade levels don’t prove it « Motivated Grammar said,
May 23, 2012 @ 11:40 am
[…] debates back in 2008, and Chad Nilep at the Society for Linguistic Anthropology and Mark Liberman at Language Log each talked about it in light of this new story. Here's an updated set of arguments why the […]
Zubon said,
May 23, 2012 @ 2:42 pm
Gah, missed a denominator and flipped a sign! A bad math morning for Zubon.
Joe said,
May 23, 2012 @ 2:44 pm
@languagehat: "I don't think one can plausibly deny that the level of complexity of both sentences and thought has gone way down"
You had me until "thought". If there is a metric for complexity of thought (presently and in the past) then I'd like to know what that is. However, if your intention is to conflate sentence complexity with thought complexity, then I'd have to disagree. John Mcwhorter did a great job of articulating the pitfalls of this particular conflation in the NYT: http://opinionator.blogs.nytimes.com/2012/04/23/talking-with-your-fingers/
Rubrick said,
May 23, 2012 @ 4:02 pm
It's unfortunate that Flesch-Kincaid is used for political misanalysis, but far more so that it's relied on heavily by the textbook publishing industry.
KeithB said,
May 23, 2012 @ 4:35 pm
How about testing Obama's love letters?
One of the comments mentions that the WND article offers 3 incorrect "corrections" to Obama's prose…
http://freethoughtblogs.com/dispatches/2012/05/23/cashill-obama-didnt-write-own-love-letters/
“Take whatever preconception you’ve decided to reinforce . . . | clusterflock said,
May 23, 2012 @ 7:11 pm
[…] This, from Language Log, explains it all so much better than I could. […]
SeekTruthFromFacts said,
May 24, 2012 @ 3:44 am
The Chinese newspaper Global Times covered the news yesterday morning inside their Mandarin edition. The headlined was (from memory) "American Congressmen not up to the level of senior high students", giving their usual anti-Western spin. I suspect many readers enjoyed it greatly.
languagehat said,
May 24, 2012 @ 7:58 am
You had me until "thought". If there is a metric for complexity of thought (presently and in the past) then I'd like to know what that is. However, if your intention is to conflate sentence complexity with thought complexity, then I'd have to disagree.
Conflate sentence complexity with thought complexity? Oh pardon me sir, it's the furthest from my mind! No, no, they're two separate issues, and I'm perfectly willing to admit I stand on shakier ground when it comes to the latter. I picture Clay and Webster listening to today's congressional "debates" (mostly orating for the cameras to an empty chamber, as I understand it) and erupting in righteous wrath. It goes without saying that empty rhetoric and simplistic ideas abounded then as now; it simply seems likely to me that there was more real Ciceronian oratory with complex chains of ideas back then to raise the average. But I freely admit I have done no breakfast experiments to check on that, and furthermore that I have no intention of doing so. Since I'm simply fulminating, I'll follow tradition and sign myself
Disgusted of Tunbridge Wells
Joe said,
May 24, 2012 @ 10:59 am
I imagine that there will always be a Disgusted of Tunbridge Wells in every generation complaining about some general degradation of thought processes. Socrates derrided writing beacuse it "will create forgetfulness in the learners' souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves". I imagine that if he were to see people engaged in discussions without having a word uttered to each other in speech, he would (in Woody Allen's words) "never stop vomitting".
This Week’s Language Blog Roundup | Wordnik said,
June 1, 2012 @ 11:14 am
[…] In politics, we learned that Mitt Romney wants a better “Amercia” (improved, apparently, by a name change) and were delighted by the subsequent snarky tumblr. Ben Zimmer noted the backronym of the week, the Ex-PATRIOT Act, while at Language Log, Mark Liberman discussed the speech levels in politics. […]
[links] Link salad is identical cousins with Linkee-poo | jlake.com said,
June 14, 2012 @ 2:27 pm
[…] News flash: Congresscritters using slightly shorter words and sentences — Language Log deconstructs a current, somewhat idiotic political meme. […]