Or, as Kevin Drum put it, "New Test Shows Severe Shortcomings in Nation's Press Corps". Or maybe the headline should be "Onion Takes Over WSJ Education Beat".
According to Stephanie Banchero, "Students Fall Flat in Vocabulary Test", Wall Street Journal 12/6/2012:
U.S. students knew only about half of what they were expected to on a new vocabulary section of a national exam, in the latest evidence of severe shortcomings in the nation's reading education.
Eighth-graders scored an average of 265 out of 500 in vocabulary on the 2011 National Assessment of Educational Progress, the results of which were made public Thursday. Fourth-graders averaged a score of 218 out of 500.
The results showed that nearly half of eighth-graders didn't know that "permeates" means to "spread all the way through," and about the same proportion of fourth-graders didn't know that "puzzled" means confused—words that educators think students in those grades should recognize.
Regular readers may recall my rule of thumb ("Ignorance about ignorance", 10/9/2012):
When you read or hear in the mass media that "Only X% of Americans know Y", don't believe it without checking the references — it's probably false even as a report of the survey statistics.
I should have phrased the rule explicitly to cover cases of the form "X% of Americans don't know Y" — but anyhow, let's look at one of Ms. Banchero's examples, as described in "Vocabulary Results From the 2009 and 2011 NAEP Reading Assessments". The evidence that "nearly half of … fourth-graders didn't know that 'puzzled' means confused" comes from the pattern of answers to this question:
The correct answer is D, which 51% of the fourth graders chose. It references the third paragraph of the passage that the students are being asked about:
Here's what the NAEP report says about the wrong answers:
Option C, chosen by 32% of the fourth graders, in fact references the sixth paragraph of the first page of the passage:
It's likely that the students who answered "C" actually did know that "'puzzled' means confused" — they just remembered the second passage on the previous page where children encountered a state of affairs markedly different from what they expected to see, rather than the first one; and they failed to realize that to meet the expectations of the test makers, they needed to reference a passage where the word "puzzled" itself was used, rather than one where its meaning was appropriate. Thus a more plausible interpretation of these results is that at least 51+32 = 83% of fourth graders know what "puzzled" means, and can apply that knowledge in the interpretation of a fairly long and complex text.
OK, we've learned that Ms. Banchero swallowed the NAEP report's alarmism willingly and uncritically — that's the normal bias towards "sensationalism, conflict, and laziness". What's a little out of the ordinary here is indicated by this passage at the bottom of the story:
Corrections & Amplifications
The test results showed that nearly half of eighth-graders didn't know that "permeates" means to "spread all the way through," and about the same proportion of fourth-graders didn't know that "puzzled" means confused. An earlier version of this article incorrectly said that nearly half of eighth-graders didn't know the meaning of "puzzled."
So in an article about "severe shortcomings in the nation's reading education", the Wall Street Journal's ace reporter made an elementary mistake in reading comprehension. But wait, it gets better.
Remember the part about how "U.S. students knew only about half of what they were expected to on a new vocabulary section of a national exam", because "Eighth-graders scored an average of 265 out of 500 in vocabulary" and "Fourth-graders averaged a score of 218 out of 500"?
This is not quite as transparently brainless as the complaint about No Child Left Behind testing attributed to Prof. Fred Hess, director of NU's Center for Urban School Policy:
The tests being used are formulated so that 50 percent of the test-takers will fall below the median score — in effect setting school districts up for failure no matter how much preparation students receive, he said.
But as a flaw in the practical application of elementary quantitative reasoning, it's close. Let me turn the floor over to Kevin Drum:
I'm not crying for the students. I'm crying for the reporter, who apparently believes that students "knew only about half of what they were expected to" because they scored in the vicinity of 250 out of 500. And since 250 is half of 500, that must mean students only knew half of what they should.
This is wrong on so many levels I don't even know where to start. First, these are scale scores, not percentages of correct answers. Second, they're normed scores. Third, this is the same way all the NAEP tests are done, and they all produce scores in the same vicinity. The current eighth grade math average is 284. The reading average is 265. The history average is 266. Etc. And since the scores are scaled so that ten points roughly equals a grade level, fourth graders scored a little more than 40 points lower by definition.
In other words, these numbers in isolation don't tell us anything at all about whether the vocabulary skills of our children are weak or strong. It's like saying someone who scored 100 out of 200 on an IQ test must be a moron. Unfortunately, the reporter was flatly ignorant of all this, so she simply hauled out standard hysterical template No. 4 and decided that the test results represented "severe shortcomings in the nation's reading education" even though they show no such thing.
The facts about what the NAEP numbers really mean are explained at length in "An Overview of Technical Procedures for the NAEP Assessment", easily found from the NAEP's "about" page.