A reading comprehension test
« previous post | next post »
Or maybe it's a writing comprehension test. Anyhow, it's past the jump.
Read the following passage from Sam Dillon, "U.S. Students Remain Poor at History, Tests Show", NYT 6/14/2011:
Diane Ravitch, an education historian who was invited by the national assessment’s governing board to review the results, said she was particularly disturbed by the fact that only 2 percent of 12th graders correctly answered a question concerning Brown v. Board of Education, which she called “very likely the most important decision” of the United States Supreme Court in the past seven decades.
Dr. Ravitch is referring to a question on the 2010 NAEP 12th grade U.S. History test in reference to the following quotation from a Supreme Court decision:
To separate them from others of similar age and qualifications solely because of their race generates a feeling of inferiority . . . that may affect their hearts and minds in a way unlikely ever to be undone. . . . We conclude that in the field of public education separate but equal has no place. Separate educational facilities are inherently unequal.
The high-school seniors were asked:
The quotation is from which Supreme Court decision?
A. Miranda v. Arizona
B. Gideon v. Wainwright
C. Mapp v. Ohio
D. Brown v. Board of Education of Topeka
A reading comprehension question (for you, not for the high-school seniors):
What percentage of the students taking the test gave the correct answer, identifying Brown v. Board of Education as the source of the quotation?
A. 82%
B. 51%
C. 28%
D. 2%
The discussion below is in white letters on a white background, so that you can consider your own answer before seeing the spoilers. Highlight the text to read it.
If you answered "2%", you've been taken in by Mr. Dillon's masterful deployment of "kids today" rhetoric. The correct answer in fact is "82%".
Answer B, "51%", is the proportion who correctly answered the next question:
The 1954 Supreme Court decision overturned which earlier decision?
A. Marbury v. Madison , 1803
B. McCulloch v. Maryland , 1819
C. Dred Scott v. Sandford , 1857
D. Plessy v. Ferguson , 1896
Answer C, "28%", is the proportion who were judged to have given a correct or partially-correct answer to a third question about the quotation:
Based on the quotation and your knowledge of history, describe the conditions that this 1954 decision was designed to correct. Be as specific as possible in your answer.
An example of an answer scored as "partial" but not "correct":
The Brown girl had to walk past the white school every day to get to her "equal" black school. Her father took the issue to court — separate but equal is not really equal.
The scorer's comment:
[This] response mentions that having separate schools for African American and White students is supposed to be equal, but that it is not really equal. However, it does not say that the Brown case was meant to desegregate schools specifically, making this a partial explanation and not a complete explanation.
An example of an answer scored as completely wrong:
Separate but equal → segragation
The scorer's comment:
This response does not provide enough information to indicate an understanding of the Brown case. The phrase “separate but equal” and word “segregation” are present, but the response makes no attempt to indicate whether the Brown case supported or struck down these ideas.
It's important to note that these were chosen by the NAEP officials as representative examples of incorrect or partial answers.
Answer D to our reading comprehension question, "2%", is the proportion of students whose answer to this third question was scored as fully "correct".
I agree with Diane Ravitch that we need to view this situation with considerable concern. It suggests that test designers and scorers are not as skillful as they should be. It suggests that educational experts like Diane Ravitch don't read reports as carefully as they ought to. And it suggests that even the best of our news organizations are, as Jon Stewart said, biased toward "sensationalism, conflict, and laziness".
Lucy Kemnitzer said,
June 22, 2011 @ 10:18 am
I have suspected for a long time that this kind of stupid shenanigans was behind most of the news reports on the abjectness of our high school students.
Test designers and education pundits like to think that if they can mark more students as failures, they're somehow being "rigorous." They're suspicious that if a large number of students seem to be successful — other than the ones theyu've already designated as fated to succeed — there must be something dishonest going on.
And news media view young people merely as characters in a morality play and their education as the scenery and props with which to play out their thrilling tragedies.
[(myl) Yes, I think that "morality play" is exactly the right metaphor. The moral is that "kids aren't learning enough/anymore because of laziness/over-scheduling/poverty/wealth/teachers/parents/TV/video-games/conservatives/liberals/…"
This is closely related to the bible-story model of scientific research interpretation.
That's not to deny that educational outcomes are sometimes bad and in serious need of improvement, or that scientific research sometimes has moral or ethical consequences.]
Adam B said,
June 22, 2011 @ 10:43 am
With a multiple choice question with four answers, it'd've been pretty surprising to get significantly below 25% correct
[(myl) It's fairly easy to do, if you design your distractors correctly.
But in this case, the crucial question was free text, not multiple choice.]
Damon said,
June 22, 2011 @ 11:00 am
I'm impressed that so many 12th graders were able to correctly identify which case Brown v. Board of Education reversed; it took me a few minutes to remember for sure that it wasn't the Dred Scott case.
I do think that the answer cited as "incorrect" should be considered incorrect, although maybe I'm being persnickety. The "partially correct" answer, though, makes it clear that the student knew what the decision was even if they weren't as explicit as they could have been. That was poor grading.
[(myl) I think you're right — though even the quoted "incorrect" answer doesn't really warrant Dr. Ravitch's anxiety, since it suggests that the student knew (or at least inferred from the quotation) what the case was about.]
Jeff S said,
June 22, 2011 @ 11:37 am
This may be quibbling, but don't reading comprehension questions generally contain the answer or some method of reasonably deducing the answer in the text? I'm not sure that the reading comprehension question can be answered correctly with the [unhighlighted] text on this page. Were you assuming that we would look through the questions on the NAEP website?
[(myl) The point is exactly that from the quoted NYT text, it's essentially impossible to avoid serious misunderstanding of the actual situation — unless you're sufficiently cynical to recognize an instance of "test confirms that kids/people today are incredibly ignorant" meme, and to expect that the cited "facts" will therefore turn out to be false or at least highly misleading.]
rpsms said,
June 22, 2011 @ 11:41 am
The answer "Separate but equal → segragation" is not incorrect, but perhaps opaque to the test scorer. The arrow is a logical operator and seems to be used in this manner in the answer. It is the operator for implication (if…then")
Kenny Easwaran said,
June 22, 2011 @ 11:52 am
The entire post is in white text for me after the one hidden answer – perhaps there's a missing tag?
[(myl) No, that's how it's supposed to be, since there are multiple spoilers in that stretch.]
J. W. Brewer said,
June 22, 2011 @ 12:07 pm
The Brown v. Board discussion evokes for me another Jim Crow parallel, namely the claim that literacy/civics tests required as a prerequisite for voting (the desirability of which could be debated in the abstract, if the test were well-designed and fairly applied) were administered in practice in certain Southern jurisdictions by arbitrarily raising the bar of what would count as a passing answer until it got to the desired end state of a sufficiently small percentage of black would-be voters passing. There are also, however, systematic pressures the other direction. I suspect that any standardized test a state mandatorily imposes as a prerequisite for a high school diploma will be designed or tweaked in practice to make sure no more than X% of students fail it (and perhaps no more than Y% in the school and/or legislative districts with the lowest-performing pool of students), where X and Y will be set based on inherently political considerations.
Philip said,
June 22, 2011 @ 12:38 pm
I can't supply the exact information because I heard it on National Public Radio the other day, but the very first time US students participated in the NAEP (sometime in the non-existent "good old days" in the 60s or 70s), they scored at the very bottom compared with students in other countries. Today, students are somewhere in the low middle.
Complaints about the decline of public education, statements like "students today can't even . . . " are routine and recurrent. Pick any time in American history, and you'll find the same thing.
Philip said,
June 22, 2011 @ 12:43 pm
Oops. I didn't read the previous post before commenting on this one.
KevinM said,
June 22, 2011 @ 1:08 pm
@JW Brewer: Ha! Both illustrate the point that nobody can score well on a test if the graders are wilfully pig-headed.
"Separate but equal → segragation" is entirely correct if you bear in mind the way most people answer questions. Most people assume that the questioner understands his own question, and possesses the knowledge that is implied by the premise of the question. That is why "yup" can be either a correct or incorrect answer. The rules of this game, however, require the student not just to answer the question but to parrot the question's premise (because we're supposedly testing "reading comprehension"). To grade it, the grader suspends his own reading comprehension and adopts the kind of brainless literalism that would lead inevitably to a flunking grade on the test. This grader is guilty of underestimating the student, who quite clearly understood what was going on in the quotation (but could use a little help with his/her spelling!). I graduated at the top of my law school class, and can recited from memory whole passages from the Plessy and Brown cases. I have no doubt that I would have lost points on this question.
Spell Me Jeff said,
June 22, 2011 @ 1:22 pm
Fail.
There were 2 Brown decisions in the mid-1950s, one in '54 and one in '55. The quote is accurately taken from 1954 (Brown I), but the decision settled only the constitutionality of separate but equal. This is of course the decision that everyone remembers, and with reason, because of its great philosophical import.
Nevertheless, the matter of correction ("relief") was postponed until further arguments could be made, and the individual cases were not remanded to the lower courts (where the actual details of correction would be hammered out) until the decision of 1955 (Brown II).
The distinction may seem persnickety from a layman's perspective, but we are discussing a matter of historical fact, from an academic perspective. The test people themselves were persnickety when discussing what made one answer better than another. I find it inconsistent, therefore, that the same level of scrutiny applied to the answers was not applied to the question itself.
(Consider your reaction if the test had asked students to name the 3 ships commanded by Columbus when he discovered the New World in 1493.)
It's further disturbing that Ravitch, the noted education historian, should accept the question at face value when she really should know better. My point is simply that her own sin is more egregious than this discussion so far has pointed out.
mollymooly said,
June 22, 2011 @ 1:23 pm
Having read the NYT article, I agree that "from the quoted NYT text, it's essentially impossible to avoid serious misunderstanding of the actual situation". However, I don't think the OP presentation is fair.
The NYT article refers to the actual question, not the multiple-choice question. The article can rightly be condemned for giving far too little context to make sense of the 2% statistic; and Revith's quote "The answer was right in front of them" hardly tallies with the sample answers provided.
But it is the OP which misleadingly implies that it is the multiple-choice question which relates to the 2% statistic. The particular trap into which we are invited to fall was laid by Prof. Liberman rather than Sam Dillon. It's just a simple Trick Question. As an example of the kind of error which can arise when shock-stats are tossed around, I find it too contrived to convince.
[(myl) I wish you were right. But let's look at how the information in the article (or perhaps the information in Diane Ravitch's statement) has been replicated elsewhere in the media. In Pat Buchanan's syndicated column for 6/22/2011, "The dumbing down of America", we read:
The Christian Science Monitor's 6/14/2011 editorial "Jobs-focused education leaves history in the dustbin" begins:
Cheryl Miller and Daniel Lautzenheiser, at The American Enterprise Institute's blog, write that
Louise Mirrer, in the Huffington Post:
Kevin Drum's reaction was properly skeptical but reflects the same (mis-)understanding of the facts fostered by the NYT article. He starts by giving this quotation from the article:
and then comments:
I could go on — there are dozens of other items on the web showing that nearly everyone who read that article drew the same incorrect conclusion about what is being claimed about 12th-graders knowledge, without any prompting from me.]
Spell Me Jeff said,
June 22, 2011 @ 1:47 pm
Oops. I'm not so original as I thought. Burke's article, cited in Marks previous post, does mention the difference between cases.
[(myl) But wait, it gets worse. Sam Wineberg points out that one of the questions on the 4th-grade test was:
The correct answer is supposed to be C, and 42% of the 4th-graders figured this out. But Wineberg observes that
The scholarly details of the "Drinkin' Gourd" song can be found here.]
Spell Me Jeff said,
June 22, 2011 @ 3:48 pm
^^ nice. The drinking gourd site is pretty clutch on its own merit, too.
Jon Weinberg said,
June 22, 2011 @ 5:48 pm
@JW Brewer: Southern literacy tests didn't work by "arbitrarily raising the bar of what would count as a passing answer until it got to the desired end state of a sufficiently small percentage of black would-be voters passing." Rather, as the Court put it in South Carolina v. Katzenbach,
To the extent that those tools weren't sufficient, there were others — in particular, requirements that applicants "explain" constitutional provisions, graded in a remarkably NAEP-like manner, and "grandfather clauses" exempting white applicants from the tests altogether.
J. W. Brewer said,
June 22, 2011 @ 6:49 pm
Jon Weinberg: I meant to smuggle all of that supporting detail into the word "arbitrarily." If it sounded like I was suggesting an objectively "fair" test but just tweaking what raw score counted as passing, I expressed myself badly.
Lucy Kemnitzer said,
June 22, 2011 @ 9:03 pm
@ J.W. Brewer: as to the claim that high school exit tests will inevitably be tweaked so that an unacceptable number of students don't fail them, that is actually not what has happened in the wake of No Child Left Behind.
Instead, what has happened is that students have been shuffled out of the pool — they are transferred out of their home high schools to continuation schools or other programs outside the district's primary mandate, and the efforts we used to be making to prevent dropouts have been sacrificed as well.
The point of the exit exams as they exist in the US today is not to assure that students are emerging with what they need to know in order to take up their next task in life, but to sell tests and programs and charters and other crap to schools which are not being funded for their actual purpose.
Jonathan D said,
June 22, 2011 @ 9:06 pm
While it's very clear that article is sensationalised, given the marking scheme, and even misleadingly incorrect in that the question what to describe specifically, not simply answer what the social problem was, I have to agree with Mollymooly that the Dillon article gives a description of the question that is quite clearly inconsistent with the claim in the OP that Ravitch referred to the first multiple choice question.
Ravitch herself also misreprents the question, but at least refers to complete and partial answers, rather than just 2% correct. This makes it even clearer that we're not talking about a multiple choice question, so you have to wonder where the error in Buchanan's column comes form.
Jerry Friedman said,
June 22, 2011 @ 9:08 pm
I disagree with the people who said "Separate but equal → segragation" was a correct answer. It's a tautology. In this notation, a summary of the Brown case might be "Separate but equal → unequal → unconstitutional". So I would have marked that student wrong. (On the other hand, I'd have given full credit to the student who put equal in quotation marks.)
[(myl) Let's keep our eye on the ball here. This is not really about how to award credit for various answers to this question, it's about whether American high school seniors understand what Brown v. Board of Education was about. Does the student who gave this answer need some coaching about how to answer test questions (and how to spell segregation? Absolutely. How well does this student understand the Brown ruling? It's hard to tell. Is this student aware of what Diane Ravitch wants students to be aware of, namely that Brown was about ending public-school segregation. Almost certainly yes.]
Stephen Nicholson said,
June 22, 2011 @ 9:41 pm
it does not say that the Brown case was meant to desegregate schools specifically, making this a partial explanation and not a complete explanation.
This part bothers me. The question says to "describe the conditions that this 1954 decision was designed to correct." And the answer: "[t]he Brown girl had to walk past the white school every day to get to her "equal" black school", does that.
It seems to me that the question is getting at what it was like before the decision, not what was the goal of the decision itself.
Also, I don't understand why the scorer is calling this an explanation, the question doesn't ask the student to explain the decision, but to describe the school system prior to the decision. Of course, maybe explanation is a term of art for answer in this context.
Chad Nilep said,
June 22, 2011 @ 10:00 pm
@Jerry Friedman
The question asks students to describe the social problem Brown was meant to address. "Segregation" names (though doesn't necessarily describe) that social problem. I don't see how containing a tautology disqualifies that answer.
Jerry Friedman said,
June 22, 2011 @ 11:10 pm
@MYL and Chad Nilep: I'll admit I didn't reread the test question carefully, and that the problem with the student's answer isn't the tautology. But I still think the answer doesn't reveal the awareness that we would hope for.
The "conditions that this 1954 decision [with the 1955 one, I gather] was designed to correct" are not just segregation in schools. They also—crucially, as I understand it—include unequal treatment based on race. This is explicit in the quotation from the decision. As far as I know, the court didn't rule that segregation in itself needed to be corrected; it ruled that separate schools were not equal and could not, in the circumstances, be made equal, and therefore were unconstitutional. And based on my knowledge of history, the court supported this argument with sociological studies of the condition of African Americans in all parts of life, not just education. So even without "Be as specific as possible", I think what students are supposed to be aware of is that the court ruled that "separate but equal" is not equal and designed its ruling to correct the inequality.
Also, I think that could easily be what Dr. Ravitch thinks students should be aware of. I see nothing in the paragraph that says a correct answer needs to mention segregation but doesn't need to mention inequality. Of course, that's the least of the article's problems.
ann said,
June 23, 2011 @ 2:57 am
In Ontario, far from using low test scores to demonstrate rigor, schools with low scores are subjected to intensive intervention in order to improve results. High scores are used to boast of 'a system working well'.
The Ontario Ministry of Education identifies specific areas of weakness through analysis of standardized test results. A major weakness turned out to be composing appropriate answers to comprehension questions. Like the 'separate but equal — segragation' student, students had a reasonable idea of what was required but omitted supporting detail that would develop their ideas into answers appropriate to the task.
Now, in an incremental programme from K to grade 8, students are taught how to formulate answers. In a nutshell:
1. Figure out what is being asked and answer the question, directly. Using part of the question in the answer makes for clarity but isn't mandatory. It is amazing how seldom students directly answer a question independently. Just look at the answers quoted in the article.
2. Back up the answer with a proof. A quotation from the text often works for this.
3. Write a short explanation of how the proof backs up the answer.
4. A top level response may end with a connection between the text and either personal experience or other readings.
And so the steps are: Answer Prove Explain Connect (APEC). With direct teaching even very young students understand these basics and can formulate clear responses.
Strong students use the framework as a jumping off point for clear, precise, stylish personalized responses. Young or weak students use it, as is, to express ideas that otherwise would go undeveloped. Because students now give the markers exactly what they are looking for, test results have improved. Teaching to the test has very negative implications but in this instance what is being tested is not specific knowledge but skills and it seems reasonable to me to test those. I wish that the materials used were less along the lines of 'The Drinkin' Gourd' but I am deeply grateful that improved test results did not come at the expense of, but as a result of, good teaching. A good educational outcome directly inspired by 'the need to improve test scores' — something I didn't think I'd ever see but which is becoming the norm rather than the exception in Ontario. Perhaps Ravitch should be suggesting steps for improvement rather than over-exciting the media with statistics.
Evan said,
June 23, 2011 @ 10:49 am
somehow I suspect the test makers haven't thought to blame themselves for the 2% figure. After all, they wrote "be specific as possible". Kids these days can't follow directions.
Alex said,
June 23, 2011 @ 3:38 pm
It's very hard to design a fair general test for a nuanced and complicated subject like history. That's probably one of the reasons it's not among the subjects tested in No Child Left Behind, but of course that means that less time and money is spent of teaching it because it doesn't count for a school's performance rating. I was in one of the first cohorts required to do the NCLB testing for graduation. I was the only student out of 150 in my class who scored honors on all the sections of the test; most were struggling with proficiency. That seems likely to be a test problem to me.
Even given the obvious problems on this NEAP test, history education in the US is deeply flawed. I went to a rural Midwestern high school (class of 98), and all the social studies classes were taught by people who had really been hired as coaches. We deliberately spent lots and lots of time on the Civil War, learning play-by-play breakdowns of battles. (I can tell you all about Pickett's Charge.) Then, of course, there was no time at the end of the year to learn about anything after the end of World War II. Convenient, since none of the teachers wanted to touch Korea, Vietnam, McCarthy, Watergate, the Civil Rights movement, feminism, or anything else that might be controversial or make people who are still alive look bad. This was also how American history was handled in lower grade levels.
So I wouldn't be at all surprised if people from my high school struggled to articulate the details of Brown vs Board of Education. Not only were we not taught any post-1945 history, we also weren't taught to write. That's not tested under No Child Left Behind, either.