Don't know much about history psychometrics
« previous post | next post »
Sam Dillon, "U.S. Students Remain Poor at History, Tests Show", NYT 6/14/2011
American students are less proficient in their nation’s history than in any other subject, according to results of a nationwide test released on Tuesday, with most fourth graders unable to say why Abraham Lincoln was an important figure and few high school seniors able to identify China as the North Korean ally that fought American troops during the Korean War.
Over all, 20 percent of fourth graders, 17 percent of eighth graders and 12 percent of high school seniors demonstrated proficiency on the exam, the National Assessment of Educational Progress. Federal officials said they were encouraged by a slight increase in eighth-grade scores since the last history test, in 2006. But even those gains offered little to celebrate because, for example, fewer than a third of eighth graders could answer even a “seemingly easy question” asking them to identify an important advantage American forces had over the British during the Revolution, the government’s statement on the results said.
Sam Wineberg, "Can even educators answer these lame questions?", HNN 6/19/2011:
“Ignorance of U.S. History Shown” was the New York Times headline. Less than a quarter of American students were able to name two contributions to U.S. history by Abraham Lincoln and Thomas Jefferson.
A report about last week’s NAEP findings? Try again.
What about this? “Test Shows Knowledge of American History Limited.”
Neither headline was written about the latest NAEP. For the first, we’d have to dust off a test given to college students by the New York Times in 1943. For the second, we’d have to reach back to 1976, our bicentennial year, when the Times ran another survey with Bernard Bailyn's help. Bailyn threw up his hands in disbelief that more students thought the Puritans “guaranteed religious freedom” than not. “I don’t know how to explain it,” he said.
With all due respect to the acclaimed Harvard historian, I do.
Indeed, my powers exceed explanation. I can predict the future. In March 2004, seven years before the release of the last week’s NAEP findings, I wrote the following in the pages of the Journal of American History: “When the next national assessment rolls around in 2010 do not hold your breath for the headline announcing ‘U.S. Children Score Well on the 100 Most Basic Facts of American History.’ The architecture of modern psychometrics ensures that that will never happen—no matter how good a job we do in the classroom.”
Let me come clean. Here's two tips for predicting the future about students’ historical knowledge.
First, know a bit about modern test construction. No matter what the NAEP spokespeople tell us about the NAEP being “criterion referenced”—or tied to an absolute standard rather than fitted to a bell curve—multiple choice items work by tricking test-takers with “distracters.” An item is “bad” if almost everyone gets it right. So, if during the piloting testing of NAEP, it is determined that most twelfth-graders can identify “The Star-Spangled Banner,” Rosa Parks, the dropping of the bomb on Hiroshima, slavery as a main cause of the Civil War, the purpose of Auschwitz, and Harriet Tubman, these items are all thrown out because they fail to “discriminate” among students.
And I myself have the psychic power of predicting how the results will be spun — by the media, by the education establishment, by the domain experts, and by the politicians. The media's take is predicted by the bias towards "sensationalism, conflict, and laziness", as Jon Stewart put it in a recent interview with Chris Wallace. The education professionals will see things in terms of the urgent need for more of their guidance in reforming the educational system. The domain experts will point out that their vital field needs more emphasis and more funding.
As for the politicians, they'll blame the sad effects of others' mistaken or even evil agendas, and the failure to give enough power to their own faction. A lovely recent example by Rick Santorum:
"America's a very young country. Our memory is very short. And as a result, we- we don't know our- my goodness, we don’t even know our own history. There was a report that just came out last week that the worst subject of children in- in American schools is — not math and science — it's history. It’s the worst subject. How can we be a free people, how can we be a people that fight for America, if we don’t know who America is or what we’re all about? This is a-, ((in)) my opinion, a conscious effort on the part of the left, who has a huge influence on our curriculum, to desensitize America to what American values are, so they're more pliable to the new values that they would like to impose on America."
Of course, any of these self-interested reactions might happen to be correct in a particular case, since a stopped clock sometimes tells the correct time. But for a more complete analysis of the response to earlier test result, in which the stopped clocks were not very accurate, see "Freedom of speech: More famous than Bart Simpson", 3/3/2006.
And if you want to dig into the most recent NAEP U.S. History results in order to evaluate the rhetoric of the various responses, you can start here. Right up front you'll find the first unused possible headline: "Student performance improves at all grades since 1994". Download the full report, and you'll find striking numbers and graphs like this one:
Or this one:
Of course, in order to join the (extensionally non-existent) celebration of these encouraging trends, you'd need to accept that the NAEP's psychometricians have succeeded in their goal of making the tests "criterion referenced" in a way that yields numbers that are comparable across the decades. Without looking further into the details, I don't know whether this is a plausible belief or not. My goal here is just to echo Prof. Wineberg's observation that the responses to the NAEP 2010 U.S. History test results have been spun, essentially by everyone involved, in a predictably negative direction.
[I should note that the History News Network deserves kudos for publishing not only Sam Wineberg's evaluation, but Paul Burke's "Wrong 'Correct' Answers: The Scourge of the NAEP", 6/21/2011:
The National Assessment of Educational Progress (NAEP) has three big flaws in measuring history knowledge. First, students get neither a grade nor any result from the test, so student effort is entirely voluntary. Second, NAEP seeks proficiency on so many topics it encourages shallowness, not depth. Third, NAEP marks wrong many of the correct answers. […]
In addition to giving some telling examples of question-design and scoring problems, Burke notes the serious problem that these tests share with the infamous Collegiate Learning Assessment:
For students these tests do not count towards a grade, or even a star, so while students have to show up, they have no incentive to give their best answers, or indeed to make any effort whatsoever. The tests are given in late winter, often a time of low energy and motivation in school anyway. The tests are one more intrusion in the most dismal part of the school year. It would not be surprising for students to put little energy in the answers, and even to give wrong answers as a protest.
If you're looking for something to bewail, forget about American schoolchildren's knowledge of U.S. history — consider instead the fact that in order to get a rational assessment of the NAEP U.S. History results, you need to go to some relatively obscure new-media website rather than to elite mainstream outlets such as the New York Times or the Wall Street Journal.]
Chris Brew said,
June 22, 2011 @ 8:53 am
Is HNN really a cable channel's website? Looks to me like a particularly well-organized collection of resources backed by a center at GMU's history department.
[(myl) You're right, this is confusing, and I may be confused. HNN's "about us" page indicates that the organization is based in Seattle, Washington (whereas George Mason University is in Virginia); and states that "History News Network (HNN) operates independently of George Mason University. The views expressed are those of its authors and editors and not GMU or the Center for History and New Media."
But the same page also identifies the organization as "George Mason University's History News Network (which is popularly known as HNN)" and states that "The [HNN] website resides on GMU's server".
Meanwhile, the HNN front page (and other basic pages) features a prominently centered link to history.com, which is indeed the cable channel formerly known as the History Channel. I assumed, apparently falsely, that this was a sort of organizational umbrella, in which the History News Network was in some sense an activity of the History Channel, but perhaps it's just commercial sponsorship or perhaps even plain old advertising space…
Anyhow, HNN is definitely some kind of New Media, and this is yet another case where New Media is more careful and more reliable than Old Media; which was my main point. So I've revised the post to reflect this.]
Jimbino said,
June 22, 2011 @ 8:57 am
I'd like to see how Europeans or Chinese score on these same tests.
richard howland-bolton said,
June 22, 2011 @ 9:14 am
@ Jimbino
Do you think they'd know much more about US history? :-)
KevinM said,
June 22, 2011 @ 1:43 pm
Trouble with Chinese history is that there's so much more of it…
greg said,
June 22, 2011 @ 3:02 pm
KevinM – not necessarily. we just don't learn much about the development of the tribal divisions and cultures and rise and fall of their localized dominance at any point in school unless a person is specializing in anthropology of the North American continent.
Mark P said,
June 22, 2011 @ 4:51 pm
"America's a very young country"
What is with this? It's not true.
Most European countries are younger than the US (Germany, Italy, Latvia, Lithuania, etc). Most Asian countries are too.
If we want countries with no previous alternative nation states, then Australia and New Zealand show what young countries look like.
What, as an outsider, I most notice about the US is how much it cares about it's history. All sorts of arguments are phrased around what people 200 years ago did. Very few other countries do that, concentrating instead on how to make their laws reflect modern situations. No English legislator will argue that a law should be retained because George IV thought it was a good idea!
Mark Mandel said,
June 22, 2011 @ 8:48 pm
Mark P "All sorts of arguments are phrased around what people 200 years ago did. Very few other countries do that, concentrating instead on how to make their laws reflect modern situations. No English legislator will argue that a law should be retained because George IV thought it was a good idea!"
Might that be because most European nations did not have a founding event, a historically decisive mo(ve)ment such as our Revolution, Declaration, and Constitution? Our "Founding Fathers" were all contemporaries, though of different ages. Nothing similar, I think, can be said of France or England or Spain. Maybe Turkey…
Marty Perlman said,
June 22, 2011 @ 10:20 pm
Yes, every couple of years, a similar study comes out referencing our lack of historical knowledge and the dire state of the Republic. But are the tests accurate? Well, it's time for the definitive exam to see just where we stand on the Plain of History.
This mini-quiz is designed to quickly assess your historical bearings.
1. True or False: During the American Revolution, one third of the colonists were pro-rebellion, one third were loyal to the Crown and one third were playing whist and waiting to see which way the winds of war would blow.
The quiz continues at the whimsical Thinking Out Loud,
http://marperl.blogspot.com/2011/06/american-history-mini-quiz.html
Jerry Friedman said,
June 23, 2011 @ 12:38 am
I saw a cartoon today, in a newspaper a friend was reading, about the "new low" in students' knowledge of history. (Can't find it on line.) I wonder whether that's the cartoonist's fault or some journalist's.
Although there are valid criticisms of the tests and the reporting, I wonder about the supposed low motivation in late winter. Is there really evidence that motivation is lower at that season than others?
Michael Schiffer said,
June 23, 2011 @ 12:15 pm
@Marty Perlman The answer to the first question of the quiz is given as "true". But while those proportions are often given, I've never seen a cite, and those too-convenient numbers (equal thirds, really?) never struck me as plausible. How do you win a war against a side with all the structural advantages (government already in place, backed by a standing army and major navy) without numerical superiority?
For what it's worth, in A Companion to the American Revolution, Robert M. Calhoon (a historian at UNC) gives "historians' best estimates" (albeit without cites) at 15-20% Loyalist and somewhere between 40% and "a bare majority" Patriot. (Though he also claims "approximately half the colonists of European ancestry tried to avoid involvement in the struggle", which doesn't add up with the other numbers.) An article (admittedly now pretty old) that tries to do a rough statistical estimate of Loyalist numbers at http://www.jstor.org/stable/1919095?seq=11 likewise gives about 20%. (If anyone has pointers to more recent research, I'd appreciate it.)
Those numbers are still large (and they weren't evenly distributed– some colonies were much more Loyalist than others), but are more consonant with the end result than an even division.
Dakota said,
June 23, 2011 @ 11:53 pm
"Distracters", huh. We used to call these "trick questions". It seems there is a polite word for every evil practice.
Alas, none of my coursework covered the mystical art of psychometrics, but thanks to the above links, at least now I can go to my administrator (and I'm still not sure if it's wise to do this) and question why students are being given "norm-referenced assessments" periodically throughout the semester, rather than "criteria-based tests".
Jerry Friedman said,
June 24, 2011 @ 5:32 pm
@Dakota: I believe distracter is just a technical term for the wrong options on a multiple-choice test. See for instance this guide.
Whether multiple-choice questions are an evil practice is another matter.
Dakota said,
June 25, 2011 @ 12:47 pm
@Jerry Friedman, yes, that makes more sense, thank you. I was conflating the distractors with "An item is 'bad' if almost everyone gets it right." A multiple-guess test is actually pretty low on the Bloom’s Taxonomy scheme of things since it only involves recognizing the correct answer, not remembering it or making inferences based on it. Although it is more susceptible to cheating, it is actually more fair for low-level learners, assuming the writers of the test aren't marking correct answers as incorrect or doing the old "choose the best correct answer" game. In which case, what you are really testing is test-taking skills. A pity there isn't more accessible about test writing theory. For those of us who have to explain someone else's tests to the students.