IELTS: The test that sets the standard?
« previous post | next post »
Here's a case that I'm hoping will turn out to be an epic example of journalistic misunderstanding. Because the alternative is that the International English Language Testing System is a really, really bad way to measure English language proficiency. And that would be a shame, because IELTS, a product of University of Cambridge ESOL Examinations and the British Council, is pretty much the standard English proficiency test outside the U.S.
Commenting on this morning's post "Language tests for immigrants in Canada", IS pointed us to a .pdf of a "sample test" on the web site of The Toronto Star. It includes four questions — and I bet you can't get more than half of them right without peeking at the answers. I certainly couldn't. Here's the test, minus the answer key:
Let's try to correct these sentences one at a time. The first one is
1. Tomoko spends an equal amount of money on rent, food, study materials and entertainment.
I thought of several ways to "correct" this sentence, which is ambiguous but false under all available interpretations. One alternative among many would be:
1a. Tomoko spends more money on rent and food than she does on study materials and entertainment.
But according to the answer key, the (only?) right answer is:
1b. Tomoko spends an unequal amount of money on rent, food, study materials and entertainment.
I'm not sure that this is even a grammatical English sentence. I could say "Tomoko spends unequal amounts of money on A, B, C and D", but "… spends an unequal amount of money on A, B, C(,) and D" just seems wrong to me, with or without the extra comma. And at best, the "corrected" sentence strikes me as awkward and uninformative.
I got the second sentence right. It simply gives the wrong percentage for "study materials", and so the obvious thing to do is to substitute 25 for 15:
2. Tomoko spends 45 per cent of her money on rent and food, but she only spends 15 25 per cent of her money on study materials.
However, some people might prefer to move only so that it's adjacent to its focus: "… but she spends only 25 per cent of her money on study materials". Apparently this would be marked as a wrong answer, even though some prescriptive authorities would mandate the change.
I also got the third sentence right, though it was a near thing:
3. Tomoko spends more less on clothes than she does on study materials.
I was tempted to leave more and invert the order of the categories compared. And again, apparently that would have been wrong — though in making that correction, I'd demonstrate that I understand the text and the pie chart, and can compose an English sentence to express my understanding.
4. Tomoko spends as much on rent and food as she does on everything else put together.
I got this one wrong. I decided to add "almost" before "as much". But the right answer, apparently, is to change "as much" to "less". [As a commenter notes below, the necessary accompanying change of "as" to "than" is omitted from the answer key.] I like my edit better, although it would apparently have been marked wrong.
So my score was 50%. How'd *you* do?
Getting back to the reason that I'm hoping for misleading journalism, it would be nice if the cited answers were simply examples of possible correct answers, rather than the only answers accepted as correct. Then the fact that the Toronto Star presented the answers as shown below would be rather misleading, but the questions and answers could still be part of a plausible evaluation of English proficiency.
I'm hoping that someone who knows how the IELTS is graded — maybe even someone who works as an IELTS grader — will tell me that I would have scored 100%, not 50%. But I'd need some favorable odds to be willing to bet money on that outcome.
Quicksand said,
July 29, 2010 @ 8:25 pm
I, for one, hope that it's a multiple choice exam, and that the Toronto Star simply left out the other (clearly incorrect) answer options to save space.
Evan Harper said,
July 29, 2010 @ 8:38 pm
I would wager that a substantial proportion of native English speakers who are perfectly capable of using the language in everyday situations would nonetheless fail this test, even if any reasonable "correction" were scored as a full mark — simply because they don't know how to read a graph.
army1987 said,
July 29, 2010 @ 8:51 pm
I would have corrected 2, 3 and 4 the same way as you, but I'd have corrected 1 by replacing "spends" with "does not spend", which comes closer to the "expected" answer.
Chris said,
July 29, 2010 @ 8:56 pm
Even accepting any 'reasonable' answer, the task itself strikes me as a bit ambiguous. What exactly does it mean to 'correct' a sentence like that? For the first one, could I say "Tomoko spends an equal amount of money on rent and clothes"? "Tomoko spends most of her money on entertainment, clothes and study materials"? "Tomoko spends the most money on rent and food"?
Identifying the 'correct' answer from the set of 'true' answers seems more like a test of test-taking acumen than of English ability. If it's meant to be multiple-choice, that might be more reasonable, though the line "see if you can correct them" suggests to me that it isn't.
Katherine said,
July 29, 2010 @ 8:59 pm
Actually I first thought of the correct answer to 1, thought it sounded awkward, but was probably what they wanted. I wonder if there was something accompanying it asking you to only change one word in the sentence.
John said,
July 29, 2010 @ 9:08 pm
In fact, this is not from an IELTS exam, but one of the first exercises from the writing section of Insight into IELTS, which I've used to teach IELTS exam prep classes. The unit, if I remember rightly, is part of the introduction to Writing Task 1 which is about the understanding and interpretation of data. The actual IELTS exam doesn't have any questions like this.
I can't remember how I've dealt with this particular exercise in the past, but I probably used my own judgement and allowed a range of answers so long as they've been a reflection of the data as it is. So yes, you're IELTS 9.
Marcio Baraco said,
July 29, 2010 @ 9:10 pm
From a fairly proficient but non-native speaker of English, i'll say the questions seem fairly easy if you take them as test questions, as in don't try to actually practice the English language but instead just "do what they want" and speak some sort of awkward "i'm learning, i'm learning" English to-be.
alice said,
July 29, 2010 @ 9:22 pm
The IELTS website gives samples too.
Both the "academic" and the "general training" writing tests consist of two writing tasks, the first requiring at least 150 words, the second 250.
Task 1 in the academic version does involve interpreting and re-presenting information from a graph, but there is no "correct" way of framing that information. In the sample of the general version, which the Canadians will surely be using for immigration purposes, there are no graphs or diagrams – you are asked to write a letter of complaint to a housing rental agency.
Task 2 in both samples asks you to produce an argument in response to a proposition.
Freddy Hill said,
July 29, 2010 @ 9:24 pm
Not being a native English speaker I got all of them right. I interpreted the last sentence of the instructions as "See if you can correct them making the smallest number of changes possible."
Maybe it's because I'm more familiar with the wicked minds of ESL test writers than the natives.
Freddy Hill said,
July 29, 2010 @ 9:26 pm
Or is it, making the smallest possible number of changes? See, getting 100% right does not mean nothin'.
Tim Leonard said,
July 29, 2010 @ 9:33 pm
Almost all 18-year-olds have sufficient *language skills* to function as productive members of society. In fact, almost all 14-year-olds do. So unless the vast majority of 14-year-olds can pass the test, it's demanding more than is necessary to function productively. (I concede that "proficient" may intentionally be a higher barrier.)
Clayton Burns said,
July 29, 2010 @ 9:49 pm
In discussions with federal civil servants in Canada I have suggested that as soon as someone becomes a candidate to immigrate to Canada, that person should begin intensive English and/or French, with materials donated if necessary. Even if potential business immigrants worked on the third season of Mad Men (all 13 episodes), it would be something.
Thanks to Mark for his sharp analysis. Too many people in Canada are commenting on this without handling the IELTS materials. If you do so, you will begin to understand that the implicit ideas about language behind IELTS are wrong.
Over the past 20 years, the power to focus on good sentences has exploded with the corpus revolution. With IELTS, it is as if we are back in the 1950s: if you hold IELTS sentences in your hands and get a palpable sense of their heft, you often find they are weightless. They just sound like dead sentences.
At John Wells's excellent blog, there has been some discussion of connected text as the basis of phonetic analysis. Cambridge should construct a 40-lyric Internet database with full descriptions of the sound system(s) of English, including sound symbolism, for testing.
The strength in English teaching comes from minute and imaginative focus at each level of the language: sound, word, sentence, discourse. Human brain plasticity will perform integration. If you separate out the levels in IELTS, what you see is mediocrity everywhere. What is there to provide sustenance?
English professors from New York do not want IELTS imposed on them. But are they equally concerned that it not be forced on others? It is a deep irony that such professors have turned in the other direction as this massive bogus testing industry has been unfolding. Now that they are getting bitten by it naturally they do not like it.
CBK said,
July 29, 2010 @ 9:56 pm
The Toronto Star gives the source for the “test item” as “Cambridge IELTS Course.” This does not appear to be the test company “IELTS.” In particular, I found the “test item” on page 68 of “Insight Into IELTS” by Vanessa Jakeman and Clare McDowell which is posted here. I did not find any claims in the book that its items were released IELTS items.
So, it seems possible that the Toronto Star item was never on an IELTS exam and is not a product of the IELTS test company.
[(myl) Then for the Star to present it as a "Sample Test" which is "an example of Part 1 of the writing test" is indeed a striking example of misleading journalism. I guess I got my wish, consistent with my general rule of thumb: "When in doubt, blame the journalist".]
Dave M said,
July 29, 2010 @ 10:21 pm
Surely 4*: "Tomoko spends less on rent and food as she does on everything else put together" can't be what they want.
Clayton Burns said,
July 29, 2010 @ 10:30 pm
Mark, I would want to look into this. It appears that just below we have Vanessa as a Cambridge author of one of the very expensive audio CDs.
In theory, someone could have ripped off the Cambridge logo, but I would look carefully:
Insight into IELTS Audio CD: The Cambridge IELTS Course [Audiobook] [Audio CD] Vanessa Jakeman (Author)…
In fact, I have just looked, and Vanessa Jakeman is the author of the Cambridge University Press "Insight into IELTS" (2001).
David Fried said,
July 29, 2010 @ 10:42 pm
I'm surprised that no one so far has commented on the problem that the first example apparently refers to four separate categories of expense, whereas the graph lumps two of the categories, "rent" and "food," into one. I found myself puzzling over the possibility that Tomoko's spending on rent and food, considered separately, might equal her expenditures on either study materials or entertainment, considered separately. The sentence can't be "corrected" without redefining the categories either as Mark did or in some other way. What an awful question, and how characteristic of test writers!
The Ridger said,
July 29, 2010 @ 10:58 pm
Hmmm. I corrected them to "Tomoko does not spend".
Jer said,
July 29, 2010 @ 11:24 pm
Do you have much experience with the IELTS? The parts I have looked at were dismal. I helped a friend who was working through them, and was horrified by the content. I can't remember the name of the module, but it was something to do with business writing. The examples they gave were so bad that I had to point out that if she sent an email like that today, she would be teased by her co-workers.
The IELTS examples weren't even wrong. They floundered around, used odd or outdated style, and didn't even use common business jargon that a native speaker would choose. I recall being given composition classes in how to write better (I realise the peril of using that phrase around here), and IELTS looked nothing like that. It appears we are giving immigrants different lessons to the ones we give ourselves. That can't be healthy.
I also helped her grammar on a few essays, and when she wasn't trying to write IELTS style, she was pretty good. IELTS actually seemed to be reducing her writing ability.
[self: native english speaker, friend: chinese]
Ingrid Piller said,
July 29, 2010 @ 11:30 pm
Welcome to the world of the so-called non-native speaker, Mark!
I'm constantly amazed how different and isolated from each other the language-segregated worlds inhabited by L1 and L2 users are. In the way that White people don't need to confront White privilege, native speakers rarely confront native-speaker-privilege. Being free to ignore the language testing industry certainly is a huge privilege – so, I'm glad you've taken it up. Good on you! :-)
Jer said,
July 29, 2010 @ 11:30 pm
Eeek, I should clarify that I don't know for sure that the examples I saw were genuine IELTS. They could have been thrid-party training materials, although they did have the IELTS logo stamped all over them. The main point still stands, that IELTS students appear to be learning a dialect spoken nowhere on the planet…
unekdoud said,
July 29, 2010 @ 11:34 pm
@Dave M: Very true, even though I wouldn't have realized it myself.
I just went ahead and assumed that the requirement for being correct was the smallest number of changes. If that's true, then the test favors "lazy" students slightly. Even so, most of the corrected sentences do not express information in the way laymen would expect. For instance, in a pie chart scenario, it is usually the larger portions which are more important, which is why many of us prefer to correct the sentences with the word "more" rather than "less". (Just my theory.)
Grammatical correctness doesn't seem to be explicitly tested, so perhaps they're lenient about the sentence structure shown in 1b?
dw said,
July 30, 2010 @ 12:33 am
When I was preparing for the US Citizenship test, one of the sample questions (given by the US government agency administering the test) was along the lines of "From which country did the United States declare independence". The "correct" answer given was England.
The person writing these questions obviously hadn't actually read the Declaration of Indepdence, which says that "all political connection between [the United States] and the State of Great Britain, is and ought to be totally dissolved".
When I took the test in person, I answered "Great Britain". No one objected, so "England" evidently wasn't the only answer that was accepted. Well, I'm a citizen now anyway :)
(You can see the old sample questions here". This particular error seems to have been corrected now.)
Ben said,
July 30, 2010 @ 12:38 am
Surely a sentence pattern should go:
less… than….
not
less… as….
How is that correct?
Steve said,
July 30, 2010 @ 12:58 am
That is not the real test, but a practice task for classroom use, just to get students talking about pie charts and how to interpret them. A range of vocabulary and 'chunks' could be elicited through this task, and put on the board. In the actual test, the candidates are not expected to correct sentences, but write their own 150 word interpretation of a graph, bar chart or pie diagram. As someone else has pointed out, there are two types of IELTS test, academic and general, and this is a practice task from the academic paper, which immigrants would (should?) not be expected to take.
J. Goard said,
July 30, 2010 @ 1:05 am
@dw:
Were they really exactly like this?
Can you name thirteen original states?
Seems semantically anomalous without the definite article, since thirteen original states would be exhaustive of the set.
Name the right guaranteed by the first amendment.
Opposite problem. There's only one?
Name one benefit of being citizen of the United States.
To become American citizen is great honor, Comrade.
What holiday was celebrated for the first time by the Americans colonists?
Hmm…
owltoucan said,
July 30, 2010 @ 1:33 am
I worked as an IELTS grader for several years, finishing relatively recently, and I have never seen a question or answers like that on an IELTS test before.
Sybil said,
July 30, 2010 @ 2:33 am
Good heavens, I hope this is an aberration due to a testing prep service (as has been suggested above) and not representative of the actual test.
That having been said, I took a civil service exam about 20 years ago (no, I won't tell you which one!) and found an ambiguous question, in the sense that more than one of its multiple-choice answers could be considered to be correct. The person administering the exam would hear none of it, though. Had I not found employment during the couple of years I was waiting to hear from them, I might have brought action. As it was…
UK lawyer said,
July 30, 2010 @ 2:50 am
I got 2,3 and 4 right. I got 1 right at first, but like another contributor I found this too clumsy so changed it to "different" then thought that "amount" needed to become plural, so over-complicated it.
Like another contributor, I was distracted by the "rent, food" wording – just poor drafting/editing.
Bob said,
July 30, 2010 @ 2:52 am
There are two things that need to be considered – whether or not the example is from an IELTS test and what the IELTS test is actually used for. It's clear from the previous posts that this is not from an actual test but rather from a book which is commonly used when preparing for the test. This book is used by most colleges, my own included, as part of a range of material demonstrating techniques for interpretting and discussing data. It isn't, in my experience, typical of a real IELTS test where the writing is considerably more extensive than simply "correcting" mistakes in sentences.
What I haven't seen discussed is what IELTS is for. In the UK, IELTS is a test that is mainly taken by students who wish to get a good level of academic English in preparation for entry to a University course in any discipline but taught in English. This is a completely different way of reading to, say, general English when reading a novel or a newspaper. The level of analytical ability needed is far higher and this, rather than points about phrasing, is the main purpose. Students have to be able to demonstrate their understanding of graphs, pie charts and other presentation techniques by discussing and analysing the data.
My college for example runs two courses at this level. One leads to an IELTS qualification and is intended for students going on to Higher Education. The other is a general English course intended for everyday use and lacks the academic components so that this kind of reading isn't included.
Alan Palmer said,
July 30, 2010 @ 6:53 am
I don't understand how the interpretation of pie charts can be used in a test of English. Surely, while they can be useful in classroom discussions, they are part of statistics, not English.
Tom Saylor said,
July 30, 2010 @ 7:13 am
I note that the approved correction to sentence 4 yields an ungrammatical sentence:
Tomoko spends less money on rent and food as she does on everything else put together.
If "as much" is changed to "less," then the second "as" has to be changed to "than."
To David Fried, who said
"What an awful question, and how characteristic of test writers!"
I think that's unfair to test writers. You won't find anything like this on tests produced by reputable test development companies in the U.S. (like ETS), though you will occasionally find such in practice materials for those tests that are published by unaffiliated test-prep companies. It's clear in many cases that those who create "practice tests" for companies like Kaplan and Princeton Review have not very carefully studied the "real tests" they've used as models.
Jacob said,
July 30, 2010 @ 7:40 am
@Alan Palmer IELTS is a test for non-native speakers hoping to pursue secondary or post-secondary education in UK/Aus/Canada. It is not a "general English" test.
And as many others have already commented, there is no section on the IELTS that would require test takers to rewrite sentences in such a way.
tarako said,
July 30, 2010 @ 7:40 am
Tomoko spends unequal amounts of money on rent plus food, study materials, and entertainment, respectively.
Dierk said,
July 30, 2010 @ 7:44 am
@Alan Palmer It could be – successfully – argued that language comprehension is more than knowing the basics of grammar and spelling, like, say, actually understanding it. This test is not about the statistics or the graphical representation but about the natural language description; the graphics only give the facts to compare the description to – independent of the first language of the testee.
Nick Lamb said,
July 30, 2010 @ 8:01 am
"Aside from the obvious schism between the abolitionists and the anti-abolitionists, there were economic factors, …"
"Wait, wait… just say slavery"
"Slavery it is, sir"
It seems that like the citizenship test the IELTS is marked by trained administrators who are themselves fluent. So answers that show you are in fact fluent are likely to get full or near-full marks. The abuse in newspaper reports of "model" answers as "correct" answers with the implication that all other answers are wrong is one of the recurring injustices of test design. It penalises any test which can't be reduced to a machine-marked multiple choice form.
Also tests tend to work best on a limited ability range. They may appear broken if your ability is far outside the range they're supposed to test. The test must be made with certain assumptions, and being far more (or far less) able than expected will tend to violate those assumptions. Also, being contrary and trying to find ways to fail (most test takers are trying to pass) will tend to violate the assumptions.
SeanH said,
July 30, 2010 @ 9:03 am
I'm not sure that this is even a grammatical English sentence. I could say "Tomoko spends unequal amounts of money on A, B, C and D", but "… spends an unequal amount of money on A, B, C(,) and D" just seems wrong to me, with or without the extra comma.
This reminds me of some dialogues of Plato, in which speakers refer to something as "equal" or "unequal" without adding "to [whatever]". This seems to have been actual philosophical confusion; Plato represents Zeno in the Parmenides as finding a serious paradox in the apparent fact that one and the same object can be like and unlike (that is, an apple is like an orange, but unlike a cat).
James O'Hearn said,
July 30, 2010 @ 9:18 am
I'm surprised that so many readers of a blog such as this would have so little familiarity with the IELTS exam. Then again, you can't blame people for being fooled by the Toronto Star's (usual) inability to properly understand and present information on just about any given topic.
While a chart like the one shown above could easily be on an IELTS exam as a part of the task 1 writing, the task itself would never include grammar oriented questions like the ones shown. The real task is simply to explain and interpret the data on the given chart (or graph) in an essay of 150 words.
While some may question what a task like this has to do with English, the point of task 1 writing is to see how well the test taker can think, reason, interpret, and communicate an academic topic in an academic manner, in a foreign language (English). That is, the aim of this task is to determine how well a test taker will be able to function in English with university level topics, in a university environment. That's why this specific task is only found on the Academic version of the IELTS exam. The task 1 writing on the General Training exam usually involves writing a letter or an email to express a complaint or an opinion.
Those who want to emigrate to an English speaking country to live and work would usually be required to take the General Training IELTS exam, which focuses on the type of English encountered in everyday life. Those wishing to study in an English speaking country take the Academic exam, which focuses on high level language use, critical thinking, and reasoning in English.
If it is true that the Academic IELTS test is being administered to immigrants to Canada, then the reasoning may be that those wishing to immigrate to Canada based on merit (as opposed to family class immigration) often have to have achieved a certain level of education and professional experience. This is because (for better or for worse) Canada generally prefers to admit doctors (professionals) rather than plumbers (tradesmen).
The beauty of the Academic IELTS exam, to my mind, is that it not only separates native speakers from non-native speakers, but it also highlights how much (or how little) understanding native speakers have of their own language. While a band 9 would indicate a native or near native level grasp of the language, the truth is that most of the people I know, born and raised in Canada and the United States, would probably struggle to achieve a band 7 or 7.5 on the Academic IELTS exam.
John Cowan said,
July 30, 2010 @ 9:31 am
SeanH: Your examples are reminiscent of the Chinese white-horse paradox, where 白馬非馬 'white horse [is] not horse' seems to be both true and false. It is false in the sense that a white horse is obviously a horse, but it is true in the sense that the class of white horses is distinct from the class of all horses. Classical Chinese, however, did not readily make this distinction between individuals and classes.
army1987 said,
July 30, 2010 @ 11:54 am
@SeanH:
“A triangle with three equal side lengths is equilateral, one with two equal side lengths and an unequal one is isosceles and one with three unequal side lengths is scalene.” “What about one with two unequal side lengths and an equal one?”
SeanH said,
July 30, 2010 @ 1:38 pm
John Cowan: could that be described as a use/mention problem? White horse is horse, but "white horse" is not "horse"?
Mary Bull said,
July 30, 2010 @ 1:56 pm
@army1987, who said,
July 30, 2010 @ 11:54 am
'@SeanH:
“A triangle with three equal side lengths is equilateral, one with two equal side lengths and an unequal one is isosceles and one with three unequal side lengths is scalene.” “What about one with two unequal side lengths and an equal one?” '
LOL
Jarek Weckwerth said,
July 30, 2010 @ 2:14 pm
In general, I second Bob and James O'Hearn above, and others who have expressed similar opinions. However, I wouldn't be as charitable as James in expecting "native speakers" to be capable of failing only the Academic test. A considerable proportion of "native speakers" (of any language) will fail a plain language test, as is evident from all kinds of school-leaving exams etc. After all, LL has just discussed how "native speakers" can get things wrong.
What bugs me is that even plain vanilla language tests seem to test logic, or even general intelligence, rather than just language. Is it possible at all to build a language test that does not do that?
Peter Taylor said,
July 30, 2010 @ 2:18 pm
Alan Palmer wrote:
I'm surprised at the suggestion that they'd be used in classrooms, at least at university level. I suppose MBAs may be an exception; can you suggest any other subjects which deal with data which is best presented as a pie chart?
David Fried said,
July 30, 2010 @ 5:45 pm
To Tom Saylor:
You're right–I now understand that this example is not from a standardized test. If you're actually a test writer, I apologize. I know very little about standardized tests these days, apart from the "horribles" published in the papers from time to time.
I think my comment was influenced by the awful homework assignments and test questions my kids bring home from middle school and high school. When I try to help them, I constantly find myself struggling with questions that range from ambiguous to incomprehensible. The inability of many math teachers to construct coherent word problems is particularly distressing.
James O'Hearn said,
July 30, 2010 @ 6:04 pm
@Peter Taylor
Most foreign students in North America, Australia, and the UK generally avoid liberal arts subjects, and focus on science, math, computers, engineering, and business related courses of study. In any of these, the ability to take data in one form, and translate that data into an easily understandable narrative, is essential. Be it an engineering schematic, a collection of raw statistical data, or a quarterly profit/loss report, the information a student reads will usually be non-narrative in nature, which they will have to interpret and explain in plain English.
Stephen Jones said,
July 30, 2010 @ 7:21 pm
What is being tested is the ability to understand information in one form (in this case numeric) and transfer it to English words.
It's like many things on the IELTS academic test. People will fail because they couldn't do the exercise in their native language.
It's still impossilbe however to have a content free language examination.
Kapitano said,
July 31, 2010 @ 12:31 pm
I've taught IELTS, and other forms of ESL. Often I've had to teach that real speakers speak in this way, but to pass the exam you need to use this form, which hasn't been used in real life for 50 years.
Frequently I've found that the most proficient learners did badly in the exams, while those who could chain together cliched chunks but not conduct a conversation did well. Being a native speaker, I scored a level 4 when I took the papers myself.
No, the language taught in IELTS classes bears as much relation to English as Newspeak.