Speaking (in)coherently
« previous post | next post »
Yesterday, LizardBreath at Unfogged made an excellent point in response to my recent post about Sarah Palin's (in)coherence ("I Think There's A Problem With the Methodology Here", 11/19/2008):
If even the clearest speakers' speech often looks incoherent when transcribed, then this argument establishes that no one can ever be validly criticized as an unusually incoherent speaker. And that can't possibly be right — some people do sound clear and logical when they talk, and other people sound error-ridden and confused.
I agree with her. It would be nice to be able to tag speakers with some sort of Standard Coherence Index — and I'm sure we'd find that there are large individual differences. But we need a better methodology than relying on journalists or bloggers to pick illustrative examples in a polemical context, especially if the examples are quoted from transcripts.
Among the reasons:
1) Speakers are variable. Each of us is sometimes clearer and more logical, and sometimes more error-ridden and confused. Anyone can be made to look better or worse, depending on what examples are chosen.
2) Writing is not speech. Many sorts of disfluencies (false starts, self-corrections, filled or unfilled pauses) can be spoken in ways that make them nearly transparent to the listener. They can even be an aid to understanding in some cases — but they can also be very disruptive. The same is true for parenthetical asides and other meaningful interpolations. We don't even have good ways to quantify the amount of disfluency and complexity, much less the quality of its performance and its impact on listeners.
3) Transcripts are unreliable. They're often created or edited in ways that make them easier to read, but they can also be produced in ways that make them harder. And some of the rhetorical techniques that are effective in speaking are hard or impossible to render effectively in a transcript, where we're missing the effects of pitch contour, speaking rate, voice quality, and so on.
4) Listeners are also variable and unreliable. The same passage may strike one person as entirely lucid and another as so incoherent as to be hardly human. This effect is especially strong for people, groups, ideas, or ways of talking that elicit strong negative emotions: a hated celebrity, pronunciations or usages associated with a despised group, conceptual pet peeves, and so on. We've discussed many examples over the years, but perhaps none has been clearer than the Plain English Campaign's 2003 Foot in Mouth award to Donald Rumsfeld for his "unknown unknowns" remark.
5) Coherence is complicated. Whether in speech or in writing, there are lots of levels on which a passage can cohere or fall apart. Does the underlying argument make sense? Are its parts and their relationship phrased clearly? Are there too many qualifications and explanatory interpolations, or not enough? Is the performance fluent? Do listeners find the disfluencies helpful or harmful or neutral?
Independent of any political application, it would be nice to have an intersubjectively valid way of evaluating the coherence of spoken discourse in terms or dimensions like these, or (more plausibly) in terms of useful proxies for them. Meanwhile, it seems to me to be entirely reasonable to try to analyze someone's speaking style in a more informal mode, as ShadowFox does in this comment. I'm not sure whether that analysis is correct, but it seems sensible and to some extent testable.
LizardBreath again:
Doesn't it seem more persuasive to assume that there is a difference between competent, coherent speech and stumbly, confused speech, but that looking at transcripts, rather than listening to the speech as produced, is a poor way of distinguishing the two?
Yes. But this is only part of it. Our reaction to" the speech as produced" may be part of a cycle of confirmation bias, where we decide that person X is characteristically Y, and so we notice when X is Y, but not when X isn't Y, or when others are or aren't Y. And journalists or bloggers may be guiding us into this cycle, by converging on the "X is Y" narrative, and plying us with example after example to illustrate it.
LizardBreath's conclusion:
Now, this doesn't mean that any particular criticism of a politician as a bad speaker is justified. But you can't point to a bad-looking transcript from a good speaker and use it to prove that there's no such thing as a poor speaker.
Agreed. My point was only that a single passage from a (mispunctuated) transcript is not good evidence that someone is a characteristically incoherent speaker, and we should be especially wary of accepting it as reliable confirmation of a widely-held belief to that effect.
N said,
November 20, 2008 @ 2:19 pm
We should also add to this that listeners/comprehenders are bad at extracting information from even the most basic passages. Not just incoherence, but complexity itself can lead to perceived incoherence.
blahedo said,
November 20, 2008 @ 2:37 pm
I wonder if you could gauge coherence a bit better—ability to think on one's feet, at least—by comparing the "transcript" that was the prepared version of remarks to the "transcript" that is an actual transcription (minus transparent disfluency)? I was recently re-reading the commencement speech Obama gave here at Knox a few years ago, which is interesting because the Obama website version appears to be the prepared one, and the Knox website version has what he actually said. (I had a hand in that; the Knox website originally had the prepared version, but I really feel that's not a transcript then, so I complained.)
What is remarkable when you scan the differences is that you can see how he (usually) seamlessly diverged from the prepared speech and performed any necessary repairs on the fly. In a few cases, there was a word change that may have just been a misspeak (listening to the audio helps confirm this, but you can often tell from the words themselves) but rather than repairing the misspeak directly, he continues the sentence with suitable modifications so that the whole thing coheres.
It's still not going to be perfect, of course, and it doesn't measure coherence in extemporaneous context. But if you're looking to assemble some sort of Standard Coherence Index that's actually deterministic and measurable from text, one of these before-and-after jobs might be the way to go.
Sili said,
November 20, 2008 @ 3:48 pm
Hmmm – the BBC had to dub member of Sin Fein for years because of some peculiarity of the law. While those speeches might not be appropriate for the experiment, they do point the way.
By that I mean that it should be possible to get actors to reliably reproduce interviews/extemporal speeches verbatim – false start, verbal ticks and all – and thus remove the bias towards the original speaker.
That would also allow for 'cleaning up' the transcripts to judge whether it's the contents or the delivery that turns people off.
Rubrick said,
November 20, 2008 @ 4:49 pm
It's clear to me that playing actual recordings of speech to a large sample of listeners is the way to go; transcripts of any kind introduce all kinds of pollution. The first-order idea would be to have the listeners simply rate the speech on a fluency scale. This of course runs into problems of confirmation bias when the speakers are famous and easily identified. (Of course, for real research it shouldn't be necessary to use famous speakers; but to settle the current argument, it's hard to get around.)
A way around this— and an improvement over an "impression rating", I think, in any case— would be to present the listener with a series of multiple-choice questions concerning whether a certain point was actually made in the speech. I would expect more fluent speech to produce less variance in the responses.
Nathan Myers said,
November 20, 2008 @ 5:31 pm
As an aside, what struck me most about the Rumsfeld quote was not what he said, but, perhaps appropriately given the initial topic, what he omitted: things you think you know that are not, in fact, true. The omission seems revealing, and even symbolic of the entire, eight-year administration. We have a word for interest in what might not be so: doubt, and we have other words for failure to exercise it.
On topic, asking a variety of listeners to summarize what they heard might provide an objective measure of coherency. If they all agree on what a speaker said, the speaker made herself clear. If no two people come away with the same impression, that's failure.
Some vocal performances are not meant to communicate anything repeatable, but to inspire or to evoke something personal in the listener ("change! hope!"), but those seem easy enough to identify without experiment.
Teresa G said,
November 20, 2008 @ 10:38 pm
I have often thought that one measure of coherence is how easy it is to translate a speech in real time (as in the sort of task a UN interpreter does every day). Note that a person skilled in simultaneous translation is able to lag as much as a sentence or more behind the speaker, waiting for the sense of the speech to be clear before provding a translation. Would such a translator require more (or less?) lag time if presented with less coherent speech?
So I would suggest as an objective test of coherence that one could look at:
a) the necessary "lag time" that a skilled interpreter requires for a given speech (it is possible that this would correlate with complexity as well as or instead of coherence, but it is at least a measurable quantity to evaluate).
b) how coherent the simultaneously translated speech is judged by native speakers of the target language. This should abstract away from questions of grammaticality to questions of the intended meaning.
c) one could even try something similar with translations prepared over time rather than in the moment. I honestly suspect that if I were asked to translate one of Palin's more famous interview responses into, say, ASL, that I would find it quite difficult to do, even if given plenty of time. Whereas I believe it would be much easier for me to translate something judged more coherent, such as her RNC speech. And crucially, I think the translations of each would be judged by signers as more incoherent in the first case and more coherent in the second, and that this should hold up even if the translator were not as biased as myself.
dr pepper said,
November 21, 2008 @ 12:44 am
One of my favorite programs is "In Our Time", a radio talk show in which the host brings together a few experts, semi experts, and well read lay people, to hold an informal discussion of some topic of interest. To give an idea of the scope, here is the topic list from 2002:
2002-02-14 Anatomy – 2000 years of anatomical study
2002-02-21 The Celts – what were the Celts in Britain really like?
2002-02-28 Virtue – is it derived from reason?
2002-03-07 John Milton – poet or politician?
2002-03-14 Budhhism – why has it captured the spirit of our age?
2002-03-21 Marriage – its various forms and the role of the State
2002-03-28 The Artist – a special kind of human being?
2002-04-04 ET – new life within our solar system
2002-04-11 Bohemia – what did it mean to be Bohemian?
2002-04-25 Tolstoy – the influence of the Russian Novel
2002-05-02 Schrodinger's Cat – Quantum Mechanics
2002-05-09 The Examined Life – is an unexamined life worth living?
2002-05-16 Chaos Theory – is the universe chaotic or orderly?
2002-05-23 History of drugs – their role in medicine and the arts
2002-05-30 The Grand Tour – what drove this desire for travel?
2002-06-06 The Soul – the key to our individuality as humans?
2002-06-13 The American West – was it an "experiment of liberty"?
2002-06-20 Richard Wagner – his influence on the German spirit
2002-06-27 Cultural Imperialism – should we try to prevent it?
2002-07-04 Freedom – a principle worth fighting and dying for?
2002-07-11 Psychoanalysis – do people crave dictatorship?
2002-10-17 Slavery and empire – were Britons also captives?
2002-10-24 The scientist in history – missionary or monster?
2002-10-31 Architecture and power – imagery of imperialism
2002-11-07 Human Nature – innate or nurtured?
2002-11-14 Victorian Realism – how real?
2002-11-21 Cordoba and Muslim Spain – a culture of tolerance?
2002-11-28 Imagination – just what is it?
2002-12-05 The Scottish Enlightenment – how enlightened?
2002-12-12 Disease – the fight against diseases and plagues
2002-12-19 The Calendar – a history of the Calendar
You can download recent broadcasts from BBC4, and collections are also available. There is no editting, you hear exactly what people said, And it's full of false starts, uhs and ers, stuttering, and occasional memory failures. Nonetheless, the discussions are extremely coherent and comprehensible. It's rather how i imagine chatter in the Members' Club at Language Log Plaza must be like.
Assistant Village Idiot said,
November 22, 2008 @ 10:31 pm
Rumsfeld's phrasing comes from concepts used in Risk Evaluation. In particular, they are used in military planning. Nicholas Nassim Taleb, author of Fooled By Randomness and The Black Swan, has recently been studying this four-quadrant problem of knowns and unknowns.
As for Sarah Palin, her accent was so off-putting to me for very bad cultural reasons that I resolved early to avoid video of her, as I was sure it would unfairly prejudice me against whatever she was saying. I suspect many of her political opponents had the same reaction but did not submit themselves to that discipline. I have a similar reaction, though less pronounced, to Obama's voice.