Illusions of understanding
« previous post | next post »
Lau et al., "The extreme illusion of understanding", Journal of Experimental Psychology: General, 2022:
Though speakers and listeners monitor communication success, they systematically overestimate it. We report an extreme illusion of understanding that exists even without shared language. Native Mandarin Chinese speakers overestimated how well native English-speaking Americans understood what they said in Chinese, even when they were informed that the listeners knew no Chinese. These listeners also believed they understood the intentions of the Chinese speakers much more than they actually did. This extreme illusion impacts theories of speech monitoring and may be consequential in real-life, where miscommunication is costly.
The paper begins with a quotation attributed to George Bernard Shaw: "The single biggest problem in communication is the illusion that it has taken place." Ironically, this attribution seems to be apocryphal, though the false attribution was not invented by the authors.
In the first phase of the study, 240 native speakers of Mandarin Chinese were paired, and given 12 pragmatically ambiguous phrases (translated from an English-language source):
Each item contained an ambiguous phrase that was accompanied by four possible meanings. For example, “What have you been up to?” had four possible meanings: (a) suggesting the other person may have been unfaithful, (b) suspicious that the other person is planning a surprise for you, (c) angry that the other person is 30 minutes late, and (d) wanted to know how the other person has been recently. The speaker attempted to convey one of the four randomly assigned meanings.
Within each pair, participants were randomly assigned to the role of “speaker” or “listener.” We were only interested in how people communicated through their voice, so participants sat back-to-back to prevent communication through cues such as facial expressions. In each of the twelve rounds, speakers and listeners received an ambiguous phrase along with its four possible meanings. For the speakers, one of the meanings was flagged as the meaning they had to communicate. The speaker was instructed to say each phrase such that the listener would be able to identify the target meaning from the four options. The listeners were informed that the meanings were randomly assigned to the speakers.
Both speakers and listeners tended to overestimate the success of the verbal disambiguation:
The Chinese listeners identified the intended meanings 44% of the time which is significantly greater than chance (25%) […] Yet, both listeners and speakers overestimated the success of the communication. On average, Chinese listeners overestimated their understanding by 41 percentage points (pp), as they thought they identified the intended meanings 85% of time […] The speakers, in turn, overestimated the listeners’ success by 26pp, as they thought the listeners understood them 70% of the time […].
In the second phrase of the experiment,
We recruited 120 native English-speaking Americans as listeners. Each American listener was yoked to a Chinese speaker and was presented with an English version of the phrases and meanings. The procedure for the American listeners was identical to that of the Chinese listeners, except that they heard the speakers via audio recordings.
A similar overestimation of understanding persisted:
Next, we report the most surprising finding: the illusion of understanding persists even when the listener doesn’t know the language.
[…]
On average, American listeners who did not know Chinese identified the intended meanings 35% of the time, which was better than chance (25%) […] Though American listeners were less accurate than Chinese listeners, […] they still overestimated their success by 30pp, believing that they succeeded 65% of the time […] The Chinese speakers overestimated here as well. While Chinese speakers indicated that the American listeners would understand less (50%) than the Chinese listeners (70%),[…] they still overestimated the American listeners’ understanding by 15pp […].
[h/t Bob Shackleton]
Brett said,
April 9, 2022 @ 7:17 pm
Interesting use of yoked.
AntC said,
April 9, 2022 @ 10:44 pm
This extreme illusion [of effective communication] impacts theories of speech monitoring …
Aha! Is this the elusive language faculty unique to humans? Are other species observed to go about presuming their signaling is understood by their peers?
Do we see groups of adolescent bonobos failing to all turn up at the same termite mound for a Friday night bender?
D.O. said,
April 10, 2022 @ 8:19 am
What's with "actual accuracy" counts? Why do they have a striped structure?
Gregory Kusnick said,
April 10, 2022 @ 9:19 am
The striping arises from the fact that each pair was given just 12 samples, so there are only 13 distinct percentage values (including 0%) that accuracy can take. The individual data points are then randomly offset from thos 13 values so they can be visually distinguished.
D.O. said,
April 10, 2022 @ 1:51 pm
Thanks!
Adrian Bailey said,
April 10, 2022 @ 3:46 pm
It's shoddy of any academic paper to include a misattribution.
AntC said,
April 10, 2022 @ 4:22 pm
Corporate illusions of understanding.
Jonathan Smith said,
April 10, 2022 @ 9:17 pm
"Native Mandarin Chinese speakers overestimated how well native English-speaking Americans understood what they said in Chinese, even when they were informed that the listeners knew no Chinese."
To my mind a silly clickbaity abstract statement given the substance of the paper…
Re: the actual not-extreme-after-all conclusions, in my experience, what good language learners esp. young children are brilliant at is beginning from pragmatic understanding of the kind this paper considers, whereas dull/old learners obsess over failures to understand at propositional level…
Bob Ladd said,
April 11, 2022 @ 1:49 am
@Jonathan Smith – I don't dispute what you say about the importance of picking up on the pragmatics and not worrying about the details, but as I understand the experiment, the only thing the listeners had to base their decision on was "tone of voice", speech rate, and similar intonational or paralinguistic cues. While in an actual situation those cues might contribute to a listener's pragmatic inference, this experiment focuses specifically on the interpretability of those cues alone – and seems to show, as other studies have shown, that paralinguistic cues on their own can be interpreted better than chance, but not all that much better.
Jonathan Smith said,
April 11, 2022 @ 3:25 pm
@Bob Ladd yeah, as far as this experiment is concerned, "pragmatics" is far too broad a notion; I had turned my mind to real-world social interactions…
this study per se seems centrally interested not in the fact that paralinguistic cues can be interpreted better than chance but in the fact that people overestimate these chances (at least in such a contrived context)… maybe in realer situations, esp. given face-to-face interaction, it could be shown that people have rather more accurate senses of how well particular non-linguistic messages are being communicated.
GH said,
April 12, 2022 @ 2:22 am
In large part, this seems to me simply like a demonstration of how absurd people's probability estimates often are. (Similar to those statistics about the proportion of people who think they would be able to beat a gorilla or a bear in an unarmed fight.)
A fairly large proportion of listeners are saying that they are 100% confident of interpreting the meaning of a sentence in a language they do not understand, just off the tone, even when the alternatives have fairly considerable overlap in emotional content (anger, suspicion, concern).
John Swindle said,
April 13, 2022 @ 12:25 am
All this within a single species. I imagine we also have individual estimates of how much we understand the vocalizations of cats, dogs, various kinds of birds, and so on; and they, probably, likewise, of how much they understand ours.
Gregory Kusnick said,
April 13, 2022 @ 9:51 am
I have to say I'm skeptical of the notion that my dog is forming mental estimates of the accuracy of our mutual understanding. To the extent that her responses to my utterances are governed by some combination of instinct and conditioning, "understanding" (as we understand it) may not enter into it at all on her end.
For the converse, no doubt there are pet owners with unwarranted confidence in their ability to decode their pet's vocalizations (and perhaps I'm one of them). But given that animal vocalizations lack the open-ended compositional structure of human speech, it seems at least possible to learn a pet's entire range with high accuracy.
John Swindle said,
April 15, 2022 @ 7:15 am
Not just pets. A mynah in the parking lot yelled the equivalent of "Cat!" once as I was heading for the car. I immediately and truthfully denied being a cat, but I was surprised at the slur and surprised I recognized it, since I don't speak a word of Mynese. Then I realized they weren't talking to me and there probably was a cat around.