Statistical cognition in the media

« previous post | next post »

Bianca Nogrady, "Music preferences reveal your inner thoughts", ABC Science 7/23/2015:

There is a clear link between people's cognitive styles and the type and depth of emotion they prefer in music, say researchers.

Their work, published today in PLOS ONE, shows people who are more empathetic — have a greater ability to identify, predict and respond to the emotions of others — are drawn to more mellow, sad, poetic and sensual music, such as R&B, adult contemporary and soft rock.

However people with more analytical tendencies (called 'systemisers') go in the opposite direction, seeking punk, heavy metal, avant garde jazz and hard rock.

Makes sense, right? But here's what the underlying data looks like. Each red x represents the relationship between one individual's "empathy quotient" (on the horizontal axis) and his or her preference for "mellow" music (on the vertical axis):

Would you say that this represents a "clear link between people's cognitive styles and the type and depth of emotion they prefer in music"?

Actually, that's some fake data that I made up to match the strongest correlation (-0.14) found in Study 1 from David M. Greenberg , Simon Baron-Cohen, David J. Stillwell, Michal Kosinski, & Peter J. Rentfrow, "Musical Preferences are Linked to Cognitive Styles", PLoS ONE 7/22/2015. Here's their Table 1:

Note. Cell entries are correlations between the Empathy Quotient (EQ) and the MUSIC music-preference dimensions. S1 = Sample 1, S2 = Sample2, S3 = Sample 3, S4 = Sample 5. S1 and S2 provided preferences rating for mixed genre excerpts; S3 provided preferences ratings for only rock excepts; and S4 provided preference ratings for only jazz excerpts. Ns = 2,178 (S1), 891 (S2), 747 (S3), 320 (S4).
*p < .05;
**p < .01

The real dataset is apparently available from mypersonality.org, but I don't expect that it would look different in any relevant way.

A correlation of -0.14 means that one variable explains -0.14^2 = 1.96% of the variance in the other variable. Call it 2%. And that was highest correlation of the 40 relationships reported in Table 1.

Their second study combined "empathizing" and "systematizing" dimensions into one "brain type" dimension, and unpacked musical types into 25 "psychological attributes". Study 2 found relationships that were stronger than those in study 1:

To investigate the extent to which preferences for specific psychological attributes in music differ by brain type, we performed analyses of variance on each of the 25 psychological attributes (standardized) using brain type as the independent variable. Results revealed a significant effect of brain type on preferences for all but three of the psychological attributes (i.e., joyful, fun, and undanceable). Of those for which there was a significant effect, effect sizes ranged from F(2, 341) = 3.68, p < .05, partial eta squared = .02 (for amusing) to F(2, 341) = 8.11, p < .001, partial eta squared = .05 (for animated).

A "partial eta squared" of 0.05 — the best out of 25 tries — means that relations between the "brain type" variable and the musical attribute of animated accounted for (i.e. predicted) 5% of the variance in preferences for different pieces of music. All of the other relationships were weaker than this. 5 percent-of-variance-accounted-for is more than twice as much as the 2%  that was the strongest relationship in study 1 — but it's still not a very strong predictor.

Although the effect sizes in both studies are small, they're in line with community standards in social psychology — thus F.D. Richard, C.F. Bond, and J.J. Stokes-Zoota ("One hundred years of social psychology quantitatively described", Review of General Psychology, 2003) present this distribution of correlations:

This article compiles results from a century of social psychological research, more than 25,000 studies of 8 million people. A large number of social psychological conclusions are listed alongside meta-analytic information about the magnitude and variability of the corresponding effects. References to 322 meta-analyses of social psychological phenomena are presented, as well as statistical effect-size summaries.

That doesn't mean that such studies are wrong or without value — marketers, like politicians, are happy to exploit effects that explain only a few percent of variance in customer behavior. Tiny improvements in click-through rates or voting behavior can mean a lot.

But the popular press is unable — or unwilling — to distinguish between "a tiny but statistically significant correlation" and "a clear link", often expressing the relationship using generic plurals. "People who are more empathetic […] are drawn to more mellow, sad, poetic and sensual music", according to that ABC Science News piece.

This is true even in the more intellectual strata of the mediasphere. Thus Olga Khazan, "The Soul of the Metallica Lover: What our music tastes say about our personalities", The Atlantic 7/29/2015:

Greenberg found that people who scored high on empathy tended to prefer music that was mellow (like soft rock and R&B), unpretentious (country and folk), and contemporary (Euro pop and electronica.) What they didn’t like, meanwhile, was “intense” music, which he classified as things like punk and heavy metal. People who scored high on systemizing, meanwhile, had just the opposite preferences—they kick back to Slayer and could do without Courtney Barnett.

Or Aimee Swartz, "Do you have a mellow brain or an intense one? Cognitive style linked to preference in pop music", Popular Science 7/22/2015:

Researchers found that people who scored high on empathy preferred what researchers categorized as “mellow” music—such as R&B, soft rock, and adult contemporary tunes—“unpretentious” music—such as country, bluegrass, and folk—and “contemporary” music—which included everything from acid jazz to Euro pop. They disliked “intense” music, such as punk, hard rock, and heavy metal.

In contrast, people who scored high on systemizing liked intense music, but disliked mellow and unpretentious musical styles.

As I've pointed out many times, this way of thinking about relationships among variables is as problematic as the Pirahã's reduction of numbers to the concepts "small size", "large size", and "collection".

For further exploration of statistical cognition among the hunter-gatherers of the media savanna, see here.



16 Comments

  1. JHH said,

    August 4, 2015 @ 8:50 am

    The claim is ridiculous from the start. Is "avant garde jazz" a label for emotions stirred up by music? No, it's a musical style fully capable of expressing/evoking a rich range of emotions.

    Personally, I like "classical music." Is that mellow, sad and poetic, or would you call it hard-hitting or "analytical"?

    Can hard-hitting music be sad? Analytical music mellow?

    [(myl) Technically, the five "MUSIC" dimensions emerged from factor analysis of subjects' reactions to a sample of "musical stimuli":

    Rentfrow, Goldberg, & Levitin [49] measured musical preferences across four independent samples by asking participants to report their preferential reactions to musical stimuli that were representative of a variety of genres and subgenres. Separately, judges rated these excerpts based on their perceptions of various sonic (e.g. instrumentation, timbre, and tempo) and psychological (e.g. joyful, sad, deep, and sophisticated) attributes in the music. Findings across all of the samples converged to suggest that a robust five-factor structure underlies musical preferences, and that each of the five dimensions are defined and differentiated by configurations of their perceived musical attributes. These dimensions (coined the MUSIC model after the first letter of each dimension label) are: Mellow (featuring romantic, relaxing, unaggressive, sad, slow, and quiet attributes; such as in the soft rock, R&B, and adult contemporary genres); Unpretentious (featuring uncomplicated, relaxing, unaggressive, soft, and acoustic attributes; such as in the country, folk, and singer/songwriter genres); Sophisticated (featuring inspiring, intelligent, complex, and dynamic attributes; such as in the classical, operatic, avant-garde, world beat, and traditional jazz genres); Intense (featuring distorted, loud, aggressive, and not relaxing, romantic, nor inspiring attributes; such as in the classic rock, punk, heavy metal, and power pop genres); and Contemporary (featuring percussive, electric, and not sad; such as in the rap, electronica, Latin, acid jazz, and Euro pop genres).

    So a particular sample of "classical music" or "avant garde jazz" could in principle be anywhere in that five-dimensional space. There are reasons to be skeptical of the reification of factor-analytic dimensions, but at least this set has been checked in a few different ways and seems reasonable stable.]

  2. ThomasH said,

    August 4, 2015 @ 9:03 am

    You did not even mention that possibility that the reported, statistically significant, relationships were found after an extensive search for different ways of categorizing empathy and musical taste which would vitiate the significance tests.

    [(myl) Yes, there's that. It's a general issue with Big Data that "data dredging" is sure to turn up some "significant" results. See e.g. here, here, here, here.

    But I don't think that happened in this case, since (a) they tested the initial hypothesis in a straightforward way (in Study 1), and then went in another fairly obvious direction in Study 2; and (b) the resulting effect sizes remain quite small, and probably could be increased by assiduous dredging of comparably-sized tables of random numbers.]

  3. bratschegirl said,

    August 4, 2015 @ 9:26 am

    What about those of us who listen to Bach and Mahler to the exclusion of all of the above? And how many words do we have for "snow?"

    [(myl) You're apparently one the rare breed of individuals who systematize their empathy. Have you tried Josquin Des Prez?]

  4. David L said,

    August 4, 2015 @ 9:40 am

    Here is a sentence from the abstract of the paper:

    Results across samples showed that empathy levels are linked to preferences even within genres and account for significant proportions of variance in preferences over and above personality traits for various music-preference dimensions.

    Your reflexive animus against journalists is pretty clear to anyone who follows this blog, but I fail to see why you scold journalists for reporting the claims made by the scientists themselves.

    [(myl) If "significant" is interpreted to have its (misleading) statistical meaning of "unlikely to have arisen by chance", rather than its ordinary-language meaning of "meaningful; noteworthy", then the quoted statement is true.

    But you're right that the scientists often write in misleading ways, and the people who write their press releases are generally much worse.

    Still, compare the relationship between journalists and politicians. It's common to complain about journalists who simply transcribe politicians' claims without checking them, and it's important to recognize that scientists are also often advocates for their claims. See e.g. Paul Krugman on the "willful obscurantism" of some economists.

    The review process for scientific publications has some safeguards against adversarial excesses, though they're relatively ineffective, as the recent concern with reproducibility indicates. Regular readers will have noted that I'm often pretty harsh in criticizing the authors of scientific publications.

    But if journalists weren't so easy to fool into promoting semi-meaningless studies, some of the motivation for Bad Science would be removed.]

  5. Jeff W said,

    August 4, 2015 @ 11:23 am

    @ Mark Liberman

    But the popular press is unable — or unwilling — to distinguish between "a tiny but statistically significant correlation" and "a clear link", often expressing the relationship using generic plurals.

    But doesn’t a correlation mean there is a clear link? If that’s the case, it seems to me that distinguishing between the two would be a sort of category error. Isn’t the problem that the effect of the link is so small—a few percentage points—that we can’t say the variable counts for a lot? So the statement “People who are more empathetic […] are drawn to more mellow, sad, poetic and sensual music” is true but insufficiently qualified by “to a very, very small degree.” Would “a clear but extremely weak link” express the findings better?

    I’m not criticizing your statement. I’m just trying to figure out if I understand the issue correctly.

    [(myl) Given the well-known problems with experimental replication, I wouldn't say that a statistically-significant correlation in a single survey constitutes demonstration of a "clear link", even in a scientific context. But a deeper problem is that most people are not thinking about such relationships in terms of the statistical properties of (samples of) groups, they're thinking in terms of group prototypes representing causally-effective essential properties of the individuals who belong to those groups. And talk about a "clear link" between X and Y, in that context, leads immediately to generic statements of the form "Xs prefer Y", or "Xs are Y", or etc.

    When we're talking about musical preferences, this doesn't do much harm, except maybe to people's confidence in what scientists have to say. But the same process is more problematic when it's applied to findings about sex or race or age or medical condition. And that happens all the time.]

  6. mike said,

    August 4, 2015 @ 11:46 am

    How is it that the word "emo" has failed to make an appearance here yet?

  7. KevinM said,

    August 4, 2015 @ 2:02 pm

    @mike: Because this is not a NY Times crossword puzzle.

  8. Jerry Friedman said,

    August 4, 2015 @ 3:30 pm

    KevinM: Does that also explain why the name "Eno" has failed to make an appearance yet?

  9. Rubrick said,

    August 4, 2015 @ 10:18 pm

    Coupled with the hyperbolizing of tiny effect sizes in this case is the rather breathtaking banality of the conclusion. I'd paraphrase the results as "To a very small degree, different kinds of people like different kinds of music." Startling!

    This is undoubtedly an unfair paraphrase — but not, I think, by much. Researchers and journalists alike seem awfully desperate….

  10. bratschegirl said,

    August 5, 2015 @ 1:25 am

    @myl: Josquin never did much for me. Gesualdo, however…

  11. richardelguru said,

    August 5, 2015 @ 5:52 am

    "Have you tried Josquin Des Prez?"
    Did you know that the high-point of Western music occurs during the Credo of his Missa L’homme Armé? (that's sexti toni, rather than super voces musicales where he's merely 'pre-Nyaaaa!-Nyaaaa!-ing' Bach).
    This explains a lot about the later history of Western music, and probably this study…
    See this.

  12. Michael said,

    August 5, 2015 @ 7:09 am

    Intelligent and responsible users of statistical significance tests know well that "significance" doesn't mean important, meaningful, useful, etc. Increasing sample size will make totally useless results statistically significant. For instance, a correlation coefficient of 0.01 (variance accounted for is 1% of 1%!) is statistically significant when N (sample size) is 27200 (using a one-tail test, i.e. declaring in advance whether the correlation is positive or negative).

    [(myl) Big Data!!! (Though really, N=27,200 is small potatoes…)

    And 1% of 1% is probably an optimistic over-estimate for the proportion of the U.S. population that counts as "intelligent and responsible users of statistical significance tests".]

  13. J. W. Brewer said,

    August 5, 2015 @ 7:20 pm

    They claim that the correlation (however faint) still exists when you separate males from females, but don't seem to have tried to factor out age or race/ethnicity, both of which are commonly thought to have some correlation with musical tastes. Maybe they didn't or couldn't collect race/ethnicity data, but they had age data (and had samples which were heavily overweighted with younger respondents, which could skew results).

    I am mildly skeptical about the five-factor model, but I think my skepticism was exacerbated by the Bad-Science/Journalism nexus as manifested in the desire of the prior researchers to use the too-clever acronym MUSIC and thus name their five dimensions to end up with one starting with M, one starting with U, etc. The fit between the cutesy and value-laden labels thus given to the dimensions and the phenomena the dimensions are trying to capture is imperfect and distracting, and I might have followed the argument better if they'd had really boring names (dimensions alpha through eta, or something like that). But note also that they acknowledge some difficulties of applying the five-dimension model consistently across genres: "A close examination of the Sophisticated jazz dimension reveals that it shares many of the same musical properties as the Intense dimension from mixed genres, and the Intense rock dimension. For example, as reported in Rentfrow et al. [50], Sophisticated jazz is highly correlated with fast tempo (r = .59), strong (r = .55), and aggressive (r = .56) features. These patterns are unique to Sophisticated jazz and are not present in the Sophisticated dimensions from mixed genres and Sophisticated rock dimensions."

    A final difficulty is that they were having people listen to 15-second bursts, not complete pieces, but in some (but not all) genres of music some particularly well-known or popular pieces may shift significantly in ways that affect the five-dimensional classification. So to take a very obvious classic-rock warhorse as an example, "Stairway to Heaven" transitions from let's say more "Mellow" to more "Intense" as it goes along and that transition is itself part of its appeal for many listeners whose "empathy quotient" is presumably remaining stable throughout the song.

  14. Zizoz said,

    August 6, 2015 @ 8:32 am

    I think the relevant meaning of "significant" here is "less than 0.05". :p

  15. Mike Maxwell said,

    August 6, 2015 @ 8:05 pm

    Bach and Des Prez I like; now I gotta go check out Mahler.

    Let's hear it for systematizing empathy!

  16. sirk oraz said,

    August 7, 2015 @ 7:32 am

    This is another pathetic attempt to define & shelf the human condition… and it perhaps holds an agenda for usage for "marketing" & other bla bla…

    Personally… Having been a musician for over 40 years, worked in various aspects of the business, hold various scholastic degrees in music, was a teacher and conversed about this topic with acclaimed artists…

    And the synopsis: This is complete HOG WASH… ridiculous…

RSS feed for comments on this post