Textual narcissism, replication 2
« previous post | next post »
Yesterday, I tried replicating one of the experiments in Jean M. Twenge et al., "Increases in Individualistic Words and Phrases in American Books, 1960–2008", PLoS One 7/10/2012, and got results that seem to be significantly at variance with their conclusions ("Textual narcissism", 7/13/2012).
This morning, I thought I'd try getting a replication with word counts from a different source of historical data. I used the Corpus of Historical American English (Mark Davies, The Corpus of Historical American English: 400 million words, 1810-2009., 2010). Some of the problems with the Google Books source are removed here: the COHA collection is balanced by genre, and a detailed list of its 107,000 sources is available.
And the results remain hard to square with Twenge et al.'s main conclusion, which they expressed like this:
This study demonstrates that language use in books reflects increasing individualism in the U.S. since 1960. Language use in books reflects the larger cultural ethos, and that ethos has been increasingly characterized by a focus on the self and uniqueness.
On the contrary, the time-series of changes in the frequency of the words in their "individualistic" and "communal" lists suggests a very different narrative, one that sees an increasing focus on the group through the first half of the 20th century, and little change since then.
Here's the decade-by-decade of sum of word counts from COHA, for Twenge et al.'s "communal" and "individualistic" lists. The "communal" word counts are the blue line with the 'C' plotting characters; and the "individualistic" word counts are the red line with the 'I' plotting character:
It's perhaps clearer to look at decade-by-decade changes in the ratio between the summed counts of the two sets. The last five points, plotted in red, are the decades from 1960 to 2009 — it's really hard for me to see how to make this support an argument that "the larger cultural ethos […] has been increasingly characterized by a focus on the self and uniqueness".
The COHA counts that I used are here and here, and the R code generating the plots is here.
For comparison, the same type of plots using decade-by-decade data from the Google Book American English collection are below, as presented in yesterday's post (where you can find the data and code). First the historical word counts:
And the ratios, where again we see an apparent rise in "communal" words through the middle of the 20th century, and not a lot of change since then. Certainly there's not much of a trend in the last five points, representing the decades since 1960:
None of this fixes problems that some may see in the basic method. But there's not a lot of point in debating the methodology, if the resulting numbers don't support the conclusion in any event.
Meanwhile, there's been surprisingly little uptake in the mass media for the amiable hypothesis that kids today are self-obsessed etc. There's Sharon Jayson, "What's on Americans' mind? Increasingly, 'me'", USA Today 7/10/2012 — illustrated with the image and caption below:
And also Andrea Johnson, "Kids are hearing more 'me first' than 'united we stand' in the books they read", Minot Daily News 7/10/2012; Tom Jacobs, "Books Increasingly Show It's All About Me", Pacific Standard 7/11/2012.
But so far, neither David Brooks nor the Daily Mail has taken the bait.
suz said,
July 14, 2012 @ 9:12 am
Likewise, I cannot replicate the results in searches in the BROWN and FROWN corpora. Elements from both fields are in general increasing (most individual plus the total increases are, however, not statistically significant). Some words are declining, through, in both areas.
Another point that I'll raise in my own blogpost (tomorrow, probably) is that looking at these words in isolation may lead them nowhere. In other words, mere number crunching seems to mask how the words are used. (Apart from the fact that the elicitation methods seem questionable and that salience and frequency in language are two different things.)
Family, for instance, increases considerably, but is much more often used with negative prosody in the 2000s as opposed to the 1960s. Doesn't this lead to the paradox that they could argue for their hypothesis with both results, increase and decrease, making their hypothesis potentially non-falsifiable?
Rosie Redfield said,
July 14, 2012 @ 1:46 pm
I've put a comment on the Twenge et al. paper's PLoS ONE site, saying that attempts at replication are being discussed here.
M B Donnellan said,
July 22, 2012 @ 5:27 pm
I found all of your analyses by accident when a science writer contacted me to comment on the Twenge et al PLoS One paper. I declined but directed her/him to contact Mark. I am not an expert in this topic – just someone who published a few papers critical of Twenge’s other work.
I hope this post publication review material makes into the article written by the journalist.
These additional word count analyses are compelling and cast considerable doubt on the sturdiness of the original paper. I hope Mark will try to publish these in a psychology journal. The problem is that Twenge et al. will keep citing their own work and these failures to replicate may not reach a wider audience. For example, the DeWall et al. lyrics paper has some potential flaws but few in Psychology seem to know or care.
Indeed, I tried to replicate the DeWall et al. lyrics paper and I ran into several problems. It was very hard in terms of getting the song lyrics reliably coded by undergraduates. Dealing with slang was a nightmare in terms of getting consistency. We didn’t originally include song titles in the LIWC analyses but I found out DeWall did. I shelved the project but your posts on that paper made me think about resurrecting the critique (I just discovered those as well – it was good to see convergence).
A few other possible problems with DeWall that might catch you interest.
1. DeWall sent me his genre codings and several seemed invalid. Examples: Captain and Tennille’s “Do That To Me One More Time” was HipHop as was Whitney Houston’s “How Will I Know” but Rick Astley’s “Never Gonna Give You Up” was Pop. Bon Jovi’s “Living on a Prayer” was Other but his “Blaze of Glory” was Rock. I asked about their coding and I got a semi-lame response from his grad student: “The song genres were coded by research assistants as the song lyrics were retrieved. Inter-rater reliability for the ratings was good (ICC = .71).”
2. The fact that the figures are predicted values from regression models was confirmed by his grad student. This is completely deceptive reporting in my book. A big clue was that the antisocial composite are all negative predicted values. (Never mind that actual violent crime in the United States declined over their interval from 1989 to 2007 using both the UCR and the NCVS – facts get in the way of a good story about the US culture.)
3. Their song list is probably wrong for about 30 or so songs as best as I can tell. I first had my undergrads use Wikipedia to get the Year-End Hot 100 Singles (e.g., http://en.wikipedia.org/wiki/Billboard_Year-End_Hot_100_singles_of_1998). Our song list didn’t completely match theirs once I saw the genre codes. DeWall said he purchased a subscription to Billboard. I felt like he was blowing off my concerns so I went to my university library to search the microfilm copies of the end of the year Billboard lists to deal with the discrepancies. Our list was correct if you go by the Hot 100 Singles List published in the paper copies of Billboard. For example, they included “My Heart Will Go” by Celine Dion in 1998 but that song charted at number 13. They included “Iris” by the Goo Goo Dolls in 1998 but that song wasn’t in the Top 10. DeWall didn’t respond when I sent him the pdf of the 1998 list. What is kind of worrisome is that the most narcissistic song in their entire database is Lovefool by the Cardigans (1997) but that song didn’t make the Top 10 Hot 100 singles (http://en.wikipedia.org/wiki/Billboard_Year-End_Hot_100_singles_of_1997) based on my search! It had 24.57% I words if I can recall.
So thanks for doing all of the hard work on this line of research. I hope more people in psychology take notice.
Jean Twenge said,
August 5, 2012 @ 1:42 pm
Thanks for an interesting discussion. Without conducting regression analyses, however, this (and the previous blog post) cannot be considered a replication attempt. The main point of the PLoS ONE paper was that individualistic language increases when *controlled* for the increase in communal language in a regression equation.
We report in the paper that both communal and individualistic language increased. The crucial question is whether individualistic language increased when the common variance shared with communal language was accounted for.
In the paper, we discuss why both types of language increased. The most likely reason is a recent sample generated and rated the words and phrases, and they are most likely to think of modern language. For example, two of the communal words were "team" and "teamwork" — very modern ways to think about groups. The main point is that the individualistic language increased even when communal language was controlled in a regression equation.
We could have looked at only individualistic language, but that would have created the possibility that it increased because the sample thought of modern language. So this was actually a fairly conservative test of the hypothesis.
Yastreblyansky said,
May 21, 2013 @ 9:57 pm
David Brooks (http://www.nytimes.com/2013/05/21/opinion/brooks-what-our-words-tell-us.html?hp&_r=0) finally went there. What a delight to find that Language Log had already been by and dismissed the study without difficulty.
I suppose the answer to Twenge's objection here is that while a regression analysis might be a good way of determining the significance of any findings, myl's proportions graph shows plainly that the finding is just not there, significant or otherwise.
Novel Data: Promise and Perils | MADE IN AMERICA said,
June 18, 2013 @ 10:38 pm
[…] and self-absorption have been on the rise since 1960 (see here and critiques here and here). Engal shows that the words I and he were about equally frequent in American books until about the […]
Novel data: promise and perils « The Berkeley Blog said,
June 20, 2013 @ 1:17 pm
[…] and self-absorption have been on the rise since 1960 (see here and critiques here and here). Engal shows that the words I and he were about equally frequent in American books until about the […]