Language Log

Grammaticality Representation in ChatGPT as Compared to Linguists and Laypeople, Zhuang Qiu, Xufeng Duan & Zhenguang G. Cai, Humanities and Social Sciences Communications 12, no. 617 (May 6, 2025). 

Abstract

Large language models (LLMs) have demonstrated exceptional performance across various linguistic tasks. However, it remains uncertain whether LLMs have developed human-like fine-grained grammatical intuition. This preregistered study (link concealed to ensure anonymity) presents the first large-scale investigation of ChatGPT’s grammatical intuition, building upon a previous study that collected laypeople’s grammatical judgments on 148 linguistic phenomena that linguists judged to be grammatical, ungrammatical, or marginally grammatical (Sprouse et al., 2013). Our primary focus was to compare ChatGPT with both laypeople and linguists in the judgment of these linguistic constructions. In Experiment 1, ChatGPT assigned ratings to sentences based on a given reference sentence. Experiment 2 involved rating sentences on a 7-point scale, and Experiment 3 asked ChatGPT to choose the more grammatical sentence from a pair. Overall, our findings demonstrate convergence rates ranging from 73% to 95% between ChatGPT and linguists, with an overall point-estimate of 89%. Significant correlations were also found between ChatGPT and laypeople across all tasks, though the correlation strength varied by task. We attribute these results to the psychometric nature of the judgment tasks and the differences in language processing styles between humans and LLMs.

Jianwei Xun, the supposed philosopher behind the hypnocracy theory, does not exist and is a product of artificial intelligence
A collaboration between an essayist and two AI platforms produced a book that reflects on new forms of manipulation

Raúl Limón, EL PAÍS (4/7/25)

The entire proposition behind this scheme is so preposterous and diabolical that I am rendered virtually speechless.

The French city of Cannes hosted a roundtable discussion on February 14 called “The Metamorphosis of Democracy – How Artificial Intelligence is Disrupting Digital Governance and Redefining Our Policy.”

The debate was covered in an article by EL PAÍS after Gianluca Misuraca, Vice President of Technology Diplomacy at Inspiring Futures, introduced the concept of “hypnocracy” — a new form of manipulation outlined in a book by Jianwei Xun called Hypnocracy: Trump, Musk, and the New Architecture of Reality. However, this Hong Kong philosopher does not exist, as revealed by Sabina Minardi, editor-in-chief of the Italian magazine L’Espresso.

Posted by MIT Linguistics:

Sad and momentous news has reached us of the passing of alum and former faculty member John R. Ross (PhD 1967) at the age of 87. Known to one and all as "Haj Ross", his dissertation and subsequent papers on syntax and related topics laid the groundwork for many — one might even say "most" — of the core research topics under central investigation today.

While attending an international conference on the application of AI to the study of the Silk Road and its history, at which most of the papers were delivered in Korean, I was struck by the frequent occurrence of one distinctive word:  hajiman.  For some speakers, it almost seemed like a kǒutóuchán 口頭禪 ("catchphrase").  I had no idea what it meant, but its frequency led me to believe that it must be some sort of function word.  However, the fact that it is three syllables long militated against such a conclusion.  Also its sentence / phrase final position (though not always) made me think that it wasn't just a simple function word.

I kept trying to extract hajiman's purpose / meaning from its position and intonation (usually not emphasized, almost like an afterthought).

When, during coffee / tea breaks I asked some Korean colleagues about it, their reply — "Oh, hajiman" (with an offhand smile) only added to the word's mystique.

In "Linguistic relativity: snow and horses" (4/15/25), I summarized and assessed the following paper:

Temuulen Khishigsuren et al, "A computational analysis of lexical elaboration across languages", Proceedings of the National Academy of Sciences (2025). DOI: 10.1073/pnas.2417304122

My post was picked up by Cody Cottier, who was doing a critique of the Khishigsuren et al. article for Scientific American.  Cottier interviewed me and incorporated some of what I said to him in this review:

Linguists Find Proof of Sweeping Language Pattern Once Deemed a ‘Hoax’
Inuit languages really do have many words for snow, linguists found—and other languages have conceptual specialties, too, potentially revealing what a culture values
Scientific American (5/9/25)

After visiting the massive National Museum of Korea in Seoul, I was eager to go to the National Hangeul Museum nearby.  Alas, it is under renovation, so I was unable to enter it this time, but I will go back on some future occasion when I travel to Korea.  I did, however, manage to buy two facsimile versions of the Hunminjeongeum 훈민정음 / 訓民正音 ("The Correct / Proper Sounds for the Instruction of the People), a 15th-century manuscript that introduced the Korean script Hangul, one for the populace and one for the literati.

Several of the comments to this post, "How to say 'Seoul'" (5/12/25), prompted me to think some more about a problem that had perplexed me from the time I did a review of The Korean Buddhist Canon: A Descriptive Catalogue, by Lewis R. Lancaster, in collaboration with Sung-bae Park (Berkeley and Los Angeles:  University of California Press, 1979).  That was nearly half a century ago, but I still remember keenly how difficult it was to romanize the titles and the proper nouns.  The hardest part of that was dealing with what happened at syllable boundaries.  It was obvious that different authorities romanized the sounds in discrepant ways.

Nick Tursi suggested that I visit Sungsimdang in Daejeon, so I went two hours out of my way as I was travelling to Seoul. Sungsimdang (Korean: 성심당; Hanja: 聖心堂; lit. Sacred Heart Hall) is a phenomenally popular bakery that could easily establish branch stores all around Korea and, indeed, the world, but it refuses to do so, not expanding beyond the city of Daejeon.

We were lucky that it was raining that day, which made the line outside the store only stretch for one block, whereas in good weather it may stretch back and forth for a length equal to three blocks or more, and you'd have to wait for 2-3 hours to make your way through it.

(photo courtesy of Song Yaoxue)

On March 27, horrificgoth posted on tumblr

(crawls on all fours with blood drenched on me) I have to do arts and crafts

resulting in 56,876 notes so far. One of them, posted Saturday 5/10 by Seebs, was

i’m more mad about this than i might otherwise be because someone pointed out the “-ench” suffix in English a while back:

drink -> drench

cling -> clench

we used to have a form for “to cause-to” on verbs. and yes, there was apparently a q verb for fire-going-out that led to “quench”.

sadly, people refuse to acknowledge my other example:

wink -> wench

So far as I know, most Americans pronounce the name of the capital city of the Republic of Korea as "soul".

(Received Pronunciation) IPA(key): /səʊl/
(General American) enPR: sōl, IPA(key): /soʊl/ 
 
Rhymes: -əʊl
Homophones: sole, soul, sowl

From Korean 서울 (Seoul, literally “capital city”), originally from Claude-Charles Dallet's French-based romanization of Korean, reinforced by the 1959 South Korean Ministry of Education romanization of Korean, which transcribed the Korean vowel ㅓ (/⁠ʌ⁠/) with the digraph "eo" and which was official until 1984.

Note that English Seoul predates the Revised Romanization romanization of Seoul. The two romanization systems simply produce identical forms.

(Wiktionary)

I recall that when I began learning Mandarin, one of the things (!) that troubled me greatly was why the word for "thing" was written with the characters for "east" and "west":  dōngxi 東西.  My classmates came up with all sorts of outlandish, speculative explanations for the supposed etymology.  All along, I suspected that the meaning "thing" for the disyllabic word dōngxi 東西 was not derived from the characters used to write it but was the phonetic reflection of a borrowing or the representation of some colloquial, topolectal term.

From Mok Ling:

A friend of mine, Lucy, is in a Mandarin learning group. She told me about the bizarre etymology she was taught for the word dōngxi 東西. Apparently, 東西 being used to mean "thing, item" is based on the conception of the Five Phases (wǔxíng 五行 [VHM:  formerly translated as Five Agents or Five Elements, which brings out the correspondences with the Four Elements of Western classical thought, also in the metaphysics of Indian, Tibetan, and other cultures]): East is represented by the element of Wood (木) and West is represented by the element of Metal (金). Objects are made of metal and wood, therefore "east-west" became a shorthand "thing" — obviously pretty ridiculous.

Prof. Qiu Xigui 裘錫圭 has passed away today at the age of 89. His groundbreaking work on ancient Chinese scripts has significantly shaped the study of Chinese palaeography and grapholinguistics. A monumental loss for academia. https://t.co/Jn0aLn19kI pic.twitter.com/OmHxTgAqbz

— Maxim Persikov (Panates) (@gyankotsu) May 8, 2025

[Update — mistaken identity corrected…] Someone with the same name as Robert Prevost, now Pope Leo XIV, published a paper in 1985 evaluating the application of Bayes' Theorem to the question of God's existence.  The paper ("Swinburne, Mackie and Bayes' Theorem" ) was published in the International journal for philosophy of religion.

 Thomas Bayes (1701-1761) was a Presbyterian minister, but the theorem that bears his name was presented in a posthumously-published work on gambling, "An Essay Towards Solving a Problem in the Doctrine of Chances".  The Economist once called Bayes' Theorem "the most important equation in the history of mathematics", but Rev. Prevost's paper argued that "the Bayesian method of evaluating the adequacy of theistic explanation … [falls] short both in practice and in principle".

From r/chatgpt:

I tried the "Create the exact replica of this image, don't change a thing" 101 times, but with Dwayne Johnson
byu/Foreign_Builder_2238 inChatGPT