Space v. Time in the grammar of emojis
« previous post | next post »
Benjamin Weissman, Jan Englelen, Lena Thamsen, & Neil Cohn, "Compositional Affordances of Emoji Sequences", 12/19/2024:
Abstract: Emoji have become ubiquitous in digital communication, and while research has explored how emoji communicate meaning, relatively little work has investigated the affordances of such meaning-making processes. We here investigate the constraints of emoji by testing participant preferences for emoji combinations, comparing linearly sequenced, “language-like” emoji strings to more “picture-like” analog representations of the same two emoji. Participants deemed the picture-like combinations more comprehensible and were faster to respond to them compared to the sequential emoji strings. This suggests that while in-line sequences of emoji are on the whole interpretable, combining them in a linear, side-by-side, word-like way may be relatively unnatural for the combinatorial affordances of the graphic modality.
One of the authors posted this summary:
The basic idea is this: putting pictographs in a sequence like words in writing is unnatural compared to making a single picture, which is the more natural way pictures combine information. So, we did an experiment and indeed that's what we found!
— Neil Cohn (@neilcohn.bsky.social) December 22, 2024 at 8:34 AM
And added:
Some of these limitations to emoji combination are addressed by Emoji Kitchen created by @jenniferdaniel.bsky.social, and maybe the new AI emoji, but emoji also have constraints in their sequences that will always hamper an "emoji grammar" cognitiveresearchjournal.springeropen.com/articles/10….
— Neil Cohn (@neilcohn.bsky.social) December 22, 2024 at 8:34 AM
The reference there is to Neil Cohn, Jan Engelen, & Joost Schilperoord, "The grammar of emoji? Constraints on communicative pictorial sequencing", 2019:
Abstract: Emoji have become a prominent part of interactive digital communication. Here, we ask the questions: does a grammatical system govern the way people use emoji; and how do emoji interact with the grammar of written text? We conducted two experiments that asked participants to have a digital conversation with each other using only emoji (Experiment 1) or to substitute at least one emoji for a word in the sentences (Experiment 2). First, we found that the emoji-only utterances of participants remained at simplistic levels of patterning, primarily appearing as one-unit utterances (as formulaic expressions or responsive emotions) or as linear sequencing (for example, repeating the same emoji or providing an unordered list of semantically related emoji). Emoji playing grammatical roles (i.e., ‘parts-of-speech’) were minimal, and showed little consistency in ‘word order’. Second, emoji were substituted more for nouns and adjectives than verbs, while also typically conveying nonredundant information to the sentences. These findings suggest that, while emoji may follow tendencies in their interactions with grammatical structure in multimodal text-emoji productions, they lack grammatical structure on their own.
And he also added:
Because pictographs like emoji are limited in space and sequencing, I don't think they will evolve into a more complex visual language, and these ideas go back to a BBC article I wrote almost 10(!) years ago on "Will emoji become a new language?" www.bbc.com/future/artic…
— Neil Cohn (@neilcohn.bsky.social) December 22, 2024 at 8:34 AM
And finally:
Does this mean that you can't have a language of pictures? No! Much of my research has argued that we *already* use visual languages in the natural systems people use to draw, but they aren't artificial systems where pictures mimic writing www.visuallanguagelab.com
— Neil Cohn (@neilcohn.bsky.social) December 22, 2024 at 8:34 AM
This discussion is at least adjacent to the literature in (psycho-)linguistic theory about "linearization", which deals in various ways with various aspects of how more abstract information (conceptual, discursive, interactional, semantic, syntactic, morphological, phonological, …) gets mapped onto serially-ordered articulatory and acoustic sequences. Descriptive and theoretical differences aside, it's clear that the need for (various types of) linearization has been a crucial constraint on the (biological and cultural) evolution of language — and it's equally clear that (still or sequential) images are affected by a very different set of time/space constraints. [Note that this is completely different from the uses of "linear" in math, engineering, and statistics…]
See also "How the first letter was written", 12/5/2012 — and also this old xkcd cartoon on "Communication"…
Philip Taylor said,
December 23, 2024 @ 6:51 am
"We here investigate the constraints of emoji by testing participant preferences for emoji combinations" — I wonder whether the set of participants included individuals such as myself who loathe and detest emoji, and who curse the Unicode consortium for ever allowing emoji to gain admission to the Unicode standard.
Mark Liberman said,
December 23, 2024 @ 7:17 am
@Philip Taylor:"I wonder whether the set of participants included individuals such as myself who loathe and detest emoji":
Quick answer: No. The experiment's participants were younger and less cranky. From the Methodology section of the cited paper:
A total of 156 participants completed the experiment. Participants were required to be fluent, though not necessarily native, speakers of English, as assessed via self-report. Participants provided informed written consent, asapproved by the Tilburg University Ethics Review Board. Three participants demonstrated patterns of consistent inattention, two of whom repeatedly responded in under 300 ms and one of whom repeatedly responded in over 20 seconds. These three participants were removed from the dataset;the final set thus includes responses from 153 participants (average age = 29.3 (SD = 8.85); 62 male, 85 female, 6 other).
Alongside a basic demographic questionnaire, participants completed an Emoji Language Fluency questionnaire created to gauge each individual’s experience and familiarity with emoji in general. Emoji Language Fluency did not correlate significantly with accuracy or response times in the experiment, nor did any of its questions; thesurvey itself appears in the supplementary materials. This questionnaire does indicate that participants were overall fairly familiar with emoji (average self-rated “emoji expertise” score = 4.8 (SD = 1.06) on a 1-7 scale).
Others: please discuss the content of the post rather than Philip's anti-modern peeves.
Jerry Packard said,
December 23, 2024 @ 8:33 am
I found it very difficult to grasp the gist of the claims given the summaries presented here. I guess I’ll have to do a closer reading maybe including the original articles.
Mark Liberman said,
December 23, 2024 @ 8:52 am
@Jerry Packard: "I found it very difficult to grasp the gist of the claims given the summaries presented here."
I agree. But one simple (and unsurprising) take-away is that pictures convey (some kinds of) interactions much more effectively when the various images are integrated into a whole scene, rather than being separated into some sequential order.
Speech is different, in that the only effective way to conceptually integrate different lexemes is to use the methods of morpho-syntax to express them in a structured temporal order.
David L said,
December 23, 2024 @ 9:43 am
It would have been more impressive if they had published their results in emoji-only form.
Mark Liberman said,
December 23, 2024 @ 10:52 am
@David L: "It would have been more impressive if they had published their results in emoji-only form.":
If I understand their work, one of the implications is that an emoji-only form of their paper would not be possible — unless of course someone did the stupid thing of extending the emoji list into a full logographic writing system.
For the possibilities and limitations in that direction, see e.g. the Emoji Bible from nearly a decade ago:
The project used a program of the translator’s own creation that linked 80 emoji with 200 corresponding words