Language Log

Registration is open for Artificial Intelligence in Language Education

Description of a General Language Model (GLM; also GLaM) project based at Tsinghua University in Beijing, but with users and collaborators around the world.

Homepage (August 4, 2022)

This prospectus is difficult for outsiders to understand because of the large number of unexplained acronyms, abbreviations, initialisms, etc. and other such participants' terminology.

GLM-130B is an open bilingual (English & Chinese) bidirectional dense model with 130 billion parameters, pre-trained using the General Language Model (GLM) algorithm¹. It is designed to support inference tasks with the 130B parameters on a single A100 (40G * 8) or V100 (32G * 8) server. As of July 3rd, 2022, GLM-130B has been trained on over 400 billion text tokens (200B each for Chinese and English) and exhibits the following unique features:

• • Bilingual: supports both English and Chinese.
  • Performance (EN): better than GPT-3 175B (+5.0%), OPT-175B (+6.5%), and BLOOM-176B (+13.0%) on LAMBADA and slightly better than GPT-3 175B (+0.9%) on MMLU.
  • Performance (CN): significantly better than ERNIE TITAN 3.0 260B on 7 zero-shot CLUE datasets (+24.26%) and 5 zero-shot FewCLUE datasets (+12.75%).
  • Fast Inference: supports fast inference on both SAT and FasterTransformer (up to 2.5X faster) with a single A100 server.
  • Reproducibility: all results (>30 tasks) can be easily reproduced with open-sourced code and model checkpoints.
  • Cross-Platform: supports training and inference on NVIDIA, Hygon DCU, Ascend 910, and Sunway.

The model checkpoints of GLM-130B and code for inference are publicly available at our GitHub repo. The code for pre-training and fine-tuning as well as the research paper are coming soon.

In a masterful Smithsonian Magazine (January-February 2023) article, Chanan Tigay documents:

How an Unorthodox Scholar Uses Technology to Expose Biblical Forgeries:  Deciphering ancient texts with modern tools, Michael Langlois challenges what we know about the Dead Sea Scrolls

This engrossing account is so rich that I can only touch on a few of the highlights.  It's about a would-be, and to some extent still is, rock musician — looking like the bassist from Def Leppard — named Michael Langlois.  But, at 46, "he is also perhaps the most versatile—and unorthodox—biblical scholar of his generation."

What makes Langlois so special?  Reading through Tigay's article, it is his relentless quest to get to the bottom of puzzles posed by tiny details of the Dead Scrolls, and his creativity in devising unconventional tools and approaches for doing so.

A #StableDiffusion model trained on images of Japanese Kanji characters came up with “Fake Kanji” for novel concepts like Skyscraper, Pikachu, Elon Musk, Deep Learning, YouTube, Gundam, Singularity, etc.

They kind of make sense. Not bad! https://t.co/ibegk4XszN pic.twitter.com/qkNcA9AAWb

— hardmaru (@hardmaru) January 6, 2023

[This is a guest post by Bill Benzon]

Everybody's talking about it.

"Meta has developed an AI translator for a primarily-spoken language

It only translates between Hokkien and English for now, but offers potential for thousands of languages without official written systems."

By Amanda Yeo, Mashable (October 20, 2022)

If true, this technology could be an enormous boon for illiterates everywhere.  It also has important theoretical and linguistic implications.

New article by Nyri Bakkalian in Unseen Japan (9/17/22):

"New App Promises Greater Convenience in Reading Old Japanese Cursive:

Kuzushiji, the 'crushed letters' found in historical Japanese documents, have long been the bane of scholars. A new app may change all that."

The author bemoans:

During my graduate education in Japanese history, interpreting handwritten primary source material from the 19th century and earlier was one of my greatest challenges. Typeset historic documents exist, especially in my period of focus during the Bakumatsu-Meiji transition. But the further back in time one’s research focus is situated, the rarer these documents become. There is a plethora of handwritten documents, written in historic cursive, but learning how to read them is a significant investment of time and resources beyond the means of most people who might otherwise have the inclination to learn.

"Google Translate learns 24 new languages"
Isaac Caswell, Google blog (5/11/22)

==========

[This is a guest post by Jenny Chu]

It came up today when I was reading this somewhat viral post on Facebook. 

Now it's getting interesting:

"China’s internet police losing man-versus-machine duel on social media"

Stephen Chen, SCMP (11/14/21)

    Hordes of bot accounts using clever dodging tactics are causing burnout among human censors, police investigative paper finds
    Authorities may respond by raising a counter-army of automated accounts or even an AI-driven public opinion leader

There may well be a dogma out there stating that all languages are equally complex, but I don't believe it, especially not if it has to be "drummed" into our minds.  I have learned many languages.  Some of them are exceedingly hard (because of their complexity) and some of them are relatively easy (because they are comparatively simple).  I have often said that Mandarin is the easiest language I ever learned to speak, but the hardest to read and write in characters (though very easy in Romanization).  And remember these posts:

"Difficult languages and easy languages" (3/4/17)

"Difficult languages and easy languages, part 2" (5/28/19)

With this post, we are already acquainted with Inspur's Yuan 1.0, "one of the most advanced deep learning language models that can generate coherent Chinese texts."  Now, with the present article, we will delve more deeply into the potentials and pitfalls of Inspur's deep learning language model:

"Inspur unveils GPT-3 equivalent for Chinese language", by Wei Sheng, TechNode (1026/21)

The model is trained with 245.7 billion parameters—the number of weights in an artificial neural network, according to the company. This is more than the Elon Musk-backed GPT-3 language model for English, which has 175 billion parameters. Inspur said the Yuan model was trained with 5 terabytes of datasets.

…

New article in EnterpriseAI (October 21, 2021):

"Language Model Training Gets Another Player: Inspur AI Research Unveils Yuan 1.0",  by Todd R. Weiss

From Pranav Mulgund: