Don't blame Google Translate
« previous post | next post »
Douglas Hofstadter has a critical article in the latest issue of The Atlantic (1/30/18):
"The Shallowness of Google Translate: The program uses state-of-the-art AI techniques, but simple tests show that it's a long way from real understanding." (1/30/18).
Hofstadter criticizes GT for not being as good as himself at translating from French, German, and Chinese into English. I will let others respond to his critique of the French and German translations, but I will comment on his critique of the Chinese to English translation.
So as not to be influenced by Hofstadter's comments on the GT translation of the Chinese and his own translation, I did my own translation first without looking at his, though it comes last here. The passage he has chosen is drawn from Wǒmen sā 我们仨 (We three), the memoir of the playwright and translator Yang Jiang (1911-2016), wife of the scholar and novelist, Qian Zhongshu (1910-1998):
Zhōngshū dào Qīnghuá gōngzuò yī nián hòu, diàorèn Máo xuǎn fānyì wěiyuánhuì de gōngzuò, zhù zài chéng lǐ, zhōumò huí xiào. Tā réng jiān guǎn yánjiūshēng.
Máo xuǎn fānyì wěiyuánhuì de lǐngdǎo shì Xú Yǒngyīng tóngzhì. Jièshào Zhōngshū zuò zhè fèn gōngzuò de shì Qīnghuá tóngxué Qiáo Guānhuá tóngzhì.
Shì dìng zhī rì, wǎnfàn hòu, yǒu yī wèi jiùyǒu tè gù huángbāochē cóng chéng lǐ gǎn lái zhùhè. Kè qù hòu, Zhōngshū huángkǒngde duì wǒ shuō:
"Tā yǐwéi wǒ yào zuò 'nán shūfáng xíngzǒu' le. Zhè jiàn shì bùshì hǎo zuò de, bù qiú yǒugōng, dàn qiú wúguò.
锺书到清华工作一年后，调任毛选翻译委员会的工作，住在城里，周末回校。 他仍兼管研究生。
毛选翻译委员会的领导是徐永煐同志。介绍锺书做这份工作的是清华同学乔冠华同志。
事定之日，晚饭后，有一位旧友特雇黄包车从城里赶来祝贺。客去后，锺书惶恐地对我说：
他以为我要做“南书房行走”了。这件事不是好做的，不求有功，但求无过。
GT regular version (from VHM):
After one year of working in Tsinghua University, Zhong Shu was transferred to Mao's translation committee to live in the city and back to school on weekends. He still holds the post of graduate student.
The leader of the Mao Selected Translation Committee is Comrade Xu Yonglian. Introducing Zhong Shu to do this job is Tsinghua classmate Qiao Guanghua.
On the appointed day, after dinner, an old friend hired a rickshaw to come from the city to congratulate. After the guests go, Zhong book said to me in fear:
He thought I had to do a "Southern study walk." This is not a good thing to do. Do not seek meritorious service, but never before.
[VHM: GT only very reluctantly translated the last sentence — I had to trick / force it — probably because it recognized that this sentence was not Mandarin, but some other language, i.e., Literary Sinitic.]
GT deep learning version (from Hofstadter):
After a year of work at Tsinghua, he was transferred to the Mao Translating Committee to live in the city and back to school on weekends. He is still a graduate student.
The leadership of the Mao Tse Translation Committee is Comrade Xu Yongjian. Introduction to the book to do this work is Tsinghua students Qiao Guanhua comrades.
On the day of the event, after dinner, an old friend hired a rickshaw from the city to congratulate. Guest to go, the book of fear in the book said to me:
He thought I had to do “South study walking.” This is not a good thing to do, not for meritorious service, but for nothing.
Hofstadter tr.:
After Zhongshu had worked at Tsinghua University for a year, he was transferred to the committee that was translating selected works of Chairman Mao. He lived in the city, but each weekend he would return to school. He also was still supervising his graduate students.
The leader of the translation committee of Mao’s works was Comrade Xu Yongying, and the person who had arranged for Zhongshu to do this work was his old Tsinghua schoolmate, Comrade Qiao Guanhua.
On the day this appointment was decided, after dinner, an old friend specially hired a rickshaw and came all the way from the city just to congratulate Zhongshu. After our guest had left, Zhongshu turned to me uneasily and said:
“He thought I was going to become a ‘South Study special aide.’ This kind of work is not easy. You can’t hope for glory; all you can hope for is to do it without errors.”
Mair tr.:
After he had been working at Tsinghua for a year, Zhongshu was transferred to work at the committee for the translation of Mao's selected works. He lived in the city, but would come back to the campus on weekends. He still continued to be in charge of graduate students.
The leader of the committee for the translation of Mao's selected works was comrade Xu Yongying. The person who had introduced Zhongshu to do this job was his Tsinghua classmate, comrade Qiao Guanhua.
On the day when the matter had been settled, an old friend hired a rickshaw expressly to come from the city after dinner to congratulate Zhongshu. After the guest left, Zhongshu anxiously said to me:
"He thought that I would have to serve as 'the emperor's literary aide'. This is not an easy task to handle. 'Don't seek for merit; just try to avoid making mistakes'."
The greatest differences between the Hofstadter and Mair translations are in the last paragraph. I attribute them to the fact that I am a Sinologist and Hofstadter is not. The final sentence is a set phrase (chéngyǔ 成语) I suspected right away that nán shūfáng 南书房 ("southern study") is a technical term pertaining to the imperial bureaucracy.
Through the wonders of Wikipedia, I was able swiftly to determine that nán shūfáng 南书房 ("southern study") is equivalent to Manchu julergi bithei booᠵᡠᠯᡝᡵᡤᡳ
ᠪᡳᡨᡥᡝᡳ
ᠪᠣᡠ
I was pleasantly surprised to find nán shūfáng 南书房 ("southern study") in Charles O. Hucker's magnificent A Dictionary of Official Titles in Imperial China (Stanford: Stanford University Press, 1985), p. 341a #4119:
Southern Study, originally the personal study of the K'ang-hsi Emperor [VHM: 1654-1722]; from 1659 members of the Hanlin Academy (han-lin yüan) served there as writers, calligraphers, copyists, etc., for a time enjoying great influence as the group through which imperial pronouncements were issued; lost such influence after establishment of the Council of State (chün-chi ch'u) in 1730. Often called nan-chai. Also see hsing-tsou.
Lexical notes by Stefan Georg:
julergi bithei boo is usually translated as:
„name of the office which held the books which were to be presented to the emperor“ (Hauer)
Another translation for the whole expression is: ‚the name of a study used by the emperor‘ (Norman)
Zakharov has: 'southern cabinet for the emperor, situated within the library'
julergi is „front, in front, by extension also ‚south‘, but the general spatial meaning is original – it may be, though, that, given the Chinese rendering, the meaning „southern“ is the intended one here – it is said that this study /office was situated in the „southwestern corner“ of the palace – this would have to be checked, though.
bithe-i is „book, piece of writing, genitive'
boo is basically ‚house‘.
It is easy to find inadequacies in the GT translations, and Hofstadter does so systematically, but I don't think anyone in their right mind would expect a machine to do as good a job as a skilled, experienced, sensitive, creative human translator who knows both the source language and the target language well.
If GT and other machine translators are unable to do a perfect job, or even one that is close to what a skilled human translator is capable of, what are their purposes? I believe that they fulfill a useful function in giving us the gist of the meaning of texts written in languages with which we are unfamiliar. As the Chinese say, "dǒng dàgài de yìsi 懂大概的意思" ("understand roughly what it means").
Previous posts on Google Translate:
"Google Translate is even better now" (9/27/16)
"Why electronic machine translation services sometimes seem to fail" (1/29/17)
"The wonders of Google Translate" (9/22/17)
[h.t. Mark Metcalf]
Yuval said,
February 4, 2018 @ 8:03 pm
If I may repeat my somewhat-buried comment from the original piece:
As someone in the relevant field of research (I'm a Natural Language Processing PhD student who's up to date on Machine Translation (MT) literature) there are two particularly irritating themes in this article:
1) On the high level, Prof. Hofstadter writes and reiterates using several different phrasings that MT, or at least Google Translate (GT), is not interested in representing anything beyond words or phrases. This is simply not true. Current MT literature is rife with methods incorporating context, world knowledge, everything DH is looking for, into MT systems. As for the GT system itself, going to newer resources than the NYT article linked to at the top, e.g. [1a/b] and derivative press coverage, demonstrates that full-sentence meaning was precisely what was on the Google's team mind when they released the most recent neural translation architecture.
2) On a lower level, there are some statements in this article purporting to explain why the specific examples failed GT (e.g. the paragraph regarding 'Ungeraden'). These confidently-stated reasonings cannot possibly be more than guesswork. They reflect not only lack of debugging the specific examples (I don't expect a writer to perform that) but some misunderstanding of what the GT architecture is capable of (in the Ungeraden case, in theory appropriate context should help get the correct translation, but there are too many other factors that may hinder the ultimate choice). What I do expect is some hedging of these statements, or consultation with people familiar with GT (or just plain MT) to see if they are plausible. DH obviously consulted experts in the writing of this piece, might as well have added some NLP experts to the language informants.
[1a] https://research.googleblog.com/zero-shot-translation-with-googles.html
[1b] https://arxiv.org/abs/1611.04558
Alex said,
February 4, 2018 @ 8:38 pm
I was curious to know how long did it take each individual to translate that short piece?
Thanks
Alex said,
February 4, 2018 @ 8:40 pm
Oh I see in that article he states
"In case you’re curious, here’s my version of the same passage (it took me hours)"
Josh Fogel said,
February 4, 2018 @ 8:41 pm
Fascinating, Victor, as always.
Joyce Melton said,
February 4, 2018 @ 11:28 pm
GT generally seems to do an adequate job translating Spanish, giving more than just the gist of the text. Less well for German even less for Swedish and still less than that for Vietnamese. I don't know French or Dutch well enough to assess but the translations are usually good enough for me to communicate with correspondents in those languages.
One thing it does very well: It's a LOT faster than I am even at Spanish.
AntC said,
February 4, 2018 @ 11:30 pm
If GT and other machine translators are unable to do a perfect job, or even one that is close to what a skilled human translator is capable of, what are their purposes?
I have to say I see very little purpose. Let's not expect it to translate literature or even memoirs. Let's stick to humdrum/formulaic reports and instructions. Victor's many 'Lost in Translation' pieces show GT can't even reliably cope with "keep off the grass" or "beware floor slippery".
I believe that they fulfill a useful function in giving us the gist of the meaning of texts written in languages with which we are unfamiliar.
Real-life example from yesterday: I'm in Taiwan; I wanted a restaurant; Yelp has very few reviews in English, so I got it to translate one from Mandarin — about 50 characters. Total fail: exactly as Hofstadter describes there were English words but nothing recognisable as a sentence or coherent phrase. I couldn't tell whether the reviewer was recommending the place or not, whether the food was spicy, was the place busy/popular, was the service friendly/tolerant of a foreigner. There was something about the owner, but nothing that helped me decide whether this was a place worth patronising.
Phil H said,
February 5, 2018 @ 12:16 am
My personal preference would be a more modernised approach, though we do risk losing some of the flavour of Qian’s reference-laden style. I second that like for the Hucker dictionary. It is an amazing piece of work.
A year after he started at Tsinghua, Zhongshu was appointed to the team translating Mao’s Selected Works. He had to stay in town during the week, but he came back to the university at the weekends, and he continued to supervise graduate students.
The man in charge of the translation committee was Comrade Xu Yongying, and it was Comrade Qiao Guanhua, a friend from Zhongshu’s own student days at Tsinghua, who had introduced him and got him the position.
On the evening of day the appointment was announced, an old friend in the city hired a rickshaw specially to trek out to the university and congratulate Zhongshu on the new job. After our guest had left, he turned to me in alarm.
“He thought I was going to work on the Chairman’s personal staff,” he said. “This job isn’t going to be easy. I’ll get none of the credit if it goes well, and all of the blame if anything goes wrong.”
Rubrick said,
February 5, 2018 @ 1:53 am
Hofstadter has for decades had something of a bee in his bonnet about any AI which attacks a problem in a way which differs from how a human would approach it. I get the sense that practical applications hold no appeal for him; if it isn't penetrating the mysteries of human creativity and thought, it's trivial and uninteresting.
boiko said,
February 5, 2018 @ 4:11 am
> Less well for German
German↔English has worked more often than not for me; it can't handle literary texts, of course, but it's more than enough to make my way through shopping and street signs and telephone bookings and online forms etc. It works well enough that I resent it, because its terrible convenience is surely damaging my German acquisition.
Victor Mair said,
February 5, 2018 @ 7:36 am
Regarding boiko's sentiments concerning GT and German↔English, they echo mine exactly, except that I also use it to skim through German newspapers, blogs, etc. If I find something that intrigues me, I'll let GT do a first pass. After reading that, and if the article really interests me, then I'l hunker down and do a more serious translation by myself. I do the same with many other languages (Russian, French, etc.).
Emily said,
February 5, 2018 @ 10:17 am
Regarding German and Google Translate, in 2010 (I mention the date because maybe the software has improved since then) I was taking a philosophy of language course and the professor assigned us a "translation" of Frege's Über Sinn und Bedeutung. I noticed right away that the text seemed off, with odd syntax and some word usages that didn't seem to make sense. And then I skimmed down and came across a sentence along the lines of "That would have the meaning of 'morning star' and 'evening star' the same, but not the meaning." Then I looked at the URL and realized it was a Google Translate version of the original German. You'd think someone who specialized in semantics would know better.
~flow said,
February 5, 2018 @ 11:43 am
Hofstadter complains about GT rendering G 'die "Ungeraden"' as E 'the "odd"', where he gives E 'the "Undesirables"' (exact wording adapted to fit this context). I disagree; especially given that the text was written by a mathematician and that the word is quoted makes me understand it as a pun on the 'odd figures', i.e. politically undesirables (from the point of the right wing professors). That pun admittedly works better in German than in English, I believe because of the negative prefix, 'un-'; still, I think that, for example, translating this as '… protect the institutions … from "Odd figures" (that is to say, "Undesirables).' does better in preserving the original (at the price of a translator's note to the reader).
Hofstadter's G/E translation has a few more instances where he prefers the most common way of putting it over the closest English equivalent; hence we have
G 'nach dem verlorenen Krieg'—E 'after the defeat' (closer: 'after the lost war', which totally works, so why not);
G 'deutschnational'—E 'with Pan-Germanistic leanings' (this reads a lot into 'German-nationalist', maybe justifiably so, but the author didn't write 'Pangermanisch' or 'Großdeutsch');
G 'über wenig war man sich einiger'—E 'nothing was clearer than that' (maybe 'few things were less controversial', or as indeed ironically proposed, but not used by Hofstadter himself: 'there was little about which people were more in agreement').
G 'am schutzlosesten'—E 'the most likely to be dismissed' (this one's outwright rong; 'schutzlos' is 'without protection', 'defenseless', as in 'den Elementen schutzlos ausgeliefert sein', 'to be at the mercy of the elements'; hence, G 'am schutzlosesten waren junge Wissenschaftler vor ihrer Habilitation' should rather become something like E 'young scholars that had not yet attained tenure were entirely defenseless'.
I'm not claiming that my suggested translations make for an easier reading by an American audience; I want to point out that I find Hofstadter's translations unnecessarily far from the original and too interpretative. It's almost as if he regards a more literally faithful rendering as too machine-ish, as if he feels that he has to explain rather than to translate.
Philipp Angermeyer said,
February 5, 2018 @ 12:58 pm
"I believe that they (=machine translators) fulfill a useful function in giving us the gist of the meaning of texts written in languages with which we are unfamiliar."
That is true, but sadly many people also routinely use Google Translate to PRODUCE texts in languages with which they are unfamiliar. Hence they are unable to recognize when the output turns out to be gibberish and incomprehensible for their addressees. When public institutions use it as a cost saving measure instead of hiring human translators or interpreters, machine translation leads to exclusion and inequality, rather than to some magic utopia where everyone is able to communicate across language barriers.
As other commentators have noted, translation quality differs by language, and that is due in part to the size of the underlying corpus of parallel texts that is available for a given language. Google now offers over 100 languages, and for many of them the corpus is too small to produce reliable results.
Andrew D. said,
February 5, 2018 @ 1:07 pm
This is like an old man fighting the cognitive dissonance between ideas he's built his entire career on and the manifest truth that machine translation is already quite excellent for plain everyday language.
"Such a development would cause a soul-shattering upheaval in my mental life. Although I fully understand the fascination of trying to get machines to translate well, I am not in the least eager to see human translators replaced by inanimate machines. Indeed, the idea frightens and revolts me. To my mind, translation is an incredibly subtle art that draws constantly on one’s many years of experience in life, and on one’s creative imagination. If, some “fine” day, human translators were to become relics of the past, my respect for the human mind would be profoundly shaken, and the shock would leave me reeling with terrible confusion and immense, permanent sadness."
David Marjanović said,
February 5, 2018 @ 4:44 pm
On his French example, Hofstadter is right: GT has no concept of emphasis. The same holds for the Wissenschaftlerinnen "female scientists"* of his German example.
* "Sciences" here including the Geisteswissenschaften, literally "mind sciences", in English humanities, so Hofstadter's choice of scholars is by no means wrong.
I'm so stealing this.
Further:
– translating gewissermaßen as "pretty much" is very free, I'd have gone for "in a way". Literally, there's "certain" in there, as in "in a certain way".
– Ungerade has, to the best of my knowledge or Hofstadter's, never before or since been used to mean "ungoodthinkful". That's almost certainly why it's in scare quotes. Otherwise, it's only used for the odd numbers. Evidently this is a consciously crafted one-off metaphor; I'd have tried to preserve or explain it in some way rather than rendering it so freely by an unremarkable word.
– The spelling in frage is odd. I'd expect either in Frage (literally "into question") or infrage (not that I understand why this spelling even exists). GT had to interpret the standalone lowercase frage as a verb form (imperative or 1sg present, among others) of fragen, "to ask", and that threw the clause into chaos.
– In Frage kommen doesn't mean "to have a place in the system". It means "to be an option". When it is negated, as it is here, it can be rendered in English straightforwardly as "to be out of the question": "and female scientists/scholars were out of the question anyway".
– "Well" for sowieso isn't a good idea either. "Anyway" fits just about perfectly – and that's what GT uses.
– How about "German nationalist" for deutschnational? Though that may require an explicit explanation that the people in question considered themselves ethnic Germans in the first place, which was very common among Austrians at that time but is pretty much limited to Nazis nowadays.
David Marjanović said,
February 5, 2018 @ 4:50 pm
…uh, except when it does. Sa […] à lui is emphasis; because it's expressed by lexical means, GT recognizes it and renders it as such ("his own" instead of just "his"). What GT doesn't have a concept of, because it isn't written, is intonation…
Mary Kuhner said,
February 5, 2018 @ 7:10 pm
In _Godel, Escher, Bach_ Hofstadter proposes that you could change the encoding between DNA and amino acids, modify the whole genome accordingly, and everything would still work. Because of DNA-binding proteins which recognize specific sequences, as well as RNA transcripts (made from the DNA) which have various activities beside coding for protein, this is far from true. It's a lovely metaphor but not a physical reality. I am pretty sure this was already known when he wrote the book, though the extent to which it is true wasn't fully appreciated then–the more we know about DNA the more meddlesome it appears, with all sorts of behaviors besides encoding proteins. It's like a text where the words are meaningful, and the arrangement of text on the page also makes a meaningful picture.
It's funny that the idea in _GEB_ is sort of a machine-translation view of the genome, which is exactly what he dislikes when applied to language!
Something that DNA and natural languages may have in common, making both of them difficult to translate, is a complete lack of "good engineering practices". If you take, say, emphasis, a natural language may indicate emphasis by tone of voice AND by pacing AND by choice of synonyms AND by position in the sentence…highly illogical, but that's what they do. Similarly a single gene may be controlled by modifying how often it's transcribed into RNA AND how often the RNA is modified into usable or unusable forms AND how long the RNA hangs around AND how fast the RNA is translated to protein AND…. This is what you get when something arises with no plan or guiding structure, just as an accretion of lots of individual-level decisions.
Gmail.com said,
February 6, 2018 @ 8:27 am
Google translate isn't perfect. That's all.
Stewart Nicol said,
February 6, 2018 @ 6:49 pm
If GT and other machine translators are unable to do a perfect job, or even one that is close to what a skilled human translator is capable of, what are their purposes?
I have found Google Translate to be extremely useful in translating old (18 & 19th Century ) scientific papers, particularly in French and German, both of which I have some familiarity with. It would take me a very long time to do a complete translation myself, but if I paste them into Google Translate I can get a quick overview, and then if there is a critical passage, I can go back to the original and check it carefully myself. The slowest part of this is correcting the optical character recognition errors in the scans of old texts.
AntC said,
February 6, 2018 @ 9:39 pm
I'm pleased to hear some are finding Google translate adequate for some purposes.
@Gmail.com, no I'm not expecting anything perfect.
@Andrew D machine translation is already quite excellent for plain everyday language.
My experience is to the contrary (trying to use it when travelling in a Mandarin-speaking country — which should have a sizable corpus). My desired plain everyday usage is restaurant reviews and recommendations for tourist sites. The alleged translation is just incomprehensible.
ktschwarz said,
February 18, 2018 @ 1:38 pm
I hope it's not too late for Professor Mair to check this thread, because I think he has a wrong assumption: what he calls "GT regular version (from VHM)" is presumably what he got as the default output on Feb. 4. But the default is the deep learning version. The difference between this and the one that Hofstadter quotes could be the effect of software updates since Hofstadter wrote the article.
I believe that the older, phrase-based, non-deep-learning version is what you get if you hover the mouse on the sentence and click for an alternate translation: He thinks I do "South study walking" up. This version translated the particle 了 in isolation at the end, producing an ungrammatical English result: it's better in the deep-learning version because it trains on entire sentences.
VHM: GT only very reluctantly translated the last sentence — I had to trick / force it — probably because it recognized that this sentence was not Mandarin, but some other language, i.e., Literary Sinitic.
Surely this is giving it far too much credit; aren't they written in the same characters? (A naive question; please set me straight if I'm confused.) Aren't these literary phrases often mixed in with modern Mandarin? I wonder if GT actually has a lot of them in its corpus, and if so, they probably haven't been annotated.
Its reluctance to translate the last 8 characters has been fluctuating for me, and even platform-dependent (Android tablet vs. Windows PC). I wonder if the problem is that that last Chinese sentence needs to be at least two sentences in English. It might be getting a low score for using a comma splice in English where it is OK in Chinese.