Uh-oh! DeepL in the classroom; it's already here

February 22, 2023 @ 1:22 pm · Filed by under Artificial intelligence, Computational linguistics

« previous post | next post »

Yesterday in my Classical Chinese class, we were reading Ouyang Xiu's (1007-1072) "Discussion on 'Biographies of Eunuchs'" in the New History of the Five Dynasties (written 1036-1039, published 1072).  Here's the relevant passage:

Móu zhī ér bùkě wéi. Wéi zhī ér bùkě chéng. Zhì qí shén zé jù shāng ér liǎng bài. ——“Xīn wǔdài shǐ huàn zhě chuán lùn”

謀之而不可為。為之而不可成。至其甚則俱傷而兩敗。 ——《新五代史宦者傳論》 

[Because of the special circumstances of this post, I will not adhere to my usual custom of providing Pinyin Romanization, Hanzi transcription, and English translation all three together.]

The student who was reciting is usually pretty good.  She is diligent, conscientious, and mostly gets things right, but occasionally makes some slipups with grammar, syntax, and lexical meaning. 

Here was her translation of the passage in question:

If the ruler is able to plan it, he might not be able to actually carry it out. Even if he does, there is a chance that it will not work. When it comes to the worst case, it is likely that both sides will fail because they are both injured.

When she got to the part that is underlined, alarm bells started going off in my head.  Even though her work is almost always acceptable or close to it, this was too good, too felicitous.  It didn't match the level of the rest of her translation.

One law of reading Literary Sinitic / Classical Chinese (LSCC) texts in my classroom is that the students have to account for every word of the original, and they have to explain how they arrived at every part of their translation.

So I asked the student where she got "there is a chance that" in her translation.  She seemed a bit flummoxed, but then said candidly, "I added it." 

I thought that was a clever response, but it wasn't good enough for me.

"On what basis, for what purpose did you add it?" I pursued.

She is an honest person, so she admitted straightforwardly, "I got it from DeepL."

Thereupon I almost fell over, since we've been talking about DeepL, ChatGPT, and other such AI translation software so intensely recently.

What's most amazing is that it was precisely the word "case" that I was probing for.  In fact, I had gone around the whole room of about ten students hoping that someone would say "case" or something like that, but not even the best students could come up with it.

Even more mind-boggling is that DeepL is not meant for LSCC, but rather for Mandarin (a very different type of language), yet somehow it extracted enough information from the overall context in which "zhì qí shén 至其甚" ("reach its extreme") that it was able to transform that into "in the worst case".

The student had actually made a number of other improvements and embellishments on the bare bones, word-for-word, literal rendering of the original, and I think she did them mostly on her own, but it is genuinely curious that neither she nor anyone else among the smart students in the class came up with "case", which is missing in the original Chinese text, though necessary for an idiomatic English translation.

I did not chastise her in the slightest for using DeepL, because she exercised good judgement in choosing the most "intelligent" part of what DeepL had to offer.  In other words, my opinion is that we should regard DeepL and its AI kindred, not as enemies of our own production, but as resources that we can draw on like dictionaries, concordances, indices, thesauruses, and so forth.  What we need to do, both for our students and for ourselves, is to learn how to discard the chaff and keep the wheat.

 

Selected readings

 

[Thanks to Runrun Tian]

February 22, 2023 @ 1:22 pm · Filed by under Artificial intelligence, Computational linguistics


8 Comments

  1. Jonathan Smith said,

    February 22, 2023 @ 3:00 pm

    It's not clear here whether you are referring to "there is a chance that" or "when it comes to the worst case". Both? It's more the case wrt the former that it doesn't correspond to anything in the original, thus "I added it."

  2. Daniel Diniz said,

    February 22, 2023 @ 3:09 pm

    During learning, the AI goes through all types of subjects. Maybe it learned from actual textbooks on classical texts.

  3. Viseguy said,

    February 22, 2023 @ 7:29 pm

    This is a bit tangential, but … I've been using DeepL for several years to polish my emails to my cousins in Sicily, and, to my inexpert eye, the results are impressive. (I often have to dumb down the DeepL translations lest it be obvious to my kin that it's not actually me but a robot talking.) Just yesterday, however, I discovered Yandex translations (https://translate.yandex.com/), and am equally impressed. I tried both with Italian and Russian. In Russian, I was struck by both systems' ability to choose a translation of the same noun object ("barn", see below)) with sensitivity to the semantic context, resulting in various renditions (хлев / коровник / сарай / амбар). The top Google search result for "deepl vs. yandex" was A Study on Machine Translation Tools: A Comparative Error Analysis Between DeepL and Yandex for Russian-Italian Medical Translation ("While DeepL provided a generally better overall translation performance, especially evident in its rendering of the context and the syntactical structure of the text at the sentence level, Yandex proved better in transliteration and was able to suggest undoubtedly more accurate cultural-specific equivalents"). Anyway, here are my comparative DeepL-Yandex results with "barn":

    The cow left the barn.
    D: Корова вышла из коровника.
    Y: Корова вышла из хлева.
    The sheep left the barn.
    D: Овца вышла из коровника.
    Y: Овцы покинули хлев.
    The horse left the barn.
    D: Лошадь вышла из сарая.
    Y: Лошадь вышла из сарая.
    The pigs left the barn.
    D: Свиньи покинули коровник.
    Y: Свинья вышла из сарая.
    The farmer took the wheat out of the barn.
    D: Фермер убрал пшеницу из амбара.
    Y: Фермер вынес пшеницу из амбара.
    Every farm needs a barn.
    D: Каждой ферме нужен амбар.
    Y: Каждой ферме нужен сарай.

  4. AntC said,

    February 22, 2023 @ 7:39 pm

    To me, the whole translation offered sounds stilted English. Perhaps it wouldn't if I had the wider context of the passage.

    There's too many 'it's — my Latin translation teacher said to avoid repeating the same word (in English). Latin allows omitting pronouns where English doesn't; but that doesn't entail merely inserting pronouns for translation.

    Is the 'it' in the first two sentences referencing some particular plan/scheme just before mentioned?

    The next two 'it's are not referencing anything in the discourse/they're syntactic placeholders — exactly as you'd get from translating Latin word-for-word. Both appearing in the same sentence and just after 'it's that are referencing, makes them awkward and caused me to pause/re-read; also the explicit 'that'. I would say sth like:

    "In the worst case, probably both sides will fail, because both are (would be?) injured."

    (Perhaps this doesn't meet Prof Mair's requirements for close to word-for-word translation?)

    IOW "When it comes to the worst case, …" is not idiomatic English for me/too wordy/it sounds like a translation. It would maybe work as the conclusion in context of a list of increasingly undesirable scenarios — introduced by enumerating 'cases'. (But Prof Mair says that's exactly the word that's missing from the LSCC.)

    I see Prof Mair also mentions "In the worst case, …". But that idiom _doesn't_ appear in the offered translation. It does appear in the DeepL offering I've just tried. I'm confused.

    I can offer one way (from my experience) of how clunky phrases appear in classroom translations: 'Joint efforts' by students, who copy from each other but switch around phrases to avoid getting detected. (It seldom fooled the teacher.)

    What DeepL (just tried) fails to capture following 'In the worst case, …' is the subjunctive/counter-factual mood of the following two phrases. I'm getting "both were injured and both were defeated." In translation for any practical purposes it's highly important to distinguish counter- vs factual. So that's a fail.

  5. Kingfisher said,

    February 22, 2023 @ 7:39 pm

    I assume he did mean to quote "when it comes to the worst case". Either way, I would be impressed, since it seems unlikely DeepL or any other program has access to much CC-to-English translation.

    Taking my own stab at the full passage (自古宦 to 可不戒哉), though I imagine I would have to justify a lot myself…

    Ever since antiquity, the potential for disruption in a state which eunuchs are able to cause has been worse than the disasters brought about by women. After all, women are nothing more than a sensual distraction, but the harms which eunuchs can inflict do not all stem from a single source. Doubtless this is because eunuchs, through the nature of their service to the sovereign, are in close proximity to him and become a familiar sight, rendering them tolerable and unchallenged in his thinking. Through petty good works they worm their way into the ruler's feelings and through little displays of trust they shore up his affections, ensuring that the ruler trusts and becomes close to them. And, having once gained the ruler's trust, eunuchs become so fearful of losing it that they grab and hold fast to the handles of doom and fortune.

    Furthermore, though the court might be lined with loyal ministers and worthy gentlemen, the ruler may be distant and aloof from such people, preferring the company of these servants who tend to him in his chambers and at his meals, for those who are always in the sovereign's presence can count on their intimacy with him. Thus the ruler's constant companions become closer to him by the day while the loyal ministers and worthy gentlemen grow ever more distant, and day after day the ruler's power becomes more isolated. And where there is such isolation, the paranoia and wrath in the ruler's heart become harsher by the day and the positions of those graspers of power become all the more unshakeable. When safety or peril emerge from the glee or rage of the ruler and doom or disaster seep from under the curtain or the door, the source of the danger comes from those very people most capable of counting upon their own position. And even if, once the threat has become particularly grave, the ruler should awaken to the reality of his situation, he would have to resort to working with ministers whom he has long held at arm's length in order to rid himself of the companions who are always near at hand; if the ruler is sluggish then he only stores up more trouble for himself and lets the problem get worse, while if he rushes into things then the eunuchs might very well take him as a hostage. No matter how wise or clever the ruler may be, it might prove challenging for him to develop any good plan to root out the eunuchs; or though he might conceive of a plan in theory, it might seem unfeasible to actually attempt; or else though the attempt might be made, success may be elusive, and the solution might even destroy his side as much as the eunuchs. But though in the end it cost him his own life or even the life of his state, where once the ruler has permitted such villains to have the resources to rise to power and in the worst instances to sow their own kind throughout the government, let him kill them one and all to slake the hearts of the realm, come what may. And past histories are filled with accounts of eunuchs that caused disasters in this manner; it has never been the problem of some particular age alone.

    It is not that any given ruler actively wishes to store up such disaster for himself within or to alienate loyal ministers and worthy gentlemen without; it can only be the result of permitting the situation to come about through one incident after another. While it is true that a ruler who has been led astray by the charms of a woman may, if he unhappily never realizes his mistake, bring disaster down even upon himself, should he but once awaken to the error of his ways he need only rouse himself and drive her out. Yet when eunuchs are the source of pending doom, though the ruler may realize and regret what has come to pass, by then he might no longer possess the power to expel them, as was the fate of Zhaozong of Tang.

    Such are the reasons why I declare that eunuchs cause more grief to a state than women. Who then will fail to take warning?

  6. jonathan silk said,

    February 24, 2023 @ 5:44 am

    I'm puzzled, because dropping this into DeepL I got something entirely different:

    To seek, but not to do. To do, but not to achieve. In the end, both will be hurt and both will lose. –New Five Dynasties History of Eunuchs

  7. AntC said,

    February 24, 2023 @ 10:26 pm

    @jonathan, that's curious. I'm getting right now

    To seek, but not to do. If we do it, we cannot achieve it. In the worst case, both were injured and both were defeated. –The New Five Dynasties History of the Eunuchs [UK English]
    To seek, but not to do. To do, but not to achieve. In the end, both will be hurt and both will lose. –New Five Dynasties History of Eunuchs [US English]

    That has the "In the worst case, …" phrase we're concentrating on **in UK English** but your "In the end, …" **in American English**. Hmm. I think it's the same UK translation offered as at the time of my earlier message — I couldn't be exactly sure. But the differences are weird: there's no specifically UK vs US usages there.

    So a coupla things:

    * DeepL is learning continually (as indeed is Google translate). If you're getting a different translation vs earlier, that might be because a bunch of people have 'voted up' one version. (Or, contrarily, a bunch of people had previously voted up some pernicious translation; their votes have been detected and removed. That might be a very short-term effect. So might be worth re-trying from time to time.)

    * DeepL I believe tries to adapt to your locale: As well as UK vs US English, there's Brazilian vs Portuguese. I'm surprised there's not a S.Am. Spanish/Mexican/etc. Try changing the locale setting on your machine/going via a VPN/etc..

  8. Victor Mair said,

    February 24, 2023 @ 11:43 pm

    @jonathan silk

    Good question! I was hoping someone would ask it.

    @AntC

    Excellent observations! I was hoping someone would make them.

RSS feed for comments on this post