Language Log

« previous post | next post »

Competition among various AI services will spur them to further heights.

ChatGPT, Bing, Bard and DeepL: Which one offers the best Japanese-to-English translation?

by Karin Kaneko, Japan Times (7/18/23)

Kaneko, working with her editors at Japan Times, devised an ingenious test for comparing the quality of several translation tools in different categories of writing.  Since this experiment is so innately interesting and inherently revelatory, I will provide extensive quotations, adding romanization of the Japanese passages from GT (not an easy task for me!).  To be fair to GT, and simply out of curiosity to see how it compares with the newer type of AI translation services, I will also invite GT to translate all three of the chosen passages.  N.B.:  All three of the GT English translations have been added by me.

Kanako Takahara, a senior bilingual Japan Times editor, compared Japanese-to-English translations from ChatGPT-4, Bing and Bard, along with DeepL, an AI-based machine translation service, using Japanese text in three different categories:

• • Literature: The opening lines of “Snow Country,” authored by Nobel laureate Yasunari Kawabata
  • Lyrics: Japan’s national anthem “Kimigayo”
  • Speech: Japanese superstar Shohei Ohtani’s speech to his teammates just before the World Baseball Classic final against the United States in March

She scored on a scale of 1 to 5, with 5 being the highest based on accuracy, how natural the English was and whether the translation reflected the context.

(Please note that both ChatGPT-4 and Bing are powered by OpenAI’s GPT-4, while Bard is currently running on its beta version. The order of the translations is based on scores.)

‘Snow Country’

[VHM:  from Yasunari Kawabata's famous novel]

Kokkyō no nagai ton'neruwonukeruto yukigunideatta. Yoru no soko ga shiroku natta. Shingōsho ni kisha ga tomatta. Mukaugawa no zaseki kara musume ga tatte kite, Shimamura no mae no garasu mado o otoshita. Yuki no reiki ga nagare konda. Musume wa mado-ippai ni noridashite, tōku e sakebu yō ni, `ekichō sā n, ekichō sā n' akari o sagete yukkuri yuki o funde kita otoko wa, erimaki de hana no ue made tsutsumi, mimi ni bōshi no kegawa o tarete ita. Mō son'na samu-sa ka to Shimamura wa soto o nagameru to, tetsudō no kansharashī barakku ga yamasuso ni samuzamu to chirabatte iru dake de, yuki no iro wa soko made ikanu uchi ni yami ni noma rete i

国境の長いトンネルを抜けると雪国であった。夜の底が白くなった。信号所に汽車が止まった。

向側の座席から娘が立って来て、島村の前のガラス窓を落した。雪の冷気が流れこんだ。娘は窓いっぱいに乗り出して、遠くへ叫ぶように、

「駅長さあん、駅長さあん」

明りをさげてゆっくり雪を踏んで来た男は、襟巻で鼻の上まで包み、耳に帽子の毛皮を垂れていた。

もうそんな寒さかと島村は外を眺めると、鉄道の官舎らしいバラックが山裾に寒々と散らばっているだけで、雪の色はそこまで行かぬうちに闇に呑まれてい

Bard
Score: 4

When the train emerged from the long tunnel at the border, it was a snowy country. The bottom of the night turned white. The train stopped at a signal station.

A girl stood up from the seat across from Shimamura and opened the glass window in front of him. The cold snow blew in. The girl leaned out the window and called out into the distance,

“Stationmaster, stationmaster!”

A man came slowly through the snow, his nose wrapped in a muffler and his ears covered with fur.

Shimamura looked out the window and saw a row of barracks that looked like railway offices scattered coldly on the mountainside. The snow was already dark before it reached there.

ChatGPT 4
Score: 3

Upon exiting the long tunnel at the national border, there was the snow country. The bottom of the night had turned white. The train had stopped at the signal station.

A young girl rose from the seat across and slid down the glass window in front of Shimamura. A rush of cold, snowy air flowed in. Leaning out as far as she could from the window, she called out into the distance.

“Stationmaster, oh stationmaster.”

A man trudging slowly through the snow, lantern in hand, appeared. His collar was wrapped up to his nose, and the fur of his hat hung over his ears.

Deep L
Score: 2

After passing through a long tunnel at the border, we were in a snowy country. The bottom of the night turned white. The train stopped at a signal station.

A girl stood up from the seat on the opposite side of the train and dropped the glass window in front of Shimamura. The cold air from the snow poured in. The daughter leaned out the window and shouted into the distance, “Ekiman-sa-an, Ekiman-sa-an!

“Station master, station master!

The man who had come slowly through the snow with his light lowered was wrapped up to the top of his nose in a collar and had the fur of his hat hanging down over his ears.

Shimamura looked outside to see if it was that cold already, but there were only a few barracks that looked like railroad buildings scattered in the cold at the foot of the mountain, and the color of the snow had been swallowed by the darkness before it reached there.

Observing the outside, Shimamura wondered if it was already that cold. Barracks resembling those of a railway official were sparsely scattered along the foot of the mountain. However, before reaching the hue of the snow, it was swallowed up by the darkness.

Bing
Score: 1

When we came out of the long tunnel and the snow country began, the night became whiter. The train stopped at a signal stop.

A girl came out of the seat on the other side and dropped the glass window in front of Shimamura. The cold air of the snow flowed in.

The girl leaned out of the window and shouted far away, “Station Master! Station Master!” The man who had come slowly down the snow with his light lowered was wrapped up to his nose with a muffler and had a fur hat hanging from his ears.

Shimamura looked outside and saw only barracks that seemed to be railway officials scattered coldly on the mountainside, and the color of the snow was swallowed up by darkness before it went that far.

GT

After passing through the long tunnel at the border, it was snow country. The bottom of the night turned white. The train stopped at the signal station. Her daughter stood up from the opposite seat and knocked down the glass window in front of Shimamura. A chill of snow poured in. Her daughter leans over the window and screams into the distance, "Station length saan, station length saan" The man who dimmed the light and slowly treaded through the snow had his scarf wrapped over his nose and the fur of his hat hanging over his ears. When Shimamura looked outside in the cold, he saw only barracks, which looked like official railway residences, were strewn coldly at the foot of the mountain.

—–

For AI, translating literature is one of the most difficult tasks, since it requires translating the nuances of a story in an eloquent manner. It also requires an understanding of the times being described — in this instance, the early Showa Era (1926-1989).

The iconic opening line is widely translated as: “The train came out of the long tunnel into the snow country. The earth lay white under the night sky.”

Although 国境 literally can be translated as “national border” as per ChatGPT 4’s effort, it actually refers to the prefectural borders back in those days, which is why the above version didn’t include it in its translation.

Takahara scored Bard the best among the four, getting the subject right for the first sentence — the train — with Bing scoring the lowest. Bing didn’t provide the eloquence required for literature, and there were more mistranslations than the other three.

 

Japanese national anthem Kimigayo

Kimigayo wa chiyo ni yachiyo ni

sazare ishi no iwao to narite

kokenomusumade

君が代は　千代に八千代に

さざれ石の　巌となりて

苔のむすまで

ChatGPT-4
Score: 5

May your reign
Continue for a thousand, eight thousand generations,
Until the pebbles
Grow into boulders
Lush with moss.

Bing
Score: 5

“May your reign continue for a thousand, eight thousand generations, Until the pebbles grow into boulders Lush with moss.”

Bard
Score: 4

May your reign last for a thousand, eight thousand years.
May it be like a small stone that becomes a great boulder, And is covered with moss.

DeepL
Score: 2

Kimigayo is for a thousand and eight thousand generations
Until it becomes a rock of gravel.
Till the moss grows

GT

Kimigayo is Chiyo and Yachiyo

Become a rock of pebbles

Moss-no-Musuma

—–

This was also a tricky one, with various interpretations of what kimigayo refers to — the emperor’s reign, the nation or the life of a loved one. The lyrics are believed to have originated from Japanese literature more than 1,000 years ago.

The government does not offer an official English translation, but has said that the lyrics of “Kimigayo” are about praying for the peace and prosperity of Japan with the emperor as a symbol. During World War II, however, kimigayo was referred to as the emperor’s reign.

Based on this, Takahara scored Bing and ChatGPT-4, as the best translations — giving each a 5 for being simple and precise. Bard scored 4, and DeepL scored 2 for translating 君が代 as simply kimigayo, which means little to English readers.

 

Ohtani’s speech

{VHM:  Japanese baseball player, Shohei Ohtani]

Boku kara ichi-ko dake. Akogareru no o yamemashou. Fāsuto ni gōrudoshumitto ga i tari, sentā o mireba maiku torauto ga irushi, gaiya ni mūkī bettsu ga i tari, yakyū o yatte itara dare shimo kiita koto ga aru yōna senshu-tachi ga iru to omou. Akogarete shimatte wa koe rarenainode, bokuraha kyō koeru tame ni, toppu ni naru tame ni kitanode. Kyōichinichi dake wa karera e no akogare o sutete, katsu koto dake kangaete ikimashou

僕から一個だけ。憧れるのをやめましょう。ファーストにゴールドシュミットがいたり、センターを見ればマイク・トラウトがいるし、外野にムーキー・ベッツがいたり、野球をやっていたら誰しも聞いたことがあるような選手たちがいると思う。憧れてしまっては超えられないので、僕らは今日超えるために、トップになるために来たので。今日一日だけは彼らへの憧れを捨てて、勝つことだけ考えていきましょう

ChatGPT-4
Score: 4

Here’s one thing from me. Let’s stop idolizing. You might see players like Paul Goldschmidt on first base, Mike Trout when you look at the center, or Mookie Betts in the outfield, the kind of players anyone who has played baseball would know. But if we keep idolizing them, we cannot surpass them. We came here today to surpass them, to reach the top. Just for today, let’s put aside our admiration for them and focus solely on winning.

Bing
Score: 3

Just one thing from me. Let’s stop admiring them. There are players like Goldschmidt in first base, Mike Trout in center field, and Mookie Betts in the outfield. I think there are players that everyone who plays baseball has heard of. If you admire them, you can’t overcome them. So we came here today to overcome them and become the top. Let’s forget about admiring them for just one day and focus on winning.

Bard
Score: 3

I have one request. Let’s stop looking up to them. There are players like Goldschmidt at first base, Mike Trout in center field, and Mookie Betts in the outfield. I think everyone who plays baseball has heard of them. But if we get too caught up in looking up to them, we won’t be able to surpass them. We’re here today to surpass them and become the best. So today, let’s just focus on winning and forget about looking up to them.

DeepL
Score: 2

Just one from me.

Let’s stop pining for them.

If you look at first there is Goldschmidt, if you look at center there is Mike Trout, if you look at the outfield there is Mookie Betts, and I think everyone who has played baseball has heard of these players.

If you admire them, you can’t surpass them, so we came here today to surpass them, to be at the top. For today, let’s put aside our admiration for them and just think about winning!

GT

Only one from me Stop dreaming. You've got Goldschmidt in first, Mike Trout in center, Mookie Betts in the outfield, and players you've heard of if you've played baseball. If you admire it, you won't be able to surpass it, so we came here to surpass it today, to become the top. Let's throw away our admiration for them for just one day today and only think about winning.

—–

Takahara scored ChatGPT as the highest among the four because of how it translated 憧れるのをやめましょう. ChatGPT is the only one that translated the nuance that best described the situation: “Let’s stop idolizing (them).”

For the Japanese baseball players, those Major League Baseball stars were their idols. “Admire” may have a similar meaning, but “idolize” was the best choice of word.

The first phrase, 僕から一個だけ, was also translated differently by the four, one of which (Bing) translated it as “Just one thing from me,” which was the most natural English.

And the winner is?

Based on the above results, ChatGPT-4 scored the highest overall, closely followed by Bard and Bing. Translations varied for “Snow Country” with all of the tools aside from Bard unable to capture the subject of the story, suggesting that the translation of literature may not be among their strengths.

Note that AI chatbots may produce different responses on different devices or with different timings or various other factors such as the way you phrase your instruction, which means that if users were to translate the above text, the result may differ from what The Japan Times had.

While this factor also makes it difficult to pick which of the AI-assisted tools is the best, ChatGPT-4 easily offers the highest quality, according to Tom Gally, an expert in Japanese and English at the University of Tokyo who has been experimenting with the AI language models.

Gally said that what makes the large language models superior compared to previous tools is the ability to interact with humans. Therefore, the quality of answers that generative AI tools produce depends largely on the way we phrase the questions or instructions to the AI, which is called “prompt engineering.”

For instance, ChatGPT-4 can produce text from “Snow Country” in various styles, from Shakespearean to something resembling a rap song. It can also provide a translation in various styles at a speedy rate, and far faster than humans ever could. Bard is also able to do this, though the quality is below that of ChatGPT-4 at the moment.

ChatGPT’s Shakespearean translation of “Snow Country"

ChatGPT translation of “Snow Country,” in a rap style

Generative AI tools are also able to understand the context of a text and appear to remember the content of the previous sentences, whereas previous language models could only translate sentence by sentence without remembering the previous one, which can change the meaning of the original text.

However, since ChatGPT has been trained with less Japanese language data, its production of Japanese is not as good as with English. On the other hand, while Bard is still not as good in its current stage, it has the potential to be great given the amount of data Google has.

Another issue that Gally pointed out is that the tools can be weak in practical and detailed areas such as explaining grammar, explaining spellings and doing arithmetic.

AI chatbots are also prone to imagining facts and making reasoning errors. According to Bard’s website, large language models may “hallucinate” and present inaccurate information as fact. OpenAI also admitted similar limitations with its ChatGPT, advising users to take great care when using its services.

“It’s like a very, very smart person who may sometimes make stupid mistakes,” Gally said. “So, you have to be aware, check the mistakes and verify.”

What was clear, though, was that AI chatbots’ translations were much better than those of DeepL — presumably because of their ability to capture the context. But what was also notable was that none of the texts were translated with 100% accuracy, which means humans would need to check and make necessary edits.

The future of translators

But beyond literature, music and casual speech, translators who handle technical documents may be most at risk of having their jobs replaced by AI in the coming years. Because of current AI limitations, those who possess specialized language knowledge are probably less likely to run the risk of being replaced.

Gally, who also works as a translator at the University of Tokyo, said that translators who do not have a relationship of trust and have less interaction with clients may be at risk, including freelancers, translators of technical documents and those hired through agencies.

The types of translators who interact and help their clients solve their language problems at a higher level will continue to be needed, especially if the translator has knowledge about their client’s needs.

The most important thing is to provide a service that surpasses the value of cheap price and speed that generative AIs offer.

“The most difficult thing about the advances in AI is that they offer both great potential and great danger at the same time,” Gally said. “The combination (of excitement and fear) is a very interesting and complex feeling.”

Table data compiled in July 2023. Please consult individual services to most up-to-date info.

 

AI translation tools are a work in progress, always improving, but never achieving perfection.

Obvious question:  when have human beings ever attained perfection?

 

Selected readings

• "ChatGPT does cuneiform studies" (5/21/23) — with a long bibliography
• "Is there no / any longer a reason / need to learn a foreign language?" (7/14/23) — also with a lengthy bibliography

[Thanks to Don Keyser]

July 19, 2023 @ 12:03 pm · Filed by Victor Mair under Artificial intelligence, Language and literature, Language and music, Language and sports, Translation

Permalink