Reading time: 5 mins

In our previous article we looked at ChatGPT from the perspective of the translation industry; does it pose a threat to human translators or is it a welcome companion? The article briefly touched on ChatGPT’s machine translation (MT) capabilities and its potential advantages over other online MT tools.

ChatGPT’s main advantage compared to other free online translators, such as Google Translate or DeepL, is that it cannot only translate a piece of text, but it can also adapt the translation according to users’ additional prompts. These may be the use of a specific register, grammatical gender, abiding by character limits, and any number of other specific instruction. In theory, the sky is the limit! However, we wanted to put ChatGPT to the test and primarily determine whether it is able to accurately fulfil users’ requests and if its translation abilities can match other free MT tools.

We approached this by giving it several translation prompts from English to Slovenian. The language pair is ideal since Slovenian, unlike English, is a heavily grammatically gendered language, as well as differentiating between a formal and informal second person singular, to name just two of its distinctive features.

Case 1: formal “you”

In English, context and a choice of register will help speakers determine the level of respect and seniority expressed between people. In contrast, in Slovenian this can be expressed using the formal or informal second personal singular pronouns. If you wish to translate “I wish you a lovely day” into Slovenian, you cannot specify with Google Translate which form to use. With ChatGPT you can enter a prompt, such as “Translate the sentence ‘I wish you a lovely day’ into Slovenian using the formal register” and it will answer with: “Želim vam lep dan.” Conversely, if you ask it to translate the same sentence in an informal register, it will answer with: “Želim ti lep dan.” DeepL provides both options but does not inform the users about the difference between the two.

Case 2: grammatical gender

In Slovenian, the grammatical gender expressed by nouns and pronouns reflects in word endings throughout the sentence. Put simply, that means that if the speaker (subject) of the sentence is feminine, the verb and adjective must also have feminine endings. English is not so heavily gendered –there is little agreement between the nouns, verbs and adjectives. You will use the same form of the verb or adjective, no matter who the speaker is, with the exception of some irregular verbs (e.g. to be) and the third person singular, which adds the suffix –s to verbs, regardless of gender.

Therefore, if we wish to translate the sentence “I am looking forward to the summer, as I have booked a three-week holiday in Indonesia with my friend.” into Slovenian, we can instruct ChatGPT which gender to use. The results below show instructions for the male and female gender, as well as the mix of the two (respectively); the instructions were duly followed.

In contrast, Google Translate and DeepL will decide on the gender seemingly arbitrarily. It is hard to pinpoint how they determine which gender they will choose in translation if no clear indication is given, but this most likely depends on the data they source from, which can have a certain bias. In this case, they both opted for the female form.

Case 3: character or word limit

Finally, we instructed ChatGPT to use a maximum number of characters or words in the translation. Character limits can be crucial, for example when translating user interfaces, slogans and catchphrases, or similar short chunks of text that may be spatially constrained.

Several tests have shown ChatGPT to be extremely unreliable in this regard. The three examples below show that in many cases the instructions were not followed and the translations produced were flawed in terms of style and grammar.

Some online users have also noted that ChatGPT’s capabilities for counting words and characters is very limited and generally untrustworthy, at least in the current free version. Speakers of Slovenian will also notice that, perhaps in an attempt to follow instructions by cutting long phrases, its translations sound unnatural.

Our experiment looked at just a handful of prompts one can give ChatGPT about translation and was limited to a single language pair. It was intended to scratch the surface and invite further exploration, but also to spotlight the model’s current advantages over other MT tools and its limitations.

It has to be said that despite following instructions reasonably well in most of our tests, ChatGPT’s translation output is very hit-and-miss and suitable for personal information use at best. This technology does, however, show promise for future use in translation with its ability to follow users’ explicit contextual instructions, provided it continues to improve. Despite traditional MT tools’ current superiority in terms of achieving accurate and naturally-sounding results, this unique skill may ultimately give AI models the edge. Still, this remains a very high and steep hill to climb.

Translations for professional use must put quality first and a skilled human translators remains unrivalled in that aspect. Additionally, a good language service provider will always follow your instructions to a tee. Reach out to us to find out more!