Does ChatGPT speak your language?

And a tool for checking the proficiency of ChatGPT in any language

In a previous post I pointed out how ChatGPT is far from proficient in Dhivehi, the language of the Maldives. The methodology was haphazard: I was just translating four sentences from English into Dhivehi and back – in a new chat, obviously - to compare their meaning to the original. It turns out that all of this can be automated and made more systematic in the process.

So I wrote a little script to do it, using the OpenAI API: lugha. This makes it easy to translate a sentence to a target language and back and, as a cherry on top, uses chatGPT in yet a new session to compare the meaning of the original sentence to that of the new sentence obtained through the round trip translation process.

I took this to a test drive on a list of 46 African languages, using two sets of sentences: ten sentences using everyday vocabulary, and ten sentences from technical domains. An example of the first type of sentence is The grocery store is in front of the school and for the second type Laparoscopic cholecystectomy is a minimally invasive surgical procedure used for the removal of a diseased gallbladder. The dataset I built can be freely downloaded from the lugha repo. You can see a summary in the plot below: for each language the bars show the number of correctly translated sentences out of a total of ten for the two categories.

The results are quite underwhelming: except for Afrikaans[1] and Swahili[2], GPT4 is far from fluent in most languages spoken on the continent. The 50 million native speakers of Hausa are out of luck: only two out of ten nontechnical sentences get translated correctly, and none of the technical ones do. Where is the bathroom? comes back from the round trip to Hausa as Are you sleepy?

Do not even think about translating a math book: A contravariant functor from the category of topological spaces (with continuous maps as morphisms) to the category of real associative algebras is given by assigning to every topological space X the algebra C(X) of all real-valued continuous functions on that space becomes absolute gibberish: The continuous function from a topological space (with forward properties like weather) to the algebraic company of truth color gives every topological space X algebra C(X) of all open truth color patches on this field.

This debacle probably boils down to the absence of training data, even though there appear to be suitable corpora available online. They probably did not make it into the common crawl.

Now tell me again that technology is neutral.

---

[1] The language is related to Dutch, making life much easier for ChatGPT

[2] Again, a lingua franca with many loanwords from Arabic, including lugha (language)