By Christian J. Meier
At first the dialogue about love sounds very common. But then says “Hal” in the youtube video by computer scientist Alif Jakir: “I believe that you can love absolutely anyone, no matter who they are.”
At least now, the conversation seems unnatural, even though the artificially generated faces look real. Because the conversation is also synthetic, generated by a special type of artificial intelligence (AI), a so-called “language model”. It’s called “GPT-3” and it stunned the world last year. The Australian philosopher of language David Chalmers wanted in the versatility paired with eloquence of GPT-3 even recognize signs of human-like intelligence.
GPT-3, developed by the Californian company OpenAI, was the most powerful language model at the time. It independently produces texts that read as if they were written by people. The AI skilfully handles any topic, answers questions, writes stories, dialogues or poems, translates or converts everyday language into programming code.
“People will talk to computers more and more naturally”
The language model showed that AI can learn and express quite a broad knowledge of the world. A leap in quality that has meanwhile triggered a race for even more computationally powerful language models, in which large American tech companies such as Google or the chip manufacturer Nvidia are participating. In the summer, China, which wants to position itself as the second AI power next to the USA, entered the race with its “Wu Dao 2.0” language model. In December, the British Google subsidiary DeepMind finally published its “Retro” language model. It is said to have an advantage thanks to an external text database that is used like a cheat sheet.
Germany also wants to join the race next year. From January 2022 the Federal Ministry of Economics is funding the OpenGPT-X project, which aims to build a European language model. In addition to the Federal AI Association, eight other partners are involved, including AI companies such as Aleph Alpha from Heidelberg, the German Research Center for Artificial Intelligence, and possible users.
“Language models are a decisive development,” says Jörg Bienert from the KI-Bundesverband, a network of German AI companies and experts. They are the basis for a whole range of applications such as chatbots or the automatic evaluation of documents.
“People will talk to computers more and more naturally,” adds Jessica Heesen, media ethicist at the University of Tübingen. A dialogue between a person and a smartphone could then sound like this: “Please find the document that I created for the lecture on Saturday.” The cell phone asks: “Do you mean that with the sales graphic in it?” Language models could become the user interface of the network. “Anyone who makes computers capable of speaking will have a great supremacy in the market,” says Heesen.
Bienert sees this as a challenge for the old continent. “Europe runs the risk of becoming dependent on American products, as it does on search engines,” he warns. US companies would then receive the data from European users and use them to further improve their language models. Bienert fears that a self-reinforcing cycle set in that led to overpowering monopolies.
Therefore, OpenGPT-X should set its own European accents – and learn the other languages of the continent after German, emphasizes project manager Nicolas Flores-Herr from the Fraunhofer Institute for Intelligent Analysis and Information Systems in Sankt Augustin. In addition, the project wants to “implement European values” and make the language model attractive for domestic industry.
“It’s not just about higher, faster, further,” says Flores-Herr. However, size is an important factor. A language model is a gigantic statistic of how words relate to one another. For example, “clouds” will often appear in the context of “raining”. The software learns such relationships from billions of pages of real text from websites and books.
Technically, learning takes place with so-called neural networks. Similar to the brain, artificial neurons exchange signals with one another via synapses. When learning, the individual synapses adjust how strongly they transmit the signal from one neuron to another.
Last year, GPT-3 had the most artificial synapses, 175 billion. This year Wu Dao 2.0 increased that number tenfold to 1.75 trillion. OpenAI is already working on the next leap: GPT-4 is to receive a hundred trillion synapses, as many as in the human brain.
The more artificial synapses a language model has and the more text it has processed during training, the better it can guess how to complete a sentence fragment. For “clouds are coming, it will be soon …”, “raining” would be a good choice. In a wintry context, however, “snowing” would be more appropriate. In this way, the AI lines up word by word so that longer texts are created.
The WDR wants to test whether AI can write table of contents for the media library
The huge neural networks eat up a lot of computing power. Microsoft built what it claims to be the fifth most powerful supercomputer in the world specifically for training GPT-3. In return, the company can offer its customers access to the language model. OpenGPT-X can also use supercomputers, for example at Forschungszentrum Jülich. The aim is to train models and optimize them for specific industrial applications. WDR, for example, is involved in the project as a user.
“We want to try out what we can do with OpenGPT-X,” says Dirk Maroni, head of the information management department at the station. He also sees a leap in quality in AI in the language models. Other algorithms can also generate legible sports or weather reports. “A large language model, however, could make such texts livelier, for example by including the special atmosphere of a local derby.” Language models are also stylistically flexible and can, for example, write texts in plain language.
Maroni’s team wants to test how well OpenGPT-X understands long texts, such as transcripts from podcasts or verbal contributions. The language model could create brief table of contents for the media library. He sees no competition for the company’s journalists, rather support for their work. “The journalists will be able to concentrate more on the content and the creative,” Maroni believes. Maroni thinks a European language model is important. By training with German, French or Italian texts, it would soak up European values.
However, language models also reflect prejudices that circulate in a society. So associated early texts from GPT-3 Occupations with a higher level of education are more likely to work with men than with women. The possible discrimination of groups turns language models into “risky applications of AI”, warns Jessica Heesen. The quality of the texts for the training is crucial. “The data are not simply available,” says the media ethicist. They would have to be carefully collected and selected by people. When training the AI, its own values are incorporated. “You have to learn to deal with things like that,” says Heesen.
The European language computer should write gender-responsively and inclusive
The makers of OpenGPT-X are aware of this. “We will try to get the issue of discrimination under control from the start,” says Flores-Herr. A research project will be dedicated to the gender-sensitive and inclusive language of OpenGPT-X, explains the researcher. The team will filter and rework training data – “refine”, as Flores-Herr says. In contrast to GPT-3, the language software is not supposed to learn the prejudices in the first place.
In order to get enough text in all 24 languages of the EU, “we will have to stretch,” fears Flores-Herr. Wikipedia articles were nowhere near enough. A good source of data is the European Language Grid, an EU-funded project that collects language technologies and data sets. “With this we should have a multilingual model in a few years,” says the researcher.
Additional user data could then be used for specific applications. “For example, knowledge networks,” says Flores-Herr, that is, a network that shows the relationships between concepts. “In this way, the specialist knowledge of a user, for example a bank, could flow into the model,” says Flores-Herr. Since such data feeds valuable knowledge directly into the language model, OpenGPT-X could trump the competition in some niches.
Flores-Herr hopes that interested parties will soon be lining up to use OpenGPT-X. “We are not only targeting industry and research, but also small and medium-sized enterprises,” says the researcher. The new language model will be open to everyone, in the sense of a European infrastructure.