Please stop reading this text for a moment. Look at the pictures you see surrounding this text here, on the website or in the printed SZ. So you look at these pictures, maybe you see a politician, an advertisement, or a brown bear. Millions and millions of nerve cells are now active in your brain, making sure that you understand what you see: politician, advertisement, brown bear.
Now imagine that a computer is able, based on the blood flow in your brain, to reproduce and display the images you have just seen one-to-one: politician, advertisement, brown bear. Gruesome? Japanese researchers from Osaka University succeeded in doing this in an experiment with the help of artificial intelligence. While not entirely accurate, the reconstructed AI images closely resemble the originals.
At least since last year, the rapid development of AI systems has surprised, excited and frightened the general public. Programs like ChatGPT or Dall-E write texts on command, create pictures or compose songs, sometimes breathtakingly well. The systems are able to do this primarily because they learn from vast amounts of data from the Internet. The technology is constantly improving. In the meantime, their developers are sometimes unsure what else will be possible with these programs. So is AI even able to read human minds?
The basic question is: How does the brain recognize, for example, that a bear is not a politician?
For decades, researchers have been trying to decipher how the brain works and how it speaks. A difficult undertaking, science still only understands a fraction of how the human brain works, billions of nerve cells form a highly complex network. In comparison, a computer still looks like a lame duck.
Still.
The two neuroscientists and authors of the study, Yu Takagi and Shinji Nishimoto, want to contribute to decoding the brain and how it works. In your research, you are concerned with the big question of how the human brain processes visual stimuli. The basic question sounds banal, but it is difficult to answer: How does a person actually recognize that a brown bear is not a politician?
For their latest study, which was published as a preprint at the end of last year and will be presented at the important AI conference CVPR in Vancouver in June, Takagi and Nishimoto used a large data set from the University of Minnesota in the USA.
This data set originally comes from another research project, US colleagues of the two Japanese had measured and recorded the blood flow in the brain of four subjects while showing them thousands of photos. A procedure called functional magnetic resonance imaging (fMRI). Depending on which photo a subject had just seen, different regions in the brain were activated and blood flowed more strongly – which the US researchers were able to measure.
The two neuroscientists from Japan used this data from their US colleagues and, when evaluating the fMRI scans, concentrated on two areas of the brain that become particularly active when it comes to processing visual stimuli. One area of the brain is credited with processing the content of the images shown, for example: brown bear or human? The other brain area processes the information about where objects are in the field of vision, how big they are and what shape they have.
Takagi and Nishimoto now tried to turn the process around again and to reconstruct the images originally shown to the subjects – in this case including a teddy bear, an airplane and a steam locomotive – from the fMRI scans.
“To a certain extent, you can use brain data to find out what someone is thinking, dreaming or seeing.”
In order to get back to the images shown from the brain scans, the researchers used an AI image generator called “Stable Diffusion”. This was developed by a research group at the Ludwig-Maximilians-University Munich (LMU) and published freely accessible in 2022. With a text command, “Stable Diffusion” can be used to create a new, photorealistic image or to change an existing image. For example, the command could be: “An astronaut should ride through space on a gray horse!” Or: “Paint an otter with a pearl earring in the style of painter Johannes Vermeer!”
The special thing about the AI system is that it can “process” text and images. You can create a new image with a text command, or change an initial image with a text command – for example, add a pearl earring to an existing otter image. To do this, the program first has to translate the text command or the original image into another mathematical representation that it can do something with. For that it has standard translators.
These standard translators could not be used to process the brain scan recordings. So the two neuroscientists trained new translators with thousands of fMRI scans of one subject each and the associated images. In this way, the AI can process an fMRI scan from the region of the visual cortex, which is more responsible for shapes and positions, in a similar way to an image. And a scan from the region that is responsible for the content of the image, just like a text (“locomotive” or “teddy bear”). Even with one of the two inputs, an approximately recognizable image could sometimes be reconstructed, but the results were far better when the researchers combined both.
So has it achieved what some AI fans dream of and others fear: that computers are able to read human thoughts? A call to brain researcher John-Dylan Haynes from the Charité Berlin: “To a certain extent, you can find out what someone thinks, dreams or sees from brain data,” says Haynes. “But there are major limitations.” After all, feeding the AI with training data has its limits, Haynes says: “A human being can see or think so many different things.” It’s simply not possible to measure all of this – because you can’t put every person in a high-performance MRI all the time.
That is why researchers are trying to understand the basic principle with which the brain processes information. According to Haynes, you try to understand the language of the brain in relation to very specific thoughts – in this case, which images people see. “We can use generative AI to capture much more detail about how the brain stores information,” says Haynes.
For example, it is now possible to reconstruct the image of a church clock from the brain scans, although the new translator’s training data did not contain such images; it was sufficient to show the subjects clocks and towers at this stage. This approach is called generative AI because these systems are used to create something new.
Brains store information in different ways
“Generative AI has made a lot of things easier here,” says Björn Ommer, whose research group developed the “Stable Diffusion” system at LMU. Although the fMRI data from the Japanese study didn’t contain particularly detailed information, Ommer said he was amazed at how well the program was able to convert the brain scans into images. “This shows how broadly our Generative AI can be applied.” The study by the Japanese researchers is a good example of how existing AI systems can be used more widely in the future. Because in this case Stable Diffusion itself was not changed at all, the researchers just switched the program to new translators. This is a rarity, because previously entire AI systems had to be rebuilt and trained for similar tests.
But despite all the success, Ommer and Haynes agree: AI can’t read minds. “It’s one thing to reconstruct an image,” says Haynes, “but it’s far from being a reliable means of reconstructing a correct thought, an emotional intention or a lie.” Because brains are very different, like a fingerprint. Their anatomy, but also the way they store information. The model of a patient’s brain cannot be easily transferred to a new person.
Studies have shown for some time that perceptions can be read from brain activity. Using a similar approach, a team led by Alexander Huth from the University of Texas at Austin has now used a language model to deduce what a person heard or thought from fMRI recordings. as the researchers on Monday in Nature Neuroscience reported. Already in 2008 determined researchers from fMRI images, which a subject had seen from several previously unknown images. Shinji Nishimoto, then still at the University of California at Berkeley, was able to use brain data as early as 2011 reconstruct simple videosthat the subjects had seen. Even then, it became apparent what potential AI had for brain research.
Nevertheless, the current study caused a stir worldwide – something to the surprise of the two authors. “We don’t read minds,” says Shinji Nishimoto of the SZ, which is a misleading term anyway. Their goal is simply to better understand the connection between perception and brain activity. One day it might be possible to develop human-machine interfaces with which completely paralyzed patients, for example, can communicate again. The two researchers, like many of their colleagues, emphasize that AI should help people, not harm them. “Whether the technology is used for evil purposes depends on how people use it.”