Dusseldorf A few months ago, Yu Takagi caught his breath. The photos on his screen looked so compelling. The Japanese researcher went into the bathroom, looked in the mirror and thought, “Okay, the reflection looks normal. Maybe I’m not going crazy after all.”
The reason for the dismay: An AI model had translated the test subjects’ brain signals into images – and they were amazingly similar to the photos previously viewed. “I really didn’t expect that,” says the neuroscientist and prospective professor at Osaka University.
The scientific field is called “Neural Decoding”. Artificial intelligence (AI) can use the brain activities of people to reconstruct not only previously seen images, but also heard music or speech. “Progress is exponential,” says Sead Ahmetovic, head of We Are Developers, Europe’s largest developer platform.
All over the world, not only computer scientists and neuroscientists are making more and more improvements to the process. Start-ups like Neuralink or Kernel and tech companies are also involved – even if commercial applications are still far in the future. It is about the ultimate communication between man and machine.
Takagi was inundated with inquiries after the publication of his study in September 2022. Marketing companies in particular wanted to use the technology to research how well their advertising was received. Defense companies were also there, says the Japanese. But: “That’s not possible for ethical reasons, everyone has to agree to an investigation – even if it’s the enemy.”
The interest of companies or venture capitalists in Takagi’s study entitled “High-resolution image reconstruction with latent diffusion models from human brain activity” is understandable given the many potential applications.
Takagi and his colleague Shinji Nishimoto carried out the “high-resolution image reconstruction” of brain activities with the program Stable Diffusion, which was developed at the LMU Munich and brought to market by the English start-up Stable AI.
>> Read also: Neuro chips – The hardware of the future for AI could come from Europe
“I’m surprised what different applications are made with stable diffusion,” says Björn Ommer, Professor of Computer Science at LMU Munich. The deep learning model is primarily there to generate images from text input. Takagi and his colleague put a kind of translator in front of Stable Diffusion that makes the brain signals understandable for the AI. “It certainly can’t read minds,” says Ommer. “But it’s exciting that it works.”
AI can reconstruct heard speech
As exciting as the results are, much remains unclear. “We don’t know exactly how the brain creates shapes or colors,” says Thirza Dado, a cognitive neuroscientist at Radbout University in Nijmegen. “But the AI is getting better and better at recognizing the patterns.”
No one knows all the details of what’s going on in both the brain and the neural networks – but the results are just fine. In a study published a few days ago, researchers at the University of Texas let three volunteers listen to podcasts and lengthy radio reports for 16 hours. A speech model evaluated the signals read out by the MRT.
The Transformer – an AI similar to that of OpenAI’s ChatGPT or Google’s Bard – was not always able to decode the brain signals correctly using the patterns, but it was surprisingly accurate. For example, one participant heard the phrase “I don’t have a driver’s license yet”. The AI read it like this: “It started learning to drive again.”
As a student, Thirza Dado was asked what she wanted to do later. “I want to build a dream machine,” she said in 2008, more for fun. Today, the 30-year-old is closer to this vision than she ever thought. She does research in the Neural Coding Lab at Radbout University and, like her colleague Takagi in Japan, uses AI models to reconstruct images.
The Neural Coding Lab, led by Dr. Umut Güclü already uses the method when reconstructing films that she has seen. The next step would be to reconstruct the images that humans only imagine. After that, one could theoretically record and evaluate dreams. According to the researcher, this is still far in the future. But even if it did, you probably never know for sure if it really was our dream.
>> Read also: How to get ChatGPT to stop writing so much nonsense
The neuroscientists measure the brain activities with a so-called fMRT – the “functional magnetic resonance tomograph”. When the brain becomes active in certain areas, it needs oxygen. Blood flows into these areas and the researchers track its movement using the iron found in red blood cells. “A direct measurement of the neuronal activities by an implant would lead to much more precise results,” says the doctoral student. “But that is still very dangerous for humans.”
Elon Musk’s Neuralink undergoes clinical testing
However, some people might be willing to take the risk if they have amyotrophic lateral sclerosis (ALS), for example. This is an incurable, serious disease of the nervous system. A few days ago, Neuralink, the biotech company founded by Tesla boss Elon Musk in 2016, received approval from the US health authority for a clinical test for an implant – a major step that no biotech company has managed to date.
Neuralink wants to use a so-called “Brain Computer Interface” (BCI) for the patient. The “brain-computer interface” is called N1 and listens to the electrical signals of the nerve cells at 1024 points – or, conversely, stimulates them with electrical impulses.
The N1 has a diameter of around 2.5 centimeters and works wirelessly. Experiments with macaques have been running since 2019. For example, the monkeys can use a BCI to control a ping-pong video game. However, Professor Ommer also warns: “Neuralink is a leader in PR communications, but not necessarily in scientific research.”
The surgical procedure carries risks that the researcher considers unacceptable in healthy people. Therefore, many companies and researchers are working on new sensors and technologies that can measure brain activity with less effort. For example, “near-infrared spectroscopy,” where the test subjects wear just a type of cap that’s connected to the computer.
Neural Decoding: Commercial application is still far in the future
According to Osaka University’s Takagi, it will be at least a decade before commercially viable methods can be expected.
One of the 34-year-old’s hopes: to decode the brain activities of animals. “I’d like to know what images my cat Lolo has in mind when she sees me,” says Takagi. His hope: Lolo sees his face as it is. His fear: The AI shows a treat.
More: How I tried to become immortal through AI.
First publication: 07/02/2023, 09:50 a.m.