Classic brain-computer interfaces like Neuralink help paralyzed people communicate or move prosthetics just by thinking, but they can’t do both at the same time because they interpret the signal from one brain area at a time.
Now a team of scientists from the University of California San Francisco has made a notable step forward in this sense, developing a neural implant capable of simultaneously decoding brain signals relating to speech and torso movements: the ones nodding and greeting, however, are not the patients, but an avatar on a screen.
To train the computer models to correctly interpret their words and gestures, the volunteers thought of up to 10 gestures and 10 words or sentences while their brain activity was recorded. By applying a grid of electrodes to the brains of paralyzed people, the researchers were able to record both speech and upper body movement signals, and decoded them with a machine learning device.
Combinations included even without training
The interesting thing is that, to work, the system did not necessarily have to record every possible combination of words and gestures during training (for example “hello” and waving), but was able to decode even combinations not presented during training.
This is an aspect that should not be underestimated, because while in this case training the machine on a hundred test combinations was relatively easy, if in the future one wanted to expand the vocabulary to make communication more fluid and complete it would be impossible to record all the possible combinations of gestures and words.
Towards better communication
Speaking through an avatar is certainly not like moving and communicating in person, but at the moment it is the best technology can give us. “The goal of our research is to help paralyzed people express themselves even more fully,” explains Sergey Stavinsky, a neural interfaces researcher at UC Davis who was not involved in the study.
According to Edward Chang, coordinator of the research, “these digital avatars are a first step towards real robotics”. And who knows, one day technology will allow those who have lost the use of their bodies to express themselves not only through a virtual figure, but physically with, for example, an exoskeleton.
