Call it “mindwriting.” The combination of mental effort and state-of-the-art technology have allowed a man with spinal injury and immobilized limbs, to communicate by text at speeds rivaling those achieved by his able-bodied peers texting on a smartphone. Scientists at Stanford University, Howard Hughes Medical Institute, and Brown University, developed an implanted brain-computer interface (BCI) technology that uses artificial intelligence to convert brain signals generated when someone visualizes the process of handwriting, into text on a computer, in real time.
The team now reports on a trial in which a paralyzed clinical trial participant with a BCI implant was able to “type” words on a computer by merely thinking about the hand motions involved in creating written letters. The software effectively decoded the information from the BCI to quickly convert the man’s thoughts about handwriting into text on a computer screen. In the reported study, the 65-year-old male participant achieved a typing rate of 90 characters per minute, more than double the previous record for typing with a brain-computer interface.
“This approach allowed a person with paralysis to compose sentences at speeds nearly comparable to those of able-bodied adults of the same age typing on a smartphone,” commented Stanford University neurosurgery professor, Jaimie Henderson, MD, the John and Jene Blume—Robert and Ruth Halperin Professor. “The goal is to restore the ability to communicate by text.”
The system is so fast because each letter elicits a highly distinctive activity pattern, making it relatively easy for the algorithm to distinguish one from another, suggested Frank Willett, PhD, a Stanford, and Howard Hughes Medical Institute research scientist, and first author of the team’s published paper in Nature.
The new study is part of the BrainGate clinical trial, directed by Leigh Hochberg, MD, a neurologist and neuroscientist at Massachusetts General Hospital, Brown University, and the Providence VA Medical Center. The BrainGate collaboration has been working for several years on systems that enable people to generate text through direct brain control. Previous incarnations have involved trial participants thinking about the motions involved in pointing to and clicking letters on a virtual keyboard. That system enabled one participant to type 40 characters per minute, which was the previous record speed.
Willet noted, “We want to find new ways of letting people communicate faster. This new system uses both the rich neural activity recorded by intracortical electrodes and the power of language models that, when applied to the neurally decoded letters, can create rapid and accurate text.”
In their published study, which is titled, “High-performance brain-to-text communication via handwriting,” the team concluded, “Here, we introduced a new approach for communication BCIs—decoding a rapid, dexterous motor behavior in an individual with tetraplegia—that sets a benchmark for communication rate at 90 characters per minute. The real-time system is general (the user can express any sentence), easy to use (entirely self-paced and the eyes are free to move) and accurate enough to be useful in the real-world.”
When a person becomes paralyzed due to spinal cord injury, the part of the brain that controls movement still works. This means that, while the participant in the trial, referred to as T5, could not move his hand or arm to write, his brain still produced similar signals related to the intended movement. Similar BCI systems have been developed to restore motor function through devices like robotic arms. As the authors noted in their Nature paper, “… previous BCI studies have shown that the motor intention for gross motor skills, such as reaching, grasping, or moving a computer cursor, remains neurally encoded in the motor cortex after paralysis.”
Study co-author Krishna Shenoy, PhD, a Howard Hughes Medical Institute (HHMI) Investigator and the Hong Seh and Vivian W. M. Lim Professor at Stanford University, stated, “Just think about how much of your day is spent on a computer or communicating with another person. Restoring the ability of people who have lost their independence to interact with computers and others is extremely important, and that is what is bringing projects like this one front and center. Shenoy and Henderson, who have been collaborating on BCIs since 2005, are the senior co-authors of the new study.
While a major focus of BCI research has been on restoring gross motor skills, as the team further noted, “rapid sequences of highly dexterous behaviors, such as handwriting or touch typing, might enable faster rates of communication.” What wasn’t known, they pointed out, was whether “… the neural representation for a rapid and highly dexterous motor skill, such as handwriting, also remains intact.”
The participant in the newly reported BrainGate2 trial had lost practically all movement below the neck because of a spinal-cord injury in 2007. Nine years later, Henderson placed two brain-computer-interface chips, each the size of a baby aspirin, on the left side of the participant’s brain. Each chip has 100 electrodes that pick up signals from neurons firing in the part of the motor cortex—a region of the brain’s outermost surface—that governs hand movement.
Those neural signals are sent via wires to a computer, where artificial intelligence algorithms decode the signals to surmise T5’s intended hand and finger motion. The algorithms were designed in Stanford’s Neural Prosthetics Translational Lab, co-directed by Henderson and Shenoy, professor of electrical engineering and the Hong Seh and Vivian W. M. Lim Professor of Engineering.
“We’ve learned that the brain retains its ability to prescribe fine movements a full decade after the body has lost its ability to execute those movements,” Willett said. “And we’ve learned that complicated intended motions involving changing speeds and curved trajectories, like handwriting, can be interpreted more easily and more rapidly by the artificial-intelligence algorithms we’re using than can simpler intended motions like moving a cursor in a straight path at a steady speed. Alphabetical letters are different from one another, so they’re easier to tell apart.”
In a previous 2017 study, three participants with limb paralysis, including T5—all with BCIs placed in the motor cortex—were asked to concentrate on using an arm and hand to move a cursor from one key to the next on a computer-screen keyboard display, then to focus on clicking on that key.
So if the paradigm underlying the 2017 study was analogous to typing, the model for the newly reported study in Nature is analogous to handwriting. T5 concentrated on trying to write individual letters of the alphabet on an imaginary legal pad with an imaginary pen, despite his inability to move his arm or hand. He repeated each letter 10 times, permitting the software to “learn” to recognize the neural signals associated with his effort to write that particular letter.
In numerous multi-hour sessions that followed, T5 was presented with groups of sentences and instructed to make a mental effort to “handwrite” each one. No uppercase letters were employed. Examples of the sentences were “i interrupted, unable to keep silent,” and “within thirty seconds the army had landed.” Over time, the algorithms improved their ability to differentiate among the neural firing patterns typifying different characters. The algorithms’ interpretation of whatever letter T5 was attempting to write appeared on the computer screen after a roughly half-second delay.
In further sessions, T5 was instructed to copy sentences the algorithms had never been exposed to. He was eventually able to generate 90 characters, or about 18 words, per minute. Later, asked to give his answers to open-ended questions, which required some pauses for thought, he generated 73.8 characters (close to 15 words, on average) per minute, tripling the previous free-composition record set in the 2017 study. For comparison, some of his able-bodied peers would be texting on a smartphone at about 23 words per minute.
T5’s sentence-copying error rate was about one mistake in every 18 or 19 attempted characters. His free-composition error rate was about one in every 11 or 12 characters. When the researchers used an after-the-fact autocorrect function—similar to the ones incorporated into our smartphone keyboards—to clean things up, those error rates were markedly lower: below 1% for copying, and just over 2% for freestyle.
Willett commented, “This method is a marked improvement over existing communication BCIs that rely on using the brain to move a cursor to “type” words on a screen. Attempting to write each letter produces a unique pattern of activity in the brain, making it easier for the computer to identify what is being written with much greater accuracy and speed.” The authors further noted, “These results suggest that time-varying patterns of movement, such as handwritten letters, are fundamentally easier to decode than point-to-point movements. We think this is one—but not necessarily the only—important factor that enabled a handwriting BCI to go faster than continuous-motion point-and-click BCIs.”
Shenoy noted, “Right now, other investigators can achieve about a 50-word dictionary using machine learning methods when decoding speech. By using handwriting to record from hundreds of individual neurons, we can write any letter and thus any word which provides a truly ‘open vocabulary’ that can be used in most any life situation … While handwriting can approach 20 words per minute, we tend to speak around 125 words per minute, and this is another exciting direction that complements handwriting. If combined, these systems could together offer even more options for patients to communicate effectively.”
BrainGate2, a collaboration of internationally recognized laboratories, universities, and hospitals working to advance brain-computer interface technologies, is testing the safety of BCIs that directly connect a person’s brain to a computer. The study was a collaboration between the research groups of Shenoy and Henderson, at Stanford University, together with Leigh Hochberg, MD, PhD, from Brown University, Massachusetts General Hospital, and Providence VA, and sponsor-investigator of the BrainGate2 trial. Henderson at performed the surgery to place the necessary electrodes.
“An important mission of our BrainGate consortium research is to restore rapid, intuitive communication for people with severe speech or motor impairments,” said Hochberg, who also directs the Center for Neurotechnology and Neurorecovery at Massachusetts General Hospital and the VA RR&D Center for Neurorestoration and Neurotechnology at the Dept. of Veterans Affairs Providence Healthcare System. “Frank’s demonstration of fast, accurate neural decoding of handwriting marks an exciting new chapter in the development of clinically useful neurotechnologies.”
The authors do acknowledge that further developments will be needed to generate a system that can be used clinically. “It is important to recognize that the current system is a proof of concept that a high-performance handwriting BCI is possible (in a single participant); it is not yet a complete, clinically viable system,” they wrote. “More work is needed to demonstrate high performance in additional people, expand the character set (for example, capital letters), enable text editing and deletion, and maintain robustness to changes in neural activity without interrupting the user for decoder retraining.” Nevertheless, they stated, “… we believe that the future of intracortical BCIs is bright.”
Shenoy’s team envisions using attempted handwriting for text entry as part of a more comprehensive system that also includes point-and-click navigation, much like that used on current smartphones, and even attempted speech decoding. “Having those two or three modes and switching between them is something we naturally do,” he says.
Next, Shenoy says, the team intends to work with a participant who cannot speak, such as someone with amyotrophic lateral sclerosis, a degenerative neurological disorder that results in the loss of movement and speech. In addition, they are looking to increase the number of characters available to the participants, such as capital letters and numbers.
