Scientists Demonstrate New ‘Mind Captioning’ System That Converts Brain Activity Into Text

Scientists Demonstrate New ‘Mind Captioning’ System That Converts Brain Activity Into Text
————————-
Researchers have developed a technique known as “mind captioning” that uses fMRI brain scans and language models to generate short text descriptions of scenes a person is viewing or recalling, Earth.com reported.

The study, published in Science Advances, involved six volunteers who watched thousands of brief video clips while their brain activity was recorded. Each clip had a human-written caption, allowing a decoder to learn how specific fMRI patterns corresponded to sentence meanings.

A second language model then converted those decoded representations into text, refining each sentence until it aligned with the predicted meaning. The system often produced simplified but accurate descriptions, sometimes misidentifying objects yet capturing overall actions and relationships.

In additional tests, the method generated sentences based on brain activity recorded during silent recall, performing above chance even without visual input. Researchers found that high-level visual and parietal regions carried key semantic information. The technology could eventually support communication for people unable to speak, though it remains in early stages.