2025/11/05 by Tomoyasu Horikawa · 1 voice · 3 citations
Computer Science · Neuroscience · Psychology · #Action Observation and Synchronization #Brain activity and meditation #Cognition #Content (measure theory) #Decoding methods #Expression (computer science) #Human brain #Interface (matter) #Mirroring #Multimodal Machine Learning Applications #Neurobiology of Language and Bilingualism #Semantics (computer science)
paper · pdf · doi:10.1126/sciadv.adw1464
published in Science Advances 11(45), eadw1464 (American Association for the Advancement of Science)
openalex created_date 2025/11/05 · openalex publication_date 2025/11/05 · openalex updated_date 2026/07/04
A central challenge in neuroscience is decoding brain activity to uncover mental content comprising multiple components and their interactions. Despite progress in decoding language-related information from human brain activity, generating comprehensive descriptions of complex mental content associated with structured visual semantics remains challenging. We present a method that generates descriptive text mirroring brain representations via semantic features computed by a deep language model. Constructing linear decoding models to translate brain activity induced by videos into semantic features of corresponding captions, we optimized candidate descriptions by aligning their features with brain-decoded features through word replacement and interpolation. This process yielded well-structured descriptions that accurately capture viewed content, even without relying on the canonical language network. The method also generalized to verbalize recalled content, functioning as an interpretive interface between mental representations and text and simultaneously demonstrating the potential for nonverbal thought-based brain-to-text communication, which could provide an alternative communication pathway for individuals with language expression difficulties, such as aphasia.