Can neurons speak? Semantic narration of vision at single-cell resolution

2026/06/17 by Arnau Marin-Llobet, Richard Hakim, Sara Matias +3 · 4 voices
#q-bio.NC #q-bio.QM

paper · pdf

Abstract

Identifying what individual neurons encode in higher-order visual cortex is an open problem. Responses resist intuitive parameterization, and the deep-network embeddings used in their place are black boxes. Here, we introduce NEURRATOR, a framework that decodes spiking activity into free-form natural-language narration of the viewed scene at single-neuron resolution. A learned encoder maps spike trains from arbitrary subsets of simultaneously-recorded neurons into the patch-embedding space of a frozen CLIP, from which a multimodal language model and sparse autoencoder generates and validates a description with no language-side training. Applied to Neuropixel recordings of mouse visual cortex during natural-movie viewing, NEURRATOR narrates from thousands of neurons, singular cortical regions, local populations, or from a molecularly-defined cell-types. We use this property to (i) quantify how decoding fidelity scales with population size and cortical region, and (ii) "neurrate", in plain language, what individual neurons and genetically-tagged inhibitory cell-types contribute to visual representation. This recasts cell identity from a classification target into a functional probe of the visual system, providing a new unit of biological insights in neural systems.

Citations

• Beyond Black Boxes: Enhancing Interpretability of Transformers Trained on Neural Data
• Steering CLIP's vision transformer with sparse autoencoders
• Neural models for detection and classification of brain states and transitions
• Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models
• InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders
• Scaling and evaluating sparse autoencoders
• MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
• BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity
• Improved Baselines with Visual Instruction Tuning
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Brain encoding models based on multimodal transformers can transfer across language and vision
• Sigmoid Loss for Language Image Pre-Training
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Multiscale Voxel Based Decoding For Enhanced Natural Image Reconstruction From Brain Activity
• Flamingo: a Visual Language Model for Few-Shot Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Survey of spiking in the mouse visual system reveals functional hierarchy
• Neuropixels 2.0: A miniaturized high-density probe for stable, long-term brain recordings
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Interpreting Encoding and Decoding Models
• Interpreting encoding and decoding models
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge

Discussions

• Check the preprint!! arxiv.org/abs/2606.18667 Shout out to all the amazing collaborators in this work Richard Hakim, Sara Matias, Na Li, Venki Murthy, Demba Ba!!!!! @harvard.edu @harvardbrainsci.bsky. [bsky, 4 points, 0 comments]
• Can neurons speak? Semantic narration of vision at single-cell resolution arxiv.org/abs/2606.18667 by @arnauya.bsky.social et al.; thread at bsky.app/profile/arna... [bsky, 4 points, 0 comments]
• Can neurons speak? Semantic narration of vision at single-cell resolution https://arxiv.org/abs/2606.18667 https://www.infiniteawareness.online/ [bsky, 0 points, 0 comments]
• Can neurons speak? Semantic narration of vision at single-cell resolution https://arxiv.org/abs/2606.18667 [bsky, 0 points, 0 comments]

Related

• Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction
• Natural Language Descriptions of Deep Visual Features
• MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs
• A Goal-Driven Approach to Systems Neuroscience
• CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks