Our research is broadly concerned with how the human brain processes and represents the natural world. In particular, we want to understand how language is processed and represented by the cortex, how those representations are grounded in other modalities such as vision and speech, and how the brain uses them to build meaning and memory over time.

The main method that we employ is the encoding model, a mathematical model that learns to predict how the brain will respond to new stimuli (such as language) based on large amounts of data. These models can then be tested for accuracy by checking how well they can predict responses in a new dataset that they were not trained on before. In general we train and test our encoding models using natural stimuli such as narrative stories or podcasts. Together, encoding models and natural stimuli give a natural gradient along which research can progress: if we can build a model that is better able to predict the brain, then we have gained some understanding of how the brain works.

This work employs a wide variety of tools drawn from machine learning, natural language processing, applied mathematics, computer graphics, physics, and neuroscience. To stay on the cutting edge of these technologies, one focus of this lab is also to develop new tools and apply them to neuroscience problems. It is our long-term goal to use the neuroscience data and results to build better algorithms and smarter machines. How is the meaning of language represented in the human brain? In earlier work we explored representations of meaning at the scale of single words, revealing complex maps across association cortex. Building on this, we now use large language models to study how phrases, sentences, and longer context are represented by activity in the human cortex.

Testing these rich, nonlinear models of language requires enormous amounts of data. We have collected some of the largest single-subject fMRI language datasets ever assembled and share them freely [LeBel et al., 2023]. Using these data, we have shown that the accuracy of language encoding models follows predictable scaling laws as both the language model and the amount of brain data grow [Antonello et al., 2024], giving us a roadmap for building ever-better models of the language system. If our encoding models truly capture how the brain represents language, we should be able to run them in reverse to decode language directly from brain activity. We showed that continuous language — the gist of what a person is hearing, imagining, or watching — can be reconstructed from non-invasive fMRI recordings [Tang et al., 2023]. More recently we have worked to make these decoders more practical, showing that they can transfer across participants and across stimulus modalities [Tang & Huth, 2025]. This line of work opens a path toward semantic brain-computer interfaces while also raising important questions about mental privacy. Language serves as a gateway to cognition. Words have the capacity to elicit incredibly complex cognitive processing in our heads, giving experimental access to many different aspects of cognition. Thus, by studying how the brain represents the meaning of language we can simultaneously explore how and where in the cortex many different cognitive processes function. We are particularly interested in the interplay between language and other modalities, such as vision, spatial reasoning, and somatosensory processing. We have found that encoding models built from multimodal transformers can transfer between language and vision [Tang et al., 2024], and that semantic and affective tuning to natural images predicts how people behave toward them [Abdel-Ghaffar et al., 2024]. Before language can be understood, the brain must transform raw sound into meaning. We use self-supervised speech models to trace this transformation, showing that models of audio effectively explain cortical responses to speech [Vaidya et al., 2022] and that these representations reveal a full acoustic-to-semantic hierarchy in speech-processing cortex [Guo & Huth, 2026]. We are also using brain responses to improve the machine models themselves, fine-tuning speech representations with brain data so that they better match the human auditory and language systems [Vattikonda et al., 2025]. Neural network models predict brain activity remarkably well, but predictive accuracy alone does not tell us why. A major focus of the lab is turning these black-box models into scientific understanding. We have argued that language models fit brain data not because of predictive coding but because they capture a broad variety of linguistic features [Antonello & Huth, 2023]. To make model-based explanations testable, we develop tools such as generative causal testing, which uses LLMs to generate stimuli that verify hypotheses about what drives a brain region [Antonello et al., 2025], and interpretable embeddings built by asking LLMs questions about language [Benara et al., 2024]. More broadly, we see accurate encoding models as a platform for in silico experimentation — simulating neuroscience experiments to generate and refine hypotheses before ever entering the scanner [Jain et al., 2023]. Understanding a story requires holding on to what came before. We study how the brain builds and retains memories from continuous natural experience, finding that efficient uniform sampling explains why some parts of narratives are remembered better than others [Mu et al., 2025]. We also bring these questions to artificial intelligence, developing tasks to assess episodic memory in large language models [Pink et al., 2024] and arguing that episodic memory is the missing piece for long-term LLM agents [Pink et al., 2025].

Interested in these questions? We share code and tutorials to help you get started with encoding models for language.