High-level visual representations in the human brain are aligned with large language models by Adrien Doerig & Tim C. Kietzmann & Emily Allen & Yihan Wu & Thomas Naselaris & Kendrick Kay & Ian Charest instant download

High-level visual representations in the human brain are aligned with large language models by Adrien Doerig & Tim C. Kietzmann & Emily Allen & Yihan Wu & Thomas Naselaris & Kendrick Kay & Ian Charest instant download

The human brain extracts complex information from visual inputs, including objects, their spatial and semantic interrelations, and their interactions with the environment. However, a quantitative approach for studying this information remains elusive. Here we test whether the contextual information encoded in large language models (LLMs) is benefcial for modelling the complex visual information extracted by the brain from natural scenes. We show that LLM embeddings of scene captions successfully characterize brain activity evoked by viewing the natural scenes. This mapping captures selectivities of diferent brain areas and is sufciently robust that accurate scene captions can be reconstructed from brain activity. Using carefully controlled model comparisons, we then proceed to show that the accuracy with which LLM representations match brain representations derives from the ability of LLMs to integrate complex information contained in scene captions beyond that conveyed by individual words. Finally, we train deep neural network models to transform image inputs into LLM representations. Remarkably, these networks learn representations that are better aligned with brain representations than a large number of state-of-the-art alternative models, despite being trained on orders-of-magnitude less data. Overall, our results suggest that LLM embeddings of scene captions provide a representational format that accounts for complex information extracted by the brain from visual inputs.