Brain-IT Reconstructs Viewed Images From fMRI With Less Subject-Specific Data

Researchers at the Weizmann Institute of Science developed Brain-IT, an artificial intelligence system that reconstructs images viewed by a person from functional magnetic resonance imaging data, using less subject-specific scanning data than earlier methods.

The system converts activity across brain voxels into localized image features through a Brain Interaction Transformer. One pathway focuses on coarse layout and visual structure, while another uses semantic features to guide diffusion-based image generation.

Brain-IT was tested on the Natural Scenes Dataset, which includes eight participants who viewed thousands of images. Each participant completed 30 to 40 scanning sessions, with six scans of about 10 minutes per session and roughly 40 images per scan. The dataset contains about 73,000 image-fMRI pairs.

Across participants, Brain-IT organized approximately 40,000 measured voxels into 128 common functional groupings. On the full-data benchmark, it recorded a 0.386 PixCorr score and a 0.486 SSIM score. Its other reported results included 98.4% on Alex(2), 99.5% on Alex(5), 97.3% on Inception and 96.4% on CLIP.

In tests, one hour of fMRI data from a new subject produced results comparable with methods trained on 40 hours of recordings. Meaningful reconstructions also appeared in transfer-learning experiments using 15 minutes of recordings.

“The new model we developed outperforms them in reconstructing both the content of the image and its details. What’s more, while every other model requires dozens of hours of brain scans to learn to ‘read’ a new person, our model needs only one hour,” Michal Irani said.

The reported experiments involve fixed images viewed under controlled scanning conditions. They do not demonstrate unrestricted decoding of thoughts, memories or dreams.

The team is also working on auditory decoding. Video decoding remains more difficult because fMRI measurements take about two seconds, while video can change many times per second.

“What remains especially challenging is decoding video – for example, during dreaming. Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it’s possible that in the future we may even be able to read dreams,” Irani said.