PhD thesis

Here you can download a digital version of my thesis

If you would like a printed version, let me know and I’ll try to get you one!

READING BRAINS FROM THE HUMAN BRAIN
By analysing MRI images of the brain with an elegant mathematical model, it is possible to reconstruct thoughts more accurately than ever before. We have succeeded in determining which letter a test subject was looking at.

Functional MRI has been used in cognition research primarily to determine which brain areas are active while test subjects perform a specific task. The question is simple: is a particular brain region on or off? We have gone a step further. We have used data from the scanner to determine what a test subject is looking at

Bayesian framework

We ‘taught’ a model how small volumes of 2x2x2 mm from the brain scans – known as voxels – respond to individual pixels. By combining all the information about the pixels from the voxels, it became possible to reconstruct the image viewed by the subject. The result was not a clear image, but a somewhat fuzzy speckle pattern.

To obtain a clear image we give the model prior knowledge: we teach the model what letters look like. This improves the recognition enormously. The model compares the letters to determine which ones correspond best to the speckle image, and then pushes the result of the image towards those selected letters. The result was the actual letter, a true reconstruction.

Gaussian Mixture Models

In order to improve the reconstructions we then made the prior more informative by grouping the letters in the prior in the six letter categories: B, R, A, I, N and S. So in this case we have six separate priors which result in 6 Gaussian mixture models. Now the model compares the speckle image to each of the six priors and then determines which prior is fitting best. Then only the prior that is most similar will get to influence the image, leading to improved reconstructions.

Semantic gating

A further improvement can be achieved by learning categorical information from other brain areas. Thus far the model only got brain data from V1 of the visual cortex as input. But we can learn how categories are represented in higher brain areas and send that knowledge to the model. This is called gating of semantic information, which translates to sending information about the category to the model.

Decoding unlabelled data

We want this model to be applicable to a large body of images, but often labelling is not available for large image databases. Therefore we investigated if we could reach the same quality in reconstruction as before with unlabelled data. We use a clustering algorithm to compute sub groups in a set of prior images. We found that the same quality of reconstructions can be achieved and the quality even increased when we choose a greater number of subgroups than the previous 6 known letter categories. The clustering algorithm found subgroups within the letter categories.

Why is this interesting?

Our approach is similar to how we believe the brain combines prior knowledge with sensory information. For example, you can recognize the lines and curves in this text as letters only after you have learned to read.

Furthermore, we hope to use the model to decode experiences like imagination or dreams and see what is precisely happening in the brain during these subjective experiences. The model can decode very quickly, so maybe decoding of thoughts while thinking will become feasible. Then we could help paralysed patients that have lost the ability to communicate with their body.

Data & Code

Data that has been collected in the context of this project can be downloaded here.

Software that has been developed for this project can be downloaded here.

Please refer to our work when using the data or code for your own purposes. And please let us know if you obtain interesting results. We love to hear that our work inspired others.

Papers

Schoenmakers, S, Heskes, T, van Gerven, MAJ.
Hidden Markov models for reading words from the human brain
Academia.edu, 2015

Schoenmakers, S, van Gerven, MAJ.
ICA Allows Rapid Identification of Functional ROIs in Task-Based fMRI Data at 3T and 7T
Academia.edu, 2015

Schoenmakers, S, Güçlü, U, van Gerven, MAJ, Heskes, T.
Gaussian mixture models and semantic gating improve reconstructions from human brain activity
Frontiers in Computational Neuroscience. 2015. 8(173).
Direct download

Schoenmakers, S, van Gerven, MAJ, Heskes, T.
Gaussian mixture models improve fMRI-based image reconstruction
IEEE Xplore, 2014
Direct download

Schoenmakers, S, Barth, M, Heskes, T, van Gerven, MAJ.
Linear reconstruction of perceived images from human brain activity
Neuroimage. 2013; 83:951-961.
Direct download