Papers
Hidden Markov models for reading words from the human brain
Recent work has shown that it is possible to re- construct perceived stimuli from human brain activity. At the same time, studies have indicated that perception and imagery share the same neural substrate. This could bring cognitive brain computer interfaces (BCIs) that are driven by direct readout of mental images within reach. A desirable feature of such BCIs is that subjects gain the ability to construct arbitrary messages.
In this study, we explore whether words can be generated from neural activity patterns that reflect the perception of individual characters. To this end, we developed a graphical model where low-level properties of individual characters are represented via Gaussian mixture models and high-level properties reflecting character co-occurrences are represented via a hidden Markov model. With this work we provide the initial outline of a model that could allow the development of cognitive BCIs driven by direct decoding of internally generated messages.
ICA Allows Rapid Identification of Functional ROIs in Task-Based fMRI Data at 3T and 7T
We showed that ICA can be used to reveal ROIs in an easy and data-driven way. The ICA analysis results in the same areas of activation as revealed by a GLM. Yet, on top of the main findings, ICA reveals secondary distributed activations that were not visible in GLM analyses. Decomposing the data into many components using ICA with a block design leads to a better isolation of task-related components compared to GLM.
In 3T data, this results in improved data analysis. At 7T, ICA results in an even bigger improvement over GLM analysis, probably because this data contains more noise artifacts than 3T, like ghosting, uneven fieldmaps and signal dropout.
Gaussian mixture models and semantic gating improve reconstructions from human brain activity
Better acquisition protocols and analysis techniques are making it possible to use fMRI to obtain highly detailed visualizations of brain processes. In particular we focus on the reconstruction of natural images from BOLD responses in visual cortex. We expand our linear Gaussian framework for percept decoding with Gaussian mixture models to better represent the prior distribution of natural images. Reconstruction of such images then boils down to probabilistic inference in a hybrid Bayesian network.
In our set-up, different mixture components correspond to different character categories. Our framework can automatically infer higher-order semantic categories from lower-level brain areas. Furthermore, the framework can gate semantic information from higher-order brain areas to enforce the correct category during reconstruction. When categorical information is not available, we show that automatically learned clusters in the data give a similar improvement in reconstruction. The hybrid Bayesian network leads to highly accurate reconstructions in both supervised and unsupervised settings.
Via Frontiers in Computational Neuroscience
Gaussian mixture models improve fMRI-based image reconstruction
New computational models have made it possible to reconstruct perceived images from BOLD responses in visual cortex. We expand a linear Gaussian framework for percept decoding with Gaussian mixture models to better represent the prior distribution of images.
In our setup, different mixture components correspond to different letter categories. Our framework not only leads to more accurate reconstructions, but also automatically infers semantic categories from low-level visual areas of the human brain.
Linear reconstruction of perceived images from human brain activity
With the advent of sophisticated acquisition and analysis techniques, decoding the contents of someone's experience has become a reality. We propose a straightforward linear Gaussian approach, where decoding relies on the inversion of properly regularized encoding models, which can still be solved analytically.
In order to test our approach we acquired functional magnetic resonance imaging data under a rapid event-related design in which subjects were presented with handwritten characters. Our approach is shown to yield state-of-the-art reconstructions of perceived characters as estimated from BOLD responses. This even holds for previously unseen characters. We propose that this framework serves as a baseline with which to compare more sophisticated models for which analytical inversion is infeasible.
Highlights
We propose a linear Gaussian framework for perceived image reconstruction.
We reconstructed handwritten characters from rapid event related fMRI.
Reconstructions are of high quality, even for previously unseen characters.
The framework is proposed as a baseline with which to compare other approaches.