Perception is profoundly multisensory and the interplay of our senses greatly facilitates the recognition of and reaction to sensory events. Multisensory interactions comprise the integration of feature-specific information across modalities and the unspecific enhancement of perception in one modality by co-stimulation in another. Our work aims to uncover the underlying computational principles and neural mechanisms.


Multisensory influences along auditory pathways
Focusing on the auditory system we found that visual inputs modulate the encoding of sounds in early auditory fields. We localized these early multisensory influences to caudal auditory fields (Kayser et al. J Neurosci 2007) and found that individual neurons and low freuqency oscillatory activity are modulated by visual stimuli in a manner that is specific to the relative audio-visusal timing (Kayser et al. CerCor 2008).
Using information theoretic analysis we then showed that this visual modulation reduces firing rates but makes neural responses more reliable (across trials and in time) and thereby increases the encoded information (Kayser Curr Biol 2010). This may well reflect the perceptual benefits known to arise from visual cues (e.g. lip movements) in noisy acoustic environments.
We further investigated these audio-visual interactions at subsequent stages of the auditory hierarchy, in superior temporal regions (Dahl et al. J Neurosci 2010), and in voice-areas (Perrodin et al J Neurosci 2014) to find more specific and topographically organized interactions.

Intriguingly, this work also showed that slow rhythmic network activity is critically involved in shaping the nature of multisensory interactions, such as by implementing the rhythmic amplification or attenuation of responses by different audio-visual delays (Perrodin et al PNAS 2015).


Noise-induced multisensory influences
Multisensory interactions can also arise from highly unrelated and unspecific sensory events. We showed that continous acoustic noise enhances visual perception in a noise-amplitude-dependent manner. The noise engages alpha-band oscillatory mechanisms thereby shifting psychometric curves and facilitating perception. Such unspecific multisensory interactions may be very frequent in daily life (Gleiss et al J Cogn Neurosci 2014).



Multisensory perception in rodents
Using a rodent model we found that rats exhibit similar audio-visual perceptual benefits as humans (Gleiss et al. Plos One 2012). Using a combination of functional intrinsic imaging and electrophysiology we then localized a multisensory responsive region within the rat parietal cortex, corresponding to Paxions area PtA (Lippert et al. Plos One 2013).





Speech rhythm and cortical entrainment
Auditory cortical activity entrains to the rhythmic structure of sounds. By manipulating the pseudo-rhythmic speech regularity imposed by pauses between syllables or words we showed that auditory cortical delta (but not theta) band entrainment is significanlty reduced when the speech structure becomes more unpredictable (Kayser et al J Neurosci 2015). This reduction in auditory entrainment correlates with left frontal alpha activity.


Slow network rhythms and auditory cortical gain control
Recent work has shown that network rhythms (e.g. delta / theta band) can influence perception. For example, we found that detetction rates for a sound in noise detection task depend on the current phase and amplitude of slow rhythmic auditory cortex activity (Ng et al. J Neurosci 2012). This suggests that local network rhythms play a causal role in sound encoding.



We recently extended this using single trial decoding analysis to dissect the different mechanistic components of prestimulis influences (Kayser et al PNAS '16). We found that pre-stimulus power influenced only the quality of the encoded task relevant sensory information in early EEG components, while phase affected only later, likely decision-related processes. This suggests that pre-stimulus effects exist at multiple stages of sensory-perceptual transformations.



We recently substantiated this hypothesis using a model-based analysis of auditory cortex neurons. We found that delta band activity relates to stimulus unspecific spiking while network activity on the alpha scale relates to sensory gain control (Kayser et al J Neurosci '15).



The functional organization of auditory cortex
The auditory cortex contains functional maps with regard to acoustic features (e.g. Petkov et al. Plos Biol 2006, Kayser et al. J Neurosci 2007) and we recently showed that human auditory cortex is organized not only with regard to sound frequency (tonotopy) but also to temporal modulation rate (periodotopy; Herdener et al. Cortex 2013).



Principles of neural coding
Understanding the neural codes (i.e. the relevant patterns of neural activity) underlying sensation is critical to link brain function with behaviour. Recent work uncovered some of the relevant spatio-temporal principles, but many questions remain open (for reviews see Panzeri et al. TINS 2010, Einevoll et al Nature Reviews Neuroscience, 2013, Panzeri et al. Phil Trans R Soc Lond B 2014), and Panzeri et al. TICS 2015).

One key question is how the brain actually reads out information carried by spatio-temporal patterns of activity. Auditory neurons seem to carry acoustic information at high temporal precision (Kayser et al. PNAS 2010). But it is unclear whether and how a downstream decoder could extract this. We found potential solutions in the relative timing between neurons (Brasselet et al J Neurosci 2012) and slow oscillatory network activity (Kayser et al. PlosCompBiol 2012). These could serve as temporal reference frames to define relevant packages of spikes in relation to intrinsically available reference time points.


Sensory representations arise from multi-neuron activity, raising the question as to how sensory information is distributed across many neurons within a given area. We recently studied the scaling of acoustic information relative to population size and found that this scalign can be predicted based on the assumption of statistically independent but un-equally informative units. This work suggests novel strategies for analyzing neural population activity (Ince et al J Neurosci 2013).