3 Multimodal Integration

The possibility of integrating multimodal sensory stimuli in meaningful units has many advantages for us — for example, improved understanding of language. The enormous selectivity of our sense of hearing makes it possible to follow a voice attentively even in a loud environment. If at the same time one watches the facial and lip movements of the person speaking, understanding improves enormously. Multimodal integration thus means that the perception in the realm of one sense is influenced by perception in another, since the two components are integrated into an interpretation which is as consistent as possible. The linkage of visual and auditory stimuli does not result solely — as was long assumed — from mental construction; it has been shown that various sensory stimuli already converge on the neuronal level in so-called multimodal neurons.^[5] Multimodal integration thus occurs on the lowest level of perception, sometimes even before the object is recognized.

In contrast to genuine synesthesia, which is considered absolute,^[6] multimodal integration is dependent on the context: if one sense provides too little or unclear information, other senses enter in as a corrective — for example, in dark surroundings, auditory perception becomes more important for orientation in space. Irrespective of modality, in a given situation the most reliable stimulus will dominate all the others. Factors in evaluating reliability naturally include attentiveness, experience, motivation, and previous knowledge, which makes it clear that multimodal integration cannot be reduced to processes taking place in multimodal neurons. In any case, the key to stable perception is the efficient combination and integration of information of different modalities.