Multi-sensory Integration in Predictive Processing

Abstract

Predictive processing is a leading account of perception, but its canonical form assumes a strict sensory hierarchy in which predictions descend and errors ascend an ordered ladder of areas [2]. Cortical connectivity is not strictly hierarchical, with dense lateral and recurrent projections and connections that skip levels. Predictive processing can in principle operate over arbitrary network topologies [3], yet a basic question remains open: when multiple populations project to a shared target, how are their converging signals combined into predictions, errors, and uncertainties? This is especially relevant to multisensory integration, and different answers imply different circuits and behaviour. We formalize two normative schemes for combining converging projections. In the joint scheme, converging streams are fused into a single prediction with a shared uncertainty (similar to [3]). In the separate scheme, each stream carries its own prediction and uncertainty, and the target integrates parallel uncertainty-weighted errors. The schemes coincide in restricted cases, but in general diverge in their dynamics, plasticity, and predicted activity. To derive experimental predictions, we adopt the interpretation of prediction uncertainty as encoded by interneurons [5, 1], mapping our abstract model onto identifiable cell populations. We demonstrate the distinction on several tasks and show how experiments could separate the schemes. In an audiovisual task tested under cross-modal mismatch [4], both schemes produce cross-modal prediction errors, but the separate scheme predicts longer error latencies and block-structured correlations among error neurons. In a cue integration task, both achieve Bayes-optimal weighting via different mechanisms. Furthermore, under heteroscedastic noise, where each cue’s reliability varies with the latent variable, the schemes diverge more substantially: the separate scheme learns each uncertainty as a function of the latent estimate and trades off the cues accordingly, while the joint scheme, constrained to a single shared uncertainty, cannot. References: [1] Arno Granier et al. "Confidence and second-order errors in cortical circuits". In: PNAS nexus 3.9 (2024), pgae404. [2] Rajesh PN Rao and Dana H Ballard. "Predictive coding in the visual cortex: a functional interpretation of some extra- classical receptive-field effects". In: Nature neuroscience 2.1 (1999), pp. 79-87. [3] Tommaso Salvatori et al. "Learning on arbitrary graph topologies via predictive coding". In: Advances in neural information processing systems 35 (2022), pp. 38232-38244. [4] Liesa Stange, José P Ossandón, and Brigitte Röder. "Crossmodal visual predictions elicit spatially specific early visual cortex activity but later than real visual stimuli". In: Philosophical Transactions of the Royal Society B: Biological Sciences 378.1886 (2023). [5] Katharina Anna Wilmes et al. "Uncertainty-modulated prediction errors in cortical microcircuits". In: Elife 13 (2025), RP95127.