Augmented Reality can induce illusory perceptions of heat when users experience virtual flames rendered on their hands. In this exploratory study, we used high-density electroencephalography to investigate the neural correlates associated with this experience. We recorded neural activity in a within-subjects design during three key audiovisual conditions: (A) lifelike virtual flames on the hand paired with a burning sound; (B) a non-fire-like control stimulus (green, downward-flowing visuals with a synthetic sound) on the hand; and (C) realistic flames presented between the hands. Our preliminary analyses revealed distinct electroencephalography signatures between the Fire-on-Hand condition and both control conditions. Our findings suggest that AR-induced thermal illusions may engage brain networks involved in somatosensation, multisensory body representation, and spatial attention. This top-down, predictive mechanism appears to recruit neural systems that overlap with those involved in the higher-order cortical processing of actual thermal stimuli.

In the Burning Hand Illusion (BHI), participants perceive illusory heat as they observe virtual flames on their hands in Augmented Reality. In this study, we investigate the role of stimulus location within peri-hand space and its resemblance to a realistic fire in the emergence of the illusion. Additionally, we examine the associated autonomic nervous responses, quantified through skin conductance and pupil dilation responses. We used the following four conditions: (A) Realistic flames on the right hand paired with the sound of a burning torch; (B) Green, flame-like visuals on the hand paired with a sound that does not evoke fire but is spectrally matched to the sound of the burning torch; (C) Realistic flames between the hands; and (D) Green, flame-like visuals between the hands. Sixteen out of twenty-five participants reported feeling heat in condition (A), while seven reported a sensation of warmth in condition (C). Condition (B) introduced ambiguity, as participants reported warmth as well as cold sensations, but overall, fewer participants reported thermal illusions. No participants reported thermal illusions in (D). These findings suggest that both the location and resemblance of the flames significantly influence the emergence of the BHI, with the response diminishing as the flames become less plausible or not placed on the actual hand. Skin conductance responses were highest in condition (A) and lower in condition (C), with the lowest responses observed in conditions (B) and (D). Pupil dilation responses were largest in conditions where the flames were directly on the hand. The observed autonomic responses reinforce the notion that the BHI elicits physiological effects that parallel those of real thermal stimuli. This provides evidence that the virtual fire in the BHI is sufficiently plausible to be perceived as affecting the hand, and is consequently processed by the brain’s homeostatic emotional systems.

It is well known that auditory nerve (AN) fibers overcome bandwidth limitations through the volley principle, a form of multiplexing. What is less well known is that the volley principle introduces a degree of unpredictability into AN neural firing patterns that may be affecting even simple stimulus categorization learning. We use a physiologically grounded, unsupervised spiking neural network model of the auditory brain with spike time dependent plasticity learning to demonstrate that plastic auditory cortex is unable to learn even simple auditory object categories when exposed to the raw AN firing input without subcortical preprocessing. We then demonstrate the importance of nonplastic subcortical preprocessing within the cochlear nucleus and the inferior colliculus for stabilizing and denoising AN responses. Such preprocessing enables the plastic auditory cortex to learn efficient robust representations of the auditory object categories. The biological realism of our model makes it suitable for generating neurophysiologically testable hypotheses.

Cortical sensory neurons are commonly characterized using the receptive field, the linear dependence of their response on the stimulus. In primary auditory cortex neurons can be characterized by their spectrotemporal receptive fields, the spectral and temporal features of a sound that linearly drive a neuron. However, receptive fields do not capture the fact that the response of a cortical neuron results from the complex nonlinear network in which it is embedded. By fitting a nonlinear feedforward network model (a network receptive field) to cortical responses to natural sounds, we reveal that primary auditory cortical neurons are sensitive over a substantially larger spectrotemporal domain than is seen in their standard spectrotemporal receptive fields. Furthermore, the network receptive field, a parsimonious network consisting of 1-7 sub-receptive fields that interact nonlinearly, consistently better predicts neural responses to auditory stimuli than the standard receptive fields. The network receptive field reveals separate excitatory and inhibitory sub-fields with different nonlinear properties, and interaction of the sub-fields gives rise to important operations such as gain control and conjunctive feature detection. The conjunctive effects, where neurons respond only if several specific features are present together, enable increased selectivity for particular complex spectrotemporal structures, and may constitute an important stage in sound recognition. In conclusion, we demonstrate that fitting auditory cortical neural responses with feedforward network models expands on simple linear receptive field models in a manner that yields substantially improved predictive power and reveals key nonlinear aspects of cortical processing, while remaining easy to interpret in a physiological context.