Affective annotation refers to the process of labeling media content based on the emotions they evoke. Since such experiences are inherently subjective and depend on individual differences, the central challenge is associating digital content with its affective, interindividual experience. Here, we present a first-of-its-kind methodology for affective annotation directly from brain signals by monitoring the affective experience of a crowd of individuals via functional near-infrared spectroscopy (fNIRS). An experiment is reported in which fNIRS was recorded from 31 participants to develop a brain-computer interface (BCI) for affective annotation. Brain signals evoked by images were used to draw predictions about the affective dimensions that characterize the stimuli. By combining annotations, the results show that monitoring crowd responses can draw accurate affective annotations, with performance improving significantly with increases in crowd size. Our methodology demonstrates a proof-of-concept to source affective annotations from a crowd of BCI users without requiring any auxiliary mental or physical interaction.

We present a dataset for the analysis of human affective states using functional near-infrared spectroscopy (fNIRS). Data were recorded from thirty-one participants who engaged in two tasks. In the emotional perception task the participants passively viewed images sampled from the standard international affective picture system database, which provided ground-truth valence and arousal annotation for the stimuli. In the affective imagery task the participants actively imagined emotional scenarios followed by rating these for subjective valence and arousal. Correlates between the fNIRS signal and the valence-arousal ratings were investigated to estimate the validity of the dataset. Source-code and summaries are provided for a processing pipeline, brain activity group analysis, and estimating baseline classification performance. For classification, prediction experiments are conducted for single-trial 4-class classification of arousal and valence as well as cross-participant classifications, and comparisons between high and low arousal variants of the valence prediction tasks. Finally, classification results are presented for subject-specific and cross-participant models. The dataset is made publicly available to encourage research on affective decoding and downstream applications using fNIRS data.

The present notion of visual similarity is based on features derived from image contents. This ignores the users' emotional or affective experiences toward the content, and how users feel when they search for images. Here we consider valence, a positive or negative quantification of affective appraisal, as a novel dimension of image similarity. We report the largest neuroimaging experiment that quantifies and predicts the valence of visual content by using functional near-infrared spectroscopy from brain-computer interfacing. We show that affective similarity can be (1)~decoded directly from brain signals in response to visual stimuli, (2)~utilized for predicting affective image similarity with an average accuracy of 0.58 and an accuracy of 0.65 for high-arousal stimuli, and (3)~effectively used to complement affective similarity estimates of content-based models; for example when fused fNIRS and image rankings the retrieval F-measure@20 is 0.70. Our work opens new research avenues for affective multimedia analysis, retrieval, and user modeling.

Information retrieval (IR) relies on a general notion of relevance, which is used as the principal foundation for ranking and evaluation methods. However, IR does not account for more a nuanced affective experience. Here, we consider the emotional response decoded directly from the human brain as an alternative dimension of relevance. We report an experiment covering seven different scenarios in which we measure and predict how users emotionally respond to visual image contents by using functional near-infrared spectroscopy (fNIRS) neuroimaging on two commonly used affective dimensions: valence (negativity and positivity) and arousal (boredness and excitedness). Our results show that affective states can be successfully decoded using fNIRS, and utilized to complement the present notion of relevance in IR studies. For example, we achieved 0.39 Balanced accuracy and 0.61 AUC in 4-class classification of affective states (vs. 0.25 Balanced accuracy and 0.5 AUC of a random classifier). Likewise, we achieved 0.684 Precision@20 when retrieving high-arousal images. Our work opens new avenues for incorporating emotional states in IR evaluation, affective feedback, and information filtering.

The present aim was to study if thermal stimuli presented to the palm can affect emotional responses when measured with emotion related subjective rating scales and changes in skin conductance response (SCR). Two target temperatures, cold and warm, were created by either decreasing or increasing the temperature of the stimulus 4 °C in respect to the participants current hand temperature. Both cold and warm stimuli were presented by using two presentation methods, i.e., dynamic and pre-adjusted. The results showed that both the dynamic and pre-adjusted warm stimuli elevated the ratings of arousal and dominance. In addition, the pre-adjusted warm and cold stimuli elevated the SCR. The results suggest that especially pre-adjusted warm stimuli can be seen as effective in activating the autonomic nervous system and arousal and dominance dimensions of the affective rating space.