Understanding human emotions from images is a challenging yet essential task for vision-language models. While recent efforts have fine-tuned vision-language models to enhance emotional awareness, most approaches rely on global visual representations and fail to capture the nuanced and multi-faceted nature of emotional cues. Furthermore, most existing approaches adopt instruction tuning, which requires costly dataset construction and involves training a large number of parameters, thereby limiting their scalability and efficiency. To address these challenges, we propose MASP, a novel framework for Multi-Aspect guided emotion reasoning with Soft Prompt tuning in vision-language models. MASP explicitly separates emotion-relevant visual cues via multi-aspect cross-attention modules and guides the language model using soft prompts, enabling efficient and scalable task adaptation without modifying the base model. Our method achieves state-of-the-art performance on various emotion recognition benchmarks, demonstrating that the explicit modeling of multi-aspect emotional cues with soft prompt tuning leads to more accurate and interpretable emotion reasoning in vision-language models.

With the growing ubiquity and broad usage, emojis are widely used as a universal visual language, which complements the intentions and emotions beyond the textual data. Despite the critical role of representing emotion, existing emojis neglect the subtle and complex properties of human emotion in that only countable and finite face emojis exist in a categorical manner. In this article, we propose a novel approach to facial emoji generation, which can control the emotional degree of generated emojis for more complex and detailed usage on online conversations. In other words, we develop a new emotion-aware emoji generative adversarial network, which is capable of generating an emoji that expresses a given emotion distribution. In this way, our approach aims to map fine-grained emotions to expressive emojis. Both quantitative and qualitative evaluation demonstrate that our approach can successfully generate highquality emoji-like images by representing a wide range of emotions. To the best of our knowledge, this is the first approach to use the deep generative model from the standpoint of the emoji’s emotional role, which can further promote more interactive and effective online communication.

Visual sentiment analysis aims to predict human emotional responses to visual stimuli. It has attracted considerable attention owing to the increasing popularity of online image sharing. Most researchers have focused on improving emotion recognition using holistic and local information derived from given images. Relatively less attention has been paid to the semantic information of objects in images, which influences human emotional responses to the images. Therefore, we propose a novel object semantic attention network (OSANet) that attempts to unravel the semantic information of objects in images that contribute to emotion detection. The OSANet combines both global representation and semantic information of objects to predict the emotion elicited by a given image. First, the holistic features that represent the entire image are extracted using convolutional blocks. Subsequently, the object-level semantic information is obtained from pre-trained word embedding and then weighted according to the relative importance of the object using the attention mechanism. Notably, a new loss function to address the subjectivity of sentiment analysis is introduced, which improves the performance of the emotion detection task. Extensive experiments on three image emotion datasets demonstrated the superiority and interpretability of the OSANet. The results show that the OSANet outperforms extant image emotion detection frameworks.