Automated interpretation and reporting of chest X-rays (CXRs) hold significant promise in reducing diagnostic errors and supporting radiologists under heavy clinical workloads. However, existing methods typically rely on global visual features and token-level supervision, limiting their sensitivity to subtle abnormalities and reducing their clinical reliability. To address these challenges, we present Reflective X-ray Network (RefleXNet), which systematically integrates multi-scale visual feature fusion and anatomical relational reasoning with a targeted self-reflective learning strategy. RefleXNet first constructs multi-scale visual representations and captures anatomical context through graph-based relational modeling. Building upon these representations, we introduce a targeted self-reflection strategy that uses clinically guided feedback from generated reports to selectively refine abnormality predictions and their associated region-level visual features. Extensive experiments on MIMIC-CXR demonstrate that RefleXNet consistently outperforms state-of-the-art baselines across clinical factual correctness metrics. Notably, our compact 3B-parameter model surpasses several recent models with over twice the parameter count. Additionally, RefleXNet exhibits strong generalization performance in zero-shot evaluations on IU-Xray compared with leading multimodal language models, highlighting its robustness and clinical effectiveness.

Refusal on harmful prompts is a key safety behaviour in instruction‑tuned large language models (LLMs), yet the internal causes of this behaviour remain poorly understood. We study two public instruction tuned models—Gemma‑2-2B‑IT and LLaMA‑3.1-8B‑IT using sparse autoencoders (SAEs) trained on residual‑stream activations. Given a harmful prompt, we search the SAE latent space for feature sets whose ablation flips the model from refusal to compliance, demonstrating causal influence and creating a jailbreak. Our search proceeds in three stages: 1. Refusal Direction - Finding a refusal mediating direction and collecting SAE features close to that direction, followed by 2. Greedy Filtering - to prune this set to obtain a minimal set and finally 3. Interaction Discovery - a factorization‑machine (FM) model that captures non‑linear interactions among the remaining active features and the minimal set. This pipeline yields a broad set of jailbreak-critical features, offering insight into the mechanistic basis of refusal. Moreover, we also find evidence of redundant features which remain dormant unless earlier features are suppressed. Our findings highlight the potential for fine-grained auditing and targeted intervention in safety behaviours by manipulating the interpretable latent space.

Collaborative reasoning with multiple agents offers the potential for more robust and diverse problem-solving. However, existing approaches often suffer from homogeneous agent behaviors and lack of reflective and rethinking capabilities. We propose Multi-Agent Personality Shaping ((MAPS), a novel framework that enhances reasoning through agent diversity and internal critique. Inspired by the Big Five personality theory, MAPS assigns distinct personality traits to individual agents, shaping their reasoning styles and promoting heterogeneous collaboration. To enable deeper and more adaptive reasoning, MAPS introduces a Critic agent that reflects on intermediate outputs, revisits flawed steps, and guides iterative refinement. This integration of personality-driven agent design and structured collaboration improves both reasoning depth and flexibility. Empirical evaluations across three benchmarks demonstrate the strong performance of MAPS, with further analysis confirming its generalizability across different large language models and validating the benefits of multi-agent collaboration.

Large language models (LLMs) typically operate in a question-answering paradigm, where the quality of the input prompt critically affects the response. Automated Prompt Optimization (APO) aims to overcome the cognitive biases of manually crafted prompts and explore a broader prompt design space. However, existing APO methods often suffer from rigid template structures and inefficient exploration in the prompt space. To this end, we propose a Multi-Agent Adaptive Reasoning with Socratic guidance framework (MARS) for APO. MARS consists of five complementary agents and formulates the optimization process as a Partially Observable Markov Decision Process (POMDP), enabling adaptive prompt refinement through explicit state modeling and interactive feedback. Specifically, a Planner agent generates flexible optimization trajectories, a Teacher-Critic-Student triad engages in Socratic-style dialogue to iteratively optimize the prompt based on pseudo-gradient signals in the text space, and a Target agent executes the prompt in downstream tasks to provide performance feedback. MARS integrates reasoning, feedback, and state transition into a unified hidden-state evolution process, improving both the effectiveness and interpretability of optimization. Extensive experiments on multiple datasets demonstrate that MARS outperforms existing APO methods in terms of optimization performance, search efficiency, and interpretability.

Navya Gupta, Rishitej Reddy Vyalla, Avinash Anand, Chhavi Kirtani, Erik Cambria, Zhengchen Zhang, Zhengkui Wang, Timothy Liu, Aik Beng Ng, Simon See, Rajiv Ratn Shah. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Wuttikorn Ponwitayarat, Peerat Limkonchotiwat, Raymond Ng, Jann Railey Montalan, Thura Aung, Jian Gang Ngui, Yosephine Susanto, William Chandra Tjhi, Panuthep Tasawong, Erik Cambria, Ekapol Chuangsuwanich, Sarana Nutanong. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

ABSTRACT Measuring semantic similarity between words or phrases is central to natural language processing, information retrieval and computational linguistics. Despite their importance, similarity and distance measures are typically chosen by default (e.g., cosine similarity) or in an ad hoc fashion, with little empirical justification. This lack of systematic evaluation creates two gaps: first, the absence of a comprehensive taxonomy of measures that spans set‐based, vector‐based and information‐theoretic approaches; second, the lack of task‐aware benchmarking that quantifies how these measures perform across different models and applications. In this paper, we address these gaps by comparing 15 similarity and distance measures on four NLP tasks (sentence similarity, kNN classification, correlation analysis and visualisation) using multiple benchmark datasets and embedding models. Our results reveal that the effectiveness of similarity measures varies substantially depending on the task and model, challenging the assumption that cosine similarity is universally optimal. These findings highlight the practical risk of relying on default measures and provide a principled basis for selecting similarity functions in NLP, information retrieval and related fields.

Personalized affective dialogue systems are critical for mental health applications, where responses must be emotionally appropriate and tailored to individual users. However, most existing large language model (LLM) based approaches rely on deterministic personalization, ignore uncertainty in affective understanding, and are difficult to deploy under privacy constraints. In this paper, we propose PALLM, a personalized affective large language modeling framework designed for privacy-preserving mental health dialogue. PALLM decouples personalization into two complementary components: a deterministic personalization layer that captures stable user preferences, and a Bayesian affective representation layer that models dynamic emotional states and uncertainty. By restricting uncertainty modeling to affective representations rather than full LLM parameters, PALLM achieves efficient and scalable personalization under federated learning. Extensive experiments on EmpatheticDialogues and a real-world mental health conversation dataset show that PALLM improves affective alignment and robustness compared with non-personalized and partially personalized baselines.

Sarcasm functions as a distinct mode of communication, intended to convey a meaning contrary to its literal interpretation. With the rapid proliferation of social networks, various manifestations of multimodal sarcasm have become widespread. Consequently, there is a growing emphasis on discerning sarcasm conveyed through multimodal data. Existing research often provides only a superficial interpretation of images, lacking a thorough exploration of the contextual nuances embedded within, particularly in understanding the scene depicted in the image. In this paper, we aim to delve deeper into the information embedded within images. Specifically, we begin by extracting entity relationships from images to capture the contextual information they convey. Additionally, we utilize image text recognition to extract textual information from the images. After conducting a comprehensive analysis of the image information, we establish consistency modeling between the image content and text using external knowledge. Finally, we employ a graph neural network to process the constructed cross-modal graph and make predictions regarding sarcasm. Extensive experiments validate the state-of-the-art performance of our model on publicly available multimodal Twitter datasets.

Large language models (LLMs) generate fluent, context-rich text but suffer from hallucinations and limited interpretability. We introduce SenticNet 9, a neurosymbolic framework that automates commonsense reasoning while preserving transparency. It leverages conceptual primitive discovery (CPD) to learn foundational concepts and a time shift mechanism (TSM) to iteratively refine them through temporal feedback. This combination yields a scalable, cognitively inspired architecture that merges symbolic interpretability with LLM generalization. Experiments show SenticNet 9 outperforming embeddings, transformers, and state-of-the-art LLMs across tasks, delivering higher accuracy without sacrificing explainability.

Automated detection of depressive moods from social media holds great promise for early mental health intervention, yet existing multimodal approaches typically require large quantities of annotated data and extensive feature engineering, impeding their deployment in real‐world settings where labels are scarce. To address this challenge, we propose prompt‐based depression detection (PBDD), a novel prompt‐based few‐shot learning framework that leverages frozen pretrained language and vision models to identify depression indicators from paired text‐image posts without fine‐tuning. Our method begins with rigorous data cleaning and sampling to construct a high‐quality few‐shot dataset, then encodes text via a masked language model and images via a self‐supervised rotation‐prediction task to capture deep semantic cues. Multimodal representations are seamlessly fused into a unified prompt template containing a [MASK] token, enabling the pre‐trained model to infer depressive states by language completion. Extensive experiments on both large‐scale and 1 % few‐shot subsets demonstrate that PBDD consistently outperforms state‐of‐the‐art baselines, achieving significant gains in accuracy and Macro‐F1. These results validate the effectiveness and scalability of our framework for depression detection under severe label scarcity, offering a practical solution for real‐time mental health monitoring in social media environments.

Multimodal emotion analysis is pivotal in decoding complex human affect by integrating diverse data sources such as text, audio, and visual signals. In this article, we introduce contextual interaction-based multimodal emotion analysis with enhanced semantic information (CIME), a novel spatio-temporal interaction network that significantly improves emotion recognition accuracy and robustness. CIME employs a text-centric cross-modal attention mechanism to refine semantic representations, while simultaneously leveraging a graph convolutional network to model contextual dialog information by capturing both intraspeaker and interspeaker relationships. This dual approach enables the effective fusion of modality-specific cues and the mining of latent emotional associations across modalities. Extensive experiments conducted on benchmark datasets—including IEMOCAP and MOSEI—demonstrate that CIME consistently outperforms existing state-of-the-art methods in terms of overall classification accuracy and weighted F1-scores. Furthermore, detailed ablation studies underscore the critical contributions of both the cross-modal attention and graph-based contextual modules.

Recognizing oracle bone scripts plays an important role in Chinese archaeology and philology. However, a significant challenge remains because of the scarcity of oracle character images. To overcome this issue, we propose Diff-Oracle, a novel multi-modal conditional diffusion model that generates a diverse range of controllable oracle characters by inputting random combinations of references. Given the challenge of accurately describing oracle character styles using natural language, Diff-Oracle departs from traditional diffusion models that rely primarily on text prompts by introducing a style encoder. This encoder extracts style prompts from existing oracle character images, where style details are converted into a text embedding format via a pre-trained language-vision model. Additionally, given the lack of explicit content information for oracle characters, ensuring that generated characters accurately represent the intended glyphs is challenging. Therefore, we pre-generate pixel-level paired oracle character images (i.e., style and content images) by an image-to-image translation model, providing content information for the generation process. Meanwhile, Diff-Oracle integrates a content encoder designed to capture specific content details from content reference images. Extensive experiments on Oracle-241 and OBC306 datasets demonstrate that Diff-Oracle significantly outperforms existing generative methods in image quality and diversity. Moreover, Diff-Oracle substantially benefits downstream recognition tasks, outperforming all existing state-of-the-art methods by a large margin. In particular, on the challenging OBC306 dataset, Diff-Oracle achieves a 7.70% accuracy gain in the zero-shot setting and reaches 84.62% accuracy for unseen oracle characters, setting a new benchmark for oracle character recognition. The code is available at https://github.com/JJJingLi/Diff-Oracle .

Sustainability reports are key for evaluating companies' environmental, social and governance (ESG) performance.To analyze these reports, NLP approaches can efficiently extract ESG insights at scale.However, even the most advanced NLP methods lack robustness against ESG content that is greenwashed -i.e.sustainability claims that are misleading, exaggerated, and fabricated.Accordingly, existing NLP approaches often extract insights that reflect misleading or exaggerated sustainability claims rather than objective ESG performance.To tackle this issue, we introduce A3CG -Aspect-Action Analysis with Cross-Category Generalization, as a novel dataset to improve the robustness of ESG analysis amid the prevalence of greenwashing.By explicitly linking sustainability aspects with their associated actions, A3CG facilitates a more fine-grained and transparent evaluation of sustainability claims, ensuring that insights are grounded in verifiable actions rather than vague or misleading rhetoric.Additionally, A3CG emphasizes cross-category generalization.This ensures robust model performance in aspectaction analysis even when companies change their reports to selectively favor certain sustainability areas.Through experiments on A3CG, we analyze state-of-the-art supervised models and LLMs, uncovering their limitations and outlining key directions for future research.

Diabetes self-management continues to exhibit wide interpersonal variability, a phenomenon insufficiently addressed by standardized Diabetes Self-Management Education (DSME) programs. The limitations of existing approaches stem largely from their implicit assumption of homogeneous cognitive structures, an assumption inconsistent with theoretical insights from conceptual metaphor theory and cognitive science indicating that individuals rely on divergent conceptual mappings to make sense of chronic illness. This study addresses this gap by conducting a large-scale computational analysis of 39,880 spontaneous patient self-disclosures from diabetes-related subreddits using MetaPro to uncover the underlying cognitive frameworks of diabetic patients. Comparative analysis demonstrated significant differences in cognitive conceptualizations between type 1 (T1D) and type 2 (T2D) diabetes patients. T1D discourse is dominated by an “action-monitoring” schema, whereas T2D discourse emphasizes a “process-mishap-component” schema. Gender comparisons further distinguish a male “metric-status-trajectory” profile from a female “burden-care-intuition” profile. These findings illuminate the deep cognitive heterogeneity underlying diabetes self-management and underscore the need for personalized, cognitively informed intervention strategies that attend to demographic and clinical distinctions.

Recent advancements in large language models (LLMs) have shown remarkable progress in reasoning capabilities, yet they still face challenges in complex, multi-step reasoning tasks. This study introduces Reasoning with Trees (RwT), a novel framework that synergistically integrates LLMs with knowledge graphs (KGs) to enhance reasoning performance and interpretability. RwT reformulates knowledge graph question answering (KGQA) as a discrete decision-making problem, leveraging Monte Carlo Tree Search (MCTS) to iteratively refine reasoning paths. This approach mirrors human-like reasoning by dynamically integrating the LLM’s internal knowledge with external KG information. We propose a real-data guided iteration technique to train an evaluation model that assesses action values, improving the efficiency of the MCTS process. Experimental results on two benchmark KGQA datasets demonstrate that RwT significantly outperforms existing state-of-the-art methods, with an average performance improvement of 9.81%. Notably, RwT achieves these improvements without requiring complete retraining of the LLM, offering a more efficient and adaptable approach to enhancing LLM reasoning capabilities.

Counterfactual reasoning typically involves considering alternatives to actual events.While often applied to understand past events, a distinct form-forward counterfactual reasoningfocuses on anticipating plausible future developments.This type of reasoning is invaluable in dynamic financial markets, where anticipating market developments can powerfully unveil potential risks and opportunities for stakeholders, guiding their decision-making.However, performing this at scale is challenging due to the cognitive demands involved, underscoring the need for automated solutions.LLMs offer promise, but remain unexplored for this application.To address this gap, we introduce a novel benchmark, FIN-FORCE-FINancial FORward Counterfactual Evaluation.By curating financial news headlines and providing structured evaluation, FIN-FORCE supports LLM based forward counterfactual generation.This paves the way for scalable and automated solutions for exploring and anticipating future market developments, thereby providing structured insights for decision-making.Through experiments on FIN-FORCE, we evaluate state-of-theart LLMs and counterfactual generation methods, analyzing their limitations and proposing insights for future research.We release the benchmark, supplementary data and all experimental codes at the following link

Enabling digital humans to express rich emotions has significant applications in dialogue systems, gaming, and other interactive scenarios.While recent advances in talking head synthesis have achieved impressive results in lip synchronization, they tend to overlook the rich and dynamic nature of facial expressions.To fill this critical gap, we introduce an endto-end text-to-expression model that explicitly focuses on emotional dynamics.Our model learns expressive facial variations in a continuous latent space and generates expressions that are diverse, fluid, and emotionally coherent.To support this task, we introduce EmoAva, a large-scale and high-quality dataset containing 15,000 text-3D expression pairs.Extensive experiments on both existing datasets and EmoAva demonstrate that our method significantly outperforms baselines across multiple evaluation metrics, marking a significant advancement in the field.

Large Language Models (LLMs) are capable of generating persuasive Natural Language Explanations (NLEs) to justify their answers.However, the faithfulness of these explanations should not be readily trusted at face value.Recent studies have proposed various methods to measure the faithfulness of NLEs, typically by inserting perturbations at the explanation or feature level.We argue that these approaches are neither comprehensive nor correctly designed according to the established definition of faithfulness.Moreover, we highlight the risks of grounding faithfulness findings on out-of-distribution samples.In this work, we leverage a causal mediation technique called activation patching, to measure the faithfulness of an explanation towards supporting the explained answer.Our proposed metric, Causal Faithfulness quantifies the consistency of causal attributions between explanations and the corresponding model outputs as the indicator of faithfulness.We experimented across models varying from 2B to 27B parameters and found that models that underwent alignment-tuning tend to produce more faithful and plausible explanations.We posit that Causal Faithfulness is a promising improvement over existing faithfulness tests by taking into account the model's internal computations and avoiding out-of-distribution concerns that could otherwise undermine the validity of faithfulness assessments.We release the code in https://github.com/ SenticNet/causal-faithfulness

Recent advancements in knowledge graph question answering (KGQA) have shown promise, yet existing methods often fail to align with human reasoning patterns. This study proposes HD-PORT (hop-level direct preference optimization for knowledge graph reasoning with trees), a novel approach that combines Monte Carlo Tree Search (MCTS) with hop-level direct preference optimization (HDPO) for KGQA tasks. MCTS simulates human-like reasoning by exploring multiple inference paths in knowledge graphs, generating interpretable reasoning chains and rich hop-level preferences. HDPO then leverages these preferences for model optimization, addressing the limitations of traditional supervised learning and existing DPO methods that focus on overall solution preferences. By optimizing preferences at each reasoning step, HD-PORT more closely mirrors human problem-solving strategies. Experimental results on benchmark datasets demonstrate that HD-PORT significantly outperforms state-of-the-art methods in both accuracy and interpretability, particularly for complex, multi-hop reasoning tasks.

Scientific discovery contributes largely to the prosperity of human society, and recent progress shows that LLMs could potentially catalyst the process. However, it is still unclear whether LLMs can discover novel and valid hypotheses in chemistry. In this work, we investigate this main research question: whether LLMs can automatically discover novel and valid chemistry research hypotheses, given only a research question? With extensive discussions with chemistry experts, we adopt the assumption that a majority of chemistry hypotheses can be resulted from a research background question and several inspirations. With this key insight, we break the main question into three smaller fundamental questions. In brief, they are: (1) given a background question, whether LLMs can retrieve good inspirations; (2) with background and inspirations, whether LLMs can lead to hypothesis; and (3) whether LLMs can identify good hypotheses to rank them higher. To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature or a similar level in 2024 (all papers are only available online since 2024). Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis. The goal is to rediscover the hypothesis given only the background and a large chemistry literature corpus consisting the ground truth inspiration papers, with LLMs trained with data up to 2023. We also develop an LLM-based multi-agent framework that leverages the assumption, consisting of three stages reflecting the more smaller questions. The proposed method can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.

Vision Language Models (VLMs), such as CLIP, are widely used for various multimodal tasks and offer significant advancements in image-text understanding. However, existing studies have revealed that VLMs inherit biases from their training data which lead to the reinforcement of harmful stereotypes and cultural misrepresentations. In the proposed work, we analyze the presence of biases associated with animals in the CLIP model. We introduce a novel taxonomy, called Animal Bias Taxonomy (ABT), which categorizes stereotyped associations of animals in three categories. We also curated an animal dataset from existing datasets and applied data-cleaning process on it to remove unwanted images. Using ABT, we evaluated the outputs of VLMs on animal dataset when prompted with animal-related stereotyped terms to assess whether CLIP propagates biased associations that align with cultural stereotypes. Our findings reveal that CLIP frequently exhibits skewed cultural interpretations, such as associating owls with wisdom. Our study underscores the necessity of bias evaluation in VLMs and calls for greater transparency and culturally diverse data curation to ensure fair and inclusive AI systems. The code is available at https://github.com/MohammadAnas5/Clip-sAnimalStereotyping

The 1st Workshop on Cognition-oriented Multimodal Affective and Empathetic Computing (CogMAEC) was held at ACM Multimedia 2025. It focused on moving emotional AI beyond basic recognition toward deeper cognitive understanding. While traditional multimodal affective computing has emphasized simple emotion detection, the rise of multimodal large language models (MLLMs) has spurred interest in modeling how emotions emerge and evolve in context. The workshop gathered researchers on emotion reasoning, multimodal understanding, and human-computer empathy, exploring how machines can not only recognize emotions but also explain their causes and simulate human-like affective reasoning. The program featured invited talks, oral presentations, and posters spanning perception, interaction, causal modeling, and cognitive grounding. CogMAEC provided a platform to connect researchers across disciplines and foster future work on cognitively aware affective computing. Materials are available at https://CogMAEC.github.io/MM2025.

MER2025 is the third year of our MER series of challenges. Previously, MER2023 (http://merchallenge.cn/mer2023) focused on multi-label learning, noise robustness, and semi-supervised learning, while MER2024 (https://zeroqiaoba.github.io/MER2024-website) introduced a new track dedicated to open-vocabulary emotion recognition. This year, MER2025 centers on the theme ''When Affective Computing Meets Large Language Models (LLMs)''. We aim to shift the paradigm from traditional categorical frameworks reliant on predefined emotion taxonomies to LLM-driven generative methods, offering innovative solutions for more accurate and reliable emotion understanding. The challenge contains four tracks: MER-SEMI focuses on fixed categorical emotion recognition enhanced by semi-supervised learning; MER-FG explores fine-grained emotions, expanding recognition from basic to nuanced emotional states; MER-DES incorporates multimodal cues (beyond emotion words) into predictions to enhance model interpretability; MER-PR reveals whether emotion prediction results can improve personality recognition performance. For the first three tracks, the baseline code is available at MERTools (https://github.com/zeroQiaoba/MERTools) and datasets can be accessed via Hugging Face (https://huggingface.co/datasets/MERChallenge/MER2025). For the last track, the dataset and baseline code are available on GitHub (https://github.com/cai-cong/MER25_personality).

Multimodal, generative, and responsible affective computing aims to enhance people's lives. In recent years, the AI revolution has already begun to impact daily life, with virtual assistants being deployed across various sectors such as healthcare, banking, transportation, and education. It is clear that, in the near future, humans may interact with AI-powered systems as much or maybe even more than direct human-to-human interactions. Affective computing has numerous applications, including innovative approaches to forecasting and preventing anxiety, stress, and mental health issues; enhancing robotic empathy; assisting individuals with communication, behavior, and emotion regulation challenges; and promoting awareness of health and well-being. Many of these applications require enhanced control and protection of sensitive, private, and personal data. Therefore, it is crucial to further develop the creation, evaluation, and deployment of emotionally intelligent systems that are both responsive and responsible. Additionally, improving the accuracy and interpretability of emotion prediction results can significantly enhance the application of this technology in the downstream tasks mentioned above. MRAC'25 is the continuation of MRAC'23 and MRAC'24. Through this workshop, we aim to bring together researchers to discuss the potential and development of affective computing.

Understanding fine-grained sentiment dynamics in human conversations is a central goal for next-generation artificial intelligence, especially in scenarios where interactions are rich in both modalities and context. To advance research in this area, we organize the Multimodal Conversational Aspect-based Sentiment Analysis (MCABSA) challenge to the community of aspect-based sentiment analysis. The MCABSA challenge introduces two novel subtasks: 1) Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder, target, aspect, opinion, sentiment, and rationale from multi-turn, multi-party multimodal dialogue; and 2) Sentiment Flipping Analysis, detecting the dynamic sentiment transformation throughout the conversation along with the causal reasons. To support these tasks, we present the PanoSent dataset, a high-quality, large-scale benchmark featuring multi-turn, multi-party dialogues annotated with both explicit and implicit sentiment elements across text, image, audio, and video modalities. PanoSent offers extensive real-world scenario coverage, providing a comprehensive testbed for multimodal conversational sentiment analysis. The challenge has attracted widespread participation from both academia and industry, with over 30 teams registered and more than 100 successful submissions. In this paper, we introduce the task, dataset, and evaluation settings, summarize the systems of the top teams, and discuss the findings of the participants. Further details of the challenge can be found at https://panosent.github.io/MM25-challenge.

Previous multimodal emotion-cause analysis in conversations (MEC-AC) has predominantly focused on English, overlooking the applicability of existing methods in multilingual contexts. To bridge this gap, we construct a Chinese contextual dataset (MEC4) to investigate how language and culture diversity influences existing MECAC approaches. Moreover, prior studies often rely on average pooling or frame sampling to extract visual and acoustic features from video and audio of long dialogues, which inevitably results in the loss of temporal dynamics and emotionally salient cues. To overcome these limitations, we propose a memory-inspired multilingual multimodal framework (M3F) based on large language model (LLM), which can effectively capture the temporal and global informative features of non-linguistic modalities through memory bank module. This module simulates the way memory is stored in human cognitive processes and incrementally aggregates past visual and acoustic features in an autoregressive manner, enabling effective reference during future sequence modeling. Through rigorous experiments and insightful analyses, we find that cultural differences cause variations in how emotional expressions in English and Chinese languages rely on modalities.

The Ava-MERG Challenge at ACM Multimedia 2025 explores avatar-based multimodal empathetic response generation by advancing dialogue systems that respond with emotional awareness across text, speech, and talking-face video. To support this goal, we constructed a high-quality dataset featuring aligned multimodal responses in diverse conversational scenarios. The challenge comprises two progressively complex subtasks centering around multimodal-aware empathetic response generation. We aim to promote research in empathetic AI and affective computing.

Sriram Ranga, Rui Mao, Erik Cambria, Anupam Chattopadhyay. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Large language models (LLMs) have shown promise in automating scientific hypothesis generation, yet existing approaches primarily yield coarse-grained hypotheses lacking critical methodological and experimental details. We introduce and formally define the new task of fine-grained scientific hypothesis discovery, which entails generating detailed, experimentally actionable hypotheses from coarse initial research directions. We frame this as a combinatorial optimization problem and investigate the upper limits of LLMs' capacity to solve it when maximally leveraged. Specifically, we explore four foundational questions: (1) how to best harness an LLM's internal heuristics to formulate the fine-grained hypothesis it itself would judge as the most promising among all the possible hypotheses it might generate, based on its own internal scoring-thus defining a latent reward landscape over the hypothesis space; (2) whether such LLM-judged better hypotheses exhibit stronger alignment with ground-truth hypotheses; (3) whether shaping the reward landscape using an ensemble of diverse LLMs of similar capacity yields better outcomes than defining it with repeated instances of the strongest LLM among them; and (4) whether an ensemble of identical LLMs provides a more reliable reward landscape than a single LLM. To address these questions, we propose a hierarchical search method that incrementally proposes and integrates details into the hypothesis, progressing from general concepts to specific experimental configurations. We show that this hierarchical process smooths the reward landscape and enables more effective optimization. Empirical evaluations on a new benchmark of expert-annotated fine-grained hypotheses from recent literature show that our method consistently outperforms strong baselines.

Recent advancements in knowledge graph question answering (KGQA) have shown promise, yet existing methods often fail to align with human reasoning patterns that involve continuous reflection and refinement. This paper proposes FD-PORT (flow-guided direct preference optimization for knowledge graph reasoning with trees), a novel approach that combines Monte Carlo Tree Search (MCTS) with flow-guided direct preference optimization (FDPO) for KGQA tasks. MCTS simulates human-like reasoning by systematically exploring multiple inference paths in knowledge graphs, while FDPO transforms the search feedback into fine-grained training signals through flow balance conditions. Unlike traditional methods focusing on end-to-end training or sequence-level preferences, FD-PORT establishes flow consistency between any states along the reasoning chain, enabling robust multi-hop reasoning that adapts to local decisions and long-range dependencies. Experimental results on three benchmark datasets demonstrate that FD-PORT significantly outperforms state-of-the-art methods, achieving up to 50.6% improvements over GPT-4 on complex multi-hop reasoning tasks with a smaller open-source language model. The framework is advanced in maintaining diverse reasoning paths while ensuring answer quality, closely mirroring human problem-solving strategies.

Empathetic Response Generation (ERG) is one of the key tasks of the affective computing area, which aims to produce emotionally nuanced and compassionate responses to user's queries. However, existing ERG research is predominantly confined to the singleton text modality, limiting its effectiveness since human emotions are inherently conveyed through multiple modalities. To combat this, we introduce an avatar-based Multimodal ERG (MERG) task, entailing rich text, speech, and facial vision information. We first present a large-scale high-quality benchmark dataset, AvaMERG, which extends traditional text ERG by incorporating authentic human speech audio and dynamic talking-face avatar videos, encompassing a diverse range of avatar profiles and broadly covering various topics of real-world scenarios. Further, we deliberately tailor a system, named Empatheia, for MERG. Built upon a Multimodal Large Language Model (MLLM) with multimodal encoder, speech and avatar generators, Empatheia performs end-to-end MERG, with Chain-of-Empathetic reasoning mechanism integrated for enhanced empathy understanding and reasoning.Finally, we devise a list of empathetic-enhanced tuning strategies, strengthening the capabilities of emotional accuracy and content, avatar-profile consistency across modalities. Experimental results on AvaMERG data demonstrate that Empatheia consistently shows superior performance than baseline methods on both textual ERG and MERG. All data and code are open at https://AvaMERG.github.io/.

ABSTRACT The focus of this study is to present the detailed process of collecting a dataset of Arabic short‐text in the tourism context and annotating this dataset for the task of sentiment analysis using an automatic zero‐shot labelling technique utilising transformer‐based models. This is benchmarked against a baseline manual annotation approach utilising native Arab human annotators. This study also introduces an approach exploiting both manual/handcrafted and automatically generated annotations of the dataset tweets for the task of sentiment analysis as part of a cross‐domain approach using a model trained on sarcasm labels and vice versa. The total collected corpus size is 2293 tweets; after annotation, these tweets were labelled in a three‐way classification approach as either positive, negative or neutral. We run different experiments to provide benchmark results of Arabic sentiment classification. Comparative results on our dataset show that the highest performing baseline model when utilising manual labels was MARBERT, with an accuracy of up to 87%, which was pre‐trained for Arabic on a massive amount of data. It should be noted that this model enhanced its performance additionally after pre‐training on a dialectical Arabic and modern standard Arabic corpus. On the other hand, zero‐shot automatically generated labels achieved an 84% accuracy rate in predicting sarcasm classes from sentiment labels.

ABSTRACT Time‐series forecasting is essential for predicting events in the future and for tracking objects. The conventional recurrent neural network model needs to pad the target with zeros when handling long inputs, resulting in a loss in accuracy. Recently, it was proposed to divide a time series input into patches and merge the learned weights. However, such a model is difficult to interpret. In this article, we consider a mixture of continuous and discrete Markov states to model long‐range time dependencies. For example, in a vehicle, each gear level can be a discrete state and the throttle input is continuously controlled to maximise the efficiency of the engine. Data collected from the sensor is prone to noise due to component faults or external disturbances. Hence, we apply a stability constraint to select samples for training. We validate our algorithm on three datasets: (1) Apple Watch, (2) Car engine and (3) Election tweets. On all datasets, we achieve an improvement in the range of 5%–20% in the F‐measure. Furthermore, the features learned are easy to explain in terms of real‐world scenarios.

Remote sensing image captioning (RSIC) is an important task in environmental monitoring and disaster assessment. However, existing methods are constrained by redundant feature interference, insufficient multi-scale feature integration, and cross-modal semantic gaps, leading to limited performance in scenarios requiring fine-grained descriptions and semantic integrity, such as disaster assessment and emergency response. In this letter, we propose a Cross-Modal Semantic Alignment Model for Remote Sensing Image Captioning (CSA-RSIC), addressing these challenges with three innovations. First, we designed an Adaptive Feature Selection Module (AFSM) that generates channel weights through dual pooling. The AFSM dynamically weights the most informative features at each scale to improve caption accuracy. Second, we propose a Cross-Scale Feature Aggregation Module (CFAM) that constructs a hierarchical feature pyramid by aligning multi-scale resolutions and performs attention-guided fusion with enhanced weighting via AFSM, ensuring the effective integration of fine-grained and global semantic information. Finally, a novel loss function that combines contrastive learning and consistency loss is proposed to enhance the semantic alignment between visual and textual features. Experiments on three datasets show the advancement of CSA-RSIC over strong baselines, indicating its effectiveness in enhancing both semantic completeness and accuracy.

Artificial intelligence-based machine learning models have been widely used to explore and address various mental health-related problems in recent years, including depression. In this study, we present an ensemble approach to complement the 90 unique input features that we proposed in a previous study on depression detection using social media texts. Our proposed Ensemble of Ensemble Classifiers (EECs) combines many ensemble models, including Bagging Predictors, Random Forests, Adaptive Boosting and Gradient Boosting, as inner ensembles. These inner ensembles are arranged in a parallel fashion, where each of them is trained using different subsets of data sampled from the training data via bootstrap sampling. After the models are trained, during the testing phase, the results of all inner ensembles are processed using two methods— majority vote or class priority threshold—to get the final result as an output. From the experiments, we find that EECs are accurate in detecting signs of depression in social media users by analysing their posts in social media platforms such as Twitter. Our approach outperforms other ensemble methods on the public datasets we used. Moreover, if set correctly, the parameters of EECs can further improve the performance of the proposed ensemble in detecting signs of depression.

Enterprises suffer tremendous economic and reputational losses in public opinion crises. However, providing specific guidance for enterprises to respond effectively to public opinion crises is challenging. This study proposes a netizen-centered approach to identify crisis response opportunities for enterprises by mining social media data. First, we identify the topics discussed by netizens with the Biterm Topic Model, thereby evaluating their importance levels. Then, the negativeness levels of crisis topics are quantified by sentiment analysis. Lastly, crisis response opportunities are quantitatively identified and prioritized by applying an opportunity algorithm that simultaneously considers each crisis topic's importance and negativeness levels. Experimental results demonstrate the validity and superiority of our approach. Moreover, we demonstrate that a very negative and less important topic in the growth stage will likely become very negative and important at the maturity of the crisis. This finding supports decision-making in response to critical topics at the early stage of a crisis, preventing the deterioration of the public opinion crisis.

Multimodal aspect-based sentiment classification (MASC) is an emerging task due to an increase in user-generated multimodal content on social platforms, aimed at predicting sentiment polarity toward specific aspect targets (i.e., entities or attributes explicitly mentioned in text-image pairs). Despite extensive efforts and significant achievements in existing MASC, substantial gaps remain in understanding fine-grained visual content and the cognitive rationales derived from semantic content and impressions (cognitive interpretations of emotions evoked by image content). In this study, we present Chimera: acognitive and aesthetic sentiment causality understanding framework to derive fine-grained holistic features of aspects and infer the fundamental drivers of sentiment expression from both semantic perspectives and affective-cognitive resonance (the synergistic effect between emotional responses and cognitive interpretations). The framework aligns visual patches with words, extracts coarse and fine-grained visual features, translates them into textual descriptions, and uses LLM-generated sentimental causes and impressions to boost sensitivity to affective cues. Experiments on MASC datasets show the model's effectiveness and greater flexibility compared to LLMs like GPT-4o. We have publicly released the complete implementation and dataset athttps://github.com/Xillv/Chimera

Dialogue act recognition and sentiment classification (DAR and DSC) are closely related tasks in dialogue systems, both benefiting from joint modeling of their interdependencies. Recent advancements have improved performance on these tasks by integrating them, yet many approaches oversimplify dialogues by treating them as monologues and assuming uniform influence across all utterances. This neglects the inherent structural and interactive nature of dialogues. To address these issues, we propose a novelDiscourseStructure- andInterlocutor-Guided (DSIG) network that fuses dialogue act recognition and sentiment classification. Our network synergizes structural dialogue relationships with interlocutor identity information, enabling effective modeling of utterance flow and cross-task interactions. Specifically, we utilize a shared encoder that functions as a dialogue discourse parser to dynamically construct utterance connections, thereby integrating dialogue structural relationships. In addition, we embed interlocutor affiliations into the fusion process during encoding, semantic modeling, and decoding, enhancing dialogue understanding and dual-task reasoning. The core component is a collaborative updating graph interaction layer, which filters redundant connections and introduces interlocutor nodes for exclusive information exchange. Experimental results on two benchmark datasets demonstrate DSIG achieves state-of-the-art performance by effectively fusing dialogue structure and interlocutor information, improving F1 scores by 6.9% and 2.1% for DSC and DAR, respectively, on Mastodon, and by 13.7% and 4.7% on DailyDialog. Model variants trained independently show promise for extension to other dialogue-related tasks.

Sexism has become a pressing issue, driven by the rapid-spreading influence of societal norms, media portrayals, and online platforms that perpetuate and amplify gender biases. Curbing sexism has emerged as a critical challenge globally. Being capable of recognizing sexist statements and behaviors is of particular importance since it is the first step in mind change. This survey provides an extensive overview of recent advancements in sexism detection. We present details of the various resources used in this field and methodologies applied to the task, covering different languages, modalities, models, and approaches. Moreover, we examine the specific challenges these models encounter in accurately identifying and classifying sexism. Additionally, we highlight areas that require further research and propose potential new directions for future exploration in the domain of sexism detection. Through this comprehensive exploration, we strive to contribute to the advancement of interdisciplinary research, fostering a collective effort to combat sexism in its multifaceted manifestations.

Multimodal medical applications have garnered considerable attention due to their potential to offer comprehensive and robust support for medical assistance. Specifically, within this domain, difference-aware medical Visual Question Answering (VQA) has emerged as a topic of increasing interest that enables the recognition of changes in physical conditions over time when compared to previous states and provides customized suggestions accordingly. However, it is challenging because samples usually exhibit characteristics of complexity, diversity, and inherent noise. Besides, there is a need for multimodal knowledge understanding of the medical domain. The difference-aware setting requiring image comparison further intensifies these situations. To this end, we propose a cross-Modal knowlEdge diffusioN-baseD gEneration netwoRk (MENDER), where the diffusion mechanism with multi-step denoising and knowledge injection from global to local level are employed to tackle the aforementioned challenges, respectively. The diffusion process is to gradually generate answers with the sequence input of questions, random noises for the answer masks and virtual vision prompts of images. The strategy of answer nosing and knowledge cascading is specifically tailored for this task and is implemented during forward and reverse diffusion processes. Moreover, the visual and structure knowledge injection are proposed to learn virtual vision prompts to guide the diffusion process, where the former is realized using a pre-trained medical image-text network and the latter is modeled with spatial and semantic graph structures processed by the heterogeneous graph Transformer models. Experiment results demonstrate the effectiveness of MENDER for difference-aware medical VQA. Furthermore, it also exhibits notable performance in the low-resource setting and conventional medical VQA tasks.

Logical reasoning consistently plays a fundamental and significant role in the domains of knowledge engineering and artificial intelligence. Recently, Large Language Models (LLMs) have emerged as a noteworthy innovation in natural language processing (NLP). However, the question of whether LLMs can effectively address the task of logical reasoning, which requires gradual cognitive inference similar to human intelligence, remains unanswered. To this end, we aim to bridge this gap and provide comprehensive evaluations in this paper. First, to offer systematic evaluations, we select fifteen typical logical reasoning datasets and organize them into deductive, inductive, abductive and mixed-form reasoning settings. Considering the comprehensiveness of evaluations, we include 3 early-era representative LLMs and 4 trending LLMs. Second, different from previous evaluations relying only on simple metrics (e.g.,accuracy), we propose fine-level evaluations in objective and subjective manners, covering both answers and explanations, includinganswer correctness,explain correctness,explain completenessandexplain redundancy. Additionally, to uncover the logical flaws of LLMs, problematic cases will be attributed to five error types from two dimensions, i.e.,evidence selection processandreasoning process. Third, to avoid the influences of knowledge bias and concentrate purely on benchmarking the logical reasoning capability of LLMs, we propose a new dataset with neutral content. Based on the in-depth evaluations, this paper finally forms a general evaluation scheme of logical reasoning capability from six dimensions (i.e.,Correct,Rigorous,Self-aware,Active,OrientedandNo hallucination). It reflects the pros and cons of LLMs and gives guiding directions for future works.

Key Point Analysis (KPA) aims for quantitative summarization that provides key points (KPs) as succinct textual summaries and quantities measuring their prevalence.KPA studies for arguments and reviews have been reported in the literature.A majority of KPA studies for reviews adopt supervised learning to extract short sentences as KPs before matching KPs to review comments for quantification of KP prevalence.Recent abstractive approaches still generate KPs based on sentences, often leading to KPs with overlapping and hallucinated opinions, and inaccurate quantification.In this paper, we propose Prompted Aspect Key Point Analysis (PAKPA) for quantitative review summarization.PAKPA employs aspect sentiment analysis and prompted in-context learning with Large Language Models (LLMs) to generate and quantify KPs grounded in aspects for business entities, which achieves faithful KPs with accurate quantification, and removes the need for large amounts of annotated data for supervised training.Experiments on the popular review dataset Yelp and the aspect-oriented review summarization dataset SPACE show that our framework achieves state-of-the-art performance.Source code and data are available at: https://github.com/ antangrocket1312/PAKPA Key Point Prevalence Matching Comments Friendly and helpful staff.46 From the minute I walked in the door the staff has treated me with such kindness and respect, there are not enough words I can say of how much gratitude I have for the staff here.It's a fine hotel, pretty basic, and all of the staff members we encountered were quite friendly.Excellent location for business travelers.37 Its a great location for business travelers since I can stay here and always be going the opposite way of traffic.*This hotel has an IDEAL location, the shuttle was perfect, and we got a great deal on priceline.Clean and comfortable rooms.32 *You will enjoy the breathtaking views from your spacious, clean and comfortable room.The rooms were nice and clean and the bed was comfortable.Convenient and helpful shuttle service.12 *This hotel has an IDEAL location, the shuttle was perfect, and we got a great deal on priceline.I would also like to give a shout out to the terrific shuttle drivers Jeff and Rod who were so great to my family group.Amazing view of Vanderbilt football stadium or The Parthenon.10 *You will enjoy the breathtaking views from your spacious, clean and comfortable room.Then the surprise came when we opened the curtains to see a full on view of the Vanderbilt football stadium.

The current charge prediction datasets mostly focus on single-defendant criminal cases.However, real-world criminal cases usually involve multiple defendants whose criminal facts are intertwined.In an early attempt to fill this gap, we introduce a new benchmark that encompasses legal cases involving multiple defendants, where each defendant is labeled with a charge and four types of crime elements, i.e., Object Element, Objective Element, Subject Element, and Subjective Element.Based on the dataset, we further develop an interpretable model called EJudge that incorporates crime elements and legal rules to infer charges.We observe that predicting crime charges while providing corresponding rationales benefits the interpretable AI system.Extensive experiments show that EJudge significantly surpasses state-of-the-art methods, which verify the importance of crime elements and legal rules in multi-defendant charge prediction.

LLMs as code generators have undergone rapid progress over the past couple of years. However, the models on their own provide no guarantees for the functional correctness of the generated code. Functional tests can not only be used by designers to assess the functional correctness of code, but also to guide them towards the solution of the problem. The same can be applied to LLMs performing automatic code generation through the use of the Reflexion technique. Reflexion is an agent-based workflow where the model generating code iterates over a loop of code generation, getting feedback from the test bench, reflecting on the feedback, and making appropriate changes to the code. The technique is known to drastically improve the performance of LLMs on software code generation. In this work, we adopt the technique for hardware description language (HDL) code generation as RTL Agent - an implementation of the workflow for Verilog generation with Reflexion. We compare multiple LLMs with standard inference vs with Reflexion on the VerilogEval benchmark. Within 5 iterations of the feedback loop, we observe a relative improvement of 33.2% for the low-performance model Llama3 and an average of 17.7% for the high-performance models GPT-4o and GPT-4o-mini in their Pass@1 performance. We present a cost analysis of the technique to enable cost-performance trade-offs.

Predicting stock price movements is a high-stakes task that demands explainability for human decision-makers. A key shortcoming in current methods is treating sub-predictions independently, without learning from accumulated experiences. We propose a novel triplet network for contrastive learning to enhance the explainability of stock movement prediction by considering instances of "integrated textual information and quantitative indicators". We refer to the target past-l-day tweet-price time series as the "anchor instance". Each anchor instance is paired with a "positive instance" characterized by highly correlated return trends yet significant differences across the entire feature space, and a "negative instance" that exhibits similar return trends along with high proximity in the feature space. The model is designed with the objective of (1) minimizing the cross entropy loss between input logits and target, (2) minimizing the distance between the anchor instances and positive instances, and (3) maximizing the distance between the anchor instances and negative instances. Our framework's effectiveness is demonstrated through extensive testing, showing superior performance on stock prediction benchmarks.

Cognitive reasoning holds a significant place within Natural Language Processing (NLP). Yet, the exploration of zero-shot scenarios, which align more closely with real-life situations than supervised scenarios, has been relatively limited. While a few studies have employed Large Language Models (LLMs) to tackle zero-shot cognitive reasoning tasks, they still grapple with two key challenges: 1) Traditional approaches rely on the chain-of-thought (CoT) mechanism, wherein LLMs are provided with a "Let's think step by step'' prompt. However, this schema may not accurately understand the meaning of a given question and ignores the possible learned knowledge (e.g., background or commonsense) of the LLMs about the questions, leading to incorrect answers. 2) Previous CoT methods normally exploit a single Large Language Model (LLM) and design many strategies to augment this LLM. We argue that the power of a single LLM is typically finite since it may not have learned some relevant knowledge about the question. To address these issues, we propose a Multi-LLM Knowledge Fusion (MLKF) approach, which resorts to heterogeneous knowledge emerging from multiple LLMs, for zero-shot cognitive reasoning tasks. Through extensive experiments and detailed analysis, we demonstrate that our MLKF can outperform the existing zero-shot or unsupervised state-of-the-art methods on four kinds of zero-shot tasks: aspect sentiment analysis, named entity recognition, question answering, and mathematical reasoning. Our code is available at https://github.com/trueBatty/MLKF

The emergence of ChatGPT has generated much speculation in the press about its potential to disrupt social and economic systems. Its astonishing language ability has aroused strong curiosity among scholars about its performance in different domains. There have been many studies evaluating the ability of ChatGPT and GPT-4 in different tasks and disciplines. However, a comprehensive review summarizing the collective assessment findings is lacking. The objective of this survey is to thoroughly analyze prior assessments of ChatGPT and GPT-4, focusing on its language and reasoning abilities, scientific knowledge, and ethical considerations. Furthermore, an examination of the existing evaluation methods is conducted, offering several recommendations for future research.

Sarcasm poses a challenge in linguistic analysis due to its implicit nature, involving an intended meaning that contradicts the literal expression. The advent of social networks has propelled the utilization of multimodal data to enhance sarcasm detection performance. In prior multimodal sarcasm detection datasets, a single label is assigned to a multimodal instance. Subsequent experiments often highlight the superiority of multimodal models by demonstrating their improvements compared to unimodal models based on these unified labels across multiple modalities. However, our investigation revealed that numerous instances of sarcasm cannot be identified using a single modality. Humans employ the conflict between a statement and factual information as a cue to detect sarcasm, and these cues can stem from different modalities. Then, a unified label for a multimodal instance may be not suitable for the associated text or image. In this work, we introduce SarcNet, a multilingual and multimodal sarcasm detection dataset in English and Chinese, consisting of 3,335 image-text pair samples. We provide annotations for sarcasm in visual, textual, and multimodal data, respectively, resulting in over 10,000 labeled instances. The separated annotation schema for unimodal and multimodal data facilitates a more accurate and reasonable assessment of unimodal and multimodal models.

Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, Furu Wei. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Cross-domain Named Entity Recognition (CD-NER) is crucial for Knowledge Graph (KG) construction and natural language processing (NLP), enabling learning from source to target domains with limited data.Previous studies often rely on manually collected entity-relevant sentences from the web or attempt to bridge the gap between tokens and entity labels across domains.These approaches are time-consuming and inefficient, as these data are often weakly correlated with the target task and require extensive pre-training.To address these issues, we propose automatically generating task-oriented knowledge (GTOK) using large language models (LLMs), focusing on the reasoning process of entity extraction.Then, we employ taskoriented pre-training (TOPT) to facilitate domain adaptation.Additionally, current crossdomain NER methods often lack explicit explanations for their effectiveness.Therefore, we introduce the concept of information density to better evaluate the model's effectiveness before performing entity recognition.We conduct systematic experiments and analyses to demonstrate the effectiveness of our proposed approach and the validity of using information density for model evaluation † .

The prediction of stock price movements is challenging due to the inherently dynamic and complex characteristics of financial markets. A current research gap is the lack of exploration into the complex interrelationships inherent in stock price dynamics, often analyzing predictions in isolation with an implicit presumption that solely the historical data of a given stock influences its future trend. However, stock prices are impacted by a diverse array of driving factors that extend beyond the traditionally examined historical prices, encompassing influences such as inter-stock correlations. In this paper, we present a predictive approach using a dynamic dual-graph neural network. The network combines textual data and quantitative metrics to capture multiple dynamic relationships. Specifically, We have developed a price relationship graph (PRG) and a semantic relationship graph (SRG), which are later integrated using a graph attention neural network. The effectiveness of our neural architecture is validated through extensive testing on two benchmark datasets for stock movement prediction, illustrating its superior performance compared to other graph-based networks for stock market prediction.

The task of multilingual emotion recognition holds significant importance in cross-cultural communication and data mining. While prior research has concentrated on enhancing classification accuracy using state-of-the-art techniques, it has often overlooked a crucial linguistic aspect—the semantic disparities across different languages. This study aims to address this gap by introducing a novel method to identify lexical semantic variations in diverse languages. The detected semantic variation features are subsequently injected into a multilingual emotion recognition model to enhance its performance within a target language. Notably, existing multilingual pre-trained language models are likely biased toward English word meanings, leading to inaccurate emotion predictions in other languages due to the misinterpretation of semantics. Our proposed semantic variation injection method tackles this limitation, resulting in improved accuracy. These findings contribute to the ongoing development of robust and culturally sensitive emotion recognition systems, offering valuable insights for both the linguistics and computational linguistics communities engaged in multilingual research.

While existing Aspect-based Sentiment Analysis (ABSA) has received extensive effort and advancement, there are still gaps in defining a more holistic research target seamlessly integrating multimodality, conversation context, fine-granularity, and also covering the changing sentiment dynamics as well as cognitive causal rationales. This paper bridges the gaps by introducing a multimodal conversational ABSA, where two novel subtasks are proposed: 1) Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder, target, aspect, opinion, sentiment, rationale from multi-turn multi-party multimodal dialogue. 2) Sentiment Flipping Analysis, detecting the dynamic sentiment transformation throughout the conversation with the causal reasons. To benchmark the tasks, we construct PanoSent, a dataset annotated both manually and automatically, featuring high quality, large scale, multimodality, multilingualism, multi-scenarios, and covering both implicit&explicit sentiment elements. To effectively address the tasks, we devise a novel Chain-of-Sentiment reasoning framework, together with a novel multimodal large language model (namely Sentica) and a paraphrase-based verification mechanism. Extensive evaluations demonstrate the superiority of our methods over strong baselines, validating the efficacy of all our proposed methods. The work is expected to open up a new era for the ABSA community, and thus all our codes and data are open at https://PanoSent.github.io/.

Medical report generation aims at automating the synthesis of accurate and comprehensive diagnostic reports from radiological images. The task can significantly enhance clinical decision-making and alleviate the workload on radiologists. Existing works normally generate reports from single chest radiographs, although historical examination data also serve as crucial references for radiologists in real-world clinical settings. To address this constraint, we introduce a novel framework that mimics the workflow of radiologists. This framework compares past and present patient images to monitor disease progression and incorporates prior diagnostic reports as references for generating current personalized reports. We tackle the textual diversity challenge in cross-modal tasks by promoting style-agnostic discrete report representation learning and token generation. Furthermore, we propose a novel spatio-temporal fusion method with multi-granularities to fuse textual and visual features by disentangling the differences between current and historical data. We also tackle token generation biases, which arise from long-tail frequency distributions, proposing a novel feature normalization technique. This technique ensures unbiased generation for tokens, whether they are frequent or infrequent, enabling the robustness of report generation for rare diseases. Experimental results on the two public datasets demonstrate that our proposed model outperforms state-of-the-art baselines.

Abstract Text classification is the process of labelling a given set of text documents with predefined classes or categories. Existing Arabic text classifiers are either applying classic Machine Learning algorithms such as k‐NN and SVM or using modern deep learning techniques. The former are assessed using small text collections and their accuracy is still subject to improvement while the latter are efficient in classifying big data collections and show limited effectiveness in classifying small corpora with a large number of categories. This paper proposes a new approach to Arabic text classification to treat small and large data collections while improving the classification rates of existing classifiers. We first demonstrate the ability of analogical proportions (AP) (statements of the form ‘x is to as is to ’), which have recently been shown to be effective in classifying ‘structured’ data, to classify ‘unstructured’ text documents requiring preprocessing. We design an analogical model to express the relationship between text documents and their real categories. Next, based on this principle, we develop two new analogical Arabic text classifiers. These rely on the idea that the category of a new document can be predicted from the categories of three others, in the training set, in case the four documents build together a ‘valid’ analogical proportion on all or on a large number of components extracted from each of them. The two proposed classifiers (denoted AATC1 and AATC2) differ mainly in terms of the keywords extracted for classification. To evaluate the proposed classifiers, we perform an extensive experimental study using five benchmark Arabic text collections with small or large sizes, namely ANT (Arabic News Texts) v2.1 and v1.1, BBC‐Arabic, CNN‐Arabic and AlKhaleej‐2004. We also compare analogical classifiers with both classical ML‐based and Deep Learning‐based classifiers. Results show that AATC2 has the best average accuracy (78.78%) over all other classifiers and the best average precision (0.77) ranked first followed by AATC1 (0.73), NB (0.73) and SVM (0.72) for the ANT corpus v2.1. Besides, AATC1 shows the best average precisions (0.88) and (0.92), respectively for the BBC‐Arabic corpus and AlKhaleej‐2004, and the best average accuracy (85.64%) for CNN‐Arabic over all other classifiers. Results demonstrate the utility of analogical proportions for text classification. In particular, the proposed analogical classifiers are shown to significantly outperform a number of existing Arabic classifiers, and in many cases, compare favourably to the robust SVM classifier.

Abstract The use of computer‐mediated communication has resulted in a new form of written text called Microtext, which is very different from well‐written text. Most previous approaches deal with microtext at the character level rather than just words resulting in increased processing time. In this paper, we propose to transform static word vectors to dynamic form by modelling the effect of neighbouring words and their sentiment strength in the AffectiveSpace. To evaluate the approach, we crawled Tweets from diverse topics and human annotation was used to label their sentiments. We also normalized the tweets to fix phonetic variations, spelling errors, and abbreviations manually. A total of 1432 out‐of‐vocabulary (OOV) texts and their IV texts made it to the final corpus with their corresponding polarity. To assess the quality of the corpus, we used several OOV classifiers such as linear regression and observed over 90% accuracy. Next, we inferred word vectors using a novel four‐gram model based on sentiment intensity and reported accuracy on both open domain and closed domain sentiment classifiers. We observed an improvement in the range of 4–20 on Twitter, Movie and Airline reviews over baselines.

Abstract There has been surge in the usage of Internet as well as social media platforms which has led to rise in online hate speech targeted on individual or group. In the recent years, hate speech has resulted in one of the challenging problems that can unfurl at a fast pace on digital platforms leading to various issues such as prejudice, violence and even genocide. Considering the acceptance of Artificial Intelligence (AI) and Natural Language Processing (NLP) techniques in varied application domains, it would be intriguing to consider these techniques for automated hate speech detection. In literature, there have been efforts to recognize and categorize hate speech using varied Machine Learning (ML) and Deep Learning (DL) techniques. Hence, considering the need and provocations for hate speech detection we aim to present a comprehensive review that discusses fundamental taxonomy as well as recent advances in the field of online hate speech identification. There is a significant amount of literature related to the initial phases of hate speech detection. The background section provides a detailed explanation of the previous research. The subsequent section that follows is dedicated to examining the recent literature published from the year 2020 onwards. The paper presents some of the hate speech datasets considered for hate speech detection. Furthermore, the paper discusses different data modalities, namely, textual hate speech detection, multi‐modal hate speech detection and multilingual hate speech detection. Apart from systematic review on hate speech detection, the paper also implement several multi‐label models to compare the performance of hate speech detection by employing classic ML technique namely, Logistic Regression and DL technique namely, Long Short‐Term Memory (LSTM) and a multiclass multi‐label architecture. In the implemented architecture, we have derived two new elements to quantify the hatefulness and intensity of hatred to improve the results for hate speech detection using Indonesian tweet dataset. Empirical Analysis of the model reveals that the implemented approach outperforms and is able to achieve improved results for the underlying dataset.

With the rise of foundation models, a new artificial intelligence paradigm has emerged, by simply using general purpose foundation models with prompting to solve problems instead of training a separate machine learning model for each problem. Such models have been shown to have emergent properties of solving problems that they were not initially trained on. The studies for the effectiveness of such models are still quite limited. In this work, we widely study the capabilities of the ChatGPT models, namely GPT-4 and GPT-3.5, on 13 affective computing problems, namely aspect extraction, aspect polarity classification, opinion extraction, sentiment analysis, sentiment intensity ranking, emotions intensity ranking, suicide tendency detection, toxicity detection, well-being assessment, engagement measurement, personality assessment, sarcasm detection, and subjectivity detection. We introduce a framework to evaluate the ChatGPT models on regression-based problems, such as intensity ranking problems, by modelling them as pairwise ranking classification. We compare ChatGPT against more traditional NLP methods, such as end-to-end recurrent neural networks and transformers. The results demonstrate the emergent abilities of the ChatGPT models on a wide range of affective computing problems, where GPT-3.5 and especially GPT-4 have shown strong performance on many problems, particularly the ones related to sentiment, emotions, or toxicity. The ChatGPT models fell short for problems with implicit signals, such as engagement measurement and subjectivity detection.

Sentiment analysis can be used to derive knowledge that is connected to emotions and opinions from textual data generated by people. As computer power has grown, and the availability of benchmark datasets has increased, deep learning models based on deep neural networks have emerged as the dominant approach for sentiment analysis. While these models offer significant advantages, their lack of interpretability poses a major challenge in comprehending the rationale behind their reasoning and prediction processes, leading to complications in the models' explainability. Further, only limited research has been carried out into developing deep learning models that describe their internal functionality and behaviors. In this timely study, we carry out a first of its kind overview of key sentiment analysis techniques and eXplainable artificial intelligence (XAI) methodologies that are currently in use. Furthermore, we provide a comprehensive review of sentiment analysis explainability.

Identifying sarcastic clues from both textual and visual information has become an important research issue, called Multimodal Sarcasm Detection. In this paper, we investigate multimodal sarcasm detection from a novel perspective, where a multimodal graph contrastive learning strategy is proposed to fuse and distinguish the sarcastic clues for textual modality and visual modality. Specifically, we first utilize object detection to derive the crucial visual regions accompanied by their captions of the images, which allows better learning of the key visual regions of visual modality. In addition, to make full use of the semantic information of the visual modality, we employ optical character recognition to extract the textual content in the images. Then, based on image regions, the textual content of visual modality, and the context of the textual modality, we build a multimodal graph for each sample to model the intricate sarcastic relations between modalities. Furthermore, we devise a graph-oriented contrastive learning strategy to leverage the correlations in the same label and differences between different labels, so as to capture better multimodal representations for multimodal sarcasm detection. Extensive experiments show that our method outperforms the previous best baseline models (with a 2.47% improvement in Accuracy, a 1.99% improvement in F-score, and a 2.20% improvement in Macro F-score). The ablation study shows that both multimodal graph structure and graph-oriented contrastive learning are important to our framework. Further, the experiments of using different pre-trained methods show that the proposed multimodal graph contrastive learning framework can directly work with various pre-trained models and achieve outstanding performance in multimodal sarcasm detection.

Prompt tuning has achieved great success in various sentence-level classification tasks by using elaborated label word mappings and prompt templates. However, for solving token-level classification tasks, e.g., named entity recognition (NER), previous research, which utilizes N-gram traversal for prompting all spans with all possible entity types, is time-consuming. To this end, we propose a novel prompt-based contrastive learning method for few-shot NER without template construction and label word mappings. First, we leverage external knowledge to initialize semantic anchors for each entity type. These anchors are simply appended with input sentence embeddings as template-free prompts (TFPs). Then, the prompts and sentence embeddings are in-context optimized with our proposed semantic-enhanced contrastive loss. Our proposed loss function enables contrastive learning in few-shot scenarios without requiring a significant number of negative samples. Moreover, it effectively addresses the issue of conventional contrastive learning, where negative instances with similar semantics are erroneously pushed apart in natural language processing (NLP)-related tasks. We examine our method in label extension (LE), domain-adaption (DA), and low-resource generalization evaluation tasks with six public datasets and different settings, achieving state-of-the-art (SOTA) results in most cases.

To date, most of the existing open-domain question answering (QA) methods focus on explicit questions where the reasoning steps are mentioned explicitly in the question. In this article, we study implicit QA where the reasoning steps are not evident in the question. Implicit QA is challenging in two aspects. First, evidence retrieval is difficult since there is little overlap between a question and its required evidence. Second, answer inference is difficult since the reasoning strategy is latent in the question. To tackle implicit QA, we propose a systematic solution denoted as DisentangledQA, which disentangles topic, attribute, and reasoning strategy from the implicit question to guide the retrieval and reasoning. Specifically, we disentangle the topic and attribute information from the implicit question to guide evidence retrieval. For answer reasoning, we propose a disentangled reasoning model for answer prediction based on retrieved evidence as well as the latent representation of the reasoning strategy. The disentangled framework empowers each module to focus on a specific latent element in the question, and thus, leads to effective representation learning for them. Experiments on the StrategyQA dataset demonstrate the effectiveness of our method in answering implicit questions, improving performance in evidence retrieval and answering inference by 31.7% and 4.5%, respectively, and achieving the best performance on the official leaderboard. In addition, our method achieved the best performance on the challenging EntityQuestions dataset, indicating the effectiveness in improving general open-domain QA tasks.

Emotion recognition in conversation (ERC) has received increasing attention from the research community. However, the ERC task is challenging, largely due to the complex and unstructured properties of multi-party conversations. Besides, the majority of daily dialogues take place in a specific context or circumstance, which requires rich external knowledge to understand the background of a certain dialogue. In this paper, we address these challenges by explicitly modeling the discourse relations between utterances and incorporating symbolic knowledge into multi-party conversations. We first introduce a dialogue parsing algorithm into ERC and further improve the algorithm through a transfer learning method. Moreover, we leverage different symbolic knowledge graph relations to learn knowledge-enhanced features for the ERC task. Extensive experiments on three benchmarks demonstrate that both dialogue structure graphs and symbolic knowledge are beneficial to the model performance on the task. Additionally, experimental results indicate that the proposed model surpasses baseline models on several indices.

In cross-lingual named entity recognition (NER), self-training is commonly used to bridge the linguistic gap by training on pseudolabeled target-language data.However, due to sub-optimal performance on target languages, the pseudo labels are often noisy and limit the overall performance.In this work, we aim to improve self-training for cross-lingual NER by combining representation learning and pseudo label refinement in one coherent framework.Our proposed method, namely ContProto mainly comprises two components: (1) contrastive self-training and (2) prototype-based pseudo-labeling.Our contrastive self-training facilitates span classification by separating clusters of different classes, and enhances crosslingual transferability by producing closelyaligned representations between the source and target language.Meanwhile, prototype-based pseudo-labeling effectively improves the accuracy of pseudo labels during training.We evaluate ContProto on multiple transfer pairs, and experimental results show our method brings in substantial improvements over current stateof-the-art methods. 1

Extrapolation reasoning on temporal knowledge graphs (TKGs) aims to forecast future facts based on past counterparts.There are two main challenges: (1) incorporating the complex information, including structural dependencies, temporal dynamics, and hidden logical rules;(2) implementing differentiable logical rule learning and reasoning for explainability.To this end, we propose an explainable extrapolation reasoning framework TEemporal logiCal grapH networkS (TECHS), which mainly contains a temporal graph encoder and a logical decoder.The former employs a graph convolutional network with temporal encoding and heterogeneous attention to embed topological structures and temporal dynamics.The latter integrates propositional reasoning and first-order reasoning by introducing a reasoning graph that iteratively expands to find the answer.A forward message-passing mechanism is also proposed to update node representations, and their propositional and first-order attention scores.Experimental results demonstrate that it outperforms state-of-the-art baselines.

Recent studies have revealed some issues of Multi-Head Attention (MHA), e.g., redundancy and over-parameterization.Specifically, the heads of MHA were originally designed to attend to information from different representation subspaces, whereas prior studies found that some attention heads likely learn similar features and can be pruned without harming performance.Inspired by the minimumredundancy feature selection, we assume that focusing on the most representative and distinctive features with minimum resources can mitigate the above issues and lead to more effective and efficient MHAs.In particular, we propose Grouped Head Attention, trained with a self-supervised group constraint that group attention heads, where each group focuses on an essential but distinctive feature subset.We additionally propose a Voting-to-Stay procedure to remove redundant heads, thus achieving a transformer with lighter weights.Moreover, our method achieves significant performance gains on three well-established tasks while considerably compressing parameters.

Persona attribute extraction is critical for personalized human-computer interaction.Dialogue is an important medium that communicates and delivers persona information.Although there is a public dataset for triplet-based persona attribute extraction from conversations, its automatically generated labels present many issues, including unspecific relations and inconsistent annotations.We fix such issues by leveraging more reliable text-label matching criteria to generate high-quality data for persona attribute extraction.We also propose a contrastive learning-and generation-based model with a novel hard negative sampling strategy for generalized zero-shot persona attribute extraction.We benchmark our model with stateof-the-art baselines on our dataset and a public dataset, showing outstanding accuracy gains.Our sampling strategy also exceeds others by a large margin in persona attribute extraction.

Pretrained language models have been shown to store knowledge in their parameters and have achieved reasonable performance in commonsense knowledge base completion (CKBC) tasks.However, CKBC is knowledge-intensive and it is reported that pretrained language models' performance in knowledge-intensive tasks are limited because of their incapability of accessing and manipulating knowledge.As a result, we hypothesize that providing retrieved passages that contain relevant knowledge as additional input to the CKBC task will improve performance.In particular, we draw insights from Case-Based Reasoning (CBR) -which aims to solve a new problem by reasoning with retrieved relevant cases, and investigate the direct application of it to CKBC.On two benchmark datasets, we demonstrate through automatic and human evaluations that our End-to-end Case-Based Reasoning Framework (ECBRF) generates more valid knowledge than the state-of-the-art COMET model for CKBC in both the fully supervised and few-shot settings.From the perspective of CBR, our framework addresses a fundamental question on whether CBR methodology can be utilized to improve deep learning models.

Learning on a massive amount of speech corpus leads to the recent success of many self-supervised speech models. With knowledge distillation, these models may also benefit from the knowledge encoded by language models that are pre-trained on rich sources of texts. The distillation process, however, is challenging due to the modal disparity between textual and speech embedding spaces. This paper studies metric-based distillation to align the embedding space of text and speech with only a small amount of data without modifying the model structure. Since the semantic and granularity gap between text and speech has been omitted in literature, which impairs the distillation, we propose the Prior-informed Adaptive knowledge Distillation (PAD) that adaptively leverages text/speech units of variable granularity and prior distributions to achieve better global and local alignments between text and speech pre-trained models. We evaluate on three spoken language understanding benchmarks to show that PAD is more effective in transferring linguistic knowledge than other metric-based distillation approaches.

The availability of new datasets and deep learning techniques have led to a surge of effort directed towards the creation of new models that can exploit the large amount of data. However, little attention has been given to the development of models that are not only accurate, but also suitable for user-specific use or geared towards resource-constrained devices. Fine-tuning deep models on edge devices is impractical and, often, user customization stands on the sub-optimal feature-extractor/classifier paradigm. Here, we propose a method to fully utilize the intermediate outputs of the popular large pre-trained models in natural language processing when used as frozen feature extractors, and further close the gap between their fine-tuning and more computationally efficient solutions. We reach this goal exploiting the concept of software-hardware co-design and propose a methodical procedure, inspired by Neural Architecture Search, to select the most desirable model taking into consideration application constraints.

Metaphors frequently appear in financial news headlines due to their ability to effectively convey complex financial concepts and market trends in a concise and memorable manner. Cognitive scientists have found that metaphors serve as the reflections of human cognition by means of concept mappings. In this work, we aim to analyze the metaphorical expressions and associated cognitive patterns employed by financial analysts in the headlines of financial analysis reports. Such an examination would enhance our comprehension of the cognitive state of financial analysts regarding various financial trends. We employ the latest computational metaphor processing tool, MetaPro to achieve this target by mining metaphors and cognitive patterns from 1,407,328 financial analyst report headlines, spanning the period from 14 February 2009 to 11 June 2020. We analyze the mined concept mappings by different time periods, and market movements, and deliver novel findings in these two dimensions.

The task of resolving repeated objects in natural languages is known as co-reference resolution. It is an important part of modern natural language processing and semantic cognition as these implicit relationships are particularly difficult in natural language understanding in downstream tasks. Mention identification and mention linking are the two sub-tasks in the general co-reference resolution research community. Gold-two-mention style co-reference resolution is a special type of co-reference resolution that focuses on linking the ambiguous pronoun to one of the two candidate antecedents. In this paper, we proposed a joint learning model that learns mention identification and mention linking tasks together, because we find that the learning of mention identification can provide supportive dependent information for the learning of mention linking. As far as we know, we propose the first model that introduces a multi-task learning framework to the gold-two-mention co-reference resolution task. We find that our proposed model outperforms state-of-the-art baselines and a single-task learning model on three gold-two-mention co-reference resolution datasets. By comparing the errors made by either the single-task learning model or the multi-task learning model, our error analysis also yields interesting findings about in which way our multi-task learning model makes fewer resolution errors.

The 4th Multimodal Sentiment Analysis Challenge (MuSe) focuses on Multimodal Prediction of Mimicked Emotions, Cross-Cultural Humour, and Personalised Recognition of Affects. The workshop takes place in conjunction with ACM Multimedia'23. We provide three datasets as part of the challenge: (i) The Hume-Vidmimic dataset which offers 30+ hours of expressive behaviour data from 557 participants. It involves mimicking and rating emotions: Approval, Disappointment, and Uncertainty. This multimodal resource is valuable for studying human emotional expressions. (ii) The 2023 edition of the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset comprises German football press conference recordings within the training set, while videos of English football press conferences are included in the unseen test set. This unique configuration offers a cross-cultural evaluation environment for humour recognition. (iii) The Ulm-Trier Social Stress Test (Ulm-TSST) dataset contains recordings of subjects under stress. It involves arousal and valence signals, with some test labels provided to aid personalisation. Based on these datasets, we formulate three multimodal affective computing challenges: (1) Mimicked Emotions Sub-Challenge (MuSe-Mimic) for categorical emotion prediction, (2) Cross-Cultural Humour Detection Sub-Challenge (MuSe-Humour) for cross-cultural humour detection, and (3) Personalisation Sub-Challenge (MuSe-Personalisation) for personalised dimensional emotion recognition. In this summary, we outline the challenge's motivation, participation guidelines, conditions, and results.

Multimodal emotion recognition has become an important research topic due to its wide applications in human-computer interaction. Over the last few decades, the technology has made remarkable progress with the development of deep learning. However, existing technologies are hard to meet the demand for practical applications. To this end, we organize this workshop to bring together researchers in this field to further discuss recent research and future directions.

The first Multimodal Emotion Recognition Challenge (MER 2023)1 was successfully held at ACM Multimedia. The challenge focuses on system robustness and consists of three distinct tracks: (1) MER-MULTI, where participants are required to recognize both discrete and dimensional emotions; (2) MER-NOISE, in which noise is added to test videos for modality robustness evaluation; (3) MER-SEMI, which provides a large amount of unlabeled samples for semi-supervised learning. In this paper, we introduce the motivation behind this challenge, describe the benchmark dataset, and provide some statistics about participants. To continue using this dataset after MER 2023, please sign a new End User License Agreement2 and send it to our official email address3. We believe this high-quality dataset can become a new benchmark in multimodal emotion recognition, especially for the Chinese research community.

Abstract A growing amount of research use pre‐trained language models to address few/zero‐shot text classification problems. Most of these studies neglect the semantic information hidden implicitly beneath the natural language names of class labels and develop a meta learner from the input texts solely. In this work, we demonstrate how label information can be utilized to extract enhanced feature representation of the input text from a Transformer‐based pre‐trained language model such as AraBERT. In addition, how this approach can improve performance when the data resources are scarce like in the Arabic language and the input text is short with little semantic information as is the case using tweets. The work also applies zero‐shot text classification to predict new classes with no training examples across different domains including sarcasm detection and sentiment analysis using the information in the last layer of a trained classifier in a transfer learning setting. Experiments show that our approach has a better performance for the few‐shot sentiment classification compared to baseline models and models trained without augmenting label information. Moreover, the zero‐shot implementation achieved an accuracy up to 0.874 in Arabic sarcasm detection from a model trained on a sentiment analysis task.

Aspect-based sentiment analysis (ABSA) means to identify fine-grained aspects, opinions, and sentiment polarities. Recent ABSA research focuses on utilizing multi-task learning (MTL) to achieve less computational costs and better performance. However, there are certain limits in MTL-based ABSA. For example, unbalanced labels and sub-task learning difficulties may result in the biases that some labels and sub-tasks are overfitting, while the others are underfitting. To address these issues, inspired by neuro-symbolic learning systems, we propose a meta-based self-training method with a meta-weighter (MSM). We believe that a generalizable model can be achieved by appropriate symbolic representation selection (in-domain knowledge) and effective learning control (regulation) in a neural system. Thus, MSM trains a teacher model to generate in-domain knowledge (e.g., unlabeled data selection and pseudo-label generation), where the generated pseudo-labels are used by a student model for supervised learning. Then, the meta-weighter of MSM is jointly trained with the student model to provide each instance with sub-task-specific weights to coordinate their convergence rates, balancing class labels, and alleviating noise impacts introduced from self-training. The following experiments indicate that MSM can utilize 50% labeled data to achieve comparable results to state-of-arts models in ABSA and outperform them with all labeled data.

Emotion recognition in conversations (ERC) has wide applications in medical care, human-computer interaction, and other fields. Unlike the general task of emotion analysis, humans usually rely on context and commonsense knowledge to convey emotions in conversations. Only when the model can connect and fully utilize a large-scale commonsense knowledge base, it can better understand latent contents in conversations. Unfortunately, there is no available knowledge selection mechanism to address such knowledge needs and to make sure the system is not flooded with irrelevant commonsense knowledge. Therefore, we propose an AutoML strategy based on emotion congruent effect to select suitable knowledge and models, called AutoML-Emo. Global exploration and local exploitation-based selection mechanisms (G&LESM) are used for automatic knowledge selection. The transformer-based architecture search (TAS) is applied to model selection, the selected transformer-based model is employed to incorporate knowledge and capture context information in conversations. The experimental results show that AutoML-Emo can effectively enhance external knowledge in different sizes and domain datasets. Moreover, the selected transformer-based model derived from TAS is superior to the most advanced models.

Conversational sentiment analysis (CSA) and emotion-cause pair extraction (ECPE) tasks have attracted increasing attention in recent years. The former aims to predict the sentiment states of speakers in a conversation, and the latter is about extracting emotion-cause clauses in a document. However, one drawback of CSA is that it cannot model the causal reasoning among emotion and neutral utterances from different speakers. In this work, we propose a new task: emotion-cause pair extraction in conversations (ECPEC), which aims to extract pairs of emotional utterances and corresponding cause utterances in conversations. The utterance-level ECPEC task is more challenging since the distance between emotion and cause utterances is larger than that of the clause-level ECPE task. To this end, we build a novel dataset ConvECPE and propose a specifically designed two-step framework for the new ECPEC task. Experimental results on ConvECPE dataset demonstrate the feasibility of the ECPEC task as well as the effectiveness of our framework.

Thanks to the breakthrough of large-scale pre-trained language model (PLM) technology, prompt-based classification tasks, e.g., sentiment analysis and emotion detection, have raised increasing attention. Such tasks are formalized as masked language prediction tasks which are in line with the pre-training objects of most language models. Thus, one can use a PLM to infer the masked words in a downstream task, then obtaining label predictions with manually defined label-word mapping templates. Prompt-based affective computing takes the advantages of both neural network modeling and explainable symbolic representations. However, there still remain many unclear issues related to the mechanisms of PLMs and prompt-based classification. We conduct a systematic empirical study on prompt-based sentiment analysis and emotion detection to study the biases of PLMs towards affective computing. We find that PLMs are biased in sentiment analysis and emotion detection tasks with respect to the number of label classes, emotional label-word selections, prompt templates and positions, and the word forms of emotion lexicons.

Neural network-based methods, especially deep learning, have been a burgeoning area in AI research and have been successful in tackling the expanding data volume as we move into a digital age. Today, the neural network-based methods are not only used for low-level cognitive tasks, such as recognizing objects and spotting keywords, but they have also been deployed in various industrial information systems to assist high-level decision-making. In natural language processing, there have been two milestones for the past decade: one is word2vec [1], a group of neural models that learn word embeddings (vector representations of words) from large datasets; and one is the most recent GPT-based models [2], which combine reinforcement learning with a generative transformer in order to enable multi-round end-to-end conversations. While producing highly accurate predictions on datasets and generating human-like utterances, those neural network-based artifacts provide little understanding of the internal features and representations of the data. Many problems and concerns subsequently emerge from this black-box issue. Because some of the problems and concerns are also relevant in the context of sentiment analysis.

Changes in human lifestyle have led to an increase in the number of people suffering from depression over the past century. Although in recent years, rates of diagnosing mental illness have improved, many cases remain undetected. Automated detection methods can help identify depressed or individuals at risk. An understanding of depression detection requires effective feature representation and analysis of language use. In this article, text classifiers are trained for depression detection. The key objective is to improve depression detection performance by examining and comparing two sets of methods: hybrid and ensemble. The results show that ensemble models outperform the hybrid model classification results. The strength and effectiveness of the combined features demonstrate that better performance can be achieved by multiple feature combinations and proper feature selection.

Human coders assign standardized medical codes to clinical documents generated during patients’ hospitalization, which is error prone and labor intensive. Automated medical coding approaches have been developed using machine learning methods, such as deep neural networks. Nevertheless, automated medical coding is still challenging because of complex code association, noise in lengthy documents, and the imbalanced class problem. We propose a novel neural network, called the Multitask Balanced and Recalibrated Neural Network, to solve these issues. Significantly, the multitask learning scheme shares the relationship knowledge between different coding branches to capture code association. A recalibrated aggregation module is developed by cascading convolutional blocks to extract high-level semantic features that mitigate the impact of noise in documents. Also, the cascaded structure of the recalibrated module can benefit learning from lengthy notes. To solve the imbalanced class problem, we deploy focal loss to redistribute the attention on low- and high-frequency medical codes. Experimental results show that our proposed model outperforms competitive baselines on a real-world clinical dataset called the Medical Information Mart for Intensive Care (MIMIC-III).

Metaphor is not only a linguistic phenomenon but also reflects the concept projection between source and target domains in human cognition. Previous sequence tagging-based metaphor identification methods could not model the concept projection, resulting in a limitation that the outputs of these models are unexplainable in the predictions of the metaphoricity labels. In this work, we propose the first explainable metaphor identification model, inspired by Conceptual Metaphor Theory. The model is based on statistic learning, a lexical resource, and a novel reward mechanism. Our model can identify the metaphoricity on the word-pair level, and explain the predicted metaphoricity labels via learned concept mappings. The use of the reward mechanism allows the model to learn the optimal concept mappings without knowing their true labels. Our method is also applicable for the concepts that are out of training domains by using the lexical resource. The automatically generated concept mappings demonstrate the implicit human thoughts in metaphoric expressions. Our experiments show the effectiveness of the proposed model in metaphor identification, and concept mapping tasks, respectively.

Human conversations are guided by short-term and long-term goals. We study how to plan short-term goal sequences as coherently as humans do and naturally direct them to an assigned long-term goal in open-domain conversations. Goal sequences are a series of knowledge graph (KG) entity-relation connections generated by KG walkers that traverse through the KG. The existing recurrent and graph attention based KG walkers either insufficiently utilize the conversation states or lack global guidance. In our work, a hierarchical model learns goal planning in a hierarchical learning framework. We present HiTKG, a hierarchical transformer-based graph walker that leverages multiscale inputs to make precise and flexible predictions on KG paths. Furthermore, we propose a two-hierarchy learning framework that employs two stages to learn both turn-level (short-term) and global-level (long-term) conversation goals. Specifically, at the first stage, HiTKG is trained in a supervised fashion to learn how to plan turn-level goal sequences; at the second stage, HiTKG tries to naturally approach the assigned global goal via reinforcement learning. In addition, we propose MetaPath as the backbone method for KG path representation to exploit the entity and relation information concurrently. We further propose Multi-source Decoding Inputs and Output-level Length Head to improve the decoding controllability. Our experiments show that HiTKG achieves a significant improvement in the performance of turn-level goal learning compared with state-of-the-art baselines. Additionally, both automatic and human evaluation prove the effectiveness of the two-hierarchy learning framework for both short-term and long-term goal planning.

The goal of building intelligent dialogue systems has largely been separately pursued under two paradigms: task-oriented dialogue (TOD) systems, which perform task-specific functions, and open-domain dialogue (ODD) systems, which focus on non-goal-oriented chitchat. The two dialogue modes can potentially be intertwined together seamlessly in the same conversation, as easily done by a friendly human assistant. Such ability is desirable in conversational agents, as the integration makes them more accessible and useful. Our paper addresses this problem of fusing TODs and ODDs in multi-turn dialogues. Based on the popular TOD dataset MultiWOZ, we build a new dataset FusedChat, by rewriting the existing TOD turns and adding new ODD turns. This procedure constructs conversation sessions containing exchanges from both dialogue modes. It features inter-mode contextual dependency, i.e., the dialogue turns from the two modes depend on each other. Rich dependency patterns such as co-reference and ellipsis are included. The new dataset, with 60k new human-written ODD turns and 5k re-written TOD turns, offers a benchmark to test a dialogue model's ability to perform inter-mode conversations. This is a more challenging task since the model has to determine the appropriate dialogue mode and generate the response based on the inter-mode context. However, such models would better mimic human-level conversation capabilities. We evaluate two baseline models on this task, including the classification-based two-stage models and the two-in-one fused models. We publicly release FusedChat and the baselines to propel future work on inter-mode dialogue systems.

Data augmentation is an effective solution to data scarcity in low-resource scenarios.However, when applied to token-level tasks such as NER, data augmentation methods often suffer from token-label misalignment, which leads to unsatsifactory performance.In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation framework for low-resource NER.To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly conditioning on their labels.Thereby, MELM generates high-quality augmented data with novel entities, which provides rich entity regularity knowledge and boosts NER performance.When training data from multiple languages are available, we also integrate MELM with codemixing for further improvement.We demonstrate the effectiveness of MELM on monolingual, cross-lingual and multilingual NER across various low-resource levels.Experimental results show that our MELM presents substantial improvement over the baseline methods. 1

Automatic depression detection on Twitter can help individuals privately and conveniently understand their mental health status in the early stages before seeing mental health professionals. Most existing black-box-like deep learning methods for depression detection largely focused on improving classification performance. However, explaining model decisions is imperative in health research because decision-making can often be high-stakes and life-and-death. Reliable automatic diagnosis of mental health problems including depression should be supported by credible explanations justifying models’ predictions. In this work, we propose a novel explainable model for depression detection on Twitter. It comprises a novel encoder combining hierarchical attention mechanisms and feed-forward neural networks. To support psycholinguistic studies, our model leverages metaphorical concept mappings as input. Thus, it not only detects depressed individuals, but also identifies features of such users’ tweets and associated metaphor concept mappings.

Cross-lingual named entity recognition (NER) suffers from data scarcity in the target languages, especially under zero-shot settings.Existing translate-train or knowledge distillation methods attempt to bridge the language gap, but often introduce a high level of noise.To solve this problem, consistency training methods regularize the model to be robust towards perturbations on data or hidden states.However, such methods are likely to violate the consistency hypothesis, or mainly focus on coarse-grain consistency.We propose ConNER as a novel consistency training framework for cross-lingual NER, which comprises of: (1) translation-based consistency training on unlabeled target-language data, and (2) dropoutbased consistency training on labeled sourcelanguage data.ConNER effectively leverages unlabeled target-language data and alleviates overfitting on the source language to enhance the cross-lingual adaptability.Experimental results show our ConNER achieves consistent improvement over various baseline methods. 1

The impacts of climate change and global warming have become more visible globally because of the increasing frequency of extreme weather events, abnormal heatwaves, and other climate crises. Besides the traditional survey method, it is beneficial to automatically distillate climate change opinions from social platforms to measure public reactions quickly. We investigate how to organize climate change opinions on Twitter into meaningful categories to support perspective summarizing tasks. We find that merely using the available taxonomy for this task is ineffective; hence we must consider the entire text content. We recommend five high-level categories (Root cause, Impact, Mitigation, Politics or Policy, Others) and assemble ClimateTweets, a dataset with category and polarity labels. In addition, we construct category classification and polarity detection tasks with a range of opinion mining baselines. The experimental results show that both tasks are challenging for existing models. We release the ClimateTweets dataset to facilitate investigation in public opinion mining using text content and artificial intelligent methods. We hope this study could pave the way for future studies in the climate change domain.

Stress detection is a basic and essential task for examining the mental health of a given population. With the rapid digitalization leading to text-based forms of communication gaining dominance over spoken ones, there is now the chance to develop analytical studies for stress detection directly from textual inputs in social media. However, only a limited number of benchmarks are publicly available. To this end, we create four high quality datasets based on Twitter and Reddit, which are designed particularly for the task of stress detection from social media texts. The main contributions are three-folds: 1) for each dataset, we provide a detailed description on our dataset construction process, including data collection, data preprocessing and annotation; 2) we perform a comparative study on the performance of different rule-based and machine learning-based approaches on the proposed datasets as the new benchmark of this field; 3) we study the feasibility and reliability of existing systems. Extensive experiments show that Transformer-based models outperform lexical-based and embedding-based methods. Also, we observe that existing methods based on sentiment polarity detection cannot be directly adapted to the task of stress detection and there remains space for further improvements. We hope this study could pave the way for future studies in the field of computational stress detection.

In recent years, AI research has demonstrated enormous potential for the benefit of humanity and society. While often better than its human counterparts in classification and pattern recognition tasks, however, AI still struggles with complex tasks that require commonsense reasoning such as natural language understanding. In this context, the key limitations of current AI models are: dependency, reproducibility, trustworthiness, interpretability, and explainability. In this work, we propose a commonsense-based neurosymbolic framework that aims to overcome these issues in the context of sentiment analysis. In particular, we employ unsupervised and reproducible subsymbolic techniques such as auto-regressive language models and kernel methods to build trustworthy symbolic representations that convert natural language to a sort of protolanguage and, hence, extract polarity from text in a completely interpretable and explainable manner.

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domainspecific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.

The 3rd Multimodal Sentiment Analysis Challenge (MuSe) focuses on multimodal affective computing. The workshop is held in conjunction with ACM Multimedia'22. Three datasets are provided as part of the challenge: (i) the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset which contains humour-tagged audio-visual data of German football coaches, (ii) the Hume-Reaction dataset, which contains annotations on how people respond to emotional stimuli in terms of seven different emotional expression intensities, and (iii) the Ulm-Trier Social Stress Test (Ulm-TSST) dataset, which consists of audio-visual recordings labelled with continuous emotion values of individuals in stressful circumstances. Based on these datasets three affective computing challenges are defined: 1) Humor Detection Sub-Challenge (MuSe-Humor), for spontaneous humour recognition, 2) Emotional Reactions Sub-Challenge (MuSe-Reaction), for prediction of seven fine-grained in-the-wild' emotions, and 3) Emotional Stress Sub-Challenge (MuSe-Stress), for continuous prediction of stressed emotion values. In this summary, we describe the motivation behind the challenge, participation and its conditions, as well as the outcomes. The complete MuSe'22 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3551876

Extracting knowledge from heterogeneous data sources is fundamental for the construction of structured biomedical knowledge graphs (BKGs), where entities and relations are represented as nodes and edges in the graphs, respectively. Previous biomedical knowledge extraction methods simply considered limited entity types and relations by using a task-specific training set, which is insufficient for large-scale BKGs development and downstream task applications in different scenarios. To alleviate this issue, we propose a joint continual learning biomedical information extraction (JCBIE) network to extract entities and relations from different biomedical information datasets. By empirically studying different joint learning and continual learning strategies, the proposed JCBIE can learn and expand different types of entities and relations from different datasets. JCBIE uses two separated encoders in joint-feature extraction, hence can effectively avoid the feature confusion problem comparing with using one hard-parameter sharing encoder. Specifically, it allows us to adopt entity augmented inputs to establish the interaction between named entity recognition and relation extraction. Finally, a novel evaluation mechanism is proposed for measuring cross-corpus generalization errors, which was ignored by traditional evaluation methods. Our empirical studies show that JCBIE achieves promising performance when continual learning strategy is adopted with multiple corpora.

Multimodal emotion recognition in conversational videos (ERC) develops rapidly in recent years. To fully extract the relative context from video clips, most studies build their models on the entire dialogues which make them lack of real-time ERC ability. Different from related researches, a novel multimodal emotion recognition model for conversational videos based on reinforcement learning and domain knowledge (ERLDK) is proposed in this paper. In ERLDK, the reinforcement learning algorithm is introduced to conduct real-time ERC with the occurrence of conversations. The collection of history utterances is composed as an emotion-pair which represents the multimodal context of the following utterance to be recognized. Dueling deep-Q-network (DDQN) based on gated recurrent unit (GRU) layers is designed to learn the correct action from the alternative emotion categories. Domain knowledge is extracted from public dataset based on the former information of emotion-pairs. The extracted domain knowledge is used to revise the results from the RL module and is transformed into other dataset to examine the rationality. The experimental results on datasets show that ERLDK achieves the state-of-the-art results on weighted average and most of the specific emotion categories.

Human knowledge provides a formal understanding of the world. Knowledge graphs that represent structural relations between entities have become an increasingly popular research direction toward cognition and human-level intelligence. In this survey, we provide a comprehensive review of the knowledge graph covering overall research topics about: 1) knowledge graph representation learning; 2) knowledge acquisition and completion; 3) temporal knowledge graph; and 4) knowledge-aware applications and summarize recent breakthroughs and perspective directions to facilitate future research. We propose a full-view categorization and new taxonomies on these topics. Knowledge graph embedding is organized from four aspects of representation space, scoring function, encoding models, and auxiliary information. For knowledge acquisition, especially knowledge graph completion, embedding methods, path inference, and logical rule reasoning are reviewed. We further explore several emerging topics, including metarelational learning, commonsense reasoning, and temporal knowledge graphs. To facilitate future research on knowledge graphs, we also provide a curated collection of data sets and open-source libraries on different tasks. In the end, we have a thorough outlook on several promising research directions.

The 2nd Multimodal Sentiment Analysis (MuSe) 2021 Challenge-based Workshop is held in conjunction with ACM Multimedia'21. Two datasets are provided as part of the challenge. Firstly, the MuSe-CaR dataset, which focuses on user-generated, emotional vehicle reviews from YouTube, and secondly, the novel Ulm-Trier Social Stress (Ulm-TSST) dataset, which shows people in stressful circumstances. Participants are faced with four sub-challenges: predicting arousal and valence in a time- and value-continuous manner on a) MuSe-CaR (MuSe-Wilder) and b) Ulm-TSST (MuSe-Stress); c) predicting unsupervised created emotion classes on MuSe-CaR (MuSe-Sent); d) predicting a fusion of human-annotated arousal and measured galvanic skin response also as a continuous target on Ulm-TSST (MuSe-Physio). In this summary, we describe the motivation, the sub-challenges, the challenge conditions, the participation, and the most successful approaches.

Abstract Medical coding translates professionally written medical reports into standardized codes, which is an essential part of medical information systems and health insurance reimbursement. Manual coding by trained human coders is time-consuming and error-prone. Thus, automated coding algorithms have been developed, building especially on the recent advances in machine learning and deep neural networks. To solve the challenges of encoding lengthy and noisy clinical documents and capturing code associations, we propose a multitask recalibrated aggregation network. In particular, multitask learning shares information across different coding schemes and captures the dependencies between different medical codes. Feature recalibration and aggregation in shared modules enhance representation learning for lengthy notes. Experiments with a real-world MIMIC-III dataset show significantly improved predictive performance.

Suicide is a critical issue in modern society. Early detection and prevention of suicide attempts should be addressed to save people's life. Current suicidal ideation detection (SID) methods include clinical methods based on the interaction between social workers or experts and the targeted individuals and machine learning techniques with feature engineering or deep learning for automatic detection based on online social contents. This article is the first survey that comprehensively introduces and discusses the methods from these categories. Domain-specific applications of SID are reviewed according to their data sources, i.e., questionnaires, electronic health records, suicide notes, and online user content. Several specific tasks and data sets are introduced and summarized to facilitate further research. Finally, we summarize the limitations of current work and provide an outlook of further research directions.

When IoT meets Industry 4.0, we talk about Industrial Internet of Things (IIoT) or the use of typical Internet of Things (IoT) solutions in the industrial sector, integrating different technologies such as machine learning, big data and analytics, sensor data, Machine to Machine communication (M2M) and automation. Its application within the fourth industrial revolution has initiated a profound change within companies making them increasingly connected and exploiting data to optimize their production processes. IIoT is revolutionizing sectors such as manufacturing, automotive and healthcare. Collecting and analyzing all the data coming from the production sensors and drawing the information for your business through business analytics tools is the new key to competitiveness.

There is a large variety of products sold online and the websites are in several languages. Hence, it is desirable to train a model that can predict sentiments in different domains simultaneously. Previous authors have used deep learning to extract features from multiple domains. Here, each word is represented by a vector that is determined using co-occurrence data. Such a model requires that all sentences have the same length resulting in low accuracy. To overcome this challenge, we model the features in each sentence using a variable length tree called a Genetic Program. The polarity of clauses can be represented using mathematical operators such as '+' or '-' at internal nodes in the tree. The proposed model is evaluated on Amazon product reviews for different products and in different languages. We are able to outperform the accuracy of baseline multi-domain models in the range of 5-20%.

Deep learning has unlocked new paths towards the emulation of the peculiarly-human capability of learning from examples. While this kind of bottom-up learning works well for tasks such as image classification or object detection, it is not as effective when it comes to natural language processing. Communication is much more than learning a sequence of letters and words: it requires a basic understanding of the world and social norms, cultural awareness, commonsense knowledge, etc.; all things that we mostly learn in a top-down manner. In this work, we integrate top-down and bottom-up learning via an ensemble of symbolic and subsymbolic AI tools, which we apply to the interesting problem of polarity detection from text. In particular, we integrate logical reasoning within deep learning architectures to build a new version of SenticNet, a commonsense knowledge base for sentiment analysis.

The recent dominance of machine learning-based natural language processing methods has fostered the culture of overemphasizing model accuracies rather than studying the reasons behind their errors.Interpretability, however, is a critical requirement for many downstream AI and NLP applications, e.g., in finance, healthcare, and autonomous driving.This study, instead of proposing any "new model", investigates the error patterns of some widely acknowledged sentiment analysis methods in the finance domain.We discover that (1) those methods belonging to the same clusters are prone to similar error patterns, and (2) there are six types of linguistic features that are pervasive in the common errors.These findings provide important clues and practical considerations for improving sentiment analysis models for financial applications.

State-of-the-art personality prediction with text data mostly relies on bottom up, automated feature generation as part of the deep learning process. More traditional models rely on hand-crafted, theory-based text-feature categories. We propose a novel deep learning-based model which integrates traditional psycholinguistic features with language model embeddings to predict personality from the Essays dataset for Big-Five and Kaggle dataset for MBTI. With this approach we achieve state-of-the-art model performance. Additionally, we use interpretable machine learning to visualize and quantify the impact of various language features in the respective personality prediction models. We conclude with a discussion on the potential this work has for computational modeling and psychological science alike.

This paper analyzes the increasing importance of mobile technologies in the tourism industry by reviewing, examining and synthesizing studies related to it. The main aim of this paper is to provide academics and practitioners with the current developments and practices of the alleged mobile tourism. In particular, we investigate the key enablers of mobile tourism, namely: social media marketing, sentiment analysis, recommender systems, Internet of Things, and virtual assistants. Finally, we focus on investigating how mobile technologies are significantly altering the tourism industry as well as providing an agenda for future research.

This paper explores nuclear energy-related Twitter discussions as a response to the 2011 Fukushima Nuclear Disaster and the 2017 Nobel Peace Prize won by the International Campaign to Abolish Nuclear Weapons. We have considered a total of 2 million tweets for these two events. In particular, we employed CNN, LSTM, and Bi-LSTM to investigate whether social media users are supportive or cynical about nuclear energy. Our AI algorithms have performed better for polarity detection (accuracy in the range of 90%) with respect to subjectivity detection (accuracy in the range of 75%). We also note that dominant aspects of supporting tweets revolve around concepts like clean energy, lower CO2 emission, and sustainable future. On the contrary, cynical users see nuclear energy as a threat to the environment, human life, and safety.

The first Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 was a Challenge-based Workshop held in conjunction with ACM Multimedia'20. It addresses three distinct 'in-the-wild` Sub-challenges: sentiment/ emotion recognition (MuSe-Wild), emotion-target engagement (MuSe-Target) and trustworthiness detection (MuSe-Trust). A large multimedia dataset MuSe-CaR was used, which was specifically designed with the intention of improving machine understanding approaches of how sentiment (e.g. emotion) is linked to a topic in emotional, user-generated reviews. In this summary, we describe the motivation, first of its kind 'in-the-wild` database, challenge conditions, participation, as well as giving an overview of utilised state-of-the-art techniques.

In the personal house renting scenario, customers usually make quick assessments based on previous customers' reviews, which makes such reviews essential for the business. If the house is assessed as popular, a Matthew effect will be observed as more people will be willing to book it. Due to the lack of definition and quantity assessment measures, however, it is difficult to make a popularity evaluation and prediction. To solve this problem, the concept of house popularity is well defined in this paper. Specifically, the house popularity is decided by inter-event timeand rating score at the same time. To make a more effective prediction over these two correlated variables, a dual-gated recurrent unit (DGRU) is employed. Furthermore, an encoder-decoder framework with DGRU is proposed to perform popularity prediction. Empirical results show the effectiveness of the proposed DGRU and the encoder-decoder framework in two-correlated sequences prediction and popularity prediction, respectively.

Social media represent a rich source of information, such as critiques, feedback, and other opinions posted online by Internet users. Such information is typically a good reflection of users’ sentiments and attitudes towards various services, topics, or products. Sentiment analysis has become an increasingly important natural language processing (NLP) task to help users make sense of what is happening in the Internet blogosphere and it can be useful for companies as well as public organizations. However, most existing sentiment analysis techniques are only able to analyze data at the aggregate level, merely providing a binary classification (positive vs. negative), and are not able to generate finer characterizations of sentiments as well as emotions involved. This paper describes a new opinion analysis scheme, i.e., a multi-level fine-scaled sentiment sensing with ambivalence handling. The ambivalence handler is presented in detail along with the strength-level tune parameters for analyzing the strength and the fine-scale of both positive or negative sentiments. It is capable of drilling deeper into text in order to reveal multi-level fine-scaled sentiments as well as different types of emotions.

Emotion detection in conversations is a necessary step for a number of applications, including opinion mining over chat history, social media threads, debates, argumentation mining, understanding consumer feedback in live conversations, and so on. Currently systems do not treat the parties in the conversation individually by adapting to the speaker of each utterance. In this paper, we describe a new method based on recurrent neural networks that keeps track of the individual party states throughout the conversation and uses this information for emotion classification. Our model outperforms the state-of-the-art by a significant margin on two different datasets.

Emotion recognition in conversations (ERC) is a challenging task that has recently gained popularity due to its potential applications.Until now, however, there has been no largescale multimodal multi-party emotional conversational database containing more than two speakers per dialogue.To address this gap, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhancement of EmotionLines.MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends.Each utterance is annotated with emotion and sentiment labels, and encompasses audio, visual, and textual modalities.We propose several strong multimodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations.The full dataset is available for use at http:// affective-meld.github.io.

Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes.In this paper, we introduce (i) an agreement score to evaluate the performance of routing processes at instance level; (ii) an adaptive optimizer to enhance the reliability of routing; (iii) capsule compression and partial routing to improve the scalability of capsule networks.We validate our approach on two NLP tasks, namely: multi-label text classification and question answering.Experimental results show that our approach considerably improves over strong competitors on both tasks.In addition, we gain the best results in low-resource settings with few training instances.1

Non-iterative methods for parameter estimation for interval type-2 neuro-fuzzy structure are fast to implement, when compared to online methods, and need no -or a few- parameters to be tuned. In this paper, a novel dynamic cost function, which defines a relationship between the current and past errors, is defined. The minimization of the aforementioned cost function results in a decreasing sequence of error which makes the proposed method numerically more stable when compared to least squares-based methods. It is a well-known phenomenon that a matrix inversion may cause problems if the matrix to be inverted is ill-defined i.e. its condition number is far bigger than one. The use of a dynamic relationship between the current and past error adds more degrees of freedom which makes it possible to improve the condition number of the matrix. Comprehensive simulation studies are presented for the prediction of financial data sets. The simulation results shows the superior numerical stability of the proposed method as the mean value of the condition number is smaller. This finding results in more accurate matrix inversion to be done in the two-step matrix inversion.

In the physical world, complex systems are generally created as the composition of multiple primitive components that interact with each other rather than a single monolithic structure. Recently, spatio-temporal graphs received a reasonable amount of attention from the research community since they emerged as a natural representational tool able to capture the interactive and interrelated structure of a complex problem. To better understand the nature of complex systems, there is the need to define models that can easily explain the learned causal relationship. To this end, we propose an attentive model able to learn and project the relational structure into a fixed-size embedding. Such representation naturally captures the dynamic influence that each neighbors exert over a given vertex providing a valuable description of the problem setting. The proposed architecture has been extensively evaluated against strong baselines on toy as well as real-world tasks, such as prediction of household energy load and traffic congestion.

Forecasting stock market behavior is an interesting and challenging problem. Regression of prices and classification of daily returns have been widely studied with the main goal of supplying forecasts useful in real trading scenarios. Unfortunately, the outcomes are not directly related with the maximization of the financial gain. Firstly, the optimal strategy requires to invest on the most performing asset every period and trading accordingly is not trivial given the predictions. Secondly, price fluctuations of different magnitude are often treated as equals even if during market trading losses or gains of different intensities are derived. In this paper, the problem of stock market forecasting is formulated as regression of market returns. This approach is able to estimate the amount of price change and thus the most performing assets. Price fluctuations of different magnitude are treated differently through the application of different weights on samples and the scarcity of data is addressed using transfer learning. Results on a real simulation of trading show how, given a finite amount of capital, the predictions can be used to invest in high performing stocks and, hence, achieve higher profits with less trades.

Microtext analysis is a crucial task for gauging social media opinion. In this paper, we compare four different deep learning encoder-decoder frameworks to handle microtext normalization problem. The frameworks have been evaluated on four different datasets in three different domains. To understand the impact of microtext normalization, we further integrate the framework into a sentiment classification task. This paper is the first of its kind to incorporate deep learning into a microtext normalization module and improve the sentiment analysis task. We show our models as a sequence to sequence character to word encoder-decoder model. We compare four deep learning models for microtext normalization task which further improve the accuracy of the sentiment analysis. Results show that the attentive LSTM and GRU cell both increase the sentiment analysis accuracy in the range of 4%–7% whereas LSTM and CNN with LSTM improve the accuracy in the range of 2%–4%.

Hiring robots for the workplaces is a challenging task as robots have to cater to customer demands, follow organizational protocols and behave with social etiquette. In this study, we propose to have a humanoid social robot, Nadine, as a customer service agent in an open social work environment. The objective of this study is to analyze the effects of humanoid robots on customers in a work environment, and see if it can handle social scenarios. We propose to evaluate these objectives through two modes, namely: survey questionnaire and customer feedback. The survey questionnaires are analyzed based on the datapoints provided in the questionnaire. We propose a novel approach to analyze customer feedback data using sentic computing. Specifically, we employ aspect extraction and sentiment analysis to analyze the data. From our framework, we detect sentiment associated to the aspects that mainly concerned the customers during their interaction. This allows us to understand customers expectations and current limitations of robots as employees.

With the recent development of deep learning, research in AI has gained new vigor and prominence. While machine learning has succeeded in revitalizing many research fields, such as computer vision, speech recognition, and medical diagnosis, we are yet to witness impressive progress in natural language understanding. One of the reasons behind this unmatched expectation is that, while a bottom-up approach is feasible for pattern recognition, reasoning and understanding often require a top-down approach. In this work, we couple sub-symbolic and symbolic AI to automatically discover conceptual primitives from text and link them to commonsense concepts and named entities in a new three-level knowledge representation for sentiment analysis. In particular, we employ recurrent neural networks to infer primitives by lexical substitution and use them for grounding common and commonsense knowledge by means of multi-dimensional scaling.

Analyzing people’s opinions and sentiments towards certain aspects is an important task of natural language understanding. In this paper, we propose a novel solution to targeted aspect-based sentiment analysis, which tackles the challenges of both aspect-based sentiment analysis and targeted sentiment analysis by exploiting commonsense knowledge. We augment the long short-term memory (LSTM) network with a hierarchical attention mechanism consisting of a target-level attention and a sentence-level attention. Commonsense knowledge of sentiment-related concepts is incorporated into the end-to-end training of a deep neural network for sentiment classification. In order to tightly integrate the commonsense knowledge into the recurrent encoder, we propose an extension of LSTM, termed Sentic LSTM. We conduct experiments on two publicly released datasets, which show that the combination of the proposed attention architecture and Sentic LSTM can outperform state-of-the-art methods in targeted aspect sentiment tasks.

Building dialogue systems that can converse naturally with humans is a challenging yet intriguing problem of artificial intelligence. In open-domain human-computer conversation, where the conversational agent is expected to respond to human utterances in an interesting and engaging way, commonsense knowledge has to be integrated into the model effectively. In this paper, we investigate the impact of providing commonsense knowledge about the concepts covered in the dialogue. Our model represents the first attempt to integrating a large commonsense knowledge base into end-to-end conversational models. In the retrieval-based scenario, we propose a model to jointly take into account message content and related commonsense for selecting an appropriate response. Our experiments suggest that the knowledge-augmented models are superior to their knowledge-free counterparts.

Multi-view sequential learning is a fundamental problem in machine learning dealing with multi-view sequences. In a multi-view sequence, there exists two forms of interactions between different views: view-specific interactions and cross-view interactions. In this paper, we present a new neural architecture for multi-view sequential learning called the Memory Fusion Network (MFN) that explicitly accounts for both interactions in a neural architecture and continuously models them through time. The first component of the MFN is called the System of LSTMs, where view-specific interactions are learned in isolation through assigning an LSTM function to each view. The cross-view interactions are then identified using a special attention mechanism called the Delta-memory Attention Network (DMAN) and summarized through time with a Multi-view Gated Memory. Through extensive experimentation, MFN is compared to various proposed approaches for multi-view sequential learning on multiple publicly available benchmark datasets. MFN outperforms all the multi-view approaches. Furthermore, MFN outperforms all current state-of-the-art models, setting new state-of-the-art results for all three multi-view datasets.

Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape the communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art results performance in all the datasets.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.

Gender-based-violence is a serious concern in recent times. Due to the social stigma attached to these assaults, victims rarely come forward. Implementing policy measures to prevent sexual violence get constrained due to lack of crime statistics. However, the recent outcry on the Twitter platform allows us to address this concern. Sexual assaults occur at workplaces, public places, educational institutes and also at home. Policy level approaches and awareness campaign for these assaults would not be similar. So, we want to identify the risk factor associated with these sexual assaults. We extracted 0.7 million tweets during the #MeToo social media movement. Next, we employ deep learning techniques to classify these sexual violences. We observe that sexual assaults by a family member at own home is a more serious concern than harassment by a stranger at public places. This study reveals assaults by a known person are more prevalent than assaults by unknown strangers.

The literature in automated sarcasm detection has mainly focused on lexical-, syntactic- and semantic-level analysis of text. However, a sarcastic sentence can be expressed with contextual presumptions, background and commonsense knowledge. In this paper, we propose a ContextuAl SarCasm DEtector (CASCADE), which adopts a hybrid approach of both content- and context-driven modeling for sarcasm detection in online social media discussions. For the latter, CASCADE aims at extracting contextual information from the discourse of a discussion thread. Also, since the sarcastic nature and form of expression can vary from person to person, CASCADE utilizes user embeddings that encode stylometric and personality features of users. When used along with content-based feature extractors such as convolutional neural networks, we see a significant boost in the classification performance on a large Reddit corpus.

Emotion recognition in conversations is crucial for building empathetic machines.Current work in this domain do not explicitly consider the inter-personal influences that thrive in the emotional dynamics of dialogues.To this end, we propose Interactive COnversational memory Network (ICON), a multimodal emotion detection framework that extracts multimodal features from conversational videos and hierarchically models the selfand interspeaker emotional influences into global memories.Such memories generate contextual summaries which aid in predicting the emotional orientation of utterance-videos.Our model outperforms state-of-the-art networks on multiple classification and regression tasks in two benchmark datasets.

Sentiment analysis has immense implications in modern businesses through user-feedback mining.Large product-based enterprises like Samsung and Apple make crucial business decisions based on the large quantity of user reviews and suggestions available in different e-commerce websites and social media platforms like Amazon and Facebook.Sentiment analysis caters to these needs by summarizing user sentiment behind a particular object.In this paper, we present a novel approach of incorporating the neighboring aspects related information into the sentiment classification of the target aspect using memory networks.Our method outperforms the state of the art by 1.6% on average in two distinct domains.

Emotion recognition in conversations is crucial for the development of empathetic machines. Present methods mostly ignore the role of inter-speaker dependency relations while classifying emotions in conversations. In this paper, we address recognizing utterance-level emotions in dyadic conversational videos. We propose a deep neural framework, termed conversational memory network, which leverages contextual information from the conversation history. The framework takes a multimodal approach comprising audio, visual and textual features with gated recurrent units to model past utterances of each speaker into memories. Such memories are then merged using attention-based hops to capture inter-speaker dependencies. Experiments show an accuracy improvement of 3-4% over the state of the art.

Information extraction in the medical domain is laborious and time-consuming due to the insufficient number of domainspecific lexicons and lack of involvement of domain experts such as doctors and medical practitioners.Thus, in the present work, we are motivated to design a new lexicon, WME 3.0 (WordNet of Medical Events), which contains over 10,000 medical concepts along with their part of speech, gloss (descriptive explanations), polarity score, sentiment, similar sentiment words, category, affinity score and gravity score features.In addition, the manual annotators help to validate the overall as well as individual category level of medical concepts of WME 3.0 using Cohen's Kappa agreement metric.The agreement score indicates almost correct identification of medical concepts and their assigned features in WME 3.0.

We find from four datasets that time expressions are formed by loose structure and the words used to express time information can differentiate time expressions from common text. The findings drive us to design a learning method named TOMN to model time expressions. TOMN defines a time-related tagging scheme named TOMN scheme with four tags, namely \tomnT,\tomnO, \tomnM,and \tomnN, indicating the constituents of time expression, namely \tomnT ime token, \tomnM odifier, \tomnN umeral, and the words \tomnO utside time expression. In modeling, TOMN assigns a word with a TOMN tag under conditional random fields with minimal features. Essentially, our constituent-based TOMN scheme overcomes the problem of inconsistent tag assignment that is caused by the conventional position-based tagging schemes (\eg BIO scheme and BILOU scheme). Experiments show that TOMN is equally or more effective than state-of-the-art methods on various datasets, and much more robust on cross-datasets. Moreover, our analysis can explain many empirical observations in other works about time expression recognition and named entity recognition.

Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, Louis-Philippe Morency. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

Extracting time expressions from free text is a fundamental task for many applications.We analyze time expressions from four different datasets and find that only a small group of words are used to express time information and that the words in time expressions demonstrate similar syntactic behaviour.Based on the findings, we propose a type-based approach named SynTime 1 for time expression recognition.Specifically, we define three main syntactic token types, namely time token, modifier, and numeral, to group time-related token regular expressions.On the types we design general heuristic rules to recognize time expressions.In recognition, SynTime first identifies time tokens from raw text, then searches their surroundings for modifiers and numerals to form time segments, and finally merges the time segments to time expressions.As a lightweight rule-based tagger, SynTime runs in real time, and can be easily expanded by simply adding keywords for the text from different domains and different text types.Experiments on benchmark datasets and tweets data show that SynTime outperforms state-of-the-art methods.

In this paper, we study an important yet largely under-explored setting of graph embedding, i.e., embedding communities instead of each individual nodes. We find that community embedding is not only useful for community-level applications such as graph visualization, but also beneficial to both community detection and node classification. To learn such embedding, our insight hinges upon a closed loop among community embedding, community detection and node embedding. On the one hand, node embedding can help improve community detection, which outputs good communities for fitting better community embedding. On the other hand, community embedding can be used to optimize the node embedding by introducing a community-aware high-order proximity. Guided by this insight, we propose a novel community embedding framework that jointly solves the three tasks together. We evaluate such a framework on multiple real-world datasets, and show that it improves graph visualization and outperforms state-of-the-art baselines in various application tasks, e.g., community detection and node classification.

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language.In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics.We introduce a novel model, termed Tensor Fusion Network, which learns both such dynamics end-to-end.The proposed approach is tailored for the volatile nature of spoken language in online videos as well as accompanying gestures and voice.In the experiments, our model outperforms state-ofthe-art approaches for both multimodal and unimodal sentiment analysis.

Multimodal sentiment analysis involves identifying sentiment in videos and is a developing field of research. Unlike current works, which model utterances individually, we propose a recurrent model that is able to capture contextual information among utterances. In this paper, we also introduce attentionbased networks for improving both context learning and dynamic feature fusion. Our model shows 6-8% improvement over the state of the art on a benchmark dataset.

Text categorization, or text classification, is one of key tasks for representing the semantic information of documents. Multi-label text categorization is finer-grained approach to text categorization which consists of assigning multiple target labels to documents. It is more challenging compared to the task of multi-class text categorization due to the exponential growth of label combinations. Existing approaches to multi-label text categorization fall short to extract local semantic information and to model label correlations. In this paper, we propose an ensemble application of convolutional and recurrent neural networks to capture both the global and the local textual semantics and to model high-order label correlations while having a tractable computational complexity. Extensive experiments show that our approach achieves the state-of-the-art performance when the CNN-RNN model is trained using a large-sized dataset.

This work tackles the seldom discussed task of predicting chaotic time series generated by dynamic systems with evolving parameters. Representative chaotic time series produced by different system dimensions are introduced with a critical parameter linearly depending on time. The evolving character of systems are qualitatively studied by phase portraits. We assess the predictability of different fuzzy neural network (FNN) architectures on several evolving chaotic time series. Experiments illustrate that FNN models can generally better approximate evolving chaotic systems comparing to the autoregression method as a benchmark. The main contribution of our work is that we found out certain FNN types, e.g., NEFCON and DENFIS, are more robust to changing system parameters. In spite of the performance, some FNN models are more vulnerable and incline to be destabilized by high order chaotic systems. This work also casts light on composing FNN structures to capture evolving characters of chaotic time series in the future.

Sentiment analysis is able to automatically extract valuable customer information from large amount of unstructured text data to support decision making in manufacturing applications such as product design and demand planning. One of the key issues of sentiment analysis is the high dimensionality of data, which can be effectively solved by feature selection. Existing feature selection techniques compute feature scores solely based on training data statistics or by modifying a specific feature metric formula to include test data information which can not be generalized to other types of feature metrics. In this paper, we propose an adaptive two-stage feature selection approach, which generates base feature scores from a training dataset and then weights them based on individual test sample so that the feature importance evaluation is adapted to the characteristic of test data as well. The proposed method is applicable to arbitrary type of feature metrics and sentiment classifiers. The experiments show that our approach can consistently outperform other methods, especially for the setting of small number of selected features.

This special section focuses on the introduction, presentation, and discussion of novel techniques that further develop and apply affective reasoning tools and techniques for big social data analysis. A key motivation for this special section, in particular, is to explore the adoption of novel affective reasoning frameworks and cognitive learning systems to go beyond a mere word-level analysis of natural language text and provide novel concept-level tools and techniques that allow a more efficient passage from (unstructured) natural language to (structured) machine-processable affective data, in potentially any domain. The selected papers aim to address the wide spectrum of issues related to affective computing research and, hence, better grasp the current limitations and opportunities related to this fast-evolving branch of artificial intelligence. Out of the 29 submissions received, 5 were accepted to appear in the special section. One of the accepted papers underwent 3 rounds of revisions, the rest were revised twice. The papers appearing in this issue are briefly summarized.

An important difference between traditional AI systems and human intelligence is the human ability to harness commonsense knowledge gleaned from a lifetime of learning and experience to make informed decisions. This allows humans to adapt easily to novel situations where AI fails catastrophically due to a lack of situation-specific rules and generalization capabilities. Commonsense knowledge also provides background information that enables humans to successfully operate in social situations where such knowledge is typically assumed. Since commonsense consists of information that humans take for granted, gathering it is an extremely difficult task. Previous versions of SenticNet were focused on collecting this kind of knowledge for sentiment analysis but they were heavily limited by their inability to generalize. SenticNet 4 overcomes such limitations by leveraging on conceptual primitives automatically generated by means of hierarchical clustering and dimensionality reduction.

Named entity typing is the task of detecting the types of a named entity in context. For instance, given “Eric is giving a presentation”, our goal is to infer that ‘Eric’ is a speaker or a presenter and a person. Existing approaches to named entity typing cannot work with a growing type set and fails to recognize entity mentions of unseen types. In this paper, we present a label embedding method that incorporates prototypical and hierarchical information to learn pre-trained label embeddings. In addition, we adapt a zero-shot learning framework that can predict both seen and previously unseen entity types. We perform evaluation on three benchmark datasets with two settings: 1) few-shots recognition where all types are covered by the training set; and 2) zero-shot recognition where fine-grained types are assumed absent from training set. Results show that prior knowledge encoded using our label embedding methods can significantly boost the performance of classification for both cases.

Sarcasm detection is a key task for many natural language processing tasks. In sentiment analysis, for example, sarcasm can flip the polarity of an “apparently positive” sentence and, hence, negatively affect polarity detection performance. To date, most approaches to sarcasm detection have treated the task primarily as a text categorization problem. Sarcasm, however, can be expressed in very subtle ways and requires a deeper understanding of natural language that standard text categorization techniques cannot grasp. In this work, we develop models based on a pre-trained convolutional neural network for extracting sentiment, emotion and personality features for sarcasm detection. Such features, along with the network’s baseline features, allow the proposed models to outperform the state of the art on benchmark datasets. We also address the often ignored generalizability issue of classifying data that have not been seen by the models at learning phase.

Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

In this paper, we propose to use contexts of superpixels as a prior to improve semantic segmentation by the CRF framework. A graphical model is constructed on over-segmented images. Our main contribution is to take the concept of “superpixel embedding” into consideration, which is formalized as a potential item for optimizing the energy of the whole graph. We also introduce two ways of calculating this embedding potential. Experiments on several popular datasets, e.g., MRSC-21 and PASCAL VOC, illustrate that our approach enhances the performance of a previously proposed segmentation model without embedding. The accuracy results are comparable to some fully supervised methods.

Objective sentences lack sentiments and, hence, can reduce the accuracy of a sentiment classifier. Traditional methods prior to 2001 used hand-crafted templates to identify subjectivity and did not generalize well for resource-deficient languages such as Spanish. Later works published between 2002 and 2009 proposed the use of deep neural networks to automatically learn a dictionary of features (in the form of convolution kernels) that is portable to new languages. Recently, recurrent neural networks are being used to model alternating subjective and objective sentences within a single review. Such networks are difficult to train for a large vocabulary of words due to the problem of vanishing gradients. Hence, in this paper we consider use of a Lyapunov linear matrix inequality to classify Spanish text as subjective or objective by combining Spanish features and features obtained from the corresponding translated English text. The aligned features for each sentence are next evolved using multiple kernel learning. The proposed Lyapunov deep neural network outperforms baselines by over 10% and the features learned in the hidden layers improve our understanding subjective sentences in Spanish.

The advent of the Social Web has provided netizens with new tools for creating and sharing, in a time- and cost-efficient way, their contents, ideas, and opinions with virtually the millions of people connected to the World Wide Web. This huge amount of information, however, is mainly unstructured as specifically produced for human consumption and, hence, it is not directly machine-processable. In order to enable a more efficient passage from unstructured information to structured data, aspect-based opinion mining models the relations between opinion targets contained in a document and the polarity values associated with these. Because aspects are often implicit, however, spotting them and calculating their respective polarity is an extremely difficult task, which is closer to natural language understanding rather than natural language processing. To this end, Sentic LDA exploits common-sense reasoning to shift LDA clustering from a syntactic to a semantic level. Rather than looking at word co-occurrence frequencies, Sentic LDA leverages on the semantics associated with words and multi-word expressions to improve clustering and, hence, outperform state-of-the-art techniques for aspect extraction.

In order to overcome the lack of medical corpora, we have developed a WordNet for Medical Events (WME) for identifying medical terms and their sense related information using a seed list.The initial WME resource contains 1654 medical terms or concepts.In the present research, we have reported the enhancement of WME with 6415 number of medical concepts along with their conceptual features viz.Parts-of-Speech (POS), gloss, semantics, polarity, sense and affinity.Several polarity lexicons viz.SentiWordNet, SenticNet, Bing Liu's subjectivity list and Taboda's adjective list were introduced with WordNet synonyms and hyponyms for expansion.The semantics feature guided us to build a semantic co-reference relation based network between the related medical concepts.These features help to prepare a medical concept network for better sense relation based visualization.Finally, we evaluated with respect to Adaptive Lesk Algorithm and conducted an agreement analysis for validating the expanded WME resource.

Predicting the affective valence of unknown multi-word expressions is key for concept-level sentiment analysis. AffectiveSpace 2 is a vector space model, built by means of random projection, that allows for reasoning by analogy on natural language con- cepts. By reducing the dimensionality of affec- tive common-sense knowledge, the model allows semantic features associated with concepts to be generalized and, hence, allows concepts to be intu- itively clustered according to their semantic and affective relatedness. Such an affective intuition (so called because it does not rely on explicit fea- tures, but rather on implicit analogies) enables the inference of emotions and polarity conveyed by multi-word expressions, thus achieving efficient concept-level sentiment analysis.

We present a novel way of extracting features from short texts, based on the activation values of an inner layer of a deep convolutional neural network.We use the extracted features in multimodal sentiment analysis of short video clips representing one sentence each.We use the combined feature vectors of textual, visual, and audio modalities to train a classifier based on multiple kernel learning, which is known to be good at heterogeneous data.We obtain 14% performance improvement over the state of the art and present a parallelizable decision-level data fusion method, which is much faster, though slightly less accurate.

This paper explores the theory of learning with similarity functions in the context of common-sense reasoning and natural language processing. Based on this theory, the proposed approach (called Sim-Predictor) is characterized by the process of remapping the input space into a new space which is able to convey the similarity between the input pattern and a number of landmarks, i.e., a subset of patterns randomly extracted from the training set. The new learning scheme exhibits the interesting property of relating the dimensionality of the remapped space to the learning abilities of the eventual predictor in a formal fashion. The evaluation phase shows that Sim-Predictor compares positively with ELM and SVM, when addressing the problem of polarity detection in the sentic computing framework, a novel approach to big social data analysis based on the interpretation of the cognitive and affective information associated with natural language (affective common-sense reasoning).

Cross-lingual document clustering is the task of automatically organizing a large collection of multi-lingual documents into a few clusters, depending on their content or topic. It is well known that language barrier and translation ambiguity are two challenging issues for cross-lingual document representation. To this end, we propose to represent cross-lingual documents through statistical word senses, which are automatically discovered from a parallel corpus through a novel cross-lingual word sense induction model and a sense clustering method. In particular, the former consists in a sense-based vector space model and the latter leverages on a sense-based latent Dirichlet allocation. Evaluation on the benchmarking datasets shows that the proposed models outperform two state-of-the-art methods for cross-lingual document clustering.

SenticNet is a publicly available semantic and affective resource for concept-level sentiment analysis. Rather than using graph-mining and dimensionality-reduction techniques, SenticNet 3 makes use of "energy flows" to connect various parts of extended common and common-sense knowledge representations to one another. SenticNet 3 models nuanced semantics and sentics (that is, the conceptual and affective information associated with multi-word natural language expressions), representing information with a symbolic opacity of an intermediate nature between that of neural networks and typical symbolic systems.

Since proposed, LDA have been successfully used in modeling text documents. So far, words are the common features to induce latent topic, which are later used in document representation. Observation on documents indicates that the polysemous words can make the latent topics less discriminative, resulting in less accurate document representation. We thus argue that the semantically deterministic word senses can improve quality of the latent topics. In this work, we proposes a series of word sense aware LDA models which use word sense as an extra latent variable in topic induction. Preliminary experiments on benchmark datasets show that word sense can indeed improve topic modeling.

The automatic detection of orientation and emotions in texts is becoming increasingly important in the Web 2.0 scenario. There is a considerable need for innovative techniques and tools capable of identifying and detecting the attitude of unstructured text. The paper tackles two crucial aspects of the sentiment classification problem: first, the computational complexity of the deployed framework; second, the ability of the framework itself to operate effectively in heterogeneous commercial domains. The proposed approach adopts empirical learning to implement the sentiment-classification technology, and uses a distance-based predictive model to combine computational efficiency and modularity. A suitably designed semantic-based metric is the cognitive core that measures the distance between two user reviews, according to the sentiment they communicate. The framework ultimately nullifies the training process; at the same time, it takes advantage of a classification procedure whose computational cost increases linearly when the training corpus increases. To attain an objective measurement of the actual accuracy of the sentiment classification method, a campaign of tests involved a pair of complex, real-world scoring domains; the goal was to compare the predicted sentiment scores with actual scores provided by human assessors. Experimental results confirmed that the overall approach attained satisfactory performances in terms of both cross-domain classification accuracy and computational efficiency.

An important difference between traditional AI systems and human intelligence is our ability to harness common sense knowledge gleaned from a lifetime of learning and experiences to inform our decision making and behavior. This allows humans to adapt easily to novel situations where AI fails catastrophically for lack of situation-specific rules and generalization capabilities. Common sense knowledge also provides the background knowledge for humans to successfully operate in social situations where such knowledge is typically assumed. In order for machines to exploit common sense knowledge in reasoning as humans do, moreover, we need to endow them with human-like reasoning strategies. In this work, we propose a two-level affective reasoning framework that concurrently employs multi-dimensionality reduction and graph mining techniques to mimic the integration of conscious and unconscious reasoning, and exploit it for sentiment analysis.