In this study, we investigate multimodal foundation models (MFMs) for emotion recognition from non-verbal sounds. We hypothesize that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs). To validate our hypothesis, we extract representations from state-of-the-art (SOTA) MFMs and AFMs and evaluated them on benchmark NVER datasets. We also investigate the potential of combining selected foundation model (FM) representations to enhance NVER further inspired by research in speech recognition and audio deepfake detection. To achieve this, we propose a framework called MATA (Intra-Modality Alignment through Transport Attention). Through MATA coupled with the combination of MFMs: LanguageBind and ImageBind, we report the topmost performance with accuracies of 76.47%, 77.40%, 75.12% and F1-scores of 70.35%, 76.19%, 74.63% for ASVP-ESD, JNV, and VIVAE datasets against individual FMs and baseline fusion techniques and report SOTA on the benchmark datasets.

Fact-checking is a crucial task as it ensures the prevention of misinformation. However, manual fact-checking cannot keep up with the rate at which false information is generated and disseminated online. Automated fact-checking by machines is significantly quicker than by humans. But for better trust and transparency of these automated systems, explainability in the fact-checking process is necessary. Fact-checking often entails contrasting a factual assertion with a body of knowledge for such explanations. An effective way of representing knowledge is the Knowledge Graph (KG). There have been sufficient works proposed related to fact-checking with the usage of KG but not much focus is given to the application of reinforcement learning (RL) in such cases. To mitigate this gap, we propose an RL-based KG reasoning approach for explainable fact-checking. Extensive experiments on FB15K-277 and NELL-995 datasets reveal that reasoning over a KG is an effective way of producing human-readable explanations in the form of paths and classifications for fact claims. The RL reasoning agent computes a path that either proves or disproves a factual claim, but does not provide a verdict itself. A verdict is reached by a voting mechanism that utilizes paths produced by the agent. These paths can be presented to human readers so that they themselves can decide whether or not the provided evidence is convincing or not. This work will encourage works in this direction for incorporating RL for explainable fact-checking as it increases trustworthiness by providing a human-in-the-loop approach.

Abstract The factors that influence a person's mental health are numerous, interconnected, and multi‐dimensional. Recognition of stress is one of the facets in developing the Mental Healthcare (MHC) system framework. With the advent of technology, smart wearable devices have paved a way to collect data in real‐time to provide the cutting‐edge reports about the individual. Due to the physiological sensors present in the smart wearable devices, it is now possible to have a robust system to recognize the stress of the smart wearable devices user thus consecutively leading to recognition of factors in leading to stress. However, the current MHC system for recognition and identification of stress have several drawbacks. First, stress recognition is mostly designed for a particular group of individuals like occupational stress, perinatal maternal stress, or health worker stress and fails to propose a framework that would not be targeted for a particular group of individual. Second, most of the previous work done on stress recognition focuses on the extraction of handcrafted features thus requiring human intervention and expertise. To address these issues, this study proposes, a hybrid deep learning based ensemble approach for automated extraction of features and classification into various state of stress for MHC system. The proposed framework takes input from wearable physiological sensors and is provided to deep learning classifier of convolutional neural network (CNN) and CNN‐long short term memory based ensemble model. The proposed framework has been experimented on the wearable stress and affect detection dataset and reports an accuracy of 91.52% that is 7.20% higher than earlier reported accuracies from other machine learning and deep learning models.