Recent advancements in Audio Language Models (ALMs) have led to significant improvements in speech-related tasks. However, their capacity for profound metaphorical reasoning, especially when derived from audio-specific cues, has yet to be thoroughly investigated. To address this gap, we introduce Unspoken, a bilingual (Chinese-English) question answering benchmark designed to assess ALMs' comprehension of non-literal, metaphor-rich audio. Unlike prior text-centric evaluations, Unspoken emphasizes prosody, phonetic ambiguity, emotional inflection, and other nuanced acoustic features critical to metaphor understanding but often lost in transcription. We construct a high-quality dataset of 2,764 manually curated and validated QA pairs, spanning three reasoning dimensions: semantic, acoustic, and contextual, and covering six common types of metaphors. Evaluation across 23 mainstream ALMs reveals a substantial performance gap: the best model achieves only 69.5% accuracy, significantly below the human average of 81.1%. By analyzing the error patterns, we identify five key failure modes that reveal fundamental limitations in current models' reasoning capabilities. Unspoken not only sets a new standard for evaluating metaphorical reasoning in audio but also pioneers a novel research direction that moves beyond transcription-based assessments. Grounding metaphor understanding in authentic human communication scenarios offers deep insight for developing more cognitively capable ALMs. The data and codes are available at https://github.com/Hongru0306/UNSPOKEN.

Abstract Marine Internet of Things (IOT) is the use of Internet technology to connect various sensing devices at sea, so as to integrate maritime information and realize the monitoring and systematic management of complex data at sea. The marine environment is complex and changeable, and marine disasters occur frequently, such as red tides. Due to the sudden and destructive nature of red tide, it plays a pivotal role to monitor the occurrence of the red tide for the marine IoT, where machine learning has been widely used to predict red tides. However, they were rarely able to catch the sudden change of chlorophyll‐a, which has important practical significance for predicting the occurrence of red tide. In order to deal with the above problems, this paper proposes the triple‐stage attention‐based recurrent neural network, which can enhance the representation ability of input sequences, selectively capture dynamic spatial correlations between input multi‐channel observations in the input sequence, meanwhile adaptively capturing dynamic temporal correlations between different time intervals in the input sequence. The results show that this method outperforms the state‐of‐art baseline methods here.

Accurate measurement of wind speed on maritime vessels is crucial for weather and sea condition forecasting, safe navigation, power generation, hydrological simulation, and other applications. However, the precision of measurements may be subject to certain errors due to factors such as vessel motion and environmental conditions. To enhance the precision of shipborne wind speed measurement, this paper introduces an innovative approach based on contrastive learning. Through proficient feature extraction and the application of a self-supervised contrastive learning algorithm, this method extracts features of varying granularity from marine observational data to predict and correct shipborne wind speed measurements. To the best of our knowledge, this study represents the first attempt to validate contrastive learning in the intelligent analysis of marine observational data. Validation experiment results demonstrate the outstanding performance of this method in both single-step and multi-step predictions, showcasing higher efficacy and robustness compared to alternative approaches.

This study introduces a novel Supervised Info-enhanced Contrastive Learning framework for EEG based Emotion Recognition (SI-CLEER). SI-CLEER employs multi-granularity contrastive learning to create robust EEG contextual representations, potentially improving emotion recognition effectiveness. Unlike existing methods solely guided by classification loss, we propose a joint learning model combining self-supervised contrastive learning loss and supervised classification loss. This model optimizes both loss functions, capturing subtle EEG signal differences specific to emotion detection. Extensive experiments demonstrate SI-CLEER’s robustness and superior accuracy on the SEED dataset compared to state-of-the-art methods. Furthermore, we analyze electrode performance, highlighting the significance of central frontal and temporal brain region EEGs in emotion detection. This study offers an universally applicable approach with potential benefits for diverse EEG classification tasks.

Deep learning is particularly important in the field of time series data analysis, and has been applied to tasks such as marine data prediction.However, there is a 'concept drift' problem in marine observation data, which leads to performance degradation and catastrophic forgetting of traditional deep learning models used in online scenarios.For this reason, this paper proposes OL-TCN, an adaptive temporal convolutional neural network online prediction deep learning model, which is more suitable for marine online learning and inference scenarios.We make the following innovations: Inside the model, model performance is enhanced by incorporating a multi-head attention mechanism and introducing an automatic machine learning enhancement in the residual cell.Outside the model, a model repository approach is used to effectively cope with the complexity and evolutionary challenges of data streams.The experimental results verify that the OL-TCN model is effective and feasible in marine time series processing.

This study presents a novel Supervised Information Enhanced Contrastive Learning Algorithm for Parkinson’s Disease Detection (SI-CLAPD) based on Electroencephalography (EEG). SI-CLAPD performs contrastive learning in a multi-granularity manner on enhanced contextual views to achieve robust contextual representations for EEG, thus could contribute to improving the effectiveness of PD detection. Unlike existing methods for constructing PD detection models guided solely by classification loss, we propose a joint learning model that combines self-supervised contrastive learning with supervised classification learning. This model is optimized using both contrastive loss and classification loss, allowing it to capture subtle differences between EEG signals and representations that are specific to PD detection. Through extensive experimental evaluations, we demonstrate that SI-CLAPD achieves robust and high accuracy in PD detection tasks on three benchmark datasets. To the best of our knowledge, this study represents the first effort in validating the effectiveness of contrastive learning for the detection of PD. Besides, within the realm of contrastive learning research in EEG, it also represents the first endeavor to fuse supervised learning with self-supervised contrastive learning for EEG classification. This investigation unveils an universally applicable approach to EEG signal processing, with the potential to confer advantages to a multitude of EEG classification tasks.

In the context of the rapid development of big data and artificial intelligence, the field of HPC is facing significant challenges in energy consumption. To address this challenge, this paper proposes a power consumption prediction framework for HPC jobs based on self-supervised contrastive learning, named Job2Vec. Firstly, HPC jobs are clustered into different clusters based on job logs and power consumption curve shapes. Next, high-quality features of power consumption data at different granularities within each cluster are captured through self-supervised contrastive learning. Lastly, an incremental update strategy is introduced to handle the dynamic changes in HPC data. The experimental results demonstrate that, on the Jinan Supercomputing dataset comprising 1170 HPC jobs, the Job2Vec framework outperforms the baseline models overall in terms of prediction performance. Additionally, we also investigated the model’s performance in single-step and multi-step forecasting, the impact of different indicators on prediction accuracy, and conducted ablation experiments to confirm the indispensability of each module. This is the first time that self-supervised contrastive learning has been introduced into the HPC field, aiming to provide support for energy scheduling in data centers.

EEG based emotion recognition technology is currently an important concept in artificial intelligence, and also holds great potential in emotional health care. Nevertheless, one major limitation of the prior approaches is they do not capture the relationships between different time-series and channels explicitly, resulting in inevitable low performance, especially in subject-independent recognition settings. In this paper, we propose a novel graph attention network based model to address this issue. Our framework includes dual-pipeline Graph Attention Network layers in parallel to learn the complex dependencies of multi-channel EEG in both temporal and spatial dimensions. The proposed method outperforms other state-of-the-art models on benchmark SEED dataset. Further analysis shows that our method also has good interpretability. As far as we know, it is the first work that introduce graph attention network into EEG based emotion detection research.

Electroencephalogram (EEG) is a precise reflection of the brain activities and has been widely studied in clinical medicine, neuroscience, brain interface, etc. Intelligent prediction of the EEG’s evolution accurately plays important roles in several application areas, such as epilepsy seizure forecasting and neonatal brain monitoring. Nevertheless, when the prediction service is deployed as a business service on the Cloud and open for public usage, there are several problems that need to be resolved: (i) how to design a computation platform to process the high-throughput multi-source EEG data, which arrives sequentially and increases rapidly when the services are rapidly promoted, namely tackling the ‘high-throughput computing’ problem; (ii) how to develop a deep learning model to capture the complex EEG distribution as well as the anomaly patterns that could evolve dynamically, namely tackling the ‘concept drift’ problem for non-stationary EEG signals. To tackle these challenges, we propose an Evolutive Convolutional Neural Network (ECNN) and the corresponding supercomputer supported distributed computation system. The ECNN model can dynamically reweighting the sub-structure of the model from data streams in an online learning fashion, by which the capacity scalability and sustainability are introduced into the model. As far as we know, it is the first work that introduce supercomputer supported online deep learning techniques into EEG prediction research.

Depression is a typical mood disease that makes people a persistent feeling of sadness and loss of interest and pleasure. Emotion thus comes into sight and is tightly entangled with depression in that one helps the understanding of the other. Depression and emotion detection has been a new research task. The central challenges in this task are multi-modal interaction and multi-task correlation. The existing approaches treat them as two separate tasks, and fail to model the relationships between them. In this paper, we propose an attentive multi-modal multitask learning framework, called AMM, to generically address such issues. The core modules are two attention mechanisms, viz. inter-modal $(I_{\mathrm{e}})$ and inter-task $(I_{t})$ attentions. The main motivation of $I_{\mathrm{e}}$ attention is to learn multi-modal fused representation. In contrast, Itattention is proposed to learn the relationship between depression detection and emotion recognition. Extensive experiments are conducted on two large scale datasets, i.e., DAIC and multi-modal Getty Image depression (MGID). The results show the effectiveness of the proposed AMM framework, and also shows that AMM obtains better performance for the main task, i.e., depression detection with the help of the secondary emotion recognition task.

The COVID-19 pandemic has had a severe impact on humans’ lives and and healthcare systems worldwide. How to early, fastly and accurately diagnose infected patients via multimodal learning is now a research focus. The central challenges in this task mainly lie on multi-modal data representation and multi-modal feature fusion. To solve such challenges, we propose a medical knowledge enriched multi-modal sequence to sequence learning model, termed MedSeq2Seq. The key components include two attention mechanisms, viz. intra-modal (Ia) and inter-model (Ie) attentions, and a medical knowledge augmentation mechanism. The former two mechanisms are to learn multi-modal refined representation, while the latter aims to incorporate external medical knowledge into the proposed model. The experimental results show the effectiveness of the proposed MedSeq2Seq framework over state-of-the-art baselines with a significant improvement of 1%-2%.

Robust cross-subject emotion recognition based on multichannel EEG has always been a hard work. In this work, we hypothesize there exists default brain variables across subjects in emotional processes. Hence, the states of the latent variables that related to emotional processing must contribute to building robust recognition models. We propose to utilize variational autoencoder (VAE) to determine the latent factors from the multichannel EEG. Through sequence modeling method, we examine the emotion recognition performance based on the learnt latent factors. The performance of the proposed methodology is verified on two public datasets (DEAP and SEED), and compared with traditional matrix factorization based (ICA) and autoencoder based (AE) approaches. Experimental results demonstrate that neural network is suitable for unsupervised EEG modeling and our proposed emotion recognition framework achieves the state-of-the-art performance. As far as we know, it is the first work that introduces VAE into multichannel EEG decoding for emotion recognition.

Identification of epithelial and stromal regions by a computerized system is quite challenging due to their irregular shape and size in histopathological images. Nowadays, although convolutional neural networks(CNNs) have significantly push forward this field, the typical use of CNNs in histopathological images is to classify an image block into a corresponding category. It fails to enable a better adherence of class boundaries due to the trade-off between networks' localization accuracy and input context. Larger blocks require more pooling layers that reduce localization accuracy, while tiny blocks allow networks to see only a small amount of context. So in this paper, we propose a generalized densely connected encoder-decoder network to deal with this problem. The main idea behind our network is captured in dense skip connections to compensate for resolution loss induced by pooling layers. So our network can have a large amount of context as input without losing localization accuracy. Finally, we show that the proposed network can outperform U-Net on Stanford Tissue Microarray Database without any further post-processing module or pretraining. Moreover, due to smart construction of the model, our approach has much fewer parameters than currently published best entries for this dataset. This means our approach is much faster than currently published entries. Segmentation of a 512×512 image only takes 34ms on an Nvidia Tesla P100 GPU.

Multimodal sentiment analysis aims to capture diversified sentiment information implied in data that are of different modalities (e.g., an image that is associated with a textual description or a set of textual labels). The key challenge is rooted on the “semantic gap” between different low-level content features and high-level semantic information. Existing approaches generally utilize a combination of multimodal features in a somehow heuristic way. However, how to employ and combine multiple information from different sources effectively is still an important yet largely unsolved problem. To address the problem, in this paper, we propose a Quantum-inspired Multimodal Sentiment Analysis (QMSA) framework. The framework consists of a Quantum-inspired Multimodal Representation (QMR) model (which aims to fill the “semantic gap” and model the correlations between different modalities via density matrix), and a Multimodal decision Fusion strategy inspired by Quantum Interference (QIMF) in the double-slit experiment (in which the sentiment label is analogous to a photon, and the data modalities are analogous to slits). Extensive experiments are conducted on two large scale datasets, which are collected from the Getty Images and Flickr photo sharing platform. The experimental results show that our approach significantly outperforms a wide range of baselines and state-of-the-art methods.

Tang poetry, as one of the most typical ways for ancient Chinese to express their emotions, has been continuously inherited for millenniums in China. Nowadays, though our way of life has changed dramatically, the traditional culture about Tang poetry is still affecting us deeply. However, the psychological effect of Tang poetry remains unclear currently. Motivated by this, we aim to investigate the impact of Tang poetry on human emotional state through designing effective experimental paradigm. We gathered 16 Tang poetry video clips with different genres, and played these to 18 subjects, meanwhile recorded their brain neural responses via Electroencephalogram (EEG). A questionnaire was designed to record subjects' subjective ratings of the emotional experience when watching the Tang poetry videos. Through analyzing the questionnaires, we found that Tang poetry can indeed induce the subjects' specific emotions. Finally, we performed a pilot analysis of the recorded EEG signals and the subjective ratings, for exploring the correlation of the brain activity with the emotional states. The results indicate that the Tang poetry can be used as a kind of emotive stimuli in affective computing research, and the EEG induced by Tang poetry can be utilized to probe the human's internal emotions.

Automatic emotion recognition based on multi-channel neurophysiological signals, as a challenging pattern recognition task, is becoming an important computer-aided method for emotional disorder diagnoses in neurology and psychiatry. Traditional approaches require designing and extracting a range of features from single or multiple channel signals based on extensive domain knowledge. This may be an obstacle for non-domain experts. Moreover, traditional feature fusion method can not fully utilize correlation information between different channels. In this paper, we propose a preprocessing method that encapsulates the multi-channel neurophysiological signals into grid-like frames through wavelet and scalogram transform. We further design a hybrid deep learning model that combines the `Convolutional Neural Network (CNN)' and `Recurrent Neural Network (RNN)', for extracting task-related features, mining inter-channel correlation and incorporating contextual information from those frames. Experiments are carried out, in a trial-level emotion recognition task, on the DEAP benchmarking dataset. Our results demonstrate the effectiveness of the proposed methods, with respect to the emotional dimensions of Valence and Arousal.