Speech-to-Speech (S2S) Large Language Models (LLMs) are foundational to natural human-computer interaction, enabling end-to-end spoken dialogue systems. However, evaluating these models remains a fundamental challenge. We propose SageLM, an end-to-end, multi-aspect, and explainable speech LLM for comprehensive S2S LLMs evaluation. First, unlike cascaded approaches that disregard acoustic features, SageLM jointly assesses both semantic and acoustic dimensions. Second, it leverages rationale-based supervision to enhance explainability and guide model learning, achieving superior alignment with evaluation outcomes compared to rule-based reinforcement learning methods. Third, we introduce SpeechFeedback, a synthetic preference dataset, and employ a two-stage training paradigm to mitigate the scarcity of speech preference data. Trained on both semantic and acoustic dimensions, SageLM achieves an 82.79% agreement rate with human evaluators, outperforming cascaded and SLM-based baselines by at least 7.42% and 26.20%, respectively.

Multi-domain machine translation (MDMT) aims to build a unified model capable of translating content across diverse domains. Despite the impressive machine translation capabilities demonstrated by large language models (LLMs), domain adaptation still remains a challenge for LLMs. Existing MDMT methods such as in-context learning and parameter-efficient fine-tuning often suffer from domain shift, parameter interference and limited generalization. In this work, we propose a neuron-efficient fine-tuning framework for MDMT that identifies and updates consensus-aligned neurons within LLMs. These neurons are selected by maximizing the mutual information between neuron behavior and domain features, enabling LLMs to capture both generalizable translation patterns and domain-specific nuances. Our method then fine-tunes LLMs guided by these neurons, effectively mitigating parameter interference and domain-specific overfitting. Comprehensive experiments on three LLMs across ten German-English and Chinese-English translation domains evidence that our method consistently outperforms strong PEFT baselines on both seen and unseen domains, achieving state-of-the-art performance.

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with LLM alignment performance, supporting it as a reliable reference for developing advanced reward models. By analyzing the evaluation results on MRMBench, we reveal that reward models struggle to simultaneously capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Furthermore, our results demonstrate that the proposed inference-time probing method provides a reliable metric for assessing the confidence of reward predictions, leading to improved alignment of large language models.

Differentially private (DP) image synthesis enables the generation of realistic images while bounding privacy leakage, facilitating secure data sharing across organizations. However, the Gaussian noise injected during DP training, such as via DP-SGD, often severely degrades synthesis quality by disrupting model convergence. To address this, we introduce RPGen, a novel framework that enhances diffusion models' parameter robustness to mitigate DP noise effects without compromising privacy guarantees. At its core, RPGen employs adversarial model perturbation (AMP) during public pre-training to build resilience against perturbations, but we identify and tackle the critical issue of robustness transferability across domains. RPGen achieves this through a three-step process: (1) A pre-trained classifier infers labels for private images, aggregated into a class distribution noised with Gaussian mechanism for DP, and public samples are selected to match this privatized distribution for domain alignment; (2) The diffusion model is pre-trained on this curated subset with adversarial model perturbation to foster robustness; (3) The model undergoes fine-tuning on private data using DP-SGD. This synergy of robustness augmentation and transferability optimization yields high-fidelity synthesis. Extensive evaluations on ImageNet for pre-training, with CelebA and CIFAR-10 for synthesis, show RPGen outperforming state-of-the-art baselines across epsilon in 1, 5, 10. On average, it achieves 20.18% lower FID and 5.45% higher classification accuracy. Ablations confirm the efficacy of domain curation and modest perturbations, establishing RPGen as a new benchmark for privacy-utility trade-offs in image generation.

Yuan Ge, Haishu Zhao, AoKai Hao, Junxiang Zhang, Bei Li, Xiaoqian Liu, Chenglong Wang, Jianjin Wang, Bingsen Zhou, Bingyu Liu, JingBo Zhu, Zhengtao Yu, Tong Xiao. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Jiecong Wang, Haoran Li, Hao Peng, Ziqian Zeng, Zihao Wang, Haohua Du, Zhengtao Yu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Single-cell RNA sequencing (scRNA-seq) has provided unprecedented insights for life science research. In scRNA-seq data analysis, clustering is a crucial step that lays the foundation for downstream tasks. However, the high dimensionality and sparsity of scRNA-seq data lead to suboptimal representations learned by existing methods, thereby limiting clustering performance. To address these issues, we propose scDGCL, a novel dual-level and graph-constrained contrastive learning method for scRNA-seq data clustering. Specifically, we first design Dual-level Contrastive Learning (DCL), which optimizes cell representations by simultaneously considering similarities and disparities at both cell and cluster levels. Then, we design Graph-constrained Contrastive Learning (GCL), which aligns the DCL-derived representations with the graph's relational priors, further enhancing cell representations. To systematically evaluate scDGCL, we perform experiments on 12 real datasets and 8 simulated datasets. Comparing scDGCL with 17 representative clustering methods, the results demonstrate it's superiority in scRNA-seq data clustering. Ablation experiments and hyperparameter experiments are performed to verify the effectiveness of each component and the overall robustness of our method. Marker gene expression and cell trajectory inference analysis verify the biological plausibility of our method from a biological perspective.

Enhancing the resolution of Magnetic Resonance Imaging (MRI) through super-resolution (SR) reconstruction is crucial for boosting diagnostic precision. However, current SR methods primarily rely on single LR images or multi-contrast features, limiting detail restoration. Inspired by video frame interpolation, this work utilizes the spatiotemporal correlations between adjacent slices to reformulate the SR task of anisotropic 3D-MRI image into the generation of new high-resolution (HR) slices between adjacent 2D slices. The generated SR slices are subsequently combined with the HR adjacent slices to create a new HR 3D-MRI image. We propose a innovative network architecture termed DGWMSR, comprising a backbone network and a feature supplement module (FSM). The backbone's core innovations include the displacement former block (DFB) module, which independently extracts structural and displacement features, and the mask-displacement vector network (MDVNet) which combines with Warp mechanism to facilitate edge pixel detailing. The DFB integrates the inter-slice attention (ISA) mechanism into the Transformer, effectively minimizing the mutual interference between the two types of features and mitigating volume effects during reconstruction. Additionally, the FSM module combines self-attention with feed-forward neural network, which emphasizes critical details derived from the backbone architecture. Experimental results demonstrate the DGWMSR network outperforms current MRI SR methods on Kirby21, ANVIL-adult, and MSSEG datasets.

Domain-specific multimodal neural machine translation (DMNMT) aims to translate source language domain sentences into target language by incorporating images as additional contextual information. However, domain-specific multimodal scenarios frequently suffer from visual imbalance issues, such as one sentence corresponding to multiple images or even no images at all. Effectively integrating visual information into text to enhance domain sentence translation performance under visual imbalance issues is one of the critical challenges for DMNMT, especially for domain-related terms. To tackle these domain-specific visual imbalance problems, this article introduces a virtual domain distillation-enhanced multimodal fusion with the awareness of multiview correlations to enhance the robustness and performance of domain machine translation across various multimodal domain scenarios. We first adopt a multiview correlation-aware cross-modal distillation strategy to generate virtual domain visual scenes by extracting visual correlations among all images through multikernel representations. Subsequently, we integrate pseudo-domain visual scenes into text to improve the performance of domain-specific machine translation. Our proposed approach has the ability to capture domain visual representations across different scenarios, and contributing to more effective domain-specific translation. We conduct expensive experiments on three domain-specific and general-domain benchmark datasets. Experimental results demonstrate that our proposed approach achieves state-of-the-art (SOTA) machine translation scores on most test sets. The in-depth analysis demonstrates the effectiveness and robustness of our proposed approach for domain machine translation.

Transformer-based methods have recently shown remarkable success in hyperspectral image classification (HSIC). However, their applications, in practice, still face two significant challenges. First, although the multihead mechanism in self-attention improves model robustness during training, it may overlook the continuity of spectral bands. Second, existing methods often struggle to effectively balance global and local information during multiscale feature extraction, limiting further improvements in classification performance. To address these issues, we propose a novel spectral-guided multiscale feature-aware Transformer (SMFAT) framework for HSIC. Specifically, a global low-rank spectral learning (GLSL) module is introduced to project hyperspectral image patches into a low-rank subspace, reducing spectral redundancy and capturing global spectral correlations. Furthermore, we introduce the multiscale feature-aware self-attention (MFASA) mechanism, which dynamically integrates fine- and coarse-grained features to enhance multiscale feature modeling. Finally, a spectral-guided fusion (SGF) module leverages the global spectral information extracted by the GLSL module to guide MFASA in more effectively capturing interspectral correlations and spectral continuity. This approach facilitates a more effective integration of spectral and spatial features in HSIs. Experiments on three well-known HSI datasets verify that the proposed SMFAT method significantly outperforms several state-of-the-art approaches in real-world HSIC tasks. The source code for this work is available at https://github.com/stellaZ77/SMFAT.

Complex claim fact-checking performs a crucial role in disinformation detection. However, existing fact-checking methods struggle with claim vagueness, specifically in effectively handling latent information and complex relations within claims. Moreover, evidence redundancy, where non-essential information complicates the verification process, remains a significant issue. To tackle these limitations, we propose Bilateral Defusing Verification (BiDeV), a novel fact-checking working-flow framework integrating multiple role-played LLMs to mimic the human-expert fact-checking process. BiDeV consists of two main modules: Vagueness Defusing identifies latent information and resolves complex relations to simplify the claim, and Redundancy Defusing eliminates redundant content to enhance the evidence quality. Extensive experimental results on two widely used challenging fact-checking benchmarks (Hover and Feverous-s) demonstrate that our BiDeV can achieve the best performance under both gold and open settings. This highlights the effectiveness of BiDeV in handling complex claims and ensuring precise fact-checking.

Pre-trained language models enhanced parsers have achieved outstanding performance in rich-resource languages. Cross-lingual dependency parsing aims to learn useful knowledge from high-resource languages to alleviate data scarcity in low-resource languages. However, effectively reducing the syntactic structure distributional bias and excavating the commonalities among languages is the key challenge for cross-lingual dependency parsing. To address this issue, we propose novel dynamic syntactic feature filtering and injecting networks based on the typical shared-private model that employs one shared and two private encoders to separate source and target language features. Concretely, a Language-Specific Filtering Network (LSFN) on private encoders emphasizes helpful information and ignores the irrelevant or harmful parts of it from the source language. Meanwhile, a Language-Invariant Injecting Network (LIIN) on the shared encoder integrates the advantages of BiLSTM and improved Transformer encoders to transcend language boundaries, thus amplifying syntactic commonalities across languages. Experiments on seven benchmark datasets show that our model achieves an average absolute gain of 1.84 UAS and 3.43 LAS compared with the shared-private model. Comparative experiments validate that both LSFN and LIIN components are complementary in transferring beneficial knowledge from source to target languages. Detailed analyses highlight that our model can effectively capture linguistic commonalities and mitigate the effect of distributional bias, showcasing its robustness and efficacy.

Multi-view document clustering (MvDC) aims to improve the accuracy and robustness of clustering by fully considering the complementarity of different views. However, in real-world clustering applications, most existing works suffer from the following challenges: 1) They primarily align multi-view data based on a single perspective, such as features and classes, thus ignoring the diversity and comprehensiveness of representations. 2) They treat each instance equally in cross-view contrastive learning without considering ambiguous ones, which weakens the model's discriminative ability. To address these problems, we propose an ambiguous instance-aware contrastive network with multi-level matching (AICN-MLM) for MvDC tasks. This model contains two key modules: a multi-level matching module and an ambiguous instance-aware contrastive learning module. The former attempts to align multi-view data from different perspectives, including features, pseudo-labels, and prototypes. The latter dynamically adjusts instance weights through a weight modulation function to highlight ambiguous instance pairs. Thus, our proposed method can effectively explore the consistency of multi-view document data and focus on ambiguous instances to enhance the model's discriminative ability. Extensive experimental results on several multi-view document datasets verify the effectiveness of our proposed method.

With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the speech features nodes within the graph and extracting the corresponding codebook by hierarchically and disentangledly minimizing 2D SE. Then, to address the issue of audio distortion, we propose a new quantization method. This method still adheres to the 2D SE minimization principle, adaptively selecting the most suitable token corresponding to the cluster for each incoming original speech node. Furthermore, we develop a Structural Entropy-based Speech Language Model (SESLM) that leverages SECodec. Experimental results demonstrate that SECodec performs comparably to EnCodec in speech reconstruction, and SESLM surpasses VALL-E in zero-shot text-to-speech tasks.

Sentence representation learning (SRL) aims to learn sentence embeddings that conform to the semantic information of sentences.In recent years, fine-tuning methods based on pre-trained models and contrastive learning frameworks have significantly advanced the quality of sentence representations.However, within the semantic space of SRL models, both word embeddings and sentence representations derived from word embeddings exhibit substantial redundant information, which can adversely affect the precision of sentence representations.Existing approaches predominantly optimize training strategies to alleviate the redundancy problem, lacking fine-grained guidance on reducing redundant representations.This paper proposes a novel approach that dynamically identifies and reduces redundant information in a dimensional perspective, training the SRL model to redistribute semantics on different dimensions, and entailing better sentence representations.Extensive experiments across seven semantic text similarity benchmarks demonstrate the effectiveness and generality of the proposed method.A comprehensive analysis of the experimental results is conducted and the code/data will be released.

Recent advancements in end-to-end text-to-speech models have made significant progress. However, these approaches based on high-resource languages, are inapplicable for low-resource languages, and existing low-resource speech synthesis methods are typically specific to single languages. Consequently, the pursuit of a universal methodology for low-resource speech synthesis emerges as a critical problem that requires immediate attention. Unlike previous works, we propose a novel and universal approach for multiple low-resource languages by leveraging phoneme fusion coordinating low-rank decomposition in a pre-trained multilingual model. Specifically, by establishing a multilingual phoneme dictionary through phoneme fusion and applying parameter freezing and matrix decomposition techniques for fine-tuning, the proposed method was extensively evaluated across four different languages, each utilizing approximately 3 hours of speech data. The experimental results demonstrate that the quality of the synthesized speech surpasses current mainstream methods for low-resource speech synthesis, achieving outcomes comparable to those trained with tens of hours of data. Audio samples are available at https://priankaan.github.io/Demo.github.io.

Voice conversion (VC) aims to transform a person’s voice to resemble that of another person while maintaining the original linguistic content. Existing methods suffer from the blurring of speech representations and the leakage of prosody information. To address this issue, this study introduces SEVC, a novel neural structural entropy-based VC framework. First, we extract self-supervised representations from both the source and reference speech. The representations of the reference speech are structured as a graph. Using two-dimensional (2D) structural entropy (SE), semantically similar representations are clustered together. For the speaker conversion process, each frame of the source speech is treated as a new node, and the most appropriate semantic cluster for each node is identified using SE. Each frame of the source representation is then replaced by the center representation of its corresponding semantic cluster from the reference speech. Finally, a pretrained vocoder synthesizes audio from the transformed representations. Quantitative and qualitative evaluations demonstrate that SEVC improves speaker similarity while maintaining intelligibility scores comparable to existing methods. Additionally, experimental results indicate that SEVC effectively disentangles paralinguistic features from the reference speech, allowing the model to better focus on the semantic content.

Multimodal Emotion Recognition aims to identify emotions by analyzing data from multiple modalities. However, real-world scenarios often involve missing modalities, which significantly degrade recognition performance. Existing methods neglect the interaction between reconstructed and available modalities and fail to address varying degrees of modality missingness. To tackle these challenges, we propose a Virtual Fusion Feature Generation model with Curriculum Learning. Instead of reconstructing individual missing modalities, our method directly generates virtual fusion features by leveraging real fused multimodal features as guidance. Specifically, we design a residual autoencoder to generate virtual fusion features under missing conditions and introduce a KL-based Similarity Alignment mechanism to align these virtual features with the real fused features extracted from complete modalities. Additionally, a curriculum learning strategy is applied to gradually adapt the model to increasingly challenging missing modality scenarios. Experiments on IEMOCAP and MSP-IMPROV demonstrate that our approach outperforms state-of-the-art methods under various missing conditions.

Unsupervised Automatic Speech Recognition (UASR) aims to align speech signals with corresponding text without relying on annotated data. However, the unknown boundaries of speech units pose challenges for cross-modal alignment, and existing methods often struggle to accurately identify these boundaries and capture structural information. To address this, we propose a reward model based on knowledge of linguistic rules. By analyzing boundary structures at three levels—phonemes, syllables, and words—we design multi-granularity scoring functions, including Sequence Reasonability (SR), Syllable Legality Rules (SLR), Syllable-to-Word Evaluation (SWE), and Syntax Grammar Rules (SGR), to generate reward signals. These signals are utilized in reinforcement learning to train a segmentation model capable of effectively capturing the latent structural patterns in speech. Experimental evaluations on two publicly available English datasets demonstrate that the proposed method effectively improves speech unit boundary detection and cross-modal alignment accuracy, validating the feasibility of integrating linguistic knowledge into the UASR framework.

Research on Graph Structure Learning (GSL) provides key insights for graph-based clustering, yet current methods like Graph Neural Networks (GNNs), Graph Attention Networks (GATs), and contrastive learning often rely heavily on the original graph structure. Their performance deteriorates when the original graph's adjacency matrix is too sparse or contains noisy edges unrelated to clustering. Moreover, these methods depend on learning node embeddings and using traditional techniques like k-means to form clusters, which may not fully capture the underlying graph structure between nodes. To address these limitations, this paper introduces DeSE, a novel unsupervised graph clustering framework incorporating Deep Structural Entropy. It enhances the original graph with quantified structural information and deep neural networks to form clusters. Specifically, we first propose a method for calculating structural entropy with soft assignment, which quantifies structure in a differentiable form. Next, we design a Structural Learning layer (SLL) to generate an attributed graph from the original feature data, serving as a target to enhance and optimize the original structural graph, thereby mitigating the issue of sparse connections between graph nodes. Finally, our clustering assignment method (ASS), based on GNNs, learns node embeddings and a soft assignment matrix to cluster on the enhanced graph. The ASS layer can be stacked to meet downstream task requirements, minimizing structural entropy for stable clustering and maximizing node consistency with edge-based cross-entropy loss. Extensive comparative experiments are conducted on four benchmark datasets against eight representative unsupervised graph clustering baselines, demonstrating the superiority of the DeSE in both effectiveness and interpretability.

Shengxiang Gao, Fang Nan, Yongbing Zhang, Yuxin Huang, Kaiwen Tan, Zhengtao Yu. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Recent advances in diffusion language models (DLMs) have presented a promising alternative to traditional autoregressive large language models (LLMs). However, DLMs still lag behind LLMs in reasoning performance, especially as the number of denoising steps decreases. Our analysis reveals that this shortcoming arises primarily from the independent generation of masked tokens across denoising steps, which fails to capture the token correlation. In this paper, we define two types of token correlation: intra-sequence correlation and inter-sequence correlation, and demonstrate that enhancing these correlations improves reasoning performance. To this end, we propose a Multi-Reward Optimization (MRO) approach, which encourages DLMs to consider the token correlation during the denoising process. More specifically, our MRO approach leverages test-time scaling, reject sampling, and reinforcement learning to directly optimize the token correlation with multiple elaborate rewards. Additionally, we introduce group step and importance sampling strategies to mitigate reward variance and enhance sampling efficiency. Through extensive experiments, we demonstrate that MRO not only improves reasoning performance but also achieves significant sampling speedups while maintaining high performance on reasoning benchmarks.

Tables are ubiquitous across various domains for concisely representing structured information. Empowering large language models (LLMs) to reason over tabular data represents an actively explored direction. However, since typical LLMs only support one-dimensional (1D) inputs, existing methods often flatten the two-dimensional (2D) table structure into a sequence of tokens, which can severely disrupt the spatial relationships and result in an inevitable loss of vital contextual information. In this paper, we first empirically demonstrate the detrimental impact of such flattening operations on the performance of LLMs in capturing the spatial information of tables through two elaborate proxy tasks. Subsequently, we introduce a simple yet effective positional encoding method, termed "2D-TPE" (Two-Dimensional Table Positional Encoding), to address this challenge. 2D-TPE enables each attention head to dynamically select a permutation order of tokens within the context for attending to them, where each permutation represents a distinct traversal mode for the table, such as column-wise or row-wise traversal. 2D-TPE effectively mitigates the risk of losing essential spatial information while preserving computational efficiency, thus better preserving the table structure. Extensive experiments across five benchmarks demonstrate that 2D-TPE outperforms strong baselines, underscoring the importance of preserving the table structure for accurate table comprehension. Comprehensive analysis further reveals the substantially better scalability of 2D-TPE to large tables than baselines.

Community detection is a critical task in graph theory, social network analysis, and bioinformatics, where communities are defined as clusters of densely interconnected nodes.However, detecting communities in large-scale networks with millions of nodes and billions of edges remains challenging due to the inefficiency and unreliability of existing methods.Moreover, many current approaches are limited to specific graph types, such as unweighted or undirected graphs, reducing their broader applicability.To address these issues, we propose a novel heuristic community detection algorithm, termed CoDeSEG, which identifies communities by minimizing the network's two-dimensional (2D) structural entropy within a potential game framework.In the game, nodes decide to stay in the current community or move to another based on a strategy that maximizes the 2D structural entropy utility function.Additionally, we introduce a structural entropy-based node overlapping heuristic for detecting overlapping communities, with a near-linear time complexity.Experimental results on real-world networks demonstrate that CoDeSEG is the fastest method available and achieves state-of-the-art performance in overlapping normalized mutual information (ONMI) and F1 scores.

In this study, we propose Multimodal Fusion-supervised Cross-modality Alignment Perception (MulFS-CAP), a novel framework for single-stage fusion of unregistered infrared-visible images. Traditional two-stage methods depend on explicit registration algorithms to align source images spatially, often adding complexity. In contrast, MulFS-CAP seamlessly blends implicit registration with fusion, simplifying the process and enhancing suitability for practical applications. MulFS-CAP utilizes a shared shallow feature encoder to merge unregistered infrared-visible images in a single stage. To address the specific requirements of feature-level alignment and fusion, we develop a consistent feature learning approach via a learnable modality dictionary. This dictionary provides complementary information for unimodal features, thereby maintaining consistency between individual and fused multimodal features. As a result, MulFS-CAP effectively reduces the impact of modality variance on cross-modality feature alignment, allowing for simultaneous registration and fusion. Additionally, in MulFS-CAP, we advance a novel cross-modality alignment approach, creating a correlation matrix to detail pixel relationships between source images. This matrix aids in aligning features across infrared and visible images, further refining the fusion process. The above designs make MulFS-CAP more lightweight, effective and explicit registration-free. Experimental results from different datasets demonstrate the effectiveness of our proposed method and its superiority over the state-of-the-art two-stage methods.

Chinese-Vietnamese cross-language event detection aims to cluster texts that describe the same events in Chinese and Vietnamese into corresponding event clusters. However, because Vietnamese is a low-resource language, directly using multilingual pre-trained models to align event representations in Chinese and Vietnamese texts yields suboptimal results, leading to poor performance in cross-lingual event detection. To address this challenge, we propose a method to enhance cross-lingual event detection between Chinese and Vietnamese by utilizing an aligned knowledge event graph. By leveraging aligned event knowledge, such as personal and place names, to establish correlations between events in different languages, we construct a cross-lingual aligned knowledge event graph. Under the constraint of relational associations, we use contrastive learning to model the similarities and differences between various events, making the representations of the same events in different languages more compact. This approach improves the model’s ability to represent Chinese-Vietnamese cross-lingual event texts and enhances the effectiveness of cross-lingual event detection. Experimental results demonstrate that our method, on multiple multilingual pre-trained models, achieves significant improvements across evaluation metrics such as normalized mutual information, adjusted normalized mutual information, and the adjusted rand coefficient.

Due to the strong representation capability of pre-trained language models, Chinese spelling correction models have significantly improved. However, pre-trained language models focus on contextualized information and treat all words equally, thus ignoring the entity information. In practical application, entity words are the most difficult part to handle in various artificial intelligence tasks, i.e., machine translation, optical character recognition, and automatic speech recognition. To address this issue, we first construct an entity-focused Chinese spelling correction (EFCSC) dataset , the first public spelling correction corpus to emphasize entity errors. Furthermore, we propose an entity knowledge injected language model (EKILM) designed for entity-focused spelling correction, which injects entity information into pre-trained language models, thus ensuring traditional spelling correction models pay more attention to entity words. Experiments on several benchmark datasets show that our proposed model outperforms all strong baseline models, leading to state-of-the-art results on all datasets. Extensive experiments and detailed analyses demonstrate that our proposed model enhances entity error correction ability without damaging the normal spelling correction performance. Our code and dataset will be released at https://github.com/DPloved/EFCSC to facilitate future research.

Multilingual document clustering (MDC) aims to partition multilingual documents into distinct clusters based on topic categories in an unsupervised manner. However, existing MDC methods still suffer from several limitations in practice tasks. Firstly, most of them optimize multiple objectives within the same feature space, thereby leading to the conflict between learning consistently shared semantics and reconstructing inconsistent view-specific information. Secondly, several methods directly integrate information from multilingual documents during the fusion stage, thereby overlooking the semantic differences between different language features. To address the aforementioned problems, we propose a novel multi-view learning method, called Semantic Feature Graph Consistency with Contrastive Cluster Assignments (SFGC 3 A), for MDC. Specifically, the proposed SFGC 3 A method implements consistency objective and reconstruction objective in different feature spaces, thus effectively avoiding conflicts between consistency learning and inconsistency reconstruction. Subsequently, we design the semantic feature graph consistency and semantic label consistency modules to further explore consistent semantic information among multilingual documents, thereby reducing the semantic differences among different language views. Extensive experiments on several multilingual document datasets have shown the effectiveness of the proposed SFGC 3 A method in MDC tasks. The source codes for this work will be released later.

Deep hashing technology has recently become an essential tool for cross-modal retrieval on large-scale datasets. However, their performances heavily depend on accurate annotations to train the hashing model. In real applications, we usually only obtain low-quality label annotations owing to labor and time consumption limitations. To mitigate the performance degradation caused by noisy labels, in this paper, we propose a robust deep hashing method, called deep hashing with ranking learning (DHRL), for cross-modal retrieval. The proposed DHRL method consists of a refined semantic concept alignment module and a ranking-swapping module. In this first module, we adopt two transformers to perform the semantic alignment tasks between different modalities on a set of refined concepts, and then convert them into hash codes to reduce heterogeneous differences between multimodalities. The second module first identifies the noisy labels in the training set and ranks them according to ranking loss. Then it swaps the ranking information of different modal network branches. Unlike existing robust hashing methods for assuming noise distribution, our proposed DHRL method requires no prior assumptions for the input data. Extensive experiments on three benchmark datasets have shown that our proposed DHRL method has stronger advantages over other state-of-the-art hashing methods.

Zero-shot sketch-based image retrieval (ZS-SBIR) aims to utilize freehand sketches for retrieving natural images with similar semantics in realistic zero-shot scenarios. Existing works focus on zero-shot semantic transfer using category word embedding and leveraging teacher-student networks to alleviate catastrophic forgetting of pre-trained models. They aim to retain rich discriminative features to achieve zero-shot semantic transfer. However, the category word embedding method is insufficient in flexibility, thereby limiting their retrieval performances in ZS-SBIR scenarios. In addition, the teacher network used for generating guidance signals results in computational redundancy, requiring repeated processing of mini-batch inputs. To address these issues, we propose a deep residual coupled prompt learning (DRCPL) for ZS-SBIR. Specifically, we leverage the text encoder of CLIP to generate category classification weights, thereby improving the flexibility and generality of zero-shot semantic transfer. To tune text and vision representations effectively, we introduce learnable prompts at the input and freeze the parameters of the CLIP encoder. This approach not only effectively prevents catastrophic forgetting, but also significantly reduces the computational complexity of the model. We also introduce the text-vision prompt coupling function to enhance the coordinated consistency between the text and vision representations, ensuring that the two branches can train collaboratively. Finally, we gradually establish stage feature relationships by learning prompts independently at different early stages to facilitate rich contextual learning. Comprehensive experimental results demonstrate that our DRCPL method achieves state-of-the-art performance in ZS-SBIR tasks.

Clustering is a crucial step in single-cell RNA sequencing (scRNA-seq) data analysis, facilitating the discovery of new cell types and the grouping of similar cells. Recently, graph convolutional networks (GCNs) have gained prominence in scRNA-seq data clustering because they effectively learn cell representations by capturing the relationship between cells. However, GCNs are sensitive to noise in scRNA-seq data and are prone to over-smoothing, resulting in the loss of cell-specific information. To overcome these challenges, we propose sigRGCN, a robust residual graph convolutional network for scRNA-seq data clustering. Specifically, we first construct a disturbed cell graph by injecting noise into a cell graph constructed from scRNA-seq data. Then, we design a graph structure optimization graph convolutional network to eliminate the impact of noise in the disturbed cell graph. It significantly improves the robustness of the proposed model in real scRNA-seq data clustering tasks. After that, we utilize a $L$-layers residual graph convolutional network to alleviate the over-smoothing problem. It allows our model to effectively capture higher-order relationships between cells, leading to better cell representations. Finally, we employ a self-supervised manner to optimize our model. The experimental results on nine real scRNA-seq datasets show that our proposed model demonstrates competitive performance in real clustering tasks.

Recently, advances in single-cell sequencing (scRNAseq) technology have enabled large-scale transcriptome analysis with high efficiency and single-cell resolution. Clustering in scRNA-seq is crucial for revealing and categorizing new cell types and gene expression patterns. However, accurate cell clustering remains a challenge due to the high dimensionality and complexity of scRNA-seq data. To overcome this challenge, in this paper, we propose a novel deep scRNA-seq clustering framework, called sparse autoencoders with gene graph embedding (scSAG2E). In our scSAG2E method, two autoencoders are firstly used to learn the low-dimensional representation of cells and genes, respectively, and the gene expression matrix of cells is reconstructed by matrix multiplication. To preserve the manifold structure of cells, we incorporate graph regularization into the cell autoencoder. Additionally, we impose sparse constraints to address the sparsity of the gene expression matrix effectively. Meanwhile, we construct the gene graph using the KNN algorithm and then feed it into the graph convolution networks (GCNs). Therefore, it effectively captures the underlying structure among genes and enhances the signal of differentially expressed genes, leading to a more accurate representation of scRNA-seq data. Extensive experimental results on several publicly available scRNA-seq datasets show that the proposed scSAG2E method outperforms several state-of-the-art single-cell analysis methods in clustering tasks. The source code for this work has been released on https://github.com/xm0312/SCAG2E

Recent hyperspectral image (HSI) classification works have focused on developing a promising architecture by combining convolutional neural networks (CNNs) with Transformers. However, most of them fail to consider the interactive fusion of global and local features, thus significantly limiting the HSI classification performance. To address this problem, in this article, we propose a dual feature aggregation network (DFAN) for HSI classification tasks. It effectively aggregates local and global spatial-spectral features to achieve efficient classification. Specifically, we design a dual feature aggregation (DFA) module to extract and aggregate local and global spatial-spectral features. In this module, the lightweight local block is responsible for extracting local spatial-spectral features, and the global block is used to extract global spatial-spectral features and aggregation tasks. Concretely, the token local aggregation multilayer perceptron (TLA-MLP) module in the global block is to extract spatial-position-aware and spectral-channel-aware global discriminative information. Meanwhile, it learns multiscale neighboring token features by aggregating token local neighborhood features. In addition, we employ the local self-global aggregation block to learn the global aggregation features and then fuse them with the local spatial-spectral features. Afterward, the cross-attention aggregation of local and global spatial-spectral features is used to further improve the classification ability of our proposed model. The experimental results on three benchmark datasets show that our proposed DFAN method outperforms other state-of-the-art HSI classification methods.

Domain generalization (DG) for person re-identification (Re-ID) aims to train models on labeled source domains that generalize well to unseen target domains. However, DG for Re-ID faces a major challenge: existing methods rely solely on labeled paired samples to train DG models and are unable to effectively leverage unpaired samples across cameras. In many cases, cross-camera paired samples are extremely scarce and difficult to annotate. To overcome this limitation, we introduce a novel method specifically tailored for Re-ID. This method leverages cross-camera unpaired samples in model training, thereby reducing the dependence on cross-camera paired samples. We refer to this technique as Unpaired-driven DG (U-DG) person Re-ID. The proposed method leverages a robust image encoder to extract identity-consistent features across various camera views. This capability is further enhanced by integrating a multi-camera person identity classifier, which boosts the encoder’s ability to capture consistent identities, even when viewed from different camera perspectives. To address the scarcity of cross-camera paired samples, we devise a unique model training strategy in our method. Specifically, we use the feature vector from the person identity classifier as a single identity prototype. This prototype serves as a reference for generating identity-related prompts across cameras, effectively compensating for the scarcity of cross-camera paired samples during model training. Additionally, we employ a learnable perturbation prompt to mimic appearance variations exhibited by the same individual across different cameras. Our U-DG offers numerous advantages: it can effectively leverage a large number of unpaired samples for model training, compensating for the scarcity of cross-camera paired samples. Moreover, it does not rely solely on cross-camera paired samples, thereby facilitating the construction of training samples. Experimental results on multiple challenging datasets demonstrate that our approach achieves performance comparable to typical DG person Re-ID, highlighting its feasibility and effectiveness. The source code of our method is available athttps://github.com/lhf12278/DGPS.

Video-text retrieval aims to precisely search for videos most relevant to text queries within a video corpus. However, existing methods are largely limited to single-text (single-event) queries and are not effective at handling multi-text (multi-event) queries. Furthermore, these methods typically focus solely on retrieval and do not attempt to locate multiple events within the retrieved videos. To address these limitations, our paper proposes a novel method named Disentangling Inter- and Intra-Video Relations, which jointly addresses multi-event video-text retrieval and grounding. This method leverages both inter-video and intra-video event relationships to enhance retrieval and grounding performance. At the retrieval level, we devise a Relational Event-Centric Video-Text Retrieval module based on the principle that comprehensive textual information leads to precise correspondence between text and video. It incorporates event relationship features at different hierarchical levels and exploits the hierarchical structure of video relationships to achieve multi-level contrastive learning between events and videos. This approach enhances the richness, accuracy, and comprehensiveness of event descriptions, improving alignment precision between text and video and enabling effective differentiation among videos. For event grounding, we propose Event Contrast-Driven Video Grounding, which accounts for positional differences among events on the 2D temporal score map and achieves precise grounding of multiple events through divergence learning for their locations. Our solution not only provides efficient text-to-video retrieval but also accurately grounds events within the retrieved videos, addressing the shortcomings of existing methods. Extensive experimental results on the ActivityNet Captions and Charades-STA benchmark datasets demonstrate the superior performance of our method, validating its effectiveness. The innovation of this research lies in introducing a new joint framework for video-text retrieval and multi-event grounding while offering new ideas for further research and applications in related fields. The code is available at https://github.com/X7J92/MVT-RG.

With long-tailed data and complex label hierarchy, hierarchical text classification (HTC) is a challenging multi-label text classification task. Applying prompts to pre-trained language models (PLMs) has recently become a mainstream approach in HTC. However, existing prompt-based models experience a significant drop in classification performance on tail labels. Due to the imbalanced data, HTC models still face two challenges. First, text embeddings, learned for classification, often lack distinctiveness for tail categories. Second, label embeddings suffer from significant degeneration, especially for tail labels. To address these issues, in this paper, we propose a novel Hierarchical Text Classification Optimization method via Structural Entropy and SIngular Spectrum Smoothing, namely SIHTC. SIHTC contains two parts: text embedding optimization and label embedding optimization. First, based on the structural information theory, we design a tree aggregation network and construct encoding trees to minimize the structural entropy of texts under the hierarchical labels. In this manner, SIHTC injects label structural information into text embeddings, hierarchically optimizing the embedding space by enclosing the text embeddings within related ground truth labels while separating them from unrelated ground truth labels. Second, we propose a global and local singular spectrum smoothing regularization method to maximize the area under the singular value curve. In this way, SIHTC decreases representation degeneration and learns label embeddings with improved label generalization capability. Extensive experiments are conducted on three popular HTC datasets. The results show that SIHTC outperforms all baseline methods, especially with an advantage in handling tail labels, indicating the effectiveness of the above two optimizations

Text-Pedestrian Image Retrieval employs textual description of pedestrian's appearance to identify the corresponding pedestrian image. This task involves modality discrepancy and the challenges posed by textual diversity of pedestrians with the same identity. Although advancements have been made in text-pedestrian image retrieval, current methods do not comprehensively address these challenges. Thus, this paper proposes a progressive feature mining and external knowledge- assisted feature purification method. Specifically, we implement a progressive mining mode, enabling the model to extract discriminative features from overlooked information. This enhances the model's feature representation capabilities and prevents the loss of discriminative information. To further mitigate the challenges posed by modality discrepancy and text diversity in cross-modal matching, we propose to use external knowledge of other samples from the same modality. This approach accentuates identity-consistent features and diminishes identity-inconsistent ones, refining feature representation and reducing interference from textual diversity and negative sample correlation features of the same modality. Extensive experiments on three challenging datasets demonstrate the effectiveness and superiority of the proposed method, with its retrieval performance outstripping that of large-scale model-based methods on large-scale datasets.

The virtual surgery engine represents a crucial research domain within biomedical and information sciences. To address the real-time and realistic demands of virtual surgical robots for soft tissue deformation and cutting, the surface information and internal structure of organ models have been redefined. An enhanced Three-Parameter Mass-Ogden model, which incorporates nonlinearity and viscoelasticity in soft tissues, has been developed based on extended position dynamics. Cluster constraints for filling particles were introduced to improve the smoothness of surgical procedures. The relaxation and creep characteristics of real soft tissues were accounted for by evaluating the responses of various biological tissues to external stress and loads using the HY-0580 high-performance mechanical testing machine. Eight experiments were conducted for each tissue type, and five sets of valid data were averaged and fitted using the Three-Parameter Mass-Ogden mixed model. Surgical simulations were conducted using Abaqus, incorporating Young's modulus, stress-strain relationships, cutting depth, pressure distribution, real-time feedback, and comprehensive visualization. The model's effectiveness was further validated. The surgical platform was integrated into a virtual reality-based digital twin robot simulator for minimally invasive surgery, achieving a surgical operation refresh rate of 78.5 Hz, a visual refresh rate of 60 Hz, and a haptic feedback refresh rate of 1000 Hz. Comparative analysis with the Mass-Spring Model (MSM) and Finite Element Method (FEM) shows our model's superior balance of accuracy and efficiency. MSM is fast but imprecise, while FEM is accurate but computationally intensive.

Visual grounding has attracted wide attention thanks to its broad application in various visual language tasks. Although visual grounding has made significant research progress, existing methods ignore the promotion effect of the association between text and image features at different hierarchies on cross-modal matching. This paper proposes a Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction Visual Grounding method. It first generates a mask through decoupled sentence phrases, and a text and image hierarchical matching mechanism is constructed, highlighting the role of association between different hierarchies in cross-modal matching. In addition, a corresponding target object position progressive correction strategy is defined based on the hierarchical matching mechanism to achieve accurate positioning for the target object described in the text. This method can continuously optimize and adjust the bounding box position of the target object as the certainty of the text description of the target object improves. This design explores the association between features at different hierarchies and highlights the role of features related to the target object and its position in target positioning. The proposed method is validated on different datasets through experiments, and its superiority is verified by the performance comparison with the state-of-the-art methods.

Despite the fact that there is a remarkable achievement on multifocus image fusion, most of the existing methods only generate a low-resolution image if the given source images suffer from low resolution. Obviously, a naive strategy is to independently conduct image fusion and image super-resolution. However, this two-step approach would inevitably introduce and enlarge artifacts in the final result if the result from the first step meets artifacts. To address this problem, in this article, we propose a novel method to simultaneously achieve image fusion and super-resolution in one framework, avoiding step-by-step processing of fusion and super-resolution. Since a small receptive field can discriminate the focusing characteristics of pixels in detailed regions, while a large receptive field is more robust to pixels in smooth regions, a subnetwork is first proposed to compute the affinity of features under different types of receptive fields, efficiently increasing the discriminability of focused pixels. Simultaneously, in order to prevent from distortion, a gradient embedding-based super-resolution subnetwork is also proposed, in which the features from the shallow layer, the deep layer, and the gradient map are jointly taken into account, allowing us to get an upsampled image with high resolution. Compared with the existing methods, which implemented fusion and super-resolution independently, our proposed method directly achieves these two tasks in a parallel way, avoiding artifacts caused by the inferior output of image fusion or super-resolution. Experiments conducted on the real-world dataset substantiate the superiority of our proposed method compared with state of the arts.

Multiple instance learning (MIL) trains models from bags of instances, where each bag contains multiple instances, and only bag-level labels are available for supervision. The application of graph neural networks (GNNs) in capturing intrabag topology effectively improves MIL. Existing GNNs usually require filtering low-confidence edges among instances and adapting graph neural architectures to new bag structures. However, such asynchronous adjustments to structure and architecture are tedious and ignore their correlations. To tackle these issues, we propose a reinforced GNN framework for MIL (RGMIL), pioneering the exploitation of multiagent deep reinforcement learning (MADRL) in MIL tasks. MADRL enables the flexible definition or extension of factors that influence bag graphs or GNNs and provides synchronous control over them. Moreover, MADRL explores structure-to-architecture correlations while automating adjustments. Experimental results on multiple MIL datasets demonstrate that RGMIL achieves the best performance with excellent explainability. The code and data are available at https://github.com/RingBDStack/RGMIL.

News recommendation aims to offer potentially interesting news items to a specific user, guided by his historical browsing behaviors. Existing methods failed to effectively address the knowledge sparsity issue that the user may have sparse behaviors and the news may own sparse features. To address the problem, we propose a graph-based news recommendation model with intrinsic interest enhancement, named GIIE , leveraging intrinsic interests and neighbor information to enhance the representation of sparse users and news. Concretely, to fully take advantage of the intrinsic interests, we design an interest encoder based on an interest-type graph with a learnable structure and explore the interest embeddings from news types. Then, we inject the obtained interest embeddings into news and represent the user by aggregating the clicked news under the same interest and across different interests sequentially. These interests can build a bridge between users so users with sparse behaviors can implicitly share knowledge with other users, thereby enhancing their representation. To properly introduce the neighbor knowledge, we propose a graph-based neighbor enhancing mechanism. First, we design a news relation graph and a user relation graph in encoders. Then, based on these graphs, we take the attention module to aggregate additional knowledge from neighbors, enhancing sparse news and user representations. To avoid feature ambiguity, we adopt a way to represent the current item (user and news) and its neighbors separately and then do adaptive aggregation. We evaluate GIIE on the public news recommendation datasets MIND-Large and MIND-Small. Experimental results show that our model can solve the knowledge-sparse problem and outperforms current state-of-the-art models in four indicators.

As a trending approach for social event detection, graph neural network (GNN)-based methods enable a fusion of natural language semantics and the complex social network structural information, thus showing SOTA performance. However, GNN-based methods can miss useful message correlations. Moreover, they require manual labeling for training and predetermining the number of events for prediction. In this work, we address social event detection via graph structural entropy (SE) minimization. While keeping the merits of the GNN-based methods, the proposed framework, HISEvent, constructs more informative message graphs, is unsupervised, and does not require the number of events given a priori. Specifically, we incrementally explore the graph neighborhoods using 1-dimensional (1D) SE minimization to supplement the existing message graph with edges between semantically related messages. We then detect events from the message graph by hierarchically minimizing 2-dimensional (2D) SE. Our proposed 1D and 2D SE minimization algorithms are customized for social event detection and effectively tackle the efficiency problem of the existing SE minimization algorithms. Extensive experiments show that HISEvent consistently outperforms GNN-based methods and achieves the new SOTA for social event detection under both closed- and open-set settings while being efficient and robust.

In recent years, Hypergraph Neural Networks (HGNNs) have achieved considerable success by manually designing architectures, which are capable of extracting effective patterns with high-order interactions from non-Euclidean data. However, such mechanism is extremely inefficient, demanding tremendous human efforts to tune diverse model parameters. In this paper, we propose a novel Hypergraph Neural Architecture Search (HyperNAS) to automatically design the optimal HGNNs. The proposed model constructs a search space suitable for hypergraphs, and derives hypergraph architectures through differentiable search strategies. A hypergraph structure-aware distance criterion is introduced as a guideline for obtaining an optimal hypergraph architecture via the leave-one-out method. Experimental results for node classification on benchmark Cora, Citeseer, Pubmed citation networks and hypergraph datasets show that HyperNAS outperforms existing HGNNs models and graph NAS methods.

The interaction between SNPs plays a key role in revealing the genetic mechanisms of complex diseases. However, as the order of SNP interactions increases, the number of SNP combinations increases exponentially, presenting a serious combinatorial explosion problem. Although swarm intelligence algorithms can alleviate the combinatorial explosion problem by optimizing search paths and have been widely used in SNP interaction detection, the performance of swarm intelligence algorithms is greatly affected by initialization and search direction. To mitigate these issues, we propose a high-order SNP interaction detection algorithm based on hierarchical clustering and multitask harmony search (abbreviated as HCMHS). In harmony memory initialization, considering that similar SNPs are more likely to form pathogenic combinations, hierarchical clustering is first used to cluster SNPs into different clusters, and then based on these clusters, harmony memory is initialized. In harmony search, considering the correlation between different orders of SNP interactions, the search directions for SNPs of various orders are optimized through a multi-task framework. To verify the performance of HCMHS, we carried out experiments on 58 simulated datasets and one real dataset, and HCMHS achieved the best results compared to nine advanced algorithms. The code of HCMHS is available at https://github.com/huoluan17-tian/HCMHS.

Personalized dialogue systems have gained significant attention in recent years for their ability to generate responses in alignment with different personas.However, most existing approaches rely on pre-defined personal profiles, which are not only time-consuming and labor-intensive to create but also lack flexibility.We propose In-Dialogue Learning (IDL), a fine-tuning framework that enhances the ability of pre-trained large language models to leverage dialogue history to characterize persona for personalized dialogue generation tasks without pre-defined profiles.Our experiments on three datasets demonstrate that IDL brings substantial improvements, with BLEU and ROUGE scores increasing by up to 200% and 247%, respectively.Additionally, the results of human evaluations further validate the efficacy of our proposed method.

Large language models (LLMs) have demonstrated remarkable capabilities in comprehensively handling various types of natural language processing (NLP) tasks.However, there are significant differences in the knowledge and abilities required for different tasks.Therefore, it is important to understand whether the same LLM processes different tasks in the same way.Are there specific neurons in a LLM for different tasks?Inspired by neuroscience, this paper pioneers the exploration of whether distinct neurons are activated when a LLM handles different tasks.Compared with current research exploring the neurons of language and knowledge, task-specific neurons present a greater challenge due to their abstractness, diversity, and complexity.To address these challenges, this paper proposes a method for task-specific neuron localization based on Causal Gradient Variation with Special Tokens (CGVST).CGVST identifies task-specific neurons by concentrating on the most significant tokens during task processing, thereby eliminating redundant tokens and minimizing interference from non-essential neurons.Compared to traditional neuron localization methods, our approach can more effectively identify task-specific neurons.We conduct experiments across eight different public tasks.Experiments involving the inhibition and amplification of identified neurons demonstrate that our method can accurately locate task-specific neurons.

End-to-end single-stage text-to-speech models have garnered significant attention in recent research, surpassing the performance of conventional two-stage pipeline systems. While prior single-stage models have made substantial advancements, there remains room for improvement in addressing intermittent issues related to unnaturalness and prosody diversity. Unlike previous works, we propose a novel single-stage TTS framework to tackle these problems via hierarchical denoising diffusion generative adversarial networks (GAN) modeling, which parameterizes the denoising model by directly predicting latent variables to improve the naturalness and diversity of the generated speech. Specifically, a conditional GAN is adopted as a non-Gaussian multimodal function to model the denoising distribution, which construct the duration predictor and speech decoder respectively. As such, it allows the TTS model learn the more natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. In addition, We show that DETS can generate high-fidelity speech waveform with only 1 denoising step. Extensive experimental results on the LJSpeech benchmark dataset demonstrate the favourable performance of the proposed method.

The northern and southern accents of the Vietnamese exhibit pronunciation differences in terms of tones and rhythms. Existing Vietnamese pre-trained speech models show a bias in accent representation, leading to a lack of generalization capability for southern accents in Vietnamese speech recognition models and a noticeable decrease in recognition accuracy. In this paper, we propose a decoupled-enhanced adaptive representation strategy guided by prosody and accent domain label information. Through Domain Adversarial Training (DAT), the pre-trained speech model decouples domain-invariant content features. Simultaneously, by incorporating prosodic features, we enhance the pronunciation information of Vietnamese, enabling adaptive representation of the distinctive characteristics of northern and southern Vietnamese accents. Evaluating our proposed method on a dataset of Vietnamese accents, the results demonstrate its superiority over existing approaches, mitigating the performance degradation in Vietnamese speech recognition models caused by accent differences.

Syllabification is a crucial task in natural language processing, and syllables also play a significant role as modeling units in speech processing. While deep learning methods have shown remarkable progress in syllabification, they face challenges in low-resource languages where ready-made segmentation datasets or rules are lacking. Large language models (LLMs) are mostly unsuitable for these low-resource languages as well. To address these challenges, this paper proposes an unsupervised syllabification approach that incorporates logical reasoning into the reinforcement learning training process, achieving knowledge-guided syllabification. By introducing logical reasoning knowledge and modeling the interaction between the Agent and the knowledge base (KB), the model gains a better understanding of language structures and patterns. The study primarily focuses on low-resource Lao language, with experiments conducted on a publicly available English dataset to validate the effectiveness of the proposed method.

Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our observation, dialogue contexts are highly structured, and the special token of End-of-Utterance (EoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as ``conversational attention sinks'' (conv-attn sinks). Accordingly, we introduce StreamingDialogue, which compresses long dialogue history into conv-attn sinks with minimal losses, and thus reduces computational complexity quadratically with the number of sinks (i.e., the number of utterances). Current LLMs already demonstrate the ability to handle long context window, e.g., a window size of 200K or more. To this end, by compressing utterances into EoUs, our method has the potential to handle more than 200K of utterances, resulting in a prolonged dialogue learning. In order to minimize information losses from reconstruction after compression, we design two learning strategies of short-memory reconstruction (SMR) and long-memory reactivation (LMR). Our method outperforms strong baselines in dialogue tasks and achieves a 4 $\times$ speedup while reducing memory usage by 18 $\times$ compared to dense attention recomputation.

Aspect Sentiment Triplet Extraction (ASTE) is an emerging task to extract a given sentence's triplets, which consist of aspects, opinions, and sentiments. Recent studies tend to address this task with a table-filling paradigm, wherein word relations are encoded in a two-dimensional table, and the process involves clarifying all the individual cells to extract triples. However, these studies ignore the deep interaction between neighbor cells, which we find quite helpful for accurate extraction. To this end, we propose a novel model for the ASTE task, called Prompt-based Tri-Channel Graph Convolution Neural Network (PT-GCN), which converts the relation table into a graph to explore more comprehensive relational information. Specifically, we treat the original table cells as nodes and utilize a prompt attention score computation module to determine the edges' weights. This enables us to construct a target-aware gridlike graph to enhance the overall extraction process. After that, a triple-channel convolution module is conducted to extract precise sentiment knowledge. Extensive experiments on the benchmark datasets show that our model achieves state-of-the-art performance. The code is available at https://github.com/KunPunCN/PT-GCN.

Clustering plays a crucial role in analyzing scRNA-seq data and has been widely used in studying cellular distribution over the past few years. However, the high dimensionality and complexity of scRNA-seq data pose significant challenges to achieving accurate clustering from a singular perspective. To address these challenges, we propose a novel approach, called multi-level multi-view network based on structural consistency contrastive learning (scMMN), for scRNA-seq data clustering. Firstly, the proposed method constructs shallow views through the $k$-nearest neighbor ($k$NN) and diffusion mapping (DM) algorithms, and then deep views are generated by utilizing the graph Laplacian filters. These deep multi-view data serve as the input for representation learning. To improve the clustering performance of scRNA-seq data, contrastive learning is introduced to enhance the discrimination ability of our network. Specifically, we construct a group contrastive loss for representation features and a structural consistency contrastive loss for structural relationships. Extensive experiments on eight real scRNA-seq datasets show that the proposed method outperforms other state-of-the-art methods in scRNA-seq data clustering tasks. Our source code has already been available at https://github.com/szq0816/scMMN.

Most methods for multi-domain adaptive neural machine translation (NMT) currently rely on mixing data from multiple domains in a single model to achieve multi-domain translation. However, this mixing can lead to imbalanced training data, causing the model to focus on training for the large-scale general domain while ignoring the scarce resources of specific domains, resulting in a decrease in translation performance. In this paper, we propose a multi-domain adaptive NMT method based on Domain Data Balancer (DDB) to address the problems of imbalanced data caused by simple fine-tuning. By adding DDB to the Transformer model, we adaptively learn the sampling distribution of each group of training data, replace the maximum likelihood estimation criterion with empirical risk minimization training, and introduce a reward-based iterative update of the bilevel optimizer based on reinforcement learning. Experimental results show that the proposed method improves the baseline model by an average of 1.55 and 0.14 BLEU (Bilingual Evaluation Understudy) scores respectively in English-German and Chinese-English multi-domain NMT.

The cooperative data download in vehicular ad hoc networks (VANETs) is attracting more and more attention since a great number of services and applications, which aim at improving user safety and promoting overall travel experience, are based on the efficient and reliable data download among vehicles. However, the characteristics of VANETs, such as fast-moving vehicles, quick-changing topologies, and unstable channel conditions, put great challenge on the realization of efficient cooperative data download, especially for large size data download in highway environment. In this article, we present an efficient and economical cooperative data download mechanism for large size data in highway environment. A multilevel cooperation strategy is designed to enable progressive cooperation and to allow continuous adding of new cooperative vehicles as needed, so that the resources of candidate cooperative vehicles can be utilized sufficiently. Moreover, a reverse auction-based vehicles selection method is proposed to work closely with the multilevel cooperation strategy, to find the most appropriate cooperative vehicles, which can accomplish the data download task with the lowest cost. In addition, the performance degradation introduced by the communication blind zones between road side units (RSUs) is resolved by a carefully designed data forwarding policy. The proposed mechanism focuses on helping vehicles download large size data not only efficiently but also economically, thus distinct it from existing works. Simulation results prove that the proposed mechanism achieves remarkable performance gain in terms of data download cost, download time, completion ratio and the amount of downloaded data in one download period.

Cross-lingual summarization (CLS) is the task of generating a summary in a target language from a document in a source language. Recently, end-to-end CLS models have achieved impressive results using large-scale, high-quality datasets typically constructed by translating monolingual summary corpora into CLS corpora. However, due to the limited performance of low-resource language translation models, translation noise can seriously degrade the performance of these models. In this paper, we propose a fine-grained reinforcement learning approach to address low-resource CLS based on noisy data. We introduce the source language summary as a gold signal to alleviate the impact of the translated noisy target summary. Specifically, we design a reinforcement reward by calculating the word correlation and word missing degree between the source language summary and the generated target language summary, and combine it with cross-entropy loss to optimize the CLS model. To validate the performance of our proposed model, we construct Chinese-Vietnamese and Vietnamese-Chinese CLS datasets. Experimental results show that our proposed model outperforms the baselines in terms of both the ROUGE score and BERTScore.

Cross-lingual summarization (CLS) , generating summaries in one language from source documents in another language, offers invaluable assistance in enabling global access to information for people worldwide. State-of-the-art neural summarization models typically train or fine-tune language models on large-scale corpora. However, this is difficult to achieve in realistic low-resource scenarios due to the lack of domain-specific annotated data. In this article, we present a novel cross-lingual summarization model that utilizes progressive training with mBART and employs reinforcement learning to optimize discrete prompts, which addresses low-resource cross-lingual summarization through a two-pronged approach. During training, we introduce a progressive approach based on mBART, which allows the pre-trained model to gradually acquire the ability to compress information, develop cross-lingual capabilities, and ultimately adapt to specific summarization tasks. During downstream summarization, we employ a discrete-prompts joint pre-trained model based on reinforcement learning optimization to achieve low-resource cross-lingual summarization. Experimental results on four cross-lingual summarization datasets demonstrate state-of-the-art performance and superiority compared to six baselines in low-resource scenarios.

Entity Alignment (EA) aims to identify pairs of entities from two distinct language knowledge graphs (KGs) that represent the same real-world objects. Current EA methods have exhibited impressive performance by leveraging both structural and semantic information. However, these approaches often falter when confronted with EA in low-resource languages. The primary challenge is that low-resource language KGs have sparse graph structures, resulting in difficulty in obtaining accurate entity representations. High-quality entity representation is the key to improving EA performance. Therefore, we propose augmenting entity representations with additional features derived from within the graph. In this paper, we introduce a novel approach: Structure and Semantic Fusion with WD for Low-Resource Language Entity Alignment (WDEA). Our method integrates structural and semantic information using the Wasserstein Distance. Specifically, we design a Wasserstein Graph Convolutional Network (WGCN), a GNN-based model that integrates multi-hop information using a message passing mechanism with WD. Additionally, our method adapts the semantic information from the pre-trained language model in the Wasserstein space to facilitate smooth integration. We also propose the Wasserstein Fusion Encoder (WFE), which effectively combines structural and semantic information in the Wasserstein space. To validate the efficacy of our proposed method, we construct low-resource language EA datasets, encompassing uncommon linguistic varieties with sparser structures compared to mainstream datasets. Experimental results show the superiority of our approach, demonstrating significant performance enhancements in low-resource language EA compared to prevailing baseline models across various information configurations.

Text normalization (TN) is a crucial preprocessing step in text-to-speech synthesis, which pertains to the accurate pronunciation of numbers and symbols within the text. Existing neural network-based TN methods have shown significant success in rich-resource languages. However, these methods are data-driven and highly rely on a large number of labeled datasets, which are not practical in zero-resource settings. Rule-based weighted finite-state transducers (WFST) are a common measure for zero-shot TN, but WFST-based TN approaches encounter challenges with ambiguous input, particularly in cases where the normalized form is context-dependent. On the other hand, conventional neural TN methods suffer from unrecoverable errors. In this paper, we propose ZSTN, a novel zero-shot TN framework based on cross-lingual knowledge distillation, which utilizes annotated data to train the teacher model on rich-resource language and unlabelled data to train the student model on zero-resource language. Furthermore, it incorporates expert knowledge from WFST into a knowledge distillation neural network. Concretely, a TN model with WFST pseudo-labels augmentation is trained as a teacher model in the source language. Subsequently, the student model is supervised by soft-labels from the teacher model and WFST pseudo-labels from the target language. By leveraging cross-lingual knowledge distillation, we address contextual ambiguity in the text, while WFST mitigates unrecoverable errors of the neural model. Additionally, ZSTN is adaptable to different zero-resource languages by using the joint loss function for the teacher model and WFST constraints. We also release a zero-shot text normalization dataset in five languages. We compare ZSTN with seven zero-shot TN benchmarks on public datasets in four languages for the teacher model and zero-shot datasets in five languages for the student model. The results demonstrate that the proposed ZSTN excels in performance without the need for labeled data.

Cross-modal hashing retrieval approaches have received extensive attention owing to their storage superiority and retrieval efficiency. To achieve better retrieval performances, hashing methods seek to embed more semantic information of multi-modal data into hash codes. Existing deep cross-modal hashing methods typically learn hash functions from the similarity of paired data to generate hash codes. However, such locally-oriented learning methods often suffer from low efficiency and incomplete acquisition of semantic information. To address these challenges, this paper presents a novel deep hashing approach, called Proxy-based Graph Convolutional Hashing (PGCH), for cross-modal retrieval. Specifically, we use global similarity to construct proxy hash codes for two different modalities. This strategy of these proxy hash codes ensures that they include data points with significant distribution differences. It helps to match data from different modalities to different proxy hash codes, which can capture the global similarity of multi-modal hash codes and improve the efficiency of hash code learning. Subsequently, we employ a multi-modal contrastive loss to learn the global similarity. Furthermore, by constructing a proxy hash matrix from the proxy hash codes, we apply graph convolution to efficiently narrow the gap between different modalities, leading to a substantial improvement in retrieval performance for cross-modal retrieval tasks. The comprehensive experiments on four benchmark multimedia datasets demonstrate that our PGCH approach achieves better retrieval performances than a bundle of state-of-the-art hashing approaches.

Multifocus image fusion is an effective method to overcome the limitations of optical lenses. The fused results can be obtained from some existing methods by generating decision maps. However, such methods assume that the focused areas of the two source images are complementary, making it impossible to achieve the simultaneous fusion of multiple images. Additionally, existing methods ignore the impact of hard pixels on the fusion performance, limiting the visual quality improvement of fusion images. To address these issues, a combined generation and recombination model called GRFusion is proposed. In GRFusion, the focus property detection of each source image can be independently implemented, enabling the simultaneous fusion of multiple source images and avoiding information loss caused by alternating fusion. It renders the GRFusion free from the limitation of the number of input images. Furthermore, GRFusion investigates the detection of hard pixels with ambiguous focus properties by analyzing the inconsistencies among the detection results of the focus areas in the source images. This allows the hard pixels to be distinguished from the source images. Besides, a multidirectional gradient embedding method is proposed for generating full-focus images. Subsequently, a hard-pixel-guided recombination mechanism for constructing the fused result is devised to integrate the complementary advantages of feature reconstruction-based and focused pixel recombination-based methods. Extensive experimental results demonstrate the effectiveness and superiority of the proposed method. The source code of the proposed method is available at: https://github.com/lhf12278/GRFusion.

Pansharpening, a type of image fusion, combines a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to produce a high-resolution multispectral output. Recent advancements in deep learning have led to the development of data-driven pansharpening methods, which have shown superior performance over traditional non-data-driven approaches. Nevertheless, current data-driven methods primarily focus on the design of single-task-driven networks. They tend to ignore the positive impact that incorporating auxiliary tasks could have on pansharpening. This oversight restricts further improvements in their performance. To break the constraints of single-task methods, we propose a dual-task collaborative promotion network (DCPNet) for pansharpening. DCPNet incorporates an LRMS image super-resolution (SR) reconstruction network into the pansharpening network, establishing a dual-task parallel collaborative framework that achieves joint collaborative optimization for both pansharpening and SR reconstruction. Additionally, we devise a feature fusion scheme that jointly reinforces spectral and spatial details, establishing a bridge for information interaction between the two task branches. This enables the two branches to extract higher quality features in a collaborative way, thus facilitating the realization of the interaction and fusion of dual-task features, as well as the exploration of potential information within the fused features. Experimental results demonstrate that our approach not only effectively transfers spectral information from the LRMS images to the fusion result but also preserves spatial details from the PAN images. The code is available at https://github.com/lhf12278/DCPNet.

Recently published graph neural networks (GNNs) show promising performance at social event detection tasks. However, most studies are oriented toward monolingual data in languages with abundant training samples. This has left the common lesser-spoken languages relatively unexplored. Thus, in this work, we present a GNN-based framework that integrates cross-lingual word embeddings into the process of graph knowledge distillation for detecting events in low-resource language data streams. To achieve this, a novel cross-lingual knowledge distillation framework, called CLKD, exploits prior knowledge learned from similar threads in English to make up for the paucity of annotated data. Specifically, to extract sufficient useful knowledge, we propose a hybrid distillation method that consists of both feature-wise and relation-wise information. To transfer both kinds of knowledge in an effective way, we add a cross-lingual module in the feature-wise distillation to eliminate the language gap and selectively choose beneficial relations in the relation-wise distillation to avoid distraction caused by teachers’ misjudgments. Our proposed CLKD framework also adopts different configurations to suit both offline and online situations. Experiments on real-world datasets show that the framework is highly effective at detection in languages where training samples are scarce.

Real-world social events typically exhibit a severe class-imbalance distribution, which makes the trained detection model encounter a serious generalization challenge. Most studies solve this problem from the frequency perspective and emphasize the representation or classifier learning for tail classes. While in our observation, compared to the rarity of classes, the calibrated uncertainty estimated from well-trained evidential deep learning networks better reflects model performance. To this end, we propose a novel uncertainty-guided class imbalance learning framework - UCL$_{SED}$, and its variant - UCL-EC$_{SED}$, for imbalanced social event detection tasks. We aim to improve the overall model performance by enhancing model generalization to those uncertain classes. Considering performance degradation usually comes from misclassifying samples as their confusing neighboring classes, we focus on boundary learning in latent space and classifier learning with high-quality uncertainty estimation. First, we design a novel uncertainty-guided contrastive learning loss, namely UCL and its variant - UCL-EC, to manipulate distinguishable representation distribution for imbalanced data. During training, they force all classes, especially uncertain ones, to adaptively adjust a clear separable boundary in the feature space. Second, to obtain more robust and accurate class uncertainty, we combine the results of multi-view evidential classifiers via the Dempster-Shafer theory under the supervision of an additional calibration method. We conduct experiments on three severely imbalanced social event datasets including Events2012_100, Events2018_100, and CrisisLexT_7. Our model significantly improves social event representation and classification tasks in almost all classes, especially those uncertain ones.

Network representation learning, also known as network embedding, aims to learn the low-dimensional representations of vertices while capturing and preserving the network structure. For real-world networks, the edges that represent some important relationships between the vertices of a network may be missed and may result in degenerated performance. The existing methods usually treat missing edges as negative samples, thereby ignoring the true connections between two vertices in a network. To capture the true network structure effectively, we propose a novel network representation learning method called WalkGAN, where random walk scheme and generative adversarial networks (GAN) are incorporated into a network embedding framework. Specifically, WalkGAN leverages GAN to generate the synthetic sequences of the vertices that sufficiently simulate random walk on a network and further learn vertex representations from these vertex sequences. Thus, the unobserved links between the vertices are inferred with high probability instead of treating them as nonexistence. Experimental results on the benchmark network datasets demonstrate that WalkGAN achieves significant performance improvements for vertex classification, link prediction, and visualization tasks.

Research on social bot detection plays a crucial role in maintaining the order and reliability of information dissemination while increasing trust in social interactions. The current mainstream social bot detection models rely on black-box neural network technology, for example, Graph Neural Network, Transformer, and so on, which lacks interpretability. In this work, we present UnDBot, a novel unsupervised, interpretable, yet effective, and practical framework for detecting social bots. This framework is built upon structural information theory. We begin by designing three social relationship metrics that capture various aspects of social bot behaviors: posting type distribution , posting influence , and follow-to-follower ratio . Three new relationships are utilized to construct a new, unified, and weighted social multi-relational graph, aiming to model the relevance of social user behaviors and discover long-distance correlations between users. Second, we introduce a novel method for optimizing heterogeneous structural entropy. This method involves the personalized aggregation of edge information from the social multi-relational graph to generate a two-dimensional encoding tree. The heterogeneous structural entropy facilitates decoding of the substantial structure of the social bots network and enables hierarchical clustering of social bots. Third, a new community labeling method is presented to distinguish social bot communities by computing the user’s stationary distribution, measuring user contributions to network structure, and counting the intensity of user aggregation within the community. Compared with 10 representative social bot detection approaches, comprehensive experiments demonstrate the advantages of effectiveness and interpretability of UnDBot on 4 real social network datasets.

Change captioning aims to describe the difference within an image pair in natural language, which combines visual comprehension and language generation. Although significant progress has been achieved, it remains a key challenge of perceiving the object change from different perspectives, especially the severe situation with drastic viewpoint change. In this article, we propose a novel full-attentive network, namely Multi-grained Representation Aggregating Transformer (MURAT), to distinguish the actual change from viewpoint change. Specifically, the Pair Encoder first captures similar semantics between pairwise objects in a multi-level manner, which are regarded as the semantic cues of distinguishing the irrelevant change. Next, a novel Multi-grained Representation Aggregator (MRA) is designed to construct the reliable difference representation by employing both coarse- and fine-grained semantic cues. Finally, the language decoder generates a description of the change based on the output of MRA. Besides, the Gating Cycle Mechanism is introduced to facilitate the semantic consistency between difference representation learning and language generation with a reverse manipulation, so as to bridge the semantic gap between change features and text features. Extensive experiments demonstrate that the proposed MURAT can greatly improve the ability to describe the actual change in the distraction of irrelevant change and achieves state-of-the-art performance on three benchmarks, CLEVR-Change, CLEVR-DC, and Spot-the-Diff.

Visual tempo characterizes the dynamics and the temporal evolution, which helps describe actions. Recent approaches directly perform visual tempo prediction on skeleton sequences, which may suffer from insufficient feature representation issue. In this paper, we observe that relative visual tempo is more in line with human intuition, and thus providing more effective supervision signals. Based on this, we propose a novel Relative Visual Tempo Contrastive Learning framework for skeleton action Representation (RVTCLR). Specifically, we design a Relative Visual Tempo Learning (RVTL) task to explore the motion information in intra-video clips, and an Appearance-Consistency (AC) task to learn appearance information simultaneously, resulting in more representative spatiotemporal features. Furthermore, skeleton sequence data is much sparser than RGB data, making the network learn shortcuts, and overfit to low-level information such as skeleton scales. To learn high-order semantics, we further design a new Distribution-Consistency (DC) branch, containing three components: Skeleton-specific Data Augmentation (S-DA), Fine-grained Skeleton Encoding Module (FSEM), and Distribution-aware Diversity (DD) Loss. We term our entire method (RVTCLR with DC) as RVTCLR+. Extensive experiments on NTU RGB+D 60 and NTU RGB+D 120 datasets demonstrate that our RVTCLR+ can achieve competitive results over the state-of-the-art methods. Code is available at https://github.com/Zhuysheng/RVTCLR.

Graph Neural Networks (GNNs) have shown great power for learning and mining on graphs, and Graph Structure Learning (GSL) plays an important role in boosting GNNs with a refined graph. In the literature, most GSL solutions either primarily focus on structure refinement with task-specific supervision (i.e., node classification), or overlook the inherent weakness of GNNs themselves (e.g., over-squashing), resulting in suboptimal performance despite sophisticated designs. In light of these limitations, we propose to study self-supervised graph structure-feature co-refinement for effectively alleviating the issue of over-squashing in typical GNNs. In this paper, we take a fundamentally different perspective of the Ricci curvature in Riemannian geometry, in which we encounter the challenges of modeling, utilizing and computing Ricci curvature. To tackle these challenges, we present a self-supervised Riemannian model, DeepRicci. Specifically, we introduce a latent Riemannian space of heterogeneous curvatures to model various Ricci curvatures, and propose a gyrovector feature mapping to utilize Ricci curvature for typical GNNs. Thereafter, we refine node features by geometric contrastive learning among different geometric views, and simultaneously refine graph structure by backward Ricci flow based on a novel formulation of differentiable Ricci curvature. Finally, extensive experiments on public datasets show the superiority of DeepRicci, and the connection between backward Ricci flow and over-squashing. Codes of our work are given in https://github.com/RiemanGraph/.

Cross-lingual sentence embedding’s goal is mapping sentences with similar semantics but in different languages close together and dissimilar sentences farther apart in the representation space. It is the basis of many downstream tasks such as cross-lingual document matching and cross-lingual summary extraction. At present, the works of cross-lingual sentence embedding tasks mainly focus on languages with large-scale corpus. But low-resource languages such as Chinese-Vietnamese are short of sentence-level parallel corpora and clear cross-lingual monitoring signals, and these works on low-resource languages have poor performances. Therefore, we propose a cross-lingual sentence embedding method based on contrastive learning and effectively fine-tune powerful pretraining mode by constructing sentence-level positive and negative samples to avoid the catastrophic forgetting problem of the traditional fine-tuning pre-trained model based only on small-scale aligned positive samples. First, we construct positive and negative examples by taking parallel Chinese Vietnamese sentences as positive examples and non-parallel sentences as negative examples. Second, we construct a siamese network to get contrastive loss by inputting positive and negative samples and fine-tuning our model. The experimental results show that our method can effectively improve the semantic alignment accuracy of cross-lingual sentence embedding in Chinese and Vietnamese contexts.

Multi-modal neural machine translation (MNMT) aims to translate sentences from the source language into the target language with the aid of corresponding images. Unfortunately, there is a considerable modality gap between the semantic-related images and texts in terms of data form and semantic expression. How to fully incorporate visual information into texts to enhance the performance of machine translation is one of the critical issues for MNMT. However, the initial visual and textual features are generally extracted with their modality-specific models; Consequently, there is a considerable representation gap between images and texts. Most previous MNMT works prefer only to adopt the feature-level fusion strategies to learn multi-modal representation, while the modality representation gap is often ignored. To this end, this paper proposes a progressive multi-modal Transformer (ProMul-Trans) with Modality Difference-Aware (MDA) to address the visual-to-textual fusion problem raised in MNMT. We first employ MDA to capture the modality-consistency information by taking visual and textual representations as inputs in each Transformer layer. Then a layer-level progressive fusion (Layer-ProFusion) strategy is adopted to progressively align visual and textual representations layer-by-layer to enhance machine translation performance. Experiment results on the Multi30k dataset are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on English$\rightarrow$German (En$\rightarrow$De), English$\rightarrow$French (En$\rightarrow$Fr) and English$\rightarrow$Czech (En$\rightarrow$Cs) tasks. We release the code athttps://github.com/JunjieYe-MMT/HierProMul-Trans.

This work focuses on the task of Video-based Visible-Infrared Person Re-Identification, a promising technique for achieving 24-hour surveillance systems. Two main issues in this field are modality discrepancy mitigating and spatial–temporal information mining. In this work, we propose a novel method, named Intermediary-guided Bidirectional spatial–temporal Aggregation Network (IBAN), to address both issues at once. Specifically, IBAN is designed to learn modality-irrelevant features by leveraging the anaglyph data of pedestrian images to serve as the intermediary. Furthermore, a bidirectional spatial–temporal aggregation module is introduced to exploit the spatial–temporal information of video data, while mitigating the impact of noisy image frames. Finally, we design an Easy-sample-based loss to guide the final embedding space and further improve the model’s generalization performance. Extensive experiments on Video-based Visible-Infrared benchmarks show that IBAN achieves promising results and outperforms the state-of-the-art ReID methods by a large margin, improving the rank-1/mAP by$1.29\%/3.46\%$at the Infrared to Visible situation, and by$5.04\%/3.27\%$at the Visible to Infrared situation. The source code of the proposed method will be released athttps://github.com/lhf12278/IBAN.

Due to incomplete appearance features, the identity matching of occluded pedestrians under multiple cross-camera views is a long-term challenge. Although existing re-identification (re-ID) solutions of occluded pedestrians have made significant progress, most of them achieve accurate identity matching by extracting pedestrian appearance features from unoccluded areas. However, when a pedestrian is partially blocked by the body of another pedestrian, existing methods cannot accurately determine whether the unoccluded body parts belong to the target pedestrian, which brings great difficulties to pedestrian identity matching. To alleviate this problem, this paper introduces the idea of adversarial attack into occluded person re-ID and proposes an adversarial training framework that can defend against attacks from obstacles to resist the interference of obstacles on pedestrian identity matching. Unlike existing solutions, the proposed framework is not limited to extracting features of unoccluded human body areas to achieve occluded person re-ID, but explores how to make the re-ID model more resistant to obstacles. In the proposed framework, the occluded pedestrian images are regarded as adversarial examples and used to attack model training. If the trained model can defend against this kind of attack, its generalization is significantly improved, and the above-mentioned issues are also effectively solved. Specifically, a single-branch dual-stream collaborative network is designed. With the cooperation of the pre-trained verification guidance network, the model realizes the attack and defense of adversarial samples. This work broadens research horizons in robust model design of occluded person re-ID, and expands the scope of adversarial attacks. Compared with existing solutions, a lot of experimental results confirm that the proposed solution achieves better performance on two occluded re-ID datasets and two partial re-ID datasets.

Change captioning aims to describe the disagreement of image pairs with a linguistic sentence. Compared with single image captioning, change captioning requires not only understanding the fine-grained information of each image, but also determining whether change occurs and further representing the differences of image pairs. Although much progress has been made, it remains a severe challenge of the precise difference representation in the distraction of viewpoint change, especially that of tiny difference. In this paper, we propose a novel Intra- and Inter-representation Interaction Network (I3N) to learn the fine difference representation and be immune to viewpoint change. In the Intra-representation Interaction stage, we design Geometry-Semantic Interaction Refining (GSIR) to explore the positional and semantic interactions of intra-image, which can be a prior knowledge of enduring viewpoint change and reinforce the cognition of semantic change. In the Inter-representation Interaction stage, to endow the model with the capability of pinpointing the latent difference in viewpoint change, Hierarchical Representation Interaction (HRI) models difference from coarse to fine representations through the Semantic Matcher and Change Amplifier module. The proposed approach outperforms the state-of-the-art methods with an encouraging performance on the existing change captioning benchmarks. Our code is available athttps://github.com/yueshengbin/I3N.

Knowledge Graph Embedding (KGE) has been proposed and successfully utilized to knowledge Graph Completion (KGC). But classic KGE paradigm often fail in unseen relation representations. Previous studies mainly utilize the textual descriptions of relations and its neighbor relations to represent unseen relations. In fact, the semantics of a relation can be expressed by three kinds of graphs: factual graph, ontology graph, textual description graph, and they can complement each other. A more common scenario in the real world is that seen and unseen relations appear at the same time. In this setting, the training set (only seen relations) and testing set (both seen and unseen relations) own different distributions. And the train-test inconsistency problem will make KGE methods easiy overfit on seen relations and under-performance on unseen relations. In this paper, we propose decoupling mixture-of-graph experts (DMoG) for unseen relations learning, which could represent the unseen relations in the factual graph by fusing ontology and textual graphs, and decouple fusing space and reasoning space to alleviate overfitting for seen relations. The experiments on two unseen only public datasets and a mixture dataset verify the effectiveness of the proposed method, which improves the state-of-the-art methods by 6.84% in Hits@10 on average.

Multi-modal neural machine translation (MNMT) aims to improve textual level machine translation performance in the presence of text-related images. Most of the previous works on MNMT focus on multi-modal fusion methods with full visual features. However, text and its corresponding image may not match exactly, visual noise is generally inevitable. The irrelevant image regions may mislead or distract the textual attention and cause model performance degradation. This paper proposes a noise-robust multi-modal interactive fusion approach with cross-modal relation-aware mask mechanism for MNMT. A text-image relation-aware attention module is constructed through the cross-modal interaction mask mechanism, and visual features are extracted based on the text-image interaction mask knowledge. Then a noise-robust multi-modal adaptive fusion approach is presented by fusion the relevant visual and textual features for machine translation. We validate our method on the Multi30K dataset. The experimental results show the superiority of our proposed model, and achieve the state-of-the-art scores in all En-De, En-Fr and En-Cs translation tasks.

Opinion target classification of microblog comments is one of the most important tasks for public opinion analysis about an event. Due to the high cost of manual labeling, opinion target classification is generally considered as a weak-supervised task. This article attempts to address the opinion target classification of microblog comments through an event graph convolution network (EventGCN) in a weak-supervised manner. Specifically, we take microblog contents and comments as document nodes, and construct an event graph with three typical relationships of event microblogs, including the co-occurrence relationship of event keywords extracted from microblogs, the reply relationship of comments, and the document similarity. Finally, under the supervision of a small number of labels, both word features and comment features can be represented well to complete the classification. The experimental results on two event microblog datasets show that EventGCN can significantly improve the classification performance compared with other baseline models.

We present a simple, efficient data augmentation approach for boosting Chinese-Vietnamese neural machine translation performance by leveraging the linguistic difference between the two languages. We first define the formalized representation of modifier symmetry, which is one of the most representative linguistic differences between Chinese and Vietnamese. We then propose and test two data augmentation strategies for leveraging the linguistic difference, which can be integrated naturally with different translation models. Results indicate that both strategies can introduce linguistic rules to boost translation accuracy. Tests on Chinese-Vietnamese benchmarks show significant accuracy improvements. To facilitate studies in this domain, we also release an open-source toolkit 1 with flexible implementation for Chinese-Vietnamese linguistic difference tagging.

Due to the importance of practical applications, unsupervised domain adaptation (UDA) person re-identification (re-ID) has attracted increasing attention. However, most of existing methods often lack the multi-view information reasoning and ignore the domain discrepancy of the pedestrian images with the same identity, which constrain the further improvement of recognition performance. So, this paper proposes a triple adversarial learning and multi-view imaginative reasoning network (TAL-MIRN) for UDA person re-ID, which consists of a multi-view imaginative reasoning module (IRM) and a triple adversarial learning module (TALM). IRM makes the classified pedestrian identity features from a single-view image extracted by a feature encoder consistent with the classification results of the aggregated multi-view pedestrian identity features, so the strong multi-view imaginative reasoning ability of the feature encoder is obtained. TALM is composed by the adversarial learning between the camera classifier and feature encoder, adversarial learning of joint distribution alignment, and adversarial learning of the difference between two classifiers used in classification. In particular, the domain-invariant features at camera level are guaranteed by the adversarial learning between the feature extractor and camera classifier. The joint alignment of identity and domain is achieved by the competition between the feature extractor and classifier integrated with identity and domain. The discriminability and robustness of the learned features are enhanced by playing a MinMax game between two different identity classifiers. Furthermore, a simple normalization operation named as cross normalization (CN) is proposed to increase both modeling and generalization capability of the proposed TAL-MIRN across multiple domains. The proposed TAL-MIRN is applied to five benchmark datasets, and the comparative experimental results confirm its superiority over the state-of-the-art methods. The related source codes is available athttps://github.com/lhf12278/TALM-IRM.

Face aging aims to estimate aged facial textures given a certain face image. A number of 2D face-aging methods have been developed, but there have been few studies on 3D face aging, which would be valuable in several real-world applications. The lack of 3D face-aging data has had a significant impact on the development of 3D face aging, but we hypothesized that the large amounts of 2D face-aging data on the internet could be leveraged for 3D aged facial textures. In this paper, we propose a novel 3D aging framework, which we call UV-transformation texture estimation based on generative adversarial networks (UVTE-GAN), to achieve 3D face aging. Specifically, the proposed framework has three parts: 1) a 3D vertex and texture estimator, which accurately estimates the face’s spatial vertices and textures; 2) a texture-aging GAN, which is responsible for aging the estimated texture map via adversarial learning; and 3) a 2D & 3D rendering rebuilder, which recovers 2D & 3D faces using the estimated facial vertex map and aged facial texture map. In addition, we also design a plugin layer that allows us to train the whole model in an end-to-end manner. Experimental results demonstrate the effectiveness of the proposed method in synthesizing visually pleasing 3D aged face pictures, and state-of-the-art performance is achieved on several public datasets.

TV show captioning aims to generate a linguistic sentence based on the video and its associated subtitle. Compared to purely video-based captioning, the subtitle can provide the captioning model with useful semantic clues such as actors’ sentiments and intentions. However, the effective use of subtitle is also very challenging, because it is the pieces of scrappy information and has semantic gap with visual modality. To organize the scrappy information together and yield a powerful omni-representation for all the modalities, an efficient captioning model requires understanding video contents, subtitle semantics, and the relations in between. In this paper, we propose an Intra- and Inter-relation Embedding Transformer (I2Transformer), consisting of an Intra-relation Embedding Block (IAE) and an Inter-relation Embedding Block (IEE) under the framework of a Transformer. First, the IAE captures the intra-relation in each modality via constructing the learnable graphs. Then, IEE learns the cross attention gates, and selects useful information from each modality based on their inter-relations, so as to derive the omni-representation as the input to the Transformer. Experimental results on the public dataset show that the I2Transformer achieves the state-of-the-art performance. We also evaluate the effectiveness of the IAE and IEE on two other relevant tasks of video with text inputs,i.e., TV show retrieval and video-guided machine translation. The encouraging performance further validates that the IAE and IEE blocks have a good generalization ability. The code is available athttps://github.com/tuyunbin/I2Transformer.

Change captioning is to use a natural language sentence to describe the fine-grained disagreement between two similar images.Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the representation of real change.In this paper, we propose a Relation-embedded Representation Reconstruction Network (R 3 Net) to explicitly distinguish the real change from the large amount of clutter and irrelevant changes.Specifically, a relation-embedded module is first devised to explore potential changed objects in the large amount of clutter.Then, based on the semantic similarities of corresponding locations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and further model the difference representation.Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the semantic interaction between change localization and caption generation.Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets 1 .

In a conventional domain adaptation person Re-identification (Re-ID) task, both the training and test images in target domain are collected under the sunny weather. However, in reality, the pedestrians to be retrieved may be obtained under severe weather conditions such as hazy, dusty and snowing, etc. This paper proposes a novel Interference Suppression Model (ISM) to deal with the interference caused by the hazy weather in domain adaptation person Re-ID. A teacher-student model is used in the ISM to distill the interference information at the feature level by reducing the discrepancy between the clear and the hazy intrinsic similarity matrix. Furthermore, in the distribution level, the extra discriminator is introduced to assist the student model make the interference feature distribution more clear. The experimental results show that the proposed method achieves the superior performance on two synthetic datasets than the state-of-the-art methods. The related code will be released online https://github.com/pangjian123/ISM-ReID.

Summary Aspect‐category sentiment classification of microblog comments aims to identify the sentiment polarity of different opinion aspects in microblog comments, which is meaningful for the analysis of public opinion. At present, most of aspect‐category sentiment classification methods need much annotation data, and regard comments as independent samples, without using of the relationship between comments. This article proposes an aspect‐category sentiment classification method based on tensor graph convolutional networks. First, the combination of a comment and its aspect category is regarded as a hybrid node, and the original representation of a hybrid node is encoded by the Bert model. Second, sentiment graph and semantic graph are constructed according to the semantic similarity and sentimental relevance between hybrid nodes, and they are stacked into a tensor. Then two convolution operations, including intra‐graph convolution and inter‐graph convolution, are performed for each layer of graph tensor. In this way, hybrid nodes can learn and merge the heterogeneous information of different graphs. Finally, under the supervision of few labeled comments, the sentiment classification can be completed based on the features of the hybrid nodes. Experimental results on two microblog datasets show that the proposed model can significantly improve the performance of sentiment classification compared with other baseline models.

This article presents a novel extended reality (XR) and deep-learning-based Internet-of-Medical-Things (IoMT) solution for the COVID-19 telemedicine diagnostic, which systematically combines virtual reality/augmented reality (AR) remote surgical plan/rehearse hardware, customized 5G cloud computing and deep learning algorithms to provide real-time COVID-19 treatment scheme clues. Compared to existing perception therapy techniques, our new technique can significantly improve performance and security. The system collected 25 clinic data from the 347 positive and 2270 negative COVID-19 patients in the Red Zone by 5G transmission. After that, a novel auxiliary classifier generative adversarial network-based intelligent prediction algorithm is conducted to train the new COVID-19 prediction model. Furthermore, The Copycat network is employed for the model stealing and attack for the IoMT to improve the security performance. To simplify the user interface and achieve an excellent user experience, we combined the Red Zone’s guiding images with the Green Zone’s view through the AR navigate clue by using 5G. The XR surgical plan/rehearse framework is designed, including all COVID-19 surgical requisite details that were developed with a real-time response guaranteed. The accuracy, recall, F1-score, and area under the ROC curve (AUC) area of our new IoMT were 0.92, 0.98, 0.95, and 0.98, respectively, which outperforms the existing perception techniques with significantly higher accuracy performance. The model stealing also has excellent performance, with the AUC area of 0.90 in Copycat slightly lower than the original model. This study suggests a new framework in the COVID-19 diagnostic integration and opens the new research about the integration of XR and deep learning for IoMT implementation.

The smallest semantic unit of the Burmese language is called the syllable. In the present study, it is intended to propose the first neural joint learning model for Burmese syllable segmentation, word segmentation, and part-of-speech ( POS ) tagging with the BERT. The proposed model alleviates the error propagation problem of the syllable segmentation. More specifically, it extends the neural joint model for Vietnamese word segmentation, POS tagging, and dependency parsing [28] with the pre-training method of the Burmese character, syllable, and word embedding with BiLSTM-CRF-based neural layers. In order to evaluate the performance of the proposed model, experiments are carried out on Burmese benchmark datasets, and we fine-tune the model of multilingual BERT. Obtained results show that the proposed joint model can result in an excellent performance.

Domain invariance and discrimination of learned features as two crucial factors affect the performance of unsupervised domain adaptation (UDA) person re-identification (Re-ID). Person attributes (such as “backpack”, “boots”, “handbag”, etc) remaining unchanged across multiple domains have been used as mid-level visual-semantic information in UDA person Re-ID. As two main challenges, both misalignment of attribute-related regions across multiple images and domain shift between source and target domains affect the learning of domain-invariant features (DIF). To address the above two challenges, this article proposes to take advantage of the stability of person attributes and the complementarity of person attributes and the corresponding low-level visual features to guide the learning of discriminative DIF. Specifically, the proposed solution contains the generation of latent attribute-correlated visual features (GLAVF), DIF learning under the guidance of person attributes, and the alignment of person attributes corresponding to the local regions of pedestrian images. Due to the gap between person attributes and visual features, person attributes are first converted into latent attribute-correlated visual features (LAVF) without any specific domain information in GLAVF, and then LAVF are used as the substitutions of person attributes to guide the learning of DIF. To enhance the discrimination of learned features, the proposed solution mainly explores the alignment between person attributes and corresponding local regions, and the alignment of the same person attributes across multiple pedestrian images. A fully connected layer is used to achieve the above two types of alignment in the proposed framework, which reduces the adverse impacts of inference information and ensures the semantic consistency between person attributes and corresponding local regions across multiple pedestrian images. The effectiveness of the proposed solution is confirmed on four existing datasets by comparative experiments.

A novel intelligent navigation technique for accurate image-guided COVID-19 lung biopsy is addressed, which systematically combines augmented reality (AR), customized haptic-enabled surgical tools, and deep neural network to achieve customized surgical navigation. Clinic data from 341 COVID-19 positive patients, with 1598 negative control group, have collected for the model synergy and evaluation. Biomechanics force data from the experiment are applied a WPD-CNN-LSTM (WCL) to learn a new patient-specific COVID-19 surgical model, and the ResNet was employed for the intraoperative force classification. To boost the user immersion and promote the user experience, intro-operational guiding images have combined with the haptic-AR navigational view. Furthermore, a 3-D user interface (3DUI), including all requisite surgical details, was developed with a real-time response guaranteed. Twenty-four thoracic surgeons were invited to the objective and subjective experiments for performance evaluation. The root-mean-square error results of our proposed WCL model is 0.0128, and the classification accuracy is 97%, which demonstrated that the innovative AR with deep learning (DL) intelligent model outperforms the existing perception navigation techniques with significantly higher performance. This article shows a novel framework in the interventional surgical integration for COVID-19 and opens the new research about the integration of AR, haptic rendering, and deep learning for surgical navigation.

Infrared and visible image fusion has gained ever-increasing attention in recent years due to its great significance in a variety of vision-based applications. However, existing fusion methods suffer from some limitations in terms of the spatial resolutions of both input source images and output fused image, which prevents their practical usage to a great extent. In this paper, we propose a meta learning-based deep framework for the fusion of infrared and visible images. Unlike most existing methods, the proposed framework can accept the source images of different resolutions and generate the fused image of arbitrary resolution just with a single learned model. In the proposed framework, the features of each source image are first extracted by a convolutional network and upscaled by a meta-upscale module with an arbitrary appropriate factor according to practical requirements. Then, a dual attention mechanism-based feature fusion module is developed to combine features from different source images. Finally, a residual compensation module, which can be iteratively adopted in the proposed framework, is designed to enhance the capability of our method in detail extraction. In addition, the loss function is formulated in a multi-task learning manner via simultaneous fusion and super-resolution, aiming to improve the effect of feature learning. And, a new contrast loss inspired by a perceptual contrast enhancement approach is proposed to further improve the contrast of the fused image. Extensive experiments on widely-used fusion datasets demonstrate the effectiveness and superiority of the proposed method. The code of the proposed method is publicly available at https://github.com/yuliu316316/MetaLearning-Fusion.

Person reidentification (Re-ID) benefits greatly from the accurate annotations of existing data sets (e.g., CUHK03 and Market-1501), which are quite expensive because each image in these data sets has to be assigned with a proper label. In this work, we ease the annotation of Re-ID by replacing the accurate annotation with inaccurate annotation, i.e., we group the images into bags in terms of time and assign a bag-level label for each bag. This greatly reduces the annotation effort and leads to the creation of a large-scale Re-ID benchmark called SYSU- 30k . The new benchmark contains 30k individuals, which is about 20 times larger than CUHK03 (1.3k individuals) and Market-1501 (1.5k individuals), and 30 times larger than ImageNet (1k categories). It sums up to 29606918 images. Learning a Re-ID model with bag-level annotation is called the weakly supervised Re-ID problem. To solve this problem, we introduce a differentiable graphical model to capture the dependencies from all images in a bag and generate a reliable pseudolabel for each person's image. The pseudolabel is further used to supervise the learning of the Re-ID model. Compared with the fully supervised Re-ID models, our method achieves state-of-the-art performance on SYSU- 30k and other data sets. The code, data set, and pretrained model will be available at https://github.com/wanggrun/SYSU-30k.

In Neural Machine Translation (NMT), due to the limitations of the vocabulary, unknown words cannot be translated properly, which brings suboptimal performance of the translation system. For resource-scarce NMT that have small-scale training corpus, the effect is amplified. The traditional approach of amplifying the scale of the corpus is not applicable, because the parallel corpus is difficult to obtain in a resource-scarce setting; however, it is easy to obtain and utilize external knowledge, bilingual lexicon, and other resources. Therefore, we propose classification lexicon approach for processing unknown words in the Chinese-Vietnamese NMT task. Specifically, three types of unknown Chinese-Vietnamese words are classified and their corresponding classification lexicon are constructed by word alignment, Wikipedia extraction, and rule-based methods, respectively. After translation, the unknown words are restored by lexicon for post-processing. Experiment results on Chinese-Vietnamese, English-Vietnamese, and Mongolian-Chinese translations show that our approach significantly improves the accuracy and the performance of NMT especially in a resource-scarce setting.

How to utilize information sufficiently is a key problem in neural machine translation (NMT), which is effectively improved in rich-resource NMT by leveraging large-scale bilingual sentence pairs. However, for low-resource NMT, lack of bilingual sentence pairs results in poor translation performance; therefore, taking full advantage of global information in the encoding-decoding process is effective for low-resource NMT. In this article, we propose a novel reread-feedback NMT architecture (RFNMT) for using global information. Our architecture builds upon the improved sequence-to-sequence neural network and consists of a double-deck attention-based encoder-decoder framework. In our proposed architecture, the information generated by the first-pass encoding and decoding process flows to the second-pass encoding process for more sufficient parameters initialization and information use. Specifically, we first propose a “reread” mechanism to transfer the outputs of the first-pass encoder to the second-pass encoder, and then the output is used for the initialization of the second-pass encoder. Second, we propose a “feedback” mechanism that transfers the first-pass decoder’s outputs to a second-pass encoder via an important weight model and an improved gated recurrent unit (GRU). Experiments on multiple datasets show that our approach achieves significant improvements over state-of-the-art NMT systems, especially in low-resource settings.

Due to the domain shift between source dataset and target dataset, most of the existing person re-identification (PRID) algorithms trained by a supervised learning framework often fail to be well generalized to another domain. To address this challenge, we propose a self-supervised learning algorithm based on attribute-identity embedding, which can incrementally optimize the model by selecting unlabeled samples from target domain. Thus the gap between source domain and target domain is bridged. Specifically, we first develop an attribute-identity joint prediction dictionary learning model for simultaneously learning a latent attribute space, a semantic attribute dictionary and an identifier. In our method, the predicted attribute from latent attribute space is used as a bridge to establish a preliminary link between different domains so as to predict the label of the target data sample. Second, to exploit the latent label contained in the predicted samples, we propose a prediction-training cycle self-supervised learning to tune the model variables to make them more adaptive in the target domain. Finally, the similarity measurement of pedestrians is achieved by combining the attribute space with latent identity space. The experiments show that the developed method outperforms some state-of-the-art supervised PRID methods and unsupervised PRID algorithms.

Despite advances in person re-identification (re-ID), it is still far from meeting the needs of real-world applications due to tremendous visual ambiguity in person appearance across cameras. To overcome this problem, we propose a person re-ID method by decomposing a pedestrian's appearance feature into different components. This method assumes that each pedestrian image is composed of person-shared components that reflect the similarities of different pedestrians and person-specific components that reflect unique identity information. Based on this assumption, we propose to reduce the ambiguity in visual features by removing person-shared components from pedestrian visual features. To this end, we develop a framework for learning a pair of commonality and specificity dictionaries, while introducing a distance constraint to force the particularities of the same person over the specificity dictionary to have the same coding coefficients and the coding coefficients of different pedestrians to have weak correlation. Furthermore, considering the similarity of the commonality dictionary and the sparsity of the specificity dictionary, low-rank and sparse regularization terms are introduced into the dictionary learning framework to improve their representation ability and discriminative ability. Extensive experimental results show that the proposed algorithm outperforms or is competitive with the state-of-the-art methods.

Recently, the Transformer machine translation system has shown strong results by stacking attention layers on both the source and target-language sides. But the inference of this model is slow due to the heavy use of dot-product attention in auto-regressive decoding. In this paper we speed up Transformer via a fast and lightweight attention model. More specifically, we share attention weights in adjacent layers and enable the efficient re-use of hidden states in a vertical manner. Moreover, the sharing policy can be jointly learned with the MT model. We test our approach on ten WMT and NIST OpenMT tasks. Experimental results show that it yields an average of 1.3X speed-up (with almost no decrease in BLEU) on top of a state-of-the-art implementation that has already adopted a cache for fast inference. Also, our approach obtains a 1.8X speed-up when it works with the AAN model. This is even 16 times faster than the baseline with no use of the attention cache.

Ultrasound (US) image segmentation plays a very important role in diagnostic imaging. In order to extract special tissues from images, this study proposes a new method for segmentation of US images. The proposed method uses multilevel threshold segmentation which is based on Otsu and differential search algorithm. Testing in simulation US images shows that the proposed algorithm has a better result of segmentation than the three existing methods, including region growing, the active contour model and k‐means technique. The proposed method gets the highest F m values and the smallest area errors in experiments. Vivo US images are also tested by the proposed method and it achieves a good segmentation result.

Because of the scarcity of bilingual corpora, current Chinese--Vietnamese machine translation is far from satisfactory. Considering the differences between Chinese and Vietnamese, we investigate whether linguistic differences can be used to supervise machine translation and propose a method of syntax-based Chinese--Vietnamese tree-to-tree statistical machine translation with bilingual features. Analyzing the syntax differences between Chinese and Vietnamese, we define some linguistic difference-based rules, such as attributive position, time adverbial position, and locative adverbial position, and create rewards for similar rules. These rewards are integrated into the extraction of tree-to-tree translation rules, and we optimize the pruning of the search space during the decoding phase. The experiments on Chinese--Vietnamese bilingual sentence translation show that the proposed method performs better than several compared methods. Further, the results show that syntactic difference features, with search pruning, can improve the accuracy of machine translation without degrading the efficiency.

Burmese is an isolated language, in which the syllable is the smallest unit. Syllable segmentation methods based on matching lead to performance subject to the syllable segmentation effect. This article proposes a word segmentation method with fusion conditions of double syllable features. It combines word segmentation and segmentation of syllables into one process, thus reducing the impact of errors on the syllable segmentation of Burmese. In the first layer of the conditional random fields (CRF) model, Burmese characters as atomic features are integrated into the Burma section of the Barkis Speech Paradigm (Backus normal form) features to realize the Burma syllable sequence tags. In the second layer of the CRFs model, with the syllable marked as input, it realizes the sequence markers through building a feature template with syllables as atomic features. The experimental results show that the proposed method has a better effect compared with the method based on the matching of syllables.

Crowdsourcing labeling systems provide an efficient way to generate multiple inaccurate labels for given observations. If the competence level or the "reputation," which can be explained as the probabilities of annotating the right label, for each crowdsourcing annotators is equal and biased to annotate the right label, majority voting (MV) is the optimal decision rule for merging the multiple labels into a single reliable one. However, in practice, the competence levels of annotators employed by the crowdsourcing labeling systems are often diverse very much. In these cases, weighted MV is more preferred. The weights should be determined by the competence levels. However, since the annotators are anonymous and the ground-truth labels are usually unknown, it is hard to compute the competence levels of the annotators directly. In this paper, we propose to learn the weights for weighted MV by exploiting the expertise of annotators. Specifically, we model the domain knowledge of different annotators with different distributions and treat the crowdsourcing problem as a domain adaptation problem. The annotators provide labels to the source domains and the target domain is assumed to be associated with the ground-truth labels. The weights are obtained by matching the source domains with the target domain. Although the target-domain labels are unknown, we prove that they could be estimated under mild conditions. Both theoretical and empirical analyses verify the effectiveness of the proposed method. Large performance gains are shown for specific data sets.

In the process of recommending review experts to projects, in order to effectively make use of the relevance among topics and the relationship among experts, a new method is proposed for review expert recommendation using topic relevance and expert relationship. In this method, firstly, the relevance among topics and the relationships among experts are used to respectively construct the Markov network of topics and the Markov network of experts. Next, the maximum topic clique is extracted from the topic Markov network and the maximum expert clique is extracted from the expert Markov network; then, with the information of the two maximum cliques, the relevance between experts and projects is calculated. After that, according to the descending order of the relevant degree, the candidates are ranked. Finally, the experts, who are the top N to projects, are recommended. The experiments on five domain datasets are made and the results show that the proposed method can improve the effect of review expert recommendation, and the F-value increases by an average of 5% than without considering the relevance among topics and the relationship among experts.

Person re-identification aims to identify the same pedestrians across different camera views at different locations. This important yet difficult intelligent video analysis problem remains a vigorous area of research due to demands for performance improvements. Person re-identification involves two main steps: feature representation and metric learning. Handcrafted features, such as color and texture histograms, are frequently used for person re-identification, but most handcrafted features are limited by not being directly applicable to practical problems. Deep learning methods have obtained the state-of-the-art performance in a wide variety of applications, including image annotation, face recognition, and speech recognition. However, deep learning features are heavily dependent on large-scale labeling of samples. In this paper, by utilizing the Cross-view Quadratic Discriminant Analysis (XQDA) metric learning, we propose a novel scheme called deep multi-view feature learning (DMVFL), which exploits the collaboration between handcrafted and deep learning features in a simple but effective way. Furthermore, we prove that the XQDA is a robust algorithm. Extensive experiments on two challenging person re-identification data sets (VIPeR and GRID) demonstrate that DMVFL improves on current state-of-the-art methods.

Millions of the smart meters, as essential components, are being deployed ubiquitously in the next generation power system. However, the public privacy concerns over the users' power consumption leakage raise, since the smart meters' unintermittent readings contain customers' behavior patterns. To alleviate this problem, the state-of-the-art techniques are common to use a rechargeable battery to hide the actual power consumption. Unfortunately, none of the existing works completely provide a rigorous privacy protection with reasonable cost under real-world battery settings, i.e., achieving the well-known differential privacy guarantee economically using batteries with limited charge/discharge rate and capacity. To attain this goal, this paper proposes a differentially private meter reading report mechanism. The main idea is to first narrow down the domain of the noise distribution parameter, in order to decrease the possibility of violating the battery limits. It also combines a multi-armed bandit algorithm to further reduce the cost as much as possible. In addition, a novel switch mechanism is proposed to prevent the meter from reporting its reading when the battery limitations might be violated. The theoretical analysis provides a formal proof of the privacy guarantee of the proposed scheme. Besides, experimental results show that the privacy protection of the proposed scheme is at least nine times stronger than that of the existing solutions with acceptable extra cost.

The popular performance profiles and data profiles for benchmarking deterministic optimization algorithms are extended to benchmark stochastic algorithms for global optimization problems. A general confidence interval is employed to replace the significance test, which is popular in traditional benchmarking methods but suffering more and more criticisms. Through computing confidence bounds of the general confidence interval and visualizing them with performance profiles and (or) data profiles, our benchmarking method can be used to compare stochastic optimization algorithms by graphs. Compared with traditional benchmarking methods, our method is synthetic statistically and therefore is suitable for large sets of benchmark problems. Compared with some sample-mean-based benchmarking methods, e.g., the method adopted in black-box-optimization-benchmarking workshop/competition, our method considers not only sample means but also sample variances. The most important property of our method is that it is a distribution-free method, i.e., it does not depend on any distribution assumption of the population. This makes it a promising benchmarking method for stochastic optimization algorithms. Some examples are provided to illustrate how to use our method to compare stochastic optimization algorithms.

Seeking multiple optima simultaneously, which multimodal optimization aims at, has attracted increasing attention but remains challenging. Taking advantage of ant colony optimization (ACO) algorithms in preserving high diversity, this paper intends to extend ACO algorithms to deal with multimodal optimization. First, combined with current niching methods, an adaptive multimodal continuous ACO algorithm is introduced. In this algorithm, an adaptive parameter adjustment is developed, which takes the difference among niches into consideration. Second, to accelerate convergence, a differential evolution mutation operator is alternatively utilized to build base vectors for ants to construct new solutions. Then, to enhance the exploitation, a local search scheme based on Gaussian distribution is self-adaptively performed around the seeds of niches. Together, the proposed algorithm affords a good balance between exploration and exploitation. Extensive experiments on 20 widely used benchmark multimodal functions are conducted to investigate the influence of each algorithmic component and results are compared with several state-of-the-art multimodal algorithms and winners of competitions on multimodal optimization. These comparisons demonstrate the competitive efficiency and effectiveness of the proposed algorithm, especially in dealing with complex problems with high numbers of local optima.

Existing differential evolution (DE) algorithms often face two challenges. The first is that the optimization performance is significantly affected by the ad hoc configurations of operators and parameters for different problems. The second is the long runtime for real-world problems whose fitness evaluations are often expensive. Aiming at solving these two problems, this paper develops a novel double-layered heterogeneous DE algorithm and realizes it in cloud computing distributed environment. In the first layer, different populations with various parameters and/or operators run concurrently and adaptively migrate to deliver robust solutions by making the best use of performance differences among multiple populations. In the second layer, a set of cloud virtual machines run in parallel to evaluate fitness of corresponding populations, reducing computational costs as offered by cloud. Experimental results on a set of benchmark problems with different search requirements and a case study with expensive design evaluations have shown that the proposed algorithm offers generally improved performance and reduced computational time, compared with not only conventional and a number of state-of-the-art DE variants, but also a number of other distributed DE and high-performing evolutionary algorithms. The speedup is significant especially on expensive problems, offering high potential in a broad range of real-world applications.

Leveraging expert attributes and their attribute-associated features, we propose an expert disambiguation method based on experts' attributed graph clustering model. In the method, firstly, the attributes and their co-occurrences are identified and extracted. Secondly, based on graph theory, the augmented expert attribute nodes are established and their correlations are connected to form a network of augmented expert attribute graph, which combines experts' attribute consistency and graph' structural consistency. Finally, we establish an entropy model to measure attribute information and structural information, and by minimizing the entropy of super nodes and super edges, we obtain the clustering partition for multiple expert nodes. The experimental results on real-world datasets show that the proposed method significantly outperforms the state-of-art spectral clustering method and the semi-supervised graph clustering method for the accuracy of disambiguation.

Product reviews have become an important resource for customers before they make purchase decisions. However, the abundance of reviews makes it difficult for customers to digest them and make informed choices. In our study, we aim to help customers who want to quickly capture the main idea of a lengthy product review before they read the details. In contrast with existing work on review analysis and document summarization, we aim to retrieve a set of real-world user questions to summarize a review. In this way, users would know what questions a given review can address and they may further read the review only if they have similar questions about the product. Specifically, we design a two-stage approach which consists of question retrieval and question diversification. We first propose probabilistic retrieval models to locate candidate questions that are relevant to a review. We then design a set function to re-rank the questions with the goal of rewarding diversity in the final question set. The set function satisfies submodularity and monotonicity, which results in an efficient greedy algorithm of submodular optimization. Evaluation on product reviews from two categories shows that the proposed approach is effective for discovering meaningful questions that are representative for individual reviews.

Person re-identification aims to match the images of pedestrians across different camera views from different locations. This is a challenging intelligent video surveillance problem that remains an active area of research due to the need for performance improvement. Person re-identification involves two main steps: feature representation and metric learning. Although the keep it simple and straightforward (KISS) metric learning method for discriminative distance metric learning has been shown to be effective for the person re-identification, the estimation of the inverse of a covariance matrix is unstable and indeed may not exist when the training set is small, resulting in poor performance. Here, we present dual-regularized KISS (DR-KISS) metric learning. By regularizing the two covariance matrices, DR-KISS improves on KISS by reducing overestimation of large eigenvalues of the two estimated covariance matrices and, in doing so, guarantees that the covariance matrix is irreversible. Furthermore, we provide theoretical analyses for supporting the motivations. Specifically, we first prove why the regularization is necessary. Then, we prove that the proposed method is robust for generalization. We conduct extensive experiments on three challenging person re-identification datasets, VIPeR, GRID, and CUHK 01, and show that DR-KISS achieves new state-of-the-art performance.

Named entity equivalent has been playing a significant role in the processing of cross-language information. However limited by the corpora resource, few in-depth studies have been made on the extraction of the bilingual Chinese-Khmer named entity equivalents. On account of this, this paper proposes a Wikipedia-based approach, utilizes the internal web links in Wikipedia and computes the feature similarity to extract the bilingual Chinese-Khmer named entity equivalents. The experimental result shows that good effect has been achieved when the entity equivalents are acquired through the internal web links in Wikipedia with F value up to 90.67%. Also it shows that the result is quite favorable when the bilingual Chinese-Khmer named entity equivalents are acquired through the computation of feature similarity, turning out that the method proposed in this paper is able to give better effect.

This paper presents a novel opinion mining research problem, which is called Contrastive Opinion Modeling (COM). Given any query topic and a set of text collections from multiple perspectives, the task of COM is to present the opinions of the individual perspectives on the topic, and furthermore to quantify their difference. This general problem subsumes many interesting applications, including opinion summarization and forecasting, government intelligence and cross-cultural studies. We propose a novel unsupervised topic model for contrastive opinion modeling. It simulates the generative process of how opinion words occur in the documents of different collections. The ad hoc opinion search process can be efficiently accomplished based on the learned parameters in the model. The difference of perspectives can be quantified in a principled way by the Jensen-Shannon divergence among the individual topic-opinion distributions. An extensive set of experiments have been conducted to evaluate the proposed model on two datasets in the political domain: 1) statement records of U.S. senators; 2) world news reports from three representative media in U.S., China and India, respectively. The experimental results with both qualitative and quantitative analysis have shown the effectiveness of the proposed model.