Multi-Hop Question Answering (MHQA) is crucial for evaluating the model's capability to integrate information from diverse sources.However, creating extensive and high-quality MHQA datasets is challenging: (i) manual annotation is expensive, and (ii) current synthesis methods often produce simplistic questions or require extensive manual guidance.This paper introduces HopWeaver, the first cross-document framework synthesizing authentic multi-hop questions without human intervention.HopWeaver synthesizes bridge and comparison questions through an innovative pipeline that identifies complementary documents and constructs authentic reasoning paths to ensure true multi-hop reasoning.We further present a comprehensive system for evaluating the synthesized multi-hop questions.Empirical evaluations demonstrate that the synthesized questions achieve comparable or superior quality to human-annotated datasets at a lower cost.Our framework provides a valuable tool for the research community: it can automatically generate challenging benchmarks from any raw corpus, which opens new avenues for both evaluation and targeted training to improve the reasoning capabilities of advanced question answering models, especially in domains with scarce resources.The code for HopWeaver is publicly available 1 .

Clinical trials are essential for discovering new treatments and advancing medical knowledge. However, the high uncertainty of carrying out clinical trials often ends with ineffective results. Therefore, the accurate prediction of clinical trial outcomes has become a significant challenge. Numerous publicly accessible clinical trial reports have been discovered to be beneficial in alleviating this challenge but lack necessary annotations to be formal datasets for deep model training. To address the issue, this paper proposes to construct a new clinical trial dataset by extracting publicly available clinical trial reports from ClinicalTrials.gov and PubMed. In addition, a new two-stage method is proposed for the prediction of clinical trial outcomes across all trial phases. Specifically, our method first employs a prompt template combined with each clinical trial report to prompt a large language model to generate a concise summarization text containing essential information related to the clinical trial outcomes. Subsequently, this summarization text is utilized to train a classifier to predict the outcomes. Extensive experiments were conducted on the dataset, and our method was compared with several state-of-the-art classification models. The results showed that our method achieved the best performance in predicting clinical trial outcomes, especially using small amounts of training data under a data imbalance difficulty.

With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

The conditional coding paradigm is widely used in learned video compression, which shows superior performance in capturing redundancies within a large context space. However, existing Conditional coding-based Learned Video Compression (C-LVC) methods ignore that the predicted motion vectors usually contain large uncertainty due to complex motions, occlusions, etc., which consequently decrease the accuracy of the generated temporal contexts. In addition, existing C-LVC methods have a weak ability to mine diverse dependencies within the context space, which are closely related to the coding efficiency. To address these issues, an efficient temporal redundancy mining method is proposed to improve the coding efficiency of C-LVC in this paper. To generate accurate temporal contexts, a Long Short-Term Motion Aggregation (LSTMA) model is proposed, in which an LSTMA-based motion estimation module is developed to capture both current and aggregated long short-term motion information to reduce the uncertainty of predicted motion vectors. Based on the dual motion information, an LSTMA-based temporal context mining module is developed to exploit the aggregated long short-term motion information and increase the accuracy of the generated temporal contexts. In order to fully eliminate spatial-temporal redundancies in a video, a Global-Local Information Decorrelation Module (GLIDM)-based context codec is proposed, in which the GLIDM is designed based on the visual state space block (namely vmamba), the residual block, and the squeeze-and-excitation block to effectively capture long-range, short-range spatial-temporal dependencies and channel-wise dependencies. Experimental results demonstrate that our proposed method can effectively improve the coding performance of C-LVC, and outperforms other state-of-the-art LVC methods.

Searching by image is popular yet still challenging in e-commerce due to the extensive interference arising from (i) data variations (e.g., background, pose, visual angle, brightness) of real-world captured images and (ii) similar images in the query dataset. This article studies a practically meaningful problem of beauty product retrieval (BPR) by neural networks. We broadly extract different types of image features and raise an intriguing question that whether these features are beneficial to (i) suppress data variations of real-world captured images and (ii) distinguish one image from others which look very similar but are intrinsically different beauty products in the dataset, therefore leading to an enhanced capability of BPR. To answer it, we present a novel v ariable-attention neural network to understand the combination of m ultiple features (termed VM-Net) of beauty product images. Considering that there are few publicly released training datasets for BPR, we establish a new dataset with more than one million images classified into more than 20K categories to improve both the generalization and anti-interference abilities of VM-Net and other methods. We verify the performance of VM-Net and its competitors on the benchmark dataset Perfect-500K, where VM-Net shows clear improvements over the competitors in terms of \(MAP@7\) . The source code and dataset will be released upon publication.

In recent years, the decreasing production costs and rapid technological advancements of Unmanned Aerial Vehicles (UAVs) have led to their widespread adoption in critical domains such as agricultural plant protection, power line inspection, emergency rescue, and homeland security. Despite their growing utility, UAVs face significant challenges when performing tasks that require real-time object detection. The images captured by their onboard vision systems often feature densely clustered, small-sized targets, which strain the capabilities of existing detection algorithms. For instance, the YOLO11n model struggles with substantial feature loss due to its downsampling operation, limiting its effectiveness in detecting small objects in complex scenarios. To address these limitations, this paper introduces P2AW-YOLO11n, an enhanced model derived from YOLO11n, specifically designed to improve small object detection. The proposed model incorporates three key innovations. First, Adaptive Feature Pyramid Network (AFPN) is integrated into the neck of the model to enable more flexible feature fusion through adaptive mechanisms such as dynamic weight learning and cross-scale interactions, effectively mitigating feature dilution. Second, the P2 detection head utilizes large-scale$160 \times 160$feature maps to capture finer target details, significantly enhancing the model's ability to detect small objects. Third, a refined version of the Wise-IoU (WIoU) loss function is introduced to address challenges posed by low-quality samples, thereby improving the model's generalization capabilities. Extensive comparative and ablation experiments conducted on the VisDrone dataset demonstrate the efficacy of the proposed algorithm. P2AWYOLO11n achieves a 7.5% improvement in Precision, a 5.5% increase in Recall, and a 6.9% boost in$\text{mAP} @ 50$compared to the baseline YOLO11n model. These results underscore the robustness and effectiveness of P2AW-YOLO11n in overcoming the challenges associated with small object detection, paving the way for enhanced UAV applications in complex environments.

Task-oriented dialogue systems have recently gained increasing attention due to their capability of using natural language to fulfill specific user demands, such as restaurant reservation and hotel booking. Recent works directly model task-oriented dialogue response as a text generation task. However, these methods, utilizing generated response tokens as an attention query to obtain the vanilla attention distribution over an entire knowledge base, frequently lead to an entity inconsistency in final response generation. To tackle this problem, we propose a novel attention mechanism called span attention and a novel model named Span Attention GEnerator (SAGE). The span attention computes an attention score between a query vector and each knowledge record vector instead of computing a vanilla attention score among word vectors, which consisted of dialogue context and knowledge base. For effective training, we propose an attention constraint strategy that utilizes the entities appearing in response as pseudo-labels to supervise the training of the span attention. Experiments based on three publicly accessible datasets demonstrate that our model, utilizing the proposed mechanism, outperforms the state-of-the-art models with improvements of 8.94%, 2.29%, and 10.22% respectively in Entity F1.

Mobile crowdsensing (MCS) networks enable largescale data collection by leveraging the ubiquity of mobile devices. However, frequent sensing and data transmission can lead to significant resource consumption. To mitigate this issue, edge caching has been proposed as a solution for storing recently collected data. Nonetheless, this approach may compromise data freshness. In this paper, we investigate the trade-off between re-using cached task results and re-sensing tasks in cacheenabled MCS networks, aiming to minimize system latency while maintaining information freshness. To this end, we formulate a weighted delay and age of information (AoI) minimization problem, jointly optimizing sensing decisions, user selection, channel selection, task allocation, and caching strategies. The problem is a mixed-integer non-convex programming problem which is intractable. Therefore, we decompose the long-term problem into sequential one-shot sub-problems and design a framework that optimizes system latency, task sensing decision, and caching strategy subproblems. When one task is re-sensing, the one-shot problem simplifies to the system latency minimization problem, which can be solved optimally. The task sensing decision is then made by comparing the system latency and AoI. Additionally, a Bayesian update strategy is developed to manage the cached task results. Building upon this framework, we propose a lightweight and time-efficient algorithm that makes real-time decisions for the long-term optimization problem. Extensive simulation results validate the effectiveness of our approach.

In this paper, we propose a Modality-aware Diffusion Distillation Network (MDDN), consisting of Diffusion-based Modality Imputation (DMI) and Margin-aware Distillation (MAD) modules, for multimodal sentiment analysis under uncertain missing modalities. More specifically, the DMI module incrementally adds Gaussian noise into the modality-specific distribution space of the available data and recovers missing modalities while adhering to their original distributions, enabling effective imputation of missing data for the network. Furthermore, the MAD module introduces the classification uncertainty of generated samples and available data to reweigh their contributions to the loss function. Consequently, the missing modalities can be reconstructed by MDDN, and be fused dynamically with the existing ones for sentiment analysis. Extensive experiments on the MOSI and MOSEI datasets demonstrate the effectiveness and robustness of the proposed network for dealing with missing modalities in sentiment analysis.

Multimodal Aspect-based Sentiment Analysis aims at identifying specific aspects like celebrities, locations, and organizations, along with their corresponding sentiments. Typically, existing methods comprehend each aspect within images and their relationship without external knowledge. To this end, we propose an Aspect-Attentioned Prompting Framework (AAPF) with Vision-language Prompting (VP) and Attention Aggregation (AA) modules to leverage the fine-grained visual semantics corresponding to the conditional sentences. More specifically, the VP constructs aspect-oriented questions from the perspectives of objects, relations, and sentence-oriented questions, leveraging chain-of-thought prompting to guide the reasoning process. Given the questions, sentences, and images, a visual language model is exploited to perceive the aspect-sensitive intricate semantics, which are further filtered by CLIP scores and condensed into an aspect-relevant summarization. Furthermore, the AA module conducts complementary cross-attention on the summarizations and the conditional sentences to model their interactions by achieving aspect-attentioned features, which can be used to make sentiment predictions. Experiments conducted on two public datasets indicate that our approach achieves superior performance against the state-of-the-art methods.

Recently, measurable quantitative information extraction from unstructured texts has attracted increasing attention in various fields of industry. However, due to the issues of error propagation and insufficient deep interactions between entities and relations, the accurate extraction of measurable quantitative information remains as a challenging task. To address these issues, this paper proposes a joint model based on table filling (JMTF) for measurable quantitative information extraction task. The core of this model introduces a co-attention mechanism to achieve bidirectional interactions between recognition and association subtasks, thereby avoiding problems such as feature confusion or insufficient interaction. Additionally, the model employs an optimized BERT-based encoder (TEncoder) for text encoding. TEncoder improves ability of the model to capture long-range contextual information of text by incorporating direction-awareness, distance-awareness, and unscaled attention. To further improve performance of the model in Chinese text, TEncoder also integrates the inherent features of Chinese characters, such as pinyin and glyphs, which help handle the ambiguity of polysemous words and homophones. The experiments evaluate the JMTF model on a standardized quantitative information dataset of 3106 Chinese text sentences. The results show that our JMTF achieves F1 values of 87.01% and 85.95% for MQI recognition and association, respectively, outperforming the best baseline at 86.42% and 84.65%, demonstrating its advantages in measurable quantitative information extraction.

Evaluating the visual quality of autostereoscopic 3D displays is crucial for quantifying their stereoscopic viewing experience and optimizing display performance. Existing quality evaluation methods primarily predict the visual quality of autostereoscopic 3D displays by indirectly learning display parameter information from image content. However, these methods fail to explicitly model the relationship between display parameters and visual quality, thereby limiting their prediction accuracy. To address this problem, a Multimodal Parameter Perception Network (MPPNet)-based visual quality assessment method is proposed in this paper, which treats display parameters as textual modalities to explicitly establish their relationship with visual quality. To effectively understand the semantic information of display parameter texts, a Contrastive Language-Image Pretraining (CLIP)-based adaptive text encoder is proposed to generate robust semantic representations by capturing both general and domain-specific semantic embeddings. In parallel, a hierarchical vision encoder is adopted to extract visual representations from display images, which simulates the human binocular perception by capturing multi-level visual features from the left and right views. To achieve comprehensive cross-modal interaction, a mamba-based cross-modal fusion module is proposed to fuse textual and visual representations of display parameters by capturing both shallow and deep correlations. Extensive experimental results demonstrate that the proposed MPPNet achieves state-of-the-art performance in evaluating the visual quality of autostereoscopic 3D displays.

The remarkable capability of generative methods to synthesize public-domain speeches that are difficult to differentiate from natural ones has raised concerns about the proliferation of disinformation. To this end, we contribute a fake speech detection dataset, denoted as the SynSpeech dataset, which includes fully fake and partially fake speeches in multiple domains covering both GANs-based and diffusion-based generative methods, serving as a prerequisite to support fake speech detectors. For the partially fake speeches, we utilize a large language model to replace interest segments like individual names, locations, adjectives, and verbs. Additionally, these segments will be substituted with fabricated clips synthesized by generative methods to create sentences with completely opposite meanings as partially fake speech. More specifically, a total of 164,924 fully fake speeches from six distinct generators, 44,070 real speeches from 108 individuals, and 99,496 partially fake speeches have been annotated as three kinds of labels, i.e., fully fake, real, and partially fake, respectively. In the experiments, we investigate quite a few fake speech detection models and customize them for two fake speech detection tasks for comparisons.

Diffusion models have achieved significant progress in image generation and the pre-trained Stable Diffusion (SD) models are helpful for image deblurring by providing clear image priors. However, directly using a blurry image or a pre-deblurred one as a conditional control for SD will either hinder accurate structure extraction or make the results overly dependent on the deblurring network. In this work, we propose a Latent Kernel Prediction Network (LKPN) to achieve robust real-world image deblurring. Specifically, we co-train the LKPN in the latent space with conditional diffusion. The LKPN learns a spatially variant kernel to guide the restoration of sharp images in the latent space. By applying element-wise adaptive convolution (EAC), the learned kernel is utilized to adaptively process the blurry feature, effectively preserving the information of the blurry input. This process thereby more effectively guides the generative process of SD, enhancing both the deblurring efficacy and the quality of detail reconstruction. Moreover, the results at each diffusion step are utilized to iteratively estimate the kernels in LKPN to better restore the sharp latent by EAC in the subsequent step. This iterative refinement enhances the accuracy and robustness of the deblurring process. Extensive experimental results demonstrate that the proposed method outperforms state-of-the-art image deblurring methods on both benchmark and real-world images.

Recent advancements in text-guided image editing have achieved notable success by leveraging natural language prompts for fine-grained semantic control. However, certain editing semantics are challenging to specify precisely using textual descriptions alone. A practical alternative involves learning editing semantics from paired source-target examples. Existing exemplar-based editing methods still rely on text prompts describing the change within paired examples or learning implicit text-based editing instructions. In this paper, we introduce PairEdit, a novel visual editing method designed to effectively learn complex editing semantics from a limited number of image pairs or even a single image pair, without using any textual guidance. We propose a target noise prediction that explicitly models semantic variations within paired images through a guidance direction term. Moreover, we introduce a content-preserving noise schedule to facilitate more effective semantic learning. We also propose optimizing distinct LoRAs to disentangle the learning of semantic variations from content. Extensive qualitative and quantitative evaluations demonstrate that PairEdit successfully learns intricate semantics while significantly improving content consistency compared to baseline methods. Code is available at https://github.com/xudonmao/PairEdit.

Knowledge graphs (KGs) typically have distinct entity and relation vocabularies, there is usually no overlap between the vocabularies of different KGs. Consequently, most existing studies develop independent reasoning models for different KGs. However, such models generally lack generalization capability in reasoning. This paper proposes a novel model Local-Global Collaborative Relational Representation for KG Reasoning(LGRR) to achieve universal reasoning by learning relational invariance in KGs. Specifically, we introduce a local-global relational graph embedding method, which employs a graph neural network with an attention mechanism to perform local message passing. Then, global attention is applied to information propagation to overcome the limitations of traditional local message passing and enhance capturing more comprehensive graph structural information. Finally, in the local message passing phase, we propose a relation-aware dynamic attention mechanism and a relation aggregation strategy. By integrating relation-type semantics with local subgraph structural features, our method dynamically generates attention coefficients among nodes, thereby enhancing the model reasoning capability. The results demonstrate that the zero-shot (0-shot) inference capability of a single pre-trained LGRR model is comparable or superior to models trained on specific KGs across most unseen KGs.

The application of metric learning in recommender systems has gained considerable attention, particularly for enabling supervised training through Euclidean distance. However, conventional collaborative translational metric learning typically represents each user and item as a single point and projects collaborative signals onto a specific translation vector within a uniform space. This approach fails to explicitly capture the high-order semantics and dynamic nature of user-item interactions derived from implicit user behavior. Given the extensive research on graph neural networks (GNNs), a natural extension is to incorporate high-order information into the generation of translation vectors. Yet, our empirical findings reveal that the performance is not as expected. In this paper, we systematically investigate the limitations of existing methods and attribute the issue to semantic confusion. Motivated by this finding, we propose a novel framework that integrates bilateral metric learning with a causal graph network. Specifically, we construct a structural causal model (SCM) to disentangle user behavior into causal features and confounders. To mitigate the influence of these confounders, we propose a conditional intervention module that generates confounder-specific conditions for the target. Moreover, inspired by this conditional intervention mechanism, we develop an additional module conditioned on user behavior to enhance the diversity of the recommended list, recognizing that diversity is also a crucial metric for evaluating recommendation quality. We conducted a series of experiments to validate the effectiveness of our proposed method. The results demonstrate that our approach achieves state-of-the-art performance in terms of both recommendation accuracy and diversity.

Infrared small target detection (ISTD) is a challenging task due to the low contrast and small size of the targets, which are often affected by complex backgrounds. UNet and its variants, known for their encoder–decoder structures, are widely used in such tasks since they can capture both local and global features. However, a significant drawback of UNet-based networks is the irreversible loss of crucial local features during downsampling, leading to missed detections and false positives, especially for small targets. Compared to other architectures like feature pyramid networks, UNet provides a more symmetric and efficient structure, allowing it to handle dense pixel-wise predictions effectively. However, standard UNet models still struggle to fully retain small target details, motivating the need for further improvements. To address this issue, we propose HintU, a novel network to recover the local features lost by various UNet-based methods for effective ISTD. HintU has two key contributions. First, it introduces the “Hint” mechanism for the first time, i.e., leveraging the prior knowledge of target locations to highlight critical local features. Second, it improves the mainstream UNet-based architecture to preserve target pixels even after downsampling. HintU can shift the focus of various networks (e.g., vanilla UNet, UNet++, UIUNet, MiM+, and HCFNet) from the irrelevant background pixels to a more restricted area from the beginning. Experimental results on three datasets NUDT-SIRST, SIRSTv2, and IRSTD1K demonstrate that HintU enhances the performance of existing methods with only an additional 1.88-ms cost (on RTX Titan). Additionally, the explicit constraints of HintU enhance the generalization ability of UNet-based methods. Code is available athttps://github.com/Wuzhou-Quan/HintU.

Point cloud analysis, arising from computer graphics, remains a fundamental but challenging problem, mainly due to the non-Euclidean property of point cloud data modality. With the snap increase in the amount and breadth of related research in deep learning for graphs, many important works come in the form of graphs representing the point clouds. In this paper, we present a sampling adaptive graph convolutional network that combines the powerful representation ability of random walk subgraph searching and the essential success of the Fisher vector. Extending from those existing graph representation learning or embedding methods with multi-hop neighbor random searching, we sample multi-scale walk fields by using asteerableexploration-exploitationsecond order random walk, which endows our model with the most flexibility compared with the original first order random walk. To encode each-scale walk field consisting of several walk paths, specifically, we characterize these paths of walk field by Gaussian mixture models (GMMs) so as to better analogize the standard CNNs on Euclidean modality. Each Gaussian component implicitly defines a direction and all of them properly encode thespatial layoutof walk fields after the gradient projecting to the space of Gaussian parameters, i.e. the Fisher vectors. Thereby, we introduce and name our deep graph convolutional network asPointFisher. Comprehensive evaluations on several public datasets well demonstrate the superiority of our proposed learning method over other state-of-the-arts for point cloud classification and segmentation.

Chroma intra prediction aims to reduce chroma redundancies within a frame, which plays an important role in improving the coding efficiency of intra coding. Existing chroma intra prediction methods typically utilize the spatial relationship between the current luma block and its neighboring reference luma blocks to predict its chroma samples. However, the spatial properties of luma components differ from those of chroma components, which limits the accuracy of chroma intra prediction. To tackle this issue, an efficient Exemplar Colorization Network (ECNet)-based chroma intra prediction method is proposed in this paper, in which the colorization relationship between reference luma and chroma components is exploited to predict the chroma components for the current luma component. Inspired by the principle that semantic information in an image exhibits short-range continuity, a Spatial-consistency-based Colorization Transfer Network (SCTNet) is proposed, which builds and transfers colorization representations of neighboring reference blocks for chroma prediction. To improve the chroma prediction capability of SCTNet, a colorization learning module is developed to learn the robust mapping relationship from the luma component to the chroma component in a region-to-pixel manner, and a weight-adaptive reconstruction module is designed to adaptively utilize reference information from neighboring blocks to generate an initial prediction result. In addition, to further improve the accuracy of chroma intra prediction, a multi-reference-based chroma refinement network is proposed, which simultaneously uses the spatial information of neighboring reference chroma blocks and the current luma block to eliminate blocking and color-bleeding artifacts in the initial prediction result. Experimental results demonstrate that our proposed ECNet outperforms the state-of-the-art chroma intra prediction methods in terms of coding performance.

The widely deployed ways to capture a set of unorganized points, e.g., merged laser scans, fusion of depth images, and structure-from- , usually yield a 3-D noisy point cloud. Accurate normal estimation for the noisy point cloud makes a crucial contribution to the success of various applications. However, the existing normal estimation wisdoms strive to meet a conflicting goal of simultaneously performing normal filtering and preserving surface features, which inevitably leads to inaccurate estimation results. We propose a normal estimation neural network (Norest-Net), which regards normal filtering and feature preservation as two separate tasks, so that each one is specialized rather than traded off. For full noise removal, we present a normal filtering network (NF-Net) branch by learning from the noisy height map descriptor (HMD) of each point to the ground-truth (GT) point normal; for surface feature recovery, we construct a normal refinement network (NR-Net) branch by learning from the bilaterally defiltered point normal descriptor (B-DPND) to the GT point normal. Moreover, NR-Net is detachable to be incorporated into the existing normal estimation methods to boost their performances. Norest-Net shows clear improvements over the state of the arts in both feature preservation and noise robustness on synthetic and real-world captured point clouds.

Word sense disambiguation (WSD) can be viewed as two subtasks: textual word sense disambiguation (Textual-WSD) and visual word sense disambiguation (Visual-WSD). They aim to identify the most semantically relevant senses or images to a given context containing ambiguous target words. However, existing WSD models seldom address these two subtasks jointly due to lack of images in Textual-WSD datasets or lack of senses in Visual-WSD datasets. To bridge this gap, we propose PolCLIP, a unified image-text WSD model. By employing an image-text complementarity strategy, it not only simulates stable diffusion models to generate implicit visual representations for word senses but also simulates image captioning models to provide implicit textual representations for images. Additionally, a disambiguation-oriented image-sense dataset is constructed for the training objective of learning multimodal polysemy representations. To the best of our knowledge, PolCLIP is the first model that can cope with both Textual-WSD and Visual-WSD. Extensive experimental results on benchmarks demonstrate the effectiveness of our method, achieving a 2.53% F1-score increase over the state-of-the-art models on Textual-WSD and a 2.22% HR@1 improvement on Visual-WSD.

Visual Word Sense Disambiguation (Visual-WSD), as a sub-task of fine-grained image-text retrieval, requires a high level of language-vision understanding to capture and exploit the nuanced relationships between text and visual features. However, the cross-linguistic background only with limited contextual information is considered the most significant challenges for this task. In this paper, we propose MTA, which employs a new approach for multilingual contrastive learning with self-distillation to align fine-grained textual features to fixed vision features and align non-English textual features to English textual momentum features. It is a lightweight and end-to-end model since it does not require updating the visual encoder or translation operations. Furthermore, a trilingual fine-grained image-text dataset is developed and a ChatGPT API module is integrated to enrich the word senses effectively during the testing phase. Extensive experiments show that MTA achieves state-of-the-art results on the benchmark English, Farsi, and Italian datasets in SemEval-2023 Task 1 and exhibits impressive generalization abilities when dealing with variations in text length and language.

Task-Oriented Dialogue commonly utilizes external knowledge bases to respond user utterances to complete specific tasks. Existing models use whole dialogue history record directly, which contain massive irrelevant noise entities, decreasing accuracy in knowledge retrieval. Thus, how to avoid noise entities and effectively identify relevant entities representing user intents within long dialogue history record remains as a crucial challenge. This paper proposes a new model named IEM consisted of a knowledge selector to retrieve relevant knowledge and a response generator to generate system responses. In addition, this paper introduces an intent retrieval mechanism that search for intent entities from dialogue history and generate intent prompts for task-oriented dialogue. Experiments on three publicly available datasets show that our IEM is superior to existing state-of-the-art baseline methods, verifying its effectiveness in task-oriented dialogue. Moreover, the results demonstrate the promising adaptability of intent retrieval mechanism in incorporating into large language models.

In this paper, we propose a two-stage network with modality-specific and -shared contrastive learning (MMCL) for multimodal sentiment analysis. MMCL comprises a category-aware modality-specific contrastive (CMC) module and a self-decoupled modality-shared contrastive (SMC) module. In the first stage, the CMC module guides the encoders to extract modality-specific representations by constructing positive-negative pairs according to sample categories. In the second stage, the SMC module guides the encoders to extract modality-shared representations by constructing positive-negative pairs based on modalities and decoupling the self-contrast of all modalities. In the aforementioned modules, we leverage self-modulation factors to focus more on hard positive pairs through assigning different loss weights to positive pairs depending on their distance. In particular, we introduce a dynamic routing algorithm to cluster the inputs of the contrastive modules during training, where a gradient stopping strategy is utilized to isolate the backpropagation process of the CMC and SMC modules. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets show that MMCL achieves the state-of-the-art performance.

Recent advances in text-to-image models have enabled high-quality personalized image synthesis based on user-provided concepts with flexible textual control. In this work, we analyze the limitations of two primary techniques in text-to-image personalization: Textual Inversion and DreamBooth. When integrating the learned concept into new prompts, Textual Inversion tends to overfit the concept, while DreamBooth often overlooks it. We attribute these issues to the incorrect learning of the embedding alignment for the concept. To address this, we introduce AttnDreamBooth, a novel approach that separately learns the embedding alignment, the attention map, and the subject identity across different training stages. We also introduce a cross-attention map regularization term to enhance the learning of the attention map. Our method demonstrates significant improvements in identity preservation and text alignment compared to the baseline methods.

The cross-attention mechanism has been widely employed in the multimodal rumor detection task, which is computation-intensive and suffers from the restricted modal receptive field. In this paper, we propose a multimodal rumor detection model (MKV), which maps multimodal key semantics with discrimination into feature vectors for rumor detection. More specifically, MKV extracts high-dimensional features for each modality separately by the Multimodal Feature Extractor (MFE). The mapping mechanism learns low-dimensional mapping scheme (Map) and key semantics (Key) with discrimination from the different modal features respectively. Subsequently, the Map and Key jointly construct a state matrix (State) containing all possible permutations of modalities. In particular, a max pooling operation is performed on State and products a feature vector (Vector). The mapping mechanism is able to incrementally learn the discriminative semantics by stacking manner. Vectors from the stacking process are leveraged in the Rumor Detection module (RD). Extensive experiments on two public datasets show that the MKV achieves the state-of-the-art performance.

Language Models (LMs) have achieved impressive success in various question answering (QA) tasks but have shown limited performance on structured reasoning. Recent research suggests that Knowledge Graph (KG) can augment text data by providing a structured background to enhance reasoning capabilities of LMs. Therefore, how to integrate and reason over KG representations and language context remains an open question. In this work, we propose MJR, a novel model to integrate encoded representations of LMs and graph neural network through multiple layers of feature interaction operations. Subsequently, the fused feature representations in two modalities are fed into a multi-head representation fusion module to comprehensively capture semantic and graph structure information, thereby enhancing language understanding and reasoning capabilities. In addition, we investigate the performance and applicability of different types of large language models as text encoder in the question-answering task. We evaluate our model on three common dataset: CommonsenseQA, OpenBookQA, and MedQA-USMLE datasets. The results demonstrate the advancements of MJR over existing LMs, LM+KG and LLMs models in reasoning for question answering.

The Versatile Video Coding (VVC) is the latest video coding standard that includes advanced intra coding techniques such as the chroma separate tree and cross-component linear model to improve the prediction accuracy of chroma components. However, these sophisticated techniques inevitably increase the computational complexity of chroma intra coding. In this paper, we propose a fast chroma intra-prediction mode decision algorithm to reduce the computational complexity of the VVC encoder. Firstly, the mode selection correlation between the current chroma Coding Unit (CU) and its nearest-encoded chroma CU is analyzed. Then, based on the variation of the sum of modulus differences, an early mode decision is proposed for the current chroma CU to determine whether the best coding mode should be traversed or directly inherited from its nearest encoded chroma CU. The experimental results show that the proposed algorithm significantly reduces the computational complexity of chroma intra-prediction and achieves an average of 28% chroma intra encoding time saving with only 0.08-1.43% performance degradation in terms of BD-rate.

Abstract Topic taxonomy discovery aims at uncovering topics of different abstraction levels and constructing hierarchical relations between them. Unfortunately, most prior work can hardly model semantic scopes of words and topics by holding the Euclidean embedding space assumption. What’s worse, they infer asymmetric hierarchical relations by symmetric distances between topic embeddings. As a result, existing methods suffer from problems of low-quality topics at high abstraction levels and inaccurate hierarchical relations. To alleviate these problems, this paper develops a Box embedding-based Topic Model (BoxTM) that maps words and topics into the box embedding space, where the asymmetric metric is defined to properly infer hierarchical relations among topics. Additionally, our BoxTM explicitly infers upper-level topics based on correlation between specific topics through recursive clustering on topic boxes. Finally, extensive experiments validate high-quality of the topic taxonomy learned by BoxTM.

While the wisdom of training an image dehazing model on synthetic hazy data can alleviate the difficulty of collecting real-world hazy/clean image pairs, it brings the well-known domain shift problem. From a different yet new perspective, this paper explores contrastive learning with an adversarial training effort to leverage unpaired real-world hazy and clean images, thus alleviating the domain shift problem and enhancing the network's generalization ability in real-world scenarios. We propose an effective unsupervised contrastive learning paradigm for image dehazing, dubbed UCL-Dehaze. Unpaired real-world clean and hazy images are easily captured, and will serve as the important positive and negative samples respectively when training our UCL-Dehaze network. To train the network more effectively, we formulate a new self-contrastive perceptual loss function, which encourages the restored images to approach the positive samples and keep away from the negative samples in the embedding space. Besides the overall network architecture of UCL-Dehaze, adversarial training is utilized to align the distributions between the positive samples and the dehazed images. Compared with recent image dehazing works, UCL-Dehaze does not require paired data during training and utilizes unpaired positive/negative data to better enhance the dehazing performance. We conduct comprehensive experiments to evaluate our UCL-Dehaze and demonstrate its superiority over the state-of-the-arts, even only 1,800 unpaired real-world images are used to train our network. Source code is publicly available at https://github.com/yz-wang/UCL-Dehaze.

Monocular depth estimation (MDE) remains a fundamental yet not well-solved problem in computer vision. Current wisdom of MDE often achieves blurred or even indistinct depth boundaries, degenerating the quality of vision-based intelligent transportation systems. This paper presents an edge-enhanced vision transformer bins network for monocular depth estimation, termed eViTBins. eViTBins has three core modules to predict monocular depth maps with exceptional smoothness, accuracy, and fidelity to scene structures and object edges. First, a multi-scale feature fusion module is proposed to circumvent the loss of depth information at various levels during depth regression. Second, an image-guided edge-enhancement module is proposed to accurately infer depth values around image boundaries. Third, a vision transformer-based depth discretization module is introduced to comprehend the global depth distribution. Meanwhile, unlike most MDE models that rely on high-performance GPUs, eViTBins is optimized for seamless deployment on edge devices, such as NVIDIA Jetson Nano and Google Coral SBC, making it ideal for real-time intelligent transportation systems applications. Extensive experimental evaluations corroborate the superiority of eViTBins over competing methods, notably in terms of preserving depth edges and global depth representations.

The acquisition of both local and global features from irregular point clouds is crucial for 3D object detection (3DOD). Current mainstream 3D detectors neglect significant local features during pooling operations or disregard many global features of the overall scene context. This paper proposes new techniques for simultaneously learning local-global features of scene point clouds to enhance 3DOD. Specifically, we propose an efficient 3DOD network in indoor point clouds, named SimLOG, which utilizes simultaneous local-global feature learning. SimLOG has two main contributions: a Dynamic Points Interaction (DPI) module to recover local features lost during pooling, and a Global Context Aggregation(GCA) module to aggregate multi-scale features from various layers of the encoder to improve scene context awareness. Unlike traditional local-global feature learning methods, our DPI and GCA modules are integrated into a single feature learning module, making it easily detachable and able to be incorporated into existing 3DOD networks to enhance their performance. SimLOG demonstrates superior performance over twenty competitors in terms of detection accuracy and robustness on both the SUN RGB-D and ScanNet V2 datasets. Specifically, SimLOG boosts the baseline VoteNet by 8.1% of [email protected] on ScanNet V2 and by 3.9% of [email protected] on SUN RGB-D. Code is publicly available athttps://github.com/chenbaian-cs/SimLOG.

There is a prevailing trend towards fusing multi-modal information for 3D object detection (3OD). However, challenges related to computational efficiency, plug-and-play capabilities, and accurate feature alignment have not been adequately addressed in the design of multi-modal fusion networks. In this paper, we present PointSee, a lightweight, flexible, and effective multi-modal fusion solution to facilitate various 3OD networks by semantic feature enhancement of point clouds (e.g., LiDAR or RGB-D data) assembled with scene images. Beyond the existing wisdom of 3OD, PointSee consists of a hidden module (HM) and a seen module (SM): HM decorates point clouds using 2D image information in an offline fusion manner, leading to minimal or even no adaptations of existing 3OD networks; SM further enriches the point clouds by acquiring point-wise representative semantic features, leading to enhanced performance of existing 3OD networks. Besides the new architecture of PointSee, we propose a simple yet efficient training strategy, to ease the potential inaccurate regressions of 2D object detection networks. Extensive experiments on the popular outdoor/indoor benchmarks show quantitative and qualitative improvements of our PointSee over thirty-five state-of-the-art methods.

Digital twins (DTs) have emerged as a promising enabler for representing the real-time states of physical worlds and realizing self-sustaining systems. In practice, DTs of physical devices, such as mobile users (MUs), are commonly deployed in multi-access edge computing (MEC) networks for the sake of reducing latency. To ensure the accuracy and fidelity of DTs, it is essential for MUs to regularly synchronize their status with their DTs. However, MU mobility introduces significant challenges to DT synchronization. Firstly, MU mobility triggers DT migration which could cause synchronization failures. Secondly, MUs require frequent synchronization with their DTs to ensure DT fidelity. Nonetheless, DT migration among MEC servers, caused by MU mobility, may occur infrequently. Accordingly, we propose a two-timescale DT synchronization and migration framework with reliability consideration by establishing a non-convex stochastic problem to minimize the long-term average energy consumption of MUs. We use Lyapunov theory to convert the reliability constraints and reformulate the new problem as a partially observable Markov decision-making process (POMDP). Furthermore, we develop a heterogeneous agent proximal policy optimization with Beta distribution (Beta-HAPPO) method to solve it. Numerical results show that our proposed Beta-HAPPO method achieves significant improvements in energy savings when compared with other benchmarks.

We propose GeoGCN, a novel geometric dual-domain graph convolution network for point cloud denoising (PCD). Beyond the traditional wisdom of PCD, to fully exploit the geometric information of point clouds, we define two kinds of surface normals, one is called Real Normal (RN), and the other is Virtual Normal (VN). RN preserves the local details of noisy point clouds while VN avoids the global shape shrinkage during denoising. GeoGCN is a new PCD paradigm that, 1) first regresses point positions by spatial-based GCN with the help of VNs, 2) then estimates initial RNs by performing Principal Component Analysis on the regressed points, and 3) finally regresses fine RNs by normal-based GCN. Unlike existing PCD methods, GeoGCN not only exploits two kinds of geometry expertise (i.e., RN and VN) but also benefits from training data. Experiments validate that GeoGCN outperforms SOTAs in terms of both noise-robustness and local-and-global feature preservation.

This research paper presents a comprehensive exploration of the application of Generative Artificial Intelligence (GAI) within the realm of language education. Through a systematic review of 20 empirical studies conducted within a structured three-step framework, this study elucidates the multifaceted integration of GAI in language learning. The examination encompasses various dimensions, including target languages, learners' educational levels, GAI applications, language skills, and practical outcomes. Additionally, the study critically assesses the key advantages and challenges inherent in GAI's role in language education. The paper provides valuable insights into human-technology interaction by delving into language learners' attitudes toward GAI. Notably, this research identifies three pivotal roles GAI assumes within the language learning process, and they are co-author, evaluator, and learning materials provider. The study concludes by charting a path toward future research endeavors within the evolving landscape of GAI-based language learning by implicating the future research direction of integrating the GAI into language education, including collaborations between humans and GAI, clarifying the definition of GAI-powered plagiarism, GAI-based language activities design, prompting strategies, and digital literacy.

In terms of the generative process, the Gamma-Gamma-Poisson Process (G2PP) is equivalent to the nonparametric topic model of Hierarchical Dirichlet Process (HDP). Considering the high computational cost of estimating parameters in HDP, a parallel G2PP was developed to generate topics efficiently via multi-threading. Unfortunately, the above model needs to predefine the number of topics. To address this issue, we first propose a Topic Self-Adaptive Model (TSAM) for nonparametric and parallel topic discovery. In TSAM, a monitor-executor mechanism is developed to manage the global topic information using a hierarchical structure of threads. Based on the apparatus of copulas, we further extend our TSAM to TSAMcop for coherent topic modeling by exploiting a copula guided parallel Gibbs sampling algorithm. Extensive experiments validate the effectiveness of both TSAM and TSAMcop.

Social media allows the public to access information conveniently, in which the false messages that are eye-catching may spread fast. In this paper, we propose a two-stage confidence-guided boundary adaption (CBA) network, consisting of a feature preprocessing (FP) module, a biased ambiguity learning (BA) module and a confidence-guided boundary adaptation (CG) module. In the first stage, the FP module obtains the textual and visual features, which are fused by conducting the visual-to-textual and textual-to-visual correlation coefficients with attention mechanism. Furthermore, BA evaluates the distribution distance between fused features and single modalities to determine the weights between modalities, capturing the semantics of key modality. In the second stage, CG leverages samples from the low-confidence interval to generate new instances using a mixup of augmentation techniques, aiming to occupy the decision space and optimize the decision boundary of the classifier. Extensive experiments on two public datasets show that our CBA model is 1.6% and 2.6% higher than the state-of-the-art methods.

With the fast progress in information technologies and artificial intelligence (AI), smart healthcare has gained considerable momentum. By using advanced technologies like AI, smart healthcare aims to promote human beings’ health and well-being throughout their life. As smart healthcare develops, big healthcare data are produced by various sensors, devices, and communication technologies constantly. To deal with these big multi-source data, automatic information fusion becomes crucial. Information fusion refers to the integration of multiple information sources for obtaining more reliable, effective, and precise information to support optimal decision-making. The close study of information fusion for healthcare with the adoption of advanced AI technologies has become an increasingly important and active field of research. The aim of this is to present a systematic description and state-of-the-art understanding of research about information fusion for healthcare with AI. Structural topic modeling was implemented to detect major research topics covered within 351 relevant articles. Annual trends and correlations of the identified topics were also investigated to identify potential future research directions. In addition, the primary research concerns of top countries/regions, institutions, and authors were shown and compared. The findings based on our analyses provide scientific and technological perspectives of research on information fusion for smart health with AI and offer useful insights and implications for its future development. We also provide valuable guidance for researchers and project managers to allocate research resources and promote effective international collaborations.

Researchers have been aware that emotion is not one-hot encoded in emotion-relevant classification tasks, and multiple emotions can coexist in a given sentence. Recently, several works have focused on leveraging a distribution label or a grayscale label of emotions in the classification model, which can enhance the one-hot label with additional information, such as the intensity of other emotions and the correlation between emotions. Such an approach has been proven effective in alleviating the overfitting problem and improving the model robustness by introducing a distribution learning component in the objective function. However, the effect of distribution learning cannot be fully unfolded as it can reduce the model’s discriminative ability within similar emotion categories. For example, “Sad” and “Fear” are both negative emotions. To address such a problem, we proposed a novel emotion extension scheme in the prior work (Li, Chen, Xie, Li, and Tao, 2021). The prior work incorporated fine-grained emotion concepts to build an extended label space, where a mapping function between coarse-grained emotion categories and fine-grained emotion concepts was identified. For example, sentences labeled “Joy” can convey various emotions such as enjoy, free, and leisure. The model can further benefit from the extended space by extracting dependency within fine-grained emotions when yielding predictions in the original label space. The prior work has shown that it is more apt to apply distribution learning in the extended label space than in the original space. A novel sparse connection method, i.e., Leaky Dropout, is proposed in this paper to refine the dependency-extraction step, which further improves the classification performance. In addition to the multiclass emotion classification task, we extensively experimented on sentiment analysis and multilabel emotion prediction tasks to investigate the effectiveness and generality of the label extension schema.

Recently, graph neural networks (GNNs) have achieved promising results in session-based recommendation. Existing methods typically construct a local session graph and a global session graph to explore complex item transition patterns. However, studies have seldom investigated the repeat consumption phenomenon in a local graph. In addition, it is challenging to retrieve relevant adjacent nodes from the whole training set owing to computational complexity and space constraints. In this study, we use a GNN to jointly model intra- and inter-session item dependencies for session-based recommendations. We construct a repeat-aware local session graph to encode the intra-item dependencies and generate the session representation with positional awareness. Then, we use sessions from the current mini-batch instead of the whole training set to construct a global graph, which we refer to as the session-level global graph. Next, we aggregate the K-nearest neighbors to generate the final session representation, which enables easy and efficient neighbor searching. Extensive experiments on three real-world recommendation datasets demonstrate that RN-GNN outperforms state-of-the-art methods.

Recently, many efforts have been devoted to promoting the Emotion-Cause Pair Extraction (ECPE) task, as jointly extracting emotions and their causes is considered more helpful than only identifying the emotions in many applications. Among the existing efforts, end-to-end approaches are getting popular as the main trend, while others like pipeline models have been overlooked due to their potential issues of cascading errors. Nevertheless, the advantages of the pipeline models, such as logically dividing a complicated task into multiple easier subtasks, are underestimated and not well exploited. Moreover, the existing end-to-end approaches fail to capture the implicit co-occurrence or exclusion patterns between multiple pairs of emotions and causes since they are extracted independently. In view of these limitations, we propose a novel two-stage model to address the ECPE task and incorporate reinforcement learning (RL) to tackle the cascading error issue. In particular, our two-stage model first detects emotion clauses and then recognizes cause clauses for each detected emotion clause sequentially. By representing the error of each decision as an explicit reward, our model clearly knows how the error at each stage affects the final performance, hence the model can adjust itself for better performance. Furthermore, the sequential prediction enables our model to use the results achieved in the previous stages as auxiliary information in the subsequent stages. Extensive experiments on the benchmark dataset demonstrate the effectiveness of our proposed two-stage model, and the ablation comparison shows the promising effect of reducing cascading errors by incorporating RL.

User intent classification is a vital task for analyzing users’ essential requirements from the users’ input query in information retrieval systems, question answering systems, and dialogue systems. Pre-trained language model Bidirectional Encoder Representation from Transformers (BERT) has been widely applied to the user intent classification task. However, BERT is compute intensive and time-consuming during inference and usually causes latency in real-time applications. To improve the inference efficiency of BERT for the user intent classification task, this article proposes a new network named one-stage deep-supervised early-exiting BERT as one-stage deep-supervised early-exiting BERT (OdeBERT). In addition, a deep supervision strategy is developed to incorporate the network with internal classifiers by one-stage joint training to improve the learning process of classifiers by extracting discriminative category features. Experiments are conducted on publicly available datasets, including ECDT, SNIPS, and FDQuestion. The results show that the OdeBERT can speed up original BERT 12 times faster at most with the same performance, outperforming state-of-the-art baseline methods.

The metaverse and its underlying blockchain technology have attracted extensive attention in the past few years. How to mine, process, and analyze the tremendous data generated by the metaverse systems has posed a number of challenges. Aiming to address them, we mainly focus on modeling and understanding the blockchain transaction network from a structural identity perspective, which represents the entire network structure and reveals the relations among multiple entities. In this article, we analyze three metaverse-related systems: non-fungible token (NFT), Ethereum (ETH), and Bitcoin (BTC) from the structural-identity perspective. First, we conduct the complex network analysis of the metaverse network and obtain several new insights (i.e., power-law degree distribution, disconnection, disassortativity, preferential attachment, and non-rich-club effect). Secondly, based on such findings, we propose a novel representation learning method named structure-to-vector with random pace (SVRP) for learning both the latent representation and structural identity of the network. Thirdly, we conduct node classification and link prediction tasks with the integration of graph neural networks (GNNs). Empirical results on three real-world datasets demonstrate that our proposed SVRP outperforms other existing methods in multiple tasks. In particular, our SVRP achieves the highest node classification accuracy (Acc) (99.3$\%$) and$F$1-score (96.7$\%$) while only requiring original non-attributed graphs.

Sentence representation learning is a crucial task in natural language processing, as the quality of learned representations directly influences downstream tasks, such as sentence classification and sentiment analysis. Transformer-based pretrained language models such as bidirectional encoder representations from transformers (BERT) have been extensively applied to various natural language processing tasks, and have exhibited moderately good performance. However, the anisotropy of the learned embedding space prevents BERT sentence embeddings from achieving good results in the semantic textual similarity tasks. It has been shown that contrastive learning can alleviate the anisotropy problem and significantly improve sentence representation performance. Therefore, there has been a surge in the development of models that utilize contrastive learning to fine-tune BERT-like pretrained language models to learn sentence representations. But no systematic review of contrastive learning models for sentence representations has been conducted. To fill this gap, this article summarizes and categorizes the contrastive learning based sentence representation models, common evaluation tasks for assessing the quality of learned representations, and future research directions. Furthermore, we select several representative models for exhaustive experiments to illustrate the quantitative improvement of various strategies on sentence representations.

As a novel paradigm for data mining and dimensionality reduction, Non-negative Matrix Tri-Factorization (NMTF) has attracted much attention due to its notable performance and elegant mathematical derivation, and it has been applied to a plethora of real-world applications, such as text data co-clustering. However, the existing NMTF-based methods usually involve intensive matrix multiplications, which exhibits a major limitation of high computational complexity. With the explosion at both the size and the feature dimension of texts, there is a growing need to develop a parallel and scalable NMTF-based algorithm for text data co-clustering. To this end, we first show in this paper how to theoretically derive the original optimization problem of NMTF by introducing the Lagrangian multipliers. Then, we propose to solve the Lagrange dual objective function in parallel through an efficient distributed implementation. Extensive experiments on five benchmark corpora validate the effectiveness, efficiency, and scalability of our distributed parallel update algorithm for an NMTF-based text data co-clustering method.

Both the metaverse and its underlying blockchain technology have attracted extensive attention in the past few years. It becomes a natural problem to extract, process, and analyze the tremendous data generated by the blockchain systems for various metaverse applications though it also poses diverse challenges. Amongst those challenges, this paper mainly focuses on modeling and understanding the blockchain transaction network from a structural identity perspective, which represents the entire network structure and reveals the relations among multiple entities. In this paper, we propose a novel representation learning method named Structure-to-Vector with Random Pace (SVRP) for learning both latent representation and structural identity of blockchain transaction networks. We then conduct node classification and link prediction tasks with integration with Graph Neural Networks (GNNs). Empirical results on three representative blockchain data sets, namely Non-fungible token (NFT), Ethereum (ETH), and Bitcoin (BTC), demonstrate that our proposed SVRP outperforms other existing methods in multiple tasks. In particular, our SVRP achieves the highest node classification accuracy (99.3%) while only requiring original non-attributed graphs (i.e., graphs without node features).

Abstract How will you repair a physical object with large missings? You may first recover its global yet coarse shape and stepwise increase its local details. We are motivated to imitate the above physical repair procedure to address the point cloud completion task. We propose a novel stepwise point cloud completion network (SPCNet) for various 3D models with large missings. SPCNet has a hierarchical bottom‐to‐up network architecture. It fulfills shape completion in an iterative manner, which 1) first infers the global feature of the coarse result; 2) then infers the local feature with the aid of global feature; and 3) finally infers the detailed result with the help of local feature and coarse result. Beyond the wisdom of simulating the physical repair, we newly design a cycle loss to enhance the generalization and robustness of SPCNet. Extensive experiments clearly show the superiority of our SPCNet over the state‐of‐the‐art methods on 3D point clouds with large missings. Code is available at https://github.com/1127368546/SPCNet .

Abstract Real‐world rain is a mixture of rain streaks and rainy haze. However, current efforts formulate image rain streaks removal and rainy haze removal as separated models, worsening the loss of image details. This paper attempts to solve the mixture of rain removal problem in a single model by estimating the scene depths of images. To this end, we propose a novel SEMI‐ supervised M ixture O f rain RE moval G enerative A dversarial N etwork (Semi‐MoreGAN). Unlike most of existing methods, Semi‐MoreGAN is a joint learning paradigm of mixture of rain removal and depth estimation; and it effectively integrates the image features with the depth information for better rain removal. Furthermore, it leverages unpaired real‐world rainy and clean images to bridge the gap between synthetic and real‐world rain. Extensive experiments show clear improvements of our approach over twenty representative state‐of‐the‐arts on both synthetic and real‐world rainy images. Source code is available at https://github.com/syy-whu/Semi-MoreGAN .

Abstract Image smoothing is a fundamental low‐level vision task that aims to preserve salient structures of an image while removing insignificant details. Deep learning has been explored in image smoothing to deal with the complex entanglement of semantic structures and trivial details. However, current methods neglect two important facts in smoothing: 1) naive pixel‐level regression supervised by the limited number of high‐quality smoothing ground‐truth could lead to domain shift and cause generalization problems towards real‐world images; 2) texture appearance is closely related to object semantics, so that image smoothing requires awareness of semantic difference to apply adaptive smoothing strengths. To address these issues, we propose a novel Contrastive Semantic‐Guided Image Smoothing Network (CSGIS‐Net) that combines both contrastive prior and semantic prior to facilitate robust image smoothing. The supervision signal is augmented by leveraging undesired smoothing effects as negative teachers, and by incorporating segmentation tasks to encourage semantic distinctiveness. To realize the proposed network, we also enrich the original VOC dataset with texture enhancement and smoothing labels, namely VOC‐smooth, which first bridges image smoothing and semantic segmentation. Extensive experiments demonstrate that the proposed CSGIS‐Net outperforms state‐of‐the‐art algorithms by a large margin. Code and dataset are available at https://github.com/wangjie6866/CSGIS-Net .

Abstract Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question – if the combination of image restoration and object detection, can boost the performance of cutting‐edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet. Different from existing efforts that intuitively apply image dehazing/deraining as a pre‐processing step, TogetherNet considers a multi‐task joint learning problem. Following the joint learning scheme, clean features produced by the restoration network can be shared to learn better object detection in the detection network, thus helping TogetherNet enhance the detection capacity in adverse weather conditions. Besides the joint learning architecture, we design a new Dynamic Transformer Feature Enhancement module to improve the feature extraction and representation capabilities of TogetherNet. Extensive experiments on both synthetic and real‐world datasets demonstrate that our TogetherNet outperforms the state‐of‐the‐art detection approaches by a large margin both quantitatively and qualitatively. Source code is available at https://github.com/yz-wang/TogetherNet .

With the growing availability and popularity of sentiment-rich resources like blogs and online reviews, new opportunities and challenges have emerged regarding the identification, extraction, and organization of sentiments from user-generated documents or sentences. Recently, many studies have exploited lexicon-based methods or supervised learning algorithms to conduct sentiment analysis tasks separately; however, the former approaches ignore contextual information of sentences and the latter ones do not take sentiment information embedded in sentiment words into consideration. To tackle these limitations, we propose a new model named Sentiment Convolutional Neural Network (SentiCNN) to analyze the sentiments of sentences with both contextual and sentiment information of sentiment words, in which, contextual information is captured from word embeddings and sentiment information is identified using existing lexicons. We incorporate a Highway Network into our model to adaptively combine sentiment and contextual information from sentences by strengthening the connection between features of both sentences and their sentiment words. Furthermore, we propose three lexicon-based attention mechanisms (LBAMs) for our SentiCNN model to find the most important indicators of sentiments and make predictions more effectively. Experiments over two well-known datasets indicate that sentiment words, the Highway Network, and LBAMs contribute to sentiment analysis.

Hierarchical Dirichlet Process (HDP) has attracted much attention in the research community of natural language processing. Given a corpus, HDP is able to determine the number of topics automatically, possessing an important feature dubbed nonparametric that overcomes the challenging issue of manually specifying a suitable topic number in parametric topic models, such as Latent Dirichlet Allocation (LDA). Nevertheless, HDP requires a much higher computational cost than LDA for parameter estimation. By taking the advantage of multi-threading, a parallel Gibbs sampling algorithm is proposed to estimate parameters for HDP based on the equivalence between HDP and Gamma-Gamma Poisson Process (G2PP) in terms of the generative process. Unfortunately, the above parallel Gibbs sampling algorithm requires to apply the finite approximation on the number of topics manually (i.e., predefine the topic number), thus can not retain the nonparametric feature of HDP. Another drawback of the above models is the lack of capturing the semantic dependencies between words, because the topic assignment of words is independent with each other. Although some works have been done in phrase-based topic modelling, these existing methods are still limited by either enforcing the entire phrase to share a common topic or requiring much complex and time-consuming phrase mining methods. In this paper, we aim to develop a copula guided parallel Gibbs sampling algorithm for HDP which can adjust the number of topics dynamically and capture the latent semantic dependencies between words that compose a coherent segment. Extensive experiments on real-world datasets indicate that our method achieves low perplexities and high topic coherence scores with a small time cost. In addition, we validate the effectiveness of our method on the modelling of word semantic dependencies by comparing the extracted topical phrases with those learned by state-of-the-art phrase-based baselines.

In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

With the development of social network platforms, discussion forums, and question answering websites, a huge number of short messages that typically contain a few words for an individual document are posted by online users. In these short messages, emotions are frequently embedded for communicating opinions, expressing friendship, and promoting influence. It is quite valuable to detect emotions from short messages, but the corresponding task suffers from the sparsity of feature space. In this article, we first generate term groups co-occurring in the same context to enrich the number of features. Then, two basic supervised topic models are proposed to associate emotions with topics accurately. To reduce the time cost of parameter estimation, we further propose an accelerated algorithm for our basic models. Extensive evaluations using three short corpora validate the efficiency and effectiveness of the accelerated models for predicting the emotions of unlabeled documents, in addition to generate the topic-level emotion lexicons.

Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics. However, the existing parameter inference method like Monte Carlo sampling is quite time-consuming. In this paper, we propose two efficient neural mixed counting models, i.e., the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are developed to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Experiments on real-world datasets indicate that our models outperform state-of-the-art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generating dispersed proportions of document topics.

This paper looks at this intriguing question: are single images with their details lost during deraining, reversible to their artifact-free status? We propose an end-to-end detail-recovery image deraining network (termed a DRDNet) to solve the problem. Unlike existing image deraining approaches that attempt to meet the conflicting goal of simultaneously deraining and preserving details in a unified framework, we propose to view rain removal and detail recovery as two seperate tasks, so that each part could specialize rather than trade-off between two conflicting goals. Specifically, we introduce two parallel sub-networks with a comprehensive loss function which synergize to derain and recover the lost details caused by deraining. For complete rain removal, we present a rain residual network with the squeeze-and-excitation (SE) operation to remove rain streaks from the rainy images. For detail recovery, we construct a specialized detail repair network consisting of welldesigned blocks, named structure detail context aggregation block (SDCAB), to encourage the lost details to return for eliminating image degradations. Moreover, the detail recovery branch of our proposed detail repair framework is detachable and can be incorporated into existing deraining methods to boost their performances. DRD-Net has been validated on several well-known benchmark datasets in terms of deraining robustness and detail accuracy. Comparisons show clear visual and numerical improvements of our method over the state-of-the-arts.

In this paper, we propose a normal estimation method for unstructured point cloud. We observe that geometric estimators commonly focus more on feature preservation but are hard to tune parameters and sensitive to noise, while learning-based approaches pursue an overall normal estimation accuracy but cannot well handle challenging regions such as surface edges. This paper presents a novel normal estimation method, under the co-support of geometric estimator and deep learning. To lowering the learning difficulty, we first propose to compute a suboptimal initial normal at each point by searching for a best fitting patch. Based on the computed normal field, we design a normal-based height map network (NH-Net) to fine-tune the suboptimal normals. Qualitative and quantitative evaluations demonstrate the clear improvements of our results over both traditional methods and learning-based methods, in terms of estimation accuracy and feature recovery.

Abstract We present a near‐lighting photometric stereo (NL‐PS) system to produce digital bas‐reliefs from a physical object (set) directly. Unlike both the 2D image and 3D model‐based modelling methods that require complicated interactions and transformations, the technique using NL‐PS is easy to use with cost‐effective hardware, providing users with a trade‐off between abstract and representation when creating bas‐reliefs. Our algorithm consists of two steps: normal map acquisition and constrained 3D reconstruction. First, we introduce a lighting model, named the quasi‐point lighting model (QPLM), and provide a two‐step calibration solution in our NL‐PS system to generate a dense normal map. Second, we filter the normal map into a detail layer and a structure layer, and formulate detail‐ or structure‐preserving bas‐relief modelling as a constrained surface reconstruction problem of solving a sparse linear system. The main contribution is a WYSIWYG (i.e. what you see is what you get) way of building new solvers that produces multi‐style bas‐reliefs with their geometric structures and/or details preserved. The performance of our approach is experimentally validated via comparisons with the state‐of‐the‐art methods.

Rate control plays an important role in high efficiency video coding (HEVC), and bit allocation is the foundation of rate control. The video content characteristics are significant for bit allocation, and modeling an accurate relationship between video content characteristics and bit allocation is essential for bit allocation optimization. Therefore, in this article, a video content characteristics–based frame-level optimal bit allocation algorithm is proposed for improving the rate distortion (RD) performance of HEVC. First, the number of search points of motion estimation is used to evaluate the motion activity of video content, and the relationship between the search points and bit allocation is modeled as the search-points model. Second, the grey level co-occurrence matrix and temporal perceptual information are used to evaluate the spatial and temporal texture complexity, and the relationship between the video content texture complexity and bit allocation is modeled as the texture-complexity model. Then, the search-points model and texture-complexity model are jointly employed to allocate the coding bits for the second and third layers of the HEVC hierarchical coding structure. Finally, the remaining coding bits of a group-of-pictures (GOP) are allocated to the first layer of HEVC coding structure. To evaluate the performance of the proposed algorithm, the RD performance and bitrate accuracy are used as evaluation criteria, and the experimental results show that when compared with the popularly used R-λ model–based bit allocation algorithm, the proposed algorithm achieves an average of -3.43% BDBR reduction and 0.13 dB BDPSNR gains with only 0.02% loss of bitrate accuracy.

Abstract This paper addresses the nontraditional but practically meaningful reversibility problem of mesh filtering. This reverse‐filtering approach (termed a DeFilter) seeks to recover the geometry of a set of filtered meshes to their artifact‐free status. To solve this scenario, we adapt cascaded normal regression (CNR) to understand the commonly used mesh filters and recover automatically the mesh geometry that was lost through various geometric operations. We formulate mesh defiltering by an extreme learning machine (ELM) on the mesh normals at an offline training stage and perform it automatically at a runtime defiltering stage. Specifically, (1) to measure the local geometry of a filtered mesh, we develop a generalized reverse Filtered Facet Normal Descriptor (grFND) in the consistent neighbors; (2) to map the grFNDs to the normals of the ground‐truth meshes, we learn a regression function from a set of filtered meshes and their ground‐truth counterparts; and (3) at runtime, we reversely filter the normals of a filtered mesh, using the learned regression function for recovering the lost geometry. We evaluate multiple quantitative and qualitative results on synthetic and real data to verify our DeFilter's performance thoroughly. From a practical point of view, our method can recover the lost geometry of denoised meshes without needing to know the exact filter used previously, and can act as a geometry‐recovery plugin for most of the state‐of‐the‐art methods of mesh denoising.

Traditional methods of annotating the sentiment of an unlabeled document are based on sentiment lexicons or machine learning algorithms, which have shown low computational cost or competitive performance. However, these methods ignore the semantic composition problem displaying in several ways such as negative reversing and intensification. In this paper, we propose a new method for sentiment classification using negative and intensive sentiment supplementary information, so as to exploit the linguistic feature of negative and intensive words in conjunction with the context information. Particularly, our method can solve the domain-specific problem without relying on the external sentiment lexicons. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed method.

Graphical models have been widely used to learn the conditional dependence structures among random variables. In many controlled experiments, such as the studies of disease or drug effectiveness, learning the structural changes of graphical models under two different conditions is of great importance. However, most existing graphical models are developed for estimating a single graph and based on a tacit assumption that there is no missing relevant variables, which wastes the common information provided by multiple heterogeneous data sets and underestimates the influence of latent/unobserved relevant variables. In this paper, we propose a joint differential network analysis (JDNA) model to jointly estimate multiple differential networks with latent variables from multiple data sets. The JDNA model is built on a penalized D-trace loss function, with group lasso or generalized fused lasso penalties. We implement a proximal gradient-based alternating direction method of multipliers to tackle the corresponding convex optimization problems. Extensive simulation experiments demonstrate that JDNA model outperforms state-of-the-art methods in estimating the structural changes of graphical models. Moreover, a series of experiments on several real-world data sets have been performed and experiment results consistently show that our proposed JDNA model is effective in identifying differential networks under different conditions.

Label-specific topics can be widely used for supporting personality psychology, aspectlevel sentiment analysis, and cross-domain sentiment classification.To generate labelspecific topics, several supervised topic models which adopt likelihood-driven objective functions have been proposed.However, it is hard for them to get a precise estimation on both topic discovery and supervised learning.In this study, we propose a supervised topic model based on the Siamese network, which can trade off label-specific word distributions with document-specific label distributions in a uniform framework.Experiments on realworld datasets validate that our model performs competitive in topic discovery quantitatively and qualitatively.Furthermore, the proposed model can effectively predict categorical or real-valued labels for new documents by generating word embeddings from a labelspecific topical space.

High-Efficiency Video Coding (HEVC) efficiently addresses the storage and transmit problems of high-definition videos, especially for 4K videos. The variable-size Prediction Units (PUs)--based Motion Estimation (ME) contributes a significant compression rate to the HEVC encoder and also generates a huge computation load. Meanwhile, high-level encoding complexity prevents widespread adoption of the HEVC encoder in multimedia systems. In this article, an adaptive fractional-pixel ME skipped scheme is proposed for low-complexity HEVC ME. First, based on the property of the variable-size PUs--based ME process and the video content partition relationship among variable-size PUs, all inter-PU modes during a coding unit encoding process are classified into root-type PU mode and children-type PU modes. Then, according to the ME result of the root-type PU mode, the fractional-pixel ME of its children-type PU modes is adaptively skipped. Simulation results show that, compared to the original ME in HEVC reference software, the proposed algorithm reduces ME encoding time by an average of 63.22% while encoding efficiency performance is maintained.

Cross-domain sentiment classification aims to tag sentiments for a target domain by labeled data from a source domain. Due to the difference between domains, the accuracy of a trained classifier may be very low. In this paper, we propose a boosting-based learning framework named TR-TrAdaBoost for cross-domain sentiment classification. We firstly explore the topic distribution of documents, and then combine it with the unigram TrAdaBoost. The topic distribution captures the domain information of documents, which is valuable for cross-domain sentiment classification. Experimental results indicate that TR-TrAdaBoost represents documents well and boost the performance and robustness of TrAdaBoost.

This paper addresses the task of cross-domain social emotion classification of online documents. The cross-domain task is formulated as using abundant labeled documents from a source domain and a small amount of labeled documents from a target domain, to predict the emotion of unlabeled documents in the target domain. Although several cross-domain emotion classification algorithms have been proposed, they require that feature distributions of different domains share a sufficient overlapping, which is hard to meet in practical applications. This paper proposes a novel framework, which uses the emotion distribution of training documents at the cluster level, to alleviate the aforementioned issue. Experimental results on two datasets show the effectiveness of our proposed model on cross-domain social emotion classification.

We have developed a method called skill2vec, which applies big data techniques to automatically analyze the learning data to discover skill relationship, leading to a more objective and data-informed decision making. Skill2vec is a neural network architecture which can transform a skill to a new vector space called embedding. The embedding can facilitate the comparison and visualization of different skills and their relationship. We conducted a pilot experiment using benchmark dataset to demonstrate the effectiveness of our method.

Social emotion classification aims to predict the aggregation of emotional responses embedded in online comments contributed by various users. Such a task is inherently challenging because extracting relevant semantics from free texts is a classical research problem. Moreover, online comments are typically characterized by a sparse feature space, which makes the corresponding emotion classification task very difficult. On the other hand, though deep neural networks have been shown to be effective for speech recognition and image analysis tasks because of their capabilities of transforming sparse low-level features to dense high-level features, their effectiveness on emotion classification requires further investigation. The main contribution of our work reported in this paper is the development of a novel model of semantically rich hybrid neural network (HNN) which leverages unsupervised teaching models to incorporate semantic domain knowledge into the neural network to bootstrap its inference power and interpretability. To our best knowledge, this is the first successful work of incorporating semantics into neural networks to enhance social emotion classification and network interpretability. Through empirical studies based on three real-world social media datasets, our experimental results confirm that the proposed hybrid neural networks outperform other state-of-the-art emotion classification methods.

With the rapid growth of massive online open courses (MOOCs) on the Web, it is essential to provide learners with appropriate assistance in courses and learning materials. The extant approaches of personalized course search mainly consider historical learnt and enrolled courses of learners. That is, those courses which are contently similar to previous courses in learner profiles will be highlighted in the ranking results of the personalized course search. However, these approaches mainly neglect two distinguished characteristics in this domain, which are (i) context-dependent: course search which is highly correlated with learner contexts, e.g., a learner may have the individual learning schedule of the courses to be retrieved depending on the temporal contexts; and (ii) knowledge-constrained: learners are more willing to search and enroll in the courses that they have sufficient pre-knowledge about. To incorporate these two domain characteristics of the personalized course search, we therefore present a novel approach based on hybrid learner profile in this paper. Furthermore, we conduct the experiments which compare the performance of different methods on a dataset to verify the effectiveness of the proposed method for the personalized course search.

Abstract Many automatic text summarization models have been developed in the last decades. Related research in information science has shown that human abstractors extract sentences for summaries based on the hierarchical structure of documents; however, the existing automatic summarization models do not take into account the human abstractor's behavior of sentence extraction and only consider the document as a sequence of sentences during the process of extraction of sentences as a summary. In general, a document exhibits a well‐defined hierarchical structure that can be described as fractals—mathematical objects with a high degree of redundancy. In this article, we introduce the fractal summarization model based on the fractal theory. The important information is captured from the source document by exploring the hierarchical structure and salient features of the document. A condensed version of the document that is informatively close to the source document is produced iteratively using the contractive transformation in the fractal theory. The fractal summarization model is the first attempt to apply fractal theory to document summarization. It significantly improves the divergence of information coverage of summary and the precision of summary. User evaluations have been conducted. Results have indicated that fractal summarization is promising and outperforms current summarization techniques that do not consider the hierarchical structure of documents.

Abstract Modern information retrieval systems use keywords within documents as indexing terms for search of relevant documents. As Chinese is an ideographic character‐based language, the words in the texts are not delimited by white spaces. Indexing of Chinese documents is impossible without a proper segmentation algorithm. Many Chinese segmentation algorithms have been proposed in the past. Traditional segmentation algorithms cannot operate without a large dictionary or a large corpus of training data. Nowadays, the Web has become the largest corpus that is ideal for Chinese segmentation. Although most search engines have problems in segmenting texts into proper words, they maintain huge databases of documents and frequencies of character sequences in the documents. Their databases are important potential resources for segmentation. In this paper, we propose a segmentation algorithm by mining Web data with the help of search engines. On the other hand, the Romanized pinyin of Chinese language indicates boundaries of words in the text. Our algorithm is the first to utilize the Romanized pinyin to segmentation. It is the first unified segmentation algorithm for the Chinese language from different geographical areas, and it is also domain independent because of the nature of the Web. Experiments have been conducted on the datasets of a recent Chinese segmentation competition. The results show that our algorithm outperforms the traditional algorithms in terms of precision and recall. Moreover, our algorithm can effectively deal with the problems of segmentation ambiguity, new word (unknown word) detection, and stop words.

Abstract Based on the salient features of the documents, automatic text summarization systems extract the key sentences from source documents. This process supports the users in evaluating the relevance of the extracted documents returned by information retrieval systems. Because of this tool, efficient filtering can be achieved. Indirectly, these systems help to resolve the problem of information overloading. Many automatic text summarization systems have been implemented for use with different languages. It has been established that the grammatical and lexical differences between languages have a significant effect on text processing. However, the impact of the language differences on the automatic text summarization systems has not yet been investigated. The authors provide an impact analysis of language difference on automatic text summarization. It includes the effect on the extraction processes, the scoring mechanisms, the performance, and the matching of the extracted sentences, using the parallel corpus in English and Chinese as the tested object. The analysis results provide a greater understanding of language differences and promote the future development of more advanced text summarization techniques.

In this paper, we introduce the fractal summarization model based on the fractal theory. In fractal summarization, the important information is captured from the source text by exploring the hierarchical structure and salient features of the document. A condensed version of the document that is informatively close to the original is produced iteratively using the contractive transformation in the fractal theory. User evaluation has shown that fractal summarization outperforms traditional summarization.

Wireless access with mobile (or handheld) devices is a promising addition to the WWW and traditional electronic business. Mobile devices provide convenience and portable access to the huge information space on the Internet without requiring users to be stationary with network connection. However, the limited screen size, narrow network bandwidth, small memory capacity and low computing power are the shortcomings of handheld devices. Loading and visualizing large documents on handheld devices become impossible. The limited resolution restricts the amount of information to be displayed. The download time is intolerably long. In this paper, we introduce the fractal summarization model for document summarization on handheld devices. Fractal summarization is developed based on the fractal theory. It generates a brief skeleton of summary at the first stage, and the details of the summary on different levels of the document are generated on demands of users. Such interactive summarization reduces the computation load in comparing with the generation of the entire summary in one batch by the traditional automatic summarization, which is ideal for wireless access. Three-tier architecture with the middle-tier conducting the major computation is also discussed. Visualization of summary on handheld devices is also investigated.