In the field of Artificial Intelligence (AI), the data used to train models may contain private information that could potentially be exposed in the model’s output. Machine Unlearning (MU) has emerged as a promising solution for removing private or obsolete data from trained models, along with their influence, thereby enforcing the “right to be forgotten” under the General Data Protection Regulation (GDPR). However, achieving a balance between privacy guarantee and model performance remains a fundamental challenge. This paper contributes to the field of AI by presenting an empirical evaluation of key families, i.e., data deletion, data perturbation, and model update of MU for privacy preservation, focusing on their impact on both classification accuracy and privacy in deep learning (DL) models. The study assesses changes in the classification performance of the convolutional neural network (CNN) architecture and the long short-term memory (LSTM) and bidirectional LSTM (Bi-LSTM) recurrent neural network (RNN) architectures when used with data deletion, data perturbation, and model update families of MU. This study also assesses these architectures’ susceptibility to membership inference attacks (MIA) before and after unlearning on PPG-DaLiA and MHEALTH (Mobile HEALTH) datasets, providing a quantitative measure of privacy leakage. Experimental results show that model update techniques offer more scalable alternatives to data deletion and perturbation, though they introduce varying levels of privacy leakage risk. In doing so, this research highlights the strengths and limitations of current targeted unlearning methods and underscores the need for more efficient and flexible approaches to privacy protection in DL models.

With the continuous growth in the number of parameters of the Transformer-based pretrained language models (PLMs), particularly the emergence of large language models (LLMs) with billions of parameters, many natural language processing (NLP) tasks have demonstrated remarkable success. However, the enormous size and computational demands of these models pose significant challenges for adapting them to specific downstream tasks, especially in environments with limited computational resources. Parameter-Efficient Fine-Tuning (PEFT) offers an effective solution by reducing the number of fine-tuning parameters and memory usage while achieving comparable performance to full fine-tuning. The demands for fine-tuning PLMs, especially LLMs, have led to a surge in the development of PEFT methods, as depicted in Fig. 1. In this paper, we present a comprehensive and systematic review of PEFT methods for PLMs. We summarize these PEFT methods, discuss their applications, and outline future directions. Furthermore, extensive experiments are conducted using several representative PEFT methods to better understand their effectiveness in parameter efficiency and memory efficiency. By offering insights into the latest advancements and practical applications, this survey serves as an invaluable resource for researchers and practitioners seeking to navigate the challenges and opportunities presented by PEFT in the context of PLMs.

The conditional coding paradigm is widely used in learned video compression, which shows superior performance in capturing redundancies within a large context space. However, existing Conditional coding-based Learned Video Compression (C-LVC) methods ignore that the predicted motion vectors usually contain large uncertainty due to complex motions, occlusions, etc., which consequently decrease the accuracy of the generated temporal contexts. In addition, existing C-LVC methods have a weak ability to mine diverse dependencies within the context space, which are closely related to the coding efficiency. To address these issues, an efficient temporal redundancy mining method is proposed to improve the coding efficiency of C-LVC in this paper. To generate accurate temporal contexts, a Long Short-Term Motion Aggregation (LSTMA) model is proposed, in which an LSTMA-based motion estimation module is developed to capture both current and aggregated long short-term motion information to reduce the uncertainty of predicted motion vectors. Based on the dual motion information, an LSTMA-based temporal context mining module is developed to exploit the aggregated long short-term motion information and increase the accuracy of the generated temporal contexts. In order to fully eliminate spatial-temporal redundancies in a video, a Global-Local Information Decorrelation Module (GLIDM)-based context codec is proposed, in which the GLIDM is designed based on the visual state space block (namely vmamba), the residual block, and the squeeze-and-excitation block to effectively capture long-range, short-range spatial-temporal dependencies and channel-wise dependencies. Experimental results demonstrate that our proposed method can effectively improve the coding performance of C-LVC, and outperforms other state-of-the-art LVC methods.

Implicit sentiment analysis (ISA) presents significant challenges due to the absence of salient cue words. Previous methods have struggled with insufficient data and limited reasoning capabilities to infer underlying opinions. Integrating multi-task learning (MTL) with large language models (LLMs) offers the potential to enable models of varying sizes to reliably perceive and recognize genuine opinions in ISA. However, existing MTL approaches are constrained by two sources of uncertainty:data-level uncertainty, arising from hallucination problems in LLM-generated contextual information, andtask-level uncertainty, stemming from the varying capacities of models to process contextual information. To handle these uncertainties, we proposeMT-ISA, a novel MTL framework that enhances ISA by leveraging the generation and reasoning capabilities of LLMs through automatic weight learning (AWL). Specifically,MT-ISAconstructs auxiliary tasks using generative LLMs to supplement sentiment elements and incorporates automatic MTL to fully exploit auxiliary data. We introduce data-level and task-level AWL, which dynamically identify relationships and prioritize more reliable data and critical tasks, enabling models of varying sizes to adaptively learn fine-grained weights based on their reasoning capabilities. Three strategies are investigated for data-level AWL, which are integrated with homoscedastic uncertainty for task-level AWL. Extensive experiments reveal that models of varying sizes achieve an optimal balance between primary prediction and auxiliary tasks inMT-ISA. This underscores the effectiveness and adaptability of our approach.

Multi-scale feature compression is essential in machine vision tasks for reducing storage and transmission costs while maintaining task performance. However, existing multi-scale feature compression methods fail to effectively extract and aggregate the local and global correlations of multi-scale features, resulting in incomplete elimination of feature redundancies. Moreover, these multi-scale feature compression methods mainly rely on mean square error-based loss functions to optimize signal fidelity, but they fail to adequately preserve semantic information critical to machine vision tasks. To address these issues, a Multi-receptive-field Convolutional Neural Network (MCNN)-based multi-scale feature compression method is proposed in this paper, which not only achieves compact fusion of multi-scale features but also enhances the semantic fidelity of reconstructed features. To effectively eliminate feature redundancies, a multi-receptive-field-based feature fusion module is designed for capturing both local and global correlations in multi-scale features. To enhance the quality of the reconstructed features, a cosine similarity-based multi-fidelity loss function is developed by considering both signal and semantic fidelity. Extensive experiments on the object detection and instance segmentation tasks show that the proposed MCNN outperforms the state-of-the-art multi-scale feature compression methods in terms of compression efficiency. The source code of our proposed MCNN is publicly available athttps://github.com/NUIST-Videocoding/MCNN.git.

Searching by image is popular yet still challenging in e-commerce due to the extensive interference arising from (i) data variations (e.g., background, pose, visual angle, brightness) of real-world captured images and (ii) similar images in the query dataset. This article studies a practically meaningful problem of beauty product retrieval (BPR) by neural networks. We broadly extract different types of image features and raise an intriguing question that whether these features are beneficial to (i) suppress data variations of real-world captured images and (ii) distinguish one image from others which look very similar but are intrinsically different beauty products in the dataset, therefore leading to an enhanced capability of BPR. To answer it, we present a novel v ariable-attention neural network to understand the combination of m ultiple features (termed VM-Net) of beauty product images. Considering that there are few publicly released training datasets for BPR, we establish a new dataset with more than one million images classified into more than 20K categories to improve both the generalization and anti-interference abilities of VM-Net and other methods. We verify the performance of VM-Net and its competitors on the benchmark dataset Perfect-500K, where VM-Net shows clear improvements over the competitors in terms of \(MAP@7\) . The source code and dataset will be released upon publication.

Users often assume that large language models (LLMs) share their cognitive alignment of context and intent, leading them to omit critical information in question-answering (QA) and produce ambiguous queries.Responses based on misaligned assumptions may be perceived as hallucinations.Therefore, identifying possible implicit assumptions is crucial in QA.To address this fundamental challenge, we propose Conditional Ambiguous Question-Answering (CondAmbigQA), a benchmark comprising 2,000 ambiguous queries and condition-aware evaluation metrics 1 .Our study pioneers "conditions" as explicit contextual constraints that resolve ambiguities in QA tasks through retrievalbased annotation, where retrieved Wikipedia fragments help identify possible interpretations for a given query and annotate answers accordingly.Experiments demonstrate that models considering conditions before answering improve answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided.These results highlight that apparent hallucinations may stem from inherent query ambiguity rather than purely model failure, and demonstrate the effectiveness of condition reasoning in QA, providing researchers with tools for rigorous evaluation.

High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). However, most existing methods leverage fixed-length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embedding size, enhancing inference efficiency. The code is available at https://github.com/SeanLee97/AnglE/blob/main/README_ESE.md.

Multimodal sentiment analysis uses text, visual, and audio data to assess user sentiment, while both the discrimination power of modalities and sample distributions over categories remain imbalanced in practice. To address these challenges, we propose a Multi-grained Perception Model with Perceived Contrastive Focal loss, denoted MGSA1. More specifically, we design a Multi-grained Cross-modal Attention Perception (MCP) module, which employs coarse-grained and fine-grained cross-modal attention to deeply explore the complementary semantics between modalities, thereby modeling sentiment polarity and intensity by fusing text-video and text-audio data, respectively. Modeling sentiment polarity and intensity helps alleviate feature interference between modalities due to their differing discriminative power. Furthermore, the Perceived Contrastive Focal (PCF) loss is designed to address the challenge of unbalanced samples. We enhance the focal loss by incorporating inverse document frequency to dynamically weight samples within each class. Furthermore, information noise contrastive estimation is introduced to replace the class probability predictions in the enhanced focal loss, thereby more effective differentiation between positive and negative samples. Experiments on the MOSI and MOSEI datasets demonstrate that MGSA outperforms all baselines across a range of metrics.

Knowledge Graphs (KGs) have revolutionized structured knowledge representation, yet their capacity to model real-world complexity and heterogeneity remains fundamentally constrained. The emerging paradigm of Multi-View Knowledge Graphs (MVKGs) addresses this gap through multi-view learning, but existing research lacks systematic integration. This survey provides the first systematic consolidation of MVKG methodologies, with four pivotal contributions: 1) The first unified taxonomy of view generation paradigms that rigorously categorizes view into four types: structure, semantic, representation, and knowledge & modality; 2) A novel methodological typology for view fusion that systematically classifies techniques by fusion targets (feature, decision, and hybrid); 3) Task-centric application mapping that bridges theoretical MVKG constructs to node/link/graph-level downstream tasks; 4) A forward-looking roadmap identifying underexplored challenges. By unifying fragmented methodologies and formalizing MVKG design principles, this survey serves as a roadmap for advancing KG versatility in complex AI-driven scenarios. In doing so, it paves the way for more efficient knowledge integration, enhanced decision-making, and cross-domain learning in real-world applications.

Evaluating the visual quality of autostereoscopic 3D displays is crucial for quantifying their stereoscopic viewing experience and optimizing display performance. Existing quality evaluation methods primarily predict the visual quality of autostereoscopic 3D displays by indirectly learning display parameter information from image content. However, these methods fail to explicitly model the relationship between display parameters and visual quality, thereby limiting their prediction accuracy. To address this problem, a Multimodal Parameter Perception Network (MPPNet)-based visual quality assessment method is proposed in this paper, which treats display parameters as textual modalities to explicitly establish their relationship with visual quality. To effectively understand the semantic information of display parameter texts, a Contrastive Language-Image Pretraining (CLIP)-based adaptive text encoder is proposed to generate robust semantic representations by capturing both general and domain-specific semantic embeddings. In parallel, a hierarchical vision encoder is adopted to extract visual representations from display images, which simulates the human binocular perception by capturing multi-level visual features from the left and right views. To achieve comprehensive cross-modal interaction, a mamba-based cross-modal fusion module is proposed to fuse textual and visual representations of display parameters by capturing both shallow and deep correlations. Extensive experimental results demonstrate that the proposed MPPNet achieves state-of-the-art performance in evaluating the visual quality of autostereoscopic 3D displays.

The application of metric learning in recommender systems has gained considerable attention, particularly for enabling supervised training through Euclidean distance. However, conventional collaborative translational metric learning typically represents each user and item as a single point and projects collaborative signals onto a specific translation vector within a uniform space. This approach fails to explicitly capture the high-order semantics and dynamic nature of user-item interactions derived from implicit user behavior. Given the extensive research on graph neural networks (GNNs), a natural extension is to incorporate high-order information into the generation of translation vectors. Yet, our empirical findings reveal that the performance is not as expected. In this paper, we systematically investigate the limitations of existing methods and attribute the issue to semantic confusion. Motivated by this finding, we propose a novel framework that integrates bilateral metric learning with a causal graph network. Specifically, we construct a structural causal model (SCM) to disentangle user behavior into causal features and confounders. To mitigate the influence of these confounders, we propose a conditional intervention module that generates confounder-specific conditions for the target. Moreover, inspired by this conditional intervention mechanism, we develop an additional module conditioned on user behavior to enhance the diversity of the recommended list, recognizing that diversity is also a crucial metric for evaluating recommendation quality. We conducted a series of experiments to validate the effectiveness of our proposed method. The results demonstrate that our approach achieves state-of-the-art performance in terms of both recommendation accuracy and diversity.

Classifying graph-structured data presents significant challenges due to the diverse features of nodes and edges and their complex relationships. While Graph Neural Networks (GNNs) are widely used for graph prediction tasks, their performance is often hindered by these intricate dependencies. Leveraging causality holds potential in overcoming these challenges by identifying causal links among features, thus enhancing GNN classification performance. However, depending solely on adjacency matrices or attention mechanisms, as commonly studied in causal prediction research, is insufficient for capturing the complex interactions among features. To address these challenges, we present HebCGNN , a Hebbian-enabled Causal GNN classification model that incorporates dynamic impact valuing . Our method creates a robust framework that prioritizes causal elements in prediction tasks. Extensive experiments on seven publicly available datasets across diverse domains demonstrate that HebCGNN outperforms state-of-the-art models.

Reinforcement learning has been increasingly applied in monitoring applications because of its ability to learn from previous experiences and make adaptive decisions. However, existing machine learning-based health monitoring applications are mostly supervised learning algorithms, trained on labels, and they cannot make adaptive decisions in an uncertain, complex environment. This study proposes a novel and generic system, predictive deep reinforcement learning (PDRL), with multiple RL agents in a time series forecasting environment. The proposed generic framework accommodates virtual Deep Q Network (DQN) agents to monitor predicted future states of a complex environment with a well-defined reward policy so that the agent learns existing knowledge while maximizing their rewards. In the evaluation process of the proposed framework, three DRL agents were deployed to monitor a subject’s future heart rate, respiration, and temperature predicted using a BiLSTM model. With each iteration, the three agents were able to learn the associated patterns, and their cumulative rewards gradually increased. It outperformed the baseline models for all three monitoring agents. The proposed PDRL framework achieves state-of-the-art performance in time series forecasting by effectively integrating reinforcement learning agents with deep learning-based prediction. The proposed DRL agents and deep learning model in the PDRL framework are customized to enable transfer learning in other forecasting applications like traffic and weather, and monitor their states. The PDRL framework is able to learn the future states of the traffic and weather forecasting, and the cumulative rewards are gradually increasing over each episode.

Under the context of the increasing social demand for fine-grained sentiment analysis (SA), implicit sentiment analysis (ISA) poses a significant challenge owing to the absence of salient cue words in expressions. Thus, reliable reasoning is required to understand how sentiment is evoked, enabling the identification of implicit sentiments. In the era of large language models (LLMs), encoder-decoder (ED) LLMs have emerged as popular backbone models for SA applications, given their impressive text comprehension and reasoning capabilities across diverse tasks. In comparison, decoder-only (DO) LLMs exhibit superior natural language generation and in-context learning capabilities. However, their responses may contain misleading or inaccurate information. To accurately identify implicit sentiments with reliable reasoning, this study introduces a two-stage reasoning framework named Reasoning and Verification for Implicit Sentiment Analysis (RVISA), which leverages the generation ability of DO LLMs and reasoning ability of ED LLMs to train an enhanced reasoner. The framework involves three-hop reasoning prompting to explicitly furnish sentiment elements as cues. The generated rationales are then used to fine-tune an ED LLM into a skilled reasoner. Additionally, we develop a straightforward yet effective answer-based verification mechanism to ensure the reliability of reasoning learning. Evaluation of the proposed method on two benchmark datasets demonstrates that it achieves state-of-the-art performance in ISA.

Infrared small target detection (ISTD) is a challenging task due to the low contrast and small size of the targets, which are often affected by complex backgrounds. UNet and its variants, known for their encoder–decoder structures, are widely used in such tasks since they can capture both local and global features. However, a significant drawback of UNet-based networks is the irreversible loss of crucial local features during downsampling, leading to missed detections and false positives, especially for small targets. Compared to other architectures like feature pyramid networks, UNet provides a more symmetric and efficient structure, allowing it to handle dense pixel-wise predictions effectively. However, standard UNet models still struggle to fully retain small target details, motivating the need for further improvements. To address this issue, we propose HintU, a novel network to recover the local features lost by various UNet-based methods for effective ISTD. HintU has two key contributions. First, it introduces the “Hint” mechanism for the first time, i.e., leveraging the prior knowledge of target locations to highlight critical local features. Second, it improves the mainstream UNet-based architecture to preserve target pixels even after downsampling. HintU can shift the focus of various networks (e.g., vanilla UNet, UNet++, UIUNet, MiM+, and HCFNet) from the irrelevant background pixels to a more restricted area from the beginning. Experimental results on three datasets NUDT-SIRST, SIRSTv2, and IRSTD1K demonstrate that HintU enhances the performance of existing methods with only an additional 1.88-ms cost (on RTX Titan). Additionally, the explicit constraints of HintU enhance the generalization ability of UNet-based methods. Code is available athttps://github.com/Wuzhou-Quan/HintU.

Knowledge-based visual question answering (KB-VQA) requires reasoning about the visual grounding relations between the images and questions by incorporating external knowledge. Existing works typically retrieve knowledge from knowledge graphs by leveraging global multimodal representations of image–text pairs for graph convolution, which neglect contextual clues at hop granularity, resulting in suboptimal spreading and leveraging of contextual information. To this end, we propose a multi-hop graph reasoning network (MGRN) for KB-VQA, which consists of a knowledge graph constructor (KGC) module, a semantic-instructed graph reasoning (SGR) module, and an answering module. MGRN exploits multimodal semantics from given images and questions as instructions for graph reasoning to obtain the knowledge representation from either the scene graph or knowledge base. Specifically, KGC fuses the scene graph with triplets from ConceptNet and Comet to construct a contextual knowledge graph for retrieving knowledge representation. Furthermore, SGR conducts multi-hop graph reasoning to select top- K knowledge items for answering by passing and filtering interplay messages on contextual knowledge graphs under the guidance of multimodal semantic representation. Extensive experiments conducted on two public datasets show the effectiveness and outperformance of our method.

Point cloud analysis, arising from computer graphics, remains a fundamental but challenging problem, mainly due to the non-Euclidean property of point cloud data modality. With the snap increase in the amount and breadth of related research in deep learning for graphs, many important works come in the form of graphs representing the point clouds. In this paper, we present a sampling adaptive graph convolutional network that combines the powerful representation ability of random walk subgraph searching and the essential success of the Fisher vector. Extending from those existing graph representation learning or embedding methods with multi-hop neighbor random searching, we sample multi-scale walk fields by using asteerableexploration-exploitationsecond order random walk, which endows our model with the most flexibility compared with the original first order random walk. To encode each-scale walk field consisting of several walk paths, specifically, we characterize these paths of walk field by Gaussian mixture models (GMMs) so as to better analogize the standard CNNs on Euclidean modality. Each Gaussian component implicitly defines a direction and all of them properly encode thespatial layoutof walk fields after the gradient projecting to the space of Gaussian parameters, i.e. the Fisher vectors. Thereby, we introduce and name our deep graph convolutional network asPointFisher. Comprehensive evaluations on several public datasets well demonstrate the superiority of our proposed learning method over other state-of-the-arts for point cloud classification and segmentation.

Machine unlearning (MU) is gaining increasing attention due to the need to remove or modify predictions made by machine learning (ML) models. While training models have become more efficient and accurate, the importance of unlearning previously learned information has become increasingly significant in fields such as privacy, security, and ethics. This article presents a comprehensive survey of MU, covering current state-of-the-art techniques and approaches, including data deletion, perturbation, and model updates. In addition, commonly used metrics and datasets are presented. This article also highlights the challenges that need to be addressed, including attack sophistication, standardization, transferability, interpretability, training data, and resource constraints. The contributions of this article include discussions about the potential benefits of MU and its future directions. Additionally, this article emphasizes the need for researchers and practitioners to continue exploring and refining unlearning techniques to ensure that ML models can adapt to changing circumstances while maintaining user trust. The importance of unlearning is further highlighted in making artificial intelligence (AI) more trustworthy and transparent, especially with the growing importance of AI across various domains that involve large amounts of personal user data.

Recent studies have extensively explored the causal connections between emotions and their underlying causes in textual data. Most research aims to identify clauses within documents that are causally related. However, these studies have overlooked the fact that such causal relationships are often context-dependent and valid only within specific contextual clauses. To bridge this gap, we present a novel task of determining the presence of a valid causal relationship between a given pair of emotion and cause clauses in different contexts, while also identifying the specific contextual clauses involved. Since this task is novel and lacks an existing dataset for testing, we manually annotate a benchmark dataset to obtain labels for our task and classify the types of context clauses, which can also be beneficial for other applications. By leveraging negative sampling, we create a balanced final dataset that includes documents with and without causal relationships. Building upon this dataset, we propose an end-to-end multi-task framework that incorporates two innovative modules aimed at achieving the objectives of our task. We introduce a context masking module to identify the contextual clauses that contribute to causal relationships and a prediction aggregation module to refine predictions by determining the reliance of emotion and cause clauses on specific contextual clauses. Extensive comparative experiments and ablation studies validate the effectiveness and robustness of our proposed framework. The annotated dataset provides a novel way for exploring complex reasoning in causal analysis.

Abstract The expansion of Graph Neural Networks (GNNs) has highlighted the importance of evaluating their performance in real-world scenarios. However, existing evaluation frameworks often overlook the integration of causality, a critical component that is essential for more robust evaluation of GNNs. To address this gap, we present a benchmark study that systematically compares standard and causal GNN models with a focus on classification tasks. Our analysis encompasses a careful selection of nine GNN models across seven diverse datasets that span three distinct domains. The results reveal the following: I) Causality-enhanced GNNs consistently outperform their traditional counterparts in graph classification tasks; II) Models integrating causal features exhibit greater generalizability across varied datasets; and III) Incorporation of causal elements significantly improves the predictive accuracy of GNNs. These findings highlight the importance of embedding causality in the evaluation and development of GNNs for improved performance and application.

Many real-world optimization problems are inherently constrained, presenting significant challenges to the application of evolutionary algorithms. Successfully managing these constraints while simultaneously optimizing the objective function requires a considerable degree of population diversity. To address this, we have developed a methodology that effectively combines an$\varepsilon$-constraint-handling method with a niching technique. The$\varepsilon$-constraint- handling method is specifically designed to manage constraints, while the niching technique aims to preserve pop-ulation diversity. In our approach, a constrained optimization problem is transformed into a tri-objective optimization challenge, introducing two additional objectives: the density objective and the overall constraint objective. The density objective is a particularly innovative aspect of our method, as it prolongs the survival of promising yet infeasible solutions. This prolongation aids the evolutionary search in converging towards the feasible region from various directions, thereby increasing the chances of identifying optimal solutions. Moreover, an improved gradient repair mutation strategy, based on a successful information reuse approach, is implemented to further refine promising solutions. To evaluate the effectiveness of our method, we tested it on 30 real-world constrained optimization problems from the CEC 2020 benchmark test suite. The results demonstrate that our approach either exceeds or is equivalent to the performance of current state-of-the-art constrained optimization algorithms.

Recently, there has been a surge in face personalization techniques, benefiting from the advanced capabilities of pretrained text-to-image diffusion models. Among these, a notable method is Textual Inversion, which generates personalized images by inverting given images into textual embeddings. However, methods based on Textual Inversion still struggle with balancing the tradeoff between reconstruction quality and editability. In this study, we examine this issue through the lens of initialization. Upon closely examining traditional initialization methods, we identified a significant disparity between the initial and learned embeddings in terms of both scale and orientation. The scale of the learned embedding can be up to 100 times greater than that of the initial embedding. Such a significant change in the embedding could increase the risk of overfitting, thereby compromising the editability. Driven by this observation, we introduce a novel initialization method, termed Cross Initialization, that significantly narrows the gap between the initial and learned embeddings. This method not only improves both reconstruction and editability but also reduces the optimization steps from 5,000 to 320. Furthermore, we apply a regularization term to keep the learned embedding close to the initial embedding. We show that when combined with Cross Initialization, this regularization term can effectively improve editability. We provide comprehensive empirical evidence to demonstrate the superior performance of our method compared to the baseline methods. Notably, in our experiments, Cross Initialization is the only method that successfully edits an individual's facial expression. Additionally, a fast version of our method allows for capturing an input image in roughly 26 seconds, while surpassing the baseline methods in terms of both reconstruction and editability. Code is available at https://github.com/lyuPang/CrossInitialization.

In this paper, we propose a multimodal contrastive network with unbiased distillation (MCUD) for knowledge-based VQA, which consists of contrastive sample construction (CSC), unbiased contrastive distillation (UCD), and hierarchical reasoning (HR) modules. Specifically, CSC constructs contrastive samples by transforming the knowledge corpus and adopting entropy-adjusted answer frequencies to identify the unbiased samples. Additionally, UCD employs a dual-branch feature extractor architecture to separately encode the knowledge corpus and image-question pairs into a shared embedding space. Subsequently, the knowledge-driven contrastive learning is supposed to bridge the modality gap between the knowledge corpus and the cross-modality of image-question pairs. Furthermore, the teacher model in UCD utilizes different distillation strategies for biased and unbiased samples, guiding the student model to establish a generalized unbiased representation. Finally, the HR module conducts chain-of-thought outputs, sequentially locates the contextual sentences in the knowledge corpus, generates the rationale, and infers the answer. Extensive experiments with two datasets, E-VQA and ScienceQA, demonstrate the effectiveness and outperformance of our method.

The increasing single-cell RNA sequencing (scRNA-seq) data enable researchers to explore cellular heterogeneity and gene expression profiles, offering a high-resolution view of the transcriptome at the single-cell level. However, the dropout events, which are often present in scRNA-seq data, remaining challenges for downstream analysis. Although a number of studies have been developed to recover single-cell expression profiles, their performance may be hindered due to not fully exploring the inherent relations between genes. To address the issue, we propose scDTL, a deep transfer learning based approach for scRNA-seq data imputation by harnessing the bulk RNA-sequencing information. We firstly employ a denoising autoencoder trained on bulk RNA-seq data as the initial imputation model, and then leverage a domain adaptation framework that transfers the knowledge learned by the bulk imputation model to scRNA-seq learning task. In addition, scDTL employs a parallel operation with a 1D U-Net denoising model to provide gene representations of varying granularity, capturing both coarse and fine features of the scRNA-seq data. Finally, we utilize a cross-channel attention mechanism to fuse the features learned from the transferred bulk imputation model and U-Net model. In the evaluation, we conduct extensive experiments to demonstrate that scDTL could outperform other state-of-the-art methods in the quantitative comparison and downstream analyses.

Abstract Topic taxonomy discovery aims at uncovering topics of different abstraction levels and constructing hierarchical relations between them. Unfortunately, most prior work can hardly model semantic scopes of words and topics by holding the Euclidean embedding space assumption. What’s worse, they infer asymmetric hierarchical relations by symmetric distances between topic embeddings. As a result, existing methods suffer from problems of low-quality topics at high abstraction levels and inaccurate hierarchical relations. To alleviate these problems, this paper develops a Box embedding-based Topic Model (BoxTM) that maps words and topics into the box embedding space, where the asymmetric metric is defined to properly infer hierarchical relations among topics. Additionally, our BoxTM explicitly infers upper-level topics based on correlation between specific topics through recursive clustering on topic boxes. Finally, extensive experiments validate high-quality of the topic taxonomy learned by BoxTM.

While the wisdom of training an image dehazing model on synthetic hazy data can alleviate the difficulty of collecting real-world hazy/clean image pairs, it brings the well-known domain shift problem. From a different yet new perspective, this paper explores contrastive learning with an adversarial training effort to leverage unpaired real-world hazy and clean images, thus alleviating the domain shift problem and enhancing the network's generalization ability in real-world scenarios. We propose an effective unsupervised contrastive learning paradigm for image dehazing, dubbed UCL-Dehaze. Unpaired real-world clean and hazy images are easily captured, and will serve as the important positive and negative samples respectively when training our UCL-Dehaze network. To train the network more effectively, we formulate a new self-contrastive perceptual loss function, which encourages the restored images to approach the positive samples and keep away from the negative samples in the embedding space. Besides the overall network architecture of UCL-Dehaze, adversarial training is utilized to align the distributions between the positive samples and the dehazed images. Compared with recent image dehazing works, UCL-Dehaze does not require paired data during training and utilizes unpaired positive/negative data to better enhance the dehazing performance. We conduct comprehensive experiments to evaluate our UCL-Dehaze and demonstrate its superiority over the state-of-the-arts, even only 1,800 unpaired real-world images are used to train our network. Source code is publicly available at https://github.com/yz-wang/UCL-Dehaze.

High dynamic range (HDR) video offers a more realistic visual experience than standard dynamic range (SDR) video, while introducing new challenges to both compression and transmission. Rate control is an effective technology to overcome these challenges, and ensure optimal HDR video delivery. However, the rate control algorithm in the latest video coding standard, versatile video coding (VVC), is tailored to SDR videos, and does not produce well coding results when encoding HDR videos. To address this problem, a data-driven λ -domain rate control algorithm is proposed for VVC HDR intra frames in this paper. First, the coding characteristics of HDR intra coding are analyzed, and a piecewise R- λ model is proposed to accurately determine the correlation between the rate (R) and the Lagrange parameter λ for HDR intra frames. Then, to optimize bit allocation at the coding tree unit (CTU)-level, a wavelet-based residual neural network (WRNN) is developed to accurately predict the parameters of the piecewise R- λ model for each CTU. Third, a large-scale HDR dataset is established for training WRNN, which facilitates the applications of deep learning in HDR intra coding. Extensive experimental results show that our proposed HDR intra frame rate control algorithm achieves superior coding results than the state-of-the-art algorithms. The source code of this work will be released at https://github.com/TJU-Videocoding/WRNN.git.

Personalized clinical decision support systems are increasingly being adopted due to the emergence of data-driven technologies, with this approach now gaining recognition in critical care. The task of incorporating diverse patient conditions and treatment procedures into critical care decision-making can be challenging due to the heterogeneous nature of medical data. Advances in Artificial Intelligence (AI), particularly Reinforcement Learning (RL) techniques, enables the development of personalized treatment strategies for severe illnesses by using a learning agent to recommend optimal policies. In this study, we propose a Deep Reinforcement Learning (DRL) model with a tailored reward function and an LSTM-GRU-derived state representation to formulate optimal treatment policies for vasopressor administration in stabilizing patient physiological states in critical care settings. Using an ICU dataset and the Medical Information Mart for Intensive Care (MIMIC-III) dataset, we focus on patients with Acute Respiratory Distress Syndrome (ARDS) that has led to Sepsis, to derive optimal policies that can prioritize patient recovery over patient survival. Both the DDQN ( RepDRL-DDQN ) and Dueling DDQN ( RepDRL-DDDQN ) versions of the DRL model surpass the baseline performance, with the proposed model’s learning agent achieving an optimal learning process across our performance measuring schemes. The robust state representation served as the foundation for enhancing the model’s performance, ultimately providing an optimal treatment policy focused on rapid patient recovery.

Monocular depth estimation (MDE) remains a fundamental yet not well-solved problem in computer vision. Current wisdom of MDE often achieves blurred or even indistinct depth boundaries, degenerating the quality of vision-based intelligent transportation systems. This paper presents an edge-enhanced vision transformer bins network for monocular depth estimation, termed eViTBins. eViTBins has three core modules to predict monocular depth maps with exceptional smoothness, accuracy, and fidelity to scene structures and object edges. First, a multi-scale feature fusion module is proposed to circumvent the loss of depth information at various levels during depth regression. Second, an image-guided edge-enhancement module is proposed to accurately infer depth values around image boundaries. Third, a vision transformer-based depth discretization module is introduced to comprehend the global depth distribution. Meanwhile, unlike most MDE models that rely on high-performance GPUs, eViTBins is optimized for seamless deployment on edge devices, such as NVIDIA Jetson Nano and Google Coral SBC, making it ideal for real-time intelligent transportation systems applications. Extensive experimental evaluations corroborate the superiority of eViTBins over competing methods, notably in terms of preserving depth edges and global depth representations.

The acquisition of both local and global features from irregular point clouds is crucial for 3D object detection (3DOD). Current mainstream 3D detectors neglect significant local features during pooling operations or disregard many global features of the overall scene context. This paper proposes new techniques for simultaneously learning local-global features of scene point clouds to enhance 3DOD. Specifically, we propose an efficient 3DOD network in indoor point clouds, named SimLOG, which utilizes simultaneous local-global feature learning. SimLOG has two main contributions: a Dynamic Points Interaction (DPI) module to recover local features lost during pooling, and a Global Context Aggregation(GCA) module to aggregate multi-scale features from various layers of the encoder to improve scene context awareness. Unlike traditional local-global feature learning methods, our DPI and GCA modules are integrated into a single feature learning module, making it easily detachable and able to be incorporated into existing 3DOD networks to enhance their performance. SimLOG demonstrates superior performance over twenty competitors in terms of detection accuracy and robustness on both the SUN RGB-D and ScanNet V2 datasets. Specifically, SimLOG boosts the baseline VoteNet by 8.1% of [email protected] on ScanNet V2 and by 3.9% of [email protected] on SUN RGB-D. Code is publicly available athttps://github.com/chenbaian-cs/SimLOG.

Being the largest Initial Coin Offering project, EOSIO has attracted great interest in cryptocurrency markets. Despite its popularity and prosperity (e.g., 26,311,585,008 token transactions occurred from June 8, 2018 to Aug. 5, 2020), there is almost no work investigating the EOSIO token ecosystem. To fill this gap, we are the first to conduct a systematic investigation of the EOSIO token ecosystem by conducting a comprehensive graph analysis of the entire on-chain EOSIO data (nearly 135 million blocks). We construct token-creator graphs, token-contract creator graphs, token-holder graphs, and token-transfer graphs to characterize token creators, holders, and transfer activities. Through graph analysis, we have obtained many insightful findings and observed some abnormal trading patterns. Moreover, we propose a fake-token detection algorithm to identify tokens generated by fake users or fake transactions and analyze their corresponding manipulation behaviors. Evaluation results also demonstrate the effectiveness of our algorithm.

Machine Unlearning, a pivotal field addressing data privacy in machine learning, necessitates efficient methods for the removal of private or irrelevant data. In this context, significant challenges arise, particularly in maintaining privacy and ensuring model efficiency when managing outdated, private, and irrelevant data. Such data not only compromises model accuracy but also burdens computational efficiency in both learning and unlearning processes. To mitigate these challenges, we introduce a novel framework: Attention-based Machine Unlearning using Federated Reinforcement Learning (FRAMU). This framework incorporates adaptive learning mechanisms, privacy preservation techniques, and optimization strategies, making it a well-rounded solution for handling various data sources, either single-modality or multi-modality, while maintaining accuracy and privacy. FRAMU's strengths include its adaptability in fluctuating data landscapes, its ability to unlearn outdated, private, or irrelevant data, and its support for continual model evolution without compromising privacy. Our experiments, conducted on both single-modality and multi-modality datasets, revealed that FRAMU significantly outperformed baseline models. Additional assessments of convergence behavior and optimization strategies further validate the framework's utility in federated learning applications. Overall, FRAMU advances Machine Unlearning by offering a robust, privacy-preserving solution that optimizes model performance while also addressing key challenges in dynamic data environments.

Exploiting long-range semantic contexts and geometric information is crucial to infer salient objects from RGB and depth features. However, existing methods mainly focus on excavating local features within fixed regions by continuously feeding forward networks. In this article, we introduce Dynamic Message Propagation (DMP) to dynamically learn context information within more flexible regions. We integrate DMP into a Siamese-based network to process the RGB image and depth map separately and design a multi-level feature fusion module to explore cross-level information between refined RGB and depth features. Extensive experiments show clear improvements of our method over 17 methods on six benchmark datasets for RGB-D salient object detection (SOD). Additionally, our method outperforms its competitors for the video SOD task. Code is available at https://github.com/chenbaian-cs/DMPNet .

There is a prevailing trend towards fusing multi-modal information for 3D object detection (3OD). However, challenges related to computational efficiency, plug-and-play capabilities, and accurate feature alignment have not been adequately addressed in the design of multi-modal fusion networks. In this paper, we present PointSee, a lightweight, flexible, and effective multi-modal fusion solution to facilitate various 3OD networks by semantic feature enhancement of point clouds (e.g., LiDAR or RGB-D data) assembled with scene images. Beyond the existing wisdom of 3OD, PointSee consists of a hidden module (HM) and a seen module (SM): HM decorates point clouds using 2D image information in an offline fusion manner, leading to minimal or even no adaptations of existing 3OD networks; SM further enriches the point clouds by acquiring point-wise representative semantic features, leading to enhanced performance of existing 3OD networks. Besides the new architecture of PointSee, we propose a simple yet efficient training strategy, to ease the potential inaccurate regressions of 2D object detection networks. Extensive experiments on the popular outdoor/indoor benchmarks show quantitative and qualitative improvements of our PointSee over thirty-five state-of-the-art methods.

How will you repair a physical object with some missings? You may imagine its original shape from previously captured images, recover its overall (global) but coarse shape first, and then refine its local details. We are motivated to imitate the physical repair procedure to address point cloud completion. To this end, we propose a cross-modal shape-transfer dual-refinement network (termed CSDN), a coarse-to-fine paradigm with images of full-cycle participation, for quality point cloud completion. CSDN mainly consists of "shape fusion" and "dual-refinement" modules to tackle the cross-modal challenge. The first module transfers the intrinsic shape characteristics from single images to guide the geometry generation of the missing regions of point clouds, in which we propose IPAdaIN to embed the global features of both the image and the partial point cloud into completion. The second module refines the coarse output by adjusting the positions of the generated points, where the local refinement unit exploits the geometric relation between the novel and the input points by graph convolution, and the global constraint unit utilizes the input image to fine-tune the generated offset. Different from most existing approaches, CSDN not only explores the complementary information from images but also effectively exploits cross-modal data in the whole coarse-to-fine completion procedure. Experimental results indicate that CSDN performs favorably against twelve competitors on the cross-modal benchmark.

We propose GeoGCN, a novel geometric dual-domain graph convolution network for point cloud denoising (PCD). Beyond the traditional wisdom of PCD, to fully exploit the geometric information of point clouds, we define two kinds of surface normals, one is called Real Normal (RN), and the other is Virtual Normal (VN). RN preserves the local details of noisy point clouds while VN avoids the global shape shrinkage during denoising. GeoGCN is a new PCD paradigm that, 1) first regresses point positions by spatial-based GCN with the help of VNs, 2) then estimates initial RNs by performing Principal Component Analysis on the regressed points, and 3) finally regresses fine RNs by normal-based GCN. Unlike existing PCD methods, GeoGCN not only exploits two kinds of geometry expertise (i.e., RN and VN) but also benefits from training data. Experiments validate that GeoGCN outperforms SOTAs in terms of both noise-robustness and local-and-global feature preservation.

Small targets are often submerged in cluttered backgrounds of infrared images. Conventional detectors tend to generate false alarms, while CNN-based detectors lose small targets in deep layers. To this end, we propose iSmallNet, a multi-stream densely nested network with label decoupling for infrared small object detection. On the one hand, to fully exploit the shape information of small targets, we decouple the original labeled ground-truth (GT) map into an interior map and a boundary one. The GT map, in collaboration with the two additional maps, tackles the unbalanced distribution of small object boundaries. On the other hand, two key modules are delicately designed and incorporated into the proposed network to boost the overall performance. First, to maintain small targets in deep layers, we develop a multi-scale nested interaction module to explore a wide range of context information. Second, we develop an interior-boundary fusion module to integrate multi-granularity information. Experiments on NUAA-SIRST and NUDT-SIRST clearly show the superiority of iSmallNet over 11 state-of-the-art detectors.

Small targets are often submerged in the cluttered backgrounds of infrared images. In this paper, we propose an iterative feedback UNet++ for infrared small target detection, dubbed ifUNet++. Unlike most of existing methods, ifU-Net++ enables to concentrate on small targets while weakening the interference of clutter backgrounds. ifUNet++ contains two parts: a simplified UNet++ and an iterative feedback strategy. We reduce the unnecessary nodes of UNet++ and have the simplified UNet++ as our backbone network, avoiding the loss of infrared small targets. Based on the simplified network, we search the infrared small targets in an iterative feedback manner, avoiding the interference of cluttered backgrounds. Besides, to optimize the iterative results, we propose Contextual Multiple Attention (CMA) to enhance the features in each iteration. Experimental results exhibit the clear promotion of ifUNet++ over eight state-of-the-art methods, in terms of noise-robustness and detection accuracy.

This research paper presents a comprehensive exploration of the application of Generative Artificial Intelligence (GAI) within the realm of language education. Through a systematic review of 20 empirical studies conducted within a structured three-step framework, this study elucidates the multifaceted integration of GAI in language learning. The examination encompasses various dimensions, including target languages, learners' educational levels, GAI applications, language skills, and practical outcomes. Additionally, the study critically assesses the key advantages and challenges inherent in GAI's role in language education. The paper provides valuable insights into human-technology interaction by delving into language learners' attitudes toward GAI. Notably, this research identifies three pivotal roles GAI assumes within the language learning process, and they are co-author, evaluator, and learning materials provider. The study concludes by charting a path toward future research endeavors within the evolving landscape of GAI-based language learning by implicating the future research direction of integrating the GAI into language education, including collaborations between humans and GAI, clarifying the definition of GAI-powered plagiarism, GAI-based language activities design, prompting strategies, and digital literacy.

Scholars suggested using social networking to assist second language and foreign language learning. Researchers in the present study designed and developed a Mini Program on the WeChat platform to assist university EFL learners in improving self-regulated English writing. This exploratory study aims to investigate the effects of WeChat-assisted Self-Regulated English Writing Instruction on university EFL learners’ motivation in English writing. A quasi-experimental design with forty-five second-school-year university students was conducted for two weeks. Participants were randomly divided into the control group, the traditional group and the WeChat-assisted group. They took a pre- and post-questionnaire survey on English writing motivation before and after the intervention. Descriptive statistic analysis was used to show the status quo of the participants’ motivation, paired sample t-test was used to analyze the differences in motivation within each group before and after the intervention, and ANOVA analysis was used to compare differences in motivation between groups after the intervention. Both the traditional approach and the WeChat approach generated positive impacts on students’ overall motivation, self-efficacy and interest. Students using the traditional approach outperformed those using the WeChat-assisted approach but the results did not reach a significant level. Implications for future studies are suggested.

In terms of the generative process, the Gamma-Gamma-Poisson Process (G2PP) is equivalent to the nonparametric topic model of Hierarchical Dirichlet Process (HDP). Considering the high computational cost of estimating parameters in HDP, a parallel G2PP was developed to generate topics efficiently via multi-threading. Unfortunately, the above model needs to predefine the number of topics. To address this issue, we first propose a Topic Self-Adaptive Model (TSAM) for nonparametric and parallel topic discovery. In TSAM, a monitor-executor mechanism is developed to manage the global topic information using a hierarchical structure of threads. Based on the apparatus of copulas, we further extend our TSAM to TSAMcop for coherent topic modeling by exploiting a copula guided parallel Gibbs sampling algorithm. Extensive experiments validate the effectiveness of both TSAM and TSAMcop.

With the fast progress in information technologies and artificial intelligence (AI), smart healthcare has gained considerable momentum. By using advanced technologies like AI, smart healthcare aims to promote human beings’ health and well-being throughout their life. As smart healthcare develops, big healthcare data are produced by various sensors, devices, and communication technologies constantly. To deal with these big multi-source data, automatic information fusion becomes crucial. Information fusion refers to the integration of multiple information sources for obtaining more reliable, effective, and precise information to support optimal decision-making. The close study of information fusion for healthcare with the adoption of advanced AI technologies has become an increasingly important and active field of research. The aim of this is to present a systematic description and state-of-the-art understanding of research about information fusion for healthcare with AI. Structural topic modeling was implemented to detect major research topics covered within 351 relevant articles. Annual trends and correlations of the identified topics were also investigated to identify potential future research directions. In addition, the primary research concerns of top countries/regions, institutions, and authors were shown and compared. The findings based on our analyses provide scientific and technological perspectives of research on information fusion for smart health with AI and offer useful insights and implications for its future development. We also provide valuable guidance for researchers and project managers to allocate research resources and promote effective international collaborations.

Researchers have been aware that emotion is not one-hot encoded in emotion-relevant classification tasks, and multiple emotions can coexist in a given sentence. Recently, several works have focused on leveraging a distribution label or a grayscale label of emotions in the classification model, which can enhance the one-hot label with additional information, such as the intensity of other emotions and the correlation between emotions. Such an approach has been proven effective in alleviating the overfitting problem and improving the model robustness by introducing a distribution learning component in the objective function. However, the effect of distribution learning cannot be fully unfolded as it can reduce the model’s discriminative ability within similar emotion categories. For example, “Sad” and “Fear” are both negative emotions. To address such a problem, we proposed a novel emotion extension scheme in the prior work (Li, Chen, Xie, Li, and Tao, 2021). The prior work incorporated fine-grained emotion concepts to build an extended label space, where a mapping function between coarse-grained emotion categories and fine-grained emotion concepts was identified. For example, sentences labeled “Joy” can convey various emotions such as enjoy, free, and leisure. The model can further benefit from the extended space by extracting dependency within fine-grained emotions when yielding predictions in the original label space. The prior work has shown that it is more apt to apply distribution learning in the extended label space than in the original space. A novel sparse connection method, i.e., Leaky Dropout, is proposed in this paper to refine the dependency-extraction step, which further improves the classification performance. In addition to the multiclass emotion classification task, we extensively experimented on sentiment analysis and multilabel emotion prediction tasks to investigate the effectiveness and generality of the label extension schema.

Recently, graph neural networks (GNNs) have achieved promising results in session-based recommendation. Existing methods typically construct a local session graph and a global session graph to explore complex item transition patterns. However, studies have seldom investigated the repeat consumption phenomenon in a local graph. In addition, it is challenging to retrieve relevant adjacent nodes from the whole training set owing to computational complexity and space constraints. In this study, we use a GNN to jointly model intra- and inter-session item dependencies for session-based recommendations. We construct a repeat-aware local session graph to encode the intra-item dependencies and generate the session representation with positional awareness. Then, we use sessions from the current mini-batch instead of the whole training set to construct a global graph, which we refer to as the session-level global graph. Next, we aggregate the K-nearest neighbors to generate the final session representation, which enables easy and efficient neighbor searching. Extensive experiments on three real-world recommendation datasets demonstrate that RN-GNN outperforms state-of-the-art methods.

Blockchain-enabled smart contracts have revolutionized the insurance industry due to their potential to streamline backend operations, mitigate fraudulent claims, and enhance data security and transparency. Guided by the design science methodology, the authors propose two specific smart contract frameworks to enhance insurance claims processing related to vehicle damage claims and personal injury claims. These proposed frameworks can improve the overall efficiency and effectiveness of insurance claims processing by automating claims submission, review, analysis, and payment, while reducing fraud and data leakage, by merging various data sources and disintermediation. Furthermore, the authors design a smart contract template supported by eight operational algorithms to facilitate the processing of insurance claims with the help of smart contracts. This template provides practitioners with a standardized prototype for the development of secure and efficient insurance applications.

Abstract Face recognition (FR) systems based on convolutional neural networks have shown excellent performance in human face inference. However, some malicious users may exploit such powerful systems to identify others' face images disclosed by victims' social network accounts, consequently obtaining private information. To address this emerging issue, synthesizing face protection images with visual and protective effects is essential. However, existing face protection methods encounter three critical problems: poor visual effect, limited protective effect, and trade‐off between visual and protective effects. To address these challenges, we propose a novel face protection approach in this article. Specifically, we design a generative adversarial network (GAN) framework with an autoencoder (AEGAN) as the generator to synthesize the protection images. It is worth noting that we introduce an interpolation upsampling module in the decoder in order to let the synthesized protection images evade recognition by powerful convolution‐based FR systems. Furthermore, we introduce an attention module with a perceptual loss in AEGAN to enhance the visual effects of synthesized images by AEGAN. Extensive experiments have shown that AEGAN not only can maintain the comfortable visual quality of synthesized images but also prevent the recognition of commercial FR systems, including Baidu and iKLYTEK.

Point normal, as an intrinsic geometric property of 3D objects, not only serves conventional geometric tasks such as surface consolidation and reconstruction, but also facilitates cutting-edge learning-based techniques for shape analysis and generation. In this paper, we propose a normal refinement network, called Refine-Net, to predict accurate normals for noisy point clouds. Traditional normal estimation wisdom heavily depends on priors such as surface shapes or noise distributions, while learning-based solutions settle for single types of hand-crafted features. Differently, our network is designed to refine the initial normal of each point by extracting additional information from multiple feature representations. To this end, several feature modules are developed and incorporated into Refine-Net by a novel connection module. Besides the overall network architecture of Refine-Net, we propose a new multi-scale fitting patch selection scheme for the initial normal estimation, by absorbing geometry domain knowledge. Also, Refine-Net is a generic normal estimation framework: 1) point normals obtained from other methods can be further refined, and 2) any feature module related to the surface geometric structures can be potentially integrated into the framework. Qualitative and quantitative evaluations demonstrate the clear superiority of Refine-Net over the state-of-the-arts on both synthetic and real-scanned datasets.

Recently, many efforts have been devoted to promoting the Emotion-Cause Pair Extraction (ECPE) task, as jointly extracting emotions and their causes is considered more helpful than only identifying the emotions in many applications. Among the existing efforts, end-to-end approaches are getting popular as the main trend, while others like pipeline models have been overlooked due to their potential issues of cascading errors. Nevertheless, the advantages of the pipeline models, such as logically dividing a complicated task into multiple easier subtasks, are underestimated and not well exploited. Moreover, the existing end-to-end approaches fail to capture the implicit co-occurrence or exclusion patterns between multiple pairs of emotions and causes since they are extracted independently. In view of these limitations, we propose a novel two-stage model to address the ECPE task and incorporate reinforcement learning (RL) to tackle the cascading error issue. In particular, our two-stage model first detects emotion clauses and then recognizes cause clauses for each detected emotion clause sequentially. By representing the error of each decision as an explicit reward, our model clearly knows how the error at each stage affects the final performance, hence the model can adjust itself for better performance. Furthermore, the sequential prediction enables our model to use the results achieved in the previous stages as auxiliary information in the subsequent stages. Extensive experiments on the benchmark dataset demonstrate the effectiveness of our proposed two-stage model, and the ablation comparison shows the promising effect of reducing cascading errors by incorporating RL.

The metaverse and its underlying blockchain technology have attracted extensive attention in the past few years. How to mine, process, and analyze the tremendous data generated by the metaverse systems has posed a number of challenges. Aiming to address them, we mainly focus on modeling and understanding the blockchain transaction network from a structural identity perspective, which represents the entire network structure and reveals the relations among multiple entities. In this article, we analyze three metaverse-related systems: non-fungible token (NFT), Ethereum (ETH), and Bitcoin (BTC) from the structural-identity perspective. First, we conduct the complex network analysis of the metaverse network and obtain several new insights (i.e., power-law degree distribution, disconnection, disassortativity, preferential attachment, and non-rich-club effect). Secondly, based on such findings, we propose a novel representation learning method named structure-to-vector with random pace (SVRP) for learning both the latent representation and structural identity of the network. Thirdly, we conduct node classification and link prediction tasks with the integration of graph neural networks (GNNs). Empirical results on three real-world datasets demonstrate that our proposed SVRP outperforms other existing methods in multiple tasks. In particular, our SVRP achieves the highest node classification accuracy (Acc) (99.3$\%$) and$F$1-score (96.7$\%$) while only requiring original non-attributed graphs.

LiDAR and camera, as two different sensors, supply geometric (point clouds) and semantic (RGB images) information of 3-D scenes. However, it is still challenging for existing methods to fuse data from the two cross sensors, making them complementary for quality 3-D object detection (3OD). We propose ImLiDAR, a new 3OD paradigm to narrow the cross-sensor discrepancies by progressively fusing the multiscale features of camera Images and LiDAR point clouds. ImLiDAR enables to provide the detection head with cross-sensor yet robustly fused features. To achieve this, two core designs exist in ImLiDAR. First, we propose a cross-sensor dynamic message propagation (CDMP) module to combine the best of the multiscale image and point features. Second, we raise a direct set prediction problem that allows designing an effective set-based detector (SD) to tackle the inconsistency of the classification and localization confidences, and the sensitivity of hand-tuned hyperparameters. Besides, the novel SD can be detachable and easily integrated into various detection networks. Comparisons on the KITTI, nuScenes, and SUN-RGBD datasets all show clear visual and numerical improvements of our ImLiDAR over 45 state-of-the-art 3OD methods.

Sentence representation learning is a crucial task in natural language processing, as the quality of learned representations directly influences downstream tasks, such as sentence classification and sentiment analysis. Transformer-based pretrained language models such as bidirectional encoder representations from transformers (BERT) have been extensively applied to various natural language processing tasks, and have exhibited moderately good performance. However, the anisotropy of the learned embedding space prevents BERT sentence embeddings from achieving good results in the semantic textual similarity tasks. It has been shown that contrastive learning can alleviate the anisotropy problem and significantly improve sentence representation performance. Therefore, there has been a surge in the development of models that utilize contrastive learning to fine-tune BERT-like pretrained language models to learn sentence representations. But no systematic review of contrastive learning models for sentence representations has been conducted. To fill this gap, this article summarizes and categorizes the contrastive learning based sentence representation models, common evaluation tasks for assessing the quality of learned representations, and future research directions. Furthermore, we select several representative models for exhaustive experiments to illustrate the quantitative improvement of various strategies on sentence representations.

Snow is one of the toughest adverse weather conditions for object detection (OD). Currently, not only there is a lack of snowy OD datasets to train cutting-edge detectors, but also these detectors have difficulties of learning latent information beneficial for detection in snow. To alleviate the two above problems, we first establish a real-world snowy OD dataset, named RSOD. Besides, we develop an unsupervised training strategy with a distinctive activation function, called$Peak Act$, to quantitatively evaluate the effect of snow on each object. Peak Act helps grade the images in RSOD into four-difficulty levels. To our knowledge, RSOD is the first quantitatively evaluated and graded real-world snowy OD dataset. Then, we propose a novel Cross Fusion (CF) block to construct a lightweight OD network based on YOLOv5s (called CF-YOLO). CF is a plug-and-play feature aggregation module, which integrates the advantages of Feature Pyramid Network and Path Aggregation Network in a simpler yet more flexible form. Both RSOD and CF lead our CF-YOLO to possess an optimization ability for OD in real-world snow. That is, CF-YOLO can handle unfavorable detection problems of vagueness, distortion and covering of snow. Experiments show that our CF-YOLO achieves better detection results on RSOD, compared to SOTAs. The code and dataset are available athttps://github.com/qqding77/CF-YOLO-and-RSOD.

In this study, sentiment classification and emotion distribution learning across domains are both formulated as a semi-supervised domain adaptation problem, which utilizes a small amount of labeled documents in the target domain for model training. By introducing a shared matrix that captures the stable association between document clusters and word clusters, non-negative matrix tri-factorization (NMTF) is robust to the labeled target domain data and has shown remarkable performance in cross-domain text classification. However, the existing NMTF-based models ignore the incompatible relationship of sentiment polarities and the relatedness among emotions. Besides, their applications on large-scale datasets are limited by the high computation complexity. To address these issues, we propose a semi-supervised NMTF framework for sentiment classification and emotion distribution learning across domains. Based on a many-to-many mapping between document clusters and sentiment polarities (or emotions), we first incorporate the prior information of label dependency to improve the model performance. Then, we develop a parallel algorithm based on message passing interface (MPI) to further enhance the model scalability. Extensive experiments on real-world datasets validate the effectiveness of our method.

As a novel paradigm for data mining and dimensionality reduction, Non-negative Matrix Tri-Factorization (NMTF) has attracted much attention due to its notable performance and elegant mathematical derivation, and it has been applied to a plethora of real-world applications, such as text data co-clustering. However, the existing NMTF-based methods usually involve intensive matrix multiplications, which exhibits a major limitation of high computational complexity. With the explosion at both the size and the feature dimension of texts, there is a growing need to develop a parallel and scalable NMTF-based algorithm for text data co-clustering. To this end, we first show in this paper how to theoretically derive the original optimization problem of NMTF by introducing the Lagrangian multipliers. Then, we propose to solve the Lagrange dual objective function in parallel through an efficient distributed implementation. Extensive experiments on five benchmark corpora validate the effectiveness, efficiency, and scalability of our distributed parallel update algorithm for an NMTF-based text data co-clustering method.

Gesture recognition by using Inertial Measurement Unit (IMU) sensors plays an important role in various Internet of Things (IOT) applications, e.g., smart home, intelligent medical system and so on. Traditional technologies usually utilize machine learning algorithms to train different gestures during the offline phase, then recognize the gesture during the online phase. However, such technologies cannot recognize these gestures without prior training. Even for the same gesture, with different gesture amplitude may result in unsuccessful recognition. Also if we change the person to perform the same gesture, the algorithms fails. In order to overcome these drawbacks, we propose an approach, which will be able to track the human body motion in real-time and also recognize complicated gestures. It utilizes the accelerometer information and proposes comprehensive localization algorithms for each deployed sensor attached on the human body. Then, it takes the correlation and limitation among body parts into account to recognize the gesture. Our experiments results show that, the successful recognition rate of our algorithm is 100%. Furthermore, any part of the human body can be well tracked, the tracking accuracy can reach$0.06m$.

In multi-site studies of Alzheimer's disease (AD), the difference of data in multi-site datasets leads to the degraded performance of models in the target sites. The traditional domain adaptation method requires sharing data from both source and target domains, which will lead to data privacy issue. To solve it, federated learning is adopted as it can allow models to be trained with multi-site data in a privacy-protected manner. In this paper, we propose a multi-site federated domain adaptation framework via Transformer (FedDAvT), which not only protects data privacy, but also eliminates data heterogeneity. The Transformer network is used as the backbone network to extract the correlation between the multi-template region of interest features, which can capture the brain abundant information. The self-attention maps in the source and target domains are aligned by applying mean squared error for subdomain adaptation. Finally, we evaluate our method on the multi-site databases based on three AD datasets. The experimental results show that the proposed FedDAvT is quite effective, achieving accuracy rates of 88.75%, 69.51%, and 69.88% on the AD vs. NC, MCI vs. NC, and AD vs. MCI two-way classification tasks, respectively.

We propose a geometry-supporting dual convolutional neural network (GeoDualCNN) for both point cloud normal estimation and denoising. GeoDualCNN fuses the geometry domain knowledge that the underlying surface of a noisy point cloud is piecewisely smooth with the fact that a point normal is properly defined only when local surface smoothness is guaranteed. Centered around this insight, we define the homogeneous neighborhood (HoNe) which stays clear of surface discontinuities, and associate each HoNe with a point whose geometry and normal orientation is mostly consistent with that of HoNe. Thus, we not only obtain initial estimates of the point normals by performing PCA on HoNes, but also for the first time optimize these initial point normals by learning the mapping from two proposed geometric descriptors to the ground-truth point normals. GeoDualCNN consists of two parallel branches that remove noise using the first geometric descriptor (a homogeneous height map, which encodes the point-position information), while preserving surface features using the second geometric descriptor (a homogeneous normal map, which encodes the point-normal information). Such geometry-supporting network architectures enable our model to leverage previous geometry expertise and to benefit from training data. Experiments with noisy point clouds show that GeoDualCNN outperforms the state-of-the-art methods in terms of both noise-robustness and feature preservation.

Can you find me? By simulating how humans to discover the so-called 'perfectly'-camouflaged object, we present a novel boundary-guided separated attention network (call BSA-Net). Beyond the existing camouflaged object detection (COD) wisdom, BSA-Net utilizes two-stream separated attention modules to highlight the separator (or say the camouflaged object's boundary) between an image's background and foreground: the reverse attention stream helps erase the camouflaged object's interior to focus on the background, while the normal attention stream recovers the interior and thus pay more attention to the foreground; and both streams are followed by a boundary guider module and combined to strengthen the understanding of boundary. The core design of such separated attention is motivated by the COD procedure of humans: find the subtle difference between the foreground and background to delineate the boundary of a camouflaged object, then the boundary can help further enhance the COD accuracy. We validate on three benchmark datasets that the proposed BSA-Net is very beneficial to detect camouflaged objects with the blurred boundaries and similar colors/patterns with their backgrounds. Extensive results exhibit very clear COD improvements on our BSA-Net over sixteen SOTAs.

Rain severely degrades the visibility of scene objects, especially when images are captured through the glass under rainy weather. We observe three intriguing phenomena: 1) rain is a mixture of raindrops, rain streaks and rainy haze; 2) the depth from the camera determines the degree of object visibility, where objects nearby and far away are visually blocked by rain streaks and rainy haze, respectively; and 3) raindrops on the glass randomly affect the object visibility of the whole image space. However, existing solutions and benchmark datasets lack full consideration of the mixture of rain (MOR). In this paper, we originally consider that the overall object visibility is determined by MOR, and enrich the RainCityscapes by considering real-world raindrops to construct the MOR dataset, named RainCityscapes++. To solve the practical rain removal problem arisen from MOR, we formulate a new rain imaging model and propose a multi-branch attention generative adversarial network (MBA-RainGAN). Extensive experiments show clear improvements of our approach over SOTAs on RainCityscapes++.

Both the metaverse and its underlying blockchain technology have attracted extensive attention in the past few years. It becomes a natural problem to extract, process, and analyze the tremendous data generated by the blockchain systems for various metaverse applications though it also poses diverse challenges. Amongst those challenges, this paper mainly focuses on modeling and understanding the blockchain transaction network from a structural identity perspective, which represents the entire network structure and reveals the relations among multiple entities. In this paper, we propose a novel representation learning method named Structure-to-Vector with Random Pace (SVRP) for learning both latent representation and structural identity of blockchain transaction networks. We then conduct node classification and link prediction tasks with integration with Graph Neural Networks (GNNs). Empirical results on three representative blockchain data sets, namely Non-fungible token (NFT), Ethereum (ETH), and Bitcoin (BTC), demonstrate that our proposed SVRP outperforms other existing methods in multiple tasks. In particular, our SVRP achieves the highest node classification accuracy (99.3%) while only requiring original non-attributed graphs (i.e., graphs without node features).

Abstract Real‐world rain is a mixture of rain streaks and rainy haze. However, current efforts formulate image rain streaks removal and rainy haze removal as separated models, worsening the loss of image details. This paper attempts to solve the mixture of rain removal problem in a single model by estimating the scene depths of images. To this end, we propose a novel SEMI‐ supervised M ixture O f rain RE moval G enerative A dversarial N etwork (Semi‐MoreGAN). Unlike most of existing methods, Semi‐MoreGAN is a joint learning paradigm of mixture of rain removal and depth estimation; and it effectively integrates the image features with the depth information for better rain removal. Furthermore, it leverages unpaired real‐world rainy and clean images to bridge the gap between synthetic and real‐world rain. Extensive experiments show clear improvements of our approach over twenty representative state‐of‐the‐arts on both synthetic and real‐world rainy images. Source code is available at https://github.com/syy-whu/Semi-MoreGAN .

Abstract Image smoothing is a fundamental low‐level vision task that aims to preserve salient structures of an image while removing insignificant details. Deep learning has been explored in image smoothing to deal with the complex entanglement of semantic structures and trivial details. However, current methods neglect two important facts in smoothing: 1) naive pixel‐level regression supervised by the limited number of high‐quality smoothing ground‐truth could lead to domain shift and cause generalization problems towards real‐world images; 2) texture appearance is closely related to object semantics, so that image smoothing requires awareness of semantic difference to apply adaptive smoothing strengths. To address these issues, we propose a novel Contrastive Semantic‐Guided Image Smoothing Network (CSGIS‐Net) that combines both contrastive prior and semantic prior to facilitate robust image smoothing. The supervision signal is augmented by leveraging undesired smoothing effects as negative teachers, and by incorporating segmentation tasks to encourage semantic distinctiveness. To realize the proposed network, we also enrich the original VOC dataset with texture enhancement and smoothing labels, namely VOC‐smooth, which first bridges image smoothing and semantic segmentation. Extensive experiments demonstrate that the proposed CSGIS‐Net outperforms state‐of‐the‐art algorithms by a large margin. Code and dataset are available at https://github.com/wangjie6866/CSGIS-Net .

Abstract Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question – if the combination of image restoration and object detection, can boost the performance of cutting‐edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet. Different from existing efforts that intuitively apply image dehazing/deraining as a pre‐processing step, TogetherNet considers a multi‐task joint learning problem. Following the joint learning scheme, clean features produced by the restoration network can be shared to learn better object detection in the detection network, thus helping TogetherNet enhance the detection capacity in adverse weather conditions. Besides the joint learning architecture, we design a new Dynamic Transformer Feature Enhancement module to improve the feature extraction and representation capabilities of TogetherNet. Extensive experiments on both synthetic and real‐world datasets demonstrate that our TogetherNet outperforms the state‐of‐the‐art detection approaches by a large margin both quantitatively and qualitatively. Source code is available at https://github.com/yz-wang/TogetherNet .

Abstract Most of the existing object detection methods generate poor glass detection results, due to the fact that the transparent glass shares the same appearance with arbitrary objects behind it in an image. Different from traditional deep learning‐based wisdoms that simply use the object boundary as an auxiliary supervision, we exploit label decoupling to decompose the original labelled ground‐truth (GT) map into an interior‐diffusion map and a boundary‐diffusion map. The GT map in collaboration with the two newly generated maps breaks the imbalanced distribution of the object boundary, leading to improved glass detection quality. We have three key contributions to solve the transparent glass detection problem: (1) We propose a three‐stream neural network (call GlassNet for short) to fully absorb beneficial features in the three maps. (2) We design a multi‐scale interactive dilation module to explore a wider range of contextual information. (3) We develop an attention‐based boundary‐aware feature Mosaic module to integrate multi‐modal information. Extensive experiments on the benchmark dataset exhibit clear improvements of our method over SOTAs, in terms of both the overall glass detection accuracy and boundary clearness.

With the growing availability and popularity of sentiment-rich resources like blogs and online reviews, new opportunities and challenges have emerged regarding the identification, extraction, and organization of sentiments from user-generated documents or sentences. Recently, many studies have exploited lexicon-based methods or supervised learning algorithms to conduct sentiment analysis tasks separately; however, the former approaches ignore contextual information of sentences and the latter ones do not take sentiment information embedded in sentiment words into consideration. To tackle these limitations, we propose a new model named Sentiment Convolutional Neural Network (SentiCNN) to analyze the sentiments of sentences with both contextual and sentiment information of sentiment words, in which, contextual information is captured from word embeddings and sentiment information is identified using existing lexicons. We incorporate a Highway Network into our model to adaptively combine sentiment and contextual information from sentences by strengthening the connection between features of both sentences and their sentiment words. Furthermore, we propose three lexicon-based attention mechanisms (LBAMs) for our SentiCNN model to find the most important indicators of sentiments and make predictions more effectively. Experiments over two well-known datasets indicate that sentiment words, the Highway Network, and LBAMs contribute to sentiment analysis.

Hierarchical Dirichlet Process (HDP) has attracted much attention in the research community of natural language processing. Given a corpus, HDP is able to determine the number of topics automatically, possessing an important feature dubbed nonparametric that overcomes the challenging issue of manually specifying a suitable topic number in parametric topic models, such as Latent Dirichlet Allocation (LDA). Nevertheless, HDP requires a much higher computational cost than LDA for parameter estimation. By taking the advantage of multi-threading, a parallel Gibbs sampling algorithm is proposed to estimate parameters for HDP based on the equivalence between HDP and Gamma-Gamma Poisson Process (G2PP) in terms of the generative process. Unfortunately, the above parallel Gibbs sampling algorithm requires to apply the finite approximation on the number of topics manually (i.e., predefine the topic number), thus can not retain the nonparametric feature of HDP. Another drawback of the above models is the lack of capturing the semantic dependencies between words, because the topic assignment of words is independent with each other. Although some works have been done in phrase-based topic modelling, these existing methods are still limited by either enforcing the entire phrase to share a common topic or requiring much complex and time-consuming phrase mining methods. In this paper, we aim to develop a copula guided parallel Gibbs sampling algorithm for HDP which can adjust the number of topics dynamically and capture the latent semantic dependencies between words that compose a coherent segment. Extensive experiments on real-world datasets indicate that our method achieves low perplexities and high topic coherence scores with a small time cost. In addition, we validate the effectiveness of our method on the modelling of word semantic dependencies by comparing the extracted topical phrases with those learned by state-of-the-art phrase-based baselines.

Currently, text classification studies mainly focus on training classifiers by using textual input only, or enhancing semantic features by introducing external knowledge (e.g., hand-craft lexicons and domain knowledge). In contrast, some intrinsic statistical features of the corpus, like word frequency and distribution over labels, are not well exploited. Compared with external knowledge, the statistical features are deterministic and naturally compatible with corresponding tasks. In this paper, we propose an Adaptive Gate Network (AGN) to consolidate semantic representation with statistical features selectively. In particular, AGN encodes statistical features through a variational component and merges information via a well-designed valve mechanism. The valve adapts the information flow into the classifier according to the confidence of semantic features in decision making, which can facilitate training a robust classifier and can address the overfitting caused by using statistical features. Extensive experiments on datasets of various scales show that, by incorporating statistical information, AGN can improve the classification performance of CNN, RNN, Transformer, and Bert based models effectively. The experiments also indicate the robustness of AGN against adversarial attacks of manipulating statistical information.

Existing methods for named entity recognition (NER) are critically relied on the amount of labeled data. However, these methods suffer from performance decline in a new domain which is fully-unlabeled. To handle the situation, we propose an entity-aware adversarial domain adaptation network, which utilizes the labeled data from source domain and then adapts to unlabeled target domain. We first apply adversarial training to reduce the distribution gap between different domains. Furthermore, we introduce an entity-aware attention to guide adversarial to achieve the alignment of entity features. The experimental results show that our model outperforms the state-of-the-art approaches.

Ziye Chen, Cheng Ding, Zusheng Zhang, Yanghui Rao, Haoran Xie. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Artificial intelligence (AI) for personalized learning has attracted increasing attention in various educational contexts and domains, including language learning. This study systematically reviewed academic studies on AI-assisted personalized language learning (PLL) from the perspectives of article trends, top journals, countries/regions and institutions, AI technology types, learning outcomes and supports, participants, scientific collaborations, and co-citation relations. Results indicated Taiwanese institutions' predominance in the field and the prevalent use of intelligent tutoring systems, natural language processing, and artificial neural network in facilitating personalized diagnosis and learning path and material recommendations in language learning. Furthermore, students' improved language outcomes and positive perception, satisfaction, or motivation towards language learning and AI technologies were commonly reported. The co-authorship analysis results indicated the close inter-regional collaborations, while the cross-regional collaborations are expected to be enhanced. The co-citation network analysis results highlighted the significance of fuzzy systems and item response theory. Additionally, learner profiling mining and learning resource adaptation were important directions to realize mobile- and web-based PLL.

In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

We present a novel direction-aware feature-level frequency decomposition network for single image deraining. Compared with existing solutions, the proposed network has three compelling characteristics. First, unlike previous algorithms, we propose to perform frequency decomposition at feature-level instead of image-level, allowing both low-frequency maps containing structures and high-frequency maps containing details to be continuously refined during the training procedure. Second, we further establish communication channels between low-frequency maps and high-frequency maps to interactively capture structures from high-frequency maps and add them back to low-frequency maps and, simultaneously, extract details from low-frequency maps and send them back to high-frequency maps, thereby removing rain streaks while preserving more delicate features in the input image. Third, different from existing algorithms using convolutional filters consistent in all directions, we propose a direction-aware filter to capture the direction of rain streaks in order to more effectively and thoroughly purge the input images of rain streaks. We extensively evaluate the proposed approach in three representative datasets and experimental results corroborate our approach consistently outperforms state-of-the-art deraining algorithms.

MOTIVATION: Reliable predictive models of protein-ligand binding affinity are required in many areas of biomedical research. Accurate prediction based on current descriptors or molecular fingerprints (FPs) remains a challenge. We develop novel interaction FPs (IFPs) to encode protein-ligand interactions and use them to improve the prediction. RESULTS: Proteo-chemometrics IFPs (PrtCmm IFPs) formed by combining extended connectivity fingerprints (ECFPs) with the proteo-chemometrics concept. Combining PrtCmm IFPs with machine-learning models led to efficient scoring models, which were validated on the PDBbind v2019 core set and CSAR-HiQ sets. The PrtCmm IFP Score outperformed several other models in predicting protein-ligand binding affinities. Besides, conventional ECFPs were simplified to generate new IFPs, which provided consistent but faster predictions. The relationship between the base atom properties of ECFPs and the accuracy of predictions was also investigated. AVAILABILITY: PrtCmm IFP has been implemented in the IFP Score Toolkit on github (https://github.com/debbydanwang/IFPscore). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Saliency detection is a fundamental and challenging task in computer vision, which aims at distinguishing the most conspicuous objects or regions in an image. Existing deep-learning methods mainly rely on the entire image to learn the global context information for saliency detection, which loses the spatial relation and results in ambiguity in predicting saliency maps. In this paper, we propose a novel deep sub-region network (DSR-Net) equipped with a sequence of sub-region dilated blocks (SRDB) by aggregating multi-scale salient context information of multiple sub-regions, such that the global context information from the whole image and local contexts from sub-regions are fused together, making the saliency prediction more accurate. Our SRDB separates the input feature map at different layers of a convolutional neural network (CNN) into different sub-regions and then designs a parallel ASPP module to refine feature maps at each sub-region. Experiments on the five widely-used saliency benchmark datasets demonstrate that our network outperforms recent state-of-the-art saliency detectors quantitatively and qualitatively on all the benchmarks.

This work presents a gated non-local deep residual learning framework for image deraining. It can avoid the over-deraining or under-deraining caused by the global residual learning in existing deraining networks, since the learned soft gate in our method adaptively adjusts the amount of global residual to be passed for generating the final derained result. To generate feature maps for global residual prediction, we develop a non-local guided attention module (NLAM), which first obtains non-local features by exploiting spatial inter-dependencies among all the feature positions of local features produced by convolutional neural network (CNN), and then leverages the attention mechanism to merge the local and non-local features based on their complementary relation. Moreover, we develop a channel-wise gated prediction module to learn a soft gate on the global residual by explicitly modelling channel inter-dependencies of the feature maps obtained from NLAM. Experiments on four deraining benchmark datasets and real-world rainy images show that our network has a quantitative and qualitative improvement over state-of-the-arts.

With the development of social network platforms, discussion forums, and question answering websites, a huge number of short messages that typically contain a few words for an individual document are posted by online users. In these short messages, emotions are frequently embedded for communicating opinions, expressing friendship, and promoting influence. It is quite valuable to detect emotions from short messages, but the corresponding task suffers from the sparsity of feature space. In this article, we first generate term groups co-occurring in the same context to enrich the number of features. Then, two basic supervised topic models are proposed to associate emotions with topics accurately. To reduce the time cost of parameter estimation, we further propose an accelerated algorithm for our basic models. Extensive evaluations using three short corpora validate the efficiency and effectiveness of the accelerated models for predicting the emotions of unlabeled documents, in addition to generate the topic-level emotion lexicons.

Abstract Exploiting hand-crafted lexicon knowledge to enhance emotional or sentimental features at word-level has become a widely adopted method in emotion-relevant classification studies. However, few attempts have been made to explore the emotion construction in the classification task, which provides insights to how a sentence’s emotion is constructed. The major challenge of exploring emotion construction is that the current studies assume the dataset labels as relatively independent emotions, which overlooks the connections among different emotions. This work aims to understand the coarse-grained emotion construction and their dependency by incorporating fine-grained emotions from domain knowledge. Incorporating domain knowledge and dimensional sentiment lexicons, our previous work proposes a novel method namedEmoChannelto capture the intensity variation of a particular emotion in time series. We utilize the resultant knowledge of 151 available fine-grained emotions to comprise the representation of sentence-level emotion construction. Furthermore, this work explicitly employs a self-attention module to extract the dependency relationship within all emotions and proposeEmoChannel-SANetwork to enhance emotion classification performance. We conducted experiments to demonstrate that the proposed method produces competitive performances against the state-of-the-art baselines on both multi-class datasets and sentiment analysis datasets.

Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics. However, the existing parameter inference method like Monte Carlo sampling is quite time-consuming. In this paper, we propose two efficient neural mixed counting models, i.e., the Negative Binomial-Neural Topic Model (NB-NTM) and the Gamma Negative Binomial-Neural Topic Model (GNB-NTM) for dispersed topic discovery. Neural variational inference algorithms are developed to infer model parameters by using the reparameterization of Gamma distribution and the Gaussian approximation of Poisson distribution. Experiments on real-world datasets indicate that our models outperform state-of-the-art baseline models in terms of perplexity and topic coherence. The results also validate that both NB-NTM and GNB-NTM can produce explainable intermediate variables by generating dispersed proportions of document topics.

This paper looks at this intriguing question: are single images with their details lost during deraining, reversible to their artifact-free status? We propose an end-to-end detail-recovery image deraining network (termed a DRDNet) to solve the problem. Unlike existing image deraining approaches that attempt to meet the conflicting goal of simultaneously deraining and preserving details in a unified framework, we propose to view rain removal and detail recovery as two seperate tasks, so that each part could specialize rather than trade-off between two conflicting goals. Specifically, we introduce two parallel sub-networks with a comprehensive loss function which synergize to derain and recover the lost details caused by deraining. For complete rain removal, we present a rain residual network with the squeeze-and-excitation (SE) operation to remove rain streaks from the rainy images. For detail recovery, we construct a specialized detail repair network consisting of welldesigned blocks, named structure detail context aggregation block (SDCAB), to encourage the lost details to return for eliminating image degradations. Moreover, the detail recovery branch of our proposed detail repair framework is detachable and can be incorporated into existing deraining methods to boost their performances. DRD-Net has been validated on several well-known benchmark datasets in terms of deraining robustness and detail accuracy. Comparisons show clear visual and numerical improvements of our method over the state-of-the-arts.

In this paper, we propose a normal estimation method for unstructured point cloud. We observe that geometric estimators commonly focus more on feature preservation but are hard to tune parameters and sensitive to noise, while learning-based approaches pursue an overall normal estimation accuracy but cannot well handle challenging regions such as surface edges. This paper presents a novel normal estimation method, under the co-support of geometric estimator and deep learning. To lowering the learning difficulty, we first propose to compute a suboptimal initial normal at each point by searching for a best fitting patch. Based on the computed normal field, we design a normal-based height map network (NH-Net) to fine-tune the suboptimal normals. Qualitative and quantitative evaluations demonstrate the clear improvements of our results over both traditional methods and learning-based methods, in terms of estimation accuracy and feature recovery.

The popularity of the Internet has brought profound influence to electronic commerce. A kind of review-oriented consumption mode is gradually expanding in the market and consumers will refer to the reviews provided by consumers who bought the product in the past. How to accurately analyze users' sentiments from massive data of e-commerce reviews has become one of the key issues for e-commerce platforms. Current standard sentiment analysis classifies overall sentiment of e-commerce reviews without an extended description of the entity. We set up an optimized Aspect-based sentiment analysis (ABSA) that includes four elements: aspect, category, polarity, and opinion. Aiming at the above problems, this paper proposes a Chinese e-commerce reviews sentiment analysis algorithm based on BERT. By using pre-training model, we use the BIO(B-begin,I-inside,O-outside) data labeling pattern to label entities and study sentiment analysis by the annotation data. Experimental results on the Taobao cosmetics review datasets show that compared with the ordinary deep learning methods, our approach in the accuracy rate and the F1 score has significant improvement.

Abstract We present a near‐lighting photometric stereo (NL‐PS) system to produce digital bas‐reliefs from a physical object (set) directly. Unlike both the 2D image and 3D model‐based modelling methods that require complicated interactions and transformations, the technique using NL‐PS is easy to use with cost‐effective hardware, providing users with a trade‐off between abstract and representation when creating bas‐reliefs. Our algorithm consists of two steps: normal map acquisition and constrained 3D reconstruction. First, we introduce a lighting model, named the quasi‐point lighting model (QPLM), and provide a two‐step calibration solution in our NL‐PS system to generate a dense normal map. Second, we filter the normal map into a detail layer and a structure layer, and formulate detail‐ or structure‐preserving bas‐relief modelling as a constrained surface reconstruction problem of solving a sparse linear system. The main contribution is a WYSIWYG (i.e. what you see is what you get) way of building new solvers that produces multi‐style bas‐reliefs with their geometric structures and/or details preserved. The performance of our approach is experimentally validated via comparisons with the state‐of‐the‐art methods.

This work-in-progress poster discusses our preliminary work on exploring the possibility of harnessing spherical video-based immersive virtual reality (SV-IVR) in learning and teaching of Physical Geography. In particular, it focuses on delineating (i) the rationale behind our initiative, and (ii) the proposed direction of how to adopt SV-IVR in the context of formal schooling in Hong Kong.

Interactive digital technology has great potentiality in providing life-like learning contexts and in-time interactions to facilitate language learning, especially for the low achievers. Contexts are essential for effective language learning. In this study, two types of digital interactive technologies, including interactive digital map (iMap) and augmented reality (AR), were employed to deliver and enhance contextualized learning experiences with gamified learning tasks for the low achievers’. The purpose of this study was to investigate the effects of type of technology-enhanced learning, including the iMap-enhanced learning and the AR-enhanced learning, on low achievers’ learning performance and attitude while learning from the gamified technology-enhanced contextualized EFL learning. A preliminary experiment showed that (a) the AR-enhanced learning group outperformed the iMap-enhanced learning group on learning performance, (b) all participants’ post attitudes toward the received technology-enhanced learning were significantly increased in all attitude aspects of confidence, preference, anxiety, attention, and learning strategy, and (c) both technology-enhanced learning group revealed similar positive in most attitude aspects, excepted that the AR group showed a higher degree of attention than the iMap group. It was concluded that technology-enhanced contextualized learning is effective in promoting learning attitudes and helping EFL learners achieve acceptable learning performance.

Distant supervised relation extraction has been an effective way to find relational facts from text. However, distant supervised method inevitably accompanies with wrongly labeled sentences. Noisy sentences lead to poor performance of relation extraction models. Though existing piecewise convolutional neural network model with sentence-level attention (PCNN+ATT) is an effective way to reduce the effect of noisy sentences, it still has two limitations. On one hand, it adopts a PCNN module as sentence encoder, which only captures local contextual features of words and might lose important information. On the other hand, it neglects the fact that not all words contribute equally to the semantics of sentences. To address these two issues, we propose a hierarchical attention-based bidirectional GRU (HA-BiGRU) model. For the first limitation, our model utilizes a BiGRU module in place of PCNN, so as to extract global contextual information. For the second limitation, our model combines word-level and sentence-level attention mechanisms, which help get accurate sentence representations. To further alleviate the wrongly labeling problem, we first calculate the co-occurrence probabilities (CP) between the shortest dependency path (SDP) and the relation labels. Based on these co-occurrence probabilities, two denoising strategies are proposed to reduce noise interference respectively from aspect of filtering labeled data and integrating CP information into model. Experimental results on the corpus of Freebase and New York Times (Freebase+NYT) show that the HA-BiGRU model outperforms baseline models, and the two co-occurrence probabilities based denoising strategies can improve robustness of HA-BiGRU model.

Cross-domain image retrieval is always encountering insufficient labelled data in real world. In this paper, we propose unsupervised embedding learning (UEL) for cross-domain beauty and personal care product retrieval to finetune the convolutional neural network (CNN). More specifically, UEL utilizes the non-parametric softmax to train the CNN model as instance-level classification, which reduces the influence of some inevitable problems (e.g., shape variations). In order to obtain better performance, we integrate a few existing retrieval methods trained on different datasets. Furthermore, a query expansion strategy (i.e., diffusion) is adopted to improve the performance. Extensive experiments conducted on a dataset including half million images of beauty and personal product items (Perfect-500K) manifest the effectiveness of our proposed method. Our approach achieves the 2nd place in the leader board of the Grand Challenge of AI Meets Beauty in ACM Multimedia 2019. Our code is available at: https://github.com/RetrainIt/Perfect-Half-Million-Beauty-Product-Image-Recognition-Challenge-2019.

Abstract This paper addresses the nontraditional but practically meaningful reversibility problem of mesh filtering. This reverse‐filtering approach (termed a DeFilter) seeks to recover the geometry of a set of filtered meshes to their artifact‐free status. To solve this scenario, we adapt cascaded normal regression (CNR) to understand the commonly used mesh filters and recover automatically the mesh geometry that was lost through various geometric operations. We formulate mesh defiltering by an extreme learning machine (ELM) on the mesh normals at an offline training stage and perform it automatically at a runtime defiltering stage. Specifically, (1) to measure the local geometry of a filtered mesh, we develop a generalized reverse Filtered Facet Normal Descriptor (grFND) in the consistent neighbors; (2) to map the grFNDs to the normals of the ground‐truth meshes, we learn a regression function from a set of filtered meshes and their ground‐truth counterparts; and (3) at runtime, we reversely filter the normals of a filtered mesh, using the learned regression function for recovering the lost geometry. We evaluate multiple quantitative and qualitative results on synthetic and real data to verify our DeFilter's performance thoroughly. From a practical point of view, our method can recover the lost geometry of denoised meshes without needing to know the exact filter used previously, and can act as a geometry‐recovery plugin for most of the state‐of‐the‐art methods of mesh denoising.

Traditional methods of annotating the sentiment of an unlabeled document are based on sentiment lexicons or machine learning algorithms, which have shown low computational cost or competitive performance. However, these methods ignore the semantic composition problem displaying in several ways such as negative reversing and intensification. In this paper, we propose a new method for sentiment classification using negative and intensive sentiment supplementary information, so as to exploit the linguistic feature of negative and intensive words in conjunction with the context information. Particularly, our method can solve the domain-specific problem without relying on the external sentiment lexicons. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed method.

Unsupervised learning with generative adversarial networks (GANs) has proven to be hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss for both the discriminator and the generator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson$\chi ^2$divergence. We also show that the derived objective function that yields minimizing the Pearson$\chi ^2$divergence performs better than the classical one of using least squares for classification. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stably during the learning process. For evaluating the image quality, we conduct both qualitative and quantitative experiments, and the experimental results show that LSGANs can generate higher quality images than regular GANs. Furthermore, we evaluate the stability of LSGANs in two groups. One is to compare between LSGANs and regular GANs without gradient penalty. We conduct three experiments, including Gaussian mixture distribution, difficult architectures, and a newly proposed method — datasets with small variability, to illustrate the stability of LSGANs. The other one is to compare between LSGANs with gradient penalty (LSGANs-GP) and WGANs with gradient penalty (WGANs-GP). The experimental results show that LSGANs-GP succeed in training for all the difficult architectures used in WGANs-GP, including 101-layer ResNet.

Label-specific topics can be widely used for supporting personality psychology, aspectlevel sentiment analysis, and cross-domain sentiment classification.To generate labelspecific topics, several supervised topic models which adopt likelihood-driven objective functions have been proposed.However, it is hard for them to get a precise estimation on both topic discovery and supervised learning.In this study, we propose a supervised topic model based on the Siamese network, which can trade off label-specific word distributions with document-specific label distributions in a uniform framework.Experiments on realworld datasets validate that our model performs competitive in topic discovery quantitatively and qualitatively.Furthermore, the proposed model can effectively predict categorical or real-valued labels for new documents by generating word embeddings from a labelspecific topical space.

Cross-domain sentiment classification aims to tag sentiments for a target domain by labeled data from a source domain. Due to the difference between domains, the accuracy of a trained classifier may be very low. In this paper, we propose a boosting-based learning framework named TR-TrAdaBoost for cross-domain sentiment classification. We firstly explore the topic distribution of documents, and then combine it with the unigram TrAdaBoost. The topic distribution captures the domain information of documents, which is valuable for cross-domain sentiment classification. Experimental results indicate that TR-TrAdaBoost represents documents well and boost the performance and robustness of TrAdaBoost.

This paper addresses the task of cross-domain social emotion classification of online documents. The cross-domain task is formulated as using abundant labeled documents from a source domain and a small amount of labeled documents from a target domain, to predict the emotion of unlabeled documents in the target domain. Although several cross-domain emotion classification algorithms have been proposed, they require that feature distributions of different domains share a sufficient overlapping, which is hard to meet in practical applications. This paper proposes a novel framework, which uses the emotion distribution of training documents at the cluster level, to alleviate the aforementioned issue. Experimental results on two datasets show the effectiveness of our proposed model on cross-domain social emotion classification.

Unsupervised learning with generative adversarial networks (GANs) has proven hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss function for the discriminator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson X2 divergence. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process. We evaluate LSGANs on LSUN and CIFAR-10 datasets and the experimental results show that the images generated by LSGANs are of better quality than the ones generated by regular GANs. We also conduct two comparison experiments between LSGANs and regular GANs to illustrate the stability of LSGANs.

We have developed a method called skill2vec, which applies big data techniques to automatically analyze the learning data to discover skill relationship, leading to a more objective and data-informed decision making. Skill2vec is a neural network architecture which can transform a skill to a new vector space called embedding. The embedding can facilitate the comparison and visualization of different skills and their relationship. We conducted a pilot experiment using benchmark dataset to demonstrate the effectiveness of our method.

The rapid growth of population has posed a challenge to people for discovering new followees in uni-directional social networks. Intuitively, a user's adoption of others as followees may motivated by her interest as well as social connection. Therefore, it is worth-while to consider both factors at the same time for better recommendations. Previous recommender works on implicit follow or not feedbacks become unqualified, mainly because of the coarse users' preferences inferring, which cannot distinguish whether one follows the other is based on her social connection or individual interest. In this paper, we present a new user recommendation method which is capable of recommending candidate followees who have similar interest and closer social connection relevant to a target user. As its core, a novel topic model namely UIS-LDA is designed to jointly model a user's preferences with respect to the set of latent interest topics and social topics. The experiments using Twitter dataset proves that our proposed method effective in improving the Precision, Conversion Rate F1 score and NDCG.

Social emotion classification aims to predict the aggregation of emotional responses embedded in online comments contributed by various users. Such a task is inherently challenging because extracting relevant semantics from free texts is a classical research problem. Moreover, online comments are typically characterized by a sparse feature space, which makes the corresponding emotion classification task very difficult. On the other hand, though deep neural networks have been shown to be effective for speech recognition and image analysis tasks because of their capabilities of transforming sparse low-level features to dense high-level features, their effectiveness on emotion classification requires further investigation. The main contribution of our work reported in this paper is the development of a novel model of semantically rich hybrid neural network (HNN) which leverages unsupervised teaching models to incorporate semantic domain knowledge into the neural network to bootstrap its inference power and interpretability. To our best knowledge, this is the first successful work of incorporating semantics into neural networks to enhance social emotion classification and network interpretability. Through empirical studies based on three real-world social media datasets, our experimental results confirm that the proposed hybrid neural networks outperform other state-of-the-art emotion classification methods.

Over the past several years, as the development of Internet, social media websites such as Twitter and Weibo have received much attention due to their enormous users. A lot of research has been done on sentiment analysis and opinion mining in these websites. However the number of research on using the data in the social media websites to predict the stock market price movement is limited. Behavioral economics and behavioral finance believe that public mood is correlated with economic indicators and financial decisions are significantly driven by emotions. This paper first presents a Chinese emotion mining approach and discusses whether the public emotions or opinions in the Chinese social media websites could be used to predict the stock market price in China. The experimental results demonstrate that the emotions automatically extracted from the large scale Weibo posts represent the real public opinions about some special topics of the stock market in China. Some public mood states extracted such as the “Happiness” and “Disgust” states are highly correlated with the change of stock price according to the Granger causality analysis. Finally, a nonlinear autoregressive model with exogenous sentiment inputs is proposed to predict the stock price movement.

With the rapid growth of massive online open courses (MOOCs) on the Web, it is essential to provide learners with appropriate assistance in courses and learning materials. The extant approaches of personalized course search mainly consider historical learnt and enrolled courses of learners. That is, those courses which are contently similar to previous courses in learner profiles will be highlighted in the ranking results of the personalized course search. However, these approaches mainly neglect two distinguished characteristics in this domain, which are (i) context-dependent: course search which is highly correlated with learner contexts, e.g., a learner may have the individual learning schedule of the courses to be retrieved depending on the temporal contexts; and (ii) knowledge-constrained: learners are more willing to search and enroll in the courses that they have sufficient pre-knowledge about. To incorporate these two domain characteristics of the personalized course search, we therefore present a novel approach based on hybrid learner profile in this paper. Furthermore, we conduct the experiments which compare the performance of different methods on a dataset to verify the effectiveness of the proposed method for the personalized course search.

In recent years, there has been a proliferation of collaborative tagging systems in Web 2.0 communities. With the increasingly large amount of social data, how to manage and organize them becomes an important and crucial problem for folksonomy applications. To better understand and meet users’ needs, multimedia resources can be organized or indexed from these user perspectives; it is thus important to find latent user communities for social media applications. In this paper, we propose the mechanism of augmented folksonomy graph (AFG) to incorporate multi-faceted relations in social media, along with a novel density-based clustering method to discover latent user community from AFG by combining contents and tags of multimedia resources. To evaluate the proposed method, we conduct experiments on a public dataset, the empirical results of which show that our approach outperforms baseline ones in terms of tag-based and content-based personalized search.

With the development of the Internet, user-generated data has been growing tremendously in Web 2.0 era. Facing such a big volume of resources in folksonomy, people need a method of fast exploration and indexing to find their demanded data. To achieve this goal, contextual information is indispensable and valuable to understand user preference and purpose. In sociolinguistics, context can be mainly categorized as verbal context and social context. Comparing with verbal context, social context not only requires domain knowledge to pre-define contextual attributes but also acquires additional data from users. However, there is no research of addressing irrelevant contextual factors for verbal context model so far. The dominating set from verbal context proposed in this paper is to fill this blank. We present the verbal context in folksonomy to capture the user intention, and propose a dominating set discovering method for this verbal context model to prune the irrelevant contextual factors and keep the major characteristics at the same time. Furthermore, the experiments, which are conducted on a public data set, show that the proposed method gives convincing results.

As a necessary part of our daily life, choose what dishes to cook that is a problem and troubles many people every day. In recent years, there has been a proliferation of multimedia recipe data on the Web 2.0 communities. To assist people to navigate and search on from large amounts of recipes, a suitable recipe model is crucial and indispensable. However, recipes have some distinct characteristics that conventional data models are inadequate to represent them for such data. For example, it is unreasonable and insufficient to measure how similar the cooking procedures of two dishes are only through text descriptions and/or their extracted terms. The main reason is that this raw data (or low-level features extracted from the raw data, e.g. term for text, color for image) do not map to the high-level semantics readily. In this paper, we argue that a recipe model should be semantic-based and behavior-oriented, preferably with domain knowledge support. A hybrid semantic item (HSI) model is next presented for addressing this problem. Based on HSI model, we devise a corresponding approach for recipe search by example. The experiment on our multimedia recipe retrieval system demonstrates that our HSI approach outperforms baseline methods.