Federated Domain Generalization (FDG) aims to collaboratively train a global model across distributed clients that can generalize well on unseen domains. However, existing FDG methods typically struggle with cross-client data heterogeneity and incur significant communication and computation overhead. To address these challenges, this paper presents a new FDG framework, dubbed FaST-PT, which facilitates local feature augmentation and efficient unseen domain adaptation in a distributed manner. First, we propose a lightweight Multi-Modal Style Transfer (MST) method to transform image embedding under text supervision, which could expand the training data distribution and mitigate domain shift. We then design a dual-prompt module that decomposes the prompt into global and domain prompts. Specifically, global prompts capture general knowledge from augmented embedding across clients, while domain prompts capture domain-specific knowledge from local data. Besides, Domain-aware Prompt Generation (DPG) is introduced to adaptively generate suitable prompts for each sample, which facilitates unseen domain adaptation through knowledge fusion. Extensive experiments on four cross-domain benchmark datasets, e.g., PACS and DomainNet, demonstrate the superior performance of FaST-PT over SOTA FDG methods such as FedDG-GA and DiPrompt. Ablation studies further validate the effectiveness and efficiency of FaST-PT.

Existing backdoor attacks on multivariate time series (MTS) forecasting enforce strict temporal and dimensional coupling between triggers and target patterns, requiring synchronous activation at fixed positions across variables. However, realistic scenarios often demand delayed and variable-specific activation. We identify this critical unmet need and propose TDBA, a temporally decoupled backdoor attack framework for MTS forecasting. By injecting triggers that encode the expected location of the target pattern, TDBA enables the activation of the target pattern at any positions within the forecasted data, with the activation position flexibly varying across different variable dimensions. TDBA introduces two core modules: (1) a position-guided trigger generation mechanism that leverages smoothed Gaussian priors to generate triggers that are position-related to the predefined target pattern; and (2) a position-aware optimization module that assigns soft weights based on trigger completeness, pattern coverage, and temporal offset, facilitating targeted and stealthy attack optimization. Extensive experiments on real-world datasets show that TDBA consistently outperforms existing baselines in effectiveness while maintaining good stealthiness. Ablation studies confirm the controllability and robustness of its design.

Combining brain-computer interface (BCI) technology with the Internet of Things (IoT) for practical motor rehabilitation applications is a critical research direction aimed at enhancing the decoding performance of BCIs and their clinical application value. Currently, BCIs for single rehabilitation movements have started to be applied in clinical settings. However, in the field of electroencephalography (EEG) signal decoding, we note that research specifically focused on motor-rest classification within single motor imagery (MI) tasks remains relatively limited. To address this gap, we propose the Bidirectional Channel Attention and Mutual-Cross-Attention Temporal Feature Extraction Network (Bi-ACTCNet), adapted for operation within IoT environments. This network excels in performing Motor-Rest classification across three datasets, yielding better outcomes compared to existing models. A key innovation of Bi-ACTCNet lies in its bidirectional temporal feature extraction method, which reveals that reverse temporal signals in EEG data contain additional features that are advantageous for improving model accuracy. Additionally, the model improves the Efficient Channel Attention (ECA) block by introducing the Efficient Temporal Channel Attention (ETCA) block, and for the first time, it utilizes Mutual-Cross-Attention (MCA) to fuse features from the bidirectional branches. Ablation experiments have verified the effectiveness of each block. To further substantiate the model’s effectiveness and generalizability, we executed 2-class classification experiments using the BCI Competition IV-2a dataset, achieved excellent performance.

Despite recent progress in semi-supervised learning (SSL), its scalability remains limited in realistic scenarios where unseen classes may appear in the unlabeled data. To address this challenge, open-world SSL (OWSSL) is proposed in recent years and attracts much attention. One core difficulty in OWSSL is to enhance the representative ability for unlabeled samples, especially for those in novel classes. More recently, several works introduce contrastive learning into OWSSL and achieve impressive performance. However, they mainly focus on conducting contrastive learning solely in either feature or prediction spaces, while ignoring the thorough exploration of information potentials in dual spaces. In this study, we propose a novel method to handle OWSSL tasks via dual-space contrastive learning (DSCL). DSCL contains two modules: intraspace contrastive learning and interspace contrastive learning. In the intraspace module, we bridge the two spaces with a learnable classifier and impose contrastive learning in the dual spaces, such that the category discriminative information could be effectively utilized to improve the representative ability. In interspace module, to further enhance the utilization of complementary information from dual spaces, we introduce neighborhood information from feature space to enhance predictive learning and meanwhile utilize the cluster structure from the prediction spaces to improve intraclass compactness of the features. Compared with state-of-the-art competitors, the proposed DSCL achieves superior performance on the popular benchmarks, i.e., CIFAR100, Imagenet100, CIFAR10, CUB-200, and Scar.

Next Point-of-Interests (POIs) recommendation task aims to provide a dynamic ranking of POIs based on users' current check-in trajectories. The recommendation performance of this task is contingent upon a comprehensive understanding of users' personalized behavioral patterns through Location-based Social Networks (LBSNs) data. While prior studies have adeptly captured sequential patterns and transitional relationships within users' check-in trajectories, a noticeable gap persists in devising a mechanism for discerning specialized behavioral patterns during distinct time slots, such as noon, afternoon, or evening. In this paper, we introduce an innovative data structure termed the ``Mobility Tree'', tailored for hierarchically describing users' check-in records. The Mobility Tree encompasses multi-granularity time slot nodes to learn user preferences across varying temporal periods. Meanwhile, we propose the Mobility Tree Network (MTNet), a multitask framework for personalized preference learning based on Mobility Trees. We develop a four-step node interaction operation to propagate feature information from the leaf nodes to the root node. Additionally, we adopt a multitask training strategy to push the model towards learning a robust representation. The comprehensive experimental results demonstrate the superiority of MTNet over eleven state-of-the-art next POI recommendation models across three real-world LBSN datasets, substantiating the efficacy of time slot preference learning facilitated by Mobility Tree.

A large number of studies have emerged for Multimodal Knowledge Graph Completion (MKGC) to predict the missing links in MKGs. However, fewer studies have been proposed to study the inductive MKGC (IMKGC) involving emerging entities unseen during training. Existing inductive approaches focus on learning textual entity representations, which neglect rich semantic information in visual modality. Moreover, they focus on aggregating structural neighbors from existing KGs, which of emerging entities are usually limited. However, the semantic neighbors are decoupled from the topology linkage and usually imply the true target entity. In this paper, we propose the IMKGC task and a semantic neighbor retrieval-enhanced IMKGC framework CMR, where the contrast brings the helpful semantic neighbors close, and then the memorize supports semantic neighbor retrieval to enhance inference. Specifically, we first propose a unified cross-modal contrastive learning to simultaneously capture the textual-visual and textual-textual correlations of query-entity pairs in a unified representation space. The contrastive learning increases the similarity of positive query-entity pairs, therefore making the representations of helpful semantic neighbors close. Then, we explicitly memorize the knowledge representations to support the semantic neighbor retrieval. At test time, we retrieve the nearest semantic neighbors and interpolate them to the query-entity similarity distribution to augment the final prediction. Extensive experiments validate the effectiveness of CMR on three inductive MKGC datasets. Codes are available at https://github.com/OreOZhao/CMR.

Multi-view clustering has become a research hotspot in recent decades because of its effectiveness in heterogeneous data fusion. Although a large number of related studies have been developed one after another, most of them usually only concern with the characteristics of the data themselves and overlook the inherent connection among samples, hindering them from exploring structural knowledge of graph space. Moreover, many current works tend to highlight the compactness of one cluster without taking the differences between clusters into account. To track these two drawbacks, in this paper, we propose a graph structure aware contrastive multi-view clustering (namely, GCMC) approach. Specifically, we incorporate the well-designed graph autoencoder with conventional multi-layer perception autoencoder to extract the structural and high-level representation of multi-view data, so that the underlying correlation of samples can be effectively squeezed for model learning. Then the contrastive learning paradigm is performed on multiple pseudo-label distributions to ensure that the positive pairs of pseudo-label representations share the complementarity across views while the divergence between negative pairs is sufficiently large. This makes each semantic cluster more discriminative, i.e., jointly satisfying intra-cluster compactness and inter-cluster exclusiveness. Through comprehensive experiments on eight widely-known datasets, we prove that the proposed approach can perform better than the state-of-the-art opponents.

Enterprises and organizations are faced with potential threats from insider employees that may lead to serious consequences. Previous studies on insider threat detection (ITD) mainly focus on detecting abnormal users or abnormal time periods (e.g., a week or a day). However, a user may have hundreds of thousands of activities in the log, and even within a day there may exist thousands of activities for a user, requiring a high investigation budget to verify abnormal users or activities given the detection results. On the other hand, existing works are mainly post-hoc methods rather than real-time detection, which can not report insider threats in time before they cause loss. In this paper, we conduct the first study towards real-time ITD at activity level, and present a fine-grained and efficient framework LAN. Specifically, LAN simultaneously learns the temporal dependencies within an activity sequence and the relationships between activities across sequences with graph structure learning. Moreover, to mitigate the data imbalance problem in ITD, we propose a novel hybrid prediction loss, which integrates self-supervision signals from normal activities and supervision signals from abnormal activities into a unified loss for anomaly detection. We evaluate the performance of LAN on two widely used datasets, i.e., CERT r4.2 and CERT r5.2. Extensive and comparative experiments demonstrate the superiority of LAN, outperforming 9 state-of-the-art baselines by at least 8.43% and 6.35% in AUC for real-time ITD on CERT r4.2 and r5.2, respectively. Moreover, LAN can be also applied to post-hoc ITD, surpassing 8 competitive baselines by at least 7.70% and 4.03% in AUC on two datasets. Finally, the ablation study, parameter analysis, and compatibility analysis evaluate the impact of each module and hyper-parameter in LAN. The source code can be obtained fromhttps://github.com/Li1Neo/LAN.

User mobility behaviors frequently exhibit a spatial clustering phenomenon, wherein points of interest (POIs) visited by the same user tend to be in close proximity. Consequently, leveraging geographical influences for user preference modeling remains a prevalent approach in POI recommendation tasks. However, existing studies often overlook users’ hidden geographical habits for the following reasons: (1) Geographical features are commonly approximated by manually partitioned regions or fixed distributions, inadequately capturing the nuanced spatial proximity among POIs. (2) POIs with high geographical correlations are not explicitly incorporated as feedback signals during the training process, resulting in a lack of spatial clustering pattern learning within users’ preference representations. This paper introduces GeoCo, aGeographicalCorrelation enhanced network for POI recommendation. First, we model POIs’ geographical features using fine-grained hierarchical sequences to capture multilevel spatial relations. Subsequently, we propose a pre-training network that employs the sentence similarity assessment technique to comprehend the semantics of geographical correlations. Second, we introduce a novel multi-objective training process that intuitively learns spatial clustering patterns through user mobility behaviors. Extensive experiments conducted on two location-based social network (LBSN) datasets, Gowalla and Foursquare, demonstrate the superiority of our proposed model over fourteen state-of-the-art baseline models in POI recommendation tasks. Compared with the baselines, GeoCo has achieved a performance improvement of at least 5$\%$in Rec@5 and HR@5 on both datasets. Furthermore, we verify the effectiveness of pre-trained location vectors and the multi-objective training process in enhancing the model's understanding of geographical correlations for user preference construction.

Quality assurance of deep neural networks (DNNs) is crucial for the deployment of DNN-based software, especially in mission- and safety-critical tasks. Inspired by structural white-box testing in traditional software, many test criteria have been proposed to test DNNs, i.e., to exhibit erroneous behaviors by activating new test units that have not been covered, such as new neurons, values, and decision paths. Many studies have been done to evaluate the effectiveness of DNN test coverage criteria. However, existing empirical studies mainly focused on measuring the effectiveness of DNN test criteria for improving the adversarial robustness of DNNs, while ignoring the correctness property when testing DNNs. To fill in this gap, we conduct a comprehensive study on 11 structural coverage criteria, 6 widely-used image datasets, and 9 popular DNNs. We investigate the effectiveness of DNN coverage criteria over natural inputs from four aspects: (1) the correlation between test coverage and test diversity; (2) the effects of criteria parameters and target DNNs; (3) the effectiveness to prioritize in-distribution natural inputs that lead to erroneous behaviors; and (4) the capability to detect out-of-distribution natural samples. Our findings include: (1) For measuring the diversity, coverage criteria considering the relationship between different neurons are more effective than coverage criteria that treat each neuron independently. For instance, the neuron-path criteria (i.e., SNPC and ANPC) show high correlation with test diversity, which is promising to measure test diversity for DNNs. (2) The hyper-parameters have a big influence on the effectiveness of criteria, especially those relevant to the granularity of test criteria. Meanwhile, the computational complexity is one of the important issues to be considered when designing deep learning test coverage criteria, especially for large-scale models. (3) Test criteria related to data distribution (i.e., LSA and DSA, SNAC, and NBC) can be used to prioritize both in-distribution natural faults and out-of-distribution inputs. Furthermore, for OOD detection, the boundary metrics (i.e., SNAC and NBC) are also effective indicators with lower computational costs and higher detection efficiency compared with LSA and DSA. These findings motivate follow-up research on scalable test coverage criteria that improve the correctness of DNNs.

Biomedical entity linking is an essential task in biomedical text processing, which aims to map entity mentions in biomedical text to standard terms in a given knowledge base.However, this task is challenging due to the rarity of many biomedical entities in real-world scenarios, which leads to a lack of annotated data for them.Limited by understanding these unseen entities, traditional biomedical entity linking models suffer from multiple types of linking errors.In this paper, we propose a novel latent feature generation framework BioFEG to address these challenges.Specifically, our BioFEG leverages domain knowledge to train a generative adversarial network, which generates latent semantic features of corresponding mentions for unseen entities.Utilizing these features, we fine-tune our entity encoder to capture fine-grained coherence information of unseen entities and better understand them.This allows models to make linking decisions more accurately, particularly for ambiguous mentions involving rare entities.Extensive experiments on the two benchmark datasets demonstrate the superiority of our proposed method.

Automatic taxonomy completion aims to attach the emerging concept to an appropriate pair of hypernym and hyponym in the existing taxonomy.Existing methods suffer from the overfitting to leaf-only problem caused by imbalanced leaf and non-leaf samples when training the newly initialized classification head.Besides, they only leverage subtasks, namely attaching the concept to its hypernym or hyponym, as auxiliary supervision for representation learning yet neglect the effects of subtask results on the final prediction.To address the aforementioned limitations, we propose TacoPrompt, a Collaborative Multi-Task Prompt Learning Method for Self-Supervised Taxonomy Completion.First, we perform triplet semantic matching using the prompt learning paradigm to effectively learn non-leaf attachment ability from imbalanced training samples.Second, we design the result context to relate the final prediction to the subtask results by a contextual approach, enhancing prompt-based multi-task learning.Third, we leverage a two-stage retrieval and re-ranking approach to improve the inference efficiency.Experimental results on three datasets show that TacoPrompt achieves state-of-the-art taxonomy completion performance.Codes are available at https://github.com/cyclexu/TacoPrompt.

Open source software (OSS) licenses regulate the conditions under which OSS can be legally reused, distributed, and modified. However, a common issue arises when incorporating third-party OSS accompanied with licenses, i.e., license incompatibility, which occurs when multiple licenses exist in one project and there are conflicts between them. Despite being problematic, fixing license incompatibility issues requires substantial efforts due to the lack of license understanding and complex package dependency. In this paper, we propose LiResolver, a fine-grained, scalable, and flexible tool to resolve license incompatibility issues for open source software. Specifically, it first understands the semantics of licenses through fine-grained entity extraction and relation extraction. Then, it detects and resolves license incompatibility issues by recommending official licenses in priority. When no official licenses can satisfy the constraints, it generates a custom license as an alternative solution. Comprehensive experiments demonstrate the effectiveness of LiResolver, with 4.09% false positive (FP) rate and 0.02% false negative (FN) rate for incompatibility issue localization, and 62.61% of 230 real-world incompatible projects resolved by LiResolver. We discuss the feedback from OSS developers and the lessons learned from this work. All the datasets and the replication package of LiResolver have been made publicly available to facilitate follow-up research.

Open source software (OSS) licenses regulate the conditions under which users can reuse, modify, and distribute the software legally. However, there exist various OSS licenses in the community, written in a formal language, which are typically long and complicated to understand. In this paper, we conducted a 661-participants online survey to investigate the perspectives and practices of developers towards OSS licenses. The user study revealed an indeed need for an automated tool to facilitate license understanding. Motivated by the user study and the fast growth of licenses in the community, we propose the first study towards automated license summarization. Specifically, we released the first high quality text summarization dataset and designed two tasks, i.e., license text summarization (LTS), aiming at generating a relatively short summary for an arbitrary license, and license term classification (LTC), focusing on the attitude inference towards a predefined set of key license terms (e.g., Distribute). Aiming at the two tasks, we present LiSum, a multi-task learning method to help developers overcome the obstacles of understanding OSS licenses. Comprehensive experiments demonstrated that the proposed jointly training objective boosted the performance on both tasks, surpassing state-of-the-art baselines with gains of at least 5 points w.r.t. F1 scores of four summarization metrics and achieving 95.13% micro average F1 score for classification simultaneously. We released all the datasets, the replication package, and the questionnaires for the community.

Drug-target interaction (DTI) prediction is an essential step in drug repositioning. A few graph neural network (GNN)-based methods have been proposed for DTI prediction using heterogeneous biological data. However, existing GNN-based methods only aggregate information from directly connected nodes restricted in a drug-related or a target-related network and are incapable of capturing high-order dependencies in the biological heterogeneous graph. In this paper, we propose a metapath-aggregated heterogeneous graph neural network (MHGNN) to capture complex structures and rich semantics in the biological heterogeneous graph for DTI prediction. Specifically, MHGNN enhances heterogeneous graph structure learning and high-order semantics learning by modeling high-order relations via metapaths. Additionally, MHGNN enriches high-order correlations between drug-target pairs (DTPs) by constructing a DTP correlation graph with DTPs as nodes. We conduct extensive experiments on three biological heterogeneous datasets. MHGNN favorably surpasses 17 state-of-the-art methods over 6 evaluation metrics, which verifies its efficacy for DTI prediction. The code is available at https://github.com/Zora-LM/MHGNN-DTI.

With the rapid development of mobile technology, it is very convenient to share people’s current locations by checking-in on Location-Based Social Networks (LBSNs). Using users’ check-in histories to study mobility preferences and recommend new locations is a typical application to LBSNs. Most existing models explore reasonable representations for users and locations. However, a lack of behavioral mobility modeling would hamper a better understanding of users’ mobility patterns. This paper proposes a location recommendation model to serve the personalized LBSNs application, called Spatio-temporal Individual mobility graph encoding network with Group Mobility Assistance (SIGMA). We design a spatio-temporal interaction enhanced graph neural network to encode the mobility graphs to represent individual mobility behaviors. Furthermore, we provide a novel stacked scoring approach to generate the recommendation score by combining the stacked individual mobility graphs with the group influences. We conduct extensive experiments on two real-world LBSNs data, Foursquare and Gowalla. The result demonstrates SIGMA outperforms ten state-of-the-art models and further confirms that both the individual and the group mobility behaviors play essential roles in the practical scenario of location recommendation.

Multimodal cross-domain sentiment analysis aims at transferring domain-invariant sentiment information across datasets to address the insufficiency of labeled data. Existing adaptation methods achieve well performance by remitting the discrepancies in characteristics of multiple modalities. However, the expressive styles of different datasets also contain domain-specific information, which hinders the adaptation performance. In this article, we propose a disentangled sentiment representation adversarial network (DiSRAN) to reduce the domain shift of expressive styles for multimodal cross-domain sentiment analysis. Specifically, we first align the multiple modalities and obtain the joint representation through a cross-modality attention layer. Then, we disentangle sentiment information from the multimodal joint representation that contains domain-specific expressive style by adversarial training. The obtained sentiment representation is domain-invariant, which can better facilitate the sentiment information transfer between different domains. Experimental results on two multimodal cross-domain sentiment analysis tasks demonstrate that the proposed method performs favorably against state-of-the-art approaches.

Early detection of lung cancer is crucial for five-year survival of patients. Compared with the pathological analysis and CT scans, the circulating tumor DNA (ctDNA) methylation based approach is noninvasive and cost-effective, and thus is one of the most promising methods for early detection of lung cancer. Existing studies on ctDNA methylation data measure the methylation level of each region with a predefined metric, ignoring the positions of methylated CpG sites and methylation patterns, thus are not able to capture the early cancer signals. In this paper, we propose a blood-based lung cancer detection method, and present the first ever study to represent methylation regions by continuous vectors. Specifically, we propose DeepMeth to regard each region as a one-channel image and develop an auto-encoder model to learn its representation. For each ctDNA methylation sample, DeepMeth achieves its representation via concatenating the region vectors. We evaluate DeepMeth on a multicenter clinical dataset collected from 14 hospitals. The experiments show that DeepMeth achieves about 5%-8% improvements compared with the baselines in terms of Area Under the Curve (AUC). Moreover, the experiments also demonstrate that DeepMeth can be combined with traditional scalar metrics to enhance the diagnostic power of ctDNA methylation classifiers. DeepMeth has been clinically deployed and applied to 450 patients from 94 hospitals nationally since April 2020.

Effective drug-target binding affinity (DTA) prediction is essential for drug discovery and development. The development of machine learning techniques considerably advances it. However, the cold-start problems in DTA prediction are still under-explored, which significantly degrades prediction performances on novel drugs and novel targets. In this paper, we propose a contrastive meta-learning (CML) framework to address these issues. We define drug-anchored tasks and target-anchored tasks, which enables the employment of meta-learning to accumulate common knowledge from various tasks so as to adapt to new tasks faster and better. Besides, we utilize a task inequality loss to measure task disparities and enhance model sensitivities to new tasks. We also propose a contrastive learning block (CLB) to explore correlations among drug-target pairs across tasks, which facilitates DTA prediction performance improvements. We compare CML with various baselines on two benchmarks and comparison results show that CML outperforms or achieves competitive results to its competitors.

Identification of drug-target interactions (DTIs) is crucial for drug discovery and drug repositioning. Existing graph neural network (GNN) based methods only aggregate information from directly connected nodes restricted in a drug-related or a target-related network, and are incapable of capturing long-range dependencies in the biological heterogeneous graph. In this paper, we propose the heterogeneous graph attention network (HGAN) to capture the complex structures and rich semantics in the biological heterogeneous graph for DTI prediction. HGAN enhances heterogeneous graph structure learning from both the intra-layer perspective and the inter-layer perspective. Concretely, we develop an enhanced graph attention diffusion layer (EGADL), which efficiently builds connections between node pairs that may not be directly connected, enabling information passing from important nodes multiple hops away. By stacking multiple EGADLs, we further enlarge the receptive field from the inter-layer perspective. HGAN advances 15 state-of-the-art methods on two heterogeneous biological datasets, achieving the results near to 1 in terms of AUC and AUPR. We also find that enlarging receptive fields from the inter-layer perspective (stacking layers) is more effective than that from the intra-layer perspective (attention diffusion) for HGAN to achieve promising DTI prediction performances. The code is available at https://github.com/Zora-LM/HGAN-DTI.

Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs.Previous works usually share relation representation across modalities.This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality.Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different.In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC.Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference.Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically.Experimental results on three KG datasets show that MoSE outperforms state-ofthe-art MKGC methods.Codes are available at https://github.com/OreOZhao/MoSE4MKGC.

The prompt-based learning paradigm has gained much research attention recently. It has achieved state-of-the-art performance on several NLP tasks, especially in the few-shot scenarios. While steering the downstream tasks, few works have been reported to investigate the security problems of the prompt-based models. In this paper, we conduct the first study on the vulnerability of the continuous prompt learning algorithm to backdoor attacks. We observe that the few-shot scenarios have posed a great challenge to backdoor attacks on the prompt-based models, limiting the usability of existing NLP backdoor methods. To address this challenge, we propose BadPrompt, a lightweight and task-adaptive algorithm, to backdoor attack continuous prompts. Specially, BadPrompt first generates candidate triggers which are indicative for predicting the targeted label and dissimilar to the samples of the non-targeted labels. Then, it automatically selects the most effective and invisible trigger for each sample with an adaptive trigger optimization algorithm. We evaluate the performance of BadPrompt on five datasets and two continuous prompt models. The results exhibit the abilities of BadPrompt to effectively attack continuous prompts while maintaining high performance on the clean test sets, outperforming the baseline models by a large margin. The source code of BadPrompt is publicly available.

Deep learning (DL) brings autonomous vehicles (AVs) close to reality. However, the witness of many safety issues has raised a big concern about the reliability of AVs. To solve this problem, much research has been done to test deep learning-driven AVs. Generally, once a test input is produced, a developer needs to manually check its expected output. However, there often exists massive unlabeled test data (e.g., raw context traces in the real world). It is impractical to manually label all test inputs. Despite some works on automatic generation of test oracles, they are either task-specific or constrained to synthetic inputs. In this paper, we present a general and extensible framework,DeepSuite, to mitigate the manual effort of generating test oracles. The intuition behind is that not all test inputs are equally worth labelling. With limited testing budget, it is desirable to label a test suite with high diversity and a reasonable size. Due to the large search space, to optimize such test suites is of great challenge. To address it,DeepSuiteemploys a three-phase optimization method (i.e., selection, crossover, and mutation) to iteratively select representative but non-redundant test suites. Such conflicting profit/cost objectives are attained through a genetic algorithm with a well-defined multi-objective fitness function. In the experiments, we first show that the diversity of tests can be revealed by test criteria. Then, experiments on three widely-used datasets demonstrated the effectiveness ofDeepSuitein generating test suites with competitive testing coverage and 68.42% smaller size, which greatly improves the data collection efficiency of testing DL-driven autonomous vehicles.

Automated medical named entity recognition and normalization are fundamental for constructing knowledge graphs and building QA systems. When it comes to medical text, the annotation demands a foundation of expertise and professionalism. Existing methods utilize active learning to reduce costs in corpus annotation, as well as the multi-task learning strategy to model the correlations between different tasks. However, existing models do not take task-specific features for different tasks and diversity of query samples into account. To address these limitations, this paper proposes a multi-task adversarial active learning model for medical named entity recognition and normalization. In our model, the adversarial learning keeps the effectiveness of multi-task learning module and active learning module. The task discriminator eliminates the influence of irregular task-specific features. And the diversity discriminator exploits the heterogeneity between samples to meet the diversity constraint. The empirical results on two medical benchmarks demonstrate the effectiveness of our model against the existing methods.

Baohang Zhou, Xiangrui Cai, Ying Zhang, Xiaojie Yuan. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Fault localization can facilitate software debugging and thus is a key technology in software maintenance. Spectrum-based fault localization (SBFL) has been known as an effective and lightweight approach. Nevertheless, the existence of multiple faults within the same system is still the bottleneck of SBFL. Inspired by feature selection in data mining, this paper improves the Relief algorithm and proposes fault-relevant statistics (FRS) to calculate the suspiciousness of program elements. Specifically, it takes test cases as samples, execution results as labels, and spectrum information as features, so that the problem of multi-fault localization can be viewed as a feature selection problem. Unlike the clustering method, this paper only takes into account the closest failing and successful case for each sampled test case when computing FRS. Experiments on open source software systems show that FRS improves the efficiency of fault localization with low computational cost.

The prevailing use of both images and text to express opinions on the web leads to the need for multimodal sentiment recognition. Some commonly used social media data containing short text and few images, such as tweets and product reviews, have been well studied. However, it is still challenging to predict the readers’ sentiment after reading online news articles, since news articles often have more complicated structures, e.g., longer text and more images. To address this problem, we propose a layout-driven multimodal attention network (LD-MAN) to recognize news sentiment in an end-to-end manner. Rather than modeling text and images individually, LD-MAN uses the layout of online news to align images with the corresponding text. Specifically, it exploits a set of distance-based coefficients to model the image locations and measure the contextual relationship between images and text. LD-MAN then learns the affective representations of the articles from the aligned text and images using a multimodal attention mechanism. Considering the lack of relevant datasets in this field, we collect two multimodal online news datasets, containing a total of 14,566 articles with 56,260 images and 251,202 words. Experimental results demonstrate that the proposed method performs favorably compared with state-of-the-art approaches. We will release all the codes, models and datasets to the community.

Visual Question Answering~(VQA) requires a simultaneous understanding of images and questions. Existing methods achieve well performance by focusing on both key objects in images and key words in questions. However, the answer also contains rich information which can help to better describe the image and generate more accurate attention maps. In this paper, to utilize the information in answer, we propose a re-attention framework for the VQA task. We first associate image and question by calculating the similarity of each object-word pairs in the feature space. Then, based on the answer, the learned model re-attends the corresponding visual objects in images and reconstructs the initial attention map to produce consistent results. Benefiting from the re-attention procedure, the question can be better understood, and the satisfactory answer is generated. Extensive experiments on the benchmark dataset demonstrate the proposed method performs favorably against the state-of-the-art approaches.

The missing values, appear in most of multivariate time series, prevent advanced analysis of multivariate time series data. Existing imputation approaches try to deal with missing values by deletion, statistical imputation, machine learning based imputation and generative imputation. However, these methods are either incapable of dealing with temporal information or multi-stage. This paper proposes an end-to-end generative model E²GAN to impute missing values in multivariate time series. With the help of the discriminative loss and the squared error loss, E²GAN can impute the incomplete time series by the nearest generated complete time series at one stage. Experiments on multiple real-world datasets show that our model outperforms the baselines on the imputation accuracy and achieves state-of-the-art classification/regression results on the downstream applications. Additionally, our method also gains better time efficiency than multi-stage method on the training of neural networks.

Embeddings of medical concepts such as medication, procedure and diagnosis codes in Electronic Medical Records (EMRs) are central to healthcare analytics. Previous work on medical concept embedding takes medical concepts and EMRs as words and documents respectively. Nevertheless, such models miss out the temporal nature of EMR data. On the one hand, two consecutive medical concepts do not indicate they are temporally close, but the correlations between them can be revealed by the time gap. On the other hand, the temporal scopes of medical concepts often vary greatly (e.g., common cold and diabetes). In this paper, we propose to incorporate the temporal information to embed medical codes. Based on the Continuous Bag-of-Words model, we employ the attention mechanism to learn a ``soft'' time-aware context window for each medical concept. Experiments on public and proprietary datasets through clustering and nearest neighbour search tasks demonstrate the effectiveness of our model, showing that it outperforms five state-of-the-art baselines.

Multivariate time series usually contain a large number of missing values, which hinders the application of advanced analysis methods on multivariate time series data. Conventional approaches to addressing the challenge of missing values, including mean/zero imputation, case deletion, and matrix factorization-based imputation, are all incapable of modeling the temporal dependencies and the nature of complex distribution in multivariate time series. In this paper, we treat the problem of missing value imputation as data generation. Inspired by the success of Generative Adversarial Networks (GAN) in image generation, we propose to learn the overall distribution of a multivariate time series dataset with GAN, which is further used to generate the missing values for each sample. Different from the image data, the time series data are usually incomplete due to the nature of data recording process. A modified Gate Recurrent Unit is employed in GAN to model the temporal irregularity of the incomplete time series. Experiments on two multivariate time series datasets show that the proposed model outperformed the baselines in terms of accuracy of imputation. Experimental results also showed that a simple model on the imputed data can achieve state-of-the-art results on the prediction tasks, demonstrating the benefits of our model in downstream applications.