Foundation models pre-trained on large-scale source datasets are reshaping the traditional training paradigm for time series classification. However, existing time series foundation models primarily focus on forecasting tasks and often overlook classification-specific challenges, such as modeling interpretable shapelets that capture class-discriminative temporal features. To bridge this gap, we propose UniShape, a unified shape-aware foundation model designed for time series classification. UniShape incorporates a shape-aware adapter that adaptively aggregates multiscale discriminative subsequences (shapes) into class tokens, effectively selecting the most relevant subsequence scales to enhance model interpretability. Meanwhile, a prototype-based pretraining module is introduced to jointly learn instance- and shape-level representations, enabling the capture of transferable shape patterns. Pre-trained on a large-scale multi-domain time series dataset comprising 1.89 million samples, UniShape exhibits superior generalization across diverse target domains. Experiments on 128 UCR datasets and 30 additional time series datasets demonstrate that UniShape achieves state-of-the-art classification performance, with interpretability and ablation analyses further validating its effectiveness.

Sequential Recommendation predicts the next item based on users' past behaviors, but sparse interaction data makes user preferences hard to learn. Recently, contrastive learning has shown promise in this area. It augments data to form positive pairs and maximizing their similarity, allowing the model to learn more generalizable user interests. However, they mainly adopt uniform augmentation and alignment to all sequences, ignoring the challenges arising from their distinct interest structure, namely semantic discrepancy and semantic bias. In this paper, we first study the impact of augmentation on sequence's semantic through Interest Entropy, which measures the diversity and density of interest distribution. Our finding shows only a small fraction of sequences are stable under perturbation. These sequences mainly exhibit low or high entropy, reflecting focused or casual interests. This limits the effectiveness of contrastive learning, which relies on semantically consistent positive pairs. Furthermore, with spectral analysis, we show that positive alignment may cause low-entropy sequences to overlook niche interests, while high-entropy sequences may amplify interest-irrelevant signals, which we term semantic bias. Finally, based on Interest Entropy, we propose IERec, a simple yet effective mutual retrieval augmented contrastive learning method that mitigates the above issues in a unified manner. For each anchor sequence (those with low or high entropy), we retrieve semantically similar sequences with complementary entropy, and concatenate them to form a positive view. Sequences that are easily affected, mainly those with medium entropy, are excluded from augmentation. This approach can avoid harmful semantic discrepancy of positive pairs and reduce the effect of the semantic bias, leading to improved performance. Moreover, using interest entropy to guide contrastive learning can further improve existing CL-based SR methods.

Time series Semi-Supervised Classification (SSC) aims to improve model performance by utilizing abundant unlabeled data in scenarios where labeled samples are limited. Previous approaches mainly focus on exploiting temporal dependencies within the time domain for SSC. However, these temporal dependencies are susceptible to sampling noise and may not effectively capture the global periodicity of features across categories. To this end, we propose a time series SSC framework called CompleMatch, leveraging the complementary information from both temporal and frequency representations for unlabeled data learning. CompleMatch simultaneously trains two deep neural networks based on time-domain and frequency-domain views, with pseudo-labels generated via label propagation in the representation space guiding the training of the opposing view's classifier. In this co-training paradigm, we incorporate a constraint term to harness the complementary nature of temporal-frequency representations, thereby enhancing the model's robustness under limited labeled data. In addition, we design a temporal-frequency contrastive learning module that integrates supervised and self-supervised signals to enhance pseudo-label quality by learning more discriminative representations. Extensive experiments demonstrate that CompleMatch surpasses state-of-the-art methods. Furthermore, analyses of model behavior (i.e., ablation studies and visualization) underscore the effectiveness of our proposed approach.

Lifelong learning, also known as continual or incremental learning, is a crucial component for advancing Artificial General Intelligence (AGI) by enabling systems to continuously adapt in dynamic environments. While large language models (LLMs) have demonstrated impressive capabilities in natural language processing, existing LLM agents are typically designed for static systems and lack the ability to adapt over time in response to new challenges. This survey is the first to systematically summarize the potential techniques for incorporating lifelong learning into LLM-based agents. We categorize the core components of these agents into three modules: the perception module for multimodal input integration, the memory module for storing and retrieving evolving knowledge, and the action module for grounded interactions with the dynamic environment. We highlight how these pillars collectively enable continuous adaptation, mitigate catastrophic forgetting, and improve long-term performance. This survey provides a roadmap for researchers and practitioners working to develop lifelong learning capabilities in LLM agents, offering insights into emerging trends, evaluation metrics, and application scenarios.

In recent years, deep learning techniques have made significant strides in molecular generation for specific targets, driving advancements in drug discovery. However, existing molecular generation methods present significant limitations: those operating at the atomic level often lack synthetic feasibility, drug-likeness, and interpretability, while fragment-based approaches frequently overlook comprehensive factors that influence protein–molecule interactions. To address these challenges, we propose a novel fragment-based molecular generation framework tailored for specific proteins. Our method begins by constructing a protein subpocket and molecular arm concept-based neural network, which systematically integrates interaction force information and geometric complementarity to sample molecular arms for specific protein subpockets. Subsequently, we introduce a diffusion model to generate molecular backbones that connect these arms, ensuring structural integrity and chemical diversity. Our approach improves synthetic feasibility and binding affinity, with a 4% increase in drug-likeness and a 6% improvement in synthetic feasibility. Furthermore, by integrating explicit interaction data through a concept-based model, our framework enhances interpretability, offering valuable insights into the molecular design process.

Huawen Feng, Pu Zhao, Qingfeng Sun, Can Xu, Fangkai Yang, Lu Wang, Qianli Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Despite recent progress in Retrieval-Augmented Generation (RAG) achieved by large language models (LLMs), retrievers often recall uncorrelated documents, regarded as "noise" during subsequent text generation. To address this, some methods train LLMs to distinguish between relevant and irrelevant documents using labeled data, enabling them to select the most likely relevant ones as context. However, they remain sensitive to noise, as LLMs can easily make mistakes when the selected document is noisy. Some approaches increase the number of referenced documents and train LLMs to perform stepwise reasoning when presented with multiple documents. Unfortunately, these methods rely on extensive and diverse annotations to ensure generalization, which is both challenging and costly. In this paper, we propose **Backtracking Correction** to address these limitations. Specifically, we reformulate stepwise RAG into a multi-step decision-making process. Starting from the final step, we optimize the model through error sampling and self-correction, and then backtrack to the previous state iteratively. In this way, the model's learning scheme follows an easy-to-hard progression: as the target state moves forward, the context space decreases while the decision space increases. Experimental results demonstrate that **Backtracking Correction** enhances LLMs' ability to make complex multi-step assessments, improving the robustness of RAG in dealing with noisy documents.

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Shapelets are discriminative subsequences (or shapes) with high interpretability in time series classification. Due to the time-intensive nature of shapelet discovery, existing shapelet-based methods mainly focus on selecting discriminative shapes while discarding others to achieve candidate subsequence sparsification. However, this approach may exclude beneficial shapes and overlook the varying contributions of shapelets to classification performance. To this end, we propose a Soft sparse Shapes (SoftShape) model for efficient time series classification. Our approach mainly introduces soft shape sparsification and soft shape learning blocks. The former transforms shapes into soft representations based on classification contribution scores, merging lower-scored ones into a single shape to retain and differentiate all subsequence information. The latter facilitates intra- and inter-shape temporal pattern learning, improving model efficiency by using sparsified soft shapes as inputs. Specifically, we employ a learnable router to activate a subset of class-specific expert networks for intra-shape pattern learning. Meanwhile, a shared expert network learns inter-shape patterns by converting sparsified shapes into sequences. Extensive experiments show that SoftShape outperforms state-of-the-art methods and produces interpretable results.

Irregular multivariate time series (IMTS) are characterized by irregular time intervals within variables and unaligned observations across variables, posing challenges in learning temporal and variable dependencies. Many existing IMTS models either require padded samples to learn separately from temporal and variable dimensions, or represent original samples via bipartite graphs or sets. However, the former approaches often need to handle extra padding values affecting efficiency and disrupting original sampling patterns, while the latter ones have limitations in capturing dependencies among unaligned observations. To represent and learn both dependencies from original observations in a unified form, we propose HyperIMTS, a Hypergraph neural network for Irregular Multivariate Time Series forecasting. Observed values are converted as nodes in the hypergraph, interconnected by temporal and variable hyperedges to enable message passing among all observations. Through irregularity-aware message passing, HyperIMTS captures variable dependencies in a time-adaptive way to achieve accurate forecasting. Experiments demonstrate HyperIMTS’s competitive performance among state-of-the-art models in IMTS forecasting with low computational cost. Our code is available at https://github.com/qianlima-lab/PyOmniTS.

Multi-scale information is crucial for multivariate time series modeling. However, most existing time series multi-scale analysis methods treat all variables in the same manner, making them unsuitable for Irregular Multivariate Time Series (IMTS), where variables have distinct origin scales/sampling rates. To fill this gap, we propose Hi-Patch, a hierarchical patch graph network. Hi-Patch encodes each observation as a node, represents and captures local temporal and inter-variable dependencies of densely sampled variables through an intra-patch graph layer, and obtains patch-level nodes through aggregation. These nodes are then updated and re-aggregated through a stack of inter-patch graph layers, where several scale-specific graph networks progressively extract more global temporal and inter-variable features of both sparsely and densely sampled variables under specific scales. The output of the last layer is fed into task-specific decoders to adapt to different downstream tasks. Experiments on 8 datasets demonstrate that Hi-Patch outperforms state-of-the-art models in IMTS forecasting and classification tasks.

Gaussian Process Latent Variable Models (GPLVMs) have become increasingly popular for unsupervised tasks such as dimensionality reduction and missing data recovery due to their flexibility and non-linear nature. An importance-weighted version of the Bayesian GPLVMs has been proposed to obtain a tighter variational bound. However, this version of the approach is primarily limited to analyzing simple data structures, as the generation of an effective proposal distribution can become quite challenging in high-dimensional spaces or with complex data sets. In this work, we propose VAIS-GPLVM, a variational Annealed Importance Sampling method that leverages time-inhomogeneous unadjusted Langevin dynamics to construct the variational posterior. By transforming the posterior into a sequence of intermediate distributions using annealing, we combine the strengths of Sequential Monte Carlo samplers and VI to explore a wider range of posterior distributions and gradually approach the target distribution. We further propose an efficient algorithm by reparameterizing all variables in the evidence lower bound (ELBO). Experimental results on both toy and image datasets demonstrate that our method outperforms state-of-the-art methods in terms of tighter variational bounds, higher log-likelihoods, and more robust convergence.

Deep Gaussian process (DGP) models offer a powerful nonparametric approach for Bayesian inference, but exact inference is typically intractable, motivating the use of various approximations. However, existing approaches, such as mean-field Gaussian assumptions, limit the expressiveness and efficacy of DGP models, while stochastic approximation can be computationally expensive. To tackle these challenges, we introduce neural operator variational inference (NOVI) for DGPs. NOVI uses a neural generator to obtain a sampler and minimizes the regularized Stein discrepancy (RSD) between the generated distribution and true posterior in $\mathcal {L}_{2}$ space. We solve the minimax problem using Monte Carlo estimation and subsampling stochastic optimization techniques and demonstrate that the bias introduced by our method can be controlled by multiplying the Fisher divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm. Our experiments on datasets ranging from hundreds to millions demonstrate the effectiveness and the faster convergence rate of the proposed method. We achieve a classification accuracy of 93.56 on the CIFAR10 dataset, outperforming state-of-the-art (SOTA) Gaussian process (GP) methods. We are optimistic that NOVI possesses the potential to enhance the performance of deep Bayesian nonparametric models and could have significant implications for various practical applications.

Multivariate time series (MTS) prediction has been studied broadly, which is widely applied in real-world applications. Recently, transformer-based methods have shown the potential in this task for their strong sequence modeling ability. Despite progress, these methods pay little attention to extracting short-term information in the context, while short-term patterns play an essential role in reflecting local temporal dynamics. Moreover, we argue that there are both consistent and specific characteristics among multiple variables, which should be fully considered for MTS modeling. To this end, we propose a multiresolution transformer (MR-Transformer) for MTS prediction, modeling MTS from both the temporal and the variable resolution. Specifically, for the temporal resolution, we design a long short-term transformer. We first split the sequence into nonoverlapping segments in an adaptive way and then extract short-term patterns within segments, while long-term patterns are captured by the inherent attention mechanism. Both of them are aggregated together to capture the temporal dependencies. For the variable resolution, besides the variable-consistent features learned by long short-term transformer, we also design a temporal convolution module to capture the specific features of each variable individually. MR-Transformer enhances the MTS modeling ability by combining multiresolution features between both time steps and variables. Extensive experiments conducted on real-world time series datasets show that MR-Transformer significantly outperforms the state-of-the-art MTS prediction models. The visualization analysis also demonstrates the effectiveness of the proposed model.

Semi-supervised time-series classification could effectively alleviate the issue of lacking labeled data. However, existing approaches usually ignore model interpretability, making it difficult for humans to understand the principles behind the predictions of a model. Shapelets are a set of discriminative subsequences that show high interpretability in time series classification tasks. Shapelet learning-based methods have demonstrated promising classification performance. Unfortunately, without enough labeled data, the shapelets learned by existing methods are often poorly discriminative, and even dissimilar to any subsequence of the original time series. To address this issue, we propose the Diffusion Language-Shapelets model (DiffShape) for semi-supervised time series classification. In DiffShape, a self-supervised diffusion learning mechanism is designed, which uses real subsequences as a condition. This helps to increase the similarity between the learned shapelets and real subsequences by using a large amount of unlabeled data. Furthermore, we introduce a contrastive language-shapelets learning strategy that improves the discriminability of the learned shapelets by incorporating the natural language descriptions of the time series. Experiments have been conducted on the UCR time series archive, and the results reveal that the proposed DiffShape method achieves state-of-the-art performance and exhibits superior interpretability over baselines.

Incremental Learning (IL) has been a longstanding problem in both vision and Natural Language Processing (NLP) communities.In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP.Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue.However, we find that this assumption is problematic.Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs.Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs.The results show that SEQ* has competitive or superior performance compared with state-ofthe-art (SOTA) IL methods yet requires considerably less trainable parameters and training time.These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs.The data, code and scripts are publicly available 1 .

Hierarchical Text Classification (HTC) is a challenging task which aims to extract the labels in a tree structure corresponding to a given text. Discriminative methods usually incorporate the hierarchical structure information into the encoding process, while generative methods decode the features according to it. However, the data distribution varies widely among different categories of samples, but current methods ignore the data imbalance, making the predictions biased and susceptible to error propagation. In this paper, we propose an IMplicitly Augmented Generativ E framework with distribution modification for hierarchical text classification (IMAGE). Specifically, we translate the distributions of original samples along various directions through implicit augmentation to get more diverse data. Furthermore, given the scarcity of the samples of tail classes, we adjust their distributions by transferring knowledge from other classes in label space. In this way, the generative framework learns a better beginning of the feature sequence without a prediction bias and avoids being misled by its wrong predictions for head classes. Experimental results show that IMAGE obtains competitive results compared with state-of-the-art methods and prove its superiority on unbalanced data.

Sequential Recommendation utilizes interaction history to uncover users' dynamic interest changes and recommend the most relevant items for their next interaction. In recent years, multi-behavior modeling and multi-interest modeling have been hot research topics. Although multi-behavior and multi-interest methods have strengths in their respective domains, both have limitations. Multi-behavior methods focus excessively on target behavior recommendation (i.e., purchase) without sufficiently leveraging auxiliary behavior interactions (i.e., click) to discern users' multi-faced interests, leading to suboptimal recommendation quality. Meanwhile, existing multi-interest methods overlook the distinct user interests behind multi-behavior when extracting interests, resulting in inaccurate interest modeling. Combining the two can not only facilitate sophisticated modeling of complex user interests but also deepen understanding of multi-behavior interactions, achieving synergistic effects. In this paper, we propose a novel approach called Multi-Interest Self-Supervised Learning (MISSL) that precisely unifies multi-behavior and multi-interest modeling to obtain more comprehensive and accurate user profiles. MISSL utilizes a hypergraph transformer network to extract behavior-specific and shared interests followed by multi-interest self-supervised learning to refine item and interest representations. Additionally, a behavior-aware training task is incorporated to enhance model stability during training. Extensive experiments on benchmark datasets demonstrate that MISSL outperforms baseline methods. The source code for MISSL is available at: https://github.com/qianlima-Iab/MISSL.

Complex Query Answering (CQA) over Knowledge Graphs (KGs) is a challenging task. Given that KGs are usually incomplete, neural models are proposed to solve CQA by performing multi-hop logical reasoning. However, most of them cannot perform well on both one-hop and multi-hop queries simultaneously. Recent work proposes a logical message passing mechanism based on the pre-trained neural link predictors. While effective on both one-hop and multi-hop queries, it ignores the difference between the constant and variable nodes in a query graph. In addition, during the node embedding update stage, this mechanism cannot dynamically measure the importance of different messages, and whether it can capture the implicit logical dependencies related to a node and received messages remains unclear. In this paper, we propose Conditional Logical Message Passing Transformer (CLMPT), which considers the difference between constants and variables in the case of using pre-trained neural link predictors and performs message passing conditionally on the node type. We empirically verified that this approach can reduce computational costs without affecting performance. Furthermore, CLMPT uses the transformer to aggregate received messages and update the corresponding node embedding. Through the self-attention mechanism, CLMPT can assign adaptive weights to elements in an input set consisting of received messages and the corresponding node and explicitly model logical dependencies between various elements. Experimental results show that CLMPT is a new state-of-the-art neural CQA model. https://github.com/qianlima-lab/CLMPT.

Irregularly Sampled Medical Time Series (ISMTS) are commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features at the variable level in ISMTS. We propose Knowledge-Empowered Dynamic Graph Network (KEDGN), a graph neural network empowered by variables' textual medical knowledge, aiming to model variable-specific temporal dependencies and inter-variable dependencies in ISMTS. Specifically, we leverage a pre-trained language model to extract semantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from a medical perspective. Based on this, we allocate variable-specific parameter spaces to capture variable-specific temporal patterns and generate a complete variable graph to measure medical correlations among variables. Additionally, we employ a density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variable-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra-variable and inter-variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods.

Hierarchical Text Classification (HTC) is an essential and challenging task due to the difficulty of modeling label hierarchy. Recent generative methods have achieved state-of-the-art performance by flattening thelocal label hierarchyinto a label sequence with a specific order. However, the order between labels does not naturally exist and the generation of the current label should incorporate the information in all other target labels. Moreover, the generative methods usually suffer from the error accumulation problem. To this end, we propose a new framework named sequence-to-label (Seq2Label) with a random generative way to learn label hierarchy for hierarchical text classification. Instead of using only one specific order, we shuffle the label sequence by a Label Sequence Random Shuffling (LSRS) mechanism so that a text will be mapped to several different order label sequences during the training phase. To alleviate the error accumulation problem, we further propose a Hierarchy-aware Negative Sampling (HNS) strategy with a negative label-aware loss to better distinguish target labels and negative labels. In this way, our model can capture the hierarchical and co-occurrence information of the target labels of each text. The experimental results on three benchmark datasets show that Seq2Label achieves state-of-the-art results.

Time-Series Mining (TSM) is an important research area since it shows great potential in practical applications. Deep learning models that rely on massive labeled data have been utilized for TSM successfully. However, constructing a large-scale well-labeled dataset is difficult due to data annotation costs. Recently, pre-trained models have gradually attracted attention in the time series domain due to their remarkable performance in computer vision and natural language processing. In this survey, we provide a comprehensive review of Time-Series Pre-Trained Models (TS-PTMs), aiming to guide the understanding, applying, and studying TS-PTMs. Specifically, we first briefly introduce the typical deep learning models employed in TSM. Then, we give an overview of TS-PTMs according to the pre-training techniques. The main categories we explore include supervised, unsupervised, and self-supervised TS-PTMs. Further, extensive experiments involving 27 methods, 434 datasets, and 679 transfer learning scenarios are conducted to analyze the advantages and disadvantages of transfer learning strategies, Transformer-based models, and representative TS-PTMs. Finally, we point out some potential directions of TS-PTMs for future work.

Semi-supervised learning (SSL) has been actively studied due to its ability to alleviate the reliance of deep learning models on labeled data. Although existing SSL methods based on pseudo-labeling strategies have made great progress, they rarely consider time-series data's intrinsic properties (e.g., temporal dependence). Learning representations by mining the inherent properties of time series has recently gained much attention. Nonetheless, how to utilize feature representations to design SSL paradigms for time series has not been explored. To this end, we propose a Time Series SSL framework via Temporal-Frequency Co-training (TS-TFC), leveraging the complementary information from two distinct views for unlabeled data learning. In particular, TS-TFC employs time-domain and frequency-domain views to train two deep neural networks simultaneously, and each view's pseudo-labels generated by label propagation in the representation space are adopted to guide the training of the other view's classifier. To enhance the discriminative of representations between categories, we propose a temporal-frequency supervised contrastive learning module, which integrates the learning difficulty of categories to improve the quality of pseudo-labels. Through co-training the pseudo-labels obtained from temporal-frequency representations, the complementary information in the two distinct views is exploited to enable the model to better learn the distribution of categories. Extensive experiments on 106 UCR datasets show that TS-TFC outperforms state-of-the-art methods, demonstrating the effectiveness and robustness of our proposed model.

Emotion-Cause Pair Extraction (ECPE) aims to identify the document's emotion clauses and corresponding cause clauses.Like other relation extraction tasks, ECPE is closely associated with the relationship between sentences.Recent methods based on Graph Convolutional Networks focus on how to model the multiplex relations between clauses by constructing different edges.However, the data of emotions, causes, and pairs are extremely unbalanced, but current methods get their representation using the same graph structure.In this paper, we propose a Joint Constrained Learning framework with Boundary-adjusting for Emotion-Cause Pair Extraction (JCB).Specifically, through constrained learning, we summarize the prior rules existing in the data and force the model to take them into consideration in optimization, which helps the model learn a better representation from unbalanced data.Furthermore, we adjust the decision boundary of classifiers according to the relations between subtasks, which have always been ignored.No longer working independently as in the previous framework, the classifiers corresponding to three subtasks cooperate under the relation constraints.Experimental results show that JCB obtains competitive results compared with state-of-theart methods and prove its robustness on unbalanced data.

Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Noisy labels seriously degrade the generalization ability of Deep Neural Networks (DNNs) in various classification tasks. Existing studies on label-noise learning mainly focus on computer vision, while time series also suffer from the same issue. Directly applying the methods from computer vision to time series may reduce the temporal dependency due to different data characteristics. How to make use of the properties of time series to enable DNNs to learn robust representations in the presence of noisy labels has not been fully explored. To this end, this paper proposes a method that expands the distribution of Confident instances by Time-Warping (CTW) to learn robust representations of time series. Specifically, since applying the augmentation method to all data may introduce extra mislabeled data, we select confident instances to implement Time-Warping. In addition, we normalize the distribution of the training loss of each class to eliminate the model's selection preference for instances of different classes, alleviating the class imbalance caused by sample selection. Extensive experimental results show that CTW achieves state-of-the-art performance on the UCR datasets when dealing with different types of noise. Besides, the t-SNE visualization of our method verifies that augmenting confident data improves the generalization ability. Our code is available at https://github.com/qianlima-lab/CTW.

Deep Neural Networks (DNNs) have been criticized because they easily overfit noisy (incorrect) labels. To improve the robustness of DNNs, existing methods for image data regard samples with small training losses as correctly labeled data (small-loss criterion). Nevertheless, time series' discriminative patterns are easily distorted by external noises (i.e., frequency perturbations) during the recording process. This results in training losses of some time series samples that do not meet the small-loss criterion. Therefore, this paper proposes a deep learning paradigm called Scale-teaching to cope with time series noisy labels. Specifically, we design a fine-to-coarse cross-scale fusion mechanism for learning discriminative patterns by utilizing time series at different scales to train multiple DNNs simultaneously. Meanwhile, each network is trained in a cross-teaching manner by using complementary information from different scales to select small-loss samples as clean labels. For unselected large-loss samples, we introduce multi-scale embedding graph learning via label propagation to correct their labels by using selected clean samples. Experiments on multiple benchmark time series datasets demonstrate the superiority of the proposed Scale-teaching paradigm over state-of-the-art methods in terms of effectiveness and robustness.

In text classification, the traditional attention mechanisms usually focus too much on frequent words, and need extensive labeled data in order to learn. This article proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overhead. Specifically, we add as much noise as possible to all the words in the sentence without changing their semantics and predictions. We hypothesize that words that tolerate more noise are less significant, and we can use this information to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly improve the performance of current attention-based models, and is more effective than existing self-supervised methods. We also provide a visualization analysis to verify the effectiveness of our approach.

Emotion-cause pair extraction (ECPE) is an emerging task born out of Emotion cause extraction (ECE), which aims to extract the emotion clause and the corresponding cause clause simultaneously. Previous methods decompose ECPE into multiple sub-tasks, namely emotion clause extraction, cause clause extraction, and emotion-cause pair extraction, and employ different modules to address them separately. However, these methods fail to effectively capture the mutuality within the three sub-tasks, which may hinder the information interaction between emotion and cause. In this paper, we revisit and analyze the mutuality between emotion and cause clauses from a linguistic perspective and further propose a novel Modularized Mutuality Network (MMN) to capture the mutuality explicitly. Specifically, the mutuality can be divided into the following categories, including position bias, sentiment consistency, and natural duality. To this end, we design three modules wrapped with various simple but effective mechanisms to address the mutuality, respectively. Extensive experiments demonstrate that MMN achieves state-of-the-art performances on the ECPE task and detailed analyzed the effect of the three modules for capturing the mutuality within sub-tasks.

In e-commerce search engines, query rewriting (QR) is a crucial technique that improves shopping experience by reducing the vocabulary gap between user queries and product catalog. Recent works have mainly adopted the generative paradigm. However, they hardly ensure high-quality generated rewrites and do not consider personalization, which leads to degraded search relevance. In this work, we present Contrastive Learning Enhanced Query Rewriting (CLE-QR), the solution used in Taobao product search. It uses a novel contrastive learning enhanced architecture based on "query retrieval-semantic relevance ranking-online ranking". It finds the rewrites from hundreds of millions of historical queries while considering relevance and personalization. Specifically, we first alleviate the representation degeneration problem during the query retrieval stage by using an unsupervised contrastive loss, and then further propose an interaction-aware matching method to find the beneficial and incremental candidates, thus improving the quality and relevance of candidate queries. We then present a relevance-oriented contrastive pre-training paradigm on the noisy user feedback data to improve semantic ranking performance. Finally, we rank these candidates online with the user profile to model personalization for the retrieval of more relevant products. We evaluate CLE-QR on Taobao Product Search, one of the largest e-commerce platforms in China. Significant metrics gains are observed in online A/B tests. CLE-QR has been deployed to our large-scale commercial retrieval system and serviced hundreds of millions of users since December 2021. We also introduce its online deployment scheme, and share practical lessons and optimization tricks of our lexical match system.

Emotion-cause pair extraction (ECPE) aims to extract emotion clauses and corresponding cause clauses, which have recently received growing attention.Previous methods sequentially encode features with a specified order.They first encode the emotion and cause features for clause extraction and then combine them for pair extraction.This lead to an imbalance in inter-task feature interaction where features extracted later have no direct contact with the former.To address this issue, we propose a novel Pair-Based Joint Encoding (PBJE) network, which generates pairs and clauses features simultaneously in a joint feature encoding manner to model the causal relationship in clauses.PBJE can balance the information flow among emotion clauses, cause clauses and pairs.From a multi-relational perspective, we construct a heterogeneous undirected graph and apply the Relational Graph Convolutional Network (RGCN) to capture the various relationship between clauses and the relationship between pairs and clauses.Experimental results show that PBJE achieves state-of-the-art performance on the Chinese benchmark corpus.

Continual Learning for Named Entity Recognition (CL-NER) aims to learn a growing number of entity types over time from a stream of data.However, simply learning Other-Class in the same way as new entity types amplifies the catastrophic forgetting and leads to a substantial performance drop.The main cause behind this is that Other-Class samples usually contain old entity types, and the old knowledge in these Other-Class samples is not preserved properly.Thanks to the causal inference, we identify that the forgetting is caused by the missing causal effect from the old data.To this end, we propose a unified causal framework to retrieve the causality from both new entity types and Other-Class.Furthermore, we apply curriculum learning to mitigate the impact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entity types and Other-Class.Experimental results on three benchmark datasets show that our method outperforms the state-of-theart method by a large margin.Moreover, our method can be combined with the existing stateof-the-art methods to improve the performance in CL-NER. 1

Incomplete time series classification (ITSC) is an important issue in time series analysis since temporal data often has missing values in practical applications. However, integrating imputation (replacing missing data) and classification within a model often rapidly amplifies the error from imputed values. Reducing this error propagation from imputation to classification remains a challenge. To this end, we propose an adversarial joint-learning recurrent neural network (AJ-RNN) for ITSC, an end-to-end model trained in an adversarial and joint learning manner. We train the system to categorize the time series as well as impute missing values. To alleviate the error introduced by each imputation value, we use an adversarial network to encourage the network to impute realistic missing values by distinguishing real and imputed values. Hence, AJ-RNN can directly perform classification with missing values and greatly reduce the error propagation from imputation to classification, boosting the accuracy. Extensive experiments on 68 synthetic datasets and 4 real-world datasets from the expanded UCR time series archive demonstrate that AJ-RNN achieves state-of-the-art performance. Furthermore, we show that our model can effectively alleviate the accumulating error problem through qualitative and quantitative analysis based on the trajectory of the dynamical system learned by the RNN. We also provide an analysis of the model behavior to verify the effectiveness of our approach.

Multivariate time series (MTSs) are widely found in many important application fields, for example, medicine, multimedia, manufacturing, action recognition, and speech recognition. The accurate classification of MTS has become an important research topic. Traditional MTS classification methods do not explicitly model the temporal difference information of time series, which is, in fact, important and reflects the dynamic evolution information. In this article, the difference-guided representation learning network (DGRL-Net) is proposed to guide the representation learning of time series by dynamic evolution information. The DGRL-Net consists of a difference-guided layer and a multiscale convolutional layer. First, in the difference-guided layer, we propose a difference gating LSTM to model the time dependency and dynamic evolution of the time series to obtain feature representations of both raw and difference series. Then, these two representations are used as two input channels of the multiscale convolutional layer to extract multiscale information. Extensive experiments demonstrate that the proposed model outperforms state-of-the-art methods on 18 MTS benchmark datasets and achieves competitive results on two skeleton-based action recognition datasets. Furthermore, the ablation study and visualized analysis are designed to verify the effectiveness of the proposed model.

Time-series clustering is an essential unsupervised technique for data analysis, applied to many real-world fields, such as medical analysis and DNA microarray. Existing clustering methods are usually based on the assumption that the data is complete. However, time series in real-world applications often contain missing values. Traditional strategy (imputing first and then clustering) does not optimize the imputation and clustering process as a whole, which not only makes per- formance dependent on the combination of imputation and clustering methods but also fails to achieve satisfactory re- sults. How to best improve the clustering performance on incomplete time series remains a challenge. This paper pro- poses a novel unsupervised temporal representation learning model, named Clustering Representation Learning on Incom- plete time-series data (CRLI). CRLI jointly optimizes the im- putation and clustering process to impute more discrimina- tive values for clustering and make the learned representa- tions possessed good clustering property. Also, to reduce the error propagation from imputation to clustering, we introduce a discriminator to make the distribution of imputation values close to the true one and train CRLI in an alternating train- ing manner. An experiment conducted on eight real-world in- complete time-series datasets shows that CRLI outperforms existing methods. We demonstrates the effectiveness of the learned representations and the convergence of the model through visualization analysis. Moreover, we reveal that the joint training strategy can impute values close to the true ones in those important sub-sequences, and impute more discrim- inative values in those less important sub-sequences at the same time, making the imputed sequence cluster-friendly.

Recently, deep neural networks (DNNs) have achieved excellent performance on time series classification. However, DNNs require large amounts of labeled data for supervised training. Although data augmentation can alleviate this problem, the standard approach assigns the same label to all augmented samples from the same source. This leads to the expansion of the data distribution such that the classification boundaries may be even harder to determine. In this paper, we propose Joint-label learning by Dual Augmentation (JobDA), which can enrich the training samples without expanding the distribution of the original data. Instead, we apply simple transformations to the time series and give these modified time series new labels, so that the model has to distinguish between these and the original data, as well as separating the original classes. This approach sharpens the boundaries around the original time series, and results in superior classification performance. We use Time Series Warping for our transformations: We shrink and stretch different regions of the original time series, like a fun-house mirror. Experiments conducted on extensive time-series datasets show that JobDA can improve the model performance on small datasets. Moreover, we verify that JobDA has better generalization ability compared with conventional data augmentation, and the visualization analysis further demonstrates that JobDA can learn more compact clusters.

Recurrent autoencoder is a popular model for time series anomaly detection, in which outliers or abnormal segments are identified by their high reconstruction errors. However, existing recurrent autoencoders can easily suffer from overfitting and error accumulation due to sequential decoding. In this paper, we propose a simple yet efficient recurrent network ensemble called Recurrent Autoencoder with Multiresolution Ensemble Decoding (RAMED). By using decoders with different decoding lengths and a new coarse-to-fine fusion mechanism, lower-resolution information can help long-range decoding for decoders with higher-resolution outputs. A multiresolution shape-forcing loss is further introduced to encourage decoders' outputs at multiple resolutions to match the input's global temporal shape. Finally, the output from the decoder with the highest resolution is used to obtain an anomaly score at each time step. Extensive empirical studies on real-world benchmark data sets demonstrate that the proposed RAMED model outperforms recent strong baselines on time series anomaly detection.

Haibin Chen, Qianli Ma, Zhenxi Lin, Jiangyue Yan. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Metaphors are ubiquitous in natural language, and detecting them requires contextual reasoning about whether a semantic incongruence actually exists.Most existing work addresses this problem using pre-trained contextualized models.Despite their success, these models require a large amount of labeled data and are not linguistically-based.In this paper, we proposed a ContrAstive pre-Trained modEl (CATE) for metaphor detection with semi-supervised learning.Our model first uses a pre-trained model to obtain a contextual representation of target words and employs a contrastive objective to promote an increased distance between target words' literal and metaphorical senses based on linguistic theories.Furthermore, we propose a simple strategy to collect large-scale candidate instances from the general corpus and generalize the model via self-training.Extensive experiments show that CATE achieves better performance against state-of-the-art baselines on several benchmark datasets.

Multi-scale information is crucial for modeling time series. Although most existing methods consider multiple scales in the time-series data, they assume all kinds of scales are equally important for each sample, making them unable to capture the dynamic temporal patterns of time series. To this end, we propose Time-Aware Multi-Scale Recurrent Neural Networks (TAMS-RNNs), which disentangle representations of different scales and adaptively select the most important scale for each sample at each time step. First, the hidden state of the RNN is disentangled into multiple independently updated small hidden states, which use different update frequencies to model time-series multi-scale information. Then, at each time step, the temporal context information is used to modulate the features of different scales, selecting the most important time-series scale. Therefore, the proposed model can capture the multi-scale information for each time series at each time step adaptively. Extensive experiments demonstrate that the model outperforms state-of-the-art methods on multivariate time series classification and human motion prediction tasks. Furthermore, visualized analysis on music genre recognition verifies the effectiveness of the model.

Nowadays, the product search service of e-commerce platforms has become a vital shopping channel in people's life. The retrieval phase of products determines the search system's quality and gradually attracts researchers' attention. Retrieving the most relevant products from a large-scale corpus while preserving personalized user characteristics remains an open question. Recent approaches in this domain have mainly focused on embedding-based retrieval (EBR) systems. However, after a long period of practice on Taobao, we find that the performance of the EBR system is dramatically degraded due to its: (1) low relevance with a given query and (2) discrepancy between the training and inference phases. Therefore, we propose a novel and practical embedding-based product retrieval model, named Multi-Grained Deep Semantic Product Retrieval (MGDSPR). Specifically, we first identify the inconsistency between the training and inference stages, and then use the softmax cross-entropy loss as the training objective, which achieves better performance and faster convergence. Two efficient methods are further proposed to improve retrieval relevance, including smoothing noisy training data and generating relevance-improving hard negative samples without requiring extra knowledge and training procedures. We evaluate MGDSPR on Taobao Product Search with significant metrics gains observed in offline experiments and online A/B tests. MGDSPR has been successfully deployed to the existing multi-channel retrieval system in Taobao Search. We also introduce the online deployment scheme and share practical lessons of our retrieval system to contribute to the community.

Cold-start recommendation is a challenging problem due to the lack of user-item interactions. Recently, heterogeneous information network~(HIN)-based recommendation methods use rich auxiliary information to enhance users and items' connections, helping alleviate the cold-start problem. Despite progress, most existing methods model HINs under traditional supervised learning settings, ignoring the gaps between training and inference procedures in cold-start scenarios. In this paper, we regard cold-start recommendation as a missing data problem where some user-item interaction data are missing. Inspired by denoising auto-encoders that train a model to reconstruct the input from its corrupted version, we propose a novel model called Multi-view Denoising Graph Auto-Encoders~(MvDGAE) on HINS. Specifically, we first extract multifaceted meaningful semantics on HINs as multi-views for both users and items, effectively enhancing user/item relationships on different aspects. Then we conduct the training procedure by randomly dropping out some user-item interactions in the encoder while forcing the decoder to use these limited views to recover the full views, including the missing ones. In this way, the complementary representations for both users and items are more informative and robust to adjust to cold-start scenarios. Moreover, the decoder's reconstruction goals are multi-view user-user and item-item relationship graphs rather than the original input graphs, which make the features of similar users (or items) in the meta-paths closer together. Finally, we adopt a Bayesian task weight learner to balance multi-view graph reconstruction objectives automatically. Extensive experiments on both public benchmark datasets and a large-scale industry dataset WeChat Channel demonstrate that MvDGAE significantly outperforms the state-of-the-art recommendation models in various cold-start scenarios. The case studies also illustrate that MvDGAE has potentially good interpretability.

Current state-of-the-art sequence labeling models are typically based on sequential architecture such as Bi-directional LSTM (BiLSTM). However, the structure of processing a word at a time based on the sequential order restricts the full utilization of non-sequential features, including syntactic relationships, word co-occurrence relations, and document topics. They can be regarded as the corpus-level features and critical for sequence labeling. In this paper, we propose a Corpus-Aware Graph Aggregation Network. Specifically, we build three types of graphs, i.e., a word-topic graph, a word co-occurrence graph, and a word syntactic dependency graph, to express different kinds of corpus-level non-sequential features. After that, a graph convolutional network (GCN) is adapted to model the relations between words and non-sequential features. Finally, we employ a label-aware attention mechanism to aggregate corpus-aware non-sequential features and sequential ones for sequence labeling. The experimental results on four sequence labeling tasks (named entity recognition, chunking, multilingual sequence labeling, and target-based sentiment analysis) show that our model achieves state-of-the-art performance.

Text classification is an important task in natural language processing. Contextual information is essential for text classification, and different words usually need different sizes of contextual information. However, most existing methods learn contextual features with predefined fixed sizes, which cannot extract the different sizes of contextual features for different words. To this end, we propose a new model named Deformable Self-Attention (DSA) to flexibly learn word-specific contextual features, rather than extracting features of fixed context sizes. Our model is mainly composed of a Deformable Local Attention Weight Generation (DLAWG) module and a Multi-Range Feature Integration (MRFI) module. The DLAWG module can adaptively determine different context sizes for different words within a particular range and then learn word-specific contextual features for each word. DLAWG then employs multiple ranges to capture context dependencies of different ranges. After that, the MRFI module integrates features from different ranges by considering the interactions with features of different ranges, which can delete irrelevant features while enhancing discriminative ones. Experiments on extensive benchmark datasets and visualizations illustrate the effectiveness of our model.

As efficient recurrent neural network (RNN) models, echo state networks (ESNs) have attracted widespread attention and been applied in many application domains in the last decade. Although they have achieved great success in modeling time series, a single ESN may have difficulty in capturing the multitimescale structures that naturally exist in temporal data. In this paper, we propose the convolutional multitimescale ESN (ConvMESN), which is a novel training-efficient model for capturing multitimescale structures and multiscale temporal dependencies of temporal data. In particular, a multitimescale memory encoder is constructed with a multireservoir structure, in which different reservoirs have recurrent connections with different skip lengths (or time spans). By collecting all past echo states in each reservoir, this multireservoir structure encodes the history of a time series as nonlinear multitimescale echo state representations (MESRs). Our visualization analysis verifies that the MESRs provide better discriminative features for time series. Finally, multiscale temporal dependencies of MESRs are learned by a convolutional layer. By leveraging the multitimescale reservoirs followed by a convolutional learner, the ConvMESN has not only efficient memory encoding ability for temporal data with multitimescale structures but also strong learning ability for complex temporal dependencies. Furthermore, the training-free reservoirs and the single convolutional layer provide high-computational efficiency for the ConvMESN to model complex temporal data. Extensive experiments on 18 multivariate time series (MTS) benchmark datasets and 3 skeleton-based action recognition datasets demonstrate that the ConvMESN captures multitimescale dynamics and outperforms existing methods.

Time series clustering is usually an essential unsupervised task in cases when category information is not available and has a wide range of applications. However, existing time series clustering methods usually either ignore temporal dynamics of time series or isolate the feature extraction from clustering tasks without considering the interaction between them. In this article, a time series clustering framework named self-supervised time series clustering network (STCN) is proposed to optimize the feature extraction and clustering simultaneously. In the feature extraction module, a recurrent neural network (RNN) conducts a one-step time series prediction that acts as the reconstruction of the input data, capturing the temporal dynamics and maintaining the local structures of the time series. The parameters of the output layer of the RNN are regarded as model-based dynamic features and then fed into a self-supervised clustering module to obtain the predicted labels. To bridge the gap between these two modules, we employ spectral analysis to constrain the similar features to have the same pseudoclass labels and align the predicted labels with pseudolabels as well. STCN is trained by iteratively updating the model parameters and the pseudoclass labels. Experiments conducted on extensive time series data sets show that STCN has state-of-the-art performance, and the visualization analysis also demonstrates the effectiveness of the proposed model.

Learning long-term and multi-scale dependencies in sequential data is a challenging task for recurrent neural networks (RNNs). In this paper, a novel RNN structure called temporal pyramid RNN (TP-RNN) is proposed to achieve these two goals. TP-RNN is a pyramid-like structure and generally has multiple layers. In each layer of the network, there are several sub-pyramids connected by a shortcut path to the output, which can efficiently aggregate historical information from hidden states and provide many gradient feedback short-paths. This avoids back-propagating through many hidden states as in usual RNNs. In particular, in the multi-layer structure of TP-RNN, the input sequence of the higher layer is a large-scale aggregated state sequence produced by the sub-pyramids in the previous layer, instead of the usual sequence of hidden states. In this way, TP-RNN can explicitly learn multi-scale dependencies with multi-scale input sequences of different layers, and shorten the input sequence and gradient feedback paths of each layer. This avoids the vanishing gradient problem in deep RNNs and allows the network to efficiently learn long-term dependencies. We evaluate TP-RNN on several sequence modeling tasks, including the masked addition problem, pixel-by-pixel image classification, signal recognition and speaker identification. Experimental results demonstrate that TP-RNN consistently outperforms existing RNNs for learning long-term and multi-scale dependencies in sequential data.

Shapelets are discriminative subsequences for time series classification. Recently, learning time-series shapelets (LTS) was proposed to learn shapelets by gradient descent directly. Although learning-based shapelet methods achieve better results than previous methods, they still have two shortcomings. First, the learned shapelets are fixed after training and cannot adapt to time series with deformations at the testing phase. Second, the shapelets learned by back-propagation may not be similar to any real subsequences, which is contrary to the original intention of shapelets and reduces model interpretability. In this paper, we propose a novel shapelet learning model called Adversarial Dynamic Shapelet Networks (ADSNs). An adversarial training strategy is employed to prevent the generated shapelets from diverging from the actual subsequences of a time series. During inference, a shapelet generator produces sample-specific shapelets, and a dynamic shapelet transformation uses the generated shapelets to extract discriminative features. Thus, ADSN can dynamically generate shapelets that are similar to the real subsequences rather than having arbitrary shapes. The proposed model has high modeling flexibility while retaining the interpretability of shapelet-based methods. Experiments conducted on extensive time series data sets show that ADSN is state-of-the-art compared to existing shapelet-based methods. The visualization analysis also shows the effectiveness of dynamic shapelet generation and adversarial training.

The central problem of sentence classification is to extract multi-scale n-gram features for understanding the semantic meaning of sentences.Most existing models tackle this problem by stacking CNN and RNN models, which easily leads to feature redundancy and overfitting because of relatively limited datasets.In this paper, we propose a simple yet effective model called Multi-scale Orthogonal inDependEnt LSTM (MODE-LSTM), which not only has effective parameters and good generalization ability, but also considers multiscale n-gram features.We disentangle the hidden state of the LSTM into several independently updated small hidden states and apply an orthogonal constraint on their recurrent matrices.We then equip this structure with sliding windows of different sizes for extracting multi-scale n-gram features.Extensive experiments demonstrate that our model achieves better or competitive performance against state-of-the-art baselines on eight benchmark datasets.We also combine our model with BERT to further boost the generalization performance.

Time series with missing values (incomplete time series) are ubiquitous in real life on account of noise or malfunctioning sensors. Time-series imputation (replacing missing data) remains a challenge due to the potential for nonlinear dependence on concurrent and previous values of the time series. In this paper, we propose a novel framework for modeling incomplete time series, called a linear memory vector recurrent neural network (LIME-RNN), a recurrent neural network (RNN) with a learned linear combination of previous history states. The technique bears some similarity to residual networks and graph-based temporal dependency imputation. In particular, we introduce a linear memory vector [called the residual sum vector (RSV)] that integrates over previous hidden states of the RNN, and is used to fill in missing values. A new loss function is developed to train our model with time series in the presence of missing values in an end-to-end way. Our framework can handle imputation of both missing-at-random and consecutive missing inputs. Moreover, when conducting time-series prediction with missing values, LIME-RNN allows imputation and prediction simultaneously. We demonstrate the efficacy of the model via extensive experimental evaluation on univariate and multivariate time series, achieving state-of-the-art performance on synthetic and real-world data. The statistical results show that our model is significantly better than most existing time-series univariate or multivariate imputation methods.

Shapelets are discriminative subsequences for time series classification (TSC). Although shapelet-based methods have achieved good performance and interpretability, they still have two issues that can be improved. First, previous methods only assess a shapelet by how accurately it can classify all the samples. However, for multi-class imbalanced classification tasks, these methods will ignore the shapelets that can distinguish minority class from other classes and will tend to use the shapelets that are useful for discriminating the majority classes. Second, the shapelets are fixed after the training phase and cannot adapt to time series with deformations, which will lead to poor matches to the shapelets. In this paper, we propose a novel end-to-end shapelet learning model called Triple Shapelet Networks (TSNs) to extract multi-level feature representations. Specifically, TSN learns the most discriminative shapelets by gradient descent similar to previous methods. In addition, it learns category-specific shapelets for each class by using auxiliary binary classifiers. Finally, it uses a shapelet generator to produce sample-specific shapelets conditioned on subsequences of the input time series. The addition of category-level and sample-level shapelets to the standard model improves the performance. Experiments conducted on extensive time series data sets show that TSN is state-of-the-art compared to existing shapelet-based methods, and the visualization analysis also shows its effectiveness.

Time series clustering is an essential unsupervised technique in cases when category information is not available. It has been widely applied to genome data, anomaly detection, and in general, in any domain where pattern detection is important. Although feature-based time series clustering methods are robust to noise and outliers, and can reduce the dimensionality of the data, they typically rely on domain knowledge to manually construct high-quality features. Sequence to sequence (seq2seq) models can learn representations from sequence data in an unsupervised manner by designing appropriate learning objectives, such as reconstruction and context prediction. When applying seq2seq to time series clustering, obtaining a representation that effectively represents the temporal dynamics of the sequence, multi-scale features, and good clustering properties remains a challenge. How to best improve the ability of the encoder is still an open question. Here we propose a novel unsupervised temporal representation learning model, named Deep Temporal Clustering Representation (DTCR), which integrates the temporal reconstruction and K-means objective into the seq2seq model. This approach leads to improved cluster structures and thus obtains cluster-specific temporal representations. Also, to enhance the ability of encoder, we propose a fake-sample generation strategy and auxiliary classification task. Experiments conducted on extensive time series datasets show that DTCR is state-of-the-art compared to existing methods. The visualization analysis not only shows the effectiveness of cluster-specific representation but also shows the learning process is robust, even if K-means makes mistakes.

Previous studies have shown that attention mechanisms and shortest dependency paths have a positive effect on relation classification. In this paper, a keyword-attentive sentence mechanism is proposed to effectively combine the two methods. Furthermore, to effectively handle the imbalanced classification problem, this paper proposes a new loss function called the synthetic stimulation loss, which uses a modulating factor to allow the model to focus on hard-to-classify samples. The proposed two methods are integrated into a bidirectional gated recurrent unit (BiGRU). As a single model is not strong in noise immunity, this paper applies the mutual learning method to our model and forces the networks to teach each other. Therefore, we call the final model SSL-KAS-MuBiGRU. Experiments on the SemEval-2010 Task 8 data set and the TAC40 data set demonstrate that the keyword-attentive sentence mechanism and synthetic stimulation loss are useful for relation classification, and our model achieves state-of-the-art results.

Text classification is a central field of inquiry in natural language processing (NLP). Although some models learn local semantic features and global long-term dependencies simultaneously, they simply combine them through concatenation either in a cascade way or in parallel while mutual effects between them are ignored. In this paper, we propose the Global-Local Mutual Attention (GLMA) model for text classification problems, which introduces a mutual attention mechanism for mutual learning between local semantic features and global long-term dependencies. The mutual attention mechanism consists of a Local-Guided Global-Attention (LGGA) and a Global-Guided Local-Attention (GGLA). The LGGA allows to assign weights and combine global long-term dependencies of word positions that are semantic related. It captures combined semantics and alleviates the gradient vanishing problem. The GGLA automatically assigns more weights to relevant local semantic features, which captures key local semantic information and filters both noises and irrelevant words/phrases. Furthermore, a weighted-over-time pooling operation is developed to aggregate the most informative and discriminative features for classification. Extensive experiments demonstrate that our model obtains the state-of-the-art performance on seven benchmark datasets and sixteen Amazon product reviews datasets. Both the result analysis and the mutual attention weights visualization further demonstrate the effectiveness of the proposed model.

Time series imputation (replacing missing data) plays an important role in time series analysis due to missing values in real world data. How to recover missing values and model the underlying dynamic dependencies from incomplete time series remains a challenge. A recent work has found that residual networks help build very deep networks by leveraging short paths due to skip connections (Veit et al., 2016). Inspired by this, we observe that these short paths can model underlying correlations between missing items and their previous non-missing observations in a graph-like way. Hence, we propose an end-to-end imputation network with residual short paths, called Residual IMPutation LSTM (RIMP-LSTM), a flexible combination of residual short paths with graph-based temporal dependencies. We construct a residual sum unit (RSU), which enables RIMP-LSTM to make full use of previous revealed information to model incomplete time series and reduce the negative impact of missing values. Moreover, a switch unit is designed to detect the missing values and a new loss function is then developed to train our model with time series in the presence of missing values in an end-to-end way, which also allows simultaneous imputation and prediction. Extensive empirical comparisons with other competitive imputation approaches over several synthetic and real world time series with various rates of missing data verify the superiority of our model.

Recognizing human actions represented by 3D trajectories of skeleton joints is a challenging machine learning task. In this paper, the 3D skeleton sequences are regarded as multivariate time series, and their dynamics and multiscale features are efficiently learned from action echo states. Specifically, first the skeleton data from the limbs and trunk are projected into five high dimensional nonlinear spaces, that are randomly generated by five dynamic, training-free recurrent networks, i.e., the reservoirs of echo state networks (ESNs). In this way, the history of the time series is represented as nonlinear echo states of actions. We then use a single multiscale convolutional layer to extract multiscale features from the echo states, and maintain multiscale temporal invariance by a max-over-time pooling layer. We propose two multi-step fusion strategies to integrate the spatial information over the five parts of the human physical structure. Finally, we learn the label distribution using softmax. With one training-free recurrent layer and only layer of convolution, our Convolutional Echo State Network (ConvESN) is a very efficient end-to-end model, and achieves state-of-the-art performance on four skeleton benchmark data sets.