Nowadays, many sequences of events are generated in areas as diverse as healthcare, finance, and social network. People have been studying these data for a long time. They hope to predict the type and occurrence time of the next event by using relationships among events in the data. recently, with the successful application of Recurrent Neural Network (RNN) in natural language processing, it has been introduced into point process. However, RNN cannot capture the long-term dependence among events well, and self-attention can partially mitigate this problem precisely. Transformer Hawkes Process (THP) using self-attention greatly improves the performance of the Hawkes Process, but THP cannot ignore the effect of irrelevant events, which will affect the computational complexity and prediction accuracy of the model. In this paper, we propose an Adaptively Sparse Transformers Hawkes Process (ASTHP). ASTHP considers the periodicity and nonlinearity of event time in the time encoding process. The sparsity of the ASTHP is achieved by substituting Softmax with [Formula: see text]-entmax: [Formula: see text]-entmax is a differentiable generalization of Softmax that allows unrelated events to gain exact zero weight. By optimizing the neural network parameters, different attention heads can adaptively select sparse modes (from Softmax to Sparsemax). Compared with the existing models, ASTHP model not only ensures the prediction performance but also improves the interpretability of the model. For example, the accuracy of ASTHP model on MIMIC-II dataset is improved by nearly 3 percentage points, and the model fitting degree and stability are also improved significantly.

In real-world application, the temporal asynchronous event sequences are ubiquitous, such as social network, financial engineering, and medical diagonostics, and so on. These data usually show certain intrinsic high-order dependency characteristics. To this end, we propose a hypergraph neural network Hawkes process (HGHP) model, which can extract the high-order correlation from the data through the hypergraph neural network and encode dependent relationships into the hypergraph structure. When processing event sequence data, this method obtains the correlation matrix between different events through hyperedge convolution, and then obtains the latent representation for the event sequence based on the correlation between the data. We conduct experiments on multiple public datasets. Our proposed HGHP model achieves 86.6% accuracy on MIMIC-II dataset, 62.42% on Financial dataset, and 46.79% on Stackoverflow, which is outperforming existing baseline models.

Recently, people pay more attention to catastrophic forgetting problem, that is, the ability of the model to recognize old tasks decreases dramatically when new tasks are added incrementally. Previous studies focused on making the outputs or intermediate features of the new model as similar as possible the old model but ignored the inner-class assignment information. We consider that the inner-class information can effectively reflect the association pattern and intrinsic nature of the samples with each other, so that maintaining the inner-class relationship among task data is helpful to alleviate the negative impact of catastrophic forgetting. Contrastive learning exhibits excellent performance under self-supervising tasks, which can enhance robustness and make representation more compact. We propose an Incremental Learning algorithm with Instance-level and Class-level Contrastive loss and Knowledge Distillation (IL-ICCKD) as common constraints. Specifically, we encourage our model to maintain the knowledge learned in the past from perspectives of instance characteristics and inner-class assignment distribution. At the same time, our model uses a spatial group-wise enhanced attention mechanism to make the learned representations grasp the spatial distribution of subfeatures. We extensively evaluate our framework on three popular benchmark datasets and demonstrate the performance beyond other models.

Continual learning aims to learn a sequence of tasks by leveraging the knowledge acquired in the past in an online-learning manner while being able to perform well on all previous tasks. This ability is crucial to the artificial intelligence (AI) system. Compared to the traditional learning pattern, continual learning is more suitable for most real-world and complex applicative scenarios. However, the current models usually learn a generic representation based on the class label on each task and an effective strategy is selected to avoid catastrophic forgetting. We postulate that selecting the related and useful parts only from the knowledge obtained to perform each task is more effective than utilizing the whole knowledge. Based on this fact, in this paper we propose a new framework, named Selecting Related Knowledge for Online Continual Learning (SRKOCL), which incorporates an additional efficient channel attention mechanism to pick the particularly related knowledge for every task. Our model also combines experience replay and knowledge distillation to circumvent catastrophic forgetting. Finally, extensive experiments are conducted on different benchmarks and the competitive experimental results demonstrate that our proposed SRKOCL is a promising approach against the state-of-the-art.

CapsNet often learns entangled representations that are unfavorable for many learning tasks. In this paper, we introduce Info-CapsNet, a novel and simple unsupervised framework for learning disentangled representations automatically by mutual information. Limiting mutual information between inputs and representations can compress the information capacity of the representations and encourage the capsules to be independent and factorized. Furthermore, variational inference is used to construct a variational bound of the additional information loss and then is replaced by a constraint on the mean of the capsule, it is a very tidy and convenient algorithm to help the intend factors to implement disentangled effect. We also introduce unsupervised dynamic routing algorithm for learning from unlabeled datasets and a novel disentanglement metric that is suitable for capsule. Empirical evaluations suggest that our method achieves the state-of-the-art disentanglement performance compared to baseline on several datasets.

Humans learning involves remembering patterns from the past to better understand recurring concepts as their knowledge grows. However, previous knowledge in deep neural networks could gradually forget when they are trained on a new concept. In this paper, we address this problem by learning a general representation that can be able to remember the previous information and promote the future learning. In this pursuit, a new controller is introduced by the meta-learning strategy that guides the network to keep balance between the previously learned concepts and the new concept, hence avoids catastrophic forgetting. In Online Recurring Concept Meta-Learning (ORCML), we propose a bi-level learning strategy, emphasizing the hidden representation learning of different concept drift in model-level learning, and obtaining a set of shared parameters through the global meta-learning strategy. Through extensive experiments, we demonstrate that our proposed framework has significant improvements over the state-of-art methods.

Abstract Multi‐view subspace clustering always performs well in high‐dimensional data analysis, but is sensitive to the quality of data representation. To this end, a two‐stage fusion strategy is proposed to embed representation learning into the process of multi‐view subspace clustering. This article first proposes a novel matrix factorization method that can separate the coupling consistent and complementary information from observations of multiple views. Based on the obtained latent representations, we further propose two subspace clustering strategies: feature‐level fusion and subspace‐level hierarchical strategy. The feature‐level method concatenates all kinds of latent representations from multiple views, and the original problem therefore degenerates to a single‐view subspace clustering process. The subspace‐level hierarchical method performs different self‐expressive reconstruction processes on the corresponding complementary and consistent latent representations coming from each view, that is, the prior constraints imposed on different types of subspace representations are related to the relevant input factors. Finally, extensive experimental results on real‐world datasets demonstrate the superiority of our proposed methods by comparing them against some state‐of‐the‐art subspace clustering algorithms.

Abstract Asynchronous event sequences widely exist in the real world, such as social networks, electronic medical records, financial data, and genome analysis. For modeling asynchronous event sequences in the continuous time domain, point process has become the underpinning. In the initial research stage, Hawkes process is widely used because it can capture the self‐triggering and mutual triggering modes between different events in a variety of point process functions. In recent years, due to the development of neural networks, deep point process (also known as neural point process) can learn models with the stronger fitting ability and reduce the dependence on prior knowledge by using the powerful capacity of neural networks. The proposal of the transformer Hawkes process (THP) has led to a huge performance improvement, so a new climax of the transformer‐based deep Hawkes process is set off. However, THP does not make full use of the event and temporal information underlying the asynchronous event sequence, meanwhile, if we simply take the event type encoding and temporal encoding as the sequence encoding, a single transformer may suffer from learning bias. In order to circumvent these problems, we propose a tri‐transformer Hawkes process model (TTHP), in which the event and temporal information are introduced to the dot‐product attention operations as auxiliary information to form different multihead attention, respectively, and are utilized to build three heterogeneous learners. A series of well‐designed experiments on synthetic and real‐world datasets validate the effectiveness of the proposed TTHP.

Multi-view learning models the relationships between various observations, and is adept to explore the underlying information of data from multiple perspectives. Since well representation is vital for self-expressive subspace clustering, we propose a method called Multi-View Subspace Clustering with Consistent and view-Specific Latent Factors and Coefficient Matrices (MVSC-CSLFCM) that explores the consensus and complementary information of multiple views, and we also impose suitable constraints on coefficient matrices corresponding to the obtained view-specific and consistent representations, respectively. Finally, an effective optimization algorithm based on augmented lagrangian multiplier is introduced to optimize our proposed MVSC-CSLFCM. Comprehensive experiments on four real-world data sets demonstrate the superiority of our proposed method by comparing with a series of state-of-art subspace algorithms.

Heterogeneous graph containing different types of nodes or links is one of graph types, which is most relevant to actual problems. However, the research for heterogeneous graph has not been studied adequately. In this paper, we propose a new model named Heterogeneous Graph Gated Attention Network (HGGAN) to process heterogeneous graph, including node feature space unification, center-neighbor nodes (C-N) aggregation and metapath-metapath (M-M) aggregation. Especially, we use multihead attention mechanism in C-N aggregation. Owing to the contribution of each attention head is different, so we use a convolutional sub-network to assign a parameter to reflect the contribution of different attention heads. Experimental results on three real-word heterogeneous datasets show that HGGAN achieves state-of-the-art results on node classification task.

The recent increase of asynchronous event sequence data in a diversity of fields, make researchers pay more attention to how to mine knowledge from them. In the initial research phase, researchers tend to make use of basic mathematical-based point process models, such as Poisson process and Hawkes process. And in recent years, recurrent neural network (RNN) based point process models are proposed which have significant model performance improvement, while it is still hard to describe the long-term relation between events. To address this issue, transformer Hawkes process is proposed. However, it is worth noting that transformer with a fixed stack of different layers is failure to implement the parallel processing, recursive learning, and abstracting the local salient properties, while they may be very important. In order to make up for this shortcoming, we present a Universal Transformer Hawkes Process (UTHP), which introduces the recurrent structure in encode process, and introduce convolutional neural network (CNN) in the position-wise-feed-forward neural network. Experiments on several datasets show that the performance of our model is improved compared to the performance of the state-of-the-art.

Weakly supervised object detection (WSOD), aiming to detect objects with only image-level annotations, has become one of the research hotspots over the past few years. Recently, much effort has been devoted to WSOD for the simple yet effective architecture and remarkable improvements have been achieved. Existing approaches using multiple-instance learning usually pay more attention to the proposals individually, ignoring relation information between proposals. Besides, to obtain pseudo-ground-truth boxes for WSOD, MIL-based methods tend to select the region with the highest confidence score and regard those with small overlap as background category, which leads to mislabeled instances. As a result, these methods suffer from mislabeling instances and lacking relations between proposals, degrading the performance of WSOD. To tackle these issues, this article introduces a multi-peak graph-based model for WSOD. Specifically, we use the instance graph to model the relations between proposals, which reinforces multiple-instance learning process. In addition, a multi-peak discovery strategy is designed to avert mislabeling instances. The proposed model is trained by stochastic gradients decent optimizer using back-propagation in an end-to-end manner. Extensive quantitative and qualitative evaluations on two publicly challenging benchmarks, PASCAL VOC 2007 and PASCAL VOC 2012, demonstrate the superiority and effectiveness of the proposed approach.

There is growing interest in low rank representation (LRR) for subspace clustering. Existing latent LRR methods can exploit the global structure of data when the observations are insufficient and/or grossly corrupted, but it cannot capture the intrinsic structure due to the neglect of the local information of data. In this paper, we proposed an improved latent LRR model with a distance regularization and a non-negative regularization jointly, which can effectively discover the global and local structure of data for graph learning and improve the expression of the model. Then, an efficiently iterative algorithm is developed to optimize the improved latent LRR model. In addition, traditional subspace clustering characterizes a fixed numbers of cluster, which cannot efficiently make model selection. An efficiently automatic subspace clustering is developed via the bias and variance trade-off, where the numbers of cluster can be automatically added and discarded on the fly.

Recently, a novel neural network called CapsNet has attracted the attention of many researchers. It is a great attempt to overcome the drawback of convolutional neural networks (CNNs) and achieves state-of-the-art performance on some simple datasets like MNIST. However, this network architecture is built specifically for MNIST and gets a poor performance on more complex datasets like CIFAR-10. To address this problem, aiming at complex data, we propose a new CapsNet architecture called Fractionally-Strided Convolutional Capsule Network (FSC-CapsNet). We modify both network structures of the encoder and decoder of CapsNet. For the purpose of extracting better features, we increase the number of convolutional layers before capsule layer in the encoder and improve the reconstruction performance by adopting two fractionally-strided convolutional layers in the decoder. In addition, no pooling layers are used in our architecture. To assess the performance of our proposed network on complex data, we conduct experiments with a single model without using any ensembled methods and data augmentation techniques on five real-world datasets, which are of higher dimensionality and larger size than MNIST. The experimental results demonstrate that our proposed method achieves better performance and improves the reconstruction performance compared with the normal CapsNet.

In this paper, we apply the method of control theory to machine learning, proposing a new multiplication update algorithm combined with adaptive control theory, we name it MRAC-MU algorithm. A new parameter updating law is obtained according to Lyapunov stability theorem. Using the same object function as the exponential gradient (EG) algorithm, which is the key online learning method to multiplicative updates algorithm, Experiments are used to validate the proposed algorithm has a better result than EG algorithm in prediction accuracy.

Currently, the research point of multi-label learning has shifted to a new direction, that is, Noisy Multi-label learning. How to solve such problem that the training examples with the flipped labels, and make full use of the information of the true label sets to improve the performance of noisy multi-label learning is a challenge. In this paper, we leverage the rank pruning to process the noisy multi-label problem called Rank Pruning Approach for Noisy Multi-Label Classification (RPANMLC). Different from the previous approaches, we adopt the rank pruning to remove the unconfident samples and estimate the noisy label rates. By this way, the original real labels have been preserved, which can improve the performance of the learning algorithm. Another obvious advantage of this approach is that we spend most time to train the "confident" samples rather than the whole corrupted samples. Besides, we also give the upper regret bound of the RPANMLC. Furthermore, other robust noisy multi-label learning methods have been compared with the RPANMLC, and experimental results on four real world data sets verify the effectiveness of our proposed algorithms.

This paper proposes a new online learning algorithm which is based on adaptive control (AC) theory, thus, we call this proposed algorithm as AC algorithm. Comparing to the gradient descent (GD) and exponential gradient (EG) algorithm which have been applied to online prediction problems, we find a new form of AC theory for online prediction problems and investigate two key questions: how to get a new update law which has a tighter upper bound on the error than the square loss? How to compare the upper bound for accumulated losses for the three algorithms? We obtain a new update law which fully utilizes model reference AC theory. Moreover, we present upper bound on the worst-case expected loss for AC algorithm and compare it with previously known bounds for the GD and EG algorithm. The loss bound we get in this paper is a time-varying function, which provides increasingly accurate estimates for upper bound. The AC algorithm has a much smaller loss only if the number of the samples meets certain conditions which can be seen in this paper. We also performed experiments which show that our update law is reasonably feasible and our upper bound is quite tight on both simple artificial and real data sets. The main contributions of this paper are twofold. First of all, we develop a new online algorithm called AC algorithm, and second, we obtain improved bounds, see Theorems 2-4 in this paper.