Skill discovery has emerged as a popular route for unsupervised reinforcement learning (URL), offering agents a diverse, reusable set of behaviours learned before any task-specific reward is experienced. However, existing methodologies tend to favour either categorical codes or unimodal skill priors, which simplifies training at the cost of limiting the variety of behaviours they can represent. We introduce Discovery of Mixture Skills (DiMS), a URL algorithm that learns a latent Gaussian mixture by training a Gaussian Mixture Variational Autoencoder (GMVAE) in tandem with the unsupervised policy. In DiMS, a hierarchical GMVAE simultaneously discovers clusters of skills, while an auxiliary macro-latent dynamically positions mixture components to prevent mode collapse. A joint loss term combining log-likelihood and curiosity rewards enables simultaneous updates of representation and policy while improving exploration. Experiments on the Unsupervised Reinforcement Learning Benchmark (URLB) show that DiMS consistently outperforms a wide range of state-of-the-art baselines. Ablation studies confirm that the mixture prior is critical to these gains, and that DiMS is robust to alternative exploration bonuses. Overall, our results suggest that Gaussian mixture skill priors offer a compelling foundation for future unsupervised RL.

High-dimensional state and action spaces com- bined with sparse reward structures in reinforcement learning (RL) environments typically require advanced control architec- tures. Hierarchical Reinforcement Learning (HRL) demonstrates superior performance compared to atomic RL approaches in these challenging scenarios. HRL can manage the complexity of commands to achieve task objectives through its hierarchical structure. One of the key challenges in HRL is efficiently training each level’s policy with optimal data collection from its experience. Off-policy correction is a critical technique for facilitating sample-efficient off-policy training in HRL, as it addresses the non-stationary issue of higher-level policy training. However, existing methods typically employ indirect probabilistic approaches that fail to accurately capture the current capability of the lower-level policy. This mismatch ultimately constrains the effectiveness of higher-level policy training. In this paper, we propose a novel HRL model that supports direct off-policy correction based on a Flow-based Deep Generative Model (FDGM). This approach leverages the inverse operation of FDGM to achieve goals aligned with the current knowledge of the lower-level policy. Additionally, our model addresses the limitations of FDGM to enable its effective use in HRL. Through comparative experiments on benchmark environments, our model demonstrates superior performance over existing models

Abstract Autonomous drone navigation with deep reinforcement learning (DRL) is hindered by the difficulty of specifying reward functions for vision-based, continuous control in complex environments. We address this by using inverse reinforcement learning (IRL) to recover a task-aligned reward directly from expert demonstrations; specifically, we employ adversarial IRL (AIRL) to learn the reward. In evaluation, the learned-reward policy improves success rate, smoothness and trajectories consistency compared with a carefully tuned human-designed reward and baselines reward function. These results indicate that learning the reward from demonstrations provides a precise and transferable objective for autonomous flight, achieving better performance and better guidance under verification of our evaluation protocol without manual reward engineering. To the best of our knowledge, this is the first work to successfully apply an AIRL framework for visual drone navigation.

Model-based reinforcement learning (MBRL) aims to promote sample efficiency and reduce the number of interactions with the true environment, via learning an environment dynamic model, compared with model-free reinforcement learning (MFRL). However, the success of MBRL heavily relies on two key aspects: model learning and planning. The former refers to learning an accurate model, and the latter aims to improve the behavior policy. In this article, we investigate these two aspects further with model ensemble learning. We design a deep residual attention U-Net (RauNet) with fewer neurons (or weights) than the widely used shallow neural network as our base models and further apply the Hilbert–Schmidt independence criterion (HSIC) as a regularization term to pursue model diversity explicitly for the model ensemble. Furthermore, we propose an adaptive trust region policy optimization (TRPO), in which the parametric Rényi alpha divergence substitutes for the Kullback–Leibler (KL) divergence for measuring the difference between two successive policies, and the alpha value can be adaptively adjusted during TRPO training iterations. This method is called diversity-driven model ensemble adaptive TRPO, or simply diversity-driven model ensemble adaptive trust region policy optimization. Our detailed experiments on six benchmark environments show that our proposed approach is optimal, compared with five state-of-the-art RL techniques.

Classical parameter-space Bayesian inference for Bayesian neural networks (BNNs) suffers from several unresolved prior issues, such as knowledge encoding intractability and pathological behaviours in deep networks, which can lead to improper posterior inference. To address these issues, functional Bayesian inference has recently been proposed leveraging functional priors, such as the emerging functional variational inference. In addition to variational methods, stochastic gradient Markov Chain Monte Carlo (MCMC) is another scalable and effective inference method for BNNs to asymptotically generate samples from the true posterior by simulating continuous dynamics. However, existing MCMC methods perform solely in parameter space and inherit the unresolved prior issues, while extending these dynamics to function space is a non-trivial undertaking. In this paper, we introduce novel functional MCMC schemes, including stochastic gradient versions, based on newly designed diffusion dynamics that can incorporate more informative functional priors. Moreover, we prove that the stationary measure of these functional dynamics is the target posterior over functions. Our functional MCMC schemes demonstrate improved performance in both predictive accuracy and uncertainty quantification on several tasks compared to naive parameter-space MCMC and functional variational inference.

Traditional parameter-space posterior inference for Bayesian neural networks faces several challenges, such as the difficulty in specifying meaningful prior, the potential pathologies in deep models and the intractability for multi-modal posterior. To address these issues, functional variational inference (fVI) and functional Markov Chain Monte Carlo (fMCMC) are two recently emerged Bayesian inference schemes that perform posterior inference directly in function space by incorporating more informative functional priors. Similar to their parameter-space counterparts, fVI and fMCMC have their own strengths and weaknesses. For instance, fVI is computationally efficient but imposes strong distributional assumptions, while fMCMC is asymptotically exact but suffers from slow mixing in high dimensions. To inherit the complementary benefits of both schemes, this work proposes a novel hybrid inference method for functional posterior inference. Specifically, it combines fVI and fMCMC successively by an elaborate linking mechanism to form an alternating approximation process. We also provide theoretical justification for the soundness of such a hybrid inference through the lens of Wasserstein gradient flows in the function space. We evaluate our method on several benchmark tasks and observe improvements in both predictive accuracy and uncertainty quantification compared to parameter/function-space VI and MCMC.

In networked multi-agent reinforcement learning (Networked-MARL), decentralized agents must act autonomously under local observability and constrained communication over fixed physical graphs. Existing methods often assume static neighborhoods, limiting adaptability to dynamic or heterogeneous environments. While centralized frameworks can learn dynamic graphs, their reliance on global state access and centralized infrastructure is impractical in real-world decentralized systems. We propose a stochastic graph-based policy for Networked-MARL, where each agent conditions its decision on a sampled subgraph over its local physical neighborhood. Building on this formulation, we introduce \textbf{BayesG}, a decentralized actor–critic framework that learns sparse, context-aware interaction structures via Bayesian variational inference. Each agent operates over an ego-graph and samples a latent communication mask to guide message passing and policy computation. The variational distribution is trained end-to-end alongside the policy using an evidence lower bound (ELBO) objective, enabling agents to jointly learn both interaction topology and decision-making strategies. BayesG outperforms strong MARL baselines on large-scale traffic control tasks with up to 167 agents, demonstrating superior scalability, efficiency, and performance.

Meta-reinforcement learning (meta-RL) algorithms extract task information from experienced context in order to reason about new tasks, and facilitate rapid adaptation. The quality of these contextual representations (or embeddings) is therefore crucial for a meta-RL agent to make effective decisions in unknown environments. Current methods predominantly assume the existence of a single underlying task, but using a single contextual embedding may not be expressive enough to fully capture the broader distribution of task variations that an agent might encounter. Decomposing that information into different representations can allow them to capture more relevant features in context space while applying additional structure that aids downstream exploitation. In this article, we develop global-local embeddings for contextual meta-RL (GLOBEX), an off-policy contextual meta-RL algorithm that decomposes the contextual representation into separate global and local embeddings. The learning process maximizes information retained by the embeddings and utilizes a mutual information constraint to encourage decoupling. Illustrative examples show that our method effectively adapts by identifying global task dynamics and exploiting temporally local signals. In addition, GLOBEX outperforms existing state-of-the-art meta-RL algorithms on standard MuJoCo benchmarks.

Model-based reinforcement learning (RL) aims to learn the underlying dynamics of a given environment. The success of most existing works is built on the critical assumption that the dynamic is fixed, which is unrealistic in many open-world scenarios, such as drone delivery and online chatting, where agents may need to deal with environments with unpredictable changing dynamics (hereafter,real non-stationary environment). Therefore, learning changing dynamics in a real non-stationary environment offers both significant benefits and challenges. This paper proposes a new model-based reinforcement learning algorithm that proactively and dynamically detects possible changes and Learns these Latent and Changing Dynamics (LLCD) in a latent Markovian space for real non-stationary environments. To ensure the Markovian property of the RL model and improve computational efficiency, we employ a latent space model to learn the environment’s transition dynamics. Furthermore, we perform online change detection in the latent space to promptly identify change points in non-stationary environments. Then, we utilize the detected information to help the agent adapt to new conditions. Experiments indicate that the rewards of the proposed algorithm accumulate for the most rapid adaptions to environmental change, among other benefits. This work has a strong potential to enhance environmentally suitable model-based reinforcement learning capabilities.

Effective agent coordination is crucial in cooperative multiagent reinforcement learning (MARL). While agent cooperation can be represented by graph structures, prevailing graph learning methods in MARL are limited. They rely solely on one-step observations, neglecting crucial historical experiences, leading to deficient graphs that foster redundant or detrimental information exchanges. In addition, high computational demands for action-pair calculations in dense graphs impede scalability. To address these challenges, we propose inferring a latent temporal sparse coordination graph (LTS-CG) for MARL. The LTS-CG leverages agents' historical observations to calculate an agent-pair probability matrix, where a sparse graph is sampled from and used for knowledge exchange between agents, thereby simultaneously capturing agent dependencies and relationship uncertainty. The computational complexity of this procedure is only related to the number of agents. This graph learning process is further augmented by two innovative characteristics: Predict-Future, which enables agents to foresee upcoming observations, and Infer-Present, ensuring a thorough grasp of the environmental context from limited data. These features allow LTS-CG to construct temporal graphs from historical and real-time information, promoting knowledge exchange during policy learning and effective collaboration. Graph learning and agent training occur simultaneously in an end-to-end manner. Our demonstrated results on the StarCraft II benchmark underscore LTS-CG's superior performance.

Trust region policy optimization (TRPO) is an iterative reinforcement learning algorithm that both maximizes a surrogate objective based on a generalized advantage function and enforces a trust region constraint between two consecutive policies based on the Kullback–Leibler divergence in each iteration. On the one hand, its surrogate objective only approximates the positive theoretical improvement and thus relaxes the theoretical guarantee. One the other hand, currently, TRPO is approximately solved via a linear approximation for its objective and a quadratic approximation for its constraint, which results in an upper bound for the linear search of the step size. However, this solution strategy does not provide a lower bound for the step size search. Intuitively, a lower bound could be determined manually, but such a setting has no physical meaning. To this end, we present an alternative solution in this article, which is to exchange the objective and the constraint in TRPO according to reciprocal optimization technique, and then derive a reciprocal TRPO. Applying a similar approximation solution of TRPO to our reciprocal TRPO produces a lower bound for the step size search that has a physical interpretation for improving the surrogate objective or return. Further, we aggregate the original TRPO with this reciprocal TRPO to construct a twin TRPO, whose step size has a lower bound from our reciprocal TRPO and an upper bound from TRPO, to facilitate the policy optimization and achieve a least return improvement. Extensive experiments on twelve benchmark environments show that our twin TRPO is superior to six existing techniques: the original TRPO, proximal policy optimization, off-policy TRPO and two entropy regularized TRPOs, as well as our reciprocal TRPO. Our code is available at https://github.com/HTXu-UTS/twinTRPO.

Unsupervised pre-training has been on the lookout for the virtue of a value function representation referred to as successor features (SFs), which decouples the dynamics of the environment from the rewards. It has a significant impact on the process of task-specific fine-tuning due to the decomposition. However, existing approaches struggle with local optima due to the unified intrinsic reward of exploration and exploitation without considering the linear regression problem and the discriminator supporting a small skill sapce. We propose a novel unsupervised pre-training model with SFs based on a non-monolithic exploration methodology. Our approach pursues the decomposition of exploitation and exploration of an agent built on SFs, which requires separate agents for the respective purpose. The idea will leverage not only the inherent characteristics of SFs such as a quick adaptation to new tasks but also the exploratory and task-agnostic capabilities. Our suggested model is termed Non-Monolithic unsupervised Pretraining with Successor features (NMPS), which improves the performance of the original monolithic exploration method of pre-training with SFs. NMPS outperforms Active Pre-training with Successor Features (APS) in a comparative experiment.

Factuality issues remain challenging to abstractive text summarization despite considerable progress in recent years. This is partly because abstractive text summarization models and methods have a limited capacity to capture complex syntactic structures. Hence, researchers have explored syntactic structures in modeling abstractive text summarization, such as learning structure-aware representations and formulating structure- derived learning objectives. However, their efforts are confined as their optimization only uses cross-entropy-based maximum likelihood estimation, which may underlie some factual problems as we reason later. This paper proposes a syntactic structure- aware encoder-decoder model that incorporates novel learning tasks on syntactic structures. By doing so, we aim at generalizing a latent semantic space that encodes both lexical semantics and complex syntactic structures holistically to tackle the syntactic structure-related factual issues. Our experiments show that our approach improves overall summaries on auto-metric evaluations over the adopted baseline. The human evaluation also indicates that our approach betters summaries on factuality and fluency overall. Further qualitative assessment sheds light on the plausible reasons underlying the quantitative evaluation results.

Bayesian deep learning (BDL) is an emerging field that combines the strong function approximation power of deep learning with the uncertainty modeling capabilities of Bayesian methods. In addition to those virtues, however, there are accompanying issues brought by such a combination to the classical parameter-space variational inference, such as the nonmeaningful priors, intricate posteriors, and possible pathologies. In this paper, we propose a new function-space variational inference solution called Functional Wasserstein Bridge Inference (FWBI), which can assign meaningful functional priors and obtain well-behaved posterior. Specifically, we develop a Wasserstein distance-based bridge to avoid the potential pathological behaviors of Kullback{–}Leibler (KL) divergence between stochastic processes that arise in most existing functional variational inference approaches. The derived functional variational objective is well-defined and proved to be a lower bound of the model evidence. We demonstrate the improved predictive performance and better uncertainty quantification of our FWBI on several tasks compared with various parameter-space and function-space variational methods.

Variational policy optimization has become increasingly attractive to the reinforcement learning community because of its strong capability in uncertainty modeling and environment generalization. However, almost all existing studies in this area rely on Kullback{–}Leibler (KL) divergence which is unfortunately ill-defined in several situations. In addition, the policy is parameterized and optimized in weight space, which may not only bring additional unnecessary bias but also make the policy learning harder due to the complicatedly dependent weight posterior. In the paper, we design a novel functional Wasserstein variational policy optimization (FWVPO) based on the Wasserstein distance between function distributions. Specifically, we firstly parameterize policy as a Bayesian neural network but from a function-space view rather than a weight-space view and then propose FWVPO to optimize and explore the functional policy posterior. We prove that our FWVPO is a valid variational Bayesian objective and also guarantees the monotonic expected reward improvement under certain conditions. Experimental results on multiple reinforcement learning tasks demonstrate the efficiency of our new algorithm in terms of both cumulative rewards and uncertainty modeling capability.

Trust region policy optimization (TRPO) is one of the landmark policy optimization algorithms in deep reinforcement learning. Its purpose is to maximize a surrogate objective based on an advantage function, subject to the limited Kullback–Leibler (KL) divergence of two consecutive policies. Although there have been many successful applications of this algorithm in the literature, the approach has often been criticized for suppressing the exploration ability of some application environments due to its strict divergence constraint. As such, most researchers prefer to use entropy regularization, which is added to the expected discounted rewards or the surrogate objectives. That said, there is much debate about whether there might be an alternative strategy for regularizing TRPOs. In this paper, we present just that. Our strategy is to regularize the KL divergence-based constraint via Shannon entropy. This approach enlarges the difference between two consecutive policies and thus derives a new TRPO scheme with entropy regularization for use with KL divergence constraint. Next, the surrogate objective and Shannon entropy are approximated linearly, while the KL divergence is expanded quadratically. An efficient conjugate gradient optimization procedure then solves two sets of linear equations, providing a detailed code-level implementation that can be used for a fair experimental comparison. Extensive experiments within eight benchmark environments demonstrate that our proposed method is superior to both the original TRPO and the entropy regularized objective TRPO. Further, theoretical and experimental analysis shows that three TRPO-like methods have an equal time complexity and a close computational burden.

Deep reinforcement learning (DRL) is a powerful tool for learning from interactions within a stationary environment where state transition and reward distributions remain constant throughout the process. Addressing the practical but challenging nonstationary environments with time-varying state transition or reward function changes during the interactions, ingenious solutions are essential for the stability and robustness of DRL agents. A key assumption to cope with nonstationary environments is that the change points between the previous and the new environments are known beforehand. Unfortunately, this assumption is impractical in many cases, such as outdoor robots and online recommendations. To address this problem, this article presents a robust DRL algorithm for nonstationary environments with unknown change points. The algorithm actively detects change points by monitoring the joint distribution of states and actions. A detection boosted, gradient-constrained optimization method then adapts the training of the current policy with the supporting knowledge of formerly well-trained policies. The previous policies and experience help the current policy adapt rapidly to environmental changes. Experiments show that the proposed method accumulates the highest reward among several alternatives and is the fastest to adapt to new environments. This work has compelling potential for increasing the environmental suitability of intelligent agents, such as drones, autonomous vehicles, and underwater robots.

Graph convolutional neural networks (GCNs) have achieved great success in graph representation learning by extracting high-level features from nodes and their topology. Since GCNs generally follow a message-passing mechanism, each node aggregates information from its first-order neighbor to update its representation. As a result, the representations of nodes with edges between them should be positively correlated and thus can be considered positive samples. However, there are more non-neighbor nodes in the whole graph, which provide diverse and useful information for the representation update. Two non-adjacent nodes usually have different representations, which can be seen as negative samples. Besides the node representations, the structural information of the graph is also crucial for learning. In this article, we used quality-diversity decomposition in determinant point processes (DPPs) to obtain diverse negative samples. When defining a distribution on diverse subsets of all non-neighboring nodes, we incorporate both graph structure information and node representations. Since the DPP sampling process requires matrix eigenvalue decomposition, we propose a new shortest-path-base method to improve computational efficiency. Finally, we incorporate the obtained negative samples into the graph convolution operation. The ideas are evaluated empirically in experiments on node classification tasks. These experiments show that the newly proposed methods not only improve the overall performance of standard representation learning but also significantly alleviate over-smoothing problems.

Most exploration research on reinforcement learning (RL) has paid attention to ‘the way of exploration’, which is ‘how to explore’. The other exploration research, ‘when to explore’, has not been the main focus of RL exploration research. The issue of ‘when’ of a monolithic exploration in the usual RL exploration behaviour binds an exploratory action to an exploitational action of an agent. Recently, a non-monolithic exploration research has emerged to examine the mode-switching exploration behaviour of humans and animals. The ultimate purpose of our research is to enable an agent to decide when to explore or exploit autonomously. We describe the initial research of an autonomous multi-mode exploration of non-monolithic behaviour in an options framework. The higher performance of our method is shown against the existing non-monolithic exploration method through comparative experimental results.

In recent years abstractive text summarization (ATS) research has made considerable progress attributed to two key improvements, deep neural modeling and likelihood estimation based sampling, in the end-to-end optimization training. While modeling has grounded on a few de facto highly capable base models within encoder-decoder architecture, novel sampling ideas, such as random masking classification and generative prediction by unsupervised learning, have also been explored. They aim at improving prior knowledge, particularly of language modeling for downstream tasks. It has led to the notable performance gain of ATS. But several challenges remain, for example, undesirable word repeats. In this paper, we propose a determinantal point process (DPP) based novel sampling method to address the issue. It can be easily integrated with the existing ATS models. Our experiments and subsequent analysis have revealed that the adopted models trained by our sampling method reduce undesirable word repeats and improve word coverage while achieving competitive ROUGE scores.

Data streams may encounter data distribution changes, which can significantly impair the accuracy of models. Concept drift detection tracks data distribution changes and signals when to update models. Many drift detection methods apply thresholds to distinguish between drift or non-drift streams and to claim their method outperforms others with non-aligned drift thresholds. We consider that selecting a proper drift threshold could be more important than developing a new drift detection algorithm, and different drift detection algorithms may end up with very similar performance with aligned drift thresholds. To better understand this process, we propose a novel threshold selection algorithm to align the drift thresholds of a set of algorithms so that they are all at the same sensitivity level. Based on comprehensive experiment evaluations, we observed that several state-of-the-art drift detection algorithms could achieve similar results by aligning their thresholds, providing a novel insight to explain how drift detection algorithms contribute to data stream learning. We noticed that a higher detection sensitivity improves accuracy for data streams with frequent distribution change. The evaluation results are showing that drift thresholds should not be fixed during stream learning. Rather, they should adjust dynamically based on the prevailing conditions of the data stream.

Causal inference from observational data lies at the heart of education, healthcare, optimal resource allocation and many other decision-making processes. Most of existing methods estimate the target treatment effect indirectly by inferring the underlying treatment response functions or the unobserved counterfactual outcome for every individual. These learning methods are subject to issues of model misspecification and high variability. As a complement of existing indirect learning methods, we propose a direct learning framework, called HTENet, for causal inference using deep multi-task learning. It is based on a novel empirical -risk for learning the causal effect model of direct interest in a supervised learning scheme. In our proposed framework, the target treatment effect model is parametrized as a neural network and learned jointly with other auxiliary models in an end-to-end manner. Moreover, we extend the nave HTENet into other two variants, HTENet-Simple and HTENet-Reg, by further incorporating shared representation learning layers and a propensity prediction regularizer. Experiments on simulated and real data demonstrate that the performances of the proposed methods match or are better than that of existing state-of-arts. Moreover, by learning the target treatment effect function directly, the proposed methods tend to obtain more stable estimates than existing methods.

Graph Convolutional Neural Networks (GCNs) have been generally accepted to be an effective tool for node representations learning. An interesting way to understand GCNs is to think of them as a message passing mechanism where each node updates its representation by accepting information from its neighbours (also known as positive samples). However, beyond these neighbouring nodes, graphs have a large, dark, all-but forgotten world in which we find the non-neighbouring nodes (negative samples). In this paper, we show that this great dark world holds a substantial amount of information that might be useful for representation learning. Most specifically, it can provide negative information about the node representations. Our overall idea is to select appropriate negative samples for each node and incorporate the negative information contained in these samples into the representation updates. Moreover, we show that the process of selecting the negative samples is not trivial. Our theme therefore begins by describing the criteria for a good negative sample, followed by a determinantal point process algorithm for efficiently obtaining such samples. A GCN, boosted by diverse negative samples, then jointly considers the positive and negative information when passing messages. Experimental evaluations show that this idea not only improves the overall performance of standard representation learning but also significantly alleviates over-smoothing problems.

Top-k words selection is a technique used to detect and return the k most similar words to a given word from a candidate set. This is a crucial and widely used tool in various tasks. The key issue in top-k words selection is how to measure the similarity between words. One popular and effective solution is to use a word embedding-based similarity measure, which represents words as low-dimensional vectors and measures the similarities between words according to the similarity of the vectors, using a metric. However, most word embedding methods only consider the local proximity properties of two words in a corpus. To mitigate this issue. In this article, we propose to use association rules for measuring word similarity at a global level, and a fuzzy similarity measure for top-k words selection that jointly encodes the local and the global similarities. Experiments on a real-world query task with three benchmark datasets, i.e., TREC-disk 4&5, WT10G, and RCV1, demonstrate the efficiency of the proposed method compared to several state-of-the-art baselines.

Online data stream mining is of great significance in practice because of its ubiquity in many real-world scenarios, especially in the big data era. Traditional data mining algorithms cannot be directly applied to data streams due to (1) the possible change of underlying data distribution over time (i.e., concept drift ) and (2) delayed, short, or even no labels for streaming data in practice. A new research area, named unsupervised concept drift detection , has emerged to tackle this difficulty mainly based on two-sample hypothesis tests, such as the Kolmogorov–Smirnov test. However, it is surprising that none of the existing methods in this area exploit the Bayesian nonparametric hypothesis test, which has clear interpretability and straightforward prior knowledge encoding ability and no strict or unrealistic requirement of prefixing the form for the underlying data distribution. In this article, we present a Bayesian nonparametric unsupervised concept drift detection method based on the Polya tree hypothesis test. The basic idea is to decompose the underlying data distribution into a multi-resolution representation that transforms the whole distribution hypothesis test into recursive and simple binomial tests. Also, an incremental mechanism is especially designed to improve its efficiency in the stream setting. The method effectively detect drifts, and it also locates where a drift happens and the posteriors of hypotheses. The experiments on synthetic data verify the desired properties of the proposed method, and the experiments on real-world data show the better performance of the method for data stream mining compared with its frequentist counterpart in the literature.

The aim of unsupervised domain adaptation is to leverage the knowledge in a labeled (source) domain to improve a model's learning performance with an unlabeled (target) domain-the basic strategy being to mitigate the effects of discrepancies between the two distributions. Most existing algorithms can only handle unsupervised closed set domain adaptation (UCSDA), i.e., where the source and target domains are assumed to share the same label set. In this article, we target a more challenging but realistic setting: unsupervised open set domain adaptation (UOSDA), where the target domain has unknown classes that are not found in the source domain. This is the first study to provide learning bound for open set domain adaptation, which we do by theoretically investigating the risk of the target classifier on unknown classes. The proposed learning bound has a special term, namely, open set difference, which reflects the risk of the target classifier on unknown classes. Furthermore, we present a novel and theoretically guided unsupervised algorithm for open set domain adaptation, called distribution alignment with open difference (DAOD), which is based on regularizing this open set difference bound. The experiments on several benchmark data sets show the superior performance of the proposed UOSDA method compared with the state-of-the-art methods in the literature.

Graph neural networks (GNNs) extends the functionality of traditional neural networks to graph-structured data. Similar to CNNs, an optimized design of graph convolution and pooling is key to success. Borrowing ideas from physics, we propose a path integral based graph neural networks (PAN) for classification and regression tasks on graphs. Specifically, we consider a convolution operation that involves every path linking the message sender and receiver with learnable weights depending on the path length, which corresponds to the maximal entropy random walk. It generalizes the graph Laplacian to a new transition matrix we call \emph{maximal entropy transition} (MET) matrix derived from a path integral formalism. Importantly, the diagonal entries of the MET matrix are directly related to the subgraph centrality, thus lead to a natural and adaptive pooling mechanism. PAN provides a versatile framework that can be tailored for different graph data with varying sizes and structures. We can view most existing GNN architectures as special cases of PAN. Experimental results show that PAN achieves state-of-the-art performance on various graph classification/regression tasks, including a new benchmark dataset from statistical mechanics we propose to boost applications of GNN in physical sciences.

Estimating causal effects by making causal inferences from observational data is common practice in scientific studies, business decision-making, and daily life. In today’s data-driven world, causal inference has become a key part of the evaluation process for many purposes, such as examining the effects of medicine or the impact of an economic policy on society. However, although the literature contains some excellent models, there is room to improve their representation power and their ability to capture complex relationships. For these reasons, we propose a novel prior called Causal DP and a model called CDP. The prior captures the complex relationships between covariates, treatments, and outcomes in observational data using a rational probabilistic dependency structure. The model is Bayesian, nonparametric, and generative and is not based on the assumption of any parametric distribution. CDP is designed to estimate various kinds of causal effects—average, conditional average, average treated, quantile, and so on. It performs well with missing covariates and does not suffer from overfitting. Comparative experiments on synthetic datasets against several state-of-the-art methods demonstrate that CDP has a superior ability to capture complex relationships. Further, a simple evaluation to infer the effect of a job training program on trainee earnings from real-world data shows that CDP is both effective and useful for causal inference.

Causal inference from observational data aims to estimate causal effects when controlled experimentation is not feasible, but it faces challenges when unobserved confounders exist. The instrumental variable method resolves this problem by introducing a variable that is correlated with the treatment and affects the outcome only through the treatment. However, existing instrumental variable methods require two stages to separately estimate the conditional treatment distribution and the outcome generating function, which is not sufficiently effective. This paper presents a one-stage approach to jointly estimate the treatment distribution and the outcome generating function through a cleverly designed deep neural network structure. This study is the first to merge the two stages to leverage the outcome to the treatment distribution estimation. Further, the new deep neural network architecture is designed with two strategies (i.e., shared and separate) of learning a confounder representation account for different observational data. Such network architecture can unveil complex relationships between confounders, treatments, and outcomes. Experimental results show that our proposed method outperforms the state-of-the-art methods. It has a wide range of applications, from medical treatment design to policy making, population regulation and beyond.

Representing the semantics of words is a fundamental task in text processing. Several research studies have shown that text and knowledge bases (KBs) are complementary sources for word embedding learning. Most existing methods only consider relationships within word-pairs in the usage of KBs. We argue that the structural information of well-organized words within the KBs is able to convey more effective and stable knowledge in capturing semantics of words. In this paper, we propose a semantic structure-based word embedding method, and introduce concept convergence and word divergence to reveal semantic structures in the word embedding learning process. To assess the effectiveness of our method, we use WordNet for training and conduct extensive experiments on word similarity, word analogy, text classification and query expansion. The experimental results show that our method outperforms state-of-the-art methods, including the methods trained solely on the corpus, and others trained on the corpus and the KBs.

Sparse nonnegative matrix factorization (SNMF) aims to factorize a data matrix into two optimized nonnegative sparse factor matrices, which could benefit many tasks, such as document-word co-clustering. However, the traditional SNMF typically assumes the number of latent factors (i.e., dimensionality of the factor matrices) to be fixed. This assumption makes it inflexible in practice. In this paper, we propose a doubly sparse nonparametric NMF framework to mitigate this issue by using dependent Indian buffet processes (dIBP). We apply a correlation function for the generation of two stick weights associated with each column pair of factor matrices while still maintaining their respective marginal distribution specified by IBP. As a consequence, the generation of two factor matrices will be columnwise correlated. Under this framework, two classes of correlation function are proposed: 1) using bivariate Beta distribution and 2) using Copula function. Compared with the single IBP-based NMF, this paper jointly makes two factor matrices nonparametric and sparse, which could be applied to broader scenarios, such as co-clustering. This paper is seen to be much more flexible than Gaussian process-based and hierarchial Beta process-based dIBPs in terms of allowing the two corresponding binary matrix columns to have greater variations in their nonzero entries. Our experiments on synthetic data show the merits of this paper compared with the state-of-the-art models in respect of factorization efficiency, sparsity, and flexibility. Experiments on real-world data sets demonstrate the efficiency of this paper in document-word co-clustering tasks.

Traditional relational topic models provide a successful way to discover the hidden topics from a document network. Many theoretical and practical tasks, such as dimensional reduction, document clustering, and link prediction, could benefit from this revealed knowledge. However, existing relational topic models are based on an assumption that the number of hidden topics is known a priori, which is impractical in many real-world applications. Therefore, in order to relax this assumption, we propose a nonparametric relational topic model using stochastic processes instead of fixed-dimensional probability distributions in this paper. Specifically, each document is assigned a Gamma process, which represents the topic interest of this document. Although this method provides an elegant solution, it brings additional challenges when mathematically modeling the inherent network structure of typical document network, i.e., two spatially closer documents tend to have more similar topics. Furthermore, we require that the topics are shared by all the documents. In order to resolve these challenges, we use a subsampling strategy to assign each document a different Gamma process from the global Gamma process, and the subsampling probabilities of documents are assigned with a Markov Random Field constraint that inherits the document network structure. Through the designed posterior inference algorithm, we can discover the hidden topics and its number simultaneously. Experimental results on both synthetic and real-world network datasets demonstrate the capabilities of learning the hidden topics and, more importantly, the number of topics.

Summary An emergency event is an unexceptional event that exceeds the capacity of normal resources and organization to cope and a situation that poses an immediate risk to health, life, property, or environment. Crowdsourcing connects unobtrusive and ubiquitous sensing technologies, advanced data management and analytics models, and novel visualization methods, to create solutions that improve urban environment, human life quality, and city operation systems. The crowdsourcing on social media can be used to detect and analyze urban emergency events. In this paper, in order to detect and describe the real‐time urban emergency event, the knowledge base model is proposed. The crowdsourcing‐based knowledge base model is firstly introduced, which uses the information from social media. Secondly, the basic definition of the proposed knowledge base model including keywords, patterns, positive sentences, and knowledge graph is given. Thirdly, the temporal information is added to the proposed knowledge base model. The case study on real data sets shows that the proposed algorithm has good performance and high effectiveness in the analysis and detection of emergency events. Copyright © 2016 John Wiley & Sons, Ltd.

The evolution potential estimation of news events can support the decision making of both corporations and governments. For example, a corporation could manage its public relations crisis in a timely manner if a negative news event about this corporation is known with large evolution potential in advance. However, existing state-of-the-art methods are mainly based on time series historical data, which are not suitable for the news events with limited historical data and bursty properties. In this article, we propose a purely content-based method to estimate the evolution potential of the news events. The proposed method considers a news event at a given time point as a system composed of different keywords, and the uncertainty of this system is defined and measured as the Semantic Uncertainty of this news event. At the same time, an uncertainty space is constructed with two extreme states: the most uncertain state and the most certain state. We believe that the Semantic Uncertainty has correlation with the content evolution of the news events, so it can be used to estimate the evolution potential of the news events. In order to verify the proposed method, we present detailed experimental setups and results measuring the correlation of the Semantic Uncertainty with the Content Change of news events using collected news events data. The results show that the correlation does exist and is stronger than the correlation of value from the time-series-based method with the Content Change. Therefore, we can use the Semantic Uncertainty to estimate the evolution potential of news events.

Webpage recommendations for hot Web events can assist people to easily follow the evolution of these Web events. At the same time, there are different levels of semantic uncertainty underlying the amount of Webpages for a Web event, such as recapitulative information and detailed information. Apparently, the grasp of the semantic uncertainty of Web events could improve the satisfactoriness of Webpage recommendations. However, traditional hit-rate-based or clustering-based Webpage recommendation methods have overlooked these different levels of semantic uncertainty. In this paper, we propose a framework to identify the different underlying levels of semantic uncertainty in terms of Web events, and then utilize these for Webpage recommendations. Our idea is to consider a Web event as a system composed of different keywords, and the uncertainty of this keyword system is related to the uncertainty of the particular Web event. Based on keyword association linked network Web event representation and Shannon entropy, we identify the different levels of semantic uncertainty, and construct a semantic pyramid (SP) to express the uncertainty hierarchy of a Web event. Finally, an SP-based Webpage recommendation system is developed. Experiments show that the proposed algorithm can significantly capture the different levels of the semantic uncertainties of Web events and it can be applied to Webpage recommendations.

Incorporating the side information of text corpus, i.e., authors, time stamps, and emotional tags, into the traditional text mining models has gained significant interests in the area of information retrieval, statistical natural language processing, and machine learning. One branch of these works is the so-called Author Topic Model (ATM), which incorporates the authors's interests as side information into the classical topic model. However, the existing ATM needs to predefine the number of topics, which is difficult and inappropriate in many real-world settings. In this paper, we propose an Infinite Author Topic (IAT) model to resolve this issue. Instead of assigning a discrete probability on fixed number of topics, we use a stochastic process to determine the number of topics from the data itself. To be specific, we extend a gamma-negative binomial process to three levels in orderto capture the author-document-keyword hierarchical structure. Furthermore, each document is assigned a mixed gamma process that accounts for the multi-author's contribution towards this document. An efficient Gibbs sampling inference algorithm witheach conditional distribution being closed-form is developed for the IAT model. Experiments on several real-world datasets show the capabilities of our IAT model to learn the hidden topics, authors' interests on these topics and the number of topics simultaneously.

Summary Online hot event discovery has become a flourishing frontier where online document streams are monitored to discover newly occurring events or assigned to previously detected events. However, hot events have the nature to evolve, and their inherent topically related words are also likely to evolve. It makes event discovery a challenging task for traditional‐mining approaches. Combining word association and semantic community, Association Link Network (ALN) organizes the loosely distributed associated resources. This paper presents an ALN‐based novel online hot event discovery approach. Technically, this approach is enacted around three stages. In the first stage, we extract significant features to represent the content of each document from the online document stream. During the second stage, we classify the online document stream into topically related detected events considering event evolution in the form of ALN. At the third stage, we create an ALN‐based event detection algorithm, which is used to timely discover newly occurring hot events. The online datasets used in our empirical studies are acquired from Baidu News, which spans a range of 1315 hot events and 236,300 documents. Experimental results demonstrate the hot events discovery ability with respect to high accuracy, good scalability, and short runtime. Copyright © 2014 John Wiley & Sons, Ltd.

Graph mining has been a popular research area because of its numerous application scenarios. Many unstructured and structured data can be represented as graphs, such as, documents, chemical molecular structures, and images. However, an issue in relation to current research on graphs is that they cannot adequately discover the topics hidden in graph-structured data which can be beneficial for both the unsupervised learning and supervised learning of the graphs. Although topic models have proved to be very successful in discovering latent topics, the standard topic models cannot be directly applied to graph-structured data due to the "bag-of-word" assumption. In this paper, an innovative graph topic model (GTM) is proposed to address this issue, which uses Bernoulli distributions to model the edges between nodes in a graph. It can, therefore, make the edges in a graph contribute to latent topic discovery and further improve the accuracy of the supervised and unsupervised learning of graphs. The experimental results on two different types of graph datasets show that the proposed GTM outperforms the latent Dirichlet allocation on classification by using the unveiled topics of these two models to represent graphs.

Similarity measures are the foundations of many research areas, e.g. information retrieval, recommender system and machine learning algorithms. Promoted by these application scenarios, a number of similarity measures have been proposed and proposing. In these state-of-the-art measures, vector-based representation is widely accepted based on Vector Space Model (VSM) in which an object is represented as a vector composed of its features. Then, the similarity between two objects is evaluated by the operations on two corresponding vectors, like cosine, extended jaccard, extended dice and so on. However, there is an assumption that the features are independent of each others. This assumption is apparently unrealistic, and normally, there are relations between features, i.e. the co-occurrence relations between keywords in text mining area. In this paper, a space geometry-based method is proposed to extend the VSM from the orthogonal coordinate system (OVSM) to affine coordinate system (AVSM) and OVSM is proved to be a special case of AVSM. Unit coordinate vectors of AVSM are inferred by the relations between features which are considered as angles between these unit coordinate vectors. At last, five different similarity measures are extended from OVSM to AVSM using unit coordinate vectors of AVSM. Within the numerous application fields of similarity measures, the task of text clustering is selected to be the evaluation criterion. Documents are represented as vectors in OVSM and AVSM, respectively. The clustering results show that AVSM outweighs the OVSM.

With the increase of information scale of web events on the time, it is extremely difficult and challenging to grasp the semantics of web events artificially, because of the limitation of the time and energy of human beings. Herein, we propose a method to map the web event to keyword level association link network (KALN) for deep analysis of the semantics of web events, such as the evolution semantics of web events. Firstly, the original KALN is constructed at a given time by traditional data mining technologies. Then, the hierarchical KALN, consisted of Theme Layer Network, Backbone Layer Network and Tidbit Layer Network, is built based on the original KALN by information entropy to identify the different semantic levels of the web event, including stable semantics, sub-stable semantics and unstable semantics. With the semantic analysis of hierarchical KALN, human could easily gain a thorough understanding of the web event. Finally, experiments show that our method can effectively capture the different level semantics of web events.