Reward shaping is a reinforcement learning technique that addresses the sparse-reward problem by providing frequent, informative feedback. We propose an efficient self-adaptive reward-shaping mechanism that uses success rates derived from historical experiences as shaped rewards. The success rates are sampled from Beta distributions, which evolve from uncertainty to reliability as data accumulates. Initially, shaped rewards are stochastic to encourage exploration, gradually becoming more certain to promote exploitation and maintain a natural balance between exploration and exploitation. We apply Kernel Density Estimation (KDE) with Random Fourier Features (RFF) to derive Beta distributions, providing a computationally efficient solution for continuous and high-dimensional state spaces. Our method, validated on tasks with extremely sparse rewards, improves sample efficiency and convergence stability over relevant baselines.

Existing reward shaping techniques for sparse-reward reinforcement learning generally fall into two categories: novelty-based exploration bonuses and significance-based hidden state values. The former promotes exploration but can lead to distraction from task objectives, while the latter facilitates stable convergence but often lacks sufficient early exploration. To address these limitations, we propose Dual Random Networks Distillation (DuRND), a novel reward shaping framework that efficiently balances exploration and exploitation in a unified mechanism. DuRND leverages two lightweight random network modules to simultaneously compute two complementary rewards: a novelty reward to encourage directed exploration and a contribution reward to assess progress toward task completion. With low computational overhead, DuRND excels in high-dimensional environments with challenging sparse rewards, such as Atari, VizDoom, and MiniWorld, outperforming several benchmarks.

Reward shaping is effective in addressing the sparse-reward challenge in reinforcement learning (RL) by providing immediate feedback through auxiliary, informative rewards. Based on the reward shaping strategy, we propose a novel multi-task reinforcement learning framework that integrates a centralized reward agent (CRA) and multiple distributed policy agents. The CRA functions as a knowledge pool, aimed at distilling knowledge from various tasks and distributing it to individual policy agents to improve learning efficiency. Specifically, the shaped rewards serve as a straightforward metric for encoding knowledge. This framework not only enhances knowledge sharing across established tasks but also adapts to new tasks by transferring meaningful reward signals. We validate the proposed method on both discrete and continuous domains, including the representative Meta-World benchmark, demonstrating its robustness in multi-task sparse-reward settings and its effective transferability to unseen tasks.

Reward shaping is a promising approach to tackle the sparse-reward challenge of reinforcement learning by reconstructing more informative and dense rewards. This paper introduces a novel dual-agent reward shaping framework, composed of two synergistic agents: a policy agent to learn the optimal behavior and a reward agent to generate auxiliary reward signals. The proposed method operates as a self-learning approach, without reliance on expert knowledge or hand-crafted functions. By restructuring the rewards to capture future-oriented information, our framework effectively enhances the sample efficiency and convergence stability. Furthermore, the auxiliary reward signals facilitate the exploration of the environment in the early stage and the exploitation of the policy agent in the late stage, achieving a self-adaptive balance. We evaluate our framework on continuous control tasks with sparse and delayed rewards, demonstrating its robustness and superiority over existing methods.

We propose a new approach to estimate causal effects from observational data. We leverage multiple data sources which share similar causal mechanisms with the scarce target observations to help infer causal effects in the target domain. The data sources may be available in sequence or some unplanned order. Causal inference can be carried out without prior knowledge of the data discrepancy between the source and target observations. We introduce three levels of knowledge transfer through modelling the outcomes, treatments, and confounders to achieve consistent positive transfer. We incorporate parametric transfer factors to adaptively control the transfer strength, thus achieving a fair and balanced knowledge transfer between the sources and the target. We also empirically show the effectiveness of the proposed method as compared with recent baselines.

We propose a new causal inference framework to learn causal effects from multiple, decentralized data sources in a federated setting. We introduce an adaptive transfer algorithm that learns the similarities among the data sources by utilizing Random Fourier Features to disentangle the loss function into multiple components, each of which is associated with a data source. The data sources may have different distributions; the causal effects are independently and systematically incorporated. The proposed method estimates the similarities among the sources through transfer coefficients, and hence requiring no prior information about the similarity measures. The heterogeneous causal effects can be estimated with no sharing of the raw training data among the sources, thus minimizing the risk of privacy leak. We also provide minimax lower bounds to assess the quality of the parameters learned from the disparate sources. The proposed method is empirically shown to outperform the baselines on decentralized data sources with dissimilar distributions.

We propose a Bayesian framework for estimating causal effects from federated observational data sources. Bayesian causal inference is an important approach to learning the distribution of the causal estimands and understanding the uncertainty of causal effects. Our framework estimates the posterior distributions of the causal effects to compute the higher-order statistics that capture the uncertainty. We integrate local causal effects from different data sources without centralizing them. We then estimate the treatment effects from observational data using a non-parametric reformulation of the classical potential outcomes framework. We model the potential outcomes as a random function distributed by Gaussian processes, with defining parameters that can be efficiently learned from multiple data sources. Our method avoids exchanging raw data among the sources, thus contributing towards privacy-preserving causal learning. The promise of our approach is demonstrated through a set of simulated and real-world examples.

Reducing domain divergence is a key step in transfer learning. Existing works focus on the minimization of global domain divergence. However, two domains may consist of several shared subdomains, and differ from each other in each subdomain. In this article, we take the local divergence of subdomains into account in transfer. Specifically, we propose to use the low-dimensional manifold to represent the subdomain, and align the local data distribution discrepancy in each manifold across domains. A manifold maximum mean discrepancy (M3D) is developed to measure the local distribution discrepancy in each manifold. We then propose a general framework, called transfer with manifolds discrepancy alignment (TMDA), to couple the discovery of data manifolds with the minimization of M3D. We instantiate TMDA in the subspace learning case considering both the linear and nonlinear mappings. We also instantiate TMDA in the deep learning framework. Experimental studies show that TMDA is a promising method for various transfer learning tasks.

This work extends causal inference in temporal models with stochastic confounders. We propose a new approach to variational estimation of causal inference based on a representer theorem with a random input space. We estimate causal effects involving latent confounders that may be interdependent and time-varying from sequential, repeated measurements in an observational study. Our approach extends current work that assumes independent, non-temporal latent confounders with potentially biased estimators. We introduce a simple yet elegant algorithm without parametric specification on model components. Our method avoids the need for expensive and careful parameterization in deploying complex models, such as deep neural networks in existing approaches, for causal inference and analysis. We demonstrate the effectiveness of our approach on various benchmark temporal datasets.

Capturing the relatedness of different domains is a key challenge in transferring knowledge across domains. In this paper, we propose an effective and efficient Gaussian process (GP) modelling framework, mTGPmk, that can explicitly model domain relatedness and adaptively control the space as well as the strength of knowledge transfer. mTGPmk takes both the discrepancy of input feature space and the discrepancy of predictive function into account in the transfer procedure. Specifically, mTGPmk adaptively selects a good latent manifold shared by different domains, and utilizes a parametric similarity coefficient to measure the predictive function covariance of different domains in this manifold. The latent shared manifold and the similarity coefficient are jointly learned in a coupled manner. By doing so, mTGPmk maximizes the strength of the shared knowledge transfer by choosing the transfer space with the best transfer capacity. More importantly, mTGPmk exploits a succinct and computationally efficient manifold learning approach so that it can be well trained with scarce target training data. Extensive experimental studies using 36 synthetic transfer tasks and 10 real-world transfer tasks show the effectiveness of mTGPmk on capturing the relatedness and the transfer adaptiveness.

Feature-based transfer is one of the most effective methodologies for transfer learning. Existing studies usually assume that the learned new feature representation is truly domain-invariant, and thus directly train a transfer model M on source domain. In this paper, we consider a more realistic scenario where the new feature representation is suboptimal and small divergence still exists across domains. We propose a new learning strategy with a transfer model called Randomized Transferable Machine (RTM). More specifically, we work on source data with the new feature representation learned from existing feature-based transfer methods. The key idea is to enlarge source training data populations by randomly corrupting source data using some noises, and then train a transfer model ~M that performs well on all the corrupted source data populations. In principle, the more corruptions are made, the higher the probability of the target data can be covered by the constructed source populations. and thus better transfer performance can be achieved by ~M An ideal case is with infinite corruptions, which however is infeasible in reality. We develop a marginalized solution with linear regression model and dropout noise. With a marginalization trick, we can train an RTM that is equivalently to training using infinite source noisy populations without truly conducting any corruption. More importantly, such an RTM has a closed-form solution, which enables very fast and efficient training. Extensive experiments on various real-world transfer tasks show that RTM is a promising transfer model.

We propose a model-based approach to hierarchical reinforcement learning that exploits shared knowledge and selective execution at different levels of abstraction, to efficiently solve large, complex problems. Our framework adopts a new transition dynamics learning algorithm that identifies the common action-feature combinations of the subtasks, and evaluates the subtask execution choices through simulation. The framework is sample efficient, and tolerates uncertain and incomplete problem characterization of the subtasks. We test the framework on common benchmark problems and complex simulated robotic environments. It compares favorably against the state-of-the-art algorithms, and scales well in very large problems.

We propose a new method for transferring a policy from a source task to a target task in model-based reinforcement learning. Our work is motivated by scenarios where a robotic agent operates in similar but challenging environments, such as hospital wards, differentiated by structural arrangements or obstacles, such as furniture. We address problems that require fast responses adapted from incomplete, prior knowledge of the agent in new scenarios. We present an efficient selective exploration strategy that maximally reuses the source task policy. Reuse efficiency is effected through identifying sub-spaces that are different in the target environment, thus limiting the exploration needed in the target task. We empirically show that SEAPoT performs better in terms of jump starts and cumulative average rewards, as compared to existing state-of-the-art policy reuse methods.

Clinical features found in brain CT scan images are widely used in traumatic brain injury (TBI) as indicators for Glasgow Outcome Scale (GOS) prediction. However, due to the lack of automated methods to measure and quantify the CT scan image features, the computerized prediction of GOS in TBI has not been well studied. This paper introduces an automated GOS prediction system for traumatic brain CT images. Different from most existing systems that perform the prognosis based on pre-processed data, our system directly works on brain CT scan images based on the image features. Our system can also be extended to large dataset with easy adaptation. For each new image of a CT scan series, our proposed system first makes use of sparse representation model that predicts the GOS of each CT image slice using Gabor features. Logistic regression, which integrates the GOS of each CT scan slice with a pre-trained model, is then applied to estimate the GOS score for the new case which contains multiple CT slices. Evaluation of the system has shown promising results in prediction of GOS of traumatic brain injury cases.

We propose a new framework for learning the world dynamics of feature-rich environments in model-based reinforcement learning. The main idea is formalized as a new, factored state-transition representation that supports efficient online-learning of the relevant features. We construct the transition models through predicting how the actions change the world. We introduce an online sparse coding learning technique for feature selection in high-dimensional spaces. We derive theoretical guarantees for our framework and empirically demonstrate its practicality in both simulated and real robotics domains.

We study how to automatically select and adapt multiple abstractions or representations of the world to support model-based reinforcement learning. We address the challenges of transfer learning in heterogeneous environments with varying tasks. We present an efficient, online framework that, through a sequence of tasks, learns a set of relevant representations to be used in future tasks. Without pre-defined mapping strategies, we introduce a general approach to support transfer learning across different state spaces. We demonstrate the potential impact of our system through improved jumpstart and faster convergence to near optimum policy in two benchmark domains.

Model minimization in Factored Markov Decision Processes (FMDPs) is concerned with finding the most compact partition of the state space such that all states in the same block are action-equivalent. This is an important problem because it can potentially transform a large FMDP into an equivalent but much smaller one, whose solution can be readily used to solve the original model. Previous model minimization algorithms are iterative in nature, making opaque the relationship between the input model and the output partition. We demonstrate that given a set of well-defined concepts and operations on partitions, we can express the model minimization problem in an analytic fashion. The theoretical results developed can be readily applied to solving problems such as estimating the size of the minimum partition, refining existing algorithms, and so on.

Medical text mining has gained increasing interest in recent years. Radiology reports contain rich information describing radiologistpsilas observations on the patientpsilas medical conditions in the associated medical images. However, as most reports are in free text format, the valuable information contained in those reports cannot be easily accessed and used, unless proper text mining has been applied. In this paper, we propose a text mining system to extract and use the information in radiology reports. The system consists of three main modules: a medical finding extractor, a report and image retriever, and a text-assisted image feature extractor. In evaluation, the overall precision and recall for medical finding extraction are 95.5% and 87.9% respectively, and for all modifiers of the medical findings 88.2% and 82.8% respectively. The overall result of report and image retrieval module and text-assisted image feature extraction module is satisfactory to radiologists.

Multi-slice computer tomography (CT) scans are widely used in todaypsilas diagnosis of head traumas. It is effective to disclose the bleeding and fractures. In this paper, we present an automated detection of CT scan slices which contain hemorrhages. Our method is robust towards various rotation, displacement and motion blur. Detection of these pathological slices will be useful for further diagnosis and retrieval.

This paper describes a new hybrid approach set-based hybrid approach (SHA) for magnetic resonance (MR) image segmentation by integrating two existing techniques, region-grow and threshold level set. To evaluate the proposed approach in performing real world image segmentation task, instead of using well-taken MR-images, we use real-life images collected in a hospital. Comparison of the performance between the two individual techniques and the new hybrid technique demonstrates the effectiveness of the latter

Translation initiation sites (TISs) are important signals in cDNA sequences. Many research efforts have tried to predict TISs in cDNA sequences. In this paper, we propose to use mixture Gaussian models for TIS prediction. Using both local features and some features generated from global measures, the proposed method predicts TISs with a sensitivity of 98 percent and a specificity of 93.6 percent. Our method outperforms many other existing methods in sensitivity while keeping specificity high. We attribute the improvement in sensitivity to the nature of the global features and the mixture Gaussian models.

In this paper we introduce a novel technique for classifying text citations without labeled training examples. We first utilize the search results of a general search engine as original training data. We then proposed a mutually reinforcing learning algorithm (MRL) to mine the classification knowledge and to "clean" the training data. With the help of a set of established domain-specific ontological terms or keywords, the MRL mining step derives the relevant classification knowledge. The MRL cleaning step then builds a naive Bayes classifier based on the mined classification knowledge and tries to clean the training set. The MRL algorithm is iteratively applied until a clean training set is obtained. We show the effectiveness of the proposed technique in the classification of biomedical citations from a large medical literature database.

In intensive care units, the patients, who are suffering from severe head injuries, usually enter a state of coma. To treat such patients, who are prone to a high risk of mortality, the neurologist adopts certain aggressive and informed decision-making procedures. Designing a decision support system (DSS) that would automate or enhance this kind of treatment procedure is difficult, due to the presence of unclear domain relationships, numerous interacting variables, time criticality and real-time multiple inputs. We illustrate how the decision analysis framework can be exploited to build a consultative DSS for severe head injury management. Specifically, we need to: (a) understand the head injury problem, with its inherent uncertainties, (b) structure the problem, and (c) discern the decision process. The designed system accepts the prognostic factors of a particular patient as input, and subsequently provides treatment advice as output. The effectiveness of the treatment is ranked in terms of patient recovery.

This work attempts to abstract medical time series using a minimum number of linear segments such that the integral square error between the abstraction and the data is a minimum. The problem is difficult since it involves a multiobjective optimization procedure, and the optimization process is affected by the presence of local minima, noise and outliers. This work proposes a greedy approach, which exploits local and global information for the optimization. Initially, the number of linear segments needed is estimated roughly by detecting the number of cycles in the data set. Then the tendency of each data point to form bends is measured locally in terms of typicality values. A global consensus in terms of clustering is used to select the breakpoints from all the data points with various typicality values. These breakpoints are utilized to partition the data set. Approximating each partition with a linear segment subsequently forms a crude abstraction. The difference between the original data set and the crude abstraction is exploited as feedback information such that the crude abstraction can be split further for refinement. The efficacy of the proposed method is demonstrated on some real life intensive care unit (ICU) data sets.