The purpose of multi-task reinforcement learning (MTRL) is to train a single policy that can be applied to a set of different tasks. Sharing parameters allows us to take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us challenges on both which tasks should share the parameters and what parameters should be shared, as well as the optimization challenges due to parameter sharing. In this work, we introduce a parameter-compositional approach (PaCo) as an attempt to address these challenges. In this framework, a policy subspace represented by a set of parameters is learned. Policies for all the single tasks lie in this subspace and can be composed by interpolating with the learned set. It allows not only flexible parameter sharing, but also a natural way to improve training.We demonstrate the state-of-the-art performance on Meta-World benchmarks, verifying the effectiveness of the proposed approach.

We consider the safe reinforcement learning (RL) problem of maximizing utility with extremely low constraint violation rates. Assuming no prior knowledge or pre-training of the environment safety model given a task, an agent has to learn, via exploration, which states and actions are safe. A popular approach in this line of research is to combine a model-free RL algorithm with the Lagrangian method to adjust the weight of the constraint reward relative to the utility reward dynamically. It relies on a single policy to handle the conflict between utility and constraint rewards, which is often challenging. We present SEditor, a two-policy approach that learns a safety editor policy transforming potentially unsafe actions proposed by a utility maximizer policy into safe ones. The safety editor is trained to maximize the constraint reward while minimizing a hinge loss of the utility state-action values before and after an action is edited. SEditor extends existing safety layer designs that assume simplified safety models, to general safe RL scenarios where the safety model can in theory be arbitrarily complex. As a first-order method, it is easy to implement and efficient for both inference and training. On 12 Safety Gym tasks and 2 safe racing tasks, SEditor obtains much a higher overall safety-weighted-utility (SWU) score than the baselines, and demonstrates outstanding utility performance with constraint violation rates as low as once per 2k time steps, even in obstacle-dense environments. On some tasks, this low violation rate is up to 200 times lower than that of an unconstrained RL method with similar utility performance. Code is available at https://github.com/hnyu/seditor.

We present temporally abstract actor-critic (TAAC), a simple but effective off-policy RL algorithm that incorporates closed-loop temporal abstraction into the actor-critic framework. TAAC adds a second-stage binary policy to choose between the previous action and a new action output by an actor. Crucially, its "act-or-repeat" decision hinges on the actually sampled action instead of the expected behavior of the actor. This post-acting switching scheme let the overall policy make more informed decisions. TAAC has two important features: a) persistent exploration, and b) a new compare-through Q operator for multi-step TD backup, specially tailored to the action repetition scenario. We demonstrate TAAC's advantages over several strong baselines across 14 continuous control tasks. Our surprising finding reveals that while achieving top performance, TAAC is able to "mine" a significant number of repeated actions with the trained policy even on continuous tasks whose problem structures on the surface seem to repel action repetition. This suggests that aside from encouraging persistent exploration, action repetition can find its place in a good policy behavior. Code is available at https://github.com/hnyu/taac.

Visual recognition under adverse conditions is a very important and challenging problem of high practical value, due to the ubiquitous existence of quality distortions during image acquisition, transmission, or storage. While deep neural networks have been extensively exploited in the techniques of low-quality image restoration and high-quality image recognition tasks respectively, few studies have been done on the important problem of recognition from very low-quality images. This paper proposes a deep learning based framework for improving the performance of image and video recognition models under adverse conditions, using robust adverse pre-training or its aggressive variant. The robust adverse pre-training algorithms leverage the power of pre-training and generalizes conventional unsupervised pre-training and data augmentation methods. We further develop a transfer learning approach to cope with real-world datasets of unknown adverse conditions. The proposed framework is comprehensively evaluated on a number of image and video recognition benchmarks, and obtains significant performance improvements under various single or mixed adverse conditions. Our visualization and analysis further add to the explainability of results.

Visual recognition from very low-quality images is an extremely challenging task with great practical values. While deep networks have been extensively applied to low-quality image restoration and high-quality image recognition tasks respectively, few works have been done on the important problem of recognition from very low-quality images.This paper presents a degradation-robust pre-training approach on improving deep learning models towards this direction. Extensive experiments on different datasets validate the effectiveness of our proposed method.

Building intelligent agents that can communicate with and learn from humans in natural language is of great value.Supervised language learning is limited by the ability of capturing mainly the statistics of training data, and is hardly adaptive to new scenarios or flexible for acquiring new knowledge without inefficient retraining or catastrophic forgetting.We highlight the perspective that conversational interaction serves as a natural interface both for language learning and for novel knowledge acquisition and propose a joint imitation and reinforcement approach for grounded language learning through an interactive conversational game.The agent trained with this approach is able to actively acquire information by asking questions about novel objects and use the justlearned knowledge in subsequent conversations in a one-shot fashion.Results compared with other methods verified the effectiveness of the proposed approach.

Camera motion introduces motion blur, degrading the quality of video. A video deblurring method is proposed based on two observations: (i) camera motion within capture of each individual frame leads to motion blur; (ii) camera motion between frames yields inter-frame mis-alignment that can be exploited for blur removal. The proposed method effectively leverages the information distributed across multiple video frames due to camera motion, jointly estimating the motion between consecutive frames and blur within each frame. This joint analysis is crucial for achieving effective restoration by leveraging temporal information. Extensive experiments are carried out on synthetic data as well as real-world blurry videos. Comparisons with several state-of-the-art methods verify the effectiveness of the proposed method.

This paper describes a robust algorithm for estimating a single latent sharp image given multiple blurry and/or noisy observations. The underlying multi-image blind deconvolution problem is solved by linking all of the observations together via a Bayesian-inspired penalty function, which couples the unknown latent image along with a separate blur kernel and noise variance associated with each observation, all of which are estimated jointly from the data. This coupled penalty function enjoys a number of desirable properties, including a mechanism whereby the relative-concavity or sparsity is adapted as a function of the intrinsic quality of each corrupted observation. In this way, higher quality observations may automatically contribute more to the final estimate than heavily degraded ones, while troublesome local minima can largely be avoided. The resulting algorithm, which requires no essential tuning parameters, can recover a sharp image from a set of observations containing potentially both blurry and noisy examples, without knowing a priori the degradation type of each observation. Experimental results on both synthetic and real-world test images clearly demonstrate the efficacy of the proposed method.

This paper presents a robust algorithm for estimating a single latent sharp image given multiple blurry and/or noisy observations. The underlying multi-image blind deconvolution problem is solved by linking all of the observations together via a Bayesian-inspired penalty function which couples the unknown latent image, blur kernels, and noise levels together in a unique way. This coupled penalty function enjoys a number of desirable properties, including a mechanism whereby the relative-concavity or shape is adapted as a function of the intrinsic quality of each blurry observation. In this way, higher quality observations may automatically contribute more to the final estimate than heavily degraded ones. The resulting algorithm, which requires no essential tuning parameters, can recover a high quality image from a set of observations containing potentially both blurry and noisy examples, without knowing a priori the degradation type of each observation. Experimental results on both synthetic and real-world test images clearly demonstrate the efficacy of the proposed method.

A nonlocal kernel regression (NL-KR) model is presented in this paper for various image and video restoration tasks. The proposed method exploits both the nonlocal self-similarity and local structural regularity properties in natural images. The nonlocal self-similarity is based on the observation that image patches tend to repeat themselves in natural images and videos, and the local structural regularity observes that image patches have regular structures where accurate estimation of pixel values via regression is possible. By unifying both properties explicitly, the proposed NL-KR framework is more robust in image estimation, and the algorithm is applicable to various image and video restoration tasks. In this paper, we apply the proposed model to image and video denoising, deblurring, and superresolution reconstruction. Extensive experimental results on both single images and realistic video sequences demonstrate that the proposed framework performs favorably with previous works both qualitatively and quantitatively.

In this paper, we propose a kernel multi-metric learning algorithm for multi-channel transient acoustic signal classification. The proposed method learns a set of metrics jointly for multi-channel transient acoustic signals in a kernel-induced feature space to exploit the non-linearity of the data for improving the classification performance. An effective algorithm is developed for the task of learning multiple metrics in the kernel space. By learning the multiple metrics jointly within a single unified optimization framework, we can learn better metrics to integrate the multiple channels of the signal for a joint classification. Experimental results compared with classical as well as recent algorithms on real-world acoustic datasets verified the effectiveness of the proposed method.

Image separation from a set of observed mixtures has important applications in many fields such as intrinsic image extraction. We investigate in this work a natural image prior based image separation algorithm. The natural image prior is modeled via a high-order Markov Random Field (MRF) and is integrated into a Bayesian framework for estimating all the component images. Due to the usage of the natural image prior, which typically leading to non-convex optimization problems, there is no closed form solution for estimating the component images. Therefore, a Markov chain Monte-Carlo based sampling algorithm is developed for solution. Based on this, a Minimum Mean Square Error (MMSE) estimation can be achieved. The proposed method exploits both the mixing observations and the prior distribution of natural images, modeled via an MRF model. Experimental results indicate that the proposed method can generate better results than state-of-the-art image separation algorithms.

Sparsity is an ubiquitous property exhibited by many natural real-world data such as images, which has been playing an important role in image and multi-media data processing. However, for many data, such as images, the sparsity pattern is not completely random, i.e., there are structures over the sparse coefficients. By exploiting this structure, we can model the data better and may further improve the performance of the recovery algorithm. In this paper, we exploit the structured sparsity of natural images for image deblurring application. Experimental results clearly demonstrate the effectiveness of the proposed approach.

We propose a new single image super resolution (SR) algorithm via Bayesian modeling with a natural image prior modeled by a high-order Markov random field (MRF). SR is one of the long-standing and active topics in image processing community. It is of great use in many practical applications, such as astronomical observation, medical imaging, and the adaptation of low-resolution contents onto high-resolution displays. One category of the conventional approaches for image SR is formulating the problem with Bayesian modeling techniques and then obtaining its maximum-a-posteriori solution, which actually boils down to a regularized regression task. Although straightforward, this approach cannot exploit the full potential offered by the probabilistic modeling, as only the posterior mode is sought. On the other hand, current Bayesian SR approaches using the posterior mean estimation typically use very simple prior models for natural images to ensure the computational tractability. In this paper, we present a Bayesian image SR approach with a flexible high-order MRF model as the prior for natural images. The minimum mean square error (MMSE) criteria are used for estimating the HR image. A Markov chain Monte Carlo-based sampling algorithm is presented for obtaining the MMSE solution. The proposed method cannot only enjoy the benefits offered by the flexible prior, but also has the advantage of making use of the probabilistic modeling to perform a posterior mean estimation, thus is less sensitive to the local minima problem as the MAP solution. Experimental results indicate that the proposed method can generate competitive or better results than state-of-the-art SR algorithms.

This paper investigates the joint-structured-sparsity-based methods for transient acoustic signal classification with multiple measurements. By joint structured sparsity, we not only use the sparsity prior for each measurement but we also exploit the structural information across the sparse representation vectors of multiple measurements. Several different sparse prior models are investigated in this paper to exploit the correlations among the multiple measurements with the notion of the joint structured sparsity for improving the classification accuracy. Specifically, we propose models with the joint structured sparsity under different assumptions: same sparse code model, common sparse pattern model, and a newly proposed joint dynamic sparse model. For the joint dynamic sparse model, we also develop an efficient greedy algorithm to solve it. Extensive experiments are carried out on real acoustic data sets, and the results are compared with the conventional discriminative classifiers in order to verify the effectiveness of the proposed method.

In this paper, we present a novel joint sparse representation based method for acoustic signal classification with multiple measurements. The proposed method exploits the correlations among the multiple measurements with the notion of joint sparsity for improving the classification accuracy. Extensive experiments are carried out on real acoustic data sets and the results are compared with the conventional discriminative classifiers in order to verify the effectiveness of the proposed method.

We address the problem of visual recognition from multiple observations of the same physical object, which can be generated under different conditions, such as frames at different time instances or snapshots from different viewpoints. We formulate the multi-observation visual recognition task as a joint sparse representation model and take advantage of the correlations among the multiple observations for classification using a novel joint dynamic sparsity prior. The proposed joint dynamic sparsity prior promotes shared joint sparsity pattern among the multiple sparse representation vectors at class-level, while allowing distinct sparsity patterns at atom-level within each class in order to facilitate a flexible representation. The proposed method can handle both homogenous as well as heterogenous data within the same framework. Extensive experiments on various visual classification tasks including face recognition and generic object classification demonstrate that the proposed method outperforms existing state-of-the-art methods

Most previous visual recognition systems simply assume ideal inputs without real-world degradations, such as low resolution, motion blur and out-of-focus blur. In presence of such unknown degradations, the conventional approach first resorts to blind image restoration and then feeds the restored image into a classifier. Treating restoration and recognition separately, such a straightforward approach, however, suffers greatly from the defective output of the ill-posed blind image restoration. In this paper, we present a joint blind image restoration and recognition method based on the sparse representation prior to handle the challenging problem of face recognition from low-quality images, where the degradation model is realistic and totally unknown. The sparse representation prior states that the degraded input image, if correctly restored, will have a good sparse representation in terms of the training set, which indicates the identity of the test image. The proposed algorithm achieves simultaneous restoration and recognition by iteratively solving the blind image restoration in pursuit of the sparest representation for recognition. Based on such a sparse representation prior, we demonstrate that the image restoration task and the recognition task can benefit greatly from each other. Extensive experiments on face datasets under various degradations are carried out and the results of our joint model shows significant improvements over conventional methods of treating the two tasks independently.

We consider the problem of automatically recognizing a human face from its multi-view images with unconstrained poses and illuminations. We formulate the multi-view face recognition problem as that of classifying among several multi-input (views) regression models by using a novel joint dynamic sparse representation method which exploits jointly the inter-correlation among all the multi-view images in order to make a decision. Extensive experiments on CMU Multi-PIE face database are conducted to verify the efficacy of the proposed method.

In this paper, we propose an extension of the Non-Local Kernel Regression (NL-KR) method and apply it to super-resolution (SR) tasks. The proposed method extends NL-KR via generalizing the self-similarity from single-scale to multi-scale, and propose an effective SR algorithm using the proposed multi-scale NL-KR model. Experimental results on both synthetic and real images demonstrate the effectiveness of the proposed method.

We propose a sparse representation based blind image deblurring method. The proposed method exploits the sparsity property of natural images, by assuming that the patches from the natural images can be sparsely represented by an over-complete dictionary. By incorporating this prior into the deblurring process, we can effectively regularize the ill-posed inverse problem and alleviate the undesirable ring effect which is usually suffered by conventional deblurring methods. Experimental results compared with state-of-the-art blind deblurring method demonstrate the effectiveness of the proposed method.

Super resolution is of great use in many visual media related scenarios, such as displaying low resolution contents on High-Definition TV(HD-TV). In these scenarios, the efficiency of the super resolution process is of vital importance. This paper presents a fast learning based super-resolution method. The proposed method speeds up the sparse representation based super-resolution method by learning a dual dictionary, and replaces the sparse recovery step by simple matrix multiplication, which is much more computationally efficient. Experiments demonstrate that the proposed method can generate desirable super-resolved images with significant computational advantages.

As the biggest sediment-laden river in the world, the river regime variation of the Yellow River is complicated which results in the difficulty in flood control at the wandering lower reaches of the Yellow River. The problem of the river regime variation detection with multi-spectral remote sensing images is investigated in this paper. Firstly, the flow characteristic of the wandering lower reaches of the Yellow River is analyzed. Then, based on the spectral similarity and the principle of spatial continuity, a main-stream detection approach called Spectral Correlation Dynamic Transmission (SCDT) algorithm is presented. Finally, bow parameters are calculated and the river regime variation is analyzed based on these parameters. Experimental results are in accordance with the periodic river regime variation of the Yellow River which indicates the effectiveness of the proposed method. The method presented in this paper can be applied directly in flood control.

A novel and convenient multi-camera self-calibration method is proposed in this paper. Different from other calibration methods, our method is done by analyzing human body motion. The only constraint is that several people of different heights are needed to walk around the experimental environment one by one in the calibration period. By this way, two kinds of corresponding points are extracted from synchronous video sequences. One is the centroid of the moving human body. The other is points on the floor, which is extracted by matching floor planes in video sequences. The floor planes registration is based on shadow detection and co-motion feature. Based on these corresponding points, camera parameters and 3D points observed are estimated. The proposed method is tested in our own experimental environment. Experimental results show the accuracy of our calibration method. Our method can satisfy many applications of multi-view computer vision.

Inspired by the observation that in image restoration, parametric models are extremely specific while pixel-level models are too loose, tending to under or over fit the underlying image respectively, in this paper, we proposed an ‘intermediate-language’ based method for image deblurring. The solution space is represented at a level higher than the pixel-grid level, while retain an enough degree of freedom (DOF), thus avoids the common local under or over fitting problem. Considering the sparseness property of images, a sparse representation based incremental iterative method is established for blurry image restoration. Comprehensive experiments demonstrate that the framework integrating the sparseness property of images significantly improves the deblurring performance.