In the vision and language navigation task (Anderson et al. 2018), the agent may encounter ambiguous situations that are hard to interpret by just relying on visual information and natural language instructions. We propose an interactive learning framework to endow the agent with the ability to ask for users' help in such situations. As part of this framework, we investigate multiple learning approaches for the agent with different levels of complexity. The simplest model-confusion-based method lets the agent ask questions based on its confusion, relying on the predefined confidence threshold of a next action prediction model. To build on this confusion-based method, the agent is expected to demonstrate more sophisticated reasoning such that it discovers the timing and locations to interact with a human. We achieve this goal using reinforcement learning (RL) with a proposed reward shaping term, which enables the agent to ask questions only when necessary. The success rate can be boosted by at least 15% with only one question asked on average during the navigation. Furthermore, we show that the RL agent is capable of adjusting dynamically to noisy human responses. Finally, we design a continual learning strategy, which can be viewed as a data augmentation method, for the agent to improve further utilizing its interaction history with a human. We demonstrate the proposed strategy is substantially more realistic and data-efficient compared to previously proposed pre-exploration techniques.

Yuheng Du, Shereen Oraby, Vittorio Perera, Minmin Shen, Anjali Narayan-Chen, Tagyoung Chung, Anushree Venkatesh, Dilek Hakkani-Tur. Proceedings of the 13th International Conference on Natural Language Generation. 2020.

In this paper, we present a deep convolutional neural network (ConvNet) based framework for estimating the bee pose from a single image. Unlike some existing human pose estimation methods that localize a fixed number of body joints, our method handles the cases with a varying number of targets. Compared to the existing bee pose estimation methods, our framework is more robust and accurate. It is effective even for some challenging images (e.g., when the bee is fed sugar water with a stick). The proposed framework learns a mapping from the global structure and local appearance of a bee to its pose. We evaluated our method on two challenging datasets. Experiments showed that it has achieved significant improvements over the existing insect pose estimation algorithms.

A vast amount of toxicological data can be obtained from feature analysis of cells treated in vitro. However, this requires microscopic image segmentation of cells. To this end, we propose a new strategy, namely Supervised Normalized Cut Segmentation (SNCS), to segment cells that partially overlap and have a large amount of curved edges. SNCS approach is a machine learning based method, where loosely annotated images are used first to train and optimise parameters, and then the optimal parameters are inserted into a Normalized Cut segmentation process. Furthermore, we compare our segmentation results using SNCS to another four classical and two state-of-the-art methods. The overall experimental result shows the usefulness and effectiveness of our method over the six comparison methods.

In this paper, we propose a strategy to optimize feature pooling and prediction models of video quality assessment (VQA) algorithms with a much smaller number of parameters than methods based on machine learning, such as neural networks. Based on optimization, the proposed mapping strategy is composed of a global linear model for pooling extracted features, a simple linear model for local alignment in which local factors depend on source videos, and a non-linear model for quality calibration. Also, a reduced-reference VQA algorithm is proposed to predict the local factors from the source video. In the IRCCyN/IVC video database of content influence and the LIVE mobile video database, the performance of VQA algorithms is improved significantly by local alignment. The proposed mapping strategy with prediction of local factors outperforms one no-reference VQA metric and is comparable to one full-reference VQA metric. Thus predicting the local factors in local alignment based on video content will be a promising new approach for VQA.

Extracting motion trajectories of insects is an important prerequisite in many behavioral studies. Despite great efforts to design efficient automatic tracking algorithms, tracking errors are unavoidable. In this paper, we propose general principles that help to minimize the human effort required for accurate multi-target tracking in the form of applications that can track the antennae and mouthparts of a honey bee based on a set of low frame rate videos. This interactive framework estimates which key frames will require user correction, i.e. those that are used for user correction, which are used for 1) incrementally learning an object classifier and 2) data association based tracking. To this framework we apply a standard classification algorithm (i.e. naive Bayesian classification) and an association optimization algorithm (i.e. Hungarian algorithm). The precision of tracking results by our framework on real-world video data is above 98%.

Automatic tracking of the movement of bee's antennae and mouthparts is necessary for studying associative learning of individuals. However, the problem of tracking them is challenging: First, the different classes of objects possess similar appearance and are close to each other. Second, tracking gaps are often present, due to the low frame-rate of the acquired video and the fast motion of the objects. Most existing insect tracking approaches have been developed for slow moving objects, and are not suitable for this application. In this paper, a novel Bayesian framework is proposed to automatically track bees' antennae and their mouthparts. This framework incorporates information about their kinematics, shape, order and temporal correlation between neighboring frames. Experimental evaluation demonstrates the effectiveness and efficiency of the proposed framework.

Detection and removal of rain is important in outdoor surveillance vision systems, since the appearance of rain strikes degrades the performance of various vision-based applications. The existing algorithms address the issue of detecting rain only in the irradiance light field, thus require dozens of successive frames to compute the temporal correlation of rain. Combining the properties of rain in irradiance light field and motion field, this paper presents a new approach for rain detection and removal using only three successive frames. In this approach, motion data are used to differentiate rain from other moving objects. A smoothing method based on anisotropic diffusion is proposed for rain removal. Experimental results verify the efficacy of our algorithm.

It has been reported that oversampling a still image before compression does not guarantee a good image quality. Similarly, down-sampling before video compression in low bit rate video coding may alleviate the blocking effect and improve peak signal-to-noise ratio of the decoded frames. When the number of discrete cosine transform coefficients is reduced in such a down-sampling based coding (DBC), the bit budget of each coefficient will increase, thus reduce the quantization error. A DBC video coding scheme is proposed in this paper, where a super-resolution technique is employed to restore the down-sampled frames to their original resolutions. The performance improvement of the proposed DBC scheme is analyzed at low bit rates, and verified by experiments.

Super-resolution reconstruction (SR) is a technique for estimating a high resolution (HR) image from multiple low resolution (LR) copies captured from the same scene. Most of the existing SR algorithms are based on the assumption that the scene moves parallel to the camera lens with translational or rotational motion. However, such an assumption may not be held if zooming exists when acquiring LR images. We present in this paper a new linear model to represent the relationship between the HR image and the LR images captured with arbitrary sampling lattices. Based on this model, a MAP based SR algorithm is proposed. Experimental results verify the improvements on the visual quality of our framework.

It has been proven that performance of video coding at low bit rates can be improved by down-sampling a video before compression and up-sampling it after decompression. The down-sampling based coding (DBC) provides more bit budget for DCT coefficients, which preserve more high order coefficients. In this paper we propose a new DBC which uses a modified example-based super-resolution (SR) algorithm to restore the original resolution of down-sampled video signals. Experiments have shown the proposed scheme achieve PSNR improvement and better visual quality at low bit rates as compared with the H.264/AVC video coding, as well as the DBC using a general locally linear embedding (LLE) method as in.

A novel scheme of scalable video coding (SVC) using super-resolution techniques is proposed in this paper. Utilizing the spatial/temporal scalability of H.264/AVC, we encode half of input high-resolution(HR) frames and their low-resolution (LR) counterparts in a video sequence and employ super-resolution (SR) method to reconstruct skipped frames during decoding. The payload saved from skipped frames can be used to improve the quality of encoded frames. This scheme provides a choice of SVC to improve the quality of HR frames while maintaining low bitrate transmission and reducing encoding complexity. Experiments show that our scheme performs well in both peak signal to noise ratio (PSNR) and subjective visual quality at low bitrate.

Super-resolution reconstruction (SR) has been widely used to produce a high resolution (HR) image from several low resolution (LR) ones. In current methods, a LR image is selected as benchmark and upsampled as the initial SR estimate. This SR estimate is then degraded and compared with the adjacent LR frames for correction. Considering LR images captured from the same HR image with different translation at different instants, SR outputs by different benchmark selection should be identical, so tighter constraints can be designed to limit SR indetermination and to produce better SR images. In this paper, we propose a novel SR framework and prove its efficiency statistically using the unbiased estimation. Experimental results indicate that the proposed algorithm outperforms some existing approaches in both subjective and objective terms.