Stable imaging in adverse environments (e.g., total darkness) makes thermal infrared (TIR) cameras a prevalent option for night scene perception. However, the low contrast and lack of chromaticity of TIR images are detrimental to human interpretation and subsequent deployment of RGB-based vision algorithms. Therefore, it makes sense to colorize the nighttime TIR images by translating them into the corresponding daytime color images (NTIR2DC). Despite the impressive progress made in the NTIR2DC task, how to improve the translation performance of small object classes is under-explored. To address this problem, we propose a generative adversarial network incorporating feedback-based object appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module and corresponding appearance consistency loss are proposed to reduce the context dependence of object translation. As a representative example of small objects in nighttime street scenes, we illustrate how to enhance the realism of traffic light by designing a traffic light appearance loss. To further improve the appearance learning of small objects, we devise a dual feedback learning strategy to selectively adjust the learning frequency of different samples. In addition, we provide pixel-level annotation for a subset of the Brno dataset, which can facilitate the research of NTIR image understanding under multiple weather conditions. Extensive experiments illustrate that the proposed FoalGAN is not only effective for appearance learning of small objects, but also outperforms other image translation methods in terms of semantic preservation and edge consistency for the NTIR2DC task. Compared with the state-of-the-art NTIR2DC approach, FoalGAN achieves at least 5.4% improvement in semantic consistency and at least 2% lead in edge consistency.

Robust imaging under challenging conditions, such as starlit nights, has broadened the adoption of thermal infrared (TIR) cameras for nighttime driving scenes. Given that TIR images are monochromatic, which makes them difficult to interpret by humans and limits the applicability of RGB-based algorithms, it is reasonable to perform colorization of nighttime TIR (NTIR) images by converting them into corresponding daytime color images (NTIR2DC). Despite the impressive results achieved by previous NTIR2DC methods, how to improve the colorization performance of small-sample categories without semantic annotation is under-explored. To address this issue, we propose a novel learning framework called Memory-guided cOllaboRative atteNtion Generative Adversarial Network (MornGAN), which is inspired by the analogical reasoning mechanisms of humans. Specifically, we first propose an online semantic distillation module to mine and refine the semantic cues of NTIR images. Then, a memory-guided sample selection strategy and adaptive collaborative attention loss are devised to enhance the semantic preservation of small-sample categories. Further, a new conditional gradient repair loss is introduced for reducing edge distortion during translation. Extensive experiments on the NTIR2DC task show that the proposed MornGAN significantly outperforms other image-to-image translation methods in terms of semantic preservation and edge consistency, which helps improve the object detection accuracy remarkably.

Recent visual odometry (VO) methods incorporating geometric algorithm into deep-learning architecture have shown outstanding performance on the challenging monocular VO task. Despite encouraging results are shown, previous methods ignore the requirement of generalization capability under noisy environment and various scenes. To address this challenging issue, this work first proposes a novel optical flow network (PANet). Compared with previous methods that predict optical flow as a direct regression task , our PANet computes optical flow by predicting it into the discrete position space with optical flow probability volume, and then converting it to optical flow. Next, we improve the bundle adjustment module to fit the self-supervised training pipeline by introducing multiple sampling, ego-motion initialization, dynamic damping factor adjustment, and Jacobi matrix weighting. In addition, a novel normalized photometric loss function is advanced to improve the depth estimation accuracy. The experiments show that the proposed system not only achieves comparable performance with other state-of-the-art self-supervised learning-based methods on the KITTI dataset, but also significantly improves the generalization capability compared with geometry-based, learning-based and hybrid VO systems on the noisy KITTI and the challenging outdoor (KAIST) scenes.

PointGoal navigation in indoor environment is a fundamental task for personal robots to navigate to a specified point. Recent studies solved this PointGoal navigation task with near-perfect success rate in photo-realistically simulated environments, under the assumptions with noiseless actuation and most importantly, perfect localization with GPS and compass sensors. However, accurate GPS signalis difficult to be obtained in real indoor environment. To improve the PointGoal navigation accuracy without GPS signal, we use visual odometry (VO) and propose a novel action integration module (AIM) trained in unsupervised manner. Sepecifically, unsupervised VO computes the relative pose of the agent from the re-projection error of two adjacent frames, and then replaces the accurate GPS signal with the path integration. The pseudo position estimated by VO is used to train action integration which assists agent to update their internal perception of location and helps improve the success rate of navigation. The training and inference process only use RGB, depth, collision as well as self-action information. The experiments show that the proposed system achieves satisfactory results and outperforms the partially supervised learning algorithms on the popular Gibson dataset.

Significant progress has been made in boundary detection with the help of convolutional neural networks. Recent boundary detection models not only focus on real object boundary detection but also “crisp” boundaries (precisely localized along the object's contour). There are two methods to evaluate crisp boundary performance. One uses more strict tolerance to measure the distance between the ground truth and the detected contour. The other focuses on evaluating the contour map without any postprocessing. In this study, we analyze both methods and conclude that both methods are two aspects of crisp contour evaluation. Accordingly, we propose a novel network named deep refinement network (DRNet) that stacks multiple refinement modules to achieve richer feature representation and a novel loss function, which combines cross-entropy and dice loss through effective adaptive fusion. Experimental results demonstrated that we achieve state-of-the-art performance for several available datasets.

Relevant physiological studies have revealed that the response of the classical receptive field (CRF) to visual stimuli could be suppressed by non‐CRF (nCRF) inhibition of the kernel in the primary visual cortex (V1). Based on this mechanism, many bio‐inspired contour detection models have been proposed, which are mainly achieved through CRF responses and nCRF surround inhibition calculation. In fact, the dynamic characteristics of neurons play an important role in contour detection in biological vision. Inspired by these visual mechanisms, the authors propose a contour detection model that emulates these dynamic characteristics. By introducing a multi‐bandwidth Gabor filter, according to the target image, they can effectively adjust the weight ratios of the filter to protect the contours and filter the background textures in the calculation of CRF responses. Additionally, they logarithmically modulate the nCRF inhibition kernel to make texture suppression more flexible and effective, thus improving the accuracy of detection algorithm as a whole. Compared with existing bio‐inspired contour detection models, the proposed model is more effective at contour detection, which will aid engineering applications that utilise pattern recognition in machine vision.

In the mammalian primary visual cortex, the response of the classical receptive field (CRF) to visual stimuli can be suppressed by inhibition of non‐CRF (nCRF) neurons. Although many biologically plausible models based on these centre–surround interaction properties have been proposed, most of these models have failed to account for two important behaviours of neurons in the primary visual cortex (V1). First, saturation properties of neuron response. Second, the properties of fixational eye movements (FEyeMs). In the present study, the authors proposed a biologically motivated counter detection approach based on these properties. The authors’ work is significant in that they utilised a simple threshold method to ensure that CRF responses were observed within a meaningful range, and multichannel filter bank was proposed to simulate the influence of FEyeMs on nCRF. Both methods effectively preserved object contours and inhibition isolated textures. Extensive experiments indicated that the authors’ model can preserve more object contours and suppress more textures than previous biologically based models.

Psychophysical and neurophysiological investigations on the human visual system show that most neurons in the primary visual cortex (V1) possess a non‐classical receptive field (nCRF) region in addition to the CRF region. The nCRF has a modulatory, normally inhibitory, effect on the responses to visual stimuli generated within the CRF. In computational terms, this mechanism suppresses the response to edges in the presence of similar edges in the surroundings. Many computational techniques have been proposed to address the surround suppression mechanism. These methods introduce an inhibition term that is required to suppress the textures and protect the contours. Several studies have found that the spatial summation properties over the receptive fields of retinal X cells are approximately linear, while they are non‐linear for Y cells. Inspired by the visual information processing in the X–Y channel and spatial summation properties of X and Y cells, the authors propose a contour detector using linear and non‐linear modulations based on nCRF suppression. Extensive experimental evaluations demonstrate that their contour detector significantly outperforms other algorithms. The methods proposed in this study are expected to facilitate the development of efficient computational models in the field of machine vision.