This paper presents an end-to-end self-supervised learning approach for cross-modality image registration and homography estimation, with a particular emphasis on registering sports field templates onto broadcast videos as a practical application. Rather then using any pairwise labelled data for training, we propose a self-supervised data mining method to train the registration network with a natural image and its edge map. Using an iterative estimation process controlled by a score regression network (SRN) to measure the registration error, the network can learn to estimate any homography transformation regardless of how misaligned the image and the template is. We further show the benefits of using pretrained weights to finetune the network for sports field calibration with few training data. We demonstrate the effectiveness of our proposed method by applying it to real-world sports broadcast videos where we achieve state-of-the-art results and real-time processing.

Person re-identification is an important yet challenging problem in visual recognition. Despite the recent advances with deep learning (DL) models for spatio-temporal and multi-modal fusion, re-identification approaches often fail to leverage the contextual information (e.g., pose and illumination) to dynamically select the most discriminant con-volutional filters (i.e., appearance features) for feature representation and inference. State-of-the-art techniques for gated fusion employ complex dedicated part- or attention-based architectures for late fusion, and do not incorporate pose and appearance information to train the backbone network. In this paper, a new DL model is proposed for pose-guided re-identification, comprised of a deep backbone, pose estimation, and gated fusion network. Given a query image of an individual, the backbone convolutional NN produces a feature embedding required for pair-wise matching with embeddings for reference images, where feature maps from the pose network and from mid-level CNN layers are combined by the gated fusion network to generate pose-guided gating. The proposed framework allows to dynamically activate the most discriminant CNN filters based on pose information in order to perform a finer grained recognition. Extensive experiments on three challenging benchmark datasets indicate that integrating the pose-guided gated fusion into the state-of-the-art re-identification backbone architecture allows to improve their recognition accuracy. Experimental results also support our intuition on the advantages of gating backbone appearance information using the pose feature maps at mid-level CNN layers.

An ongoing major challenge in computer vision is the task of person re-identification, where the goal is to match individuals across different, non-overlapping camera views. While recent success has been achieved via supervised learning using deep neural networks, such methods have limited widespread adoption due to the need for large-scale, customized data annotation. As such, there has been a recent focus on unsupervised learning approaches to mitigate the data annotation issue; however, current approaches in literature have limited performance compared to supervised learning approaches as well as limited applicability for adoption in new environments. In this paper, we address the aforementioned challenges faced in person re-identification for real-world, practical scenarios by introducing a novel, unsupervised domain adaptation approach for person re-identification. This is accomplished through the introduction of: i) k-reciprocal tracklet Clustering for Unsupervised Domain Adaptation (ktCUDA) (for pseudo-label generation on target domain), and ii) Synthesized Heterogeneous RE-id Domain (SHRED) composed of large-scale heterogeneous independent source environments (for improving robustness and adaptability to a wide diversity of target environments). Experimental results across four different image and video benchmark datasets show that the proposed ktCUDA and SHRED approach achieves an average improvement of +5.7 mAP in re-identification performance when compared to existing state-of-the-art methods, as well as demonstrate better adaptability to different types of environments.

In this letter, a novel approach to visual saliency detection via neural response divergence (NeRD) is proposed, where synaptic portions of deep neural networks, previously trained for complex object recognition, are leveraged to compute low-level cues that can be used to compute image region distinctiveness. Based on this concept, an efficient visual saliency detection framework is proposed using deep convolutional StochasticNets. Experimental results using complex scene saliency dataset and MSRA10k natural image datasets show that the proposed NeRD approach can achieve improved performance when compared to state-of-the-art image saliency approaches, while attaining low computational complexity necessary for near-real-time computer vision applications.

This paper proposes DESIRe (Discontinuous Energy Seam-carving Image Retargeting), an improved seam carving approach for content-aware image retargeting. The proposed algorithm introduces a novel discontinuous seam carving optimization process that incorporates not only a structural energy cost functional, but also a texture energy cost functional to handle scenes with complex structural and textural characteristics. Experimental results using a variety of scenes with dense image detail characteristics show that the proposed DESIRe seam carving approach can provide improved visual quality when compared to a number of existing seam carving methods. These results illustrate the potential of the proposed DESIRe approach for improving content-aware image retargeting that reduces the loss of important image information without causing significant visual distortions or artifacts.

This paper proposes DESIRe (Discontinuous Energy Seam-carving Image Retargeting), an improved seam carving approach for content-aware image retargeting. The proposed algorithm introduces a novel discontinuous seam carving optimization process that incorporates not only a structural energy cost functional, but also a texture energy cost functional to handle scenes with complex structural and textural characteristics. Experimental results using a variety of scenes with dense image detail characteristics show that the proposed DESIRe seam carving approach can provide improved visual quality when compared to a number of existing seam carving methods. These results illustrate the potential of the proposed DESIRe approach for improving content-aware image retargeting that reduces the loss of important image information without causing significant visual distortions or artifacts.

Superpixels have been widely used for compact image representation in a large number of computer vision applications such as object recognition, segmentation, and depth estimation. Recently, a novel seam carving approach for superpixel generation called Grid Seams was introduced that significantly improved image structure preservation while maintaining global spatial constraints. While Grid Seams was able to achieve state-of-the-art superpixel accuracy, it only took advantage of rudimentary edge information in its energy function and as such did not account for other important image characteristics such as texture. Motivated by this, we present Return of Grid Seams (RGS), a novel extension of Grid Seams that takes advantage of not only structural variations, but also textural variations (in the form of texture distinctiveness) into a unified multi-functional energy along with global spatial constraints. Furthermore, RGS incorporates discontinuous seams in the optimization process to allow for greater flexibility in preserving structural information. Experimental results using the Berkeley Segmentation Dataset show that RGS is able to outperform Grid Seams as well as a number of other seam carving based superpixel methods.

We present an illumination-compensation method to enable fast and reliable background subtraction under sudden, local illumination changes in wide area surveillance videos. We use Probabilistic Illumination Range Modeling (PIRM) to model the conditional probability distribution of current frame intensity given background intensity. With this model, we can identify a continuous range of current frame intensities that map to the same background intensity, and scale all pixels within that range in the current frame appropriately to enable illumination-compensated background subtraction. Experimental results using a standard academic dataset as well as very challenging industry videos show that PIRM can achieve improvements in compensating for sudden, local illumination changes.

Conditional random fields (CRFs) are one of the most powerful frameworks in image modeling. However practical CRFs typically have edges only between nearby nodes; using more interactions and expressive relations among nodes make these methods impractical for large-scale applications, due to the high computational complexity. Recent work has shown that fully connected CRFs can be tractable by defining specific potential functions. In this paper, we present a novel framework to tackle the computational complexity of a fully connected graph without requiring specific potential functions. Instead, inspired by random graph theory and sampling methods, we propose a new clique structure called stochastic cliques. The stochastically fully connected CRF (SFCRF) is a marriage between random graphs and random fields, benefiting from the advantages of fully connected graphs while maintaining computational tractability. The effectiveness of SFCRF was examined by binary image labeling of highly noisy images. The results show that the proposed framework outperforms an adjacency CRF and a CRF with a large neighborhood size.

Conditional random fields (CRF) have been used extensively for spatially coherent segmentation and classification of images. As a result, may techniques for finding the optimal inference of CRFs have been developed. However, CRFs have seen little use in clustering and segmenting of satellite imagery due to the large number of pixels in satellite images. In this paper we present a means of defining remote sensing imagery as a region based conditional random field (CRF). Unlike the pixel based CRF, our region based CRF can be optimally solved using the latest advancements in CRF inference techniques because the region based CRF is a computationally simpler problem to tackle. To reduce the loss of information when forming the region based CRF, the regions are defined using a superpixel algorithm which decreases the spatial resolution while preserving the original satellite image's structure. Furthermore, unlike previous approaches, we show that the optimal number of clusters in the satellite images can be automatically determined by formulating the number of clusters as part of the CRF cost function. This unsupervised region based approach is a non-parametric formulation which is validated using different imaging modalities: SAR and hyper-spectral imaging.

We propose a principled probabilistic formulation of object saliency as a sampling problem. This novel formulation allows us to learn, from a large corpus of unlabelled images, which patches of an image are of the greatest interest and most likely to correspond to an object. We then sample the object saliency map to propose object locations. We show that using only a single object location proposal per image, we are able to correctly select an object in over 42% of the images in the Pascal VOC 2007 dataset, substantially outperforming existing approaches. Furthermore, we show that our object proposal can be used as a simple unsupervised approach to the weakly supervised annotation problem. Our simple unsupervised approach to annotating objects of interest in images achieves a higher annotation accuracy than most weakly supervised approaches.

Most existing approaches to training object detectors rely on fully supervised learning, which requires the tedious manual annotation of object location in a training set. Recently there has been an increasing interest in developing weakly supervised approach to detector training where the object location is not manually annotated but automatically determined based on binary (weak) labels indicating if a training image contains the object. This is a challenging problem because each image can contain many candidate object locations which partially overlaps the object of interest. Existing approaches focus on how to best utilise the binary labels for object location annotation. In this paper we propose to solve this problem from a very different perspective by casting it as a transfer learning problem. Specifically, we formulate a novel transfer learning based on learning to rank, which effectively transfers a model for automatic annotation of object location from an auxiliary dataset to a target dataset with completely unrelated object categories. We show that our approach outperforms existing state-of-the-art weakly supervised approach to annotating objects in the challenging VOC dataset.

The detection of human action in videos of busy natural scenes with dynamic background is of interest for applications such as video surveillance. Taking a conventional fully supervised approach, the spatio-temporal locations of the action of interest have to be manually annotated frame by frame in the training videos, which is tedious and unreliable. In this paper, for the first time, a weakly supervised action detection method is proposed which only requires binary labels of the videos indicating the presence of the action of interest. Given a training set of binary labelled videos, the weakly supervised learning (WSL) problem is recast as a multiple instance learning (MIL) problem. A novel MIL algorithm is developed which differs from the existing MIL algorithms in that it locates the action of interest spatially and temporally by globally optimising both interand intra-class distance. We demonstrate through experiments that our WSL approach can achieve comparable detection performance to a fully supervised learning approach, and that the proposed MIL algorithm significantly outperforms the existing ones.

A conventional approach to learning object detectors uses fully supervised learning techniques which assumes that a training image set with manual annotation of object bounding boxes are provided. The manual annotation of objects in large image sets is tedious and unreliable. Therefore, a weakly supervised learning approach is desirable, where the training set needs only binary labels regarding whether an image contains the target object class. In the weakly supervised approach a detector is used to iteratively annotate the training set and learn the object model. We present a novel weakly supervised learning framework for learning an object detector. Our framework incorporates a new initial annotation model to start the iterative learning of a detector and a model drift detection method that is able to detect and stop the iterative learning when the detector starts to drift away from the objects of interest. We demonstrate the effectiveness of our approach on the challenging PASCAL 2007 dataset.

For automated surveillance, it is useful to detect specific actions performed by people in busy natural environments. This differs from and thus more challenging than the intensively studied action recognition problem in that for action detection in crowd an action of interest is often overwhelmed by large number of background activities of other objects in the scene. Motivated by the success of sliding-window based 2D object detection approaches, in this paper, we propose to tackle the problem by learning a discriminative classifier from annotated 3D action cuboids to capture the intra-class variation, and sliding 3D search windows for detection. The key innovation of our method is a novel greedy k nearest neighbour algorithm for automated annotation of positive training data, by which an action detector can be learned with only a single training sequence being annotated thus greatly alleviating the tedious and unreliable 3D manual annotation. Extensive experiments on real-world action detection datasets demonstrate that our detector trained with minimal annotation can achieve comparable results to that learned with full annotation, and outperforms existing methods.