We introduce a novel neural network architecture—spectral encoder for sensor independence (SEnSeI)—by which several multispectral instruments, each with different combinations of spectral bands, can be used to train a generalized deep learning model. We focus on the problem of cloud masking, using several preexisting datasets, and a new, freely available dataset for Sentinel-2. Our model is shown to achieve state-of-the-art performance on the satellites on which it was trained (Sentinel-2 and Landsat 8) and is able to extrapolate to sensors that it has not seen during training, such as Landsat 7, PerúSat-1, and Sentinel-3 Sea and Land Surface Temperature Radiometer (SLSTR). Model performance is shown to improve when multiple satellites are used in training, approaching, or surpassing the performance of specialized, single-sensor models. This work is motivated by the fact that the remote sensing community has access to data taken with a huge variety of sensors. This has inevitably led to labeling efforts being undertaken separately for different sensors, which limits the performance of deep learning models, given their need for huge training sets to perform optimally. Sensor independence can enable deep learning models to utilize multiple datasets for training simultaneously, boosting performance, and making them much more widely applicable. This may lead to deep learning approaches being used more frequently for onboard applications and in ground segment data processing, which generally requires models to be ready at launch or soon afterward.

In this paper, we introduce a variation of a state-of-the-art real-time tracker (CFNet), which adds to the original algorithm robustness to target loss without a significant computational overhead. The new method is based on the assumption that the feature map can be used to estimate the tracking confidence more accurately. When the confidence is low, we avoid updating the object's position through the feature map; instead, the tracker passes to a single-frame failure mode, during which the patch's low-level visual content is used to swiftly update the object's position, before recovering from the target loss in the next frame. The experimental evidence provided by evaluating the method on several tracking datasets validates both the theoretical assumption that the feature map is associated to tracking confidence, and that the proposed implementation can achieve target recovery in multiple scenarios, without compromising the real-time performance.

We address the problem of automatically coregistering planetary images to a common baseline, introducing a novel generic technique that achieves an unprecedented robustness to different image inputs, thus making batch-mode coregistration achievable without requiring the usual parameter tweaking. We introduce a novel image matching technique, which boosts matching performance even under the most strenuous circumstances, and experimentally demonstrate validation through an extensive experimental multi-instrument setup that includes images from eight high-resolution data sets of the Mars and the Moon. The technique is further tested in a batch-mode processing, in which approximately 1.6% of all high-resolution Martian imagery is coregistered to a common baseline.

In this paper, we address the problem of fast and accurate extraction of points that correspond to the same location (named tie-points) from pairs of large-sized images. First, we conduct a theoretical analysis of the performance of the full-image matching approach, demonstrating its limitations when applied to large images. Subsequently, we introduce a novel technique to impose spatial constraints on the matching process without employing subsampled versions of the reference and the target image, which we name coupled image decomposition. This technique splits images into corresponding subimages through a process that is theoretically invariant to geometric transformations, additive noise, and global radiometric differences, as well as being robust to local changes. After presenting it, we demonstrate how coupled image decomposition can be used both for image registration and for automatic estimation of epipolar geometry. Finally, coupled image decomposition is tested on a data set consisting of several planetary images of different size, varying from less than one megapixel to several hundreds of megapixels. The reported experimental results, which includes comparison with full-image matching and state-of-the-art techniques, demonstrate the substantial computational cost reduction that can be achieved when matching large images through coupled decomposition, without at the same time compromising the overall matching accuracy.

In this paper, we deal with the problem of video concept detection to use the concept detection results toward a more effective concept-based video retrieval. The key novelties of this paper are as follows: 1) the use of spatio-temporal video slices (tomographs) in the same way that visual keyframes are typically used in video concept detection schemes. These spatio-temporal slices capture in a compact way motion patterns that are useful for detecting semantic concepts and are used for training a number of base detectors. The latter augment the set of keyframe-based base detectors that can be trained using different frame representations. 2) The introduction of a generic methodology, built upon a genetic algorithm, for controlling which subset of the available base detectors (consequently, which subset of the possible shot representations) should be combined for developing an optimal detector for each specific concept. This methodology is directly applicable to the learning of hundreds of diverse concepts, while diverging from the one-size-fits-all approach that is typically used in problems of this size. The proposed techniques are evaluated on the datasets of the 2011 and 2012 Semantic Indexing Task of TRECVID, each comprising several hundred hours of heterogeneous video clips and ground-truth annotations for tens of concepts that exhibit significant variation in terms of generality, complexity, and human participation. The experimental results manifest the merit of the proposed techniques.

In this work we deal with the problem of video concept detection, for the purpose of using the detection results towards more effective concept-based video retrieval. In order to handle this task, we propose using spatio-temporal video slices, called video tomographs, in the same way that visual keyframes are typically used in traditional keyframe-based video concept detection schemes. Video tomographs capture in a compact way motion patterns that are present in the video, and are used in this work for training a number of base detectors. The latter augment the set of keyframe-based base detectors that can be trained on different image representations. Combining the keyframe-based and tomograph-based detectors, improved concept detection accuracy can be achieved. The proposed approach is evaluated on a dataset that is extensive both in terms of video duration and concept variation. The experimental results manifest the merit of the proposed approach.

In this paper, a novel approach to evaluating video temporal decomposition algorithms is presented. The evaluation measures typically used to this end are nonlinear combinations of precision-recall or coverage-overflow, which are not metrics and additionally possess undesirable properties, such as nonsymmetricity. To alleviate these drawbacks, we introduce a novel unidimensional measure that is proven to be metric and satisfies a number of qualitative prerequisites that previous measures do not. This measure is named differential edit distance (DED), since it can be seen as a variation of the well-known edit distance. After defining DED, we further introduce an algorithm that computes it in less than cubic time. Finally, DED is extensively compared with state-of-the-art measures, namely, the harmonic means (F-score) of precision-recall and coverage-overflow. The experiments include comparisons of qualitative properties, the time required for optimizing the parameters of scene segmentation algorithms with the help of these measures, and a user study gauging the agreement of these measures with the users' assessment of the segmentation results. The results confirm that the proposed measure is a unidimensional metric that is effective in evaluating scene segmentation techniques and in helping to optimize their parameters.

In this paper, a novel approach to video temporal decomposition into semantic units, termed scenes, is presented. In contrast to previous temporal segmentation approaches that employ mostly low-level visual or audiovisual features, we introduce a technique that jointly exploits low-level and high-level features automatically extracted from the visual and the auditory channel. This technique is built upon the well-known method of the scene transition graph (STG), first by introducing a new STG approximation that features reduced computational cost, and then by extending the unimodal STG-based temporal segmentation technique to a method for multimodal scene segmentation. The latter exploits, among others, the results of a large number of TRECVID-type trained visual concept detectors and audio event detectors, and is based on a probabilistic merging process that combines multiple individual STGs while at the same time diminishing the need for selecting and fine-tuning several STG construction parameters. The proposed approach is evaluated on three test datasets, comprising TRECVID documentary films, movies, and news-related videos, respectively. The experimental results demonstrate the improved performance of the proposed approach in comparison to other unimodal and multimodal techniques of the relevant literature and highlight the contribution of high-level audiovisual features toward improved video segmentation to scenes.

In this work the problem of automatic decomposition of video into elementary semantic units, known in the literature as scenes, is addressed. Two multi-modal automatic scene segmentation techniques are proposed, both building upon the Scene Transition Graph (STG). In the first of the proposed approaches, speaker diarization results are used for introducing a post-processing step to the STG construction algorithm, with the objective of discarding scene boundaries erroneously identified according to visual-only dissimilarity. In the second approach, speaker diarization and additional audio analysis results are employed and a separate audio-based STG is constructed, in parallel to the original STG based on visual information. The two STGs are subsequently combined. Preliminary results from the application of the proposed techniques to broadcast videos reveal their improved performance over previous approaches.

In this paper a novel system for image replica detection is presented. The system uses color-based descriptors in order to extract robust features for image representation. These features are used for indexing the images in a database using an R-tree. When a query about whether a test image is a replica of an image in the database is submitted, the R-tree is traversed and a set of candidate images is retrieved. Then, in order to obtain a single result and at the same time reduce the number of decision errors the system is enhanced with linear discriminant analysis (LDA). The conducted experiments show that the proposed approach is very promising