Data and code working together is fundamental to machine learning (ML), but the context around datasets and interactions between datasets and code are in general captured only rudimentarily. Context such as how the dataset was prepared and created, what source data were used, what code was used in processing, how the dataset evolved, and where it has been used and reused can provide much insight, but this information is often poorly documented. That is unfortunate since it makes datasets into black-boxes with potentially hidden characteristics that have downstream consequences. We argue that making dataset preparation more accessible and dataset usage easier to record and document would have significant benefits for the ML community: it would allow for greater diversity in datasets by inviting modification to published sources, simplify use of alternative datasets and, in doing so, make results more transparent and robust, while allowing for all contributions to be adequately credited. We present a platform, Renku, designed to support and encourage such sustainable development and use of data, datasets, and code, and we demonstrate its benefits through a few illustrative projects which span the spectrum from dataset creation to dataset consumption and showcasing.

An important step towards explaining deep image classifiers lies in the identification of image regions that contribute to individual class scores in the model's output. However, doing this accurately is a difficult task due to the black-box nature of such networks. Most existing approaches find such attributions either using activations and gradients or by repeatedly perturbing the input. We instead address this challenge by training a second deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. These attributions are provided in the form of masks that only show the classifier-relevant parts of an image, masking out the rest. Our approach produces sharper and more boundary-precise masks when compared to the saliency maps generated by other methods. Moreover, unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks in a single forward pass. This makes the proposed method very efficient during inference. We show that our attributions are superior to established methods both visually and quantitatively with respect to the PASCAL VOC-2007 and Microsoft COCO-2014 datasets.

Defining and identifying duplicate records in a dataset is a challenging task which grows more complex when the modeled entities themselves are hard to delineate. In the geospatial domain, it may not be clear where a mountain, stream, or valley ends and begins, a problem carried over when such entities are catalogued in gazetteers. In this paper, we take two gazetteers, GeoNames and SwissNames3D, and perform matching – identifying records in each that are about the same entity – across a sample of natural feature records. We first perform rule-based matching, establishing competitive results, then apply machine learning using Random Forests, a method well-suited to the matching task. We report on the performance of a wider array of matching features than has been previously studied, including domain-specific ones such as feature type, land cover class, and elevation. Our results show an increase in performance using machine learning over rules, with a notable performance gain from considering feature types, but negligible gains from other specialized matching features. We argue that future work in this area should strive to be more reproducible and report results on a realistic testing pipeline including candidate selection, feature extraction, and classification.

Classification of remotely sensed images into land cover or land use is highly dependent on geographical information at least at two levels. First, land cover classes are observed in a spatially smooth domain separated by sharp region boundaries. Second, land classes and observation scale are also tightly intertwined: they tend to be consistent within areas of homogeneous appearance, or regions, in the sense that all pixels within a roof should be classified as roof, independently on the spatial support used for the classification. In this paper, we follow these two observations and encode them as priors in an energy minimization framework based on conditional random fields (CRFs), where classification results obtained at pixel and region levels are probabilistically fused. The aim is to enforce the final maps to be consistent not only in their own spatial supports (pixel and region) but also across supports, i.e., by getting the predictions on the pixel lattice and on the set of regions to agree. To this end, we define an energy function with three terms: 1) a data term for the individual elements in each support (support-specific nodes); 2) spatial regularization terms in a neighborhood for each of the supports (support-specific edges); and 3) a regularization term between individual pixels and the region containing each of them (intersupports edges). We utilize these priors in a unified energy minimization problem that can be optimized by standard solvers. The proposed 2LCRF model consists of a CRF defined over a bipartite graph, i.e., two interconnected layers within a single graph accounting for interlattice connections. 2LCRF is tested on two very high-resolution data sets involving submetric satellite and subdecimeter aerial data. In all cases, 2LCRF improves the result obtained by the independent base model (either random forests or convolutional neural networks) and by standard CRF models enforcing smoothness in the spatial domain.

In many computer vision tasks, we expect a particular behavior of the output with respect to rotations of the input image. If this relationship is explicitly encoded, instead of treated as any other variation, the complexity of the problem is decreased, leading to a reduction in the size of the required model. In this paper, we propose the Rotation Equivariant Vector Field Networks (RotEqNet), a Convolutional Neural Network (CNN) architecture encoding rotation equivariance, invariance and covariance. Each convolutional filter is applied at multiple orientations and returns a vector field representing magnitude and angle of the highest scoring orientation at every spatial location. We develop a modified convolution operator relying on this representation to obtain deep architectures. We test RotEqNet on several problems requiring different responses with respect to the inputs' rotation: image classification, biomedical image segmentation, orientation estimation and patch matching. In all cases, we show that RotEqNet offers extremely compact models in terms of number of parameters and provides results in line to those of networks orders of magnitude larger.

Illegal wildlife poaching poses one severe threat to the environment. Measures to stem poaching have only been with limited success, mainly due to efforts required to keep track of wildlife stock and animal tracking. Recent developments in remote sensing have led to low-cost Unmanned Aerial Vehicles (UAVs), facilitating quick and repeated image acquisitions over vast areas. In parallel, progress in object detection in computer vision yielded unprecedented performance improvements, partially attributable to algorithms like Convolutional Neural Networks (CNNs). We present an object detection method tailored to detect large animals in UAV images. We achieve a substantial increase in precision over a robust state-of-the-art model on a dataset acquired over the Kuzikus wildlife reserve park in Namibia. Furthermore, our model processes data at over 72 images per second, as opposed 3 for the baseline, allowing for real-time applications.

We aim to jointly estimate height and semantically label monocular aerial images. These two tasks are traditionally addressed separately in remote sensing, despite their strong correlation. Therefore, a model learning both height and classes jointly seems advantageous and so, we propose a multitask Convolutional Neural Network (CNN) architecture with two losses: one performing semantic labeling, and another predicting normalized Digital Surface Model (nDSM) from the pixel values. Since the nDSM/height information is used only in the second loss, there is no need to have a nDSM map at test time, and the model can estimate height automatically on new images. We test our proposed method on a set of sub-decimeter resolution images and show that our model equals the performances of two separate models, but at the cost of a single one.

Semantic labeling (or pixel-level land-cover classification) in ultrahigh-resolution imagery (<;10 cm) requires statistical models able to learn high-level concepts from spatial data, with large appearance variations. Convolutional neural networks (CNNs) achieve this goal by learning discriminatively a hierarchy of representations of increasing abstraction. In this paper, we present a CNN-based system relying on a downsample-then-upsample architecture. Specifically, it first learns a rough spatial map of high-level representations by means of convolutions and then learns to upsample them back to the original resolution by deconvolutions. By doing so, the CNN learns to densely label every pixel at the original resolution of the image. This results in many advantages, including: 1) the state-of-the-art numerical accuracy; 2) the improved geometric accuracy of predictions; and 3) high efficiency at inference time. We test the proposed system on the Vaihingen and Potsdam subdecimeter resolution data sets, involving the semantic labeling of aerial images of 9- and 5-cm resolution, respectively. These data sets are composed by many large and fully annotated tiles, allowing an unbiased evaluation of models making use of spatial information. We do so by comparing two standard CNN architectures with the proposed one: standard patch classification, prediction of local label patches by employing only convolutions, and full patch labeling by employing deconvolutions. All the systems compare favorably or outperform a state-of-the-art baseline relying on superpixels and powerful appearance descriptors. The proposed full patch labeling CNN outperforms these models by a large margin, also showing a very appealing inference time.

We present a method for learning discriminative filters using a shallow Convolutional Neural Network (CNN). We encode rotation invariance directly in the model by tying the weights of groups of filters to several rotated versions of the canonical filter in the group. These filters can be used to extract rotation invariant features well-suited for image classification. We test this learning procedure on a texture classification benchmark, where the orientations of the training images differ from those of the test images. We obtain results comparable to the state-of-the-art. Compared to standard shallow CNNs, the proposed method obtains higher classification performance while reducing by an order of magnitude the number of parameters to be learned.

Traditional approaches to structured semantic segmentation employ appearance-based classifiers to provide a class-likelihood at each spatial location and then post-process it with Markov Random Fields (MRF) to enforce label smoothness and structure in the output space. The spatial support for such techniques is usually a patch of pixels, which makes the prediction over-smoothed because the borders of objects are not explicitly taken into account. This is further exacerbated by MRF post-processing employing the standard Potts model, which tends to further over-smooth predictions at boundaries. In this paper, we propose a different but related approach: we optimize an energy function finding the optimal combination of small ground truth (GT) tiles from training data over predictions at test time, effectively solving a puzzle. We optimize over a first configuration given by a Convolutional Neural Network (CNN) output.

Land cover / land use classification of remotely sensed images is inherently geographical. The use of spatial information, accounting for neighborhood relationship and spatial smoothness of geographical objects, made its proofs in countless occasions and, especially when considering very high resolution images, methods ignoring spatial context do not perform well. In this paper, we propose a hybrid dual-layer conditional random field model that enforces spatial smoothness and consistency between the pixel and region-based maps. We formulate these intuitions as a standard energy minimization problem, and we show that finding a joint solution over both output spaces leads to strong improvements in the numerical and visual senses.

Land-cover and land-use semantic labeling in centimeter resolution imagery (ultra-high resolution) is mostly performed by supervised classification of informative descriptors extracted from spatially coherent but small objects (e.g. superpixels or patches). In this paper, we propose an extension of this reasoning by proposing a class-specific, multi-scale and bottom-up object proposal strategy to perform semantic labeling. Specifically, we rely on a fully trainable boundary (edge) detector, allowing us to extract class-specific object-proposals. Such proposals enable training rich appearance and object models as well as enhanced spatial reasoning. We evaluate the proposed strategy on the Vaihingen dataset with promising results.

In this paper, we study the problem of feature extraction for knowledge transfer between multiple remotely sensed images in the context of land-cover classification. Several factors such as illumination, atmospheric, and ground conditions cause radiometric differences between images of similar scenes acquired on different geographical areas or over the same scene but at different time instants. Accordingly, a change in the probability distributions of the classes is observed. The purpose of this work is to statistically align in the feature space an image of interest that still has to be classified (the target image) to another image whose ground truth is already available (the source image). Following a specifically designed feature extraction step applied to both images, we show that classifiers trained on the source image can successfully predict the classes of the target image despite the shift that has occurred. In this context, we analyze a recently proposed domain adaptation method aiming at reducing the distance between domains, Transfer Component Analysis, and assess the potential of its unsupervised and semisupervised implementations. In particular, with a dedicated study of its key additional objectives, namely the alignment of the projection with the labels and the preservation of the local data structures, we demonstrate the advantages of Semisupervised Transfer Component Analysis. We compare this approach with other both linear and kernel-based feature extraction techniques. Experiments on multi- and hyperspectral acquisitions show remarkable cross- image classification performances for the considered strategy, thus confirming its suitability when applied to remotely sensed images.

The re-use of available labeled samples to classify newly acquired data is a hot topic in pattern analysis and machine learning. Classification algorithms developed with data from one domain cannot be directly used in another related domain, unless the data representation or the classifier have been adapted to the new data distribution. This is crucial in satellite/airborne image analysis: when confronted to domain shifts issued from changes in acquisition or illumination conditions, image classifiers tend to become inaccurate. In this paper, we introduce a method to align data manifolds that represent the same land cover classes, but have undergone spectral distortions. The proposed method relies on a semi-supervised manifold alignment technique and relaxes the requirement of labeled data in all domains by exploiting centrality measures over graphs to match the manifolds. Experiments on multispectral pixel classification at very high spatial resolution show the potential of the method.

Including spatial information is a key step for successful remote sensing image classification. In particular, when dealing with high spatial resolution, if local variability is strongly reduced by spatial filtering, the classification performance results are boosted. In this paper, we consider the triple objective of designing a spatial/spectral classifier, which is compact (uses as few features as possible), discriminative (enhances class separation), and robust (works well in small sample situations). We achieve this triple objective by discovering the relevant features in the (possibly infinite) space of spatial filters by optimizing a margin-maximization criterion. Instead of imposing a filter bank with predefined filter types and parameters, we let the model figure out which set of filters is optimal for class separation. To do so, we randomly generate spatial filter banks and use an active-set criterion to rank the candidate features according to their benefits to margin maximization (and, thus, to generalization) if added to the model. Experiments on multispectral very high spatial resolution (VHR) and hyperspectral VHR data show that the proposed algorithm, which is sparse and linear, finds discriminative features and achieves at least the same performances as models using a large filter bank defined in advance by prior knowledge.

We introduce a method for manifold alignment of different modalities (or domains) of remote sensing images. The problem is recurrent when a set of multitemporal, multisource, multisensor, and multiangular images is available. In these situations, images should ideally be spatially coregistered, corrected, and compensated for differences in the image domains. Such procedures require massive interaction of the user, involve tuning of many parameters and heuristics, and are usually applied separately. Changes of sensors and acquisition conditions translate into shifts, twists, warps, and foldings of the (typically nonlinear) manifolds where images lie. The proposed semisupervised manifold alignment (SS-MA) method aligns the images working directly on their manifolds and is thus not restricted to images of the same resolutions, either spectral or spatial. SS-MA pulls close together samples of the same class while pushing those of different classes apart. At the same time, it preserves the geometry of each manifold along the transformation. The method builds a linear invertible transformation to a latent space where all images are alike and reduces to solving a generalized eigenproblem of moderate size. We study the performance of SS-MA in toy examples and in real multiangular, multitemporal, and multisource image classification problems. The method performs well for strong deformations and leads to accurate classification for all domains. A MATLAB implementation of the proposed method is provided at http://isp. uv.es/code/ssma.htm.

The access to many sources of satellite information is nowadays a reality. However, few methods allow to consider simultaneously data coming from different sensors, due to the differences in numbers of bands, spatial resolution and changes in the acquisition conditions. In this paper, we propose a methodology to align the data structures (also called manifolds) of two (or more) images and to exploit them simultaneously in a joint latent space. The method being invertible, it also have the interesting property to allow to project the image pixels from one sensor to another, thus allowing to synthesize the bands of one sensor using the pixels of the other through the projection learned. Experiments using QuickBird and World-View II images show the properties of the method and open new opportunities for multisensor remote-sensing.

Inclusion of spatial information is known to be beneficial to the classification of hyperspectral images. However, given the high dimensionality of the data, it is difficult to know before hand which are the bands to filter or what are the filters to be applied. In this paper, we propose an active set algorithm based on a l1 support vector machine that explores the (possibily infinite) space of spatial filters and retrieves automatically the filters that maximize class separation. Experiments on hyperspectral imagery confirms the power of the method, that reaches state of the art performance with small feature sets generated automatically and without prior knowledge.

The analysis of multi-modal and multi-sensor images is nowadays of paramount importance for Earth Observation (EO) applications. There exist a variety of methods that aim at fusing the different sources of information to obtain a compact representation of such datasets. However, for change detection existing methods are often unable to deal with heterogeneous image sources and very few consider possible nonlinearities in the data. Additionally, the availability of labeled information is very limited in change detection applications. For these reasons, we present the use of a semi-supervised kernel-based feature extraction technique. It incorporates a manifold regularization accounting for the geometric distribution and jointly addressing the small sample problem. An exhaustive example using Landsat 5 data illustrates the potential of the method for multi-sensor change detection.

This paper presents an application of the kernel principal component analysis aiming at spectrally aligning optical images before the application of change detection techniques. The approach relies on the extraction of nonlinear features from a selected subset of pixels representing unchanged areas in the bi-temporal images. Both images are then projected into the new space defined by the eigenvectors associated to largest variance (eigenvalues). In the transformed space, unchanged pixels are mapped next to each other, thus reducing within-class variance. The difference image that results from subtracting the projected datasets is likely to provide a more suitable representation for detecting changes. A subset of two Landsat TM scenes validates the proposed approach. The new representation is studied thanks to the change vector analysis and to the support vector domain description.

In this letter, an unsupervised kernel-based approach to change detection is introduced. Nonlinear clustering is utilized to partition in two a selected subset of pixels representing both changed and unchanged areas. Once the optimal clustering is obtained, the learned representatives of each group are exploited to assign all the pixels composing the multitemporal scenes to the two classes of interest. Two approaches based on different assumptions of the difference image are proposed. The first accounts for the difference image in the original space, while the second defines a mapping describing the difference image directly in feature spaces. To optimize the parameters of the kernels, a novel unsupervised cost function is proposed. An evidence of the correctness, stability, and superiority of the proposed solution is provided through the analysis of two challenging change-detection problems.

In this paper, we address the problem of semi-automatic definition of training sets for the classification of remotely sensed images. We propose two approaches based on active learning aiming at removing both the proximal (low diversity) and the dense (low exploration during iterations) sampling redundancies. The first is encountered when several samples carrying similar spectral information are selected by the algorithm, while the second occurs when the heuristic is unable to explore undiscovered parts of the feature space during iterations. For this purpose, kernel$k$-means is used to cluster a set of uncertain candidates in the same space spanned by the kernel function defined in the SVM classification step. Two heuristics are proposed to maximize the speed of convergence to high classification accuracies: The first is based on binary hierarchical partitioning of the set of selected uncertain samples, while the second extends this approach by considering memory in the selection and thus dynamically adapts to the problem throughout the iterations. Experiments on both VHR and hyperspectral imagery confirm fast convergence of the algorithm, that outperforms state-of-the-art sampling schemes.

In this paper we propose an unsupervised approach to change detection by computing the difference image directly in the feature spaces. The resulting difference kernel, that is a combination of kernels computed on the coregistered and radiometrically matched input images, is used to train a nonlinear partitioning algorithm. In order to apply the kernel k-means, issues related to the initialization and to the tuning of parameters (e.g. the Gaussian RBF bandwidth) are considered. To validate the proposed unsupervised algorithm, two multitemporal VHR remote sensing images are used.

A novel approach to active sampling is proposed for the semi automatic selection of training patterns in a given pool of candidates. In the method proposed, each candidate is ranked with a double criterion: first, informativeness of the candidate is assessed using the Support Vector Machine (SVM) real valued decision function. Then, diverse sampling is ensured by considering the relative position of the candidate in the SVM feature space. Such position is evaluated by using partitioning of the feature space via nonlinear clustering. Among all candidates belonging to the same cluster, the winner is the pixel minimizing the weighted combination of the distances from both the SVM hyperplane and the cluster center. This way, the proposed approach provides a way to i) account for and minimize the redundancy of the sampled pixels and ii) maximize the speed of convergence to an optimal classification accuracy. In order to discover clusters and to evaluate the distance between the cluster center and the samples, the kernel k-means algorithm is used in a hierarchical way. By its kernel nature, the algorithm partitions data in the SVM induced space, thus ensuring coherent diversity with respect to the linear model therein. The reliability of the proposed heuristic is evaluated on a QuickBird VHR image of the city of Zurich, Switzerland.