OBJECTIVE: Modeling variable-sized regions of interest (ROIs) in whole slide images using deep convolutional networks is a challenging task, as these networks typically require fixed-sized inputs that should contain sufficient structural and contextual information for classification. We propose a deep feature extraction framework that builds an ROI-level feature representation via weighted aggregation of the representations of variable numbers of fixed-sized patches sampled from nuclei-dense regions in breast histopathology images. METHODS: First, the initial patch-level feature representations are extracted from both fully-connected layer activations and pixel-level convolutional layer activations of a deep network, and the weights are obtained from the class predictions of the same network trained on patch samples. Then, the final patch-level feature representations are computed by concatenation of weighted instances of the extracted feature activations. Finally, the ROI-level representation is obtained by fusion of the patch-level representations by average pooling. RESULTS: Experiments using a well-characterized data set of 240 slides containing 437 ROIs marked by experienced pathologists with variable sizes and shapes result in an accuracy score of 72.65% in classifying ROIs into four diagnostic categories that cover the whole histologic spectrum. CONCLUSION: The results show that the proposed feature representations are superior to existing approaches and provide accuracies that are higher than the average accuracy of another set of pathologists. SIGNIFICANCE: The proposed generic representation that can be extracted from any type of deep convolutional architecture combines the patch appearance information captured by the network activations and the diagnostic relevance predicted by the class-specific scoring of patches for effective modeling of variable-sized ROIs.

Deep learning has achieved successful performance in representation learning and content-based retrieval of histopathology images. The commonly used setting in deep learning-based approaches is supervised training of deep neural networks for classification, and using the trained model to extract representations that are used for computing and ranking the distances between images. However, there are two remaining major challenges. First, supervised training of deep neural networks requires large amount of manually labeled data which is often limited in the medical field. Transfer learning has been used to overcome this challenge, but its success remained limited. Second, the clinical practice in histopathology necessitates working with regions of interest (ROI) of multiple diagnostic classes with arbitrary shapes and sizes. The typical solution to this problem is to aggregate the representations of fixed-sized patches cropped from these regions to obtain region-level representations. However, naive methods cannot sufficiently exploit the rich contextual information in the complex tissue structures. To tackle these two challenges, we propose a generic method that utilizes graph neural networks (GNN), combined with a self-supervised training method using a contrastive loss. GNN enables representing arbitrarily-shaped ROIs as graphs and encoding contextual information. Self-supervised contrastive learning improves quality of learned representations without requiring labeled data. The experiments using a challenging breast histopathology data set show that the proposed method achieves better performance than the state-of-the-art.

The challenging task of training object detectors for fine-grained classification faces additional difficulties when there are registration errors between the image data and the ground truth. We propose a weakly supervised learning methodology for the classification of 40 types of trees by using fixed-sized multispectral images with a class label but with no exact knowledge of the object location. Our approach consists of an end-to-end trainable convolutional neural network with separate branches for learning class-specific and location-specific scoring of image regions. Comparative experiments show that the proposed method simultaneously learns to detect and classify the objects of interest with high accuracy.

Fine-grained object recognition concerns the identification of the type of an object among a large number of closely related subcategories. Multisource data analysis that aims to leverage the complementary spectral, spatial, and structural information embedded in different sources is a promising direction toward solving the fine-grained recognition problem that involves low between-class variance, small training set sizes for rare classes, and class imbalance. However, the common assumption of coregistered sources may not hold at the pixel level for small objects of interest. We present a novel methodology that aims to simultaneously learn the alignment of multisource data and the classification model in a unified framework. The proposed method involves a multisource region attention network that computes per-source feature representations, assigns attention scores to candidate regions sampled around the expected object locations by using these representations, and classifies the objects by using an attention-driven multisource representation that combines the feature representations and the attention scores from all sources. All components of the model are realized using deep neural networks and are learned in an end-to-end fashion. Experiments using RGB, multispectral, and LiDAR elevation data for classification of street trees showed that our approach achieved 64.2% and 47.3% accuracies for the 18-class and 40-class settings, respectively, which correspond to 13% and 14.3% improvement relative to the commonly used feature concatenation approach from multiple sources.

Fine-grained object recognition that aims to identify the type of an object among a large number of subcategories is an emerging application with the increasing resolution that exposes new details in image data. Traditional fully supervised algorithms fail to handle this problem where there is low between-class variance and high within-class variance for the classes of interest with small sample sizes. We study an even more extreme scenario named zero-shot learning (ZSL) in which no training example exists for some of the classes. ZSL aims to build a recognition model for new unseen categories by relating them to seen classes that were previously learned. We establish this relation by learning a compatibility function between image features extracted via a convolutional neural network and auxiliary information that describes the semantics of the classes of interest by using training samples from the seen classes. Then, we show how knowledge transfer can be performed for the unseen classes by maximizing this function during inference. We introduce a new data set that contains 40 different types of street trees in 1-ft spatial resolution aerial data, and evaluate the performance of this model with manually annotated attributes, a natural language model, and a scientific taxonomy as auxiliary information. The experiments show that the proposed model achieves 14.3% recognition accuracy for the classes with no training examples, which is significantly better than a random guess accuracy of 6.3% for 16 test classes, and three other ZSL algorithms.

Digital pathology has entered a new era with the availability of whole slide scanners that create the high-resolution images of full biopsy slides. Consequently, the uncertainty regarding the correspondence between the image areas and the diagnostic labels assigned by pathologists at the slide level, and the need for identifying regions that belong to multiple classes with different clinical significances have emerged as two new challenges. However, generalizability of the state-of-the-art algorithms, whose accuracies were reported on carefully selected regions of interest (ROIs) for the binary benign versus cancer classification, to these multi-class learning and localization problems is currently unknown. This paper presents our potential solutions to these challenges by exploiting the viewing records of pathologists and their slide-level annotations in weakly supervised learning scenarios. First, we extract candidate ROIs from the logs of pathologists' image screenings based on different behaviors, such as zooming, panning, and fixation. Then, we model each slide with a bag of instances represented by the candidate ROIs and a set of class labels extracted from the pathology forms. Finally, we use four different multi-instance multi-label learning algorithms for both slide-level and ROI-level predictions of diagnostic categories in whole slide breast histopathology images. Slide-level evaluation using 5-class and 14-class settings showed average precision values up to 81% and 69%, respectively, under different weakly labeled learning scenarios. ROI-level predictions showed that the classifier could successfully perform multi-class localization and classification within whole slide images that were selected to include the full range of challenging diagnostic categories.

Detection of compound structures that are comprised of different arrangements of simpler primitive objects has been a challenging problem as commonly used bag-of-words models are limited in capturing spatial information. We have developed a generic method that considers the primitive objects as random variables, builds a contextual model of their arrangements using a Markov random field, and detects new instances of compound structures through automatic selection of subsets of candidate regions from a hierarchical segmentation by maximizing the likelihood of their individual appearances and relative spatial arrangements. In this paper, we extend the model to handle different types of primitive objects that come from multiple hierarchical segmentations. Results are shown for the detection of different types of housing estates in a WorldView-2 image.

Fusion of aerial optical and LiDAR data has been a popular problem in remote sensing as they carry complementary information for object detection. We describe a stratified method that involves separately thresholding the normalized digital surface model derived from LiDAR data and the normalized difference vegetation index derived from spectral bands to obtain candidate image parts that contain different object classes, and incorporates spectral and height data with spatial information in a graph cut framework to segment the rest of the image where such separation is not possible. Experiments using a benchmark data set show that the performance of the proposed method that uses small amount of supervision is compatible with the ones in the literature.

A challenging problem in remote sensing image analysis is the detection of heterogeneous compound structures such as different types of residential, industrial, and agricultural areas that are composed of spatial arrangements of simple primitive objects such as buildings and trees. We describe a generic method for the modeling and detection of compound structures that involve arrangements of an unknown number of primitives in large scenes. The modeling process starts with a single example structure, considers the primitive objects as random variables, builds a contextual model of their arrangements using a Markov random field, and learns the parameters of this model via sampling from the corresponding maximum entropy distribution. The detection task is formulated as the selection of multiple subsets of candidate regions from a hierarchical segmentation where each set of selected regions constitutes an instance of the example compound structure. The combinatorial selection problem is solved by the joint sampling of groups of regions by maximizing the likelihood of their individual appearances and relative spatial arrangements. Experiments using very high spatial resolution images show that the proposed method can effectively localize an unknown number of instances of different compound structures that cannot be detected by using spectral and shape features alone.

We propose an anomaly detection method that uses Gaussian mixture models for characterizing the scene background in hyperspectral images. First, the full spectrum is divided into several contiguous band groups for dimensionality reduction as well as for exploiting the peculiarities of different parts of the spectrum. Then, sparse spectral unmixing is performed for identifying significant endmembers in the scene, and hierarchical clustering in the abundance space is used for identifying pixel groups that contain these endmembers. Next, these pixel groups are used for initializing individual Gaussian mixture models that are estimated separately for each spectral band group. Finally, the Gaussian mixture models for all groups are fused for obtaining the final anomaly map for the scene. Comparative experiments showed that the proposed method performed better than two other density-based anomaly detectors, especially for small false positive rates, on an airborne hyperspectral data set.

Whole slide imaging technology enables pathologists to screen biopsy images and make a diagnosis in a digital form. This creates an opportunity to understand the screening patterns of expert pathologists and extract the patterns that lead to accurate and efficient diagnoses. For this purpose, we are taking the first step to interpret the recorded actions of world-class expert pathologists on a set of digitized breast biopsy images. We propose an algorithm to extract regions of interest from the logs of image screenings using zoom levels, time and the magnitude of panning motion. Using diagnostically relevant regions marked by experts, we use the visual bag-of-words model with texture and color features to describe these regions and train probabilistic classifiers to predict similar regions of interest in new whole slide images. The proposed algorithm gives promising results for detecting diagnostically relevant regions. We hope this attempt to predict the regions that attract pathologists' attention will provide the first step in a more comprehensive study to understand the diagnostic patterns in histopathology.

Increasing spectral and spatial resolution of new-generation remotely sensed images necessitate the joint use of both types of information for detection and classification tasks. This paper describes a new approach for detecting heterogeneous compound structures such as different types of residential, agricultural, commercial, and industrial areas that are comprised of spatial arrangements of primitive objects such as buildings, roads, and trees. The proposed approach uses Gaussian mixture models (GMMs), in which the individual Gaussian components model the spectral and shape characteristics of the individual primitives and an associated layout model is used to model their spatial arrangements. We propose a novel expectation-maximization (EM) algorithm that solves the detection problem using constrained optimization. The input is an example structure of interest that is used to estimate a reference GMM and construct spectral and spatial constraints. Then, the EM algorithm fits a new GMM to the target image data so that the pixels with high likelihoods of being similar to the Gaussian object models while satisfying the spatial layout constraints are identified without any requirement for region segmentation. Experiments using WorldView-2 images show that the proposed method can detect high-level structures that cannot be modeled using traditional techniques.

In this paper, our aim is to discover compound structures comprised of regions obtained from hierarchical segmentations of multiple spectral bands. A region adjacency graph is constructed by representing regions as vertices and connecting these vertices that are spatially close by edges. Then, dissimilarities between neighboring vertices are computed using statistical and structural features, and are assigned as edge weights. Finally, the compound structures are detected by extracting the connected components of the graph whose edges with relatively large weights are removed. Experiments using WorldView-2 images show that grouping of these vertices according to different criteria can extract high-level compound structures that cannot be obtained using traditional techniques.

Spectral information alone is often not sufficient to distinguish certain terrain classes such as permanent crops like orchards, vineyards, and olive groves from other types of vegetation. However, instances of these classes possess distinctive spatial structures that can be observable in detail in very high spatial resolution images. This paper proposes a novel unsupervised algorithm for the detection and segmentation of orchards. The detection step uses a texture model that is based on the idea that textures are made up of primitives (trees) appearing in a near-regular repetitive arrangement (planting patterns). The algorithm starts with the enhancement of potential tree locations by using multi-granularity isotropic filters. Then, the regularity of the planting patterns is quantified using projection profiles of the filter responses at multiple orientations. The result is a regularity score at each pixel for each granularity and orientation. Finally, the segmentation step iteratively merges neighboring pixels and regions belonging to similar planting patterns according to the similarities of their regularity scores and obtains the boundaries of individual orchards along with estimates of their granularities and orientations. Extensive experiments using Ikonos and QuickBird imagery as well as images taken from Google Earth show that the proposed algorithm provides good localization of the target objects even when no sharp boundaries exist in the image data.

We describe a new procedure that combines statistical and structural characteristics of simple primitive objects to discover compound structures in images. The statistical information that is modeled using spectral, shape, and position data of individual objects, and structural information that is modeled in terms of spatial alignments of neighboring object groups are encoded in a graph structure that contains the primitive objects at its vertices, and the edges connect the potentially related objects. Experiments using WorldView-2 data show that hierarchical clustering of these vertices can find high level compound structures that cannot be obtained using traditional techniques.

We present an unsupervised hierarchical segmentation algorithm for detection of complex heterogeneous image structures that are comprised of simpler homogeneous primitive objects. An initial segmentation step produces regions corresponding to primitive objects with uniform spectral content. Next, the transitions between neighboring regions are modeled and clustered. We assume that the clusters that are dense and large enough in this transition space can be considered as significant. Then, the neighboring regions belonging to the significant clusters are merged to obtain the next level in the hierarchy. The experiments show that the algorithm that iteratively clusters and merges region groups is able to segment high-level complex structures in a hierarchical manner.

We present solutions to two problems that prevent the effective use of population-based algorithms in clustering problems. The first solution presents a new representation for arbitrary covariance matrices that allows independent updating of individual parameters while retaining the validity of the matrix. The second solution involves an optimization formulation for finding correspondences between different parameter orderings of candidate solutions. The effectiveness of the proposed solutions are demonstrated on a novel clustering algorithm based on particle swarm optimization for the estimation of Gaussian mixture models.

The key step of a computer-assisted screening system that aims early diagnosis of cervical cancer is the accurate segmentation of cells. In this paper, we propose a two-phase approach to cell segmentation in Pap smear test images with the challenges of inconsistent staining, poor contrast, and overlapping cells. The first phase consists of segmenting an image by a non-parametric hierarchical segmentation algorithm that uses spectral and shape information as well as the gradient information. The second phase aims to obtain nucleus regions and cytoplasm areas by classifying the segments resulting from the first phase based on their spectral and shape features. Experiments using two data sets show that our method performs well for images containing both a single cell and many overlapping cells.

We describe an image representation that combines the representational power of graphs with the efficiency of the bag-of-words model. For each image in a data set, first, a graph is constructed from local patches of interest regions and their spatial arrangements. Then, each graph is represented with a histogram of sub graphs selected using a frequent subgraph mining algorithm in the whole data. Using the sub graphs as the visual words of the bag-of-words model and transforming of the graphs into a vector space using this model enables statistical classification of images using support vector machines. Experiments using images cut from a large satellite scene show the effectiveness of the proposed representation in classification of complex types of scenes into eight high-level semantic classes.

We propose an algorithm for automatic detection of buildings with complex shapes and roof structures in very high spatial resolution remotely sensed images. First, an initial oversegmentation is obtained. Then, candidate building regions are found using shadow and sun azimuth angle information. Finally, the building regions are selected by clustering the candidate regions using minimum spanning trees. The experiments on Ikonos scenes show that the algorithm is able to detect buildings with complex appearances and shapes.

Gaussian mixture models (GMM) are widely used for unsupervised classification applications in remote sensing. Expectation-Maximization (EM) is the standard algorithm employed to estimate the parameters of these models. However, such iterative optimization methods can easily get trapped into local maxima. Researchers use population-based stochastic search algorithms to obtain better estimates. We present a novel particle swarm optimization-based algorithm for maximum likelihood estimation of Gaussian mixture models. The proposed approach provides solutions for important problems in effective application of population-based algorithms to the clustering problem. We present a new parametrization for arbitrary covariance matrices that allows independent updating of individual parameters during the search process. We also describe an optimization formulation for identifying the correspondence relations between different parameter orderings of candidate solutions. Experiments on a hyperspectral image show better clustering results compared to the commonly used EM algorithm for estimating GMMs.

Spatial information plays a fundamental role in building high-level content models for supporting analysts' interpretations and automating geospatial intelligence. We describe a framework for modeling directional spatial relationships among objects and using this information for contextual classification and retrieval. The proposed model first identifies image areas that have a high degree of satisfaction of a spatial relation with respect to several reference objects. Then, this information is incorporated into the Bayesian decision rule as spatial priors for contextual classification. The model also supports dynamic queries by using directional relationships as spatial constraints to enable object detection based on the properties of individual objects as well as their spatial relationships to other objects. Comparative experiments using high-resolution satellite imagery illustrate the flexibility and effectiveness of the proposed framework in image mining with significant improvements in both classification and retrieval performance.

Automatic mapping and monitoring of agricultural landscapes using remotely sensed imagery has been an important research problem. This paper describes our work on developing automatic methods for the detection of target landscape features in very high spatial resolution images. The target objects of interest consist of linear strips of woody vegetation that include hedgerows and riparian vegetation that are important elements of the landscape ecology and biodiversity. The proposed framework exploits the spectral, textural, and shape properties of objects using hierarchical feature extraction and decision-making steps. First, a multifeature and multiscale strategy is used to be able to cover different characteristics of these objects in a wide range of landscapes. Discriminant functions trained on combinations of spectral and textural features are used to select the pixels that may belong to candidate objects. Then, a shape analysis step employs morphological top-hat transforms to locate the woody vegetation areas that fall within the width limits of an acceptable object, and a skeletonization and iterative least-squares fitting procedure quantifies the linearity of the objects using the uniformity of the estimated radii along the skeleton points. Extensive experiments using QuickBird imagery from three European Union member states show that the proposed algorithms provide good localization of the target objects in a wide range of landscapes with very different characteristics.

We present an unsupervised method for discovering compound image structures that are comprised of simpler primitive objects. An initial segmentation step produces image regions with homogeneous spectral content. Then, the segmentation is translated into a relational graph structure whose nodes correspond to the regions and the edges represent the relationships between these regions. We assume that the region objects that appear together frequently can be considered as strongly related. This relation is modeled using the transition frequencies between neighboring regions, and the significant relations are found as the modes of a probability distribution estimated using the features of these transitions. Experiments using an Ikonos image show that subgraphs found within the graph representing the whole image correspond to parts of different high-level compound structures.

Development of automatic methods for agricultural mapping and monitoring using remotely sensed imagery has been an important research problem. We describe algorithms that exploit the spectral, textural and object shape information using hierarchical feature extraction and decision making steps for automatic mapping of linear strips of woody vegetation in very high-resolution imagery. First, combinations of multispectral values and multi-scale Gabor and entropy texture features are used for training pixel level statistical classifiers for characterizing individual trees and tree groups with respect to their surroundings. Then, decisions based on object level texture features and morphological shape analysis provide the final detection of woody vegetation having a linear structure. Experiments on QuickBird imagery from different sites show that the proposed algorithms provide good localization of linear strips of woody vegetation in different landscapes.

We describe a new image representation using spatial relationship histograms that extend our earlier work on modeling image content using attributed relational graphs. These histograms are constructed by classifying the regions in an image, computing the topological and distance-based spatial relationships between these regions, and counting the number of times different groups of regions are observed in the image. We also describe a selection algorithm that produces very compact representations by identifying the distinguishing region groups that are frequently found in a particular class of scenes but rarely exist in others. Experiments using Ikonos scenes illustrate the effectiveness of the proposed representation in retrieval of images containing complex types of scenes such as dense and sparse urban areas.

The object-based analysis of remotely sensed imagery provides valuable spatial and structural information that is complementary to pixel-based spectral information in classification. In this paper, we present novel methods for automatic object detection in high-resolution images by combining spectral information with structural information exploited by using image segmentation. The proposed segmentation algorithm uses morphological operations applied to individual spectral bands using structuring elements in increasing sizes. These operations produce a set of connected components forming a hierarchy of segments for each band. A generic algorithm is designed to select meaningful segments that maximize a measure consisting of spectral homogeneity and neighborhood connectivity. Given the observation that different structures appear more clearly at different scales in different spectral bands, we describe a new algorithm for unsupervised grouping of candidate segments belonging to multiple hierarchical segmentations to find coherent sets of segments that correspond to actual objects. The segments are modeled by using their spectral and textural content, and the grouping problem is solved by using the probabilistic latent semantic analysis algorithm that builds object models by learning the object-conditional probability distributions. The automatic labeling of a segment is done by computing the similarity of its feature distribution to the distribution of the learned object models using the Kullback–Leibler divergence. The performances of the unsupervised segmentation and object detection algorithms are evaluated qualitatively and quantitatively using three different data sets with comparative experiments, and the results show that the proposed methods are able to automatically detect, group, and label segments belonging to the same object classes.

This paper describes our work on classification of outdoor scenes. First, images are partitioned into regions using one-class classification and patch-based clustering algorithms where one-class classifiers model the regions with relatively uniform color and texture properties, and clustering of patches aims to detect structures in the remaining regions. Next, the resulting regions are clustered to obtain a codebook of region types, and two models are constructed for scene representation: a "bag of individual regions" representation where each region is regarded separately, and a "bag of region pairs" representation where regions with particular spatial relationships are considered together. Given these representations, scene classification is done using Bayesian classifiers. We also propose a novel region selection algorithm that identifies region types that are frequently found in a particular class of scenes but rarely exist in other classes, and also consistently occur together in the same class of scenes. Experiments on the LabelMe data set showed that the proposed models significantly outperform a baseline global feature-based approach.

Spatial information is a crucial aspect of image understanding for modeling context as well as resolving the uncertainties caused by the ambiguities in low-level features. We describe intuitive, flexible and efficient methods for modeling pairwise directional spatial relationships and the ternary between relation using fuzzy mathematical morphology. First, a fuzzy landscape is constructed where each point is assigned a value that quantifies its relative position according to the reference object(s) and the type of the relationship. Then, the degree of satisfaction of this relation by a target object is computed by integrating the corresponding landscape over the support of the target region. Our models support sensitivity to visibility to handle areas that are partially enclosed by objects and are not visible from image points along the direction of interest. They can also cope with the cases where one object is significantly spatially extended relative to others. Experiments using synthetic and real images show that our models produce more intuitive results than other techniques.

We introduce an unsupervised method that combines both spectral and structural information for automatic object detection. First, a segmentation hierarchy is constructed by combining structural information extracted by morphological processing with spectral information summarized using principal components analysis. Then, segments that maximize a measure consisting of spectral homogeneity and neighborhood connectivity are selected as candidate structures for object detection. Given the observation that different structures appear more clearly in different principal components, we present an algorithm that is based on probabilistic Latent Semantic Analysis (PLSA) for grouping the candidate segments belonging to multiple segmentations and multiple principal components. The segments are modeled using their spectral content and the PLSA algorithm builds object models by learning the object-conditional probability distributions. Labeling of a segment is done by computing the similarity of its spectral distribution to the distribution of object models using Kullback-Leibler divergence. Experiments on two data sets show that our method is able to automatically detect, group, and label segments belonging to the same object classes.

We introduce a new method for modeling the spatial arrangements of geospatial objects. As opposed to the existing approaches that are based on classifying images using pixel level methods, we propose to use objects as textural primitives and exploit their spatial patterns. First, the primitives are detected using spectral and morphological processing. Then, these primitives form the nodes of a graph where the neighborhood information is obtained through Voronoi tessellation of the image scene. Next, this graph is clustered by thresholding its minimum spanning tree. Finally, the resulting clusters are classified as regular or irregular by examining the distributions of the angles between neighboring nodes. Experiments using Ikonos images show that the application of the proposed model where buildings are used as the primitives and building groups are automatically classified as organized or unorganized can extract valuable information about urban development.

The papers in this special section are devoted to pattern recognition in remote sensing applications.

A challenging problem in image content extraction and classification is building a system that automatically learns high-level semantic interpretations of images. We describe a Bayesian framework for a visual grammar that aims to reduce the gap between low-level features and high-level user semantics. Our approach includes modeling image pixels using automatic fusion of their spectral, textural, and other ancillary attributes; segmentation of image regions using an iterative split-and-merge algorithm; and representing scenes by decomposing them into prototype regions and modeling the interactions between these regions in terms of their spatial relationships. Naive Bayes classifiers are used in the learning of models for region segmentation and classification using positive and negative examples for user-defined semantic land cover labels. The system also automatically learns representative region groups that can distinguish different scenes and builds visual grammar models. Experiments using Landsat scenes show that the visual grammar enables creation of high-level classes that cannot be modeled by individual pixels or regions. Furthermore, learning of the classifiers requires only a few training examples.

Date of Conference: August 22 - 25, 2004

The VisiMine project aims to provide infrastructure that would enable the analysis of large databases containing satellite images. Our work addresses two issues. One is the extraction of information that enables reduction of the data from multi-spectral images into a number of features. Second is the organization and selection of the features that would allow flexible and scalable discovery of the knowledge from the databases of remotely sensed images. The VisiMine architecture distinguishes between three types of feature vectors: pixel, region and tile. One of the challenges in information retrieval is the proper choice of the set of features that are the best suited for a data mining task. The VisiMine system enables extraction of a large number of features that describe textural and spectral properties of satellite information, in addition to the analysis of image information, the system can perform data fusion of image properties with auxiliary data such as DEM. Tilton et al. (2002) presented the results of the information retrieval experiments with the Hierarchical Segmentation (HSEG) algorithm that produces a hierarchical set of image segmentations. The results presented showed that the use of HSEG features improves the precision and recall of similarity searches. However, for different types of land cover, different combinations of HSEG segmentation levels and textural features provided the best results. Image analysis applications often require different levels of image segmentation detail as well as the use of different mixes of spectral, textural and shape features combined together with auxiliary information. Furthermore, a particular application may require different features and different levels of image segmentation detail depending on how the image objects are being analyzed. Thus, an automatic selection of feature sets would be very useful for satellite image analysis. In this paper, we present algorithms that allow for automatic selection of features for region and tile similarity searches. The relevance feedback technique allows for selective choices to be made in the region(s) of interest for which a good subset of features may be found in real time. The preliminary results of the experiments with LANDSAT data show improvements in both precision and recall over previously used methods.

We describe a system for content-based retrieval and classification of multispectral images. Our system models images on pixel, region and scene levels. To reduce the gap between low-level features and highlevel user semantics, and to support complex query scenarios that consist of many regions with different feature characteristics, we propose a probabilistic visual grammar that includes automatic identification of region prototypes and modeling of their spatial relationships. A Bayesian framework is used to automatically classify scenes based on these models. We demonstrate our system with query scenarios that cannot be expressed by traditional region or scene level approaches but where the visual grammar provides accurate classifications and effective retrieval.

We describe usage of DEM data in the VisiMine system for data mining and statistical analysis of the collections of remotely sensed images.

We describe VisiMine, a system for data mining and statistical analysis of large collections of remotely sensed images.

Similarity between images in image retrieval is measured by computing distances between feature vectors. This paper presents a probabilistic approach and describes two likelihood-based similarity measures for image retrieval. Popular distance measures like the Euclidean distance implicitly assign more more weighting to features with large ranges than those with small ranges. First, we discuss the effects of five feature normalization methods on retrieval performance. Then, we show that the probabilistic methods perform significantly better than geometric approaches like the nearest neighbor rule with city-block or Euclidean distances. They are also more robust to normalization effects and using better models for the features improves the retrieval results compared to making only general assumptions. Experiments on a database of approximately 10000 images show that studying the feature distributions are important and this information should be used in designing feature normalization methods and similarity measures.

Content-based image retrieval systems use low-level features like color and texture for image representation. Given these representations as feature vectors, similarity between images is measured by computing distances in the feature space. Unfortunately, these low-level features cannot always capture the high-level concept of similarity in human perception. Relevance feedback tries to improve the performance by allowing iterative retrievals where the feedback information from the user is incorporated into the database search. We present a weighted distance approach where the weights are the ratios of standard deviations of the feature values both for the whole database and also among the images selected as relevant by the user. The feedback is used for both independent and incremental updating of the weights and these weights are used to iteratively refine the effects of different features in the database search. Retrieval performance is evaluated using average precision and progress that are computed on a database of approximately 10,000 images and an average performance improvement of 19% is obtained after the first iteration.

This contest involved the running and evaluation of computer vision and pattern recognition techniques on different data sets with known groundwidth. The contest included three areas; binary shape recognition, symbol recognition and image flow estimation. A package was made available for each area. Each package contained either real images with manual groundtruth or programs to generate data sets of ideal as well as noisy images with known groundtruth. They also contained programs to evaluate the results of an algorithm according to the given groundtruth. These evaluation criteria included the generation of confusion matrices, computation of the misdetection and false alarm rates and other performance measures suitable for the problems. The paper summarizes the data generation for each area and experimental results for a total of six participating algorithms.

Image retrieval algorithms are generally based on the assumption that visually similar images are located close to each other in the feature space. Since the feature vectors usually exist in a very high dimensional space, a parametric characterization of their distribution is impossible, so non-parametric approaches, like the k-nearest neighbor search, are used for retrieval. This paper introduces a graph-theoretic approach for image retrieval by formulating the database search as a graph clustering problem by using a constraint that retrieved images should be consistent with each other (close in the feature space) as well as being individually similar (close) to the query image. The experiments that compare retrieval precision with and without clustering showed an average precision of 0.76 after clustering, which is an improvement by 5.56% over the average precision before clustering.

This paper introduces a size invariant method to recognize two-dimensional binary shapes using the recursive erosion transform. Using recursive morphology with multiple structuring elements, the method takes constant time per pixel regardless of the scale of the shape model, and also works on noisy images without requiring noise removal. Results from experiments on 100 noisy images show the methodology is able to detect every shape model's scale and position with 13 false alarms and five misdetections out of 254 total translated and scaled models.