Image classification for real-world applications often involves complicated data distributions such as fine-grained and long-tailed. To address the two challenging issues simultaneously, we propose a new regularization technique that yields an adversarial loss to strengthen the model learning. Specifically, for each training batch, we construct an adaptive batch prediction (ABP) matrix and establish its corresponding adaptive batch confusion norm (ABC-Norm). The ABP matrix is a composition of two parts, including an adaptive component to class-wise encode the imbalanced data distribution, and the other component to batch-wise assess the softmax predictions. The ABC-Norm leads to a norm-based regularization loss, which can be theoretically shown to be an upper bound for an objective function closely related to rank minimization. By coupling with the conventional cross-entropy loss, the ABC-Norm regularization could introduce adaptive classification confusion and thus trigger adversarial learning to improve the effectiveness of model learning. Different from most of state-of-the-art techniques in solving either fine-grained or long-tailed problems, our method is characterized with its simple and efficient design, and most distinctively, provides a unified solution. In the experiments, we compare ABC-Norm with relevant techniques and demonstrate its efficacy on several benchmark datasets, including (CUB-LT, iNaturalist2018); (CUB, CAR, AIR); and (ImageNet-LT), which respectively correspond to the real-world, fine-grained, and long-tailed scenarios.

This paper proposes the use of complex variables to represent votes in the Hough transform for circle detection. Replacing the positive numbers classically used in the parameter space of the Hough transforms by complex numbers allows cancellation effects when adding up the votes. Cancellation and the computation of shape likelihood via a complex number's magnitude square lead to more robust solutions than the “classic” algorithms, as shown by computational experiments on synthetic and real datasets. We note a resemblance to methods used in quantum theory.

We propose a data structure that captures global geometric properties in images: Histogram of Mirror Symmetry Coefficients. We compute such a coefficient for every pair of pixels, and group them in a 6-dimensional histogram. By marginalizing the HMSC in various ways, we develop algorithms for a range of applications: detection of nearly-circular cells, location of the main axis of reflection symmetry, detection of cell-division in movies of developing embryos, detection of worm-tips and indirect cell-counting via supervised classification. Our approach generalizes a series of histogram-related methods, and the proposed algorithms perform with state-of-the-art accuracy.

We describe a statistical approach to the problem of estimating the times of cell-division cycles in time-lapse movies of early mouse embryos. Our method is based on the likelihoods for cells of certain radii ranges to be in each frame — without actually locating or counting the cells. Computing the likelihoods consists of a voting scheme where votes come form quadruples of points in a way similar to the first step of the Randomized Hough Transform for ellipse detection. To locate divisions, we search for points of abrupt change in the matrix of likelihoods (built for all frames), and pick the two optimal division points using a dynamic programming algorithm. Our results for the first and second cell division cycles differ less than two frames from the medians of the annotated times in a database of 100 annotated videos, and outperform two other recent methods in the same set.

We present an ellipse finding and fitting algorithm that uses points and tangents, rather than just points, as the basic unit of information. These units are analyzed in a hierarchy: points with tangents are paired into triangles in the first layer and pairs of triangles in the second layer vote for ellipse centers. The remaining parameters are estimated via robust linear algebra: eigen-decomposition and iteratively reweighed least squares. Our method outperforms the state-of-the-art approach in synthetic images and microscopic images of cells.

We present DevStaR, an automated computer vision and machine learning system that provides rapid, accurate, and quantitative measurements of C. elegans embryonic viability in high-throughput (HTP) applications. A leading genetic model organism for the study of animal development and behavior, C. elegans is particularly amenable to HTP functional genomic analysis due to its small size and ease of cultivation, but the lack of efficient and quantitative methods to score phenotypes has become a major bottleneck. DevStaR addresses this challenge using a novel hierarchical object recognition machine that rapidly segments, classifies, and counts animals at each developmental stage in images of mixed-stage populations of C. elegans. Here, we describe the algorithmic design of the DevStaR system and demonstrate its performance in scoring image data acquired in HTP screens.

We present a hierarchical principle for object recognition and its application to automatically classify developmental stages of C. elegans animals from a population of mixed stages. The object recognition machine consists of four hierarchical layers, each composed of units upon which evaluation functions output a label score, followed by a grouping mechanism that resolves ambiguities in the score by imposing local consistency constraints. Each layer then outputs groups of units, from which the units of the next layer are derived. Using this hierarchical principle, the machine builds up successively more sophisticated representations of the objects to be classified. The algorithm segments large and small objects, decomposes objects into parts, extracts features from these parts, and classifies them by SVM. We are using this system to analyze phenotypic data from C. elegans high-throughput genetic screens, and our system overcomes a previous bottleneck in image analysis by achieving near real-time scoring of image data. The system is in current use in a functioning C. elegans laboratory and has processed over two hundred thousand images for lab users.

The representation of objects in images as tree structures is of great interest to vision, as they can represent articulated objects such as people as well as other structured objects like arteries in human bodies, roads, circuit board patterns, etc. Tree structures are often related to the symmetry axis representation of shapes, which captures their local symmetries. Algorithms have been introduced to detect (i) open contours in images in quadratic time (ii) closed contours in images in cubic time, and (iii) tree structures from contours in quadratic time. The algorithms are based on dynamic programming and single source shortest path algorithms. However, in this paper, we show that the problem of finding tree structures in images in a principled manner is a much harder problem. We argue that the optimization problem of finding tree structures in images is essentially equivalent to a variant of the Steiner tree problem, which is NP-hard. Nevertheless, an approximate polynomial-time algorithm for this problem exists: we apply a fast implementation of the Goemans-Williamson approximate algorithm to the problem of finding a tree representation after an image is transformed by a local symmetry mapping. Examples of extracting tree structures from images illustrate the idea and applicability of the approximate method

Representing shapes in a compact and informative form is a significant problem for vision systems that must recognize or classify objects. We describe a compact representation model for two-dimensional (2D) shapes by investigating their self-similarities and constructing their shape axis trees (SA-trees). Our approach can be formulated as a variational one (or, equivalently, as MAP estimation of a Markov random field). We start with a 2D shape, its boundary contour, and two different parameterizations for the contour (one parameterization is oriented counterclockwise and the other clockwise). To measure its self-similarity, the two parameterizations are matched to derive the best set of one-to-one point-to-point correspondences along the contour. The cost functional used in the matching may vary and is determined by the adopted self-similarity criteria, e.g., cocircularity, distance variation, parallelism, and region homogeneity. The loci of middle points of the pairing contour points yield the shape axis and they can be grouped into a unique free tree structure, the SA-tree. By implicitly encoding the (local and global) shape information into an SA-tree, a variety of vision tasks, e.g., shape recognition, comparison, and retrieval, can be performed in a more robust and efficient way via various tree-based algorithms. A dynamic programming algorithm gives the optimal solution in O(N/sup 1/), where N is the size of the contour.

We propose an on-line handwriting recognition approach that integrates local bottom-up constructs with a global top-down measure into a modular recognition engine. The bottom-up process uses local point features for hypothesizing character segmentations and the top-down part performs shape matching for evaluating the segmentations. The shape comparison, called Fisher segmental matching, is based on Fisher's linear discriminant analysis. Along with an efficient ligature modeling, the segmentations and their matching scores are integrated into a recognition engine termed Hypotheses Propagation Network, which runs a variant of the topological sort algorithm of graph search. The result is a system that is more shape-oriented less dependent on local and temporal features, modular in construction and has a rich range of opportunities for further extensions. Our system currently performs at 95% of recognition rate on cursive scripts with a 460 word dictionary.

We present a framework for 2D shape contour (silhouette) comparison that can account for stretchings, occlusions and region information. Topological changes due to the original 3D scenarios and articulations are also addressed. To compare the degree of similarity between any two shapes, our approach is to represent each shape contour with a free tree structure derived from a shape axis (SA) model, which we have recently proposed. We then use a tree matching scheme to find the best approximate match and the matching cost. To deal with articulations, stretchings and occlusions, three local tree matching operations, merge, cut, and merge-and-cut, are introduced to yield optimally approximate matches, which can accommodate not only one-to-one but many-to-many mappings. The optimization process gives guaranteed globally optimal match efficiently. Experimental results on a variety of shape contours are provided.

In human perception, convex surfaces have a strong tendency to be perceived as the "figure". Convexity has a stronger influence on figural organization than other global shape properties, such as symmetry (G. Kanizsa, 1979), and yet there has been very little work on convexity properties in computer vision. We present a model for figure/ground segregation which exhibits a preference for convex regions as the figure (i.e., the foreground). The model also shows a preference for smaller regions to be selected as figures, which is also known to hold for human visual perception (K. Koffka, 1935). The model is based on the machinery of Markov random fields/random walks/diffusion processes, so that the global shape properties are obtained via local and stochastic computations. Experimental results demonstrate that our model performs well on ambiguous figure/ground displays which were not captured before. In particular in ambiguous displays where neither region is strictly convex, the model shows preference to the "more convex" region, thus offering a continuous measure of convexity in agreement with human perception.

We have been developing a stochastic model for figure-ground separation. The model selects/constructs the foreground with preference for figures with “more convex” shapes. When these models are applied to illusory figures they yield perceptually accurate selection of figure and background. The approach is based on an “entropy” measure of a region diffusion Markov model from a set of local figure/ground hypothesis. The contour boundaries are implicitly represented, via the thresholding of the diffusion result. What optimal properties do the illusory contours satisfies? We show that the entropy criteria selects contours such as to minimize a Taylor series of the even derivatives with respect to the length of the contour. The coefficients are positive and they get exponentially smaller as the derivatives increase. The zeroth order term suggest that small length contours are preferred, the second order terms suggests that curvature-like term is minimized (with less strength compared to the zero order one), and higher order derivatives give additional contour smoothness constraints.

All illusory surface figures yield a perception of a surface occluding another one or the background. Occluded surfaces yield completion, a phenomena known as amodal completions. It is intriguing that for some images illusory surfaces are perceived, but not for other images. Also, illusory surfaces may have portions occlude. We aim to understand these phenomena. Our approach detects intensity edges and junctions. From the junctions we seek to find an optimal image organization, i.e., multiple ordered surfaces with the ordering accounting for salience. The most salient being the figure, while the other surfaces are classified as background. A decision of which surface is the visible one (on rep) is made locally, at each pixel, allowing the salient surface (figure) to have portions occluded, i.e., with amodal completions. We account for a variety of imagery not explained before.

We propose a method for segmenting gray-value images. By segmentation, we mean a map from the set of pixels to a small set of levels such that each connected component of the set of pixels with the same level forms a relatively large and "meaningful" region. The method finds a set of levels with associated gray values by first finding junctions in the image and then seeking a minimum set of threshold values that preserves the junctions. Then it finds a segmentation map that maps each pixel to the level with the closest gray value to the pixel data, within a smoothness constraint. For a convex smoothing penalty, we show the global optimal solution for an energy function that fits the data can be obtained in a polynomial time, by a novel use of the maximum-flow algorithm. Our approach is in contrast to a view in computer vision where segmentation is driven by intensity, gradient, usually not yielding closed boundaries.

We present a new type of "snake" in which the dimensionality of the shapes is scaled appropriately for the resolution of the images in which the shapes are embedded. We define shapes as an ordered list of control points and compute the principal components of the shapes in a prior training set. Our energy function is based upon the Mahalanobis distance of a given shape from the mean shape and on the Mahalanobis distance of the image attributes from image attribute values extracted from the training set. We show that the derivative of this energy function with respect to the modal weights is reduced as the image resolution is reduced, and that the derivative of the energy scales with the variance associated with each mode. We exploit this property to determine the subset of the modes which are relevant at a particular level of image resolution thereby reducing the dimensionality of the shapes. We implement a coarse-to-fine search procedure in the image and shape domains simultaneously, and demonstrate this procedure on the identification of anatomic structures in Computed Tomography images.

Representing shapes is a significant problem for vision systems that must recognize or classify objects. We derive a representation for a given shape by investigating its self-similarities, and constructing its shape axis (SA) and shape axis tree (SA-tree). We start with a shape, its boundary contour, and two different parameterizations for the contour. To measure its self-similarity we consider matching pairs of points (and their tangents) along the boundary contour, i.e., matching the two parameterizations. The matching, of self-similarity criteria may vary, e.g., co-circularity, parallelism, distance, region homogeneity. The loci of middle points of the pairing contour points are the shape axis and they can be grouped into a unique tree graph, the SA-tree. The shape axis for the co-circularity criteria is compared to the symmetry axis. An interpretation in terms of object parts is also presented.

We introduce a method for segmenting a shape from an image and simultaneously determining its symmetry axis. The symmetry is used to help the segmentation and in turn the segmentation determines the symmetry. The problem is formulated as one of minimizing a goodness of fitness function and Dijkstra's algorithm is used to find the global minimum of the cost function. The results are illustrated on real images.

Junctions are important features for image analysis and form a critical aspect of image understanding tasks such as object recognition. We present a unified approach to detecting, classifying, and reconstructing junctions in images. Our main contribution is a modeling of the junction which is complex enough to handle all these issues and yet simple enough to admit an effective dynamic programming solution. We use a template deformation framework along with a gradient criterium to detect radial partitions of the template. We use the minimum description length principle to obtain the optimal number of partitions that best describes the junction. The Kona detector presented by Parida et al. (1997) is an implementation of this model. We demonstrate the stability and robustness of the detector by analyzing its behavior in the presence of noise, using synthetic/controlled apparatus. We also present a qualitative study of its behavior on real images.

We study the problem of how to detect "interesting objects" appeared in a given image, I. Our approach is to treat it as a function approximation problem based on an over-redundant basis, and also account for occlusions, where the basis superposition principle is no longer valid. Since the basis (a library of image templates) is over-redundant, there are infinitely many ways to decompose I. We are motivated to select a sparse/compact representation of I, and to account for occlusions and noise. We then study a greedy and iterative "weighted L/sup p/ Matching Pursuit" strategy, with O<p<1. We use an L/sup p/ result to compute a solution, select the best template, at each stage of the pursuit.

A common factor in all illusory contour figures is the perception of a surface occluding part of a background. In our previous work, we have shown we could diffuse a proper set of junction hypothesis (what is salient or background) to obtain a surface where their boundaries represented illusory contours. Amodal completions emerge at the overlapping surfaces. We address the problem of selecting the best image organization (set of hypothesis). We propose an optimization criteria based on a coherence measure between pairs of junctions (correlation between the diffusion of each pair). A statistical physics approach to select the best organization is applied. The experiments suggest that despite the large number of possible organizations our approach may take only a few steps (in organization space) to select the best one.

This paper addresses the problem of recognizing articulated and deformable objects. In particular we are interested in human arm and leg articulations. Our approach is a Bayesian-Information integration of shape similarity and snakes, and naturally combines top-down and bottom-up algorithms. The bottom-up method extracts edges, then constructs snakes (or contours) by grouping edge elements and feeds the shape analysis. The top-down one uses shape analysis, by comparing the object model with the extracted snakes, to guide/prune the search for other snakes. The optimizations are based on Dijkstra algorithm and further pruning of this algorithm is obtained by "integration by parts". Our approach is general enough to handle three dimensional objects, but our focus here is on two dimensional contours.

The problem of segmenting an image into separate regions and tracking them over time is one of the most significant problems in vision. Terzopoulos et al. (1987) proposed an approach to detect the contour regions of complex shapes, assuming a user selected initial contour not very far from the desired solution. We propose to further explore the information provided by the user's selected points and apply an optimal method to detect contours which allows a segmentation of the image. The method is based on dynamic programming (DP), and applies to a wide variety of shapes. It is exact and not iterative. We also consider a multiscale approach capable of speeding up the algorithm by a factor of 20, although at the expense of losing the guaranteed optimality characteristic. The problem of tracking and matching these contours is addressed. For tracking, the final contour obtained at one frame is sampled and used as initial points for the next frame. Then, the same DP process is applied. For matching, a novel strategy is proposed where the solution is a smooth displacement field in which unmatched regions are allowed while cross vectors are not. The algorithm is again based on DP and the optimal solution is guaranteed. We have demonstrated the algorithms on natural objects in a large spectrum of applications, including interactive segmentation and automatic tracking of the regions of interest in medical images.>

The problem of obtaining high-quality images and image features at different scales is discussed. Emphasis is placed on a Markov image model described in a lattice by a Gaussian noise and a local regularization term depending upon the image discontinuities. An approximate self-similar property of the model is derived by a process of averaging over half of the lattice sites. This is known as the renormalization group approach. Two multiscale pyramid structures, one of images and the other of image discontinuities, are obtained. The coarse images generated by the proposed method are smooth and show good contrast. The approach, when applied in the reverse order, is capable of enlarging images, while accounting for the original image features. The quality of the derived pyramid is demonstrated by using it to help solve a segmentation problem.>

An optimal method to match elastic contours is proposed. The method is based on dynamic programming and on some ideas borrowed from stereo matching, where occlusions correspond to contour regions that have no match after the deformation. The matching between two circles of different sizes and centers is shown. One may consider this experiment as the matching between two images of an inflating balloon. The algorithm produces uniform matches along the perimeter despite the large difference in size of the two contours. The matching of the inner and outer walls of the heart left ventricle is also shown. This matching can give diagnostic information for the heart wall. Because of the restricted search space, the DP matching algorithm is very fast, matching two contours of 500 points in a few milliseconds.>

A Markov random field (MRF) formulation for the problem of optical flow computation is studied. An adaptive window matching scheme is used to obtain a good measure of the correlation between the two images. A confidence measure for each match is also used. Thus, the input to the system is the adaptive correlation and the corresponding confidence. The MRF model is then used to estimate the velocity field and the velocity discontinuities. The problem of occlusions is addressed, and a relationship between occlusions and motion discontinuities is established.>

Deterministic approximations to Markov random field (MRF) models are derived. One of the models is shown to give in a natural way the graduated nonconvexity (GNC) algorithm proposed by A. Blake and A. Zisserman (1987). This model can be applied to smooth a field preserving its discontinuities. A class of more complex models is then proposed in order to deal with a variety of vision problems. All the theoretical results are obtained in the framework of statistical mechanics and mean field techniques. A parallel, iterative algorithm to solve the deterministic equations of the two models is presented, together with some experiments on synthetic and real images.>

An attempt is made to unify several approaches to image segmentation in early vision under a common framework. The energy function, or Markov random field, formalism is very attractive since it enables the assumptions used to be explicitly stated in the energy functions, and it can be extended to deal with many other problems in vision. It is shown that the specified discrete formulations for the energy function are closely related to the continuous formulation. When the mean field theory approach is used, several previous attempts to solve these energy functions are effectively equivalent. By varying the parameters of the energy functions, one can obtain a class of solutions and several nonlinear diffusion approaches to image segmentation, but it can be applied equally well to image or surface reconstruction (where the data are sparse).>

It is assumed that a major goal of the early vision modules and their integration is to deliver a cartoon of the discontinuities in the scene and to label them in terms of their physical origin. The output of each of the vision modules is noisy, possibly sparse, and sometimes not unique. The authors suggest the use of a coupled Markov random field (MRF) at the output of each module (image cues)-stereo, motion, color, and texture-to achieve two goals: first, to counteract the noise and fill in sparse data, and secondly, to integrate the image within each MRF to find the module discontinuities and align them with the intensity edges. The authors outline a theory of how to label the discontinuities in terms of depth, orientation, albedo, illumination, and specular discontinuities. They present labeling results using a simple linear classifier operating on the output of the MRF associated with each vision module and coupled to the image data. The classifier has been trained on a small set of a mixture of synthetic and real data.>