This paper presents a time-causal analogue of the Gabor filter, as well as a both time-causal and time-recursive analogue of the Gabor transform, where the proposed time-causal representations obey both temporal scale covariance and a cascade property over temporal scales. The motivation behind these constructions is to enable theoretically well-founded time-frequency analysis over multiple temporal scales for real-time situations, or for physical or biological modelling situations, when the future cannot be accessed, and the non-causal access to the future in Gabor filtering is therefore not viable for a time-frequency analysis of the system. We develop a principled axiomatically determined theory for formulating these time-causal time-frequency representations, obtained by replacing the Gaussian kernel in the Gabor filtering with a time-causal kernel, referred to as the time-causal limit kernel, and which guarantees simplification properties from finer to coarser levels of scales in a time-causal situation, similar as the Gaussian kernel can be shown to guarantee over a non-causal temporal domain. We do also develop an axiomatically determined theory for implementing a discrete analogue of the proposed time-causal frequency analysis method on discrete data, based on first-order recursive filters coupled in cascade, with provable variation-diminishing properties that strongly suppress the influence from local perturbations and noise, and with specially chosen time constants to achieve self-similarity over scales and temporal scale covariance. In these ways, the proposed time-frequency representations guarantee well-founded treatment over multiple temporal scales, in situations when the characteristic scales in the signals, or physical or biological phenomena, to be analyzed may vary substantially, and additionally all steps in the time-frequency analysis have to be fully time-causal.

Spatial transformer networks (STNs) were designed to enable convolutional neural networks (CNNs) to learn invariance to image transformations. STNs were originally proposed to transform CNN feature maps as well as input images. This enables the use of more complex features when predicting transformation parameters. However, since STNs perform a purely spatial transformation, they do not, in the general case, have the ability to align the feature maps of a transformed image with those of its original. STNs are therefore unable to support invariance when transforming CNN feature maps. We present a simple proof for this and study the practical implications, showing that this inability is coupled with decreased classification accuracy. We therefore investigate alternative STN architectures that make use of complex features. We find that while deeper localization networks are difficult to train, localization networks that share parameters with the classification network remain stable as they grow deeper, which allows for higher classification accuracy on difficult datasets. Finally, we explore the interaction between localization network complexity and iterative image alignment.

The ability to handle large scale variations is crucial for many real world visual tasks. A straightforward approach for handling scale in a deep network is to process an image at several scales simultaneously in a set of scale channels. Scale invariance can then, in principle, be achieved by using weight sharing between the scale channels together with max or average pooling over the outputs from the scale channels. The ability of such scale channel networks to generalise to scales not present in the training set over significant scale ranges has, however, not previously been explored. We, therefore, present a theoretical analysis of invariance and covariance properties of scale channel networks and perform an experimental evaluation of the ability of different types of scale channel networks to generalise to previously unseen scales. We identify limitations of previous approaches and propose a new type of foveated scale channel architecture, where the scale channels process increasingly larger parts of the image with decreasing resolution. Our proposed FovMax and FovAvg networks perform almost identically over a scale range of 8, also when training on single scale training data, and do also give improvements in the small sample regime.

Scale selection methods based on local extrema over scale of scale-normalized derivatives have been primarily developed to be applied sparsely---at image points where the magnitude of a scale-normalized differential expression additionally assumes local extrema over the domain where the data are defined. This paper presents a methodology for performing dense scale selection, so that hypotheses about local characteristic scales in images, temporal signals, and video can be computed at every image point and every time moment. A critical problem when designing mechanisms for dense scale selection is that the scale at which scale-normalized differential entities assume local extrema over scale can be strongly dependent on the local order of the locally dominant differential structure. To address this problem, we propose a methodology where local extrema over scale are detected of a quasi quadrature measure involving scale-space derivatives up to order two and propose two independent mechanisms to reduce the phase dependency of the local scale estimates by (i) introducing a second layer of postsmoothing prior to the detection of local extrema over scale, and (ii) performing local phase compensation based on a model of the phase dependency of the local scale estimates depending on the relative strengths between first- and second-order differential structures. This general methodology is applied over three types of domains: (i) spatial images, (ii) temporal signals, and (iii) spatio-temporal video. Experiments demonstrate that the proposed methodology leads to intuitively reasonable results with local scale estimates that reflect variations in the characteristic scales of locally dominant structures over space and time.

Local image features or interest points provide compact and abstract representations of patterns in an image. We propose to extend the notion of spatial interest points into the spatio-temporal domain and show how the resulting features often reflect interesting events that can be used for a compact representation of video data as well as for its interpretation. To detect spatio-temporal events, we build on the idea of the Harris and Forstner interest point operators and detect local structures in space-time where the image values have significant local variations in both space and time. We then estimate the spatio-temporal extents of the detected events and compute their scale-invariant spatio-temporal descriptors. Using such descriptors, we classify events and construct video representation in terms of labeled space-time points. For the problem of human motion analysis, we illustrate how the proposed method allows for detection of walking people in scenes with occlusions and dynamic backgrounds.

This work presents two mechanisms for processing fingerprint images; shape-adapted smoothing based on second moment descriptors and automatic scale selection based on normalized derivatives. The shape adaptation procedure adapts the smoothing operation to the local ridge structures, which allows interrupted ridges to be joined without destroying essential singularities such as branching points and enforces continuity of their directional fields. The scale selection procedure estimates local ridge width and adapts the amount of smoothing to the local amount of noise. In addition, a ridgeness measure is defined, which reflects how well the local image structure agrees with a qualitative ridge model, and is used for spreading the results of shape adaptation into noisy areas. The combined approach makes it possible to resolve fine scale structures in clear areas while reducing the risk of enhancing noise in blurred or fragmented areas. The result is a reliable and adaptively detailed estimate of the ridge orientation field and ridge width, as well as a smoothed grey-level version of the input image. We propose that these general techniques should be of interest to developers of automatic fingerprint identification systems as well as in other applications of processing related types of imagery.

We present an automatic method for the classification of steel quality based on scale-space operations. The carbide distribution of microscopic specimen images is assessed by classifying according to so-called 'degree' and 'type' of the specimen. 'Degree' is represented by features extracted with automatic scale selection, and 'type' information is computed from second-moment descriptors. In combination with a morphological verification scheme, this pattern classifier shares large similarities with current manual techniques. Compared to previous work, the new classification scheme has several advantages: the significant scale of the carbide agglomeration is calculated explicitly, and the method is less sensitive to the variance of spatial connectivity than a morphological approach.

When extracting features from image data, the type of information that can be extracted may be strongly dependent on the scales at which the feature detectors are applied. This article presents a systematic methodology for addressing this problem. A mechanism is presented for automatic selection of scale levels when detecting one-dimensional features, such as edges and ridges. A novel concept of a scale-space edge is introduced, defined as a connected set of points in scale-space at which: (i) the gradient magnitude assumes a local maximum in the gradient direction, and (ii) a normalized measure of the strength of the edge response is locally maximal over scales. An important property of this definition is that it allows the scale levels to vary along the edge. Two specific measures of edge strength are analysed in detail. It is shown that by expressing these in terms of /spl gamma/-normalized derivatives, an immediate consequence of this definition is that fine scales are selected for sharp edges (so as to reduce the shape distortions due to scale-space smoothing), whereas coarse scales are selected for diffuse edges, such that an edge model constitutes a valid abstraction of the intensity profile across the edge. With slight modifications, this idea can be used for formulating a ridge detector with automatic scale selection, having the characteristic property that the selected scales on a scale-space ridge instead reflect the width of the ridge.

Summary form only given. We perceive objects in the world as meaningful entities only over certain ranges of scale. This fact that objects in the world appear in different ways depending on the scale of observation has important implications if one aims at describing them. It shows that the notion of scale is of utmost importance when processing unknown measurement data by automatic methods. In their seminal works, Witkin (1983) and Koenderink (1984) proposed to approach this problem by representing image structures at different scales in a so-called scale-space representation. Traditional scale-space theory building on this work, however, does not address the problem of how to select local appropriate scales for further analysis. After a brief review of the main ideas behind a scale-space representation, I describe a systematic methodology for generating hypotheses about interesting scale levels in image data based on a general principle stating that local extrema over scales of different combinations of normalized derivatives are likely candidates to correspond to interesting image structures. Specifically, I show how this idea can be used for formulating feature detectors which automatically adapt their local scales of processing to the local image structure. I show how the scale selection approach applies to various types of feature detection problems in early vision. In many computer vision applications, the poor performance of the low-level vision modules constitutes a major bottleneck.

This article deals with the problem of estimating deformations of brightness patterns using visual front-end operations. Estimating such deformations constitutes an important subtask in several computer vision problems relating to image correspondence and shape estimation. The following subjects are treated: The problem of decomposing affine flow fields into simpler components is analysed in detail. A canonical parametrization is presented based on singular value decomposition, which naturally separates the rotationally invariant components of the flow field from the rotationally variant ones. A novel mechanism is presented for automatic selection of scale levels when estimating local affine deformations. This mechanism is expressed within a multiscale framework where disparity estimates are computed in a hierarchical coarse-to-fine manner and corrected using iterative techniques. Then, deformation estimates are selected from the scales that minimize a certain normalized residual over scales. Finally, the descriptors so obtained serve as initial data for computing refined estimates of the local deformations.>

The subject of scale selection is essential to many aspects of multi-scale and multi-resolution processing of image data. This article shows how a general heuristic principle for scale selection can be applied to the problem of detecting and localizing junctions. In a first uncommitted processing step initial hypotheses about interesting scale levels (and regions of interest) are generated from scales where normalized differential invariants assume maxima over scales (and space). Then, based on this scale (and region) information, a more refined processing stage is invoked tuned to the task at hand. The resulting method is the first junction detector with automatic scale selection. Whereas this article deals with the specific problem of junction detection, the underlying ideas apply also to other types of differential feature detectors, such as blob detectors, edge detectors, and ridge defectors.>

The problem of scale in shape from texture is addressed. The need for two scale parameters is emphasized: a local scale, for describing the amount of smoothing used for suppressing noise and irrelevant details when computing primitive texture descriptors from image data; and an integration scale, for describing the size of the region in space over which the statistics of the local descriptors is accummulated. The proposed computational model is expressed completely in terms of different invariants defined from Gaussian derivatives at multiple scales in scale-space. The resulting texture description can be combined with various assumptions about surface texture to estimate local surface orientation. Two specific assumptions, weak isotropy and constant area, are explored in more detail. Results from experiments on real and synthetic reference data with known geometry that demonstrate the viability of the approach are presented.>

A manner in which a notion of effective scale can be introduced in a formal way is developed. For continuous signals, a scaling argument directly gives a natural unit for measuring scale-space lifetime in terms of the logarithm of the ordinary scale parameter. That approach is, however, not appropriate for discrete signals since an infinite lifetime would be assigned to structures existing in the original signal. It is shown how such an effective scale parameter can be defined to give consistent results for both discrete and continuous signals. The treatment is based on the assumption that the probability that a local extremum disappears during a short-scale interval should not vary with scale. As a tool for the analysis, estimates are given of how the density of local extrema can be expected to vary with scale in the scale-space representation of different random noise signals both in the continuous and discrete cases.>

We present a multi-scale representation of grey-level shape, called scale-space primal sketch, that makes explicit features in scale-space as well as the relations between features at different levels of scale. The representation gives a qualitative description of the image structure that allows for extraction of significant image structure — stable scales and regions of interest — in a solely bottom-up data-driven manner. Hence, it can be seen as preceding further processing, which can then be properly tuned. Experiments on real imagery demonstrate that the proposed theory gives perceptually intuitive results. Scale-space theory [11, 3] provides a well-founded framework for dealing with image structures, which naturally occur at different scales. According to this theory one can from a given signal generate a family of derived

The authors present: (1) a multi-scale representation of gray-level shape, called a scale-space primal sketch, which makes explicit both features in scale-space and the relations between features at different levels of scales; (2) a theory for extraction of significant image structure from this representation; and (3) applications to edge detection, histogram analysis and junction classification demonstrating how the proposed method can be used for guiding later stage processing. The representation gives a qualitative description of the image structure that allows for detection of stable scales and regions of interest in a solely bottom-up data-driven way. In other words, it generates coarse segmentation cues and can be hence seen as preceding further processing, which can then be properly tuned. The authors argue that once such information is available many other processing tasks can become much simpler. Experiments on real imagery demonstrate that the proposed theory gives perceptually intuitive results.>

A basic and extensive treatment of discrete aspects of the scale-space theory is presented. A genuinely discrete scale-space theory is developed and its connection to the continuous scale-space theory is explained. Special attention is given to discretization effects, which occur when results from the continuous scale-space theory are to be implemented computationally. The 1D problem is solved completely in an axiomatic manner. For the 2D problem, the author discusses how the 2D discrete scale space should be constructed. The main results are as follows: the proper way to apply the scale-space theory to discrete signals and discrete images is by discretization of the diffusion equation, not the convolution integral; the discrete scale space obtained in this way can be described by convolution with the kernel, which is the discrete analog of the Gaussian kernel, a scale-space implementation based on the sampled Gaussian kernel might lead to undesirable effects and computational problems, especially at fine levels of scale; the 1D discrete smoothing transformations can be characterized exactly and a complete catalogue is given; all finite support 1D discrete smoothing transformations arise from repeated averaging over two adjacent elements (the limit case of such an averaging process is described); and the symmetric 1D discrete smoothing kernels are nonnegative and unimodal, in both the spatial and the frequency domain.>