The DEtection TRansformer (DETR) opened new possibilities for object detection by modeling it as a translation task: converting image features into object-level representations. Previous works typically add expensive modules to DETR to perform Multi-Object Tracking (MOT), resulting in more complicated architectures. We instead show how DETR can be turned into a MOT model by employing an instance-level contrastive loss, a revised sampling strategy and a lightweight assignment method. Our training scheme learns object appearances while preserving detection capabilities and with little overhead. Its performance surpasses the previous state-of-the-art by +2.6 mMOTA on the challenging BDD100K dataset and is comparable to existing transformer-based methods on the MOT17 dataset.

During training, supervised object detection tries to correctly match the predicted bounding boxes and associated classification scores to the ground truth. This is essential to determine which predictions are to be pushed towards which solutions, or to be discarded. Popular matching strategies include matching to the closest ground truth box (mostly used in combination with anchors), or matching via the Hungarian algorithm (mostly used in anchor free methods). Each of these strategies comes with its own properties, underlying losses, and heuristics. We show how Unbalanced Optimal Transport unifies these different approaches and opens a whole continuum of methods in between. This allows for a finer selection of the desired properties. Experimentally, we show that training an object detection model with Unbalanced Optimal Transport is able to reach the state-of-the-art both in terms of Average Precision and Average Recall as well as to provide a faster initial convergence. The approach is well suited for GPU implementation, which proves to be an advantage for large-scale models.

With the advent of deep learning, many dense prediction tasks, i.e., tasks that produce pixel-level predictions, have seen significant performance improvements. The typical approach is to learn these tasks in isolation, that is, a separate neural network is trained for each individual task. Yet, recent multi-task learning (MTL) techniques have shown promising results w.r.t. performance, computations and/or memory footprint, by jointly tackling multiple tasks through a learned shared representation. In this survey, we provide a well-rounded view on state-of-the-art deep learning approaches for MTL in computer vision, explicitly emphasizing on dense prediction tasks. Our contributions concern the following. First, we consider MTL from a network architecture point-of-view. We include an extensive overview and discuss the advantages/disadvantages of recent popular MTL models. Second, we examine various optimization methods to tackle the joint learning of multiple tasks. We summarize the qualitative elements of these works and explore their commonalities and differences. Finally, we provide an extensive experimental evaluation across a variety of dense prediction benchmarks to examine the pros and cons of the different methods, including both architectural and optimization based strategies.

While ground truth depth data remains hard to obtain, self-supervised monocular depth estimation methods enjoy growing attention. Much research in this area aims at improving loss functions or network architectures. Most works, however, do not leverage self-supervision to its full potential. They stick to the standard closed world train-test pipeline, assuming the network parameters to be fixed after the training is finished. Such an assumption does not allow to adapt to new scenes, whereas with self-supervision this becomes possible without extra annotations.In this paper, we propose a novel self-supervised Continuous Monocular Depth Adaptation method (CoMoDA), which adapts the pretrained model on a test video on the fly. As opposed to existing test-time refinement methods that use isolated frame triplets, we opt for continuous adaptation, making use of the previous experience from the same scene. We additionally augment the proposed procedure with the experience from the distant past, preventing the model from overfitting and thus forgetting already learnt information.We demonstrate that our method can be used for both intra- and cross-dataset adaptation. By adapting the model from train to test set of the Eigen split of KITTI, we achieve state-of-the-art depth estimation performance and surpass all existing methods using standard architectures. We also show that our method runs 15 times faster than existing test-time refinement methods. The code is available at https://github.com/Yevkuzn/CoMoDA.

Current state-of-the-art instance segmentation methods are not suited for real-time applications like autonomous driving, which require fast execution times at high accuracy. Although the currently dominant proposal-based methods have high accuracy, they are slow and generate masks at a fixed and low resolution. Proposal-free methods, by contrast, can generate masks at high resolution and are often faster, but fail to reach the same accuracy as the proposal-based methods. In this work we propose a new clustering loss function for proposal-free instance segmentation. The loss function pulls the spatial embeddings of pixels belonging to the same instance together and jointly learns an instance-specific clustering bandwidth, maximizing the intersection-over-union of the resulting instance mask. When combined with a fast architecture, the network can perform instance segmentation in real-time while maintaining a high accuracy. We evaluate our method on the challenging Cityscapes benchmark and achieve top results (5% improvement over Mask R-CNN) at more than 10 fps on 2MP images.

Modern cars are incorporating an increasing number of driver assist features, among which automatic lane keeping. The latter allows the car to properly position itself within the road lanes, which is also crucial for any subsequent lane departure or trajectory planning decision in fully autonomous cars. Traditional lane detection methods rely on a combination of highly-specialized, hand-crafted features and heuristics, usually followed by post-processing techniques, that are computationally expensive and prone to scalability due to road scene variations. More recent approaches leverage deep learning models, trained for pixel-wise lane segmentation, even when no markings are present in the image due to their big receptive field. Despite their advantages, these methods are limited to detecting a pre-defined, fixed number of lanes, e.g. ego-lanes, and can not cope with lane changes. In this paper, we go beyond the aforementioned limitations and propose to cast the lane detection problem as an instance segmentation problem - in which each lane forms its own instance - that can be trained end-to-end. To parametrize the segmented lane instances before fitting the lane, we further propose to apply a learned perspective transformation, conditioned on the image, in contrast to a fixed ”bird's-eye view” transformation. By doing so, we ensure a lane fitting which is robust against road plane changes, unlike existing approaches that rely on a fixed, predefined transformation. In summary, we propose a fast lane detection algorithm, running at 50 fps, which can handle a variable number of lanes and cope with lane changes. We verify our method on the tuSimple dataset and achieve competitive results.

Research on visual material recognition has traditionally been based on texture analysis. Whereas older work has focused on uncluttered scenes, more recent contributions allowed for material recognition 'in the wild'. Quite some objects have untextured surfaces, however. Especially man-made examples are legion. The most obvious cue to use in such cases would be reflection information. Yet, methods to that effect are lacking. It is clear that more than an estimate of a scalar albedo is needed, and a more complete reflectance model has to be derived. Rather than using an extensive lab setup, we propose a system that only requires the 3D shape of objects and a regular, commercial camera to capture their appearance in a single image, to perform material classification under unknown illumination. To this end, we rely on a Gaussian Process Latent Variable Model (GPLVM) with a discriminative prior to learn a low-dimensional manifold suitable for material classification of reflectance maps, i.e. from a 2D image of a singlematerial sphere under natural illumination. We evaluated our method based on experiments generated from synthetic and real-life data. Although recognizing materials without texture (or object recognition) is not a trivial problem, our method achieves about 75% recognition accuracy, about 27% higher than human performance.

The problem of estimating a full BRDF from partial observations has already been studied using either parametric or non-parametric approaches. The goal in each case is to best match this sparse set of input measurements. In this paper we address the problem of inferring higher order reflectance information starting from the minimal input of a single BRDF slice. We begin from the prototypical case of a homogeneous sphere, lit by a head-on light source, which only holds information about less than 0.001% of the whole BRDF domain. We propose a novel method to infer the higher dimensional properties of the material's BRDF, based on the statistical distribution of known material characteristics observed in real-life samples. We evaluated our method based on a large set of experiments generated from real-world BRDFs and newly measured materials. Although inferring higher dimensional BRDFs from such modest training is not a trivial problem, our method performs better than state-of-the-art parametric, semi-parametric and non-parametric approaches. Finally, we discuss interesting applications on material re-lighting, and flash-based photography.

In this work, we investigate the use of simple flash-based photography to capture an object's 3D shape and reflectance characteristics at the same time. The presented method is based on the principles of Structure from Motion (SfM) and Photometric Stereo (PS), yet, we make sure not to use more than readily-available consumer equipment, like a camera with flash. Starting from a SfM-generated mesh, we apply PS to refine both geometry and reflectance, where the latter is expressed in terms of data-driven Bidirectional Reflectance Distribution Function (BRDF) representations. We also introduce a novel approach to infer complete BRDFs starting from the sparsely sampled data-driven reflectance information captured with this setup. Our approach is experimentally validated by modeling several challenging objects, both synthetic and real.

3D shape models are often put together from several partial reconstructions. There are good algorithms available now to perform the necessary, precise registration automatically, but only after the partial reconstructions have been brought into approximate positions. This paper proposes a technique to do precisely that. Automatic 'crude registration' is illustrated for Euclidean and affine transformations between parts. The technique is based on the extraction and invariant characterisation of bitangent curve pairs. These can be found and matched efficiently. Registration of these curves yields the crude initialisation that more precise registration algorithms like ICP or mutual information maximisation can start from.

The media and communications providers share an increasing interest in 3D models of people, objects, and scenes. The paper focuses on features that 3D acquisition systems ought to have in order to optimally serve these markets, where emphasis is on realistic visualisation. It is argued that 3D acquisition techniques developed for traditional applications such as visual inspection aren't necessarily the best option. Techniques should be developed that are dedicated to visualisation-specific requirements. This is exemplified with two systems that have been developed recently. One takes uncalibrated video data as input from which it generates a 3D model. A second system projects a grid of lines and gets dense 3D from a single image. This system needs some calibration, but the corresponding procedure is extremely simple. It can also be used to capture detailed, 3D scene dynamics.

A method is presented that extracts the 3D shape of objects, together with the surface texture. Both shape and texture are obtained from a single image. The underlying principle is based on an active technique. A high resolution, pattern is projected onto the object and the deformations as observed by a single camera yield the 3rd dimension. Furthermore, the surface texture is extracted from the same image. Because the whole procedure is based on a single image, a frame-by-frame reconstruction of a video taken with the pattern projected throughout, yields 3D shape dynamics. The paper sketches the complete system but focuses on the problem of texture extraction.

Cartography and other applications of remote sensing have led to an increased interest in the (semi-)automatic interpretation of structures in aerial images of urban and suburban areas. Although these areas are particularly challenging because of their complexity, the degree of regularity in such man-made structures also helps to tackle the problems. The paper presents the iterated application of the Hough transform as a means to exploit such regularities. It shows how such 'Cascaded Hough Transform' (or CHT for short) yields straight lines, vanishing points, and vanishing lines. It also illustrates how the latter assist in improving the precision of the former. The examples are based on real aerial photographs.

When using the original slope-intercept parameterisation for the Hough transform, the resulting parameter space actually corresponds to the dual space. Indeed, lines are transformed into points, and for every point there is also a corresponding line. This paper presents a way of exploiting this special property, by the introduction of the cascaded Hough transform (CHT). This allows to look for the overall structure in an image, such as lines intersecting in a point or intersection points lying on a line. An interesting example is the detection of vanishing points and vanishing lines.

A method is presented that extracts the 3D shape of objects, together with their surface texture.Both shape and texture are obtained from a single image.The paper sketches the complete system but focuses on the problem of texture extraction.The underlying principle is based on an active technique.A high resolution pattern is projected onto the object and the deformations as observed by a single camera yield the 3rd dimension.The surface texture is extracted from the same image by literally reading between the lines that are used for the shape extraction.This is done using a combination of interpolation and non-linear diffusion techniques.Because the whole procedure is based on a single image, a frame-byframe reconstruction of a video taken with the pattern projected throughout, yields 3D shape dynamics. A paradigm shift in 3DMost methods for three-dimensional shape extraction go via the explicit calculation of the distance between the sensor and the object.The 3D shape of the surface then follows from the variation of this distance.This is what 3D acquisition devices for reverse engineering, shape inspection, and mobile robotics wotild typically do.More recent applications, however, focus on visualization, such as virtual and augmented reality, 3D on the Internet, special effects in movies, etc.These kind of applications come with different priorities: l Extracting the absolute scale of objects usually is not crucial.Objects.will be shown at completely different scales anyway.

Over recent years, symmetry research has shifted from the detection of affinely to perspectively skewed mirror symmetry. Also, links between invariance research and symmetry-specific geometric constraints have been established. The paper aims to contribute to both strands. Several sets of symmetry specific invariants are derived, that can be used in different situations, depending on the a priori assumptions made. It is also argued that all the results directly apply to the case of perspectively skewed point symmetry.

An active 3D acquisition system is presented that projects a simple pattern of squares on a scene and views it from a different angle. This paper describes how the observed pattern can be extracted from the image data. The underlying algorithm automatically detects the lines and crossings of the projected pattern in the image. Experiments show that the algorithm is robust and provides accurate three-dimensional (3D) reconstructions. Its one-shot operation principle enables the system to retrieve the shape of moving objects.

An active 3D acquisition system is presented that is based on the projection a simple pattern of squares. Through the observation of the projected pattern, and assuming pseudo-orthographic projection, the 3D shape of the scene can be retrieved up to a single parameter, that can be determined through a particularly simple calibration. An algorithm has been developed to automatically detect the lines and crossings of the projected pattern in the image. Experiments on a variety of scenes show that the algorithm is robust and provides accurate 3D reconstructions. Its one-shot operation enables the system to retrieve the shape of moving objects.

A method is described to recover the three-dimensional affine structure of a scene consisting of at least five points identified in two perspective views with a relative object-camera translation in between. When compared to the results for arbitrary stereo views, a more detailed reconstruction is possible using less information. The method presented only assumes that the two images are obtained by identical cameras, but no knowledge about the intrinsic parameters of the camera(s) or about the performed translation is assumed. By the same method, affine 3D reconstruction from a single view can be achieved for parallel structures. In that case, four points suffice for affine reconstruction.

The paper describes a systematic approach to the analysis of symmetries in planar shapes. The shapes can be observed from arbitrary viewpoints. Hence, the results encompass the case of skewed symmetries. In fact, skewing is assumed to arise from perspective distortions, whereas most of the literature restricts the analysis to affine skewing. The point of departure is the identification of structures that symmetries keep fixed in an image. These define subgroups of the projectivities, which in turn assume simpler invariants than their projective counterparts. These invariants allow both simpler and more selective detection and checking of symmetries.

The three-dimensional structure of a scene consisting of at least five points whose images are identified in two perspective views obtained by a translating camera, can be reconstructed up to a 3D affine transformation. Hence, a more detailed reconstruction is possible using less information when compared to the results for arbitrary stereo pairs taken with uncalibrated, perspective cameras.

The potential of coupled nonlinear diffusion processes to provide an efficient and implementable mechanism capable of edge preserving smoothing and noise suppression is investigated. The basic idea is that several maps undergo coupled development towards an equilibrium state. These maps could contain intensity, local edge strength, range, or another quantity. All these maps, including the edge map, contain continuous rather than all-or-nothing information, following a strategy of least commitment. Each of the approaches is developed and tested on a parallel transputer network.>

Line drawings of 3D environments are often sufficient for scene interpretation. The VLSI implementation of an algorithm for extracting edge information from grey level images is presented. It yields one pixel wide connected edge point strings and is applicable in real-time.>

This paper introduces a number of systems of coupled, non-linear diffusion equations and investigates their role in edge-preserving smoothing and noise suppression. The basic idea is that several maps describing the image undergo coupled development towards an equilibrium state representing the enhanced image. These maps could contain: e.g., intensity, local edge strength, range or another quantity. All these maps, including the edge map, contain continuous rather than all-or-nothing information, following a strategy of least commitment. Each of the approaches has been developed and tested on a parallel transputer network.