Semantic segmentation is crucial for comprehending a 2D image or a 3D point cloud by categorizing each pixel/point into a specific semantic class. A key objective of machine learning models used for segmentation is to improve the model accuracy and reliability by reducing misclassifications. Existing approaches treat all misclassification errors uniformly, lacking consideration for specific application requirements. Recognizing that some misclassifications may be more acceptable than others, depending on the particular class and application, we propose a novel Similarity-weighted Intersection over Union metric (sIOU). This approach provides a comprehensive evaluation framework, particularly in challenging scenarios with indistinct class boundaries, incorporating wrongly segmented classes with a similarity score. The metric offers a more refined assessment of a model’s segmentation performance in complex, real-world environments. Validation with an in-house indoor dataset demonstrates its superiority over existing metrics, highlighting its flexibility for diverse application contexts.

Recently, transformer networks have outperformed traditional deep neural networks in natural language processing and show a large potential in many computer vision tasks compared to convolutional backbones. In the original transformer, readout tokens are used as designated vectors for aggregating information from other tokens. However, the performance of using readout tokens in a vision transformer is limited. Therefore, we propose a novel fusion strategy to integrate radar data into a dense prediction transformer network by reassembling camera representations with radar representations. Instead of using readout tokens, radar representations contribute additional depth information to a monocular depth estimation model and improve performance. We further investigate different fusion approaches that are commonly used for integrating additional modality in a dense prediction transformer network. The experiments are conducted on the nuScenes dataset, which includes camera images, lidar, and radar data. The results show that our proposed method yields better performance than the commonly used fusion strategies and outperforms existing convolutional depth estimation models that fuse camera images and radar.

In this paper, we present a Computer Vision (CV) based tracking and fusion algorithm, dedicated to a 3D printed gimbal system on drones flying in nature. The whole gimbal system can stabilize the camera orientation robustly in challenging environments by using skyline and ground plane as references. Our main contributions are the following: a) a light-weight Resnet-18 backbone network model was trained from scratch, and deployed onto the Jetson Nano platform to segment the image specifically into binary parts (ground and sky); b) our geometry assumption from the skyline and ground cues delivers the potential for robust visual tracking in the wild by using the skyline and ground plane as references; c) a manifold surface-based adaptive particle sampling can fuse orientation from multiple sensor sources flexibly. The whole algorithm pipeline is tested on our 3D-printed gimbal module with Jetson Nano. The experiments were performed on top of a building in a real landscape. The public code link: https://github.com/alexandor91/gimbal-fusion.git.

We integrate sparse radar data into a monocular depth estimation model and introduce a novel preprocessing method for reducing the sparseness and limited field of view provided by radar. We explore the intrinsic error of different radar modalities and show our proposed method results in more data points with reduced error. We further propose a novel method for estimating dense depth maps from monocular 2D images and sparse radar measurements using deep learning based on the deep ordinal regression network by Fu et al. Radar data are integrated by first converting the sparse 2D points to a height-extended 3D measurement and then including it into the network using a late fusion approach. Experiments are conducted on the nuScenes dataset. Our experiments demonstrate state-of-the-art performance in both day and night scenes.

It is generally understood that interpreting depth from monoscopic images inherently induces the problem of object segmentation and context recognition. A common way to deal with this complexity is to focus on one aspect (e.g. segmentation) and largely constrain or skip the other (e.g. scene classification). In our approach, we practically reconcile the two paths. We start by defining several depth profiles, providing rough estimates for both background and foreground, while reflecting common photographic and cinematic practices and rules. The idea is to assign one or more of these profiles to an image - based on the output of multiple classifiers, and apply an image-adaptive bilateral filter to align the depth with object edges. From extensive trials, we conclude that robust classification and visual performance can be achieved with this approach on a variety of content

Super-resolution algorithms combine multiple low resolution images into a single high resolution image. They have received a lot of attention recently in various application domains such as HDTV, satellite imaging, and video surveillance. These techniques take advantage of the aliasing present in the input images to reconstruct high frequency information of the resulting image. One of the major challenges in such algorithms is a good alignment of the input images: subpixel precision is required to enable accurate reconstruction. In this paper, we give an overview of some subspace techniques that address this problem. We first formulate super-resolution in a multichannel sampling framework with unknown offsets. Then, we present three registration methods: one approach using ideas from variable projections, one using a Fourier description of the aliased signals, and one using a spline description of the sampling kernel. The performance of the algorithms is evaluated in numerical simulations.

How often have you been able to implement an algorithm as it is described in a paper? And when you did, were you confident that you had exactly the same parameter values and results as the authors of the paper? All too often, articles do not describe all the details of an algorithm and thus prohibit an implementation by someone else. In this paper, we describe our experience with reproducible research, a paradigm to allow other people to reproduce with minimal effort the results we have obtained. We discuss both the reproducibility of data and algorithms, and give examples for each of them. The effort required to make research reproducible is compensated by a higher visibility and impact of the results.

We present a new method for signal reconstruction from multiple sets of samples with unknown offsets. We rewrite the reconstruction problem as a set of polynomial equations in the unknown signal parameters and the offsets between the sets of samples. Then, we construct a Grobner basis for the corresponding affine variety. The signal parameters can then easily be derived from this Grobner basis. This provides us with an elegant solution method for the initial nonlinear problem. We show two examples for the reconstruction of polynomial signals and Fourier series

Aliasing artifacts in images are visually very disturbing. Therefore, most imaging devices apply a low-pass filter before sampling. This removes all aliasing from the image, but it also creates a blurred image. Actually, all the image information above half the sampling frequency is removed. In this paper, we present a new method for the reconstruction of a high resolution image from a set of highly undersampled and thus aliased images. We use the information in the entire frequency spectrum, including the aliased part, to create a sharp, high resolution image. The unknown relative shifts between the images are computed using a subspace projection approach. We show that the projection can be decomposed into multiple projections onto smaller subspaces. This allows for a considerable reduction of the overall computational complexity of the algorithm. A high resolution image can then be reconstructed from the registered low resolution images. Simulation results show the validity of our algorithm.

In signal processing systems, aliasing is normally treated as a disturbing signal. That motivates the need for effective analog, optical and digital anti-aliasing filters. However, aliasing also conveys valuable information on the signal above the Nyquist frequency. Hence, an effective processing of the samples, based on a model of the input signal, would virtually allow the sampling frequency to be increased using slower and cheaper converters. We present such an algorithm for bandlimited signals that are sampled below twice the maximum signal frequency. Using a subspace method in the frequency domain, we show that these signals can be reconstructed from multiple sets of samples. The offset between the sets is unknown and can have arbitrary values. This approach can be applied to the creation of super-resolution images from sets of low resolution images. In this application, registration parameters have to be computed from aliased images. We show that parameters and high resolution images can be computed precisely, even when high levels of aliasing are present on the low resolution images.