In this work, we explore the technical feasibility of implementing end-to-end 3D object detection (3DOD) with surround-view fisheye camera system. Specifically, we first investigate the performance drop incurred when transferring classic pinhole-based 3D object detectors to fisheye imagery. To mitigate this, we then develop two methods that incorporate the unique geometry of fisheye images into mainstream detection frameworks: one based on the bird's-eye-view (BEV) paradigm, named FisheyeBEVDet, and the other on the query-based paradigm, named FisheyePETR. Both methods adopt spherical spatial representations to effectively capture fisheye geometry. In light of the lack of dedicated evaluation benchmarks, we release Fisheye3DOD, a new open dataset synthesized using CARLA and featuring both standard pinhole and fisheye camera arrays. Experiments on Fisheye3DOD demonstrate that our fisheye-compatible modeling improves accuracy by up to 6.2% compared to baseline methods.

Top-view fisheye cameras are cost-efficient devices used for omnidirectional perception. However, the wide field of view (FOV) of these cameras causes significant image distortion, while the top-view setup introduces rotational symmetry, resulting in the degradation of performance of the standard convolution neural network when processing fisheye images. To address this problem, we present ARFC, a novel method called adaptive receptive field convolution, specifically designed to extract rotation- and scale-equivariant representations from top-view fisheye images. Unlike traditional orientation-static convolutions, ARFC incorporates an adaptive rotating kernel (ARK) to separate rotation from distortion and capture rotational equivariant features. Additionally, a multi-scale fusion module (MSFM) is implemented to combine scale-distorted features. Experimental evaluations conducted on THEODORE for segmentation tasks illustrate the superior performance of ARFC compared to the current state-of-the-art methods. Code is available at: https://github.com/LinMenwill/ARFC.git.

Keypoint detection, description and depth estimation are fundamental parts for many computer vision tasks, like image matching and simultaneous localization and mapping (SLAM) systems. Early methods are based on human heuristics, extracted handcrafted feature from image pixel, which may lead to unstable and confusable results. Recent years learned-based methods have achieved remarkable results in this field. However, most of these methods cannot ensure real-time application, and the lightweight deployment of network models remains challenging. Additionally, the black-box nature of neural network operations lacks interpretability, which constrains further application expansion. This paper proposed a super lightweight framework, FastPoint, used shallow binary neural network combine with handcrafted differentiable module FastLayer, which outputs key-points location, descriptor and depth estimation simultaneously. Our FastPoint is trained on data synthetically created form KITTI Raw dataset and evaluated on HPatches. The results demonstrate that our method achieves competitive performance compared to existing approaches while significantly reducing computational complexity.

Sparse local feature matching has seen significant progress with the rise of deep learning. Unfortunately, the performance could suffer from significant degradation under rainy conditions. Most of the existing image-level methods focus on pixel-level similarity rather than feature discrimination, limiting their effectiveness for matching tasks. Meanwhile, contrastive learning offers a promising feature-level solution to domain discrepancies but often overlooks pixel-wise contrast and explicit geometrical constraints, resulting in unstable feature detection and description in rainy conditions. To address these issues, we propose an effective contrastive learning framework, called PRED, for local feature matching under rainy conditions, which leverages the inherent differences between geometrical and rainy discrepancies as an additional constraint for deraining contrastive learning, while preserving geometrical invariance in feature detection and description. To achieve this, the PRED framework integrates an Error-Aware Rainy Distillation (EARD) module and Pixel-wise Selective Contrast (PSC) module to explore mutual properties of pixel-wise geometrical- and rainy-invariant local features between clean and rainy images. Extensive experiments on the rainy-distorted HPatches and MegaDepth datasets demonstrate the effectiveness of the PRED framework, achieving state-of-the-art performance in sparse local feature matching under rainy conditions.

Understanding the environments through interactions has been one of the most important human intellectual activities in mastering unknown systems. Deep reinforcement learning (DRL) has already been known to achieve effective control through human-like exploration and exploitation in many applications. However, the opaque nature of deep neural network (DNN) often hides critical information about feature relevance to control, which is essential for understanding the target systems. In this article, a novel online feature selection framework, namely, the dual-world-based attentive feature selection (D-AFS), is first proposed to identify the contribution of the inputs over the whole control process. Rather than the one world used in most DRL, D-AFS has both the real world and its virtual peer with twisted features. The newly introduced attention-based evaluation (AR) module performs the dynamic mapping from the real world to the virtual world. The existing DRL algorithms, with slight modification, can learn in the dual world. By analyzing the DRL's response in the two worlds, D-AFS can quantitatively identify respective features' importance toward control. A set of experiments is performed on four classical control systems in OpenAI Gym. Results show that D-AFS can generate the same or even better feature combinations than the solutions provided by human experts and seven recent feature selection baselines. In all cases, the selected feature representations are closely correlated with the ones used by underlying system dynamic models.

Extracting local features for accurate correspondences between image pairs is an essential basis for various computer vision tasks. Recent works have shown that deep neural networks (DNNs) have demonstrated promising performance in challenging environments. However, these state-of-the-art DNN-based approaches are not well suited for the scenario with geometry rotations due to their intrinsic deficiencies of square kernel structure. That is, square kernel structures in standard DNNs cannot fully identify the essentials of the rotations in geometry. To address this problem, we present RICNN, a novel deep learning framework that encodes invariance against the rotations in geometry explicitly into convolutional neural networks. Rather than using the square-shaped kernel structure, RICNN adopts sectorshaped convolutional kernels to achieve encoding invariance in all rotations. With the explicitness of such rotation encoding, RICNN enables the transfer of perspective DNN models to obtain rotation-invariant descriptions. Furthermore, we propose a novel multi-level hinge triplet loss function to strengthen the matching constraints against geometry rotations. Comprehensive experiments demonstrate the strong generalization ability of the RICNN descriptor on the HPatches dataset. Toward the rotation invariance evaluation, our method shows state-of-the-art results.

Reducing sensor requirements while keeping optimal control performance is crucial to many industrial control applications to achieve robust, low-cost, and computation-efficient controllers. However, existing feature selection solutions for the typical machine learning domain can hardly be applied in the domain of control with changing dynamics. In this paper, a novel framework, namely the Dual-world embedded Attentive Feature Selection (D-AFS), can efficiently select the most relevant sensors for the system under dynamic control. Rather than the one world used in most Deep Reinforcement Learning (DRL) algorithms, D-AFS has both the real world and its virtual peer with twisted features. By analyzing the DRL's response in two worlds, D-AFS can quantitatively identify respective features' importance towards control. A well-known active flow control problem, cylinder drag reduction, is used for evaluation. Results show that D-AFS successfully finds an optimized five-probes layout with 18.7% drag reduction than the state-of-the-art solution with 151 probes and 49.2% reduction than five-probes layout by human experts. We also apply this solution to four OpenAI classical control cases. In all cases, D-AFS achieves the same or better sensor configurations than originally provided solutions. Results highlight, we argued, a new way to achieve efficient and optimal sensor designs for experimental or industrial systems. Our source codes are made publicly available at https://github.com/G-AILab/DAFSFluid.