Abstract 3D Gaussian Splatting (3DGS) [KKLD23] has transformed novel‐view synthesis from RGB images, yet remains restricted to the visible spectrum. Many applications, including agricultural monitoring, rely on multi‐spectral imaging, where spectral camera alignment and scalability pose major challenges. We present MS‐Splatting—a multi‐spectral 3DGS framework enabling unified multi‐view consistent reconstruction and rendering across both visible and invisible spectra. Our key component is a neural color representation that encodes per‐primitive features shared across spectral bands, decoded through a shallow multi‐layer perceptron into spectrum‐specific radiance. By leveraging inter‐band correlations, this formulation enhances detail while reducing memory consumption compared to independent band modeling via per‐channel modeling with spherical harmonics. Our method enables accurate parallax‐free novel‐view vegetation index rendering for plant monitoring and enhances RGB novel view synthesis quality by exploiting details revealed through multi‐spectral bands. Our evaluation demonstrates that MS‐Splatting exceeds the current leading methods in both categories. In addition, we introduce a multi‐spectral dataset from aerial captures covering outdoor environments, specifically designed for evaluating these applications. We will release our code and dataset to facilitate further research. The project page is located at: https://meyerls.github.io/ms_splatting

We introduce FruitNeRF++, a novel fruit-counting approach that combines contrastive learning with neural radiance fields to count fruits from unstructured input photographs of orchards. Our work is based on FruitNeRF [6], which employs a neural semantic field combined with a fruit-specific clustering approach. The requirement for adaptation for each fruit type limits the applicability of the method, and makes it difficult to use in practice. To lift this limitation, we design a shape-agnostic multi-fruit counting framework, that complements the RGB and semantic data with instance masks predicted by a vision foundation model. The masks are used to encode the identity of each fruit as instance embeddings into a neural instance field. By volumetrically sampling the neural fields, we extract a point cloud embedded with the instance features, which can be clustered in a fruit-agnostic manner to obtain the fruit count. We evaluate our approach using a synthetic dataset containing apples, plums, lemons, pears, peaches, and mangoes, as well as a real-world benchmark apple dataset. Our results demonstrate that FruitNeRF++ is easier to control and compares favorably to other state-of-the-art methods.

We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DoF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting. Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. PEGASUS allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that capture images from both the object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.

We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count. The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit. We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mango. Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

Robots with elasticity in structural components can suffer from undesired end-effector positioning imprecision, which exceeds the accuracy requirements for successful manipulation. We present the Probabilistic-Product-Of-Exponentials robot model, a novel approach for kinematic modeling of robots. It does not only consider the robot's deterministic geometry but additionally models time-varying and configuration-dependent errors in a probabilistic way. Our robot model allows to propagate the errors along the kinematic chain and to compute their influence on the end-effector pose. We apply this model in the context of sensor fusion for manipulator pose correction for two different robotic systems. The results of a simulation study, as well as of an experiment, demonstrate that probabilistic configuration-dependent error modeling of the robot kinematics is crucial in improving pose estimation results.

For wheeled mobile systems, the wheel odometry is an important source of information about the current motion of the vehicle. It is used e.g. in the context of pose estimation and self-localization of planetary rovers, which is a crucial part of the success of planetary exploration missions. Depending on the wheel-soil interaction properties, wheel odometry measurements are subject to inherent errors such as wheel slippage. In this paper, a parameter-based approach for whole-body slip modeling and calibration is applied to a four-wheeled lightweight rover system. Details on the method for slip parameter calibration as well as the system-specific implementation are given. Experimental results from a test campaign on Mt. Etna are presented, showing significant improvements of the resulting wheel odometry measurements. The results are validated during a long range drive of approx. 900 m and discussed w. r. t. the advantages but also limitations of the method within a space exploration scenario.