Digitizing the physical world into accurate simulation‑ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness.

We introduce OpenBot-Fleet, a comprehensive open-source cloud robotics system for navigation. OpenBot-Fleet uses smartphones for sensing, local compute and communication, Google Firebase for secure cloud storage and off-board compute, and a robust yet low-cost wheeled robot to act in real-world environments. The robots collect task data and upload it to the cloud where navigation policies can be learned either offline or online and can then be sent back to the robot fleet. In our experiments we distribute 72 robots to a crowd of workers who operate them in homes, and show that OpenBot-Fleet can learn robust navigation policies that generalize to unseen homes with >80% success rate. OpenBot-Fleet represents a significant step forward in cloud robotics, making it possible to deploy large continually learning robot fleets in a cost-effective and scalable manner. All materials can be found at https://www.openbot.org/.

Current visual navigation systems often treat the environment as static, lacking the ability to adaptively interact with obstacles. This limitation leads to navigation failure when encountering unavoidable obstructions. In response, we introduce IN-Sight, a novel approach to self-supervised path planning, enabling more effective navigation strategies through interaction with obstacles. Utilizing RGB-D observations, IN-Sight calculates traversability scores and incorporates them into a semantic map, facilitating long-range path planning in complex, maze-like environments. To precisely navigate around obstacles, IN-Sight employs a local planner, trained imperatively on a differentiable costmap using representation learning techniques. The entire framework undergoes end-to-end training within the state-of-the-art photorealistic Intel SPEAR Simulator. We validate the effectiveness of IN-Sight through extensive benchmarking in a variety of simulated scenarios and ablation studies. Moreover, we demonstrate the system’s real-world applicability with zero-shot sim-to-real transfer, deploying our planner on the legged robot platform ANYmal, showcasing its practical potential for interactive navigation in real environments.

Providing functionality through articulation and interaction with objects is a key objective in 3D generation. We introduce MIDGArD (Modular Interpretable Diffusion over Graphs for Articulated Designs), a novel diffusion-based framework for articulated 3D asset generation. MIDGArD improves over foundational work in the field by enhancing quality, consistency, and controllability in the generation process. This is achieved through MIDGArD's modular approach that separates the problem into two primary components: structure generation and shape generation. The structure generation module of MIDGArD aims at producing coherent articulation features from noisy or incomplete inputs. It acts on the object's structural and kinematic attributes, represented as features of a graph that are being progressively denoised to issue coherent and interpretable articulation solutions. This denoised graph then serves as an advanced conditioning mechanism for the shape generation module, a 3D generative model that populates each link of the articulated structure with consistent 3D meshes. Experiments show the superiority of MIDGArD on the quality, consistency, and interpretability of the generated assets. Importantly, the generated models are fully simulatable, i.e., can be seamlessly integrated into standard physics engines such as MuJoCo, broadening MIDGArD's applicability to fields such as digital content creation, meta realities, and robotics.

In this work, we propose a new estimation method for second-order kinematics for floating-base robots, based on highly redundant distributed inertial feedback. The linear acceleration of each robot link is measured at multiple points using a multimodal, self-configuring and self-calibrating artificial skin. The proposed algorithm is two-fold: i) the skin acceleration data is fused at the link level for state dimensionality reduction; ii) the estimated values are then fused limb-wise with data from the joint encoders and the main inertial measurement unit (IMU), using a Sigma-point Kalman filter. In this manner, it is possible to estimate the joint velocities and accelerations while avoiding the lag and noise amplification phenomena associated with conventional numerical derivation approaches. Experiments performed on the right arm and torso of a REEM-C humanoid robot, demonstrate the consistency of the proposed estimation method.

Multiple approaches to the estimation of high-order motion derivatives for innovative control applications now rely on the data collected by redundant arrays of inertial sensors mounted on robots, with promising results. However, most of these works suffer scalability issues induced by the considerable amount of data generated by such large-scale distributed sensor systems. In this article, we propose a new adaptive sensor-selection algorithm, for distributed inertial measurements. Our approach consists in using the data of a subset of sensors, selected among a larger collection of inertial sensing elements covering a rigid robot link. The sensor selection process is formulated as an optimization problem, and solved using a projected gradient heuristics. The proposed method can run online on a robot and be used to recalculate the selected sensor arrangement on the fly when physical interaction or potential sensor failure is detected. The tests performed on a simulated UR5 industrial manipulator covered with a multimodal artificial skin, demonstrate the consistency and performance of the proposed sensor-selection algorithm.

This article presents a holistic approach to the engineering of an artificial robot skin for robots. An example of a multimodal skin cell is given, one that supports multiple human-like sensing modalities, and support for skin cell network is also provided; this is essential to form large-area skin patches in order to cover the surfaces of robots. The essential elements of efficiently handling a large amount of tactile data are explained. A general control framework, which supports robots commanded in position, velocity, and torque, is provided and validated. Several applications of this robot skin will be presented, demonstrating the effectiveness and efficiency of our artificial robot skin to support a wide number of robotic platforms as well as its ease of use across different domains.

In this paper, we propose a control method, providing mobile robots with whole-body compliance capabilities, in response to multicontact physical interactions with their environment. The external forces applied to the robot, as well as their localization on its kinematic tree, are measured using a multimodal, self-configuring, and self-calibrating artificial skin. We formulate a compliance control law in Cartesian space, as a set of quadratic optimization problems, solved in parallel for each limb involved in the interaction process. This specific formulation makes it possible to determine the torque commands required to generate the desired reactive behaviors, while taking the robot kinematic and dynamic constraints into account. When a given limb fails to produce the desired compliant behavior, the generalized force residual at the considered contact points is propagated to a parent limb in order to be adequately compensated. Hence, the robot's compliance range can be extended in a both robust and easily adjustable manner. The experiments performed on a dual-arm velocity-controlled mobile manipulator show that our methodology is robust to nullspace interactions and robot physical constraints.