Reconstructing realistic digital twins has become crucial as advances in mixed reality, metaverse, and robotics demand more accurate simulations for the physical world. Despite technical progress, building high-fidelity digital twins from a systematic and human-centered perspective remains underexplored. Drawing from the human processing model, we decompose human-centric reality into perception, motion, and cognition, and define a reality-preserving digital twin (RPDT) as a reconstruction integrating these dimensions. We present RealTwin, an attribute-graph-based representation and inference framework for RPDT. Leveraging the grounding capabilities of Multimodal Large Language Models (MLLMs), RealTwin chains AI tools to construct attribute graphs that faithfully encode real-world properties. We validate RealTwin through both technical evaluation, showing promising success in graph parsing and attribute inference, and a user study, assessing its applicability across diverse user groups. Enlightened by RealTwin, we discuss critical issues, including ecology, interaction space, and real-world adoption, for future end-to-end, fine-grained, and scalable digital twin reconstruction.

Differentiable simulation has become a powerful tool for system identification. While prior work has focused on identifying robot properties using robot-specific data or object properties using object-specific data, our approach calibrates object properties by using information from the robot, without relying on data from the object itself. Specifically, we utilize robot joint encoder information, which is commonly available in standard robotic systems. Our key observation is that by analyzing the robot's reactions to manipulated objects, we can infer properties of those objects, such as inertia and softness. Leveraging this insight, we develop differentiable simulations of robot-object interactions to inversely identify the properties of the manipulated objects. Our approach relies solely on proprioception – the robot's internal sensing capabilities – and does not require external measurement tools or vision-based tracking systems. This general method is applicable to any articulated robot and requires only joint position information. We demonstrate the effectiveness of our method on a low-cost robotic platform, achieving accurate mass and elastic modulus estimations of manipulated objects with just a few seconds of computation on a laptop.

Modeling and analyzing object and shape has been well studied in the past. However, manipulation of these complex tools and articulated objects remains difficult for autonomous agents. Our human hands, however, are dexterous and adaptive. We can easily adapt a manipulation skill on one object to all objects in the class and to other similar classes. Our intuition comes from that there is a close connection between manipulations and topology and articulation of objects. The possible articulation of objects indicates the types of manipulation necessary to operate the object. In this work, we aim to take a manipulation perspective to understand everyday objects and tools. We collect a multi-modal visual-tactile dataset that contains paired full-hand force pressure maps and manipulation videos. We also propose a novel method to learn a cross-modal latent manifold that allow for cross-modal prediction and discovery of latent structure in different data modalities. We conduct extensive experiments to demonstrate the effectiveness of our method.

Despite recent exponential advancements in computer vision and reinforcement learning, it remains challenging for robots to interact with liquids. These challenges are particularly pronounced due to the limitations imposed by opaque containers, transparent liquids, fine-grained splashes, and visual obstructions arising from the robot’s own manipulation activities. Yet, there exists a substantial opportunity for robotics to excel in liquid identification and manipulation, given its potential role in chemical handling in laboratories and various manufacturing sectors such as pharmaceuticals or beverages. In this work, we present a novel approach for liquid class identification and state estimation leveraging electrical impedance sensing. We design and mount a digitally embroidered electrode array to a commercial robot gripper. Coupled with a customized impedance sensing board, we collect data on liquid manipulation with a swept frequency sensing mode and a frequency-specific impedance measuring mode. Our developed learning-based model achieves an accuracy of 93.33% in classifying 9 different types of liquids (8 liquids + air), and 97.65% in estimating the liquid state. We investigate the effectiveness of our system with a series of ablation studies. These findings highlight our work as a promising solution for enhancing robotic manipulation in liquid-related tasks.

Soft robotic manipulators with many degrees of freedom can carry out complex tasks safely around humans. However, manufacturing of soft robotic hands with several degrees of freedom requires a complex multi-step manual process, which significantly increases their cost. We present a design of a multi-material 15 DoF robotic hand with five fingers including an opposable thumb. Our design has 15 pneumatic actuators based on a series of hollow chambers that are driven by an external pressure system. The thumb utilizes rigid joints and the palm features internal rigid structure and soft skin. The design can be directly 3D printed using a multi-material additive manufacturing process without any assembly process and therefore our hand can be manufactured for less than 300 dollars. We test the hand in conjunction with a low-cost vision-based teleoperation system on different tasks.

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Truss robots are highly redundant parallel robotic systems that can be applied in a variety of scenarios. The variable topology truss (VTT) is a class of modular truss robots. As self-reconfigurable modular robots, a VTT is composed of many edge modules that can be rearranged into various structures depending on the task. These robots change their shape by not only controlling joint positions as with fixed morphology robots but also reconfiguring the connectivity between truss members in order to change their topology. The motion planning problem for VTT robots is difficult due to their varying morphology, high dimensionality, the high likelihood for self-collision, and complex motion constraints. In this article, a new motion planning framework to dramatically alter the structure of a VTT is presented. It can also be used to solve locomotion tasks that are much more efficient compared with previous work. Several test scenarios are used to show its effectiveness.

This paper introduces ActionSense, a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It provides rich, synchronized data streams along with ground truth data to facilitate learning pipelines that could extract insights about how humans interact with the physical world during activities of daily living, and help lead to more capable and collaborative robot assistants. The wearable sensing suite captures motion, force, and attention information; it includes eye tracking with a first-person camera, forearm muscle activity sensors, a body-tracking system using 17 inertial sensors, finger-tracking gloves, and custom tactile sensors on the hands that use a matrix of conductive threads. This is coupled with activity labels and with externally-captured data from multiple RGB cameras, a depth camera, and microphones. The specific tasks recorded in ActionSense are designed to highlight lower-level physical skills and higher-level scene reasoning or action planning. They include simple object manipulations (e.g., stacking plates), dexterous actions (e.g., peeling or cutting vegetables), and complex action sequences (e.g., setting a table or loading a dishwasher). The resulting dataset and underlying experiment framework are available at https://action-sense.csail.mit.edu. Preliminary networks and analyses explore modality subsets and cross-modal correlations. ActionSense aims to support applications including learning from demonstrations, dexterous robot control, cross-modal predictions, and fine-grained action segmentation. It could also help inform the next generation of smart textiles that may one day unobtrusively send rich data streams to in-home collaborative or autonomous robot assistants.

The Variable Topology Truss (VTT) is a new class of self-reconfigurable robot that can reconfigure its truss shape and topology depending on the task or environment requirements. Motion planning and avoiding self-collision are difficult as these systems usually have dozens of degrees-of-freedom with complex intersecting parallel actuation. There are two different types of shape changing actions for a VTT: geometry reconfiguration and topology reconfiguration. This paper focuses on the geometry reconfiguration actions. A new cell decomposition approach is presented based on a fast and complete method to compute the collision-free space of a node in a truss. A simple shape-morphing method is shown to quickly create motion paths for reconfiguration by moving one node at a time.

This paper is an initial step toward the realization of an aerial robot that can perform lateral physical work, such as drilling a hole or fastening a screw in a wall. Aerial robots are capable of high maneuverability and can provide access to locations that would be difficult or impossible for ground-based robots to reach. However, to fully utilize this mobility, systems would ideally be able to perform functional work in those locations, requiring the ability to exert lateral forces. To substantially improve a hovering vehicle's ability to stably deliver large lateral forces, we propose the use of a versatile suction-based gripper that can establish pulling contact on featureless surfaces. Such contact enables access to environmental forces that can be used to further stabilize the vehicle and also increase the lateral force delivered to the surface through a possible secondary mechanism. This paper introduces the concept, describes the design of a new self-sealing suction cup based on a previous design, details the design of a gripper using those cups, and describes the arm and flight vehicle. It then evaluates the cup and gripper performance in several ways, culminating in physical grasping demonstrations using the arm and gripper, including one in the presence of simulated flight noise based on data from preliminary indoor flight experiments.

This paper presents an algorithm to do motion planning for a new class of self-reconfigurable modular robot: the variable topology truss (VTT). Modular robots consist of many modules that can be configured into various structures, and motion planning problem for modular robots with many degrees of freedom and many motion constraints is a significant challenge. In this paper, we propose a novel motion planning algorithm for modular robots to handle this problem with huge state space inspired by DNA replication process - the topology of DNA can be changed by cutting and resealing strands as tanglements form. In a variable topology truss, a single node with enough edge modules can split into a pair of nodes and two separate nodes can be merged to become an individual one. This self-reconfiguration ability results in more potential applications for this type of robots in unstructured environment, such as space and underseas but also leads to more challenges for reconfiguration planning. A novel way to model the robot in a nonuniform grid space is presented and a simple local planner is also developed to check the validation of possible actions. This approach significantly simplifies the problem and some experiment results show that the complicated problem can be solved in a reasonable time.

This paper presents a novel manipulator for aerial vehicles to perform grasping and manipulation tasks. The goal is to design a low-cost, relatively light but strong manipulator with a large workspace and compact storage space that can be mounted on an unmanned aerial system. A novel design solution based on the Spiral Zipper, an expanding tube, combined with tether actuators is presented. A model of the system is introduced and the control method and pose estimator are developed and tested with some experiments showing the reliable performance of the overall system. An experiment with a self-sealing suction cup gripper demonstrates manipulation while mounted on the aerial vehicle frame.

The PaintPot manufacturing process is a new way to create low-cost, low-profile, highly customizable potentiometers for position sensing in robotic applications. It uses widely accessible materials, requires no special expertise, and creates custom potentiometers in a variety of shapes and sizes, including curved surfaces. PaintPots offer accuracy and precision performance comparable with commercial (non-customizable) options through a calibration process that trades small computation for cost. This paper includes detailed PaintPot manufacturing and calibration processes, and experiments that validate the accuracy, precision, and lifetime performance of PaintPots, comparable to commercial sensors. We also provide a case-study application in the SMORES-EP modular robot, and show how the PaintPot process can be used to create resistive surfaces capable of sensing position in 2D on planes and spheres.

We present the EP-Face connector, a novel connector for hybrid chain-lattice type modular robots that is highstrength (88.4N), compact, fast, power efficient, and robust to position errors. The connector consists of an array of electro-permanent magnets (EP magnets) embedded in a planar face. EP magnets are solid-state magnets that can be turned on and off and require power only when changing state. In this paper, we present the design of the connector, manufacturing process, detailed experimental characterization of the connector strength under different loading conditions, and compare its performance to existing magnetic and mechanical connectors. We also illustrate the functional benefits of the EPFace by demonstrating reconfiguration with the SMORES-EP robot.