Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (≈ 7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

Grasping a variety of objects remains a key challenge in the development of versatile robotic systems. The human hand is remarkably dexterous, capable of grasping and manipulating objects with diverse shapes, mechanical properties, and textures. Inspired by how humans use two fingers to pick up thin and large objects such as fabric or sheets of paper, we aim to develop a gripper optimized for grasping such deformable objects. Observing how the soft and flexible fingertip joints of the hand approach and grasp thin materials, a hybrid gripper design that incorporates both soft and rigid components was proposed. The gripper utilizes a soft pneumatic ring wrapped around a rigid revolute joint to create a flexible two-fingered gripper. Experiments were conducted to characterize and evaluate the gripper's performance in handling sheets of paper and other objects. Compared to rigid grippers, the proposed design improves grasping efficiency and reduces the gripping distance by up to eightfold.

Liquid phase change pouch actuators (liquid pouch motors) hold great promise for a wide range of robotic applications, from artificial organs to pneumatic manipulators for dexterous manipulation. However, the usability of liquid pouch motors remains challenging due to the nonlinear intrinsic properties of liquids and their highly dynamic implications for liquid-gas phase changes, which complicate state modeling and estimation. To address these issues, we propose a reservoir computing-based method for modeling the inflation states of a customized liquid pouch motor, which serves as an actuator, featuring four Peltier heating junctions. We use a motion capture system to track the landmark movements on the pouch as a proxy for its volumetric profile. These movements represent the internal liquid-gas phase changes of the pouch at stable room temperature, atmospheric pressure, and in the presence of electrical noise. The motion coordinates are thus learned by our reservoir computing framework, PhysRes, to model the states based on prior observations. Through training, our model achieves excellent results on the test set, with a normalized root mean squared error of 0.0041 in estimating the states and a corresponding volumetric error of 0.0160%. To further demonstrate how such actuators could be implemented in the future, we also design a dual-pouch actuator-based robotic gripper to control the grasping of soft objects. Our design and source code are available at: https://github.com/tatung/liquidpouch_reservoir.

Robot-assisted endovascular navigation provides significant advantages, including reduced radiation exposure for surgeons and improved patient safety. However, a major challenge is to control curvilinear instruments like guidewires precisely for smooth and accurate navigation while adapting to anatomical variations and external forces. Traditional segmentation-based approaches struggle with real-time prediction of the guidewire’s evolving shape, limiting their effectiveness in navigation tasks. In this paper, we propose SplineFormer, an explainable transformer network that predicts the continuous, structured representation of the guidewire as a B-spline. This formulation enables a compact, smooth, and explainable state representation that facilitates downstream navigation. By leveraging SplineFormer’s predictions within an imitation learning framework, our system successfully performs autonomous endovascular navigation. Experimental results show that SplineFormer achieves a 50% success rate when fully autonomously cannulating the Brachiocephalic Artery in a real robotic setup, demonstrating its potential for improved autonomous navigation in endovascular interventions.

Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or fine-tuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner, responsible for strategizing complex queries; Coder, which generates and executes source code; and Observer, which evaluates the outcomes and provides feedback. Intensive experiments on two large-scale datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach. Our project page is available at https://zquang2202.github.io/GraspMAS.

Soft robots - due to their intrinsic flexibility of the body - can adaptively navigate unstructured environments. One of the most popular locomotion gaits that has been implemented in soft robots is undulation. The undulation motion in soft robots resembles the locomotion gait of stringy creatures such as snakes, eels, and C. Elegans. Typically, the implementation of undulation locomotion on a soft robot requires many actuators to control each segment of the stringy body. The added weight of multiple actuators limits the navigating performance of soft-bodied robots. In this paper, we propose a simple tendon-driven flexible beam with only one actuator (a DC motor) that can generate a mechanical traveling wave along the beam to support the undulation locomotion of soft robots. The beam will be precompressed along its axis by shortening the length of the two tendons to form an S-shape, thus pretensioning the tendons. The motor will wind and unwind the tendons to deform the flexible beam and generate traveling waves along the body of the robot. We experiment with different pre-tension to characterize the relationship between tendon pre-tension forces and the DC-motor winding/unwinding. Our proposal enables a simple implementation of undulation motion to support the locomotion of soft-bodied robots.

Soft robots made of flexible materials are highly adaptive, easy to fabricate, and safer to interact with. One of the ways for soft robots to interact with the surrounding environment is through their deformable bodily characteristics including internal body stiffness and external body friction. Though the flexibility of soft-bodied robots has been rigorously studied, the frictional skin of such soft-bodied robots, acting as a mechanical interface between the robot and the environment, remains unexplored. Being able to design the frictional skin will make soft-bodied robots more versatile in environmental navigation, more dexterous in manipulation tasks, and more flexible in haptic feedback. In this paper, we propose a robotic skin that can be programmed dynamically to change the mode of friction. The robotic skin is based on bistable bellow structures that can be switched between two folding states to change the contact points between the robotic skin and the ground. Our robotic skin can dynamically change its anisotropic frictional behavior to add another dimension to the designing space of soft robotics.

Origami/kirigami robotics are opening a path that leads to lightweight, compact, and expandable robots. However, it is generally challenging to design agile motions for origami/kirigami robots due to their size and the intrinsic limitation of the materials. In this paper, we propose to use the bistability of the waterbomb base structure to generate the swift motion of the robots. We evaluate the bistability of the waterbomb-based structure and build origami jumpers with different configurations of the body to help analyze the behavior of the waterbomb base bistable structure. The jumper is actuated by a phase change liquid pouch actuator. Our jumper is lightweight (0.3 g), flattenable, and able to jump to more than 12 times of its diameter and 112 times of its height.

This paper presents PlushPal, a web-based design tool for children to make plush toys interactive with machine learning (ML). With PlushPal, children attach micro:bit hardware to stuffed animals, design custom gestures for their toy, and build gesture-recognition ML models to trigger their own sounds. We describe how, in the context of storytelling, PlushPal introduces core concepts in ML including data sampling and model evaluation. We conducted online workshops and in-person play sessions with 11 children between 8-14 years old building interactive stuffed animals with PlushPal. In these play sessions, we observed how children imagined bringing their toys to life using ML, as well as how children’s data literacy changed as a result of experimenting with sensors, data sampling, and building their own ML models. Our work contributes a novel design space for children to express their ideas using gesture, as well as a description of observed debugging practices, building on efforts to support children using ML to enhance creative play.

Kirigami Haptic Swatches demonstrate how kirigami and origami based structures enable sophisticated haptic feedback through simple cut-and-fold fabrication techniques. We leverage four types of geometric patterns: rotational erection system (RES), split-fold waterbomb (SFWB), the overlaid structure of SFWB and RES (SFWB+RES), and cylindrical origami, to render different sets of haptic feedback (i.e. linear, bistable, bouncing snap-through, and rotational force behaviors, respectively). In each structure, not only the form factor but also the force feedback properties can be tuned through geometric parameters. We experimentally analyzed and modeled the structures, and implemented software to automatically generate 2D patterns for desired haptic properties. We also demonstrate five example applications including an assistive custom keyboard, rotational switch, multi-sensory toy, task checklist, and phone accessories. We believe the Kirigami Haptic Swatches helps tinkerers, designers, and even researchers to create interactions that enrich our haptic experience.

Personal identification numbers (PINs) and grid patterns have been used for user authentication, such as for unlocking smartphones. However, they carry the risk that attackers will learn the PINs and patterns by shoulder surfing. We propose a secure authentication method called SheetKey that requires complicated and quick touch inputs that can only be accomplished with a sheet that has a pattern printed with conductive ink. Using SheetKey, users can input a complicated combination of touch events within 0.3 s by just swiping the pad of their finger on the sheet. We investigated the requirements for producing SheetKeys, e.g., the optimal disc diameter for generating touch events. In a user study, 13 participants passed through authentication by using SheetKeys at success rates of 78-87%, while attackers using manual inputs had success rates of 0-27%. We also discuss the degree of complexity based on entropy and further improvements, e.g., entering passwords on alphabetical keyboards.

The exploration of spatially limited terrestrial or aquatic environments requires miniature and lightweight robots. Soft-bodied robot research is paving ways for a new class of small-scale robots that can navigate a variety of environments with minimum influence on the environment itself. However, it is generally challenging to design miniature soft-bodied robots that efficiently adapt to the change between viscous environments. A small-scale soft-bodied robot, which could slowly move on dry land, will need rapid motions to be able to swim in a wet environment. Although using snap-through buckling of a deformable body could help to create swift motions of the robot, merely applying the snap-through buckling does not improve the swimming speed of the robot so much. Here we propose a design of a stringy soft-bodied robot that can crawl on dry surfaces and swim in liquid environments. Besides taking advantage of the snap-through buckling using coil shape memory alloys (SMAs), we design the body of the robot with a geometrical overlapping of the active body segments and control the frequency of the undulation movement, which is crucial for the swimming locomotion. We evaluate the performance of the robot in different density and viscosity liquids such as cooking oil and Glycerin solution. We found that the robot needs to drastically change its undulation from low to high frequency when it moves from high to low viscosity environments. Our robot can swim at a speed of 3. 37 body-lengths per minute (BL/min) and crawl at a speed of 1. 74 BL/min. We anticipate our findings will help shed light on the design of soft-bodied robots that adapt to the changing environments efficiently.

Soft-bodied robots are getting attention from researchers as their potential in designing compliant and adaptive robots. However, soft-bodied robots also pose many challenges not only in non-linear controlling but also in design and fabrication. Especially, the non-compatibility between soft materials and rigid sensors/actuators makes it more difficult to design a fully compliant soft-bodied robot. In this paper, we propose an all-printed sensor and actuator for designing soft-bodied robots by printing silver nano-particle ink on top of a flexible plastic film. We can print bending sensors and thermal based actuators instantly with home-commodity inkjet printers without any pre/post-processing. We exemplify the application of this fabrication method with an all-printed paper caterpillar robots which can inch forward and sense its body bending angle.

We present PEP (Printed Electronic Papercrafts), a set of design and fabrication techniques to integrate electronic based interactivities into printed papercrafts via 3D sculpting. We explore the design space of PEP, integrating four functions into 3D paper products: actuation, sensing, display, and communication, leveraging the expressive and technical opportunities enabled by paper-like functional layers with a stack of paper. We outline a seven-step workflow, introduce a design tool we developed as an add-on to an existing CAD environment, and demonstrate example applications that combine the electronic enabled functionality, the capability of 3D sculpting, and the unique creative affordances by the materiality of paper.

Soft-bodied and continuum robots have shown great adaptability to the environment thanks to its flexibility of the body. They have great potential in environment exploring or rescuing mission. One of those robots is snake-like soft-bodied robots. A snake robot is often made by attaching passive wheels along a long body to achieve frictional anisotropy. This anisotropic structure helps to propel the body with serpentine locomotion and prevents it from sliding laterally. However, with a snake-like soft-bodied robot, attaching wheels is not only clumsy but also adding weight to the robot. In this paper, being inspired by the scales on the skin of a snake, we propose a designing scheme to achieve an all-printed wriggle soft-bodied robot by patterning high and low friction material to the ventral side of the robot. Compared to a totally flat ventral, we are able to speed-up the serpentine locomotion 2.8 times. Besides, by changing the configuration of high/low friction material, our wriggle soft-bodied robot can easily move forward or backward just by switching the controlling signal. The fabrication time is just less than 1 hour and the robot can achieve the speed of 26 mm/s.

Instant Inkjet Circuits by silver nano-particle ink realized home-brew electric circuit fabrication. However, current method can support only single-layered patterns, and conventional inter-layer connection methods are not suitable. In this paper, we will evaluate various easy-to-use inter-layer connection methods by making via holes, especially the ones made by different drilling mechanisms. We show that the felting needle is the best candidate as it can establish good conductivity immediately after nano-particle ink is printed into the hole, without using any curing process.