New tangible input techniques are transforming human-computer interaction. Point-contact devices such as joysticks or buttons are simple and scalable, but they capture limited spatial information. In contrast, surface-based contact interfaces such as touchpads provide richer spatial input but require larger instrumented surfaces. We present MagBall, a magnetic-ball sensor that captures fine-grained interactions, including displacement and force, through the rotation of a magnet-embedded ball over a 3D Hall-effect sensor array. Our design localizes diverse physical interactions to a single point-contact yet operates at multiple scales from millimeters to meters. Our machine learning models can infer the displacement and force with root-mean-squared errors of 0.15 mm and 0.67 N. Furthermore, our device supports interactions across diverse surfaces such as glass, metal and human skin, without additional instrumentation. We demonstrate applications in stylus pens, wearable trackballs and smart massage tools, which naturally aligns with the rolling mechanism of MagBall.

Smart yarns hold the potential to transform everyday textiles into functional platforms, yet current methods remain constrained. These include conductive yarns, made from silver or stainless steel, which retain the feel of conventional yarns but offer limited functions, and PCB-based solutions, which add capability at the cost of bulk and rigidity. We present Circuit2Yarn, a fabrication framework that transforms planar printed circuits into flexible yarns by rolling copper-traced TPU films with soldered surface-mount components, preserving the capabilities of rigid electronics while producing yarn-like forms suitable for textile integration. We demonstrate yarns as small as 0.8 mm that integrate LEDs and sensors, including temperature, humidity, light, IMU, and capacitive sensing modules, enabling applications ranging from smart garments and interactive musical instruments to responsive tea bags. Characterization confirms durability under bending/stretching. By rolling planar circuits into yarns, Circuit2Yarn paves the way toward comfortable, multifunctional, and interactive textiles in everyday life.

Past research has widely explored the design and fabrication of resistive matrix-based tactile sensors for creating touch-sensitive devices. However, real-world deployment of resistive tactile sensing systems remains difficult for individuals with limited prior experience in embedded sensing due to challenges of portability, adaptivity, and efficiency. We introduce the WiReSens Toolkit, an accessible, open-source platform to bridge this gap. Central to our approach is adaptive hardware for interfacing with resistive sensors and a web-based GUI that streamlines access to advanced features for building scalable tactile sensing systems, including multi-device programming and wireless visualization across three communication protocols, autocalibration for adaptive sensitivity, and intermittent data transmission for low-power use. We validated the toolkit’s usability through a user study with 11 novice participants, who, on average, configured a tactile sensor with over 95% accuracy in under five minutes, calibrated sensors 10× faster than baseline methods, and showed improved sense-making of tactile data.

CHI ’25, Yokohama, Japan

CHI ’25, Yokohama, Japan

In this paper, we present an adaptive walker system designed to address limitations in current intelligent walker technologies. While recent advancements have been made in this field, existing systems often struggle to seamlessly interpret user intent for speed control and lack adaptability across diverse scenarios and terrain. Our proposed solution incorporates high-resolution tactile sensors, deep learning algorithms, IMU sensors, and linear motors to dynamically adjust to the user's intentions and terrain changes. The system is capable of predicting the user's desired speed with an error margin of only 20.99%, relying solely on tactile input from hand and arm contact points. Additionally, it maintains the walker's horizontal stability with an error of less than 1 degree by adjusting leg lengths in response to variations in ground angle. This adaptive walker enhances user safety and comfort, particularly for individuals with reduced strength or cognitive abilities, and offers reliable assistance on uneven terrain such as uphill and downhill paths.

Despite recent exponential advancements in computer vision and reinforcement learning, it remains challenging for robots to interact with liquids. These challenges are particularly pronounced due to the limitations imposed by opaque containers, transparent liquids, fine-grained splashes, and visual obstructions arising from the robot’s own manipulation activities. Yet, there exists a substantial opportunity for robotics to excel in liquid identification and manipulation, given its potential role in chemical handling in laboratories and various manufacturing sectors such as pharmaceuticals or beverages. In this work, we present a novel approach for liquid class identification and state estimation leveraging electrical impedance sensing. We design and mount a digitally embroidered electrode array to a commercial robot gripper. Coupled with a customized impedance sensing board, we collect data on liquid manipulation with a swept frequency sensing mode and a frequency-specific impedance measuring mode. Our developed learning-based model achieves an accuracy of 93.33% in classifying 9 different types of liquids (8 liquids + air), and 97.65% in estimating the liquid state. We investigate the effectiveness of our system with a series of ablation studies. These findings highlight our work as a promising solution for enhancing robotic manipulation in liquid-related tasks.

Tactile sensing plays a pivotal role in human perception and manipulation tasks, allowing us to intuitively understand task dynamics and adapt our actions in real time. Transferring such tactile intelligence to robotic systems would help intelligent agents understand task constraints and accurately interpret the dynamics of both the objects they are interacting with and their own operations. While significant progress has been made in imbuing robots with this tactile intelligence, challenges persist in effectively utilizing tactile information due to the diversity of tactile sensor form factors, manipulation tasks, and learning objectives involved. To address this challenge, we present a unified tactile embedding space capable of predicting a variety of task-centric qualities over multiple manipulation tasks. We collect tactile data from human demonstrations across various tasks and leverage this data to construct a shared latent space for task stage classification, object dynamics estimation, and tactile dynamics prediction. Through experiments and ablation studies, we demonstrate the effectiveness of our shared tactile latent space for more accurate and adaptable tactile networks, showing an improvement of up to 84% over the single-task training.

Touch is an important channel for human-robot interaction, while it is challenging for robots to recognize human touch accurately and make appropriate responses. In this paper, we design and implement a set of large-format distributed flexible pressure sensors on a robot dog to enable natural human-robot tactile interaction. Through a heuristic study, we sorted out 81 tactile gestures commonly used when humans interact with real dogs and 44 dog reactions. A gesture classification algorithm based on ResNet is proposed to recognize these 81 human gestures, and the classification accuracy reaches 98.7%. In addition, an action prediction algorithm based on Transformer is proposed to predict dog actions from human gestures, reaching a 1-gram BLEU score of 0.87. Finally, we compare the tactile interaction with the voice interaction during a freedom human-robot-dog interactive playing study. The results show that tactile interaction plays a more significant role in alleviating user anxiety, stimulating user excitement and improving the acceptability of robot dogs.

In this paper, we introduce MagKnitic, a novel approach to integrate passive force feedback and binary sensing into fabrics via digital machine knitting. Our approach utilizes digital fabrication technology to enable haptic interfaces that are soft, flexible, lightweight, and conform to the user’s body shape. Despite these characteristics, our interfaces provide diverse, interactive, and responsive force feedback, expanding the design space for haptic experiences.MagKnitic provides scalable and customizable passive haptic sensations by utilizing the attractive force between ferromagnetic yarns and permanent magnets, both of which are seamlessly integrated into knitted fabrics. Moreover, we present a binary sensing capability based on the resistance drop resulting from the activated electrical path between the integrated magnets and ferromagnetic yarn upon direct contact. We offer parametric design templates for users to customize MagKnitic layouts and patterns. With various design layouts and combinations, MagKnitic supports passive haptics interactions of linear, polar, angular, planar, radial, and user-defined motions. We perform a technical evaluation of the passive force feedback and the binary sensing capabilities with different machine knitting layouts and patterns, embedded magnet sizes, and interaction distances. In addition, we conduct two user studies to validate the effectiveness of MagKnitic. Finally, we demonstrate various application scenarios, including wearable input interfaces, game controllers, passive VR/AR wearables, and interactive furniture coverings.

Soft actuators with integrated sensing have shown utility in a variety of applications such as assistive wearables, robotics, and interactive input devices. Despite their promise, these actuators can be difficult to both design and fabricate. As a solution, we present a workflow for computationally designing and digitally fabricating soft pneumatic actuators via a machine knitting process. Machine knitting is attractive as a fabrication process because it is fast, digital (programmable), and provides access to a rich material library of functional yarns for specified mechanical behavior and integrated sensing. Our method uses elastic stitches to construct non-homogeneous knitting structures, which program the bending of actuators when inflated. Our method also integrates pressure and swept frequency capacitive sensing structures using conductive yarns. The entire knitted structure is fabricated automatically in a single machine run. We further provide a computational design interface for the user to interactively preview actuators’ quasi-static shape when authoring elastic stitches. Our sensing-integrated actuators are cost-effective, easy to design, robust to large actuation, and require minimal manual post-processing. We demonstrate five use-cases of our actuators in relevant application settings.

Traditional robotic manipulator design methods require extensive, time-consuming, and manual trial and error to produce a viable design. During this process, engineers often spend their time redesigning or reshaping components as they discover better topologies for the robotic manipula-tor. Tactile sensors, while useful, often complicate the design due to their bulky form factor. We propose an integrated design pipeline to streamline the design and manufacturing of robotic manipulators with knitted, glove-like tactile sensors. The proposed pipeline allows a designer to assemble a collection of modular, open-source components by applying predefined graph grammar rules. The end result is an intuitive design paradigm that allows the creation of new virtual designs of manipulators in a matter of minutes. Our framework allows the designer to fine-tune the manipulator's shape through cage-based geometry deformation. Finally, the designer can select surfaces for adding tactile sensing. Once the manipulator design is finished, the program will automatically generate 3D printing and knitting files for manufacturing. We demonstrate the utility of this pipeline by creating four custom manipulators tested on real-world tasks: screwing in a wing nut, pouring water from a bottle, picking up an egg, and cutting paper with scissors.

This paper introduces ActionSense, a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It provides rich, synchronized data streams along with ground truth data to facilitate learning pipelines that could extract insights about how humans interact with the physical world during activities of daily living, and help lead to more capable and collaborative robot assistants. The wearable sensing suite captures motion, force, and attention information; it includes eye tracking with a first-person camera, forearm muscle activity sensors, a body-tracking system using 17 inertial sensors, finger-tracking gloves, and custom tactile sensors on the hands that use a matrix of conductive threads. This is coupled with activity labels and with externally-captured data from multiple RGB cameras, a depth camera, and microphones. The specific tasks recorded in ActionSense are designed to highlight lower-level physical skills and higher-level scene reasoning or action planning. They include simple object manipulations (e.g., stacking plates), dexterous actions (e.g., peeling or cutting vegetables), and complex action sequences (e.g., setting a table or loading a dishwasher). The resulting dataset and underlying experiment framework are available at https://action-sense.csail.mit.edu. Preliminary networks and analyses explore modality subsets and cross-modal correlations. ActionSense aims to support applications including learning from demonstrations, dexterous robot control, cross-modal predictions, and fine-grained action segmentation. It could also help inform the next generation of smart textiles that may one day unobtrusively send rich data streams to in-home collaborative or autonomous robot assistants.

With the recent interest in wearable electronics and smart garments, digital fabrication of sensing and interactive textiles is in increasing demand. Recently, advances in digital machine knitting offer opportunities for the programmable, rapid fabrication of soft, breathable textiles. In this paper, we present KnitUI, a novel, accessible machine-knitted user interface based on resistive pressure sensing. Employing conductive yarns and various machine knitting techniques, we computationally design and automatically fabricate the double-layered resistive sensing structures as well as the coupled conductive connection traces with minimal manual post-processing. We present an interactive design interface for users to customize KnitUI’s colors, sizes, positions, and shapes. After investigating design parameters for the optimized sensing and interactive performance, we demonstrate KnitUI as a portable, deformable, washable, and customizable interactive and sensing platform. It obtains diverse applications, including wearable user interfaces, tactile sensing wearables, and artificial robot skin.

Daily human activities, e.g., locomotion, exercises, and resting, are heavily guided by the tactile interactions between the human and the ground. In this work, leveraging such tactile interactions, we propose a 3D human pose estimation approach using the pressure maps recorded by a tactile carpet as input. We build a low-cost, high-density, large-scale intelligent carpet, which enables the real-time recordings of human-floor tactile interactions in a seamless manner. We collect a synchronized tactile and visual dataset on various human activities. Employing a state-of-the-art camera-based pose estimation model as supervision, we design and implement a deep neural network model to infer 3D human poses using only the tactile information. Our pipeline can be further scaled up to multi-person pose estimation. We evaluate our system and demonstrate its potential applications in diverse fields.

Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We propose a framework aiming at predicting the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, esti-mate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.

We present a novel workflow to design and program knitted garments for industrial whole-garment knitting machines. Inspired by traditional garment making based on cutting and sewing, we propose a sketch representation with additional annotations necessary to model the knitting process. Our system bypasses complex editing operations in 3D space, which allows us to achieve interactive editing of both the garment shape and its underlying time process. We provide control of the local knitting direction, the location of important course interfaces, as well as the placement of stitch irregularities that form seams in the final garment. After solving for the constrained knitting time process, the garment sketches are automatically segmented into a minimal set of simple regions that can be knitted using simple knitting procedures. Finally, our system optimizes a stitch graph hierarchically while providing control over the tradeoff between accuracy and simplicity. We showcase different garments created with our web interface.