Robotic manipulation tasks commonly rely on computer vision or tactile sensing to extract the physical characteristics of an object. However, this additional sensing capability adds complexity and financial cost to a robotic system. Our work investigates the inexpensive alternative of feature extraction via proprioceptive sensing. Our goal is to determine whether proprioceptive data combined with in-hand-manipulation provides sufficient information to enable geometric reconstruction of object profiles. We use a newly designed 3-DOF robotic gripper with variable-friction finger surfaces to perform model-free in-hand-manipulation on a set of test objects comprised of two dimensional convex prisms. We have devised a manipulation sequence based on the rotation and sliding of test objects to allow side-counting with the successful measurement of shapes and sizes with average angle and size errors of 1.64% and 6.76% respectively. In addition, we have outlined potential research directions aimed at resolving inherent limitations of proprioceptive approaches and making our algorithm generalisable to any arbitrary shape.

Dexterous in-hand manipulation (IHM) for arbitrary objects is challenging due to the rich and subtle contact process. Variable-friction manipulation is an alternative approach to dexterity, previously demonstrating robust and versatile 2D IHM capabilities with only two single-joint fingers. However, the hard-coded manipulation methods for variable friction hands are restricted to regular polygon objects and limited target poses, as well as requiring the policy to be tailored for each object. This paper proposes an end-to-end learning-based manipulation method to achieve arbitrary object manipulation for any target pose on real hardware, with minimal engineering efforts and data collection. The method features a diffusion policy-based imitation learning method with cotraining from simulation and a small amount of real-world data. With the proposed framework, arbitrary objects including polygons and non-polygons can be precisely manipulated to reach arbitrary goal poses within 2 hours of training on an A100 GPU and only 1 hour of real-world data collection. The precision is higher than previous customized object-specific policies, achieving an average success rate of 71.3 % with average pose error being 2.676 mm and 1.902°. Code and videos can be found at: https://sites.google.com/view/vf-ihm-il/home.

Generalizable algorithms for tactile sensing remain underexplored, primarily due to the diversity of sensor modalities. Recently, many methods for cross-sensor transfer between optical (vision-based) tactile sensors have been investigated, yet little work focus on non-optical tactile sensors. To address this gap, we propose an encoder-decoder architecture to unify tactile data across non-vision-based sensors. By leveraging sensor-specific encoders, the framework creates a latent space that is sensor-agnostic, enabling cross-sensor data transfer with low errors and direct use in downstream applications. We leverage this network to unify tactile data from two commercial tactile sensors: the Xela uSkin uSPa 46 and the Contactile PapillArray. Both were mounted on a UR5e robotic arm, performing force-controlled pressing sequences against distinct object shapes (circular, square, and hexagonal prisms) and two materials (rigid PLA and flexible TPU). Another more complex unseen object was also included to investigate the model’s generalization capabilities. We show that alignment in latent space can be implicitly learned from joint autoencoder training with matching contacts collected via different sensors. We further demonstrate the practical utility of our approach through contact geometry estimation, where downstream models trained on one sensor’s latent representation can be directly applied to another without retraining.

The adoption of tactile sensors in robotics is hindered by their high cost and fragility. We designed and validated a cost-effective and robust barometric tactile sensor array, whose material cost is below 80 USD. Unlike past work, we do not mold the rubber surface over the barometers but instead keep it as a separate element, leading to a design that is easy to fabricate and repair. Machine learning techniques are applied to enhance the sensor's localization precision, increasing the effective resolution from 6 mm (the distance between adjacent barometers) to 0.284 mm. To investigate the localization model's robustness, we utilized anE-TRollrobotic gripper to roll differently shaped prismatic objects across the sensing surface mounted on one finger. Under these uncontrolled settings, we achieved a satisfactory average real-time localization resolution of within 2.66 mm. Furthermore, we demonstrate a novel practical application: The E-TRoll mimics a one-DoF parallel gripper inferring a cube's orientation relative to the sensor. The range of orientations is split into four classes, which a trained CNN-LSTM model can predict with an 86.91% five-fold cross-validated accuracy.

With the goal of providing an alternative to optical and other tactile sensors, we set out to stress test the object shape identification capabilities of barometric tactile arrays in robotic manipulation tasks. These sensors are superior to optical devices in terms of form factor, ease of fabrication, and data reading/processing speeds, but lack the necessary spatial resolution to identify surface shapes via a single contact. To compensate, we utilize in-hand-manipulation, specifically in- hand-rolling to identify object shapes via a spatiotemporal approach. To increase task difficulty, we only use three neighboring barometric sensors and designed strict experiment requirements with the purpose of creating a set of extremely confusable test objects. The E- TRoll robotic hand, equipped with a barometric tactile array on one finger, was used to roll test objects within its grasp, taking just under 3.4 seconds for data collection under the fastest tested speed setting, compared to 33 seconds in our previous work. We also designed and implemented a feature extraction algorithm, based and improved upon our recently published algorithm. This captures enough information from the collected spatiotemporal data samples for successful classification with only 13 features. Finally, a bagged tree classification algorithm was trained and optimized with data from 1,164 trials of rolling 9 prismatic test objects, leading to a five-fold cross validation accuracy of 90.5% for identifying the 9 object classes.

Robot grippers are widely used in industrial automation for pick-and-place tasks on a variety of objects. Whilst the majority of commercial grippers are capable of establishing stable grasps, few can perform in-hand-manipulation (IHM). IHM is has the potential to increase robotic motion efficiency, yet most IHM-capable manipulation platforms are anthropomorphic in nature and cost over $10,000, posing a barrier to entry for many. In this work we propose a IHM capable gripper platform that is open-source and may be assembled for £150 ($162) and access to a 3D printer. The gripper consists of two fully actuated 2DOF fingers, each of which is based on a five-bar linkage mechanism with one link extended. The fingers are modular, allowing the gripper to be easily expanded into 3+ finger configurations via simple modification of the central mount. We define the inverse kinematics and effective workspace of the gripper (via the use of Freudenstein equations), providing guidance for translation and rotation of gripped objects. We demonstrate the gripper's ability to manipulate a 1-inch cube's pose within a ±5% error margin and rotate various other YCB objects via open-loop position control.

Fabricating existing and popular open-source adaptive robotic grippers commonly involves using multiple professional machines, purchasing a wide range of parts, and tedious, time-consuming assembly processes. This poses a significant barrier to entry for some robotics researchers and drives others to opt for expensive commercial alternatives. To provide both parties with an easier and cheaper (under £100) solution, we propose a novel adaptive gripper design where every component (with the exception of actuators and the screws that come packaged with them) can be fabricated on a hobby-grade 3D printer, via a combination of inexpensive and readily available PLA and TPU filaments. This approach means that the gripper's tendons, flexure joints and finger pads are now printed, as a replacement for traditional string-tendons and molded urethane flexures / pads. A push-fit systems results in an assembly time of under 10 minutes. The gripper design is also highly modular and requires only a few minutes to replace any part, leading to extremely user-friendly maintenance and part modifications. An extensive stress test has shown a level of durability more than suitable for research, whilst grasping experiments (with perturbations) using items from the YCB object set has also proven its mechanical adaptability to be highly satisfactory.

Screen-based pedestrian navigation assistance can be distracting or inaccessible to users. Shape-changing haptic interfaces can overcome these concerns. The S-BAN is a new handheld haptic interface that utilizes a parallel kinematic structure to deliver 2-DOF spatial information over a continuous workspace, with a form factor suited to integration with other travel aids. The ability to pivot, extend and retract its body opens possibilities and questions around spatial data representation. We present a static study to understand user perception of absolute pose and relative motion for two spatial mappings, showing the highest sensitivity to relative motions in the cardinal directions. We then present an embodied navigation experiment in virtual reality (VR). User motion efficiency when guided by the S-BAN was statistically equivalent to using a vision-based tool (a smartphone proxy). Although haptic trials were slower than visual trials, participants’ heads were more elevated with the S-BAN, allowing greater visual focus on the environment.

Robotic tactile sensing provides a method of recognizing objects and their properties where vision fails. Prior work on tactile perception in robotic manipulation has frequently focused on exploratory procedures (EPs). However, the also-human-inspired technique of in-hand-manipulation can glean rich data in a fraction of the time of EPs. We propose a simple 3-DOF robotic hand design, optimized for object rolling tasks via a variable-width palm and associated control system. This system dynamically adjusts the distance between the finger bases in response to object behavior. Compared to fixed finger bases, this technique significantly increases the area of the object that is exposed to finger-mounted tactile arrays during a single rolling motion (an increase of over 60% was observed for a cylinder with a 30-millimeter diameter). In addition, this paper presents a feature extraction algorithm for the collected spatiotemporal dataset, which focuses on object corner identification, analysis, and compact representation. This technique drastically reduces the dimensionality of each data sample from$\boldsymbol{10\times 1500}$time series data to 80 features, which was further reduced by Principal Component Analysis (PCA) to 22 components. An ensemble subspace k-nearest neighbors (KNN) classification model was trained with 90 observations on rolling three different geometric objects, resulting in a three-fold cross-validation accuracy of 95.6% for object shape recognition.

Attempts to achieve robotic Within-Hand-Manipulation (WIHM) generally utilize either high-DOF robotic hands with elaborate sensing apparatus or multi-arm robotic systems. In prior work we presented a simple robot hand with variable friction robot fingers, which allow a low-complexity approach to within-hand object translation and rotation, though this manipulation was limited to planar actions. In this work we extend the capabilities of this system to 3D manipulation with a novel region-based WIHM planning algorithm and utilizing extrinsic contacts. The ability to modulate finger friction enhances extrinsic dexterity for three-dimensional WIHM, and allows us to operate in the quasi-static level. The region-based planner automatically generates 3D manipulation sequences with a modified A* formulation that navigates the contact regions between the fingers and the object surface to reach desired regions. Central to this method is a set of object-motion primitives (i.e. within-hand sliding, rotation and pivoting), which can easily be achieved via changing contact friction. A wide range of goal regions can be achieved via this approach, which is demonstrated via real robot experiments following a standardized in-hand manipulation benchmarking protocol.

In this paper we present a novel method of categorizing naturalistic human arm motions during activities of daily living using clustering techniques. While many current approaches attempt to define all arm motions using heuristic interpretation, or a combination of several abstract motion primitives, our unsupervised approach generates a hierarchical description of natural human motion with well recognized groups. Reliable recommendation of a subset of motions for task achievement is beneficial to various fields, such as robotic and semi-autonomous prosthetic device applications. The proposed method makes use of well-known techniques such as dynamic time warping (DTW) to obtain a divergence measure between motion segments, DTW barycenter averaging (DBA) to get a motion average, and Ward's distance criterion to build the hierarchical tree. The clusters that emerge summarize the variety of recorded motions into the following general tasks: reach-to-front, transfer-box, drinking from vessel, on-table motion, turning a key or door knob, and reach-to-back pocket. The clustering methodology is justified by comparing against an alternative measure of divergence using Bezier coefficients and K-medoids clustering.

The human wrist contributes greatly to the mobility of the arm/hand system, empowering dexterity and manipulation capabilities. However, both robotic and prosthetic research communities tend to favor the study and development of end-effectors/terminal devices (hands, grippers, etc.) over wrists. Wrists can improve manipulation capabilities, as they can orient the end-effector of a system without imparting significant translational motion. In this paper, we review the current state of the art of wrist devices, ranging from passive wrist prostheses to actuated robotic wrist devices. We focus on the mechanical design and kinematic arrangements of said devices and provide specifications when available.

Flatland was an immersive “in-the-wild” experimental theater and technology project, undertaken with the goal of developing systems that could assist “real-world” pedestrian navigation for both vision-impaired (VI) and sighted individuals, while also exploring inclusive and equivalent cultural experiences for VI and sighted audiences. A novel shape-changing handheld haptic navigation device, the “Animotus,” was developed. The device has the ability to modify its form in the user's grasp to communicate heading and proximity to navigational targets. Flatland provided a unique opportunity to comparatively study the use of novel navigation devices with a large group of individuals (79 sighted, 15 VI) who were primarily attending a theater production rather than an experimental study. In this paper, we present our findings on comparing the navigation performance (measured in terms of efficiency, average pace, and time facing targets) and opinions of VI and sighted users of the Animotus as they negotiated the 112 m2production environment. Differences in navigation performance were nonsignificant across VI and sighted individuals and a similar range of opinions on device function and engagement spanned both groups. We believe more structured device familiarization, particularly for VI users, could improve performance and incorrect technology expectations (such as obstacle avoidance capability), which influenced overall opinion. This paper is intended to aid the development of future inclusive technologies and cultural experiences.

Immersive theatre is a growing trend within theatre entertainment: audience members can now wander around performances and choose how the story unfolds in front of them. Technology can be used to create novel, multi-modal experiences for audiences in these performances; but when the rules of such an experience are ill-defined, how do users react to this technology? We present an evaluation of 25 performances of an immersive, in the dark performance. Issues of control can arise in situations where technology becomes an important part of such a performance. Participants take and relinquish control in three key areas: navigation, exploration and attention during the performance, and this affects their perception of both technology and the piece itself. We discuss how technology can play a positive role in immersive theatre and other cultural settings, yet its use must be carefully choreographed to ensure the audience experience matches the intended goal.

This paper presents a minimalistic handheld haptic interface designed to provide pedestrian navigation assistance via the intuitive and unobtrusive stimulus of shape-changing. The new device, named the Haptic Taco, explores a novel region of robotic interfaces which we believe to have benefits over other communication methods. In previous work, we demonstrated the use of a 2DOF shape changing interface for navigation without the use of sight. In this paper we seek to explore the potential of minimal 1DOF interfaces, whose simplicity may increase intuitiveness and performance despite conveying less information. The Haptic Taco utilizes the same `variable volume' concept as a previous device, the Haptic Lotus (2010), but with reduced body compliance and higher force exertion capability. Both devices modulate their perceived volume in relation to proximity to a navigational target (a destination or waypoint). As users walk within an environment, they also attempt to minimize the device volume, finding targets via an embodied `steepest descent' method. Experimental comparison of the Lotus and Taco in a target-finding study revealed that the Taco interface increased motion path efficiency by 24% over the Lotus, to 47% average efficiency. This result is highly comparable to the mean motion efficiency of 43.6-48% observed in prior experiments with the 2DOF shape-changing interface, the Animotus. The findings indicate the potential for minimalistic interfaces in this emerging field.

It has been frequently argued that the addition of haptic sensing to tele-operated surgical robots would benefit surgeon performance. Conventional haptic sensing technologies are impractical for application to minimally invasive surgery, due to size, sterilization robustness and cost vs. tool disposability. In this work we validate the concept of remote force measurement, where force interactions at the tip of a surgical tool are observed via simple torque sensors near the tool's actuators. This method provides reusable sensors located outside of the human body and so sidesteps many key issues that have limited practical haptic sensing in this scenario. Though such methods have been proposed and criticized by several groups in the past, we have been unable to locate quantitative results in the literature. Here, we provide initial remote force interaction measurements on a da Vinci EndoWrist needle driver tool. The measurements were obtained via simple custom torque sensors which easily attach to any EndoWrist tool. The complex cable-pulley transmission of the tool introduces expected nonlinearities over distal measurements. These effects are more pronounced in flexion than abduction DOFs. Despite these effects, the unprocessed torque data identifies contact with synthetic soft tissue at various actuator velocities and during external shaft loading.

In this paper we present a methodology for discriminating between different objects using only a single force closure grasp with an underactuated robot hand equipped with force sensors. The technique leverages the benefits of simple, adaptive robot grippers (which can grasp successfully without prior knowledge of the hand or the object model), with an advanced machine learning technique (Random Forests). Unlike prior work in literature, the proposed methodology does not require object exploration, release or re-grasping and works for arbitrary object positions and orientations within the reach of a grasp. A two-fingered compliant, underactuated robot hand is controlled in an open-loop fashion to grasp objects with various shapes, sizes and stiffness. The Random Forests classification technique is used in order to discriminate between different object classes. The feature space used consists only of the actuator positions and the force sensor measurements at two specific time instances of the grasping process. A feature variables importance calculation procedure facilitates the identification of the most crucial features, concluding to the minimum number of sensors required. The efficiency of the proposed method is validated with two experimental paradigms involving two sets of fabricated model objects with different shapes, sizes and stiffness and a set of everyday life objects.

The human finger pad is known to be highly sensitive to lateral skin deformation, which is present during the palpation of soft objects containing hard lumps. This study investigates the effects of this lateral information on the accuracy and reliability of lump localization through a teletaction system. The results are expected to benefit future telesurgical robots for remote palpation. A previously proven deformation-based tactile feedback system was modified to include laterotactile feedback using a new version of a biologically-inspired fingertip sensor (TACTIP) and a remotely-actuated electromechanical shape display. These novel devices were mounted on to a newly developed teleoperated system equipped with visual and force feedback, through which test subjects palpated a number of soft artificial tissue samples. The level of lateral feedback was altered between trials, with constant visual, force, and normal-direction tactile feedback provided throughout. The surprising results show that the addition of lateral feedback offered no benefit to the subjects' ability to detect and localise embedded objects. Reasons for this observation are discussed. It is concluded that normal-direction tactile feedback is likely to be sufficient for lump detection even when lateral motions are made, although feedback from the subjects indicates that the addition benefited their perception of interactions with the system.

An immersive theatre experience was designed to raise awareness and question perceptions of 'blindness', through enabling both sighted and blind members to experience a similar reality. A multimodal experience was created, comprising ambient sounds and narratives -- heard through headphones -- and an assortment of themed tactile objects, intended to be felt. In addition, audience members were each provided with a novel haptic device that was designed to enhance their discovery of a pitch-black space. An in the wild study of the cultural experience showed how blind and sighted audience members had different 'felt' experiences, but that neither was a lesser one. Furthermore, the haptic device was found to encourage enactive exploration and provide reassurance of the environment for both sighted and blind people, rather than acting simply as a navigation guide. We discuss the potential of using haptic feedback to create cultural experiences for both blind and sighted people; rethinking current utilitarian framing of it as assistive technology.