Independent control of microrobots is a cardinal challenge for manipulation at micro/nano scale. In this paper, we design and assemble an electromagnetic setup to overcome some of the major obstacles in the independent control of microrobots. The demanding magnetic requirements are met by the presented experimental testbed that is able to produce magnetic fields and gradients of, respectively, 160 mT and 3.6 T/m at the center of the workspace. Through the design process of this testbed, we analyze the importance of design parameters and derive a quantitative analysis of the requirements for the dissipation of the generated heat. Further, we present and develop the model and software infrastructure, capable of running at 25 Hz, necessary for independent control of multiple microrobots. We also introduce two novel techniques for current-minimizing mapping of the desired forces into currents at the electromagnet. Finally, the capabilities of the setup are demonstrated through independent control of two, both identical and nonidentical, soft-magnetic microspheres in three-dimensional space—with average root mean square errors of 102$\mu$m and peak velocities of up to 331$\mu$m/s.

Minimally invasive surgery can benefit greatly from utilizing micro-agents. These miniaturized agents need to be clearly visualized and precisely controlled to ensure the success of the surgery. Since medical imaging modalities suffer from low acquisition rate, multi-rate sampling methods can be used to estimate the intersample states of micro-agents. Hence, the sampling rate of the controller can be virtually increased even if the position data is acquired using a slow medical imaging modality. This study presents multi-rate Luenberger and Kalman state estimators for visual tracking of micro-agents. The micro-agents are tracked using sum of squared differences and normalized cross correlation based visual tracking. Further, the outputs of the two methods are merged to minimize the tracking error and prevent tracking failures. During the experiments, the micro-agents with different geometrical shapes and sizes are imaged using a 2D ultrasound machine and a microscope, and manipulated using electromagnetic coils. The multi-rate state estimation accuracy is measured using a high speed camera. The precision of the tracking and multi-rate state estimation are verified experimentally under challenging conditions. For this purpose, an elliptical shaped magnetic micro-agent with a length of 48 pixels is used. Maximum absolute error in x and y axes are 2.273 and 2.432 pixels for an 8-fold increase of the sample rate (25 frames per second), respectively. During the experiments, it was observed that the micro-agents could be tracked more reliably using normalized cross correlation based visual tracking and inters ample states could be estimated more accurately using Kalman state estimator. Experimental results show that the proposed method could be used to track micro-agents in medical imaging modalities and estimate system states at intermediate time instants in real-time.

Soft miniaturized untethered grippers can be used to manipulate and transport biological material in unstructured and tortuous environments. Previous studies on control of soft miniaturized grippers employed cameras and optical images as a feedback modality. However, the use of cameras might be unsuitable for localizing miniaturized agents that navigate within the human body. In this paper, we demonstrate the wireless magnetic motion control and planning of soft untethered grippers using feedback extracted from B-mode ultrasound images. Results show that our system employing ultrasound images can be used to control the miniaturized grippers with an average tracking error of 0.4±0.13 mm without payload and 0.36±0.05 mm when the agent performs a transportation task with a payload. The proposed ultrasound feedback magnetic control system demonstrates the ability to control miniaturized grippers in situations where visual feedback cannot be provided via cameras.

The human hand represents a complex fascinating system with highly sensitive sensory capabilities and dexterous grasping and manipulation functionalities. As a consequence, estimating the hand pose and at the same time having the capability to provide haptic feedback in a wearable way may benefit areas such as rehabilitation, human-robot interaction, gaming, and many more. Existing solutions allow us to accurately measure the hand configuration and provide effective force feedback to the user. However, they have limited wearability/portability. In this paper, we present the wearable sensing/actuation system glove for enhanced sensing and touching (GESTO). It is based on inertial and magnetic sensors for hand tracking, coupled with cutaneous devices for the force feedback rendering. Unlike vision-based tracking systems, the sensing glove does not suffer from occlusion problems and lighting conditions. We properly designed the cutaneous devices in order to reduce possible interferences with the magnetic sensors and performed an experimental validation on ten healthy subjects. In order to measure the estimation accuracy of GESTO, we used a high-precision optical tracker. A comparison between using the glove with and without the haptic devices shows that the presence of them does not induce a statistically significant increase in the estimation error. Experimental results revealed the effectiveness of the proposed approach. The accuracy of our system, 3.32° mean estimation error in the worst case, is comparable with the human ability of discriminating finger joint angle.

In this paper, we present a novel cooperative navigation control for human-robot teams. Assuming that a human wants to reach a final location in a large environment with the help of a mobile robot, the robot must steer the human from the initial to the target position. The challenges posed by cooperative human-robot navigation are typically addressed by using haptic feedback via physical interaction. In contrast with that, in this paper, we describe a different approach, in which the human-robot interaction is achieved via wearable vibrotactile armbands. In the proposed work, the subject is free to decide her/his own pace. A warning vibrational signal is generated by the haptic armbands when a large deviation with respect to the desired pose is detected by the robot. The proposed method has been evaluated in a large indoor environment, where 15 blindfolded human subjects were asked to follow the haptic cues provided by the robot. The participants had to reach a target area, while avoiding static and dynamic obstacles. Experimental results revealed that the blindfolded subjects were able to avoid the obstacles and safely reach the target in all of the performed trials. A comparison is provided between the results obtained with blindfolded users and experiments performed with sighted people.

Micro-sized agents can benefit robotic minimally invasive surgery since they can be inserted into the human body and use natural pathways such as arteries and veins or the gastrointestinal tract, to reach their target for drug delivery or diagnosis. Recently, miniaturized agents with shape-changing and gripping capabilities have provided significant advantages in performing grasping, transportation, and manipulation tasks. In order to robustly perform such tasks, it is of utmost importance to properly estimate their overall configuration. This paper presents a novel solution to the problem of estimating and tracking the 3D position, orientation and configuration of the tips of miniaturized grippers from RGB marker-less visual observations obtained by a microscope. We consider this as an optimization problem, seeking for the gripper model parameters that minimize the discrepancy between hypothesized instances of the gripper model and actual observations of the miniaturized gripper. This optimization problem is solved using a variant of the Particle Swarm Optimization algorithm. The proposed approach has been evaluated on several image sequences showing the grippers moving, rotating, opening/closing and grasping biological material.

In the last years, wearable haptic technologies became very promising since they provide the users with tactile force feedback via small and wearable interfaces. However, they have no position sensing thus additional technologies like motion capture systems or expensive gloves are required. Recently, low cost hand tracking systems based on RGB-D cameras have been developed, however they suffer from lighting conditions and occlusions. In this paper we present a sensing glove based on inertial and magnetic sensors for hand tracking which can be combined with cutaneous devices for the rendering of the force feedback, thus producing a wearable sensing/actuation system. The sensing glove does not suffer from occlusion problems, it is wearable and cost effective however, since the employed sensors use the magnetometer to compensate the drift, they are sensitive to variations in the magnetic field. This property makes it challenging to interface the sensing glove with wearable devices since their motors generate variations in the magnetic field. Preliminary experiments showed the effectiveness of using inertial and magnetic sensors for hand tracking. A comparison between using the glove with and without the haptic devices was presented in order to compare the tracking performance when cutaneous devices are used.

In this paper, we present a haptic guidance policy to steer the user along predefined paths, and we evaluate a predictive approach to compensate actuation delays that humans have when they are guided along a given trajectory via sensory stimuli. The proposed navigation policy exploits the nonholonomic nature of human locomotion in goal directed paths, which leads to a very simple guidance mechanism. The proposed method has been evaluated in a real scenario where seven human subjects were asked to walk along a set of predefined paths, and were guided via vibrotactile cues. Their poses as well as the related distances from the path have been recorded using an accurate optical tracking system. Results revealed that an average error of 0.24 m is achieved by using the proposed haptic policy, and that the predictive approach does not bring significant improvements to the path following problem for what concerns the distance error. On the contrary, the predictive approach achieved a definitely lower activation time of the haptic interfaces.

The flying hand is a robotic hand consisting of a swarm of UAVs able to grasp an object where each UAV contributes to the grasping task with a single contact point at the tooltip. The swarm of robots is teleoperated by a human hand whose fingertip motions are tracked, e.g., using an RGB-D camera. We solve the kinematic dissimilarity of this unique master-slave system using a multi-layered approach that includes: a hand interpreter that translates the fingertip motion in a desired motion for the object to be manipulated; a mapping algorithm that transforms the desired object motions into a suitable set of virtual points deviating from the planned contact points; a compliant force control for the case of quadrotor UAVs that allows to use them as indirect 3D force effectors. Visual feedback is also used as sensory substitution technique to provide a hint on the internal forces exerted on the object. We validate the approach with several human-in-the-loop simulations including the full physical model of the object, contact points and UAVs.

This paper proposes a novel use of haptic feedback for human navigation with a mobile robot. Assuming that a path-planner has provided a mobile robot with an obstacle-free trajectory, the vehicle must steer the human from an initial to a desired target position by only interacting with him/her via a custom-designed vibro-tactile bracelet. The subject is free to decide his/her own pace and a warning vibrational signal is generated by the bracelet only when a large deviation with respect to the planned trajectory is detected by the vision sensor on-board the robot. This leads to a cooperative navigation system that is less intrusive, more flexible and easy-to-use than the ones existing in literature. The effectiveness of the proposed system is demonstrated via extensive real-world experiments.

This paper presents a new algorithm based on previous results of the authors, for the estimation of the yaw angle of an omnidirectional camera/robot undergoing a 6-DoF rigid motion. Our real-time algorithm is uncalibrated, robust to noisy data, and it only relies on the projection of 3-D parallel lines as image features. Numerical and real-world experiments conducted with an eye-in-hand robot manipulator, which we used to simulate the 3-D motion of a Micro unmanned Aerial Vehicle (MAV), show the accuracy and reliability of our estimation algorithm.

The Kuka Control Toolbox (KCT) is a collection of MATLAB functions for motion control of KUKA robot manipulators, developed to offer an intuitive and high-level programming interface to the user. The toolbox, which is compatible with all 6 DOF small and low payload KUKA robots that use the Eth.RSIXML, runs on a remote computer connected with the KUKA controller via TCP/IP. KCT includes more than 30 functions, spanning operations such as forward and inverse kinematics computation, point-to-point joint and Cartesian control, trajectory generation, graphical display and diagnostics. The flexibility, ease of use and reliability of the toolbox is demonstrated through two applicative examples.

Haptic interfaces in Augmented Reality scenarios are an efficient way to enhance the interaction with a virtual world. We introduce a new system that involves both hands in virtual object experience. We render contemporary shape and weight of a virtual object, merged in a real world environment, using a commercial haptic interface and a new kind of finger-worn device. We test the system with an experimental setup where a virtual cube is placed over a real marker. The system developed can be used in several applications of augmented reality where rendering only the object shape is not enough to perceive the virtual world. It can also lead towards the introduction of more wearable devices in the haptics augmented reality context.

Planar catadioptric stereo vision sensors (PCS) combine a pinhole camera with two or more planar mirrors. PCS have recently received an increasing attention since a stereo view can be easily obtained without the need of exact multi-camera synchronization and calibration. In this paper we present a rigorous analytical treatment of the imaging geometry of PCS, propose new mirror calibration algorithms and introduce new multi-view properties that can be used for eye-in-hand camera localization. The effectiveness of the algorithms is shown via extensive simulation and real-data experiments on a robotic manipulator.