Tactile sensing plays a crucial role in robot grasping and manipulation by providing essential contact information between the robot and the environment. In this paper, we present AllTact Fin Ray, a novel compliant gripper design with omni-directional and local tactile sensing capabilities. The finger body is unibody-casted using transparent elastic silicone, and a camera positioned at the base of the finger captures the deformation of the whole body and the contact face. Due to the global deformation of the adaptive structure, existing vision-based tactile sensing approaches that assume constant illumination are no longer applicable. To address this, we propose a novel sensing method where the global deformation is first reconstructed from the image using edge features and spatial constraints. Then, detailed contact geometry is computed from the brightness difference against a dynamically retrieved reference image. Extensive experiments validate the effectiveness of our proposed gripper design and sensing method in contact detection, force estimation, object grasping, and precise manipulation.

Depth reconstruction for transparent objects is a challenging problem, where surface feature matching methods are hindered by complex refraction and reflection. Existing learning-based reconstruction methods by regressing or completing depth maps for entire scenes are data-costly and lack generalization in different environments. To solve this problem, we propose a novel transparent object reconstruction pipeline with a guided object-centric 3D diffusion model. Specifically, we train an unconditional 3D diffusion model with only 3D point cloud data. To control the output of the diffusion model, we design a silhouette-based guidance function and a completion framework with outline points for each step of diffusion process. Specifically, for each step, we design a re-projection pipeline to estimate a silhouette with uncertainty and constrain the partially-noised point cloud to align with it. We further apply stereo matching to compute the outline points in the stereo silhouettes and use a completion framework to fuse them with the partially-denoised point cloud. Finally, we transform the transparent objects to the world frame by applying the transformation from pose estimation. Experiment results show that our method can achieve state-of-the-art performance for transparent object depth reconstruction compared to existing depth regression and completion methods.

To support the increasing demand for smart manufacturing in wire arc additive manufacturing, such as digital twins, high-frequency real-time bead measurement is a long-standing challenge due to the protracted processing time of laser scans and vision-based approaches. This article introduces a pioneering approach for high-frequency, real-time bead geometry measurements. Utilizing high-frequency electric signal sensors, welding current, and voltage are captured. Time and frequency features are subsequently extracted and channeled into multilayer perceptron regressors to predict bead height and width. The model is trained using ground truth data derived from a laser profilometer. Furthermore, a feature dimension reduction algorithm coupled with an incremental learning framework is incorporated to optimize time efficiency and adaptability. Comprehensive practical experiments and a comparative analysis have been conducted. The results demonstrate that the proposed measurement system offers faster measuring speeds than vision-based methods while maintaining accuracy comparable to laser scanning techniques.

Articulated objects are ubiquitous in daily life. In this paper, we present DexSim2Real$^{\mathbf{2}}$, a novel framework for goal-conditioned articulated object manipulation. The core of our framework is constructing an explicit world model of unseen articulated objects through active interactions, which enables sampling-based model predictive control to plan trajectories achieving different goals without requiring demonstrations or RL. It first predicts an interaction using an affordance network trained on self-supervised interaction data or videos of human manipulation. After executing the interactions on the real robot to move the object parts, we propose a novel modeling pipeline based on 3D AIGC to build a digital twin of the object in simulation from multiple frames of observations. For dexterous hands, we utilize eigengrasp to reduce the action dimension, enabling more efficient trajectory searching. Experiments validate the framework's effectiveness for precise manipulation using a suction gripper, a two-finger gripper and two dexterous hands. The generalizability of the explicit world model also enables advanced manipulation strategies like manipulating with tools.

Vision-based tactile sensors have drawn increasing interest in the robotics community. However, traditional lens-based designs impose minimum thickness constraints on these sensors, limiting their applicability in space-restricted settings. In this article, we propose ThinTact, a novel lensless vision-based tactile sensor with a sensing field of over 200 mm${}^{2}$and a thickness of less than 10 mm. ThinTact utilizes the mask-based lensless imaging technique to map the contact information to CMOS signals. To ensure real-time tactile sensing, we propose a real-time lensless reconstruction algorithm that leverages a frequency-spatial-domain joint filter based on discrete cosine transform. This algorithm achieves computation significantly faster than existing optimization-based methods. In addition, to improve the sensing quality, we develop a mask optimization method based on the generic algorithm and the corresponding system matrix calibration algorithm. We evaluate the performance of our proposed lensless reconstruction and tactile sensing through qualitative and quantitative experiments. Furthermore, we demonstrate ThinTact's practical applicability in diverse applications, including texture recognition and contact-rich object manipulation.

Visuotactile sensors can provide rich contact information, having great potential in contact-rich manipulation tasks with reinforcement learning (RL) policies. Sim2Real technique tackles the challenge of RL's reliance on a large amount of interaction data. However, most Sim2Real methods for manipulation tasks with visuotactile sensors rely on rigid-body physics simulation, which fails to simulate the real elastic deformation precisely. Moreover, these methods do not exploit the characteristic of tactile signals for designing the network architecture. In this paper, we build a general-purpose Sim2Real protocol for manipulation policy learning with marker-based visuotactile sensors. To improve the simulation fidelity, we employ an FEM-based physics simulator that can simulate the sensor deformation accurately and stably for arbitrary geometries. We further propose a novel tactile feature extraction network that directly processes the set of pixel coordinates of tactile sensor markers and a self-supervised pre-training strategy to improve the efficiency and generalizability of RL policies. We conduct extensive Sim2Real experiments on the peg-in-hole task to validate the effectiveness of our method. And we further show its generalizability on additional tasks including plug adjustment and lock opening. The protocol, including the simulator and the policy learning framework, will be open-sourced for community usage.

Accurately manipulating articulated objects is a challenging yet important task for real robot applications. In this paper, we present a novel framework called Sim2Real2to enable the robot to manipulate an unseen articulated object to the desired state precisely in the real world with no human demonstrations. We leverage recent advances in physics simulation and learning-based perception to build the interactive explicit physics model of the object and use it to plan a long-horizon manipulation trajectory to accomplish the task. However, the interactive model cannot be correctly estimated from a static observation. Therefore, we learn to predict the object affordance from a single-frame point cloud, control the robot to actively interact with the object with a one-step action, and capture another point cloud. Further, the physics model is constructed from the two point clouds. Experimental results show that our framework achieves about 70% manipulations with < 30% relative error for common articulated objects, and 30% manipulations for difficult objects. Our proposed framework also enables advanced manipulation strategies, such as manipulating with different tools. Code and videos are available on our project webpage: https://ttimelord.github.io/Sim2Real2-site/

Transparent objects are common in daily life. However, depth sensing for transparent objects remains a challenging problem. While learning-based methods can leverage shape priors to improve the sensing quality, the labor-intensive data collection in real world and the sim-to-real domain gap restrict these methods' scalability. In this paper, we propose a method to finetune a stereo network with sparse depth labels automatically collected using a probing system with tactile feedback. We present a novel utility function to evaluate the benefit of touches. By approximating and optimizing the utility function, we can optimize the probing locations given a fixed touching budget to better improve the network's performance on real objects. We further combine tactile depth supervision with a confidence-based regularization to prevent over-fitting during finetuning. To evaluate the effectiveness of our method, we construct a real-world dataset including both diffuse and transparent objects. Experimental results on this dataset show that our method can significantly improve real-world depth sensing accuracy, especially for transparent objects.

Learning-based stereo methods usually require a large scale dataset with depth, however obtaining accurate depth in the real domain is difficult, but groundtruth depth is readily available in the simulation domain. In this article we propose a new framework, ActiveZero++, which is a mixed domain learning solution for active stereovision systems that requires no real world depth annotation. In the simulation domain, we use a combination of supervised disparity loss and self-supervised loss on a shape primitives dataset. By contrast, in the real domain, we only use self-supervised loss on a dataset that is out-of-distribution from either training simulation data or test real data. To improve the robustness and accuracy of our reprojection loss in hard-to-perceive regions, our method introduces a novel self-supervised loss called temporal IR reprojection. Further, we propose the confidence-based depth completion module, which uses the confidence from the stereo network to identify and improve erroneous areas in depth prediction through depth-normal consistency. Extensive qualitative and quantitative evaluations on real-world data demonstrate state-of-the-art results that can even outperform a commercial depth sensor. Furthermore, our method can significantly narrow the Sim2Real domain gap of depth maps for state-of-the-art learning based 6D pose estimation algorithms.

Contextual policy search methods have demonstrated the potential to acquire robotic skill generalization on trajectory-shaping-based tasks. However, it is still challenging for robotic contact-rich manipulation tasks because contact force regulation, reference trajectory adaptation, and task generalization must be fulfilled simultaneously. To this end, a hierarchical compliance-based contextual policy search (HC-CPS) approach is proposed to learn the robotic compliant skills for force, motion, and task adaptation. Specifically, the parameterized impedance-conditioned action space is proposed for reinforcement learning lower-level policy to obtain the compliance for reference motion regulation and contact force control, while a linear Gaussian contextual policy is formulated as the higher-level policy to optimize the context-conditioned impedance parameters for task generalization; therefore, a family of contact-rich manipulation tasks with multiple objectives is achieved. Moreover, data efficiency is further improved by two aspects: first, a variation encoder-decoder model is proposed to estimate the underlying constraints of impedance parameters over the actions, leading to the mitigated extrapolation error for lower-level policy off-policy learning; second, a composite forward model is proposed to generate artificial trajectories and reduce the reward bias for higher-level contextual policy learning. The HC-CPS approach is validated by three simulated manipulation tasks and the real-world dual peg-in-hole assembly tasks with two kinds of objectives, and the results demonstrate the effectiveness of HC-CPS.

In this article, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by the underlying mechanism of the sensors, we designed a fully physics-grounded simulation pipeline that includes material acquisition, ray-tracing-based infrared (IR) image rendering, IR noise simulation, and depth estimation. The pipeline is able to generate depth maps with material-dependent error patterns similar to a real depth sensor in real time. We conduct real experiments to show that perception algorithms and reinforcement learning policies trained in our simulation platform could transfer well to the real-world test cases without any fine-tuning. Furthermore, due to the high degree of realism of this simulation, our depth sensor simulator can be used as a convenient testbed to evaluate the algorithm performance in the real world, which will largely reduce the human effort in developing robotic algorithms. The entire pipeline has been integrated into the SAPIEN simulator and is open-sourced to promote the research of vision and robotics communities.

Reinforcement learning (RL) has been increasingly used for single peg-in-hole assembly, where assembly skill is learned through interaction with the assembly environment in a manner similar to skills employed by human beings. However, the existing RL algorithms are difficult to apply to the multiple peg-in-hole assembly because the much more complicated assembly environment requires sufficient exploration, resulting in a long training time and less data efficiency. To this end, this article focuses on how to predict the assembly environment and how to use the predicted environment in assembly action control to improve the data efficiency of the RL algorithm. Specifically, first, the assembly environment is exactly predicted by a variable time-scale prediction (VTSP) defined as general value functions (GVFs), reducing the unnecessary exploration. Second, we propose a fuzzy logic-driven variable time-scale prediction-based reinforcement learning (FLDVTSP-RL) for assembly action control to improve the efficiency of the RL algorithm, in which the predicted environment is mapped to the impedance parameter in the proposed impedance action space by a fuzzy logic system (FLS) as the action baseline. To demonstrate the effectiveness of VTSP and the data efficiency of the FLDVTSP-RL methods, a dual peg-in-hole assembly experiment is set up; the results show that FLDVTSP-deep Q-learning (DQN) decreases the assembly time about 44% compared with DQN and FLDVTSP-deep deterministic policy gradient (DDPG) decreases the assembly time about 24% compared with DDPG.Note to Practitioners—The complicated assembly environment of the multiple peg-in-hole assembly results in a contact state that cannot be recognized exactly from the force sensor. Therefore, contact-model-based methods that require tuning of the control parameters based on the contact state recognition cannot be applied directly in this complicated environment. Recently, reinforcement learning (RL) methods without contact state recognition have recently attracted scientific interest. However, the existing RL methods still rely on numerous explorations and a long training time, which cannot be directly applied to real-world tasks. This article takes inspiration from the manner in which human beings can learn assembly skills with a few trials, which relies on the variable time-scale predictions (VTSPs) of the environment and the optimized assembly action control strategy. Our proposed fuzzy logic-driven variable time-scale prediction-based reinforcement learning (FLDVTSP-RL) can be implemented in two steps. First, the assembly environment is predicted by the VTSP defined as general value functions (GVFs). Second, assembly action control is realized in an impedance action space with a baseline defined by the impedance parameter mapped from the predicted environment by the fuzzy logic system (FLS). Finally, a dual peg-in-hole assembly experiment is conducted; compared with deep Q-learning (DQN), FLDVTSP-DQN can decrease the assembly time about 44%; compared with deep deterministic policy gradient (DDPG), FLDVTSP-DDPG can decrease the assembly time about 24%.

We introduce VA-Point-MVSNet, a novel visibility-aware point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Furthermore, our visibility-aware multi-view feature aggregation allows the network to aggregate multi-view appearance cues while taking into account visibility. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. The code of VA-Point-MVSNet proposed in this work will be released at https://github.com/callmeray/PointMVSNet.

Current latent representation methods using unsupervised learning have no semantic meaning; thus, it is difficult to directly express their physical task in the real world. To this end, this paper attempts to propose a specified latent representation with physical semantic meaning. First, a few labeled samples are used to generate the framework of the latent space, and these labeled samples are mapped to framework nodes in the latent space. Second, a self-learning method using structured unlabeled samples is proposed to shape the free space between the framework nodes in the latent space. The proposed specified latent representation therefore possesses the advantages provided by both supervised and unsupervised learning. The proposed method is verified by numerical simulations and real-world experiments.

We introduce Point-MVSNet, a novel point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. Our source code and trained models are available at https://github.com/callmeray/PointMVSNet.

The automatic completion of multiple peg-in-hole assembly tasks by robots remains a formidable challenge because the traditional control strategies require a complex analysis of the contact model. In this paper, the assembly task is formulated as a Markov decision process, and a model-driven deep deterministic policy gradient algorithm is proposed to accomplish the assembly task through the learned policy without analyzing the contact states. In our algorithm, the learning process is driven by a simple traditional force controller. In addition, a feedback exploration strategy is proposed to ensure that our algorithm can efficiently explore the optimal assembly policy and avoid risky actions, which can address the data efficiency and guarantee stability in realistic assembly scenarios. To improve the learning efficiency, we utilize a fuzzy reward system for the complex assembly process. Then, simulations and realistic experiments of a dual peg-in-hole assembly demonstrate the effectiveness of the proposed algorithm. The advantages of the fuzzy reward system and feedback exploration strategy are validated by comparing the performances of different cases in simulations and experiments.

This article develops a set space visual servoing method that is quiet different from state-of-the-art approaches. Our approach does not require complex image processing techniques for the extraction, matching and tracking of image features. Instead, it only requires a simple matching algorithm and builds visual errors in set space. Each error is mainly related to one degree of freedom of the camera; therefore, we can design a decoupled control law. This control law is robust and does not require calibrated inner parameters of the camera. Our approach has been validated in 4-degree-of-freedom (DOF) visual servoing simulations with common image patterns and 6-DOF visual servoing experiments with specific image patterns. These visual servoing tasks are properly achieved even when partial occlusions occur.

This paper presents an algorithm to optimize the base position of a mobile manipulator to meet the requirements of local painting tasks. Considering the physical limits and singularity of the manipulator, the feasible base positions are first discretely calculated with the given poses of the end effector by inverse kinematics. Then, the joint-level performance criteria are proposed with respect to the requirements of the painting process. The weight coefficients are also determined by the critic method to balance the contribution of every criterion. Thus, the globally near-optimal base position is selected by sorting all feasible positions according to the evaluation criteria. The experimental results show that the planning result is well executed and has an acceptable computation time, thus demonstrating that the algorithm is both practical and effective compared with previous methods. Note to Practitioners-Finding a proper base position is pivotal for achieving automatic offline planning for a manipulator mounted on a mobile platform. We present an algorithm that leads to an appropriate base position for the manipulator by which the manipulator could reach all the target points with specific orientations and keep a stable velocity of the end effector. This will greatly facilitate the offline planning for mobile manipulator systems such as the painting system implemented in this paper. With no need to manually participate in the algorithm implementation, operators can simply import the path-point data and obtain the optimized base position. The experimental results presented in this paper are encouraging in that the velocity of the end effector is more stable and the planning efficiency is highly increased.

Though the structured light measurement system has been successfully applied to the profile measurement of diffuse objects, it is still a challenge to measure shiny objects due to the mix of both specular and diffuse reflections. To this end, we propose a robust encoding and decoding method in this paper. First, the monochromatic stripe patterns are utilized to eliminate the effect of texture and color of objects. Second, an intensity mask, dynamically adjusting the intensity of a projected pattern, is applied to avoid overexposure without any pre-knowledge of the workpiece. Thus, it is more flexible and efficient, compared with the existing methods. Third, to solve the internal reflection of the shiny part, an extrapolation model, combined with the intensity mask, is developed to detect the stripe edge for pattern decoding, resulting in accurate and robust 3D reconstruction. Compared with traditional polarization based methods, it does not need to readjust for a new part. The experimental results show that the proposed method is capable of measuring various parts without surface pretreatment.

A humanoid neck design with low motion noise yields a cable-driven parallel manipulator to imitate two axis rotational motion of a human neck. The fixed base and moving platform of the manipulator are connected by four cables and a column compression spring. The four cables are actuated separately, while the spring can support weight on the moving platform. Although similar manipulators exist in literature, the analysis for them is scarce. The difficulty is that while traditional parallel manipulator usually has a rigid kinematic chain as the spine, this manipulator has a flexible spring. With the spring's lateral bending motion, new approach must be adopted to solve the kinematics. In this paper, we propose a method which combines the kinematics with the statics to solve them simultaneously. We parameterize the posture of the moving platform with four parameters and consider one of them as parasitic motion. Using the spring bending equation, we can obtain the parasitic motion and solve the inverse position problem. The analysis in this paper provides a novel way to analyze parallel manipulator with a spring spine, and it can be applied to other manipulators with flexible spines.

The objective of this paper is to propose a robust one shot structured light pattern for real time 3D shape inspection system. To reduce the influence of inspected part reflectance property and ambient light, the pattern is constructed by using monochromatic light. The corner of the chessboard is utilized as the primitive of the pattern since it can provide highly accurate position. Additionally, the orientation of the corner is used to encode the primitive of the pattern. Compared with ordinary two dimensional patterns, the pattern is developed in one dimension, along the epipolar line so that the search of the corresponding pixels between the projector and the camera is speeded up. Last, experiments were conducted to evaluate the robustness and accuracy of the inspection system using the proposed pattern. The results demonstrate that the system has high accuracy performance.

Three dimensional optic measurement system's accuracy is highly related with the field of inspection. Increasing of field inspection costs increasing camera / projector pixel area on the test surface. Small surface changes within one pixel area cannot be directly detected, which will lower the system accuracy. A pixel-to-pixel strategy is developed to solve this problem. Increasing field of inspection also costs a longer standoff distance. The random image noise from the environment, uncertainties functions by lens distortion and resolution variation are all amplified. Therefore, a more complicated calibration model for each pixel is proposed to calibrate the system. In traditional structured light vision systems, a single sensor usually detects around 10,000 - 50,000 mm2, and the 3D vision sensor in this paper needs to detect around 2,400,000 mm2. Larger detection range gives more challenge to finish the calibration tasks. This paper proposes a clear calibration procedure to a large field of inspection structured light system. Last the comparison with the CMM measured results is used to prove that the calibration tasks have been successfully achieved.

The non-contact 3D sensing technology, though achieved many success in a variety of applications, needs an automation system to expand its applications to automotive industries for 3D shape inspection. The reason is the difficulty for an operator to find an optimal solution by the manual control of sensor viewpoints. The problem of this industrial application is the capability for the sensing system to simultaneously satisfy all requirements of competence, efficiency, and cost. A robot-aided 3D sensing system can provide such a solution. A CAD-guided robot view planner can automatically generate viewpoints. Measurement accuracy can be satisfied in a certain range. However, the unpredictable image noises still need to be compensated for better measurement performance. In this paper, a feedback planning system is designed and applied to the CAD-guided robot sensor planning system. The feedback controller can automatically evaluate the accuracy of obtained point clouds and generate new viewpoints. This feedback-based inspection system had been successfully implemented in filling holes of a point cloud, caused by shadows and light reflections. Such a system had been implemented on an ABB industrial robot for a 3D measurement of an automotive glass and a pillar. This paper introduces the developed planning system and our current results.

The objective of this paper is to propose a wind- shield surface shape and optical parameters inspection system. In this paper, a white board works as a diffuse planar light source projecting structured light patterns because physical properties of specular surface do not allow us to directly apply the triangulation-based techniques used in ordinary diffuse surface inspection. The board is placed at two different positions and the distorted structured light patterns are observed by a fixed camera so that the incident vector for every point on the windshield surface is determined using corresponding points in the board at two different positions. Likewise, the reflection vector is determined by camera calibration. Hence, the 3D shape and normal of the windshield surface are obtained by the intersection of the incident and reflection vectors. The normal of the surface denotes the optical reflective property of the windshield. Last, accuracy and consistence experiments are conducted. The experiment results demonstrate the efficiency and accuracy of our system.

In the robot-aided ultrasound-guided microwave coagulation therapy system for liver cancer, manual manipulation of the passive robot should be as easy as possible, to improve the efficiency of therapy and popularize such a micro injury therapy technique. The robot manipulation is separated into three steps: the wrist point, the forth joint and the fifth joint are dragged or twisted to their expected position or orientation in turn. During the guide process, the expected position and orientation of robot end-effector can be decided from the focus position and body structure, but expected position or orientation for each manipulating step should be planned through particular calibration and computation. A magnetic tracker with three receivers is installed on the robot to record position and orientation of the robot, which will supply original data to planning and mapping arithmetic of the guide system. The first receiver on the robot base displays position and orientation of the robot base to the magnetic transmitter, which is imperative for the destination planning. Through simple conversion, record of the second receiver on the forearm represents the wrist position, and the third receiver on the end-effector represents real-time position and orientation of the robot tool axis. Based on such an arrangement, planning and mapping arithmetic of the guide system are provided. At last, an independent test with other error sources obviated is introduced to validate accuracy of the system. Test result shows that the error is acceptable, and then proves efficiency of the robot-aid manipulating bed and corresponding arithmetic.