Visual localization is essential in many vision-driven interaction domains, including automation, AR (augmented reality), and surgical navigation. This paper presents a visual marker specifically designed for deformable surface tracking. It has three advantages: 1) IFPs (inner false positives) detection is avoided through grayscale integration design, leading to more robust results. 2) Without relying on thresholds, our detection algorithm enhances the method’s reliability in deformable surface with complex shading. 3) We use a position-sensing marker design with higher information density than the self-identifing marker to ensure the supply of features. In the experiments, our marker achieved superior localization accuracy and zero IFPs. Additionally, we present two compelling case studies showcasing the marker’s practical applications in augmented reality and surgical instrument tracking. Our work offers a significant advancement in visual localization, especially in challenging scenarios involving deformable surfaces, providing valuable solutions for researchers and developers across various application domains.

This paper presents a learning-based approach for accurately estimating the 3D shape of flexible continuum robots subjected to external loads. The proposed method introduces a spatiotemporal neural network architecture that fuses multimodal inputs, including current and historical tendon displacement data and RGB images, to generate point clouds representing the robot’s deformed configuration. The network integrates a recurrent neural module for temporal feature extraction, an encoding module for spatial feature extraction, and a multimodal fusion module to combine spatial features extracted from visual data with temporal dependencies from historical actuator inputs. Continuous 3D shape reconstruction is achieved by fitting Bézier curves to the predicted point clouds. Experimental validation demonstrates that our approach achieves high precision, with mean shape estimation errors of 0.08 mm (unloaded) and 0.22 mm (loaded), outperforming state-of-the-art methods in shape sensing for TDCRs. The results validate the efficacy of deep learning-based spatiotemporal data fusion for precise shape estimation under loading conditions.

Non-cooperative target herding is one of the concerns in the robotics field. For the non-cooperative target herding problem, a general planning framework is proposed to compel the target to the destination by a mobile robot team with pursuit, encirclement, and guidance operations. In the proposed planning framework, the encirclement strategy and guidance strategy are designed to deal with the unpredictability of the target. Firstly, the mobile robots approach the randomly moving target in the pursuit process. Secondly, the encirclement strategy is applied in the encirclement process to enable the mobile robot team to encircle the non-cooperative target. Finally, the guidance strategy is employed in the guidance process to enable the mobile robot team to form a favorable encirclement formation and compel the non-cooperative target to the destination. During the herding task, the mobile robots encircle the target or other mobile robots in a satellite-like motion. Moreover, two queues (orbiters and wanders) are maintained to change the formation adaptively, improving the flexibility of the framework. Various environments are designed to verify the effectiveness of the planning framework both in simulations and real-world experiments. Furthermore, the proposed framework is a general platform having great potential for various applications, where the numbers of mobile robots in the herding team are changeable, and different planning algorithms can be integrated into the framework.Note to Practitioners—The motivation for this paper is to propose a planning framework to efficiently herd the non-cooperative target in various scenarios for applications such as the maintenance of public safety and the facilitation of wildlife migration. The existing cooperative herding methods typically maintain relatively fixed formations, while the planning framework proposed in this paper introduces a satellite-based motion pattern and diverse formation transformation strategies, thereby enhancing formation flexibility and adaptability. Consequently, this framework can be effectively applied to challenging environments such as those with restricted areas and dynamic obstacles. Furthermore, the proposed planning framework allows for flexible adjustments in path planning methods and the number of mobile robots to meet various practical application requirements. Through a series of simulations and real-world experiments, the reliability and favorable performance of the proposed algorithm framework in practical applications are demonstrated.

The refined aerial operations of the unmanned aerial manipulator (UAM) have been extensively studied for the last decades. Usually, UAM operations are accompanied by several phases, such as free flight, contact operations, and separation. A great challenge is proposed for switching operations in different environments and the high-precision contact force requirements of UAM control, so this paper conducts control stability research in the case of dynamic differences between free flight and contact operation for the UAM system. First, a hybrid force/position control strategy is proposed for switchable UAM system, among them, the adaptive sliding mode control method based on the interference observer is utilized in the free flight phase, and the adaptive impedance force control method is employed in the contact operation phase, where the adaptive estimation method is designed to perform on-board manipulator contact force estimation. Then a robust adaptive control strategy is proposed for the attitude loop to compensate for the torque disturbance generated during the contact operation phase. Meanwhile, the stability of the switching system is analyzed through the continuous Lyapunov function to prove the stability of the switching process. Finally, the effectiveness and superiority of the proposed schemes are verified through contact operation simulations and experiments.Note to Practitioners—This work is motivated by the contact force tracking of an UAM without force sensor. In recent years, hybrid force/position control has been widely used in UAM. However, force measurement is required at the end-effector and the object in most studies. The proposed method divides the contact operation process into free flight and contact operation stages. In the free flight stage, it is not necessary to know the prior information of the environment accurately (i.e. the external disturbance of slowly varying or known upper bound), and the disturbance observer is adopted to compensate for the disturbance caused by the external environment. In the contact operation stage, the impedance force control method is used for force tracking, which reduces the complexity and quality of the end-effector. At the same time, The shock caused by the switch from free flight to contact operation is reduced by calculating the appropriate controller parameters using the Lyapunov function. The proposed method is a promising solution for real applications and is validated via simulation and indoor contact experiments. The experimental results show that the proposed method has better stability and accuracy than the existing methods, and can be extended to industrial inspection, component processing, aerial operation, etc.

An n-order marker field is a special binary matrix whose n×n subregions are all distinct from each other in four orientations. It is commonly used to guide the composing process of position-sensing markers, which can be detected and identified in a camera image with very limited scope or severe visibility problems. Despite the advantages, position-sensing markers are rare and overlooked because generating marker fields is difficult. In this article, we broaden the definition of marker field, making it more powerful and flexible. Then, we propose bWFC (binary wave function collapse) and its high-speed version, fast-bWFC, to solve the generation problem. The methods are packaged into an open-sourced toolkit named HydraMarker, with which, users not only can generate marker fields on laptops within a short period of time, but also can highly customize them: preset values; fields and subregions in any shape; multifold local uniqueness. Comparative results indicate that the proposed method has superior efficiency, quality, and capability. It makes marker field generation accessible to common marker designers, opening up more possibilities for fiducial markers.

This article presents a trajectory tracking control method for the unmanned aerial manipulator robot system (UAMRS) under model uncertainty and unknown disturbances. More specifically, a low-complexity prescribed performance controller is proposed to effectively reduce the design complexity and achieve the prescribed transient and steady-state performance. First, the dynamics model of the UAMRS is analyzed and modeled, where the unmeasured internal interaction generated by the coupling effect and the random environmental disturbances are considered simultaneously. Then, utilizing the property of prescribed performance, the UAMRS with model uncertainty and external disturbances can guarantee preferable trajectory tracking responses, where the nonlinear disturbance observer is used to estimate and compensate uncertainties and external disturbances. Moreover, the proposed controller defined by simple expressions does not require accurate knowledge of the UAMRS, which is of low complexity and can effectively reduce the amount of calculation. The stability of the proposed controller is analyzed. Finally, the performances of the proposed scheme are demonstrated by the numerical simulation comparisons and real-world experiments, where a quadrotor with a 3-DOF onboard active manipulator is adopted in outdoor experimental validations.

In this paper, we propose a novel edge consistency metric for multi-modal correspondence. It is based on a novel observation on image truncated SVD (singular value decomposition) termed regression robustness, which describes the fact that, a good approximation from image truncated SVD can be inherited even if the eigen-images change due to expansion and channel-dependent offsets. Compared to state-of-the-arts, multi-modal edge consistency metric can simultaneously handle multiple images with complex modality changes, including local variation, gradient reverse, intensity order change, and texture loss. Its complexity is almost linear to pixel number. Remarkable accuracies have been achieved in experiments.

Common assumptions in most of the previous nonlinear networked leader–follower systems are that the leader is provided as a constant signal and that the controllers are designed for either the joint-space regulation or the translational Cartesian-space motion control. This article addresses the control issue of the delay-induced horizontal shift effect when the leader is moving with a changing speed. A novel mixed-type feedback is introduced to reduce the horizontal shift effect so as to minimize tracking errors experienced by the follower agents. In this article, we also study the complete pose control using a nonsingular terminal sliding mode (NTSM) method. A stability analysis is provided to prove the finite-time boundedness of the tracking error signals. Quantitative evaluations of the multiple effects on the error bound are conducted to facilitate the subsequent control design to improve the performance. Numerical simulation results of a team of two degree-of-freedoms (DOFs) manipulators demonstrate the improved tracking performance of the end effectors with small bounded errors which are affected by network delays, maximum assigned accelerations, and the selection of control gains. The experimental results are provided to demonstrate the performance of the developed controller.

Dark channel prior is a highly valued theory mostly used in the area of haze removal. It reveals the wide existence of pixels with very small channel. However, the definition is in a decorated way that relies on the concept of dark channel. The dark channel works like an image domain wherein non-sky regions of clear outdoor images are close to zero. In this letter we show that, such a concept is unnecessary and misleading, and can be improved by using a more basic concept named dark pixel. A novel transmission estimation method based on finding dark pixels is proposed. The method is concise and easy to understand. In comparisons with state-of-the-art methods, our method shows significant improvements on both transmission estimation and haze removal results.

The vision‐based mobile robot's simultaneous localisation and mapping and navigation capability in dynamic environments are highly problematic elements of robot vision applications. The goal of this study is to reconstruct a static map and track the dynamic object for a camera and laser scanner system. An improved automatic calibration is designed to merge image and laser point clouds. Then, the fusion data is exploited to detect the slowly moved object and reconstruct static map. Tracking‐by‐detection requires the correct assignment of noisy detection results to object trajectories. In the proposed method, occluded regions are combined 3D motion models with object appearance to manage difficulties in crowded scenes. The proposed method was validated by experimental results gathered in a real environment and on publicly available data.

Estimating the three‐dimensional (3D) motion from sparse laser point clouds is a highly challenging endeavour facing computer and robotic vision engineers. In this study, a novel method is proposed for robustly estimating the scene flow from a laser scanner assisted by a camera. Conditional random field (CRF) is constructed by a spatial structure of point clouds, the energy of which is minimised by a synchronous calibrated image. With the high frame rate of a laser scanner, the authors’ method allows for estimating the potential motion field as the CRF label. The authors ran an experiment on a public dataset to demonstrate that their method can accurately estimate rigid motion in outdoor scenes. They also tested the method on a laser scanner and omni‐directional camera system to find that it also accurately estimates the rigid and semi‐rigid motion of objects in a controlled indoor environment.

Hazy images have limited visibility and low contrast. The degradation is expressed by transmission map, which is one of the most important estimates of single image dehazing. Transmission map estimation is an underconstraint problem, and lots of priors have been proposed. Among them, the dark channel prior is widely recognized. However, traditional methods have not fully exploited its power due to improper assumptions or operations, which cause unwanted artifacts. The postrefinement algorithms employed to remove these artifacts in turn undermine the merits of the prior. In this letter, a novel method for estimating transmission map by energy minimization is proposed to solve this problem. The energy function combines the dark channel prior with piecewise smoothness. The method is compared to the state-of-the-art methods and shows outstanding performance.

Real-time 3D mapping with MAV (Micro Aerial Vehicle) in GPS-denied environment is a challenging problem. In this paper, we present an effective vision-based 3D mapping system with 2D laser-scanner. All algorithms necessary for this system are on-board. In this system, two cameras work together with the laser-scanner for motion estimation. The distance of the points detected by laser-scanner are transformed and treated as the depth of image features, which improves the robustness and accuracy of the pose estimation. The output of visual odometry is used as an initial pose in the Iterative Closest Point (ICP) algorithm and the motion trajectory is optimized by the registration result. We finally get the MAV's state by fusing IMU with the pose estimation from mapping process. This method maximizes the utility of the point clouds information and overcomes the scale problem of lacking depth information in the monocular visual odometry. The results of the experiments prove that this method has good characteristics in real-time and accuracy.

Tracking multiple moving targets in a video is a challenge because of several factors, including noisy video data, varying number of targets, and mutual occlusion problems. The Gaussian mixture probability hypothesis density (GM-PHD) filter, which aims to recursively propagate the intensity associated with the multi-target posterior density, can overcome the difficulty caused by the data association. This paper develops a multi-target visual tracking system that combines the GM-PHD filter with object detection. First, a new birth intensity estimation algorithm based on entropy distribution and coverage rate is proposed to automatically and accurately track the newborn targets in a noisy video. Then, a robust game-theoretical mutual occlusion handling algorithm with an improved spatial color appearance model is proposed to effectively track the targets in mutual occlusion. The spatial color appearance model is improved by incorporating interferences of other targets within the occlusion region. Finally, the experiments conducted on publicly available videos demonstrate the good performance of the proposed visual tracking system.

Tracking multiple moving targets in video is still a challenge because of mutual occlusion problem. This paper presents a Gaussian mixture probability hypothesis density-based visual tracking system with game theory-based mutual occlusion handling. First, a two-step occlusion reasoning algorithm is proposed to determine the occlusion region. Then, the spatial constraint-based appearance model with other interacting targets¶ interferences is modeled. Finally, an n-person, non-zero-sum, non-cooperative game is constructed to handle the mutual occlusion problem. The individual measurements within the occlusion region are regarded as the players in the constructed game competing for the maximum utilities by using the certain strategies. The Nash Equilibrium of the game is the optimal estimation of the locations of the players within the occlusion region. Experiments conducted on publicly available videos demonstrate the good performance of the proposed occlusion handling algorithm.

Multi-target tracking in video is a challenge due to noisy video data, varying number of targets, and the data association problems. In this paper, a multi-target visual tracking system that incorporates object detection with the Gaussian mixture PHD filter is developed. The main contribution of this paper is to propose a new birth intensity online estimation method that based on the entropy distribution and the coverage rate. First, the birth intensity is initialized by using the previously obtained targets' states and measurements. The measurements are obtained by object detection and classified into the birth measurements and the survival measurements. Then it is updated according to the currently obtained birth measurements. In the update stage, the instability of the entropy distribution is applied to remove components like noises within the birth intensity which are irrelevant with the currently obtained birth measurements. And the coverage rate between each birth intensity component and corresponding birth measurement is computed to further eliminate the noises. Finally, experiments are implemented to show the performance of the proposed visual tracking system, especially to show the good performance for tracking the newborn targets.

Automatic reconstruction of unknown 3-D objects has been of great importance in the areas of machine vision, object recognition, and automatic modeling. In this paper, a new planning approach of generating 3-D models automatically is proposed. The new algorithm incorporates the limit visual surfaces of unknown model which are obtained according to both of the known object boundary knowledge and the visual region of the vision system and selects the suitability of viewpoints as the next best view on scanning coverage. The limit visual surfaces are used to predict the maximal information of unknown model and then the visibility criterion of next viewpoint is determined. And the position which can obtain the maximal visual surface area is defined as the next best view position. The reconstruction result of real model with proposed method show the efficiency in practical implementation.