Left ventricular segmentation in echocardiographic videos plays a crucial role in assessing various heart functions and disease diagnoses. However, due to the dynamic nature of echocardiography, maintaining consistent target segmentation across context shifts between frames poses a significant challenge. This task becomes even more difficult in the presence of noise interference in ultrasound images. In this paper, we propose DiffSegMem, which leverages the generative advantages of conditional diffusion model and a contextual memory storage architecture to enhance the accuracy of cross-frame segmentation in dynamic echocardiographic videos. Specifically, we introduce a noise-frequency domain aware dual-branch conditional encoding that establishes noise-resistant conditions at each sampling step, providing a reliable mask for the first frame of the echocardiographic sequence. As a result, our method does not require any prompting for the first video frame. Additionally, we propose a dynamic memory propagation strategy that utilizes memory transfer switches to extract memory from deeply linked video frames using a positional switch, with memory blocks carrying contextual clues prompting segmentation frame by frame. We evaluate our method on publicly available echocardiographic segmentation datasets and demonstrate state-of-the-art performance compared to existing models, outperforming current supervised methods for prompt-based segmentation.

Neural implicit scene representations have recently shown promising results in dense visual SLAM. However, existing implicit SLAM algorithms are constrained to single-agent scenarios, and fall difficulty in large indoor scenes and long sequences. Existing multi-agent SLAM frameworks cannot meet the constraints of communication bandwidth. To this end, we propose the first distributed multi-agent collaborative SLAM framework with distributed mapping and camera tracking, joint scene representation, intra-to-inter loop closure, and multi-submap fusion. Specifically, our proposed distributed neural mapping and tracking framework only needs peer-to-peer communication, which can greatly improve multi-agent cooperation and communication efficiency. A novel intra-to-inter loop closure method is designed to achieve local (single-agent) and global (multi-agent) consistency. Furthermore, to the best of our knowledge, there is no real-world dataset for NeRF-based/GS-based SLAM that provides both continuous-time trajectories groundtruth and high-accuracy 3D meshes groundtruth. To this end, we propose the first real-world indoor neural slam (INS) dataset covering both single-agent and multi-agent scenarios, ranging from small room to large-scale scenes, with high-accuracy ground truth for both 3D mesh and continuous-time camera trajectory. This dataset can advance the development of the community. Experiments on various datasets demonstrate the superiority of the proposed method in both mapping, tracking, and communication. The dataset and code will be open-source on https://github.com/dtc111111/MNESLAM.

4D imaging radar is a type of low-cost millimeter-wave radar(costing merely 10-20% of lidar systems) capable of providing range, azimuth, elevation, and Doppler velocity information. Accurate extrinsic calibration between millimeter-wave radar and camera systems is critical for robust multimodal perception in robotics, yet remains challenging due to inherent sensor noise characteristics and complex error propagation. This paper presents a systematic calibration framework to address critical challenges through a spatial 3d uncertainty-aware PnP algorithm (3DUPnP) that explicitly models spherical coordinate noise propagation in radar measurements, then compensating for non-zero error expectations during coordinate transformations. Finally, experimental validation demonstrates significant performance improvements over state-of-the-art CPnP baseline, including improved consistency in simulations and enhanced precision in physical experiments. This study provides a robust calibration solution for robotic systems equipped with millimeter-wave radar and cameras, tailored specifically for autonomous driving and robotic perception applications.

Recent research has begun exploring novel view synthesis (NVS) for LiDAR point clouds, aiming to generate realistic LiDAR scans from unseen viewpoints. However, most existing approaches do not reconstruct semantic labels, which are crucial for many downstream applications such as autonomous driving and robotic perception. Unlike images, which benefit from powerful segmentation models, LiDAR point clouds lack such large-scale pre-trained models, making semantic annotation time-consuming and labor-intensive. To address this challenge, we propose SN-LiDAR, a method that jointly performs accurate semantic segmentation, high-quality geometric reconstruction, and realistic LiDAR synthesis. Specifically, we employ a coarse-to-fine planar-grid feature representation to extract global features from multi-frame point clouds and leverage a CNN-based encoder to extract local semantic features from the current frame point cloud. Extensive experiments on SemanticKITTI and KITTI-360 demonstrate the superiority of SN-LiDAR in both semantic and geometric reconstruction, effectively handling dynamic objects and large-scale scenes. Codes will be available on https://github.com/dtc111111/SN-Lidar.

Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, and robotics navigation. Existing dense reconstruction methods are primarily designed for small room scenarios, but in practice, the scenes encountered by robots are typically large-scale environments. Most existing methods have difficulties in large-scale scenes due to three core challenges:(a) inaccurate depth input. Depth information is crucial for both scene geometry reconstruction and pose estimation. Accurate depth input is impossible to get in real-world large-scale scenes.(b) inaccurate pose estimation. Existing methods are not robust enough with the growth of cumulative errors in large scenes and long sequences.(c) insufficient scene representation capability. A single global radiance field lacks the capacity to scale effectively to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale dense scene reconstruction. For depth estimation, a vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes and eliminates pose drift. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. In local radiance fields, we propose a tri-plane based scene representation method to further improve the accuracy and efficiency of scene reconstruction. We conduct extensive experiments on various datasets, including our own collected data, to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction. The code has been open-sourced on https://github.com/dtc111111/incre-dpsr.

In recent years, there have been significant advancements in 3D reconstruction and dense RGB-D SLAM systems. One notable development is the application of Neural Radiance Fields (NeRF) in these systems, which utilizes implicit neural representation to encode 3D scenes. However, the depth images obtained from consumer-grade RGB-D sensors are often sparse and noisy, which poses significant challenges for 3D reconstruction and affects the accuracy of the representation of the scene geometry. Furthermore, existing methods select random pixels for camera tracking, leading to inaccurate localization in real-world indoor environments. To this end, we present NeSLAM, an advanced framework that achieves accurate and dense depth estimation, robust camera tracking, and realistic synthesis of novel views. First, a depth completion and denoising network is designed to provide dense geometry prior and guide the neural implicit representation optimization. Second, we propose a NeRF-based self-supervised feature tracking algorithm for robust real-time tracking. Experiments on various indoor datasets demonstrate the effectiveness and accuracy of the system in reconstruction, tracking quality, and novel view synthesis. Note to Practitioners—Traditional SLAM methods usually use the sparse point cloud to represent the scene, resulting in poor scene representation capability. Our method proposes a neural implicit representation method with depth completion and denoising network and feature tracking method, achieves accurate scene reconstruction and accurate pose estimation in various indoor scenes. The depth completion and denoising network provide accurate depth information associated with depth uncertainty, which is used to improve the geometry consistency. The NeRF-based self-supervised feature tracking method improve the accuracy and robustness for camera tracking. The experimental results demonstrate the accuracy and effectiveness of this method in different scenes.

In this work, we propose an optimization-based trajectory planner for tractor-trailer vehicles on curvy roads. The lack of analytical expression for the trailer’s errors to the center line pose a great challenge to the trajectory planning for tractor-trailer vehicles. To address this issue, we first use geometric representations to characterize the lateral and orientation errors in Cartesian frame, where the errors would serve as the components of the cost function and the road edge constraints within our optimization process. Next, we generate a coarse trajectory to warm-start the subsequent optimization problems. On the other hand, to achieve a good approximation of the continuous-time kinematics, optimization-based methods usually discretize the kinematics with a large sampling number. This leads to an increase in the number of the variables and constraints, thus making the optimization problem difficult to solve. To address this issue, we design a Progressively Increasing Sampling Number Optimization (PISNO) framework. More specifically, we first find a nearly feasible trajectory with a small sampling number to warm-start the optimization process. Then, the sampling number is progressively increased, and the corresponding intermediate Optimal Control Problem (OCP) is solved in each iteration. Next, we further resample the obtained solution into a finer sampling period, and then use it to warm-start the intermediate OCP in next iteration. This process is repeated until reaching a threshold sampling number. Simulation and experiment results show the proposed method exhibits a good performance and less computational consumption over the benchmarks.

Predictive analysis of pre-hospital care can optimize resource allocation, address challenges in assessing ambulance demand, and significantly improve emergency care efficiency and patient survival rates. However, the lack of high-quality publicly available datasets, combined with the influence of complex factors such as urban spatial patterns, weather, and time, makes predictive analysis of pre-hospital emergency care a challenge. To address this, we introduce the Pre-hospital Emergency Information Mart, a large-scale, publicly accessible repository of authentic emergency records. Given the complexity of such analysis, we propose a Multivariate Spatiotemporal Prediction Network. Specifically, we design a multifactor fine-grained sequence learning branch to model historical sequences, time factors, and external factors. The spatial distribution prediction branch, based on spatiotemporal attention, effectively captures spatial features with hyperconnections. The global knowledge transfer module transfers global spatial knowledge to the sequence prediction module, increasing spatial dependencies for reliable predictions under multidimensional constraints. Experiments on the Pre-hospital Emergency Information Mart dataset show that our proposed Multivariate Spatiotemporal Prediction Network outperforms six well-known methods.

Neural implicit scene representations have recently shown encouraging results in dense visual SLAM. However, existing methods produce low-quality scene reconstruction and low-accuracy localization performance when scaling up to large indoor scenes and long sequences. These limitations are mainly due to their single, global radiance field with finite capacity, which does not adapt to large scenarios. Their end-to-end pose networks are also not robust enough with the growth of cumulative errors in large scenes. To this end, we introduce PLGSLAM, a neural visual SLAM system capable of high-fidelity surface reconstruction and robust camera tracking in real-time. To handle large-scale indoor scenes, PLGSLAM proposes a progressive scene representation method which dynamically allocates new local scene representation trained with frames within a local sliding window. This allows us to scale up to larger indoor scenes and improves robustness (even under pose drifts). In local scene representation, PLGSLAM utilizes tri-planes for local high-frequency features with multilayer perceptron (MLP) networks for the low-frequency feature, achieving smoothness and scene completion in unobserved areas. Moreover, we propose local-to-global bundle adjustment method with a global keyframe database to address the increased pose drifts on long sequences. Experimental results demonstrate that PLGSLAM achieves state-of-the-art scene reconstruction results and tracking performance across various datasets and scenarios (both in small and large-scale indoor environments).

Nonlinear dynamics bring difficulties to controller design for control-affine systems such as tractor-trailer vehicles, especially when the parameters in the dynamics are unknown. To address this constraint, we propose a derivative-based lifting function construction method, show that the corresponding infinite dimensional Koopman bilinear model over the lifting function is equivalent to the original control-affine system. Further, we analyze the propagation and bounds of state prediction errors caused by the truncation in derivative order. The identified finite dimensional Koopman bilinear model would serve as predictive model in the next step. Koopman Bilinear Model Predictive control (K-BMPC) is proposed to solve the trajectory tracking problem. We linearize the bilinear model around the estimation of the lifted state and control input. Then the bilinear Model Predictive Control problem is approximated by a quadratic programming problem. Further, the estimation is updated at each iteration until the convergence is reached. Moreover, we implement our algorithm on a tractor-trailer system, taking into account the longitudinal and side slip effects. The open-loop simulation shows the proposed Koopman bilinear model captures the dynamics with unknown parameters and has good prediction performance. Closed-loop tracking results show the proposed K-BMPC exhibits elevated tracking precision with the commendable computational efficiency. The experimental results demonstrate the feasibility of K-BMPC.

Prior structural reference like floor plan is readily accessible in indoor scene, which exhibits the potential of improving localization quality without the requirements of a previously-built high-precision map. This paper introduces a novel optimal transport-based framework for prior structural reference-based localization, aiming to improve the robustness for the robot localization. Leveraging the spacial relations of structures, a matching method based on optimal transport theory is proposed and it improves the robustness of matching results in dynamic scene and rapid rotation conditions. Additionally, this paper handles metric inaccuracies in the known structural reference by implementing an prior guided plane adjustment-based updating strategy. This strategy combines prior and observational information to jointly optimize the structural information within a sliding window. The performance of the framework is validated through real-world experiments, demonstrating superior accuracy and robustness to disturbances from dynamic occlusion and rapid rotation compared to common state-of-the-art SLAM and localization methods.

The main content of the research in this paper is the estimation of depth and pose based on monocular vision and Inertial Measurement Unit (IMU). The usual depth estimation network and pose estimation network require depth ground truth or pose ground truth as a supervised signal for training, while the depth ground truth and pose ground truth are hard to obtain, and monocular vision based depth estimation cannot predict absolute depth. In this paper, with the help of IMU, which is inexpensive and widely used, we can obtain angular velocity and acceleration information. Two new supervision signals are proposed and the calculation expressions are given. Among them, the model trained with acceleration constraint shows a good ability to estimate the absolute depth during the test. It can be considered that the model can estimate the absolute depth. We also derive the method of estimating the scale factor during the test from the acceleration constraint, and also achieve good results as the acceleration constraint does. In addition, this paper also studies the method of using IMU information as pose network input and as selecting conditions. Moreover, it analyzes and discusses the experimental results. At the same time, we also evaluate the effect of the pose estimation of the relevant models. This article starts by reviewing the achievements and deficiencies of the work in this field, combines the use of IMU, puts forward three new methods such as a new loss function, and conducts a test analysis and discussion of relevant indicators on the KITTI data set.

With the rapid development of autonomous vehicle technologies, how to perform high-precision localization in unknown complex outdoor environment has become an important issue. Visual odometry is one of the low-cost and the most widely utilized localization methods. Traditional methods predict relative pose based on the principle of multi-view geometry, which is sensitive to camera parameters and environmental changes. This paper studies deep learning-based methods which can be more robust. A novel end-to-end unsupervised visual odometry framework based on confidence evaluation is proposed. Its process can be divided into two stages. The first is predicting the initial relative pose transformation with the help of confidence mask which is generated by measuring the relative similarity of geometric corresponding regions in associated images. The second is evaluating the confidence of the output pose estimate based on the trajectory geometric consistency and then refining it. Quantitative and qualitative evaluation of the proposed approach on KITTI dataset are presented to demonstrate its effectiveness in improving pose estimation accuracy and robustness.

In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at:https://github.com/guangmingw/DOPlearning.

In this article, to solve trajectory tracing problem and vibration suppression for a planar two-link rigid-flexible manipulator subject to joint-velocity measurement noise, a novel uncalibrated visual servoing control is proposed. To begin with, the manipulator’s dynamic model is established by the assumed mode method (AMM). On this basis, based on the singular perturbation theory, two subsystem controllers are designed, one is slow subsystem controller, and the other one is fast subsystem controller. In the slow subsystem, to cope with the complication of the camera calibration, an adaptive algorithm is formulated to evaluate the parameters of a fixed camera online. Aiming to overcome the challenge that exact joint-velocity measurement may be disturbed by external noise, a nonlinear sliding observer is developed to estimate the state of joint velocity accurately. The asymptotic convergence of image tracking error is proved by means of Lyapunov analysis. Additionally, for the purpose of restraining the flexible beam’s elastic vibration, a linear quadratic regulator (LQR) approach is adopted in the fast subsystem control design. The realistic comparing simulation experiments are presented to demonstrate the performance of the proposed controller.

This study proposes a hybrid vision/force control scheme for interaction with the inner surface of the bottle-like object. Based on the geometry of the object, a new generalized constraint called the bottleneck (BN) constraint is proposed, which ensures the tool passes through a fixed 3-D region and avoid collisions with the boundary of the region. To realize the hybrid vision/force control under the BN constraint, a novel dynamic controller is designed inspired by the hierarchical operational space, which can complete the different tasks defined in the decoupled subspace. To enhance the robustness of the algorithm, we develop a data-driven method and an adaptive method to estimate the Jacobian matrix online in force space and image space, respectively. The asymptotic stability of the closed-loop system is rigorously proved by the Lyapunov theory. Experiments are conducted to validate the performance of the proposed method.

This article proposes a novel image-based visual servoing for a flexible aerial refueling boom with an eye-in-hand camera. The dynamic model of the flexible refueling boom is established by absolute nodal coordinate formulation (ANCF), after which the dynamic model is decomposed into a slow subsystem and a fast subsystem based on the singular perturbation approach. With respect to slow subsystem, the image feedback is used to control the flexible refueling boom so that the projection of the point marker on the back of the receiver converges to the desired position. With respect to fast subsystem, linear quadratic regulator (LQR) is applied to stabilize the vibration of the flexible refueling boom. The asymptotic convergence of the image error to zero is verified based on the Lyapunov theory. Simulation is used to demonstrate the effectiveness of the proposed method.

Accurate, robust and real-time localization under constrained-resources is a critical problem to be solved. In this paper, we present a new sparse pose-graph visual-inertial SLAM (SPVIS). Unlike the existing methods that are costly to deal with a large number of redundant features and 3D map points, which are inefficient for improving positioning accuracy, we focus on the concise visual cues for high-precision pose estimating. We propose a novel hierarchical quadtree based optical flow tracking algorithm, it achieves high accuracy and robustness within very few concise features, which is only about one fifth features of the state-of-the-art visual-inertial SLAM algorithms. Benefiting from the efficient optical flow tracking, our sparse pose-graph optimization time cost achieves bounded complexity. By selecting and optimizing the informative features in sliding window and local VIO, the computational complexity is bounded, it achieves low time cost in long-term operation. We compare with the state-of-the-art VIO/VI-SLAM systems on the challenging public datasets by the embedded platform without GPUs, the results effectively verify that the proposed method has better real-time performance and localization accuracy.

With the advance in the field of mobile robots, autonomous robots are required for long-term deployment in dynamic and complex environments. However, the performance of Visual Inertial SLAM systems in long-term operation is not satisfactory, and most long-term SLAM systems assumes periodic changes in the environment. This paper presents a novel solution for long-term monocular VI SLAM system in dynamic environment based on autoregression(AR) modeling and map prediction. Map points are first classified into static and semi-static map points according to a memory model. Modeling and prediction of the different states of semi-static map points are performed that are derived from time series models. The predicted map is then fused with the current map to achieve a better forecast for the next frame if the prediction is not satisfactory enough. Experiments are carried out on an embedded system. The results indicate that the map prediction is reliable and the proposed approach improves the performance of long-term localization and mapping in dynamic environments.

During autonomous navigation, environmental information and map noises at different locations may have dissimilar influence on a vehicle's localization process. This implies that the vehicle's localizability, i.e., the ability to localize itself in an environment using laser range finder (LRF) readings, varies over a given map. It is essential to take this factor into consideration when planning a path to avoid large localization errors or placing the vehicle at risk of failure to perform localization. We propose a localizability constraint (LC)-based path planning method for autonomous vehicles which plans the navigation path according to LRF sensor model of the vehicle in an effort to maintain a satisfactory level of localizability throughout the path, as well as to reduce the overall localization error. Our method is not limited to any specific algorithm in the optimization stage. Paths planned with and without LC are compared, and the influence of the LRF sensor model on planning outcomes is discussed through simulations. By conducting comparative experiments on a “JiaoLong” intelligent wheelchair in both indoor and outdoor environments, we show that the proposed method effectively lowers the localization error along the planned paths.

In this paper, we study vision-based localization for robots. We anticipate that numerous mobile robots will serve or interact with humans in indoor scenarios such as healthcare, entertainment, and public service. Such scenarios entail accurate and scalable indoor visual robot localization, the subject of this work. Most existing vision-based localization approaches suffer from low localization accuracy and scalability issues due to visual environmental features' limited effective range and detection accuracy. In light of infrastructural cameras' wide indoor deployment, this paper proposes BRIDGELOC, a novel vision-based indoor robot localization system that integrates both robots' and infrastructural cameras. BRIDGELOC develops three key technologies: robot and infrastructural camera view bridging, rotation symmetric visual tag design, and continuous localization based on robots' visual and motion sensing. Our system bridges robots' and infrastructural cameras' views to accurately localize robots. We use visual tags with rotation symmetric patterns to extend scalability greatly. Our continuous localization enables robot localization in areas without visual tags and infrastructural camera coverage. We implement our system and build a prototype robot using commercial off-the-shelf hardware. Our real-world evaluation validates BRIDGELOC's promise for indoor robot localization.

Shared control is a common used method for human and wheelchair cooperation. However, most of the previous shared control methods didn't think much of the effect caused by the difference in the user's control ability. The control weight of a user in these methods is irrelevant to the user's capability or the driving conditions. In this paper, a dynamic shared control method is proposed to adapt wheelchair's assistance to the variations of user performance and the environmental changes. Three evaluation indices including safety, comfort and obedience are designed to evaluate wheelchair performance in real time. A minimax multi-objective optimization algorithm is adopted to calculate the user's control weight. The results of lab experiments and elderly home field tests show that this method can adapt the degree of wheelchair's autonomy to the user's control ability and it makes driving wheelchair much easier for elder people.

To keep up with the wide-ranging, fast-moving robotics field, education must be adaptive and multidisciplinary. In this paper, integration of PSoC technology with educational robotics is presented. Both of the advantage: flexibility in hardware for PSoC and modularized robot components are fused. This integration balances research and implementation fundamentals by reinforcing course work with intensive projects focused on robotics technology. This education fostered students' teamwork skills, while project completion and competition success greatly enhanced the students' self-confidence.

The navigation system of family indoor mobile robot includes localization, path planning, collision avoidance. The hybrid localization method of straight line matching, corner matching and odometry is proposed. The hardware and software configuration is introduced. Robot detects environment using a 2D laser range finder. Line feature extraction process including area divided, iterative end point fit (IEPF) and a least square technique is introduced. Based on line feature, straight lines and corners as geometry features are obtained. The odometry localization algorithm, straight line localization algorithm and corner localization algorithm are discussed. Artificial Potential Field (APF) based path planning algorithms is implemented. As a result stable localization is achieved with position and orientation resolution as 50mm, 5 degree. A good performance for the method is also achieved with cycle time as 120ms. Experiment shows the effectiveness of the hybrid localization method.