Precise shape control of tendon-driven continuum robots (TDCRs) remains challenging due to their inherent nonlinearity and environmental uncertainties. Analytical kinematic models fail to accurately characterize nonlinear deformation behavior. This paper proposes a model-less optimal shape control (MLOSC) framework using an offline-online learning strategy to achieve precise shape tracking under unknown loading conditions. The method approximates the TDCR’s shape Jacobian by a Radial Basis Function Neural Network (RBFNN), pretrained offline. Online adaptation of network parameters is performed via neurodynamic optimization to enhance prediction accuracy in the presence of uncertainties. The shape control problem is subsequently formulated as a constrained quadratic program (QP) that incorporates the robot’s physical limits. A neurodynamic solver is designed to compute optimal control inputs in real time. Control stability is analyzed via Lyapunov’s method. Experimental evaluations under static and dynamic loading conditions achieve average shape tracking errors of 2.04 mm (1.4% of robot total length) and 5.09 mm (3.4%), respectively. Comparative results confirm improved convergence speed and tracking accuracy over state-of-the-art methods.

Accurate segmentation of thoracoabdominal anatomical structures in three-dimensional medical imaging modalities is fundamental for informed clinical decision-making across a wide array of medical disciplines. Current approaches often struggle to efficiently and comprehensively process this region's intricate and heterogeneous anatomical information, leading to suboptimal outcomes in diagnosis, treatment planning, and disease management. To address this challenge, we introduce SegTom, a novel volumetric segmentation framework equipped with a cutting-edge SegTom Block specifically engineered to effectively capture the complex anatomical representations inherent to the thoracoabdominal region. This SegTom Block incorporates a hierarchical anatomical-representation decomposition to facilitate efficient information exchange by decomposing the computationally intensive self-attention mechanism and cost-effectively aggregating the extracted representations. Rigorous validation of SegTom across nine diverse datasets, encompassing both computed tomography (CT) and magnetic resonance imaging (MRI) modalities, consistently demonstrates high performance across a broad spectrum of anatomical structures. Specifically, SegTom achieves a mean Dice similarity coefficient (DSC) of 87.29% for cardiac segmentation on the MM-WHS MRI dataset, 83.48% for multi-organ segmentation on the BTCV abdominal CT dataset, and 92.01% for airway segmentation on a dedicated CT dataset.

Multimodal ultrasound imaging, combining B-mode ultrasound, shear wave velocity, and shear wave time, is crucial for diagnosing and treating breast lesions, providing insights into lesion characteristics and tissue properties. However, challenges arise from inter-modal feature misalignment and attention shifts due to varied capture methods and an overemphasis on vibrant color data. To tackle these issues, we introduce two innovations: a novel segmentation framework and a comprehensive dataset. The UltraMamba framework utilizes bidirectional alignment between modalities and enhances region-specific information to improve breast lesion segmentation accuracy. Key components include the Cross-Modal Knowledge Interaction module for robust information exchange and the Region-Aware Feature Excitation module to focus on relevant features. We also present the BreLS dataset, the first two-dimensional multimodal ultrasound breast lesion dataset, with paired images from 506 cases, serving as a valuable resource for analysis. UltraMamba shows strong performance on the BreLS dataset, achieving a Dice Similarity Coefficient of 72.16% and an HD95 of 42.02 mm, reflecting improvements of 2.59% in DSC and a 6.78 mm reduction in HD95 compared to the second-best framework, MMCA-NET. These results highlight UltraMamba's potential to enhance segmentation accuracy in clinical settings, facilitating precise treatment planning and, ultimately, leading to improved outcomes. Code: https://github.com/deepang-ai/UltraMamba.

Polyp segmentation in endoscopic imaging is essential for the early detection of colorectal cancer, as polyps are precursor lesions in the colon and rectum, yet the task is complicated by the morphological variability and indistinct boundaries of polyps, which often blend into surrounding tissues. Conventional approaches struggle with these complexities, as fixed scale and window sizes are unable to adapt to the diverse and irregular structures of polyps. To address this challenge, we introduce the Endoscopic Adaptive Transformer, EAT, a novel framework specifically engineered for polyp segmentation. EAT incorporates an adaptive perception module, APM, that employs an adaptive perceptive-field mechanism to dynamically capture both fine-grained local details and broad contextual information, enhancing segmentation accuracy across diverse polyp morphologies. EAT demonstrates comprehensive performance by achieving a Dice coefficient of 97.77% and an HD95 of 4.50mm in single-target segmentation, while also excelling in multi-target scenarios with a Dice coefficient of 88.02% and an HD95 of 53.75mm, significantly outperforming state-of-the-art methods across both single- and multi-target segmentation scenarios. This performance underscores EAT's critical role in improving the accuracy of polyp segmentation, highlighting its potential to advance diagnostic precision and treatment planning in clinical endoscopy applications. Code: https://github.com/deepang-ai/EAT.

Medical portable devices are increasingly requiring high accuracy, speed, and low inference jitter to meet the urgent demands of healthcare. Modern hybrid attention-based segmentation frameworks enhance segmentation accuracy but add complexity that can slow operational speed, complicating practical deployment in resource-limited settings. We propose Slim UNETRV2, a simplified framework that utilizes only basic convolutional operations in both the encoder and decoder, thereby reducing execution time and inference jitter. The Slim UNETRV2 block, placed in skip connections at each hierarchical stage, aggregates extracted representations and improves global processing. Experiments demonstrate that Slim UNETRV2 outperforms state-of-the-art models in terms of accuracy, speed, and inference jitter for resource-constrained medical devices. Notably, Slim UNETRV2 achieves 93.89% dice accuracy and 2.90 mm HD95 on BraTS 2021, being 16.7 times faster with only 0.225 ms of inference jitter compared to SegMamba. Code: https://github.com/deepang-ai/Slim-UNETRV2https://github.com/deepang-ai/Slim-UNETRV2.

Pelvic fracture reduction surgery (PFRS) has always been one of the most challenging procedures in trauma orthopedics. Excellent preoperative planning is crucial for surgery, especially with the increasingly mature robot-assisted surgical systems. However, current preoperative reduction planning heavily relies on surgeons’ experience. This paper proposes an automatic preoperative planning framework for PFRS. Firstly, an enlarged RRT$^\ast$(ERRT$\ast)$algorithm is proposed to search feasible paths, which adopts a synchronized exploration and asynchronous adjustment strategy in 6D space to change the position and orientation of fragments during the reduction process. Secondly, a collision detection method based on surface point cloud is proposed to improve the safety of the reduction path by taking into account the actual volume of fragments. Finally, a post-processing method combining path shortening (PS) algorithm and cubic spline interpolation is proposed to optimize and smooth the reduction path. The clinical case simulation results show that the ERRT$^\ast$algorithm can find a feasible reduction path within a few seconds ($<$10s), and the length of the path is reduced by an average of 11.11% with the PS algorithm. Furthermore, repeated experimental results demonstrate that the method has good consistency. The proposed preoperative planning method can serve as a powerful tool to provide references for surgeons and also provide a quantitative basis for robot-assisted PFRS.Note to Practitioners—This work addresses the challenge of automating preoperative planning for pelvic fracture reduction surgery, specifically in determining the target reduction pose and operative path for repositioning the fragment. The motion of the fragment is decoupled into translation along a point and rotation through the coordinate system of that point to quantify the difference between the initial pose and target pose and determine the planning requirements. An enlarged RRT$^\ast$algorithm based on six-dimensional generalized coordinates is proposed, combined with an efficient collision avoidance algorithm, to quickly find a safe and feasible reduction operation path. The proposed planning method not only provides guidance to physicians but also establishes a basis for robot-assisted fracture reduction surgery.

Computer-aided ultrasound (US) imaging is an important prerequisite for early clinical diagnosis and treatment. Due to the harsh ultrasound (US) image quality and the blurry tumor area, recent memory-based video object segmentation models (VOS) achieve frame-level segmentation by performing intensive similarity matching among the past frames which could inevitably result in computational redundancy. In this paper, we first build a larger annotated benchmark dataset for breast lesion segmentation in ultrasound videos, then we propose a lightweight clip-level VOS framework for achieving higher segmentation accuracy while maintaining the speed. Then an Inner-Outer Clip Retformer is proposed to extract spatial-temporal tumor features in parallel. Specifically, the proposed Outer Clip Retformer extracts the tumor movement feature from past video clips to locate the current clip tumor position, while the Inner Clip Retformer detailedly extracts current tumor features that can produce more accurate segmentation results. Then a Clip Contrastive loss function is further proposed to align the extracted tumor features along both the spatial-temporal dimensions to improve the segmentation accuracy. In addition, the Global Retentive Memory is proposed to maintain the complementary tumor features with lower computing resources which can generate coherent temporal movement features. In this way, our model can significantly improve the spatial-temporal perception ability without increasing a large number of parameters, achieving more accurate segmentation results while maintaining a faster segmentation speed. Finally, we conduct extensive experiments to evaluate our proposed model on several video object segmentation datasets, the results show that our framework outperforms state-of-the-art segmentation methods.

Medical video segmentation is fundamentally important in clinical diagnosis and treatment procedures, offering dynamic tracking of breast lesions across frames in ultrasound videos for improved segmentation performance. However, existing approaches face challenges in striking a balance between segmentation performance and inference speed, hindering real-time application in resource-constrained medical environments. In order to address these limitations, we present BaS, a blazing-fast on-device breast lesion segmentation model. BaS integrates the Stem module and BaSBlock to refine representations through inter- and intra-frame analysis on ultrasound videos. In addition, we release two versions of BaS: the BaS-S for superior segmentation performance and the BaS-L for accelerated inference times. Experimental Results indicate that BaS surpasses the top-performing models in terms of segmenting efficiency and accuracy of predictions on devices with limited resources. This work advances the development of efficient medical video segmentation frameworks applicable to multiple medical platforms.

In the specialized domain of brain tumor segmentation, supervised segmentation approaches are hindered by the limited availability of high-quality labeled data, a condition arising from data privacy concerns, significant costs, and ethical issues. In response to this challenge, this paper presents a training framework that adeptly integrates a plug-and-play component, MOD, into current supervised learning models, boosting their efficacy in scenarios with limited data. The MOD consists of an Online Tokenizer and a Dense Predictor, which employs self-distillation and self-modeling on masked patches, promoting swift convergence and efficient representation learning. During the inference phase, the plug-and-play MOD component is excluded, preserving the computational efficiency of the original model without incurring extra processing costs. We substantiated the value of our approach through experiments on leading 3D brain tumor segmentation baselines. Remarkably, models augmented with the MOD consistently showcased superior results, achieving elevated Dice coefficients and HD95 scores on two datasets: BraTS 2021 and MSD 2019 Task-01 Brain Tumor.

Automatic report generation has arisen as a significant research area in computer-aided diagnosis, aiming to alleviate the burden on clinicians by generating reports automatically based on medical images. In this work, we propose a novel framework for automatic ultrasound report generation, leveraging a combination of unsupervised and supervised learning methods to aid the report generation process. Our framework incorporates unsupervised learning methods to extract potential knowledge from ultrasound text reports, serving as the prior information to guide the model in aligning visual and textual features, thereby addressing the challenge of feature discrepancy. Additionally, we design a global semantic comparison mechanism to enhance the performance of generating more comprehensive and accurate medical reports. To enable the implementation of ultrasound report generation, we constructed three large-scale ultrasound image-text datasets from different organs for training and validation purposes. Extensive evaluations with other state-of-the-art approaches exhibit its superior performance across all three datasets. Code and dataset are valuable at this link.

Puncture robots pave a new way for stable, accurate and safe percutaneous liver tumor puncture operation. However, affected by respiratory motion, intraoperative accurate location of the tumor and its surrounding anatomical structures remains a difficult problem in existing robot-assisted puncture operations. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D ultrasound (US) and preoperative 3D computed tomography (CT) fusion is proposed, addressing the shortcomings of existing puncture robots. To deal with the challenge of cross-modal and cross-dimensional registration between 2D US and 3D CT, a decoupled two-stage registration approach combining initial vessel structure-based 3D US – 3D CT registration with intraoperative intensity-based 2D US -3D US registration is proposed. To achieve fast and robust ultrasound probe calibration, a method based on an improved N-wire phantom is proposed. Twenty puncture experiments are performed in different breath-holding positions on a respiratory motion simulation platform, and experimental results show that the mean puncture error is 2.48 mm, which can meet the requirements in a wide of clinical scenariosNote to Practitioners—In clinical percutaneous liver tumor puncture operation, due to the lack of real-time and clear image guidance, it is difficult to locate the tumor and its surrounding vital anatomical structures. In addition, the stability and accuracy of manual operation are poor. The development of a puncture robot is an effective solution for these problems. However, existing CT and magnetic resonance imaging (MRI) guided robots do not consider the tumor localization errors caused by inconsistent breath-holding positions between preoperative scan period and intraoperative puncture period, and US guided robots are limited by the poor image quality and the narrow field of vision. In this paper, a dual-arm robotic needle insertion system with guidance of intraoperative 2D US and preoperative 3D CT fusion is proposed. This system can take advantage of the real-time ultrasound and clear CT images at the same time, and can provide real-time, clear and all-round guidance for percutaneous liver tumor puncture operation, which has obvious advantages over the existing puncture robots. Phantom experiments have been completed and animal experiments will be carried out in the future.

Offline reinforcement learning (ORL) has been getting increasing attention in robot learning, benefiting from its ability to avoid hazardous exploration and learn policies directly from precollected samples. Approximate policy iteration (API) is one of the most commonly investigated ORL approaches in robotics, due to its linear representation of policies, which makes it fairly transparent in both theoretical and engineering analysis. One open problem of API is how to design efficient and effective basis functions. The broad learning system (BLS) has been extensively studied in supervised and unsupervised learning in various applications. However, few investigations have been conducted on ORL. In this article, a novel incremental ORL approach with sparse broad critic approximation (BORL) is proposed with the advantages of BLS, which approximates the critic function in a linear manner with randomly projected sparse and compact features and dynamically expands its broad structure. The BORL is the first extension of API with BLS in the field of robotics and ORL. The approximation ability and convergence performance of BORL are also analyzed. Comprehensive simulation studies are then conducted on two benchmarks, and the results demonstrate that the proposed BORL can obtain comparable or better performance than conventional API methods without laborious hyperparameter fine-tuning work. To further demonstrate the effectiveness of BORL in practical robotic applications, a variable force tracking problem in robotic ultrasound scanning (RUSS) is investigated, and a learning-based adaptive impedance control (LAIC) algorithm is proposed based on BORL. The experimental results demonstrate the advantages of LAIC compared with conventional force tracking methods.

Robot-assisted decompressive laminectomy is a new strategy in clinical applications. However, a stable contact force between the bone-cutting device and lamina is needed to keep it working correctly, which may be affected by respiration and deformation. The surgeon can adapt to this dynamic process quickly, but the robot could cause a considerable force that may damage the patient. This paper proposes a compensation control method based on a respiration-spine model to first improve the stability of the contact force. The model is established based on human morphology and ventilator parameters for anaesthetised patients. The control method is a combination of a surgery sleeve and active fuzzy control to improve the robustness of the robot. The control of the sleeve is related to the thickness of the bone layer, which can be calculated from the image. Furthermore, the lower boundary of the lamina in the CT image is extracted as a safety constraint that can protect the spinal nerves. Finally, an experiment is conducted to verify the safety constraint and compare the changes in contact force with or without the control method. The statistical experiment shows that the control error is 2.47 N without the force control method, while the force control error is 0.223 N when the target control force is 2 N. The robot will hover on the surface of the spine after completing the laminectomy of the planned area. These results show that the robot can be controlled safely and stably. Note to Practitioners—The purpose of this paper is to propose a stable and safe control method for lamina grinding robots under the influence of breathing and deformation factors. Most previous research only considered one of the two factors, and both factors are considered in this paper. Unlike the surgeon, who has the ability to adapt to physiological movements, the robot may cause a considerable force to be exerted on the device, which may damage the patient. Therefore, a controller based on human experience is longitudinally designed, and position constraints are added in three directions. This application can improve the stability and safety of robot-assisted surgery, which may be suitable for remote surgery and can help rural areas solve the problem of a lack of medical resources. Spinal model bone experiments have been completed and will be tested on animals in the future.

Goal: In the context of robot-assisted laminectomy surgery, an analytical force model is introduced to guarantee procedural safety. The aim of the method is to intraoperatively monitor the cutting depth via modeling the milling status. Methods: The theoretical dynamic model for the surgical milling process is based on the flute geometry of the ball-end milling tool. A particle swarm optimization algorithm is exploited to calibrate the model using the local average force, and to validate it using the denoised dynamic force. A wear detection method based on the fast Fourier transform is proposed to determine the quality of the tool geometry and to avoid using worn tools, which may lead to imprecise and unsafe operations. Results: Milling experiments were performed on machined fresh bovine femur bones. The experimental results thus obtained from the mechanical model are in good accordance with the numerical model. The proposed method can monitor the current cutting depth with an accuracy of ±0.1 mm in regions located within the depth [0.8-1.2 mm], and ±0.2 mm within [1.2-1.6 mm]. Conclusion: The proposed model can successfully estimate the milling force and the cutting depth intraoperatively in experimental conditions. Significance: This approach has the potential to improve the safety of laminectomy operations in humans, and make it more accessible to younger surgeons by lowering the required manual skills threshold.

Pedicle drilling is a crucial and high-risk process in spinal surgery. Due to the respiration and cardiac cycle, the position of spine would fluctuate during operations, which result in an increase of the difficulty in state recognition of pedicle drilling. To guarantee the safety and validity, a model-based compensation method is proposed in this paper. To build the empirical model of vertebral motion, vertebral displacement and tidal volume (Tv) signals are collected from volunteers. To rule out disturbances in original signal, FFT and wavelet transform (DWT) are used to process the experimental signals. In order to select the apt basis for different signals, the root mean square error decision-making method (RMSE-DMM) is introduced. When the filtered vertebral displacement signal is obtained, the particle swarm optimization (PSO) algorithm is used to figure out the empirical model. The robot assisted systems (RAS) can easily compensate vertebral fluctuation based on the empirical model. Due to the goal of pedicle drilling is to drill a hole from surface of first cortical layer to the interior of second cortical layer, a new key point recognition algorithm, based on force, proposed in this paper. To verify the effectiveness of compensation and the recognition algorithm, 3 sets of comparison experiments are carried out. And the results of experiment show the compensation method and new key point recognition algorithm perform effectively.

For a robot serving in a complex environment such as in a restaurant, it is difficult to perform a task like tabletop object manipulation completely by itself, in that some information may be missing. An approach to deal with this is to use a tele-control system and method to control the robot or demonstrate. In this paper, a LeapMotion sensor based non-contact tele-control method is developed for a robot to perform tabletop object manipulation tasks. A coordinate system for mapping from the operation space of the LeapMotion sensor to the workspace of the robot is established. A gesture recognition and action generating algorithm is proposed for control or to demonstrate the motion to the robot. To evaluate the performance of the LeapMotion sensor and proposed method for tele-control of a robot, a comprehensive assessment index based on entropy weighting is proposed. Three common tele-control modes, including demonstration mode, teleoperation mode and semi-teleoperation mode, are developed on a PR2 robot. The experimental results show that the proposed tele-control system is more appropriate for use in task demonstration.

Screw path drilling is an important process among many orthopedic surgeries. To guarantee the safety and correctness of this process, a model-based drilling state recognition method is proposed in this paper. The thrust force in the drilling process is modeled based on an accurate 3D bone model restructured by means of Micro-CT images. In theoretical modeling of the thrust force, the resistance and the elasticity of the bone tissues are considered. The cutting energy and elastic modulus are defined as the material parameters in the theoretical model, which are identified via a least square method. Some key parameters are proposed to support the state recognition: the peak forces in the first and the second cortical layers, the average force in the cancellous layer and the thickness of each layer. Based on these key parameters in the model, a state recognition strategy with a robotic orthopedic surgery system is proposed to recognize the switch position of each layer. Experiments are performed to demonstrate the effectiveness of the modeling approach and the state recognition method.

Bone drilling is an important and difficult process in orthopedic surgeries. To detect the drilling state of a Robotic Orthopedic Surgery System (ROSS) in real-time, a state recognition method based on audio signals, the Acoustic Emission (AE) signals generated in drilling process, is proposed in this paper. By an analysis via power spectral density of the AE signals, an appropriate frequency band is selected for state recognition. The Exponential Mean Amplitude (EMA) and the Hurst Exponent (HE) are used to illustrate the energy characteristics and stability of the AE signals in the chosen frequency band, respectively. The recognition algorithm combines the two different features is performed on a embedded device in the experiments. Finally, the experiments are carried out to demonstrate the effectiveness of the proposed drilling state recognition method.

Category hierarchies are commonly used to compactly represent large numbers of categories and reduce the complexity of the classification problem. In this paper we introduce a novel and extended application of category hierarchies which is a powerful novel framework developed to construct dynamic category hierarchies and automatically discover novel visual categories. The dynamic is a characteristic of category hierarchies which can facilitate an important cognitive ability, the discovering of novel categories. We develop a constrained hierarchical latent Dirichlet allocation to build accurate category hierarchies. We employ object attributes as features to describe objects, which can transfer knowledge across categories and can efficiently describe novel categories. By combining them in the novel framework, novel visual object categories can be efficiently discovered and described. Extensive experiments based on PASCAL VOC 2008 and the LabelMe image database show the satisfactory performance of the proposed framework.

Tasks planning under uncertainties is one of fundamental skills for enabling autonomous robots to make proper manipulations in the complex environment. But owing to inexpressive representations, autonomous robots hardly conduct efficient tasks planning, especially in unknown conditions. The application of semantic knowledge in task planning is critically required in artificial intelligence research. In this paper, we focus on two topics: semantic knowledge representations and parallel planning for uncertainties. Firstly, a semantic memory system which is called EpistemeBase is proposed for indoor tasks planning, it includes five parallel agents: Assertion, Plan, Anticipation, Behaviour and Effect. Its framework is an evolving process, which consists of Datum, Information, Knowledge and Intelligence. Secondly, the same task planning is synchronously represented by five paralleled agents. This paralleled structure can well accelerate the process of tasks planning as well as better handle it under uncertainties. Finally, the experiment of tasks planning is conducted for measuring the reaction time of planning and uncertainties by using the EpistemeBase and the Open Mind Common Sense (OMCS) respectively.