Autonomous Driving Systems (ADS) are critical components of Intelligent Transportation Systems (ITS), where vehicle-level reliability has a direct bearing on road traffic safety. Evaluating ADS performance in complex environments remains challenging due to the absence of test oracles and the heavy reliance on deep learning. To address these challenges, this study proposes a novel metamorphic testing framework tailored for vision-based ADS. First, causal inference is employed to extract key environmental factors from high-dimensional observational traffic data, thereby reducing the test space. Second, a multi-objective optimization algorithm integrating causal counterfactual reasoning is developed to quantify the challenges associated with specific combinations of causal factors, enabling cost-effective exploration of test conditions. Third, low-risk source images are systematically transformed into hazardous driving scenes through a fine-tuned diffusion model, allowing ADS evaluation to be guided by metamorphic relations (MRs). Empirical experiments show that the proposed method achieves a higher fault detection ratio than the strongest baseline in four out of five ADS models, with relative gains ranging from 18.1% to 88.9%. Data augmentation experiments further demonstrate that incorporating MR-violating test cases can reduce ADS prediction errors by up to 13.67%, with these benefits preserved in real-world road traffic datasets through domain adaptation. This study highlights a new pathway for validating the reliability of vision-based ADS driven by deep learning, thereby supporting the deployment of safer road transportation. The source code for our methods and baselines is available athttps://github.com/SafeDL/AutoMetTest

Driving decision-making in mixed traffic, characterized by high-dynamic interactions and stochastic behaviors of human-driven vehicles, poses significant challenges for autonomous driving systems. To address these issues, we propose a novel Transformer-based Spatial Temporal Fusion (TSTF) module integrated with an auxiliary contrastive learning task within a multi-agent reinforcement learning (MARL) framework. The TSTF module captures interaction-aware behaviors and long-term temporal dependencies that tackle mixed cooperative driving scenarios, while the auxiliary contrastive learning task refines feature representations to enhance exploration efficiency and decision stability. Experimental evaluations on the MetaDrive platform demonstrate that the proposed approach outperforms baseline algorithms in safety, adaptability and robustness to dynamic traffic scenarios. The results highlight the effectiveness of the TSTF module in enabling robust and context-aware collaborative driving behaviors, offering a scalable solution for real-world mixed traffic. This work advances MARL by addressing key challenges in interaction modeling and driving decision-making under uncertainty, with significant implications for the development of intelligent transportation systems.

Magnetic continuum robots (MCRs) with active steering capability hold great promise for improving interventional surgery due to their flexibility and controllability. However, achieving real-time tracking and automatic navigation of MCRs in tissue-mimicking multi-bifurcated vessels remains a significant challenge. This work proposes an ultrasound-guided real-time automatic navigation framework for magnetic guidewire robots to improve interventional surgery, including modeling, simulation, tracking and control. An ultrasound-guided magnetically controlled guidewire robot system (UMCGRS) is designed and validated in 3D vascular phantom. An equilibrium guidewire model is established to describe the quasi-static behavior of MCRs in a permanent magnetic field and to derive the control Jacobian for guidewire tip control, which is validated by magnetic navigation simulation. A network-based real-time ultrasound tracking method is developed for accurate guidewire detection (average detection error of 0.81 mm across various vessels), and a model-based path tracking control strategy is proposed for guidewire navigation. Experiments in a femoral artery gelatin phantom with tissue-mimicking environments demonstrate the effectiveness of the tracking and control (average tracking error of 1.50 ± 0.30 mm). The proposed UMCGRS and automatic navigation framework are expected to enhance the autonomy of MCRs, and will provide a reliable solution for improving interventional surgery.

Magnetic microrobots have garnered significant attention and hold great potential for biomedical research applications. However, achieving precise manipulation in vivo poses significant challenges, particularly in medical image-based real-time feedback control, because it is difficult for a visual camera to track the motion of magnetic microrobots inside the body in biomedical applications. To realize the precise control of magnetic microrobots, it is also necessary to design and implement a simple and powerful control method. This approach allows for avoiding resource-intensive and complex control strategies. In this article, we present a learning-based real-time control method utilizing ultrasound images. Inspired by the ADboost concept, we use a reinforcement learning approach to integrate two simple control methods: a proportional-integral-derivative controller and a guiding vector field controller. We develop a novel$Q$-learning method called average$Q$-learning that incorporates average operation and$n$-step bootstraps. Its primary objective is to dynamically adjust the outputs of the different simple controllers. While each controller individually offers a straightforward solution, their integration contributes to a powerful control approach. To demonstrate its scalability, a nonsmooth path is utilized to investigate the integration performance of three simple controllers. In addition, we enhance a classic segmentation module, U-net, by incorporating an atrous spatial pyramid pooling module. To validate the effectiveness of the proposed control method, we conduct simulations and experiments using various planar paths. The quantitative analysis of the results demonstrates the efficacy of our approach in achieving precise manipulation, leveraging real-time control based on medical images for magnetic microrobots. Overall, this study provides a preliminary investigation into the field of medical image-based precise manipulation of magnetic microrobots in vivo applications.

Test the safety of Autonomous Driving Systems (ADS) with realistic traffic conditions is important to the insurance industry, legislators, and third-party technical services. Approaches for ADS testing can be divided into two main categories: physical test and virtual test, as shown in Fig. 1.

Visual-based object detection has become a crucial component in the realm of autonomous vehicles. However, conducting reliable testing for such systems remains unresolved. In this paper, we advocate for the application of causal inference to investigate the pivotal environmental factors influencing detection accuracy. Through the integration of diffusion models, we address the specialized conditional generalization of hazardous testing images. Our approach involves the construction of observational data to attribute key factors and fine-tune the diffusion model. Additionally, we introduce an optimal prompt words search method that strikes a balance between test coverage and level of challenge. Subsequently, leveraging these optimal prompts, we propose a cost-effective testing image generation through both "Text2Scene" and "Image2Scene" fashions. The experimental results indicate that, on the generalized dataset, the performance of object detection algorithms is the poorest, with the average detection accuracy decreasing from 0.81 to 0.285. Moreover, retraining object detection models on our generalized critical test cases can ultimately enhance algorithm performance, achieving a median accuracy improvement of up to 8.13%. Overall, our research proposes a novel approach to generalize test cases, thereby contributing to the advancement and deployment of safer autonomous vehicles.

Reinforcement learning has been shown to have great potential applications in autonomous driving. For collaborative driving scenarios, multi-agent reinforcement learning can be used to explore efficient and collaborative driving strategies. However, it still faces the challenge of non-stationary. Traditional methods focus on evaluating the similarities between the real state of the teammate and the modeled state. There is also the issue of partial observability. It can be addressed by establishing communication to share information with other surrounding agents. However, prior approaches overlook the efficient communication problem caused by unprocessed and redundant information. To tackle these two challenges, we propose an approach named Multi-Agent Collaboration via Cross-Attention and Communication (MACAC). MACAC leverages the agent’s local observations to analyze and capture environment and interaction information, while also incorporating teammate modeling through the exchange of concise state information via communication. In addition, to improve the learning process, we integrate the noisy advantage technique into MACAC to enhance the agent’s exploration capabilities. As a result, vehicles can effectively adapt to dynamic environments and exhibit efficient collaborative driving skills. In all, experiments conducted on an autonomous driving simulator demonstrate that our approach surpasses the performance of the baseline algorithms.

Magnetic micro/mini-swimmers have a great potential application in biomedical research and have gained broad attention. Recent research studies focus on the automatic control methods of magnetic micro/mini-swimmers, such as artificial intelligence methods. However, their autonomous manipulation remains a challenge since they are subject to various disturbances from the external environment and model uncertainties. The current methods employ a status observer to estimate these disturbances and uncertainties. In this paper, we apply a data-driven technique that utilizes the nonlinear approximation ability of neural networks (NNs). To be specific, a flexible structure of NNs, Broad Learning System (BLS), is employed to model the input-output mapping relationship between the direction of the rotating magnetic field and the swimming direction of the helical miniature swimmer. Then, according to the dual-variable decoupling control, a levitation controller is formulated and a path following controller is proposed based on a planar three-degree-of-freedom model of magnetic helical swimmers, which is inspired by the model of wheeled mobile robots. Simulations and experiments are conducted to quantitatively validate the proposed control method using different planar paths. The experiment results show that the mean absolute error of the path following control is about 3% of the body length which is less than$0.5mm$. Our proposed control method provides a preliminary study to alleviate the impact of disturbances and uncertainties on the control performance of magnetic micro/mini-swimmers. Note to Practitioners—This paper is mainly motivated by the potential application of artificial intelligence methods in magnetic micro/mini-robot community, especially for the applications that reject disturbances and uncertainties. In practice, a path planner is employed to compute a pre-defined reference path that connects the start and the targeted locations. Then our control method is applied to the magnetic helical swimmer and it is guided to follow the reference path. Tackling external disturbances and model uncertainties is still challenging during the control progress. Neural network-based technique is an intuitive approach since their nonlinear modeling ability and generalization are suitable for estimating these disturbances and uncertainties. As for training the neural-network model, the training data about the angle parameters should be recorded by manual control of the helical swimmer. These angle parameters define the rotating axis of the uniform rotating magnetic field and the self-rotating axis of the magnetic helical swimmer. According to the planar motion model, the optimal controller is formulated using the feedback distance error and angle error, and the sum of the control signal and the prediction of the compensating model is used as the final control input. Our electromagnetic coil system features easy operation and configuration of cameras or other sensors. Simulations and experiments validate the performance of the neural network-based compensating method and the proposed optimal control method using magnetic micro/mini-swimmers.

Automated driving systems (ADS) have made remarkable progress in recent years, yet their reliability and testability remain as significant challenges. The environmental conditions that ADS face are highly complex and may result in the disruption of autonomous vehicles. In this study, we propose an approach that leverages causal inference theory to analyze the impact of causal factors on automated driving visual modules. Our method uncovers the root key factors that affect visual perception performance. We further establish a Challenging Index to quantitatively characterize the causal effects of the key factors on perception failures. This quantitative index is subsequently utilized to generate risky scenarios. Through extensive experiments on various state-of-the-art automated driving visual algorithms, we demonstrate the effectiveness of the challenge index in evaluating the level of hazard in the deployment environment. Additionally, the proposed “challenge index guided search” method improves test efficiency by up to 8.95 times compared to the baselines while maintaining a balance between coverage diversity and the hazardous level of test scenarios. Our research offers a new perspective for analyzing and evaluating the impact of key factors on visual perception. This contributes to the reduction of test space and efficiency of the generation of high-value test scenarios, ultimately advancing the deployment of safer automated vehicles.

Microswimmer and miniswimmer toward precision-targeted medicine have attracted extensive attention recently. We have developed an autonomous manipulation approach for magnetic-driven helical miniswimmer at low Reynolds number in the horizontal plane ($H$-plane). Different from our previous work which just makes the barycenter of miniswimmer on the reference path as well as takes the swimming direction not into consideration in planar path following, our control policy in this article can make the miniswimmer to follow the reference path and, at the same time, its swimming direction is also along the reference path. A robust tracking method is employed to locate the helical miniswimmer in real time. Due to different external disturbances, an angle compensating model in the global coordinate frame is developed by radial basis function (RBF) networks trained by backpropagation algorithms, which is used to express the swimming model of the helical miniswimmer facing the gravity and lateral disturbances. A discrete-time optimal controller is formulated based on the linear-quadratic feedback control. Simulations and experiments are conducted to quantitatively validate the autonomous manipulation, and the results show the control performance with submillimeter accuracy in the$H$-plane.Note to Practitioners:This article is motivated by the potential application of precision-targeted medicine using magnetic-driven microswimmer/miniswimmer. The formulated controller employs the error model in the horizontal plane to design the control law. Simulations and experiments validate the effectiveness of the proposed discrete-time optimal control scheme using magnetic-driven miniswimmers.

As the controller parameter adjustment process is simplified significantly by using learning algorithms, the studies about learning-based control attract a lot of interest in recent years. This article focuses on the intelligent servo control problem using learning from desired demonstrations. Compared with the previous studies about the learning-based servo control, a control policy using the broad learning system (BLS) is developed and first applied to a microrobotic system, since the advantages of the BLS, such as simple structure and no-requirement for retraining when new demos' data is provided. Then, the Lyapunov theory is skillfully combined with the complex learning algorithm to derive the controller parameters' constraints. Thus, the final control policy not only can obtain the movement skills of the desired demonstrations but also have the strong ability of generalization and error convergence. Finally, simulation and experimental examples verify the effectiveness of the proposed strategy using MATLAB and a microswimmer trajectory tracking system.

Steering microswimmers toward 3-D autonomous manipulation tasks has received extensive attention. Our previous works have accomplished autonomously manipulating microswimmers in the 2-D space. This article aims to extend the 2-D autonomous manipulation to 3-D autonomous manipulation. Specifically, this article addresses the problem of an autonomous system that consists of 3-D path planning and 3-D path following for magnetically driven helical microswimmers. The path-planning algorithm called optimal Bidirectional RRT* is formulated to explore the shortest route in the confined 3-D space. A proxy-based sliding mode control (PSMC) approach is developed to design stable controllers based on the error model in the Serret–Frenet frame. We transport the swimming model trained by a kind of neural network to another new helical microswimmer according to an online updating scheme. The updating scheme can identify and refine compensating angles between the swimming direction of the microswimmer and the magnetic direction in the 3-D space facing the weight disturbances of the swimmer and lateral disturbances. The experiments are conducted to quantitatively validate the 3-D autonomous manipulation system. Experimental results show the effectiveness of path planning and path following with submillimeter accuracy in a 3-D space. Future works will focus on autonomous manipulations in dynamic environments.Note to Practitioners—This article is motivated by the issue of 3-D autonomous manipulation tasks for magnetically driven helical microswimmers. The formulated path planning is responsible for finding the shortest route in the 3-D confined space. The closed-loop controller is charge of steering the helical microswimmers on a reference path based on an online updating model trained by neural networks. It is demonstrated that the helical microswimmer can find the shortest path and follow it in a 3-D space with submillimeter accuracy.

Controlling magnetic microswimmers toward 3-D manipulation tasks has received considerable attention. Although related studies on manipulating helical microswimmers have been developed, stable closed-loop controls and accuracy swimming models should be still investigated. This article addresses the problem of 3-D path following for magnetically driven helical microswimmers with an adaptive-compensation scheme. The orientation-compensation model in the global coordinate frame is learned by radial basis function (RBF) networks trained with backpropagation algorithms, which is used to express the motion of the helical microswimmer in the presence of the weight of the swimmer and lateral disturbances from the boundary effects. A proxy-based sliding-mode control (PSMC) approach is developed to design stable controllers based on the kinematic error model. The effects of variable parameters and boundary effects are also considered. Experimental results including different paths in 3-D space validated the path following with submillimeter accuracy using the helical microswimmer. Note to Practitioners-This article is motivated by the issue of the following predefined paths for magnetically driven helical microswimmers in 3-D space. The proposed closed-loop controller employs the error model in 3-D space to formulate the control law according to an orientation-compensation model learned by neural networks. It is demonstrated that the helical microswimmer is able to follow different paths in 3-D space with submillimeter accuracy using the proposed control scheme.

Magnetically actuated microswimmers have attracted researchers to investigate their swimming characteristics and controlled actuation. Although plenty of studies on actuating helical microswimmers have been carried out, robust closed-loop controls should be still explored for practical applications. In this paper, we proposed a data-driven model-free method using Image-Based Visual Servoing (IBVS), which uses features directly extracted in the image space as feedbacks. The IBVS method can eliminate camera calibration errors. We have demonstrated with experiments that the proposed IBVS method can enable velocity-independent path following of an arbitrarily given path on the plane, which permits a better experience of user interaction. The proposed control method is successfully applied to obstacle avoidance tasks and has the potential for the application in complex circumstances. This approach is promising for biomedical applications.

In this paper, controlling miniature swimmers via electromagnetic actuation has received considerable attention due to their minor invasive trait in various biomedical applications and ease of passing through the complex environments. Studying the navigation and control system is an essential step towards such applications. Currently, navigation and control for magnetically driven miniature swimmers are still challenging research issues. This paper aims to formulate a navigation and control system of magnetically driven helical miniature swimmers. First, a global planning algorithm named informed optimal random exploring tree (Informed RRT*) is applied to compute the feasible path in cluttered environments. Second, a closed-loop control algorithm is presented to follow various of reference paths using visual feedback on a planar substrate. In particular, a single-hidden layer feedforward neural networks is employed to approximate the mapping relationship between the magnetic self-rotation direction and the actual moving direction of helical miniature swimmers. The neural network is first implemented to control the magnetically driven miniature swimmers in this paper. Experiments are conducted to verify the ability of navigation and visual feedback control tasks.

Untethered and wirelessly-controlled microrobots have many applications in the field of biomedicine. Therefore, many laboratories and scientists have invested more scientific research into magnetic microrobots which can make more contributions to medical care. Many magnetic field devices and microrobots are manufactured. In the development of micro-robots, helical microrobots have been well developed. Rigid-body robots account for the majority of these, but they may cause damage to human organs during treatment. However, soft and deformable robots can relieve more medical restrictions. In general, helical microrobots are driven by uniform fields which have their own limitations while the gradient magnetic field can relieve more restrictions and have more functions. This paper presents a flexible deformable helical swimmer controlled in a rotating gradient magnetic field. Helical swimmers are covered with magnetic nano-particles and the helical structure is derived from the inner fiber structure of the lotus root. The soft helical swimmers are controlled to swim several special trajectories in the rotating gradient magnetic field and we analyze the frequency and other factors for velocity or other effects.

Magnetically actuated microswimmers have shown great potentials in multiple application scenarios, attracting researchers to investigate their swimming characteristics and controlling methods. However, among those studies, only a small number of closed-loop control schemes have been applied, which is crucial for the accuracy and repeatability in applications. In this paper, we proposed an Image-Based Visual Servoing(IBVS) method for arbitrary planar path following using features directly presented in image space as feedbacks. During experiments we found that IBVS guarantees convergence while not requiring camera calibrations. Furthermore, the proposed path following method is intuitive and flexible, and provides great potentials in various applications.