Accurate battery capacity estimation is key to alleviating consumer concerns about battery performance and reliability of electric vehicles (EVs). However, practical data limitations imposed by stringent privacy regulations and labeled data shortages hamper the development of generalizable capacity estimation models that remain robust to real-world data distribution shifts. While self-supervised learning can leverage unlabeled data, existing techniques are not particularly designed to learn effectively from challenging field data—let alone from privacy-friendly data, which are often less feature-rich and noisier. In this work, we propose a first-of-its-kind capacity estimation model based on self-supervised pretraining, developed on a large-scale dataset of privacy-friendly charging data snippets from real-world EV operations. Our pre-training framework,snippet similarity-weighted masked input reconstruction, is designed to learn rich, generalizable representations even from less feature-rich and fragmented privacy-friendly data. Our key innovation lies in harnessing contrastive learning to first capture high-level similarities among fragmented snippets that otherwise lack meaningful context. With our snippet-wise contrastive learning and subsequent similarity-weighted masked reconstruction, we are able to learn rich representations of both granular charging patterns within individual snippets and high-level associative relationships across different snippets. Bolstered by this rich representation learning, our model consistently outperforms state-of-the-art baselines, achieving 31.9% lower test error than the best-performing benchmark, even under challenging domain-shifted settings affected by both manufacturer and age-induced distribution shifts.

Traditional reinforcement learning-based robotic control methods are often task-specific and fail to generalize across diverse environments or unseen objects and instructions. Visual Language Models (VLMs) demonstrate strong scene understanding and planning capabilities but lack the ability to generate actionable policies tailored to specific robotic embodiments. To address this, Visual-Language-Action (VLA) models have emerged, yet they face challenges in long-horizon spatial reasoning and grounded task planning. In this work, we propose the Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning, EMMA-X. EMMA-X leverages our constructed hierarchical embodiment dataset based on BridgeV2, containing 60,000 robot manipulation trajectories auto-annotated with grounded task reasoning and spatial guidance. Additionally, we introduce a trajectory segmentation strategy based on gripper states and motion trajectories, which can help mitigate hallucination in grounding subtask reasoning generation. Experimental results demonstrate that EMMA-X achieves superior performance over competitive baselines, particularly in real-world robotic tasks requiring spatial reasoning.

Traditional exteroceptive sensors in mobile robots, such as LiDARs and cameras often struggle to perceive the environment in poor visibility conditions. Recently, radar technologies, such as ultra-wideband (UWB) have emerged as potential alternatives due to their ability to see through adverse environmental conditions (e.g. dust, smoke and rain). However, due to the small apertures with low directivity, the UWB radars cannot reconstruct a detailed image of its field of view (FOV) using a single scan. Hence, a virtual large aperture is synthesized by moving the radar along a mobile robot path. The resulting synthetic aperture radar (SAR) image is a high-definition representation of the surrounding environment. Hence, this paper proposes a pipeline for mobile robots to incorporate UWB radar-based SAR imaging to map an unknown environment. Finally, we evaluated the performance of classical feature detectors: SIFT, SURF, BRISK, AKAZE and ORB to identify loop closures using UWB SAR images. The experiments were conducted emulating adverse environmental conditions. The results demonstrate the viability and effectiveness of UWB SAR imaging for high-resolution environmental mapping and loop closure detection toward more robust and reliable robotic perception systems.

To perform target-following tasks in unknown environments, a robot must identify the target’s position and plan an efficient path to reach it. Traditional LiDAR-based localization systems face challenges in distinguishing the target from objects with similar appearances. Meanwhile, existing target-following approaches often neglect target visibility during path planning, leading to target occlusion by obstacles and ultimately resulting in following failure. In this paper, we propose a sequence matching method for target-localization using LiDAR and Ultra-Wideband (UWB) ranging. We determine the position of the target by analyzing the similarities between UWB ranging sequence and LiDAR cluster trajectories. To achieve visibility-aware target-following, we incorporate a visibility objective function into the Dynamic Window Approach (DWA) to generate a following path that minimizes the risk of target loss. This function evaluates the target loss risk based on the positional relationships between the robot, the target, and the nearest obstacle to the target. Extensive experiments were conducted using both human and robot as targets. The results show that our approach achieves higher completion rates when compared to the target-following using traditional DWA.

Ultraviolet (UV) germicidal radiation is an established non-contact method for surface disinfection in medical environments. Traditional approaches require substantial human intervention to define disinfection areas, complicating automation, while deep learning-based methods often need extensive fine-tuning and large datasets, which can be impractical for large-scale deployment. Additionally, these methods often do not address scene understanding for partial surface disinfection, which is crucial for avoiding unintended UV exposure. We propose a solution that leverages foundation models to simplify surface selection for manipulator-based UV disinfection, reducing human involvement and removing the need for model training. Additionally, we propose a VLM-assisted segmentation refinement to detect and exclude thin and small non-target objects, showing that this reduces mis-segmentation errors. Our approach achieves over 92% success rate in correctly segmenting target and non-target surfaces, and real-world experiments with a manipulator and simulated UV light demonstrate its practical potential for real-world applications.

Collaborative multiple robots for unknown environment exploration have become mainstream due to their remarkable performance and efficiency. However, most existing methods assume perfect robots’ communication during exploration, which is unattainable in real-world settings. Though there have been recent works aiming to tackle communication-constrained situations, substantial room for advancement remains for both information-sharing and exploration strategy aspects. In this paper, we propose a Communication-Constrained Multi-Robot Entropy-Field-Based Exploration (MEF-Explore). The first module of the proposed method is the two-layer inter-robot communication-aware information-sharing strategy. A dynamic graph is used to represent a multi-robot network and to determine communication based on whether it is low-speed or high-speed. Specifically, low-speed communication, which is always accessible between every robot, can only be used to share their current positions. If robots are within a certain range, high-speed communication will be available for inter-robot map merging. The second module is the entropy-field-based exploration strategy. Particularly, robots explore the unknown area distributedly according to the novel forms constructed to evaluate the entropies of frontiers and robots. These entropies can also trigger implicit robot rendezvous to enhance inter-robot map merging if feasible. In addition, we include the duration-adaptive goal-assigning module to manage robots’ goal assignment. The simulation results demonstrate that our MEF-Explore surpasses the existing ones regarding exploration time and success rate in all scenarios. For real-world experiments, our method leads to a 21.32% faster exploration time and a 16.67% higher success rate compared to the baseline.

Introducing robotics in the healthcare sector revolutionizes medical services by providing advanced treatments, medication management, and robotic assistance while overcoming resource limitations. In the current healthcare domain, an intermediate robotic communication platform is essential for distributing equal medical services, facilitating remote consultations, and maintaining the integrity of medical education, especially in rural areas and during pandemics. This work introduces ARIS, a multitasking medical service robot designed for telemedicine aspects and to facilitate remote medical education activities such as ward rounds. The prototype called ARIS 1.0 was developed, including a three-wheeled omnidirectional mobile platform, a torso and a novel movable neck mechanism with a face. The prototype robot can generate an online summarized report using its integrated language interaction and IoT-based vital sign extraction modules. The ROS-based semi-autonomous navigation facilitates the robot to be an assistive agent, allowing it to either accompany doctors or visit patients individually. Ultimately, ARIS 1.0 serves telepresence and novel regional language capabilities, specifically Sinhala-based self-communication features. This enables inter-party communication among doctors, medical students, and patients. The functionalities of ARIS 1.0 were validated in an emulated indoor environment to evaluate their feasibility. The results indicate that ARIS 1.0 is feasible for providing remote medical services. Furthermore, the paper discusses several promising research directions related to the proposed concept.

Cleaning glass surfaces is a prevailing maintenance problem in high-rise buildings. In the traditional methods of cleaning windows, hanging on ropes poses significant occupational hazards to workers. Furthermore, most glass facades feature window frames to securely fasten the glass panels to the building structure, ensuring durability and elegance. In this context, existing robotic cleaning methods are limited by their capability to move-over window frames and need more flexibility to access tight corners and curved surfaces. This paper presents a novel reconfigurable glass cleaning robot called "WiBot" to address these limitations. WiBot is a kinematic chain comprising modular linkages with a prismatic joint and two revolute joints at each end. Each revolute joint has a suction unit that enables locomotion and adhesion. Window frames are detected using image processing with an onboard camera, and design optimizations were performed to improve the robot’s capabilities. The prototype WiBot 1.0 was developed, and several experiments were conducted to evaluate the feasibility of the proposed system focusing on robot motion, window frame detection and move-over mechanism. The results show that WiBot can overcome the limitations of existing window cleaning solutions. Finally, several promising research directions are mentioned involving the proposed reconfigurable robot architecture in cleaning operations.

The rapid advancements in unmanned aerial vehicles (UAVs) have unlocked numerous applications, including environmental monitoring, disaster response, and agricultural surveying. Enhancing the collective behavior of multiple decentralized UAVs can significantly improve these applications through more efficient and coordinated operations. In this study, we explore a Recurrent PPO model for target localization in perceptually degraded environments like places without GNSS/GPS signals. We first developed a single-drone approach for target identification, followed by a decentralized two-drone model. Our approach can utilize two types of sensors on the UAVs, a detection sensor and a target signal sensor. The single-drone model achieved an accuracy of 93%, while the two-drone model achieved an accuracy of 86%, with the latter requiring fewer average steps to locate the target. This demonstrates the potential of our method in UAV swarms, offering efficient and effective localization of radiant targets in complex environmental conditions.

Odometry is commonly used in localization applications especially with wheeled platforms since encoders are readily available. It is often used by itself or fused with other sensor data to obtain a better estimate. However, its limitation is its exclusivity to wheeled platforms whereas it is often desired to have similar encoder odometry options on other systems. Given that WiFi is ubiquitous in most commercial and industrial areas, in this paper, a method is proposed for obtaining odometry from WiFi scans for position estimation. The method is not constrained to wheel robots such as the case for wheeled odometry and does not rely on the traditional fingerprinting method. The proposed method involves training a neural network model to predict the distance moved based on features extracted from WiFi scans in the environment. These distances moved are then summed up to obtain the trajectory. Experiments are conducted and the methods are evaluated based on Root Mean Square Error (RMSE). Experimental results showed that the proposed method is able to achieve an RMSE of at most 8.39m for the various test cases.Note to Practitioners—This paper was motivated by the limited sensors available for odometry. Existing methods of odometry either require a wheeled platform or exteroceptive sensors to be placed outside of the robot so that it can see the environment. This paper proposes a new and low-cost method of performing odometry using a WiFi receiver and Inertial Measurement Unit (IMU) with a neural network model. This provides an alternative that exploits existing WiFi infrastructure and thus more flexibility in robot design without wheels and sensor placement constraints. We show how the features are selected as well as propose several similarity methods to choose from. We then show how the neural network model is trained and used during implementation. Preliminary physical experiments suggest that the method was able to obtain the trajectory of a robot in two different environments using the same model and different speeds.

General Visual Inspection is a manual inspection process regularly used to detect and localise obvious damage on the exterior of commercial aircraft. There has been increasing demand to perform this process at the boarding gate to minimise the downtime of the aircraft and automating this process is desired to reduce the reliance on human labour. Automating this typically requires estimating a camera’s pose with respect to the aircraft for initialisation but most existing localisation methods require infrastructure, which is very challenging in uncontrolled outdoor environments and within the limited turnover time (approximately 2 hours) on an airport tarmac. Additionally, many airlines and airports do not allow contact with the aircraft’s surface or using UAVs for inspection between flights, and restrict access to commercial aircraft. Hence, this paper proposes an on-site method that is infrastructure-free and easy to deploy for estimating a pan-tilt-zoom camera’s pose and localising scan images. This method initialises using the same pan-tilt-zoom camera used for the inspection task by utilising a Deep Convolutional Neural Network fine-tuned on only synthetic images to predict its own pose. We apply domain randomisation to generate the dataset for fine-tuning the network and modify its loss function by leveraging aircraft geometry to improve accuracy. We also propose a workflow for initialisation, scan path planning, and precise localisation of images captured from a pan-tilt-zoom camera. We evaluate and demonstrate our approach through experiments with real aircraft, achieving root-mean-square camera pose estimation errors of less than 0.24 m and 2$^\circ$for all real scenes.

With the growing prevalence of online learning, ensuring student engagement and attention during online lectures has become a challenge. Unlike traditional classroom settings, where students are physically present and can interact with their peers and the instructor, online lectures can feel more passive and isolating. To address this challenge, we investigate the use of eye-tracking data to identify distractions during online lectures. By analyzing this data, it becomes possible to gain insights into a person's attention and focus. To achieve this, we employed a technique to approximate the raw gaze data using piecewise linear functions, where each segment represents an eye movement event such as fixation and saccade. These segments are then used to extract important features that distinguish between eye gaze time series before and after the distractor stimuli in the online lecture. We then train a binary classifier using the extracted features and also rank the importance of the features. The classifier achieves an accuracy of 73.6 % in classifying gaze timeseries as a distraction or no distraction.

To accomplish task efficiently in a multiple robots system, a problem that has to be addressed is Simultaneous Localization and Mapping (SLAM). LiDAR (Light Detection and Ranging) has been used for many SLAM solutions due to its superb accuracy, but its performance degrades in featureless environments, like tunnels or long corridors. Centralized SLAM solves the problem with a cloud server, which requires a huge amount of computational resources and lacks robustness against central node failure. To address these issues, we present a distributed SLAM solution to estimate the trajectory of a group of robots using Ultra-WideBand (UWB) ranging and odometry measurements. The proposed approach distributes the processing among the robot team and significantly mitigates the computation concern emerged from the centralized SLAM. Our solution determines the relative pose (also known as loop closure) between two robots by minimizing the UWB ranging measurements taken at different positions when the robots are in close proximity. UWB provides a good distance measure in line-of-sight conditions, but retrieving a precise pose estimation remains a challenge, due to ranging noise and unpredictable path traveled by the robot. To deal with the suspicious loop closures, we use Pairwise Consistency Maximization (PCM) to examine the quality of loop closures and perform outlier rejections. The filtered loop closures are then fused with odometry in a distributed pose graph optimization (DPGO) module to recover the full trajectory of the robot team. Extensive experiments are conducted to validate the effectiveness of the proposed approach.

General Visual Inspection is a manual inspection process regularly used to detect and localise obvious damage on the exterior of commercial aircraft. There has been increasing demand to perform this process at the boarding gate to minimize the downtime of the aircraft and automating this process is desired to reduce the reliance on human labour. This automation typically requires the first step of estimating a camera’s pose with respect to the aircraft for initialisation. However, localisation methods often require infrastructure, which can be very challenging when performed in uncontrolled outdoor environments and within the limited turnover time (approximately 2 hours) on an airport tarmac. In addition, access to commercial aircraft can be very restricted, causing development and testing of solutions to be a challenge. Hence, this paper proposes an on-site infrastructure-less initialisation method, by using the same pan-tilt-zoom camera used for the inspection task to estimate its own pose. This is achieved using a Deep Convolutional Neural Network trained with only synthetic images to regress the camera’s pose. We apply domain randomisation when generating our dataset for training our network and improve prediction accuracy by introducing a new component to an existing loss function that leverages on known aircraft geometry to relate position and orientation. Experiments are conducted and we have successfully regressed camera poses with a median error of 0.22 m and 0.73°.

Relative localization between autonomous robots without infrastructure is crucial to achieve their navigation, path planning, and formation in many applications, such as emergency response, where acquiring a prior knowledge of the environment is not possible. The traditional Ultra-WideBand (UWB)-based approach provides a good estimation of the distance between the robots, but obtaining the relative pose (including the displacement and orientation) remains challenging. We propose an approach to estimate the relative pose between a group of robots by equipping each robot with multiple UWB ranging nodes. We determine the pose between two robots by minimizing the residual error of the ranging measurements from all UWB nodes. To improve the localization accuracy, we propose to utilize the odometry constraints through a sliding window-based optimization. The optimized pose is then fused with the odometry in a particle filtering for pose tracking among a group of mobile robots. We have conducted extensive experiments to validate the effectiveness of the proposed approach.

There has been an increasing demand to automate the non-patient care matters so that the clinical staff can focus on delivering patient care. For example, out-patients undergoing chemotherapy increases their toilet usage frequency due to the treatment. As they are undergoing chemotherapy, their output waste contains a level of chemical. This task is compulsory yet troublesome and time-consuming so it is often desired to be removed from the nursing staff for them to focus on patient care. Hence, in this paper, we propose a manipulator system to automatically dispose the bedpan used by patients undergoing chemotherapy. The main technical challenge lies in the removal of the bedpan from the commode as the interaction of the grasping is highly dynamic, along with the different conditions of the bedpans. To address this manipulation issue, a Residual Reinforcement Learning (RRL) method that leverages vision-based commode pose estimation and the reinforcement learning (RL)-based uncertainty compensation for improvement of the grasping accuracy is proposed to increase the robustness of the disposal. The experiments conducted show that the manipulator can dispose the bedpan without human intervention and the proposed method achieves a 100 % success rate while the traditional method without RL is only 50 %.

This work demonstrates the feasibility and benefits of using pointing gestures, a naturally-generated additional input modality, to improve the multi-modal comprehension accuracy of human instructions to robotic agents for collaborative tasks.We present M2Gestic, a system that combines neural-based text parsing with a novel knowledge-graph traversal mechanism, over a multi-modal input of vision, natural language text and pointing. Via multiple studies related to a benchmark table top manipulation task, we show that (a) M2Gestic can achieve close-to-human performance in reasoning over unambiguous verbal instructions, and (b) incorporating pointing input (even with its inherent location uncertainty) in M2Gestic results in a significant (30%) accuracy improvement when verbal instructions are ambiguous.

Place recognition is essential for SLAM system since it is critical for loop closure and can help to correct the accumulated drift and result in a globally consistent map. Unlike the visual slam which can use diverse feature detection methods to describe the scene, there are limited works reported to represent a place using single LiDAR scan. In this paper, we propose a segmentation-based egocentric descriptor termed Seed by using a single LiDAR scan to describe the scene. Through the segmentation approach, we first obtain different segmented objects, which can reduce the noise and resolution effect, making it more robust. Then, the topological information of the segmented objects is encoded into the descriptor. Unlike other reported approaches, the proposed method is rotation invariant and insensitive to translation variation. The feasibility of proposed method is evaluated through the KITTI dataset and the results show that the proposed method outperforms the state-of-the-art method in terms of accuracy.

Simultaneous localization and mapping (SLAM) has been extensively researched in past years particularly with regard to range-based or visual-based sensors. Instead of deploying dedicated devices that use visual features, it is more pragmatic to exploit the radio features to achieve this task, due to their ubiquitous nature and the widespread deployment of the Wi-Fi wireless network. This article presents a novel approach for collaborative simultaneous localization and radio fingerprint mapping (C-SLAM-RF) in large unknown indoor environments. The proposed system uses received signal strengths (RSS) from Wi-Fi access points (APs) in the existing infrastructure and pedestrian dead reckoning (PDR) from a smartphone, without a prior knowledge about map or distribution of AP in the environment. We claim a loop closure based on the similarity of the two radio fingerprints. To further improve the performance, we incorporate the turning motion and assign a small uncertainty value to a loop closure if a matched turning is identified. The experiment was done in an area of 130 m by 70 m and the results show that our proposed system is capable of estimating the tracks of four users with an accuracy of 0.6 m with Tango-based PDR and 4.76 m with a step counter-based PDR.

This article proposes a confidence-based approach for combining two visual tracking techniques to minimize the influence of unforeseen visual tracking failures to achieve uninterrupted vision-based control. Despite research efforts in vision-guided micromanipulation, existing systems are not designed to overcome visual tracking failures, such as inconsistent illumination condition, regional occlusion, unknown structures, and nonhomogenous background scene. There remains a gap in expanding current procedures beyond the laboratory environment for practical deployment of vision-guided micromanipulation system. A hybrid tracking method, which combines motion-cue feature detection and score-based template matching, is incorporated in an uncalibrated vision-guided workflow capable of self-initializing and recovery during the micromanipulation. Weighted average, based on the respective confidence indices of the motion-cue feature localization and template-based trackers, is inferred from the statistical accuracy of feature locations and the similarity score-based template matches. Results suggest improvement of the tracking performance using hybrid tracking under the conditions. The mean errors of hybrid tracking are maintained at subpixel level under adverse experimental conditions while the original template matching approach has mean errors of 1.53, 1.73, and 2.08 pixels. The method is also demonstrated to be robust in the nonhomogeneous scene with an array of plant cells. By proposing a self-contained fusion method that overcomes unforeseen visual tracking failures using pure vision approach, we demonstrated the robustness in our developed low-cost micromanipulation platform.

Automatic targeting of plant cells to perform tasks like extraction of chloroplast is often desired in the study of plant biology. Hence, this paper proposes an improved cell segmentation method combined with a robust tracking algorithm for vision-guided micromanipulation in plant cells. The objective of this work is to develop an automatic plant cell detection and localization technique to complete the automated workflow for plant cell manipulation. The complex structural properties of plant cells make both segmentation of cells and visual tracking of the microneedle immensely challenging, unlike single animal cell applications. Thus, an improved version of watershed segmentation with adaptive thresholding is proposed to detect the plant cells without the need for staining of the cells or additional tedious preparations. To manipulate the needle to reach the identified centroid of the cells, tracking of the needle tip is required. Visual and motion information from two data sources namely, template tracking and projected manipulator trajectory are combined using score-based normalized weighted averaging to continuously track the microneedle. The selection of trackers is influenced by their complementary nature as the former and latter are individually robust against physical and visual uncertainties, respectively. Experimental results validate the effectiveness of the proposed method by detecting plant cell centroids accurately, tracking the microneedle constantly and reaching the plant cell of interest despite the presence of visual disturbances.

There has been increasing demand in using multiple robots with human intervention for higher robustness in sectors like agriculture, clinical applications, environment surveillance, military operations, security. The benefits of using multiple robots include increased speed of missions largely due to ability to do parallel tasks and redundancy due to being able to replace a robot with another. However, at the same time, multiple robots substantially increase amount of information exchange with their operator. Therefore, designing interactions between human and multiple robots to achieve their effective cooperation has been a difficult research issue. In this paper, we propose an intuitive complementary user-interface design using speech and touch to control multiple robots simultaneously for a mission. This is achieved using a list of touch complement speech guidelines. We tested the system with six subjects who operated one, two and three robots in a simulation environment built using Unity. The mean performance score for various tasks and perceived cognitive load for each experiment iteration measured using the NASA-TLX questionnaire is then shown to illustrate the benefits of touch complement speech.

With the increase in autonomous car technology, and advance driver assistance systems (ADAS), the demand for vehicle trajectory prediction is increasing. These systems mostly use many sensors such as lidar, radar, stereo cameras which are big and expensive to detect the location of the ego vehicles with respect to other vehicles. Such sensors are not readily available to Vulnerable Road Users (VRU) such as motorcyclist, cyclist as they mostly only rely on their phones or a small device for navigation and safety warnings. Trajectory prediction is important to VRU as it is able to predict if the trajectory of other vehicles is a threat or not. Most of the current Trajectory prediction for vehicles involves plotting a relative position of other vehicles with respect to the ego vehicle and this is not possible with just a mobile device. Hence, this paper proposes a method to predict trajectory of other vehicles based on detected vehicles center coordinates. This is achieved using Gaussian Process Auto-Regression based on past data of vehicle center coordinates to predict future coordinates on an x-y pixel plane using only a camera sensor and You Only Look Once (YOLO) vehicle detection.

This paper presents a novel sliding mode controller for trajectory tracking of the piezo-driven stage. The tracking performance of piezoelectric actuator is mainly affected by the hysteresis nonlinearity. Sliding mode control is a possible solution to achieve better tracking performance. However, conventional sliding mode control generates discontinuous control signal which results in chattering. Hence, the hysteresis nonlinearity is first compensated with a hysteresis model, and an uncertainty and disturbance estimator is designed and included to devise a smooth control action. The stability of the proposed method is demonstrated via Lyapunov analysis. Both simulation and experiment are also conducted to verify the effectiveness of the proposed approach. The results are compared with a conventional sliding mode controller and a proportional-integral control with notch filter (PIC-NF).

Percutaneous Nephrolithotomy is a recommended treatment method for large kidney stone removal. However, the first and most important step, i.e., getting the percutaneous access to create the tract between the targeted calyx and the flank skin, is challenging as the surgeon is often occupied by several tasks at a given time. Therefore, in this paper, we propose a robotic assisted system that collaborates with the surgeon and provides assistance in order for the surgeons to focus on more critical jobs resulting in better surgical performance. A procedure for this robot including three working stages is described. This procedure allows the surgeon to choose a suitable percutaneous target using an ultrasound probe based on his or her experience and the robot will track the respiratory motion of the target kidney stone and insert the needle automatically after the surgeon releases the probe. Experiments are conducted to demonstrate the procedure with the proposed assisted robot for PCNL.

In this paper, an automatic vision-guided micromanipulation approach to facilitate versatile deployment and portable setup is proposed. This paper is motivated by the importance of micromanipulation and the limitations in existing automation technology in micromanipulation. Despite significant advancements in micromanipulation techniques, there remain bottlenecks in integrating and adopting automation for this application. An underlying reason for the gaps is the difficulty in deploying and setting up such systems. To address this, we identified two important design requirements, namely, portability and versatility of the micromanipulation platform. A self-contained vision-guided approach requiring no complicated preparation or setup is proposed. This is achieved through an uncalibrated self-initializing workflow algorithm also capable of assisted targeting. The feasibility of the solution is demonstrated on a low-cost portable microscope camera and compact actuated microstages. Results suggest subpixel accuracy in localizing the tool tip during initialization steps. The self-focus mechanism could recover intentional blurring of the tip by autonomously manipulating it 95.3% closer to the focal plane. The average error in visual servo is less than a pixel with our depth compensation mechanism showing better maintaining of similarity score in tracking. Cell detection rate in a 1637-frame video stream is 97.7% with subpixels localization uncertainty. Our work addresses the gaps in existing automation technology in the application of robotic vision-guided micromanipulation and potentially contributes to the way cell manipulation is performed.

Creating an environment to allow students to appreciate the linkage between subjects has been gaining increasing importance because interdisciplinary approaches are necessary to address socio-technological challenges. Subjects like Physics and Mathematics have often been taught as separate subjects. This results in students viewing various subjects as individual subjects, which is not ideal because there is no clear distinction between the subjects when dealing with real-life problems. For example, a number of students have a tendency to view Mathematics as only formula without applications, which result in them losing interest as they are unable to appreciate the vast number of applications that Mathematics can be applied in. In addition, it has been observed that a number of students are able to solve the Mathematics portion during Mathematics lesson, but are unable to evaluate similar Mathematics questions during Physics lessons. Hence, this paper proposes an integrated Physics and Mathematics learning and aims to help students establishing linkage between the two subjects. In order to achieve the learning objective of the students being able to appreciate the linkage between the two mentioned subjects, the syllabus is planned such that Physics is used as an application of Mathematics. The team has performed a preliminary study and implemented the proposed idea with a group of students in a bridging course. The bridging course is conducted for a duration of five days, whereby both Mathematics and Physics topics are covered. The students involved are incoming undergraduate students, and the data collected is analyzed. The preliminary results indicate a clear shift in enabling the students to appreciate the linkage between Physics and Mathematics with the integrated teaching of Physics and Mathematics.

In monocopter flight, two important parameters are required for control: angular rates and heading direction. Small monocopters fly at a very high speed (more than 600rpm), which can be out of the typical gyroscope limit. Very high speed gyroscopes do exist, but the price is high and it can only measure a single axis rotation. This paper presents an alternative approach to measure angular rates by using three accelerometers. The readings of the accelerometers are subtracted to calculate the angular rates in all three axes (x, y, and z). This paper also proposes to use the Extended Kalman Filter (EKF) to estimate the heading direction based on the magnetometer reading and the angular rates. The angular rates direction is used as the vertical direction reference. The proposed method has been applied on two setups: DC Motor setup (for quantifying the method's performance) and Monocopter setup. In the DC Motor setup, the motor encoder is used as the ground truth for the heading direction. The result is compared with the usual method of using only the magnetometer to obtain the heading direction of monocopters. The EKF result is more accurate and stable even in the presence of strong magnetic disturbances. In addition, the angle of attack and the coning angle can also be determined by the proposed method.

Relative positioning between multiple mobile users is essential for many applications, such as search and rescue in disaster areas or human social interaction. Inertial-measurement unit (IMU) is promising to determine the change of position over short periods of time, but it is very sensitive to error accumulation over long term run. By equipping the mobile users with ranging unit, e.g. ultra-wideband (UWB), it is possible to achieve accurate relative positioning by trilateration-based approaches. As compared to vision or laser-based sensors, the UWB does not need to be with in line-of-sight and provides accurate distance estimation. However, UWB does not provide any bearing information and the communication range is limited, thus UWB alone cannot determine the user location without any ambiguity. In this paper, we propose an approach to combine IMU inertial and UWB ranging measurement for relative positioning between multiple mobile users without the knowledge of the infrastructure. We incorporate the UWB and the IMU measurement into a probabilistic-based framework, which allows to cooperatively position a group of mobile users and recover from positioning failures. We have conducted extensive experiments to demonstrate the benefits of incorporating IMU inertial and UWB ranging measurements.

Robotic image-guided micromanipulation contributes towards the ease of operation, speed, accuracy, and repeatability in cell manipulation. However, such technology is not fully exploited because of the challenges in the integration of robotic modules with existing microscope systems, and the difficulty in incorporating robot assistance seamlessly into the workflow. In this paper, we propose a vision-based workflow algorithm termed Detect-Focus-Track-Servo (DFTS). It facilitates easy integration of robotic modules. It also supports user interactions while minimizing the need for manual intervention and disruption to workflow through automatic detection, focusing, tracking and servoing. Experimental results suggest satisfactory detection accuracy of 99.0 % at 70 μm tolerance. The robustness test suggests no difference in the accuracy under blurred and cluttered images. The self-focus algorithm is also demonstrated to bring the tip into focus consistently. The track-servo algorithm achieves low sub-pixel uncertainty. By proposing the DFTS workflow algorithm, we hope that the level of autonomy and ease of deployment in robot and vision modules for micromanipulation can be improved so as to open up new possibilities in the development of robotic image-guided cell manipulation.

In this paper, we propose a workflow algorithm for timely tracking of the tool tip during cell manipulation using a template-based approach augmented with low level feature detection. Doing so addresses the problem of adverse influences on template-based tracking during tool-cell interaction while maintaining an efficient track-servo framework. This consideration is important in developing autonomous robotic vision-guided micromanipulators. Our method facilitates vision-guided micromanipulation autonomously without manual interventions even during tool-cell interaction. This is done by decomposing the process to four scenarios that operate on their respective mode. The self-initializing mode is first used to localize and focus a region of interest (ROI) which the tip lies in. Once in focus, the tip is manipulated using a unified visual track-servo template-based approach. A reinitialization mechanism will be triggered to prevent tracking from being interrupted by partial cell occlusion of the tracking ROI. This mechanism uses the self-initializing concept combining motion cue and low-level feature detection to localize the needle tip. Following the reinitialization, we further recover tracking of the needle tip using a mechanism that updates the base template. This adaptive approach ensures uninterrupted tracking even when the cell is interacting with the tool and under deformation. Results demonstrated that with the newly incorporated mechanisms, the localized position improved from an error of more than 50% to less than 10% of the specimen size. When there is no specimen in the scene the new workflow shows no adverse effect on the localization through 270 tracked frames. By incorporating reinitialization and recovery to this workflow algorithm, we hope to initiate the first step towards uncalibrated autonomous vision-guided micromanipulation process.

Micromanipulation during live microscopic imaging relies heavily on good manual controls, dexterity, and hand-eye coordination. However, unassisted manual operations in these procedures greatly limit the speed, repeatability, and ease of operation. This is especially challenging in the case of microinjection where the insertion path needs to be in precise alignment with the imaging plane to avoid damage to cells. In this paper, we proposed an assistive robotic system that facilitates micromanipulation under microscopy. This comes in the form of intelligent robotic vision and guided manipulation. Using user-selected patch similarity, the system registers target templates and provides online coordinated depth compensation that ensures in-plane microinjection without the need for any prior calibration. This vision-based auto-registration approach readily integrates to any existing microscope system uncalibrated. It can also work as a standalone imaging solution with any general digital microscope camera. Experiments show that the similarity-score based depth compensation performed better than the uncompensated method. The method was shown to self-recover from an unfocused position. By robotizing conventional microscopy and micromanipulation procedures, we hope to address traditional latent needs and open up new possibilities in the ways experimental biology is performed.

Magnetic Resonance Imaging (MRI) is superior to other imaging modalities such as Ultrasound and Computed Tomography and is used for both diagnostic and therapeutic procedures. However, current breast biopsy procedures based on MR images obtained apriori, use a blind targeting approach, which can be long and painful. Current approaches, due to possible patient motion, can lead to tool tip positioning errors thereby affecting diagnostic accuracy and causing significant patient discomfort, if repeated procedures are required. Hence, it is desired to develop a MRI-compatible robot for breast biopsy procedures without removing the patient from the MRI bore. This approach could potentially avoid multiple biopsy needle insertions and minimize sampling errors. Due to the working principle of MRI, material, actuation, and sensing techniques are limited as the MR images must not be affected significantly during the procedure. In addition, the limited space of the MRI bore requires the robot to be compact. This paper presents a four degrees of freedom robot with a compact parallel mechanism of which three degrees of freedom are pneumatically actuated while the needle driver mechanism is actuated by a piezo motor. Fiber-optic force sensor is also designed, developed, and mounted on the top mobile platform of the parallel mechanism to sense the needle and tissue interaction forces. Position control of the pneumatic cylinders is implemented using PI control with a modified integration term to achieve a slow and smooth motion.

Magnetic resonance imaging (MRI) has been gaining popularity over standard imaging modalities like ultrasound and CT because of its ability to provide excellent soft-tissue contrast. However, due to the working principle of MRI, a number of conventional force sensors are not compatible. One popular solution is to develop a fiber-optic force sensor. However, the measurements along the principal axes of a number of these force sensors are highly cross-coupled. One of the objectives of this paper is to minimize this coupling effect. In addition, this paper describes the design of elastic frame structures that are obtained systematically using topology optimization techniques for maximizing sensor resolution and sensor bandwidth. Through the topology optimization approach, we ensure that the frames are linked from the input to output. The elastic frame structures are then fabricated using polymers materials, such as ABS and Delrin(®), as they are ideal materials for use in MRI environment. However, the hysteresis effect seen in the displacement-load graph of plastic materials is known to affect the accuracy. Hence, this paper also proposes modeling and addressing this hysteretic effect using Prandtl-Ishlinskii play operators. Finally, experiments are conducted to evaluate the sensor's performance, as well as its compatibility in MRI under continuous imaging.

Magnetic resonance imaging (MRI) has been gaining popularity over standard imaging modalities like ultrasound and CT because of its ability to provide excellent soft-tissue contrast. However, due to the working principle of MRI, a number of conventional force sensors are not compatible. One popular solution is to develop a fiber-optic force sensor. However, the measurements along the principal axes of a number of these force sensors are highly cross-coupled. One of the objectives of this paper is to minimize this coupling effect. In addition, this paper describes the design of an elastic frame structure that is obtained systematically by an algorithm and not purely based on design intuition. We used a topology optimization technique, which has two major advantages: 1) aids engineers in design when given a constrained boundary, and 2) optimize the displacement amplification, which will in turn increase stiffness, bandwidth, and improve sensing resolution. To ensure that the frames are linked from the input to output, a solution for topology optimization is proposed. The sensor is then fabricated using plastic material (ABS) as it is one of the ideal material for MRI environment. However, the hysteresis effect seen in the displacement-load graph of plastic materials is known to affect the accuracy. Hence, this paper also proposes modeling and addressing this hysteretic effect using Prandtl-Ishlinskii play operators. Finally, experiments are conducted to evaluate the sensor's performance, as well as its compatibility in MRI under continuous imaging.

Inertial measurement units (IMU) consist of accelerometers. Estimation accuracy of acceleration in a particular direction depends on how accurately accelerometers are placed at desired or ideal orientations. The estimation inaccuracy which results from inaccurate orientation of an accelerometer can be eliminated if the orientation error or the angle between the actual and the ideal orientations is known. This paper presents a method of identification of the accelerometer orientation errors without requiring any rotational motion of the IMU in which accelerometers are placed. It also presents a method of compensation for the inaccuracy of acceleration estimation due to the accelerometer orientation errors using angular motion information.

This paper presents a 3-DOF low-cost hand-held micromanipulator driven by 3 piezoelectric actuators and built using rapid prototyping. Traditional pin and ball joints have been commonly replaced by flexure-based methods in the field of micromanipulation. Utilization of flexure-based joints have several advantages like the non-existence of backlash and assembly errors. However, most of the present flexure-based mechanisms are bulky and not suitable for hand-held applications. It is difficult and expensive to make such compact mechanism using traditional machining methods. In additional, traditional machining methods are limited to simple design. To reduce the cost of fabrication and also to allow more complex designs, Objet (a rapid prototyping machine) is proposed to be used to build the mechanism. With regards to hand-held applications, the size of the mechanism is a constraint. Hence, a parallel manipulator design is the preferred choice as compared to a serial mechanism because of its rigidity, compactness, and simplicity in design. For the illustration of an application, the mechanism is designed with an intraocular needle attached to it. Possible applications of this design include enhancement of performance in microsurgery and cell micromanipulation. Experiments are also conducted to evaluate the manipulator's tracking performance of the needle tip at a frequency of 10 Hz.

With the increasing popularity of actuators involving smart materials like piezoelectric, control of such materials becomes important. The existence of the inherent hysteretic behavior hinders the tracking accuracy of the actuators. To make matters worse, the hysteretic behavior changes with rate. One of the suggested ways is to have a feedforward controller to linearize the relationship between the input and output. Thus, the hysteretic behavior of the actuator must be first modeled by sensing the relationship between the input voltage and output displacement. Unfortunately, the hysteretic behavior is dependent on individual actuator and also environmental conditions like temperature. In this fast moving world, time is money and it is very costly to model the hysteresis regularly. In addition, the hysteretic behavior of the actuators also changes with age. Base on the studies done on the phenomena hysteretic behavior with rate, this paper proposes an adaptive rate-dependent feedforward controller with Prandtl-Ishlinskii (PI) hysteresis operators for piezoelectric actuators. This adaptive controller is achieved by adapting the coefficients to manipulate the weights of the play operators. Actual experiments are conducted to demonstrate the effectiveness of the adaptive controller.

An optical sensing system has been developed using a pair of orthogonally placed position sensitive detectors (PSD) to track 3D displacement of a microsurgical instrument tip in real-time. An infrared (IR) diode is used to illuminate the workspace. A ball is attached to the tip of an intraocular shaft to reflect IR rays onto the PSDs. Instrument tip position is then calculated from the centroid positions of reflected IR light on the respective PSDs. The system can be used to assess the accuracy of hand-held microsurgical instruments and operator performance in micromanipulation tasks, such as microsurgeries. In order to eliminate inherent nonlinearity of the PSDs and lenses, calibration is performed using a feedforward neural network. After calibration, percentage RMS error is reduced from about 5.46 % to about 0.16%. The system RMS noise is about 0.7 mum. The sampling rate of the system is 250 Hz.