Robotic-assisted medical systems (RAMS) have gained significant attention for their advantages in alleviating surgeons’ fatigue and improving patients’ outcomes. These systems comprise a range of human-computer interactions, including medical scene monitoring, anatomical target planning, and robot manipulation. However, despite its versatility and effectiveness, RAMS demands expertise in robotics, leading to a high learning cost for the operator. In this work, we introduce a novel framework using mixed reality technologies to ease the use of RAMS. The proposed framework achieves real-time planning and execution of medical instruments by providing 3D anatomical image overlay, human-robot collision detection, and robot programming interface. These features, integrated with an easy-to-use calibration method for head-mounted display, improve the effectiveness of human-robot interactions. To assess the feasibility of the framework, two medical applications are presented in this work: 1) coil placement during transcranial magnetic stimulation and 2) drill and injector device positioning during femoroplasty. Results from these use cases demonstrate its potential to extend to a wider range of medical scenarios.

In Augmented Reality, a robust virtual-to-real calibration that aligns the virtual and real spaces is crucial to ensure accurate overlays. Inspired by popular methods in robotics, we propose establishing the virtual-to-real calibration as a hand-eye/robot-world calibration problem using an $A X=Y B$ formulation. This formulation uses both the self-localization of a head-mounted display and the tracking functionality of an external tracking system. Additional techniques are also provided to address the data synchronization issue between the two measurement systems. To further improve the results, we integrate a post-acquisition outlier filter based on the $A X-Y B$ Frobenius norm. Improvements resulting from the filter were first validated by simulation. For the assessment of the complete pipeline, both subjective evaluation based on human perception and objective evaluation based on computer vision methods were used. The results show that the proposed method is accurate and noise-resistant. With only 100 measurement samples collected, the overlay of a tracked object at the farthest distance (1300 mm) in front of the tracking system has an average rotation error of $1.77 \pm 0.54^{\circ}$ and an average translation error of $4.82 \pm 1.71 \mathrm{~mm}$.