Fetoscopy laser photocoagulation is a widely adopted procedure for treating Twin-to-Twin Transfusion Syndrome (TTTS). The procedure involves photocoagulation pathological anastomoses to restore a physiological blood exchange among twins. The procedure is particularly challenging, from the surgeon's side, due to the limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to amniotic fluid turbidity, and variability in illumination. These challenges may lead to increased surgery time and incomplete ablation of pathological anastomoses, resulting in persistent TTTS. Computer-assisted intervention (CAI) can provide TTTS surgeons with decision support and context awareness by identifying key structures in the scene and expanding the fetoscopic field of view through video mosaicking. Research in this domain has been hampered by the lack of high-quality data to design, develop and test CAI algorithms. Through the Fetoscopic Placental Vessel Segmentation and Registration (FetReg2021) challenge, which was organized as part of the MICCAI2021 Endoscopic Vision (EndoVis) challenge, we released the first large-scale multi-center TTTS dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms with a focus on creating drift-free mosaics from long duration fetoscopy videos. For this challenge, we released a dataset of 2060 images, pixel-annotated for vessels, tool, fetus and background classes, from 18 in-vivo TTTS fetoscopy procedures and 18 short video clips of an average length of 411 frames for developing placental scene segmentation and frame registration for mosaicking techniques. Seven teams participated in this challenge and their model performance was assessed on an unseen test dataset of 658 pixel-annotated images from 6 fetoscopic procedures and 6 short clips. For the segmentation task, overall baseline performed was the top performing (aggregated mIoU of 0.6763) and was the best on the vessel class (mIoU of 0.5817) while team RREB was the best on the tool (mIoU of 0.6335) and fetus (mIoU of 0.5178) classes. For the registration task, overall the baseline performed better than team SANO with an overall mean 5-frame SSIM of 0.9348. Qualitatively, it was observed that team SANO performed better in planar scenarios, while baseline was better in non-planner scenarios. The detailed analysis showed that no single team outperformed on all 6 test fetoscopic videos. The challenge provided an opportunity to create generalized solutions for fetoscopic scene understanding and mosaicking. In this paper, we present the findings of the FetReg2021 challenge, alongside reporting a detailed literature review for CAI in TTTS fetoscopy. Through this challenge, its analysis and the release of multi-center fetoscopic data, we provide a benchmark for future research in this field.

Accurate needle insertion is an important task in many medical procedures. This paper studies the case of an autonomous needle insertion system for central venous access, which is a risky and challenging procedure involving the simultaneous manipulation of an ultrasound probe and of a catheterization needle. The goal of this medical operation is to provide access to a deep central vein, which is a key step in cardiovascular treatments or for the administration of drugs and treatments for cancer or infections. Accordingly, in this work we propose an autonomous dual-arm system for central venous access. The system is composed of two Franka robotic arms that are precisely co-registered and collaborate to achieve accurate needle insertion by combining ultrasound and bioimpedance sensing to ensure robust deep vessels visualization and venipuncture detection. The proposed system performance is evaluated on a phantom trainer through experiments simulating the jugular vein access for cardiac catheterization purposes. Quantitative results show the system is able to autonomously scan the area of interest, localize the vein and perform autonomous needle insertion with high accuracy and placement error below 1.7mm, proving the potential of the technology for real clinical use.

Augmented reality (AR) is considered one of the most promising solutions for safer procedures in several surgical specialities. Fusing patient-specific pre-operative information, typically 3D models extracted from CT scans or MRI, with real-time surgical images allows the surgeon to have detailed information on the anatomical structure of the surgical target intra-operatively. The coupling of AR and Robotics represents the next step towards introducing awareness into the surgical room, thus enhancing the surgeon's perceptual, cognitive and manipulative capabilities. This paper presents a novel integrated system for real-time AR navigation in robotic minimally invasive surgery (RMIS), composed of a robotic endoscopic camera, a robotic teleoperation implementing a software-based Remote Center of Motion (RCM), and an AR navigation software based on an initial manual registration of virtual 3D models with the real anatomy. The integrated system, as well as the individual modules, were evaluated in simulated surgical-like setups for accuracy and repeatability. The proposed system can perform high-precision tasks (position accuracy around$1 mm$and AR error lower than 7%), showing potential for application in different surgical procedures and setting the basis for autonomous robotic surgery operations.

Endoscopic laser tools have been recently proposed in order to overcome the limitations of state-of-the-art laser tools, by integrating fiber-coupled lasers into flexible endoscopic systems. One of the main challenges in designing such endoscopic tools consists in the focusing of the laser, that requires to be frequently adjusted reducing the reliability of the system and increasing surgeons’ mental workload. To avoid these problems, compact auto-focusing tools have been recently developed, taking advantage of MEMS varifocal mirrors (VM) to allow integration with endoscopic tools. In this paper, we integrate such VM-based tool with a distance sensing algorithm based on 3D surface reconstruction to achieve a complete autofocusing system. We evaluate the performance of the proposed integrated system by ablating lines on plaster block targets at variable distance and comparing the obtained ablation depth and width with that of a fixed focus system. Preliminary results show that the proposed system is able to keep the laser in focus resulting in uniform ablation lines for distance ranges from 14mm to 22mm.

BACKGROUND AND OBJECTIVES: During Twin-to-Twin Transfusion Syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the fetuses. In the current practice, this syndrome is surgically treated by closing the abnormal connections using laser ablation. Surgeons commonly use the inter-fetal membrane as a reference. Limited field of view, low fetoscopic image quality and high inter-subject variability make the membrane identification a challenging task. However, currently available tools are not optimal for automatic membrane segmentation in fetoscopic videos, due to membrane texture homogeneity and high illumination variability. METHODS: To tackle these challenges, we present a new deep-learning framework for inter-fetal membrane segmentation on in-vivo fetoscopic videos. The framework enhances existing architectures by (i) encoding a novel (instance-normalized) dense block, invariant to illumination changes, that extracts spatio-temporal features to enforce pixel connectivity in time, and (ii) relying on an adversarial training, which constrains macro appearance. RESULTS: We performed a comprehensive validation using 20 different videos (2000 frames) from 20 different surgeries, achieving a mean Dice Similarity Coefficient of 0.8780±0.1383. CONCLUSIONS: The proposed framework has great potential to positively impact the actual surgical practice for TTTS treatment, allowing the implementation of surgical guidance systems that can enhance context awareness and potentially lower the duration of the surgeries.

Objective: In the last years, Robot-assisted partial nephrectomy (RAPN) is establishing as elected treatment for renal cell carcinoma (RCC). Reduced field of view, field occlusions by surgical tools, and reduced maneuverability may potentially cause accidents, such as unwanted vessel resection with consequent bleeding. Surgical Data Science (SDS) can provide effective context-aware tools for supporting surgeons. However, currently no tools have been exploited for automatic vessels segmentation from nephrectomy laparoscopic videos. Herein, we propose a new approach based on adversarial Fully Convolutional Neural Networks (FCNNs) to kidney vessel segmentation from nephrectomy laparoscopic vision. Methods: The proposed approach enhances existing segmentation framework by (i) encoding 3D kernels for spatio-temporal features extraction to enforce pixel connectivity in time, and (ii) perform training in adversarial fashion, which constrains vessels shape. Results: We performed a preliminary study using 8 different RAPN videos (1871 frames), the first in the field, achieving a median Dice Similarity Coefficient of 71.76%. Conclusions: Results showed that the proposed approach could be a valuable solution with a view to assist surgeon during RAPN.

Laser tattoo removal is a standard non-invasive method for removing color pigments on the skin. Increasing number of tattooed people who want to remove their tattoo has driven the medical laser market to develop new technologies for painless, scar-free and complete tattoo removal. However, manual use of such laser systems creates post-operative complications since they do not guarantee (i) protection on nontattooed skin from laser exposure, nor (ii) precise control of the laser focus during the operations for best performance. This paper introduces deTattoo, a robotic system to improve tattoo removal operations. A robotic arm is equipped with a RGB-D camera and a visible laser, in eye-in-end configuration. A hybrid visual servoing control is proposed to guarantee the correct pose of the laser with respect to the tattooed tissue while compensating body motions. 2D features tracked with a mass-spring-damper deformable mesh model are combined with the 3D reconstruction retrieved from a RGB-D camera in order to build the control law. Several experiments were conducted to evaluate the performance of the system with a fixed or moving tattooed surface, at different inclinations. Results showed that the proposed framework is able to fulfil the laser-based tattoo removal requirements, providing high positioning accuracy (<; 1mm) orientation (<; 0.2°) and body motion compensation.

Shared control and shared autonomy play an important role in assistive technologies, allowing the offloading of the cognitive burden required for control from the user to the intelligent robotic device. In this context, electrophysiological measures of error detection, directly measured from a person's brain activity as Error-related Potentials (ErrPs), can be exploited to provide passive adaptation of an external semi-autonomous system to the human. This concept was implemented in an online robot learning task, where user's evaluation of the robot's actions, in terms of detected ErrP, was exploited to update a reward function in a Reinforcement Learning (RL) framework. Results from both simulated and experimental studies show that the introduction of human evaluation in the robot learning loop allows for: (1) the acceleration of optimal policy learning in a target reaching task, (2) the introduction of a further degree of control in robot learning, namely identification of one among multiple targets, according to the user's will. Overall, presented results support the potential of human-robot co-adaptive and co-operative strategies to develop human-centered assistive technologies.

The use of robots in operating rooms improves safety and decreases patient recovery time and surgeon fatigue, but it introduces new potential hazards that can lead to severe injury or even the loss of human life. Thus, safety has been perceived as a crucial system property since the early days by the industry, the medical community, and the regulatory agents. In this article, we discuss the application of the mathematically rigorous technique known as Formal Verification to analyze the safety properties of a laser incision case study, and we assess its safe and predictable operation. Like all formal methods approaches, our analysis has three distinct components: a method to create a model of the system, a language to specify the properties, and a strategy to prove rigorously that the behavior of the model fulfills the desired properties. The model of the system takes the form of a hybrid automaton consisting of a discrete control part that operates in a continuous environment. The safety constraints are formalized as reachability properties of the hybrid automaton model, while the verification strategy exploits the capabilities of the tool A riadne to address the verification problem and answer the related questions ranging from safety to efficiency and effectiveness.

Scanning lasers increase the quality of the laser microsurgery enabling fast tissue ablation with less thermal damage. However, the possibility to perform scanning laser microsurgery in confined workspaces is restricted by the large size of currently available actuators, which are typically located outside the patient and require direct line-of-sight to the microsurgical area. Here, a magnetic scanner tool is designed to allow endoscopic scanning laser microsurgery. The tool consists of two miniature electromagnetic coil pairs and permanent magnets attached to a flexible optical fiber. The actuation mechanism is based on the interaction between the electromagnetic field and the permanent magnets. Controlled and high-speed laser scanning is achieved by bending of the optical fiber with magnetic torque. Results demonstrate the achievement of a 3×3 mm2scanning range within the laser spot is controlled with 35μm precision. The system is also capable of automatically executing high-speed laser scanning operations over customized trajectories with a root-mean-squared-error (RMSE) in the order of 75μm. Furthermore, it can be teleoperated in real-time using any appropriate user interface device. This new technology enables laser scanning in narrow and difficult to reach workspaces, promising to bring the benefits of scanning laser microsurgery to laparoscopic or even flexible endoscopic procedures. In addition, the same technology can be potentially used for optical fiber based imaging, enabling for example the creation of new family of scanning endoscopic OCT or hyperspectral probes.

Error-related electroencephalographic (EEG) potentials (ErrPs) have been explored to improve the reliability of modern Brain-Computer Interfaces (BCIs), thanks to the information they carry about user awareness of erroneous responses. ErrPs detection on a single-trial basis has been successfully demonstrated, and proved to effectively enhance human-computer interaction and BCI performance. Previous studies tested ErrPs elicited by providing either visual or tactile feedback, showing similar results for all feedback modalities. In the present work, we tested: 1) whether the addition of tactile feedback can improve the detection of ErrP, when used in combination and not alternatively to visual feedback; 2) whether a mismatch between the two different sensory channels can enhance ErrP detection. Results on a study carried out on 12 healthy subjects show that the addition of tactile stimuli significantly affects single-trial ErrP recognition (AUC increment of 4.3%) without significant difference in case of concordant or discordant visual and tactile stimuli.

The development of the narrow-band imaging (NBI) has been increasing the interest of medical specialists in the study of laryngeal microvascular network to establish diagnosis without biopsy and pathological examination. A possible solution to this challenging problem is presented in this paper, which proposes an automatic method based on anisotropic filtering and matched filter to extract the lesion area and segment blood vessels. Lesion classification is then performed based on a statistical analysis of the blood vessels' characteristics, such as thickness, tortuosity, and density. Here, the presented algorithm is applied to 50 NBI endoscopic images of laryngeal diseases and the segmentation and classification accuracies are investigated. The experimental results show the proposed algorithm provides reliable results, reaching an overall classification accuracy rating of 84.3%. This is a highly motivating preliminary result that proves the feasibility of the new method and supports the investment in further research and development to translate this study into clinical practice. Furthermore, to our best knowledge, this is the first time image processing is used to automatically classify laryngeal tumors in endoscopic videos based on tumor vascularization characteristics. Therefore, the introduced system represents an innovation in biomedical and health informatics.

In laser-based laryngeal surgeries, motorized laser scanners offer greater aiming accuracy and efficiency. In this paper, a new motorized laser micromanipulator is presented, which is based on a spherical orienting device. It is a 2 degrees-of-freedom roll/pitch mechanism which actuates the laser beamsplitter mirror for improved aiming control and automated intraoperative planning. The combination of this device with state-of-the-art reflective laser focusing optics overcomes the drawbacks of an earlier prototype, providing increased operating distance and surgical range. This makes the device more suitable to real surgical scenarios in the operating room (OR). Improved system accuracy and usability is successfully demonstrated through comparative user trials against the traditional manual laser micromanipulator. The new device offers greater than 57% improvement in accuracy demonstrating its safety and usability. Preliminary ex-vivo trials were also performed with expert surgeons with the new mechanism. The surgeons evaluated the system positively and provided valuable and favourable feedback pointing to the suitability of the device for the OR and its potential to enhance the capacity of laser-based transoral microsurgeries.

In this paper we present a feed forward controller to regulate the depth of laser incisions in soft tissue. Such a controller is compatible with the requirements of laser microsurgery, where space constraints limit the use of sensing devices. The controller is based on an inverse model that maps the desired incision depth to the required laser exposure time. This model is extracted from experimental data through the use of statistical learning methods. To prove the concept, the controller is implemented in a robot-assisted laser microsurgery system that enables precision control of exposure time and laser motion. The validity and the accuracy of the controller is verified experimentally on ex-vivo muscle tissue (chicken breast), revealing an RMSE of 0.12 mm for incisions ranging up to 1 mm. In addition, we demonstrate how the model can be used to implement the automatic ablation of entire volumes of tissue, through the superposition of controlled laser incisions.

Brain-computer interfaces (BCIs) offer high potential for enhancing training in many tasks, especially those that require maintaining high levels of concentration such as surgery. Training focus and attention can play a critical role in surgery since concentration on the task at hand is fundamental to prevent life-threatening errors. In this paper we propose a new method for concentration training in the context of robot-assisted laser microsurgery associated to a feedback design that makes the interaction more intuitive. This approach couples augmented reality (AR) features to both BCI-based on-line measurement of the user's mental focus and the control of the surgical robot. The methodology is described as a brain-controlled augmented reality (BcAR) training system. AR is used to maintain the surgeon's perceptual contact with the real operating setting, while focus stimulation is provided by modifying features of an AR item based on real-time monitoring of the user's mental state. In this research a low-cost EEG device is used and the BcAR is implemented in the form of an AR scalpel that behaves as a "retractable" knife according to the user's mental focus: low concentration levels retract the knife and prevent cutting. This design provides directional compatibility between the AR feedback animation and the spontaneous motion of user's attention along the AR tool, resulting in an intuitive system with real impact on the training outcome. This is demonstrated through user trials and comparison with training based on simple AR feedback (no EEG). Results demonstrate the potential of the approach, showing a significant improvement in post-training task execution time without any detriment to user experience. Subjective questionnaires also confirmed the critical role of directional compatibility in the AR feedback. Such findings allow the identification of further improvements and novel potential applications of this interaction paradigm.

Ergonomic and human-centered approaches are increasingly important in the design of surgeon-machine interfaces. In the case of microsurgeries, the procedures suffer from susceptibility to variation in surgeon skill and equipment characteristics. This paper presents a novel, computer-assisted surgical interface for laser-based microsurgeries, called the “μRALP Surgical System”. With the system, surgeries can be performed with improved safety and precision using a three-part architecture: (i) a 3D viewer device providing stereoscopic visualization; (ii) a graphics stylus that controls a motorized micromanipulator for laser aiming and activation; and (iii) a configuration interface allowing system setup and modifications in real-time. The system combines the advantages of a computer-assisted platform while respecting the visualization and manipulation requirements of a microsurgical procedure. The features include intraoperative planning for automatic laser incisions and ablations as well as safety regions based on virtual overlays in the surgeon's field-of-view. A comparative evaluation of the proposed system against the traditional system points to the clear superiority of the new interface. The quantitative comparison shows that the proposed interface is safer, more precise, and better controlled. The qualitative comparison demonstrates that the interface is easier to use, easier to learn, and has a minimal training requirement. The technological advances presented here shall lead to enhanced interfaces, increasing the capacity of surgical systems through user-centered design approaches.

This paper proposes an automated robotic system to perform cell microinjections to relieve human operators from this highly difficult and tedious manual procedure. The system, which uses commercial equipment currently found on most biomanipulation laboratories, consists of a multitask software framework combining computer vision and robotic control elements. The vision part features an injection pipette tracker and an automatic cell targeting system that is responsible for defining injection points within the contours of adherent cells in culture. The main challenge is the use of bright-field microscopy only, without the need for chemical markers normally employed to highlight the cells. Here, cells are identified and segmented using a threshold-based image processing technique working on defocused images. Fast and precise microinjection pipette positioning over the automatically defined targets is performed by a two-stage robotic system which achieves an average injection rate of 7.6 cells/min with a pipette positioning precision of 0.23 μm. The consistency of these microinjections and the performance of the visual targeting framework were experimentally evaluated using two cell lines (CHO-K1 and HEK) and over 500 cells. In these trials, the cells were automatically targeted and injected with a fluorescent marker, resulting in a correct cell detection rate of 87% and a successful marker delivery rate of 67.5%. These results demonstrate that the new system is capable of better performances than expert operators, highlighting its benefits and potential for large-scale application.

State-of-the-art laser phonomicrosurgery (LP) used for the treatment of laryngeal abnormalities involves complex otolaryngological surgical techniques. It relies heavily on surgeon dexterity, requiring significant psychomotor skills. Equipment scale and size, laser operative distance, and the anatomically small nature of the vocal folds all combine to compound the surgical challenges. An objective measurement is therefore necessary to understand the impact of equipment design, its usability, surgeon skill, and learning, on performing LP effectively. This paper introduces imaging based feature extraction as a method to establish metrics to assess surgical performance in LP. Experimental analysis demonstrates the utility of these metrics in measuring surgical task execution vis-à-vis the task objectives. The metrics also provide for a combined rating scale giving a robust quantitative classification of the levels of surgical performance.

Robot-assisted surgical procedures, such as Laser Phonomicrosurgery (LP), suffer from susceptibility to variation in surgeon skill and equipment characteristics. Ergonomic and human-centered approaches acquire increased importance in the design of surgeon-machine interfaces. This paper proposes a protocol for comparative evaluation of surgeon-machine interfaces based on two criteria: (i) the subjective evaluation of their usability using questionnaires, and (ii) the objective evaluation of their performance using an imaging-based feature extraction method. Two interfaces in LP, the traditional (“AcuBlade”) interface and the novel (“Virtual Scalpel”) interface, were evaluated to demonstrate the effectiveness of the proposed scheme. A series of experimental trials were conducted using the interfaces in surgery-like tasks in a controlled environment. The subjective evaluation pointed to the superiority of the Virtual Scalpel interface (score: 83.06) in terms of confidence and ease of use, and learnability, over the AcuBlade interface (score: 65.56). The objective evaluation showed the Virtual Scalpel interface having an overall score (55.96) significantly superior to the AcuBlade (51.37). It is thus shown that the multidimensional evaluation approach allowed to clearly distinguish between levels of perceived usability and effective performance of surgeon-machine interfaces from a user-centered perspective.

A medical robotic system for teleoperated laser microsurgery based on a concept we have called “virtual scalpel” is presented in this paper. This system allows surgeries to be safely and precisely performed using a graphics pen directly over a live video from the surgical site. This is shown to eliminate hand-eye coordination problems that affect other microsurgery systems and to make full use of the operator's manual dexterity without requiring extra training. The implementation of this system, which is based on a tablet PC and a new motorized laser micromanipulator offering 1μm aiming accuracy within the traditional line-of-sight 2D operative space, is fully described. This includes details on the system's hardware and software structures and on its calibration process, which is essential for guaranteeing precise matching between a point touched on the live video and the laser aiming point at the surgical site. Together, the new hardware and software structures make both the calibration parameters and the laser aiming accuracy (on any plane orthogonal to the imaging axis) independent of the target distance and of its motions. Automatic laser control based on new intraoperative planning software and safety improvements based on virtual features are also described in this paper, which concludes by presenting results from sets of path following evaluation experiments conducted with 10 different subjects. These demonstrate an error reduction of almost 50% when using the virtual scalpel system versus the traditional laser microsurgery setup, and an 80% error reduction when using the automatic laser control routines, evidencing great improvements in terms of precision and controllability, and suggesting that the technological advances presented herein will lead to a significantly enhanced capacity for treating a variety of internal human pathologies.

This paper presents a mixed-reality system for the training of operators (biologogist/neuroscientists) on fully teleoperated biomanipulations. These tasks are traditionally performed via direct manual control of the biomanipulation equipment while looking through the binoculars of a microscope. However, direct manual control makes the conventional systems susceptible to even very small operator errors, and extensive training is normally required to attain a satisfactory proficiency. To improve this area, a fully teleoperated biomanipulation system has been previously developed, but efficient operation of that system also requires some training. Therefore, the system presented here has been created to help new operators became familiar with the teleoperated system environment, introducing them to the system controls and joysticks functions. Two mixed-reality training games were designed, implemented and tested for this purpose: A ¿move-and-shoot¿ game focused on precise positioning training; and a trajectory following game intended to develop precise motion control skills on new operators. Preliminary experiments performed with 20 totally novice operators demonstrated that this new training system is effective in terms of the initial development of control skills for real teleoperated biomanipulations. Experimental metrics demonstrated an exponential learning curve for these novice operators, who achieved good performance values after only two practice runs on the system. In addition, training here was proven safe and inexpensive since no real cells, biochemical products, or several pipettes were needed for this initial training phase.

This paper shows a novel application of the diffusion tensor for anisotropic image processing. The designed system aims at spotting and localizing injection points on a population of adherent cells lying on a Petri's dish. The overall procedure is described including pre-filtering, ridge enhancement, cell segmentation, shape analysis and injection point detection. The anisotropic contour completion (ACC) employed is equivalent to a dilation with a continuous elliptic structural element that takes into account the local orientation of the contours to be closed, preventing extension towards the normal direction. Experiments carried out on real images from an optical microscope revealed a remarkable reliability with up to 86% of cells in the field of view correctly segmented and targeted for microinjection.

Current challenges in biomanipulations for life-sciences research include extensive operator training, low success rates and low consistency of operations. These problems were tackled here through the use of teleoperation techniques and the development of a unified interface for simultaneous control of all devices used for standard biomanipulations. The developed system was created with high-end commercial biomanipulation equipment similar to those currently in use at many research laboratories, and also included game joysticks for teleoperated control. These were integrated into a single system through the design of modular component abstractions and the implementation of a central control structure. This structure enabled the creation of an open, flexible and user-friendly biomanipulation system for improved operation performance. This paper describes the design and implementation of such system.

Blastocyst microinjections are routinely involved in the process of creating genetically modified mice for biomedical research, but their efficiency is highly dependent on the skills of the operators. As a consequence, much time and resources are required for training microinjection personnel. This situation has been aggravated by the rapid growth of genetic research, which has increased the demand for mutant animals. Therefore, increased productivity and efficiency in this area are highly desired. Here, we pursue these goals through the automation of a previously developed teleoperated blastocyst microinjection system. This included the design of a new system setup to facilitate automation, the definition of rules for automatic microinjections, the implementation of video processing algorithms to extract feedback information from microscope images, and the creation of control algorithms for process automation. Experimentation conducted with this new system and operator assistance during the cells delivery phase demonstrated a 75% microinjection success rate. In addition, implantation of the successfully injected blastocysts resulted in a 53% birth rate and a 20% yield of chimeras. These results proved that the developed system was capable of automatic blastocyst penetration and retraction, demonstrating the success of major steps toward full process automation.

Blastocyst microinjections are routinely involved in the process of creating genetically modified mice for biomedical research, but their efficiency is typically poor and highly dependent on the skills of the operators. Here, the goal of increasing the consistency and efficiency rates of these operations was pursued through robotics and automation. Initially, a teleoperated microinjection system was designed and implemented. Then, this system was evaluated under the control of a microinjection expert. The evaluation consisted of a series of blastocyst microinjections, which are presented and analyzed in this paper. The knowledge gained from these experiments was then used to: 1) create a new system setup more appropriate for automating consecutive blastocyst microinjections; 2) derive rules to guide automatic microinjections; and 3) define future research directions and system upgrades to further increase the system performance. Finally, experiments with the encoded algorithm for automatic blastocyst microinjections attested the validity of the derived rules by demonstrating a 75% success rate. The good quality of the microinjections was demonstrated by the 53% birth rate and the 20% yield of chimeras obtained from the implanted blastocysts.

This paper presents results related to our latest semi-automated blastocyst microinjection system. Here, the improvements made to the microinjection system are described and evaluated. First, after replacing the original piezo-electric kinematic stage by a DC motor-based robot manipulator, experimentation showed that the speed and the precise motion control of pipettes were improved. Second, by introducing an X-Y stage into the system, to manipulate the Petri dish around the microscope's field of view, multiple microinjection speed was improved. Third, by using SSD template matching to track the injection pipette, rather than the cross-correlation template matching algorithm used in the original system, improvements were made to pipette localization. Under human control, this new semi-automated system gives improved microinjection performance metrics compared to previously obtained results. The system is also providing implicit human knowledge of the microinjection process via the human-control interface. It is the encoding of this knowledge that will lead to the first fully automated system. The semi-automated microinjection system is being tested and evaluated in the AMC at UNC-Chapel Hill.

The focus of this paper is the design and development of a semi-automated system for microinjection of embryonic stem cells into blastocysts. Semi-automation is achieved through treating cell microinjection as a computer game. In this first phase, cell manipulation and microinjection is carried out using a joystick and an interactive graphical user interface (GUI). For this system to be developed further, to achieve full automation, this first phase had to succeed. The interactive GUI records and displays: real-time video images of the cells; processed images of cells; and the microinjection strategies adopted by operators during cell manipulation through a real-time video processing algorithm. Experiments showed that this first phase of the research was successful. Future research develop knowledge-based controllers using machine-learning techniques as the research drives towards its goal of fully automated cell microinjection

This paper describes machine vision techniques that provide fast visual feedback for an automatic blastocyst microinjection system. The goals of the vision processing were to locate and track both the blastocysts and manipulation pipettes within the images and throughout the microinjection process. This was successfully accomplished using Hough transforms and cross-correlation template matching. Emphasis here is placed on a detailed description of the techniques applied to the algorithm to speed-up the matching and to enable real-time visual feedback. These techniques allowed the algorithm to be, on average, 857 times faster than the original algorithm. This converts into a 99.88% reduction in the template matching processing time

Transference of controllers evolved in simulation to real vehicles is an important issue in evolutionary robotics (ER). We have previously evolved autonomous navigation controllers for fixed wing UAV applications using multi-objective genetic programming (GP). Controllers were evolved to locate a radar source, navigate the UAV to the source efficiently using on-board sensor measurements, and circle around the emitter. We successfully tested an evolved UAV controller on a wheeled mobile robot. A passive sonar system on the robot was used in place of the radar sensor, and a speaker emitting a tone was used as the target in place of a radar. Using the evolved navigation controller, the mobile robot moved to the speaker and circled around it. The results from this experiment demonstrate that our evolved controllers are capable of transference to real vehicles. Future research will include testing the best evolved controllers by using them to fly real UAVs.

This paper demonstrates the use of small area acoustic array technology as passive sonar for an autonomous mobile robot sound localization and direction control. Real-time target tracking is based solely on received audio signals.