In modern automation settings, jobs are processed across machines with interdependencies and are subject to limited equipment availability. When transportation between machines is considered, the problem evolves into a complex multi-agent routing task with operational constraints. Existing Multi-Agent Path Finding (MAPF) algorithms address challenges such as robustness and uncertainty, but practical applications involving scheduling constraints often require considerable manual effort for adaptation and heuristic design. In this paper, we introduce MAPF-SC, an extension of MAPF that incorporates scheduling constraints for continuous task flows. We explore the challenges of applying existing techniques to this problem, emphasizing the engineering effort involved in addressing these constraints. Our evaluation investigates the impact of temporal and topological variations on performance, highlighting key factors that influence real-world automation scenarios.

ECG-imaging has been receiving increasing clinical and commercialization interest as a non-invasive technology for computing cardiac electrical activity and facilitating clinical management of heart rhythm disorders. Despite its potential, the standard ECG-imaging pipeline requires thorax imaging for constructing patient-specific heart-thorax geometry - being outside standard-of-care clinical workflow, this component constitutes a major barrier to the clinical adoption of ECG-imaging. The advent of 3D cameras into ECG-imaging workflow to replace thorax imaging has shown promise, although existing works largely neglect the registration between camera-derived thorax models with an individual's heart geometry. In this work, we address this gap with a novel system that generates patient-specific torso geometry using a 3D camera, registers the torso to heart geometry obtained from pre-existing cardiac scans, and optimally deforms the torso to the skin surface of the patient. We evaluated the presented camera-based ECG-imaging system on five patients undergoing ablation of scar-related ventricular tachycardia, where we evaluate both the accuracy of the surface electrode localization and the ECG-imaging solutions in comparison to those obtained from computed tomography based thorax imaging. We further evaluate the use of the presented camera-based patient-specific heart-thorax model versus a generic heart-thorax model, highlighting the importance of the registration between the camera-based thorax models with cardiac scans.

In this paper, we propose to use constrained inpainting methods to recover usability of corrupted images. Here we focus on the example of face images that are masked for privacy protection but complete images are required for further algorithm development. The task is tackled in a progressive manner: 1) the generated images should look realistic; 2) the generated images must satisfy spatial constraints, if available; 3) when applied to video data, temporal consistency should be retained. We first present a spatial inpainting framework to synthesize face images which can incorporate spatial constraints, provided as positions offa- cial markers and show that it outperforms state-of-the-art methods. Next, we propose Spatial-Temporal Nested GAN (STN-GAN) to adapt image inpainting framework, trained on ~200k images, to video data by incorporating temporal information using residual blocks. Experiments on multiple public datasets show STN-GAN attains spatio-temporal consistency effectively and efficiently. Furthermore, we show that the spatial constraints can be perturbed to obtain different inpainted results from a single source.1

We describe a convex programming framework for pose estimation in 2D/3D point-set registration with unknown point correspondences. We give two mixed-integer nonlinear program (MINLP) formulations of the 2D/3D registration problem when there are multiple 2D images, and propose convex relaxations for both the MINLPs to semidefinite programs that can be solved efficiently by interior point methods. Our approach to the 2D/3D registration problem is non-iterative in nature as we jointly solve for pose and correspondence. Furthermore, these convex programs can readily incorporate feature descriptors of points to enhance registration results. We prove that the convex programs exactly recover the solution to the MINLPs under certain noiseless condition. We apply these formulations to the registration of 3D models of coronary vessels to their 2D projections obtained from multiple intra-operative fluoroscopic images. For this application, we experimentally corroborate the exact recovery property in the absence of noise and further demonstrate robustness of the convex programs in the presence of noise.

Direct volume visualization techniques offer powerful insight into volumetric medical images and are part of the clinical routine for many applications. Up to now, however, their use is mostly limited to tomographic imaging modalities such as CT or MRI. With very few exceptions, such as fetal ultrasound, classic volume rendering using one-dimensional intensity-based transfer functions fails to yield satisfying results in case of ultrasound volumes. This is particularly due its gradient-like nature, a high amount of noise and speckle, and the fact that individual tissue types are rather characterized by a similar texture than by similar intensity values. Therefore, clinicians still prefer to look at 2D slices extracted from the ultrasound volume. In this work, we present an entirely novel approach to the classification and compositing stage of the volume rendering pipeline, specifically designed for use with ultrasonic images. We introduce point predicates as a generic formulation for integrating the evaluation of not only low-level information like local intensity or gradient, but also of high-level information, such as non-local image features or even anatomical models. Thus, we can successfully filter clinically relevant from non-relevant information. In order to effectively reduce the potentially high dimensionality of the predicate configuration space, we propose the predicate histogram as an intuitive user interface. This is augmented by a scribble technique to provide a comfortable metaphor for selecting predicates of interest. Assigning importance factors to the predicates allows for focus-and-context visualization that ensures to always show important (focus) regions of the data while maintaining as much context information as possible. Our method naturally integrates into standard ray casting algorithms and yields superior results in comparison to traditional methods in terms of visualizing a specific target anatomy in ultrasound volumes.

This paper presents a constrained optimization framework that enables the implementation of multi-robot constraints, as virtual fixtures, to assist human operators, in a teleoperated scenario. The collaborative constraints guide the motion of multiple robots such that the spatial and temporal relationships are maintained between them, while following human input motion objectives. We demonstrate this control architecture for the task of manipulating a surgical knot to a target point. The teleoperation system uses four arms from a da Vinci Surgical System® (two master manipulators and two slave manipulators), with custom electronics and software. It extends previous work, which focused on a cooperatively controlled system where the motions of two robots were directly controlled by user-applied forces. Our current system enables us to effectively evaluate the accuracy of the knot positioning task and completion time in a clinically realistic setup for Minimally Invasive Surgery

Transapical aortic valve replacement under MRI guidance in a beating heart is a recent minimally invasive technique that could benefit from a surgical assistant system. We present a robotic surgical assistant system that can precisely and repeatably deliver aortic valve prostheses. The surgical system consists of an imaging system, an Innomotion robotic arm, a 3-DoF valve delivery module and user interfaces. Interactive control allows the physician to remain in the loop and adjust the orientation and position using real-time MR feedback. The 3-DoF valve delivery module is developed to deploy both balloon-expandable and self-expanding stented prostheses. We use a new compact fiducial that can be placed close to the volume of interest and requires a single image plane for image based robot registration. We evaluate the MRI compatible valve delivery module for both types of prostheses. The accuracy for prosthesis delivery is about 0.8 mm and 1.5 mm, for self-expanding and balloon-expandable prostheses, respectively. Preliminary results in ex-vivo experimentation suggest that the robotic system can be translated into animal and clinical models.

MRI compatible robots are emerging as useful tools for image guided interventions. A shared control between a user and the MRI compatible robot makes it more intuitive instrument especially during setup phases of interventions. We present a MRI compatible, hands-on cooperative system using Innomotion robotic arm. An economic MRI compatible user input sensor was developed and its functionality was tested under typical application conditions. Performance improvement in phantom tasks shows promise of adding hands-on interface in MRI compatible robots.

In this work, we have extended the concept of constrained motion control of robots to surgical tasks that require multiple robots. We present virtual fixtures to guide the motion of multiple robots such that spatial and temporal relationship is maintained between them. At the same time, our algorithm keeps the surgeon in the control loop. Moreover, we show that these virtual fixtures allow bimanual tasks to be completed using input for a single robot. That is, the user requires only one hand to cooperatively control multiple robots. This reduces the cognitive load on the surgeon and makes multiple-robot setup for surgery more relevant. We demonstrate this architecture by using an example of manipulating a surgical knot to position it at a target point. Significant improvement is observed in the accuracy when bimanual virtual fixture assistance is provided. Moreover, the accuracy when using a single input from user is similar to the accuracy obtained from bimanual assistance.

This paper describes the development and initial testing of a new and optimized version of a steady-hand manipulator for retinal microsurgery. In the steady-hand paradigm, the surgeon and the robot share control of a tool attached to the robot through a force sensor. The robot controller senses forces exerted by the operator on the tool and uses this information in various control modes to provide smooth, tremor-free, precise positional control and force scaling. The steady-hand manipulator reported here has been specifically designed with the unique constraints of retinal microsurgery in mind. In particular, the system makes use of a compact wrist design that places the bulk of the robot away from the operating field. The resulting system has high efficacy, flexibility and ergonomics while meeting the accuracy and safety requirements of microsurgery. We have now tested this robot on a biological model system and we report a protocol for reliably cannulating ~80 mum OD veins (the size of veins in the human retina) using the system

Robots are increasingly being used in computer integrated surgery (CIS) systems, yet to our knowledge, there is no open source software that is specifically targeted at this application domain. In this paper, we derive unique requirements for medical robot controllers that are based on our experiences in the field. We present an overview of the second-generation open source software package (the cisst package), currently under development, that would include a family of application frameworks with dynamically loaded software components. We then focus on some key design details, which include extensive application of the command pattern as well as a dynamic interface query mechanism, and present a simple example to illustrate the concepts

This paper presents an approach to implement virtual fixtures for surgical robot assistants. Our approach uses a weighted, multi-objective (both linear and nonlinear) constrained optimization framework to formalize a library of virtual fixtures for task primitives. By our formulation, we provide a library of virtual fixtures on task primitives and a way to assemble multiple virtual fixture objects. We implement the constrained optimization problem with both linear and nonlinear constraints, and discuss the trade-offs between them. Moreover, we introduce the notion of soft virtual fixture mechanism for robotic surgical assistance. The soft virtual fixtures enable a surgical tool to have some resistance inside safety regions and no resistance in preferred regions. I. INTRODUCTION This paper presents an approach to implement virtual fix- tures for surgical robot assistants. Most robotic assisted sur- gical procedures are characterized by restricted access to the workspace as well as constrained manipulation of a surgical tool. In such cases, the surgeons' ability can be augmented by techniques such as virtual fixtures (VF). Virtual fixtures (1), which have been discussed previously in the literature for both telerobotic and cooperative robots, are algorithms which provide anisotropic behavior to surgeons' motion commands in addition to filtering out tremor to provide safety and precision. An important case of virtual fixtures is forbidden regions, where the surgical tool is restricted to certain regions in the workspace. Davies et al. (2) set active constraints to constrain the robot to cut the femur and tibia within a permitted region for prosthetic knee surgery. Park et al. (3) developed sensor- mediated virtual fixtures that constrain the robot's motion or create haptic feedback directing the surgeon to move the surgical instruments in a desired direction. The recent work by Bettini et al. on virtual fixtures (4) used admittance control laws to implement guidance virtual fixtures. These works are based either on a specific robot type or on a specific task. Path planning and motion control is a well discussed area with a wide variety of proposed optimality criteria (5), (6), (7), (8). Funda et al. (9) presented an optimal motion control method to control both redundant and deficient robotic systems in constrained working volumes. We extend Funda's work by applying the method to generate complicated virtual fixtures based on user input for surgical assistant robots. Typically, surgical tasks have a certain degree of uncertainty that arises from factors such as registration errors, variations in anatomy and changes during procedures. Consider an example task of placing a surgical tool at a point in space. Depending on the nature of the procedure, one can define a region and tool placement within this region that would lead to the expected outcome. This region could be on the order of a few microns for retinal vein cannulation or hundreds of microns for a biopsy procedure. Furthermore, we can define another region where the surgeon might deliberately want to place the instrument to account for some uncertainties inherent in surgical procedures. In other words, we would like to have some compliance in the virtual fixture, while maintaining a preferred motion. Therefore, we define 3 different regions: A) Preferred region: this region defines expected outcome. B) Safety region: the tool could temporarily be in this region for fulfilling some expected task. C) Forbidden region: The tool never could be here for safety purposes. The relationship of these three regions depends on the surgical task. Figure 1 shows two typical examples.

We describe a new method to generate virtual fixtures for surgical robot control which provide sophisticated ways to assist the surgeon. Different spatial motion constraints for human machine collaborative systems can be implemented by using this method if we know the required geometric constraints and the instantaneous kinematics of the robot. It is independent of manipulator types: teleoperative or cooperative controlled; admittance or impedance type. Our method uses weighted, linearized, multi-objective optimization framework to formalize a library of virtual fixtures for task primitives. We set the cost function based on the user's inputs, and set linearized subject function based on a combination of five basic geometric constraints. In this paper, we also illustrate the implementation for two sample tasks, which are useful for surgical applications, and provide the experimental results for these tasks.

This paper reports preliminary experiments with a cooperative robot system to augment single cell manipulation tasks. The JHU "Steady-Hand" robot configuration for cell manipulation is reported. Stable force control laws for the "Steady Hand" are revisited. Preliminary experiments validating stable insertion of a micropipette in a mouse embryo are detailed along with formulation of vision based tracking and augmentation. These preliminary experiments demonstrate promise of cooperative augmentation in single cell manipulation tasks.