This paper presents a novel robotic system designed to address the challenges of a robotic-assisted Laparo-Endoscopic single-site (LESS) surgery. While numerous studies have explored mechanisms enabling Remote Center of Motion (RCM) for minimally invasive procedures, few have investigated the interactions of multiple (≥2) manipulators within an integrated system. The proposed robotic system features an arc-shaped mainframe for positioning and mounting of five-bar Spherical Parallel Mechanism (SPM). A conventional straight-shaft surgical tool is inserted through a linear guide rail aligned with the pointing axis leading into the RCM. To account for the physical dimensions of the closed-chain linkages, a modified Denavit-Hartenberg parameterization is adopted to assign spherical linkage frames. This design approach ensures self-collision avoidance within the parallel mechanism and enables systematic evaluation of the distal end-effector’s linear motion characteristics. Furthermore, we investigate the basic functions of SPM manipulators through a prototype experiment, providing preliminary insights that could inform future enhancement of surgical techniques for robotic-assisted LESS procedures.

Instrument-tissue interaction detection, a task aimed at understanding surgical scenes from videos, holds immense importance in constructing computer-assisted surgery systems. Existing methods for this task consist of two stages: instance detection and interaction prediction. This sequential and separate model structure limits both effectiveness and efficiency, making it difficult to deploy on surgical robotic platforms. In this paper, we propose an end-to-end Action-Instance Progressive Learning Network (AIPNet) for the task. The model operates in three steps: action detection, instance detection, and action class refinement. Starting with coarse-scale proposals, the model progressively refines them into coarse-grained actions, which then serve as proposals for instance detection. The action prediction results are further refined using instance features through late fusion. These progressive learning processes improve the performance of the end-to-end model. Additionally, we introduce Dynamic Proposal Generators (DPG) to create dynamic adaptive learnable proposals for each video frame. To address the training challenges of this multi-task model, semantic supervised training is introduced to transfer prior language knowledge, and a training label strategy is proposed to generate unrelated instrument-tissue pair labels for enhanced supervision. Experimental results on PhacoQ and CholecQ datasets show that the proposed method achieves superior accuracy and faster processing speed than state-of-the-art models.

Deep reinforcement learning (DRL) has been researched for computer room air conditioning unit control problems in data centers (DCs). However, two main issues limit the deployment of DRL in actual systems. First, a large amount of data is needed. Next, as a mission-critical system, safe control needs to be guaranteed, and temperatures in DCs should be kept within a certain operating range. To mitigate these issues, this article proposes a novel control method RP-SDRL. First, Residual Physics, built using the first law of thermodynamics, is integrated with the DRL algorithm and a Prediction Model. Subsequently, a Correction Model adapted from gradient descent is combined with the Prediction Model as Post-Posed Shielding to enforce safe actions. The RP-SDRL method was validated using simulation. Noise is added to the states of the model to further test its performance under state uncertainty. Experimental results show that the combination of Residual Physics and DRL can significantly improve the initial policy, sample efficiency, and robustness. Residual Physics can also improve the sample efficiency and the accuracy of the prediction model. While DRL alone cannot avoid constraint violations, RP-SDRL can detect unsafe actions and significantly reduce violations. Compared to the baseline controller, about 13% of electricity usage can be saved.

Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra- and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as 〈 instrument class, instrument bounding box, tissue class, tissue bounding box, action class 〉 quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.

For surgeries which require general anesthesia, airway management is imperative. Difficult airway, which inhibits proper intubation, can be fatal. As such, pre-operative airway assessments are conducted by clinicians to determine the ease of intubation as well as to identify patients with difficult airway. To improve the process, artificial intelligence (AI) methods can be employed to predict such difficult airway situations so that suitable preparations can be made beforehand. However, due to the need for explainability of AI models required by healthcare regulations, typical black box models which work best with most data-driven AI methods cannot be used. Therefore, in the current work, a machine learning model has been established to predict the specific medical facial landmarks that are currently used by clinicians. These include the eyes, mentum, thyroid notch, suprasternal notch, forehead, tragus and radix. The model is based on convolutional neural network and a practical facial landmark detector concept. Furthermore, k-fold cross-validation sampling and the Adabelief optimizer have been utilized. The model prediction results display accurate prediction of the features, with the testing loss exhibiting good stability and maintaining well below 0.01 throughout. Attributed to that, the current model can lead to meaningful diagnosis of difficult airway during airway assessments.

Blockchain can be used to store data safely while ensuring data transparency in quality control of advanced manufacturing. A smart contract running on blockchain can prevent data from being tampered with, along with specifying data transmission rules efficiently and securely. This paper proposes a smart contract system for manufacturing quality control that encompasses machine learning to solve dynamic multi-objective combinatorial optimization problem in production. The proposed system was experimented in various production scenarios. Experimental results showed that the system can effectively restore data through smart contracts when the data were artificially tampered with. In addition, the machine learning algorithm can improve the efficiency of the productions and achieve the combined optimization of multiple objectives.

The Capital Expenditure (CapEx) of offshore floating wind turbine generation (WTG) and battery energy storage system (BESS) are declining over the years. This can mitigate gas turbine power generation in offshore oil and gas (O&G) platforms with lower cost impact. This paper proposes integrated systems consisting a WTG and O&G production platforms with BESS onboard to meet the load demand. Cost analysis shows that this integrated system paradigm with BESS can lower overall cost in CapEx and Operational Expenditure (OpEx) compared with typical system. Moreover, transient stability in simulation shows that the system 2 has a significant reduction in term of both voltage and frequency transient deviation, with transient recovery time that could meet the IEC standards 61892-1 for O&G platforms.

With the mindset of constant improvement in efficiency and safety in the workspace and training in Singapore, there is a need to explore varying technologies and their capabilities to fulfil this need. The ability of Virtual Reality (VR) and Augmented Reality (AR) to create an immersive experience of tying the virtual and physical environments coupled with information filtering capabilities brings a possibility of introducing this technology into the training process and workspace. This paper surveys current research trends, findings and limitation of VR and AR in its effect on human performance, specifically in Singapore, and our experience in the National University of Singapore (NUS).

Flexible needle insertion with bevel tips is becoming a preferred method for approaching targets in the human body in the least invasive manner. However, to successfully implement needle insertion, surgeons require prolonged training processes and long-term experience to develop essential handling skills. This paper presents a new path planning approach with Deep Reinforcement Learning (DRL) to implement automatic needle insertion using a surgical robot. In this paper, Deep Q-Network (DQN) algorithm is utilized to learn the control policy for flexible needle steering with needle-tissue interaction. As the human body is composed of a complex environment such as tissues, blood vessels, bones, and muscles, the uncertainty of the needle-tissue interaction should be considered during insertion. To model this complex interaction in path planning, utilizing a neural network to approximate the action-value function is more efficient than using traditional array methods in terms of time and accuracy. In our simulation, the agent (needle) can be controlled with 2 degrees of freedom (bevel direction rotation and insertion) and received negative rewards when it collides with obstacles, goes out of range, or exceeds a predefined number of rotations. During the training, the agent demonstrates the accuracy and efficiency of the learned policy through feedback scores in every episode. In addition, this system incorporates the uncertainty within flexible needle-tissue interaction using a stochastic environment. Compared with other traditional methods for flexible needle path planning, we demonstrated that motion planning of bevel-tip flexible needles in complex human bodies using DRL has better efficiency and accuracy.

This paper proposes a "Gate"-based human-in-the-loop (HiTL) cyber-physical system (CPS) framework that is human-centric. Human, a complex component in CPS, contributes errors to the CPS due to human fatigue, behaviour and false sense of security. The proposed framework maximizes human performance by considering the human behavior and his/her health. A simulated water pouring task is performed to evaluate the performance of the framework. The simulation results showed that the proposed framework allows the simulated operator to consistently complete the task efficiently with higher accuracy as compared to a common framework which generalizes existing HiTL architecture.

Region-based active contour models are effective in segmenting images with poorly defined boundaries but often fail when applied to images containing intensity inhomogeneity. The traditional models utilize pixel intensity and are very sensitive to parameter tuning. On the other hand, machine learning algorithms are highly effective in handling inhomogeneities but often result in noise from misclassified pixels. In addition, there is no objective function. We propose a framework which integrates machine learning with a region-based active contour model. Classification probability scores from machine learning algorithm, which are regularized using a non-linear function, are used to replace the pixel intensity values during energy minimization. In our experiments, we integrate the k-nearest neighbours and the support vector machine with the Chan-Vese method and compare the results obtained with the traditional methods of Chan-Vese and Li et al. The proposed framework gives better accuracy and less sensitive to parameter tuning.

Brain tumor segmentation, an essential but challenging task, has long attracted much attention from the medical imaging community. Recently, successful applications of sparse coding and dictionary learning has emerged in various vision problems including image segmentation. In this paper, a superpixel-based framework for automated brain tumor segmentation is introduced. The kernel trick is adopted in dictionary learning to transform superpixel-level features to a high-dimensional feature space where their nonlinear similarities are considered to generate discriminative sparse codes. A graph is constructed from the approximation errors given by dictionaries modeling different brain tumor structures so that superpixels belonging to particular tumor regions can be efficiently identified. The proposed framework is evaluated on brain magnetic resonance images of high-grade glioma (HGG) patients provided by the multi-modal Brain Tumor Segmentation (BRATS) Benchmark. Results show that the proposed framework achieves competitive performance when compared with the state-of-the-art methods.

In order to develop an effective and user-friendly control method for surgical robotic system, we propose a new framework of cognitive engine to supervise and regulate the surgical processes. The framework aims to make the surgical processes understandable by both human operators and robots. A prototype cognitive engine was implemented using ontology and SPARQL query language on JAVA and tested in ex-vivo phantom experiments with a robotic RF needle insertion system. The prototype cognitive engine has successfully guided the robot in execution of surgical procedures.

Edge-based active contour models are effective in segmenting images with intensity inhomogeneity but often fail when applied to images containing poorly defined boundaries, such as in medical images. Traditional edge-stop functions (ESFs) utilize only gradient information, which fails to stop contour evolution at such boundaries because of the small gradient magnitudes. To address this problem, we propose a framework to construct a group of ESFs for edge-based active contour models to segment objects with poorly defined boundaries. In our framework, which incorporates gradient information as well as probability scores from a standard classifier, the ESF can be constructed from any classification algorithm and applied to any edge-based model using a level set method. Experiments on medical images using the distance regularized level set for edge-based active contour models as well as the k-nearest neighbours and the support vector machine confirm the effectiveness of the proposed approach.

An interactive surgical simulation system needs to meet three main requirements, speed, accuracy, and stability. In this paper, we present a stable and accurate method for animating mass-spring systems in real time. An integration scheme derived from explicit integration is used to obtain interactive realistic animation for a multiobject environment. We explore a predictor-corrector approach by correcting the estimation of the explicit integration in a poststep process. We introduce novel constraints on positions into the mass-spring model (MSM) to model the nonlinearity and preserve volume for the realistic simulation of the incompressibility. We verify the proposed MSM by comparing its deformations with the reference deformations of the nonlinear finite-element method. Moreover, experiments on porcine organs are designed for the evaluation of the multiobject deformation. Using a pair of freshly harvested porcine liver and gallbladder, the real organ deformations are acquired by computed tomography and used as the reference ground truth. Compared to the porcine model, our model achieves a 1.502 mm mean absolute error measured at landmark locations for cases with small deformation (the largest deformation is 49.109 mm) and a 3.639 mm mean absolute error for cases with large deformation (the largest deformation is 83.137 mm). The changes of volume for the two deformations are limited to 0.030% and 0.057%, respectively. Finally, an implementation in a virtual reality environment for laparoscopic cholecystectomy demonstrates that our model is capable to simulate large deformation and preserve volume in real-time calculations.

High-fidelity patient specific model for blood flow simulation is important in medical applications. In this paper, a software component approach is proposed to perform physics based simulation of blood flow using GPU for general computing. The method comprises three independently developed software components, namely image processing component, object model component and GPU acceleration component. The fluid dynamics of blood flow is simulated using a mesh free modeler based on an improved Smoothed Particle Hydrodynamics method, and the interaction with arterial wall is simulated using a Finite Element modeler based on lumped element approach. Real-time simulation of blood flow has been demonstrated in an application on patients with abdominal aorta aneurysm. The internal fluidic structure and pressure distribution analysis have also been achieved using the proposed method.

Abstract In volume visualization, transfer functions are used to classify the volumetric data and assign optical properties to the voxels. In general, transfer functions are generated in a transfer function space, which is the feature space constructed by data values and properties derived from the data. If volumetric objects have the same or overlapping data values, it would be difficult to separate them in the transfer function space. In this paper, we present a rule‐enhanced transfer function design method that allows important structures of the volume to be more effectively separated and highlighted. We define a set of rules based on the local frequency distribution of volume attributes. A rule‐selection method based on a genetic algorithm is proposed to learn the set of rules that can distinguish the user‐specified target tissue from other tissues. In the rendering stage, voxels satisfying these rules are rendered with higher opacities in order to highlight the target tissue. The proposed method was tested on various volumetric datasets to enhance the visualization of important structures that are difficult to be visualized by traditional transfer function design methods. The results demonstrate the effectiveness of the proposed method.

Biometric recognition can be used to improve gesture-based interfaces by automatically identifying operators. Traditional palm biometric recognition techniques depend on palm appearance features, but these features are not available in an operating theater where gloves are worn. We propose a depth-based solution for palm biometric recognition. Based on the depth image, our system automatically segments the user's palm and extracts finger dimensions. The finger dimensions are further scaled according to the sensed depth to obtain the true finger dimensions, which are then used as features to characterize the palm. Finally, a modified$k$-nearest neighbors algorithm that assigns class labels based on the centroid displacement of each class in the neighboring points is applied to recognize the palm based on the geometric features. An accuracy of 96.24% was achieved for the biometric recognition of 4057 gloved palm samples captured at different angles and depths from 27 users. This accuracy is comparable with those of other state-of-the-art classification algorithms and demonstrates that biometric recognition may be viable for settings with gloved hands such as surgery.

Medical implants are devices manufactured to replace damaged biological organs or structures. Once implanted, these devices communicate with other devices outside the body wirelessly. In this paper, we present our work on a particular type of WMI (wireless medical implants): artificial pancreas. We investigate challenges and issues in the context of artificial pancreas designs. In particular, we focus on how to achieve a low power design for WMI and propose a Virtual Patient system to evaluate control algorithms and communication protocols without an actual implantation of the device. Simulation and experiment results presented in this paper serves as design guidelines for wireless medical implant development.

Although dual-energy X-ray absorptiometry (DXA) offers an effective measurement of bone mineral density, it only provides a 2-D projected measurement of the bone mineral density. Clinical computed tomography (CT) imaging will have to be employed for measurement of 3-D bone mineral density. The typical dual energy process requires precise measurement of the beam spectral intensity at the 80 kVp and 120 kVp settings. However, this is not used clinically because of the extra radiation dosage and sophisticated hardware setup. We propose an accurate and fast approach to measure bone material properties with single energy scans. Beam hardening artifacts are eliminated by incorporating the polychromatic characteristics of the X-ray beam into the reconstruction process. Bone mineral measurement from single energy CT correction is compared with that of dual energy correction and the commonly used DXA. Experimental results show that single energy correction is compatible with dual energy CT correction in eliminating beam hardening artifacts and producing an accurate measurement of bone mineral density. We can then estimate Young's modulus, yield stress, yield strain and ultimate tensile stress of the bone, which are important data for patient specific therapy planning.

This paper proposes a new method for steering a flexible needle in soft tissue according to an analogy of the swimming wave motion of fish locomotion. Assuming small deflection, the flexible needle is modeled as a linear beam. The governing equation for cantilever deflection is used to minimize the strain energy of the needle. Our simulation demonstrated that the repetitive motion has presented a more flexible agility to avoid obstacles, and possess better ability to find the appropriate path according to the external forces applied on the tissue during navigation. The motion of the needle can be tracked by comparing the desired deflection with needle position from x-ray images. Point-to-point algorithm is used to achieve real time control of the flexible needle.

Blood loss is a concern in the liver resections. In line with the wide implementation of ablation process, bleeding during liver resections can be minimized. However, the division process has to be accurately monitored to provide efficient liver resection in terms of time and safety. The modeling of liver tissue and division tool interaction along with the deformation of the liver tissue is studied in this research. As liver tissue is made of complex viscoelastic material, we determined that it could be modeled using standard viscoelastic model based on biomechanics experiment on fresh porcine livers. This result is further extended to incorporate a modeled division tool. Simulation of the modeled interaction is discussed. A state control feedback system which can help monitor the combined ablation and division process is also proposed.

A straight line needle trajectory is typically used in medical needle insertion for percutaneous intervention. Flexible needle steering may be able to avoid obstacles, and reach regions that are currently inaccessible using straight line trajectory. The success of flexible needle insertion in clinical application, for example, biopsy to obtain a tissue sample from human liver organ is dependent on how accurate the motion path can be planned and simulated. We developed a motion planning algorithm for flexible needle insertion with an integrative model of human liver organ, and finite element models of needle and needle-tissue interaction. The aim of image based integrative modeling is to have a unified organ model of patient comprising finite element and implicit surface models of blood vessels, normal and pathological tissues. The trajectory path can be determined in an interactive manner. The generated trajectory for flexible needles avoids obstacles (vessels) to reach target (tumor) inaccessible to rigid needles. The algorithm can also be used to simulate needle bending due to dynamic contact with tumor

Medical simulators are increasingly being used for surgical training. For interactive surgical simulation involving haptic rendering, the force at the needle tip has to be computed very fast. We are developing biomechanical models for bone needle insertion. The cortical bone can be regarded as a dense form of cancellous bone that can be modeled using a linear elastic material. The porosity of the bone determines the resistance felt as the user inserts the needle into the bone. The bar element method that represents each trabecular bone as a FE beam is most computationally efficient. With 1000 FE elements, the computed force feedback were close to the insertion force measured during experiments. However, the extended bar element method may be the more appropriate choice for taking into consideration the trabecular distribution and hence, inhomogeneous of bone. The simulation studies on bone-needle interaction also showed that a diamond bevel needle may penetrate the bone with less force