Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from Computed Tomography (CT) or Magnetic Resonance (MR) imaging data are registered to the intraoperative laparoscopic images during this process. Regarding 3D-2D fusion, most algorithms use anatomical landmarks to guide registration, such as the liver's inferior ridge, the falciform ligament, and the occluding contours. These are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and prone to error. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion challenge (P2ILF), held during the Medical Image Computing and Computer Assisted Intervention (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: (1) A 2D and 3D landmark segmentation task and (2) a 3D-2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated in the challenge, whose results were assessed for each task independently. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. The proposed methods were evaluated on 16 test images and 2 preoperative 3D models from 2 patients. In Task 1, the teams were able to segment most of the 2D landmarks, while the 3D landmarks showed to be more challenging to segment. In Task 2, only one team obtained acceptable qualitative and quantitative registration results. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

Obtaining feedback from time-constrained end-users is a major challenge in evaluating novel systems for specialised applications. The performance and feedback of engineers and surgeons was evaluated through an experiment where participants were asked to identify tumour locations within an anatomically realistic silicon liver model across three different conditions of an Augmented Reality (AR) prototype system (Baseline, Split AR and Full AR). Our findings show that engineers and surgeons share some similarities in their performance, feedback and behaviour, particularly when reliance on the AR system is high for both groups. However, engineers typically focus more on accuracy of the image alignment and are more accurate in their responses when supported by AR. Senior surgeons typically perform faster and use AR as supplementary information, while the performance of junior surgeons is more closely aligned to the performance of engineers. We conclude that engineers could be involved in preliminary evaluations of a surgical system or in evaluations of systems which are aimed at training junior surgeons, but that it is essential to involve surgeons in later evaluations, where ecological validity is a more important consideration.

Minimally-invasive surgery (MIS) and robot-assisted minimally invasive (RAMIS) surgery offer well-documented benefits to patients such as reduced post-operative pain and shorter hospital stays.However, the automation of MIS and RAMIS through the use of AI has been slow due to difficulties in data acquisition and curation, partially caused by the ethical considerations of training, testing and deploying AI models in medical environments.We introduce \texttt{SARAMIS}, the first large-scale dataset of anatomically derived 3D rendering assets of the human abdominal anatomy.Using previously existing, open-source CT datasets of the human anatomy, we derive novel 3D meshes, tetrahedral volumes, textures and diffuse maps for over 104 different anatomical targets in the human body, representing the largest, open-source dataset of 3D rendering assets for synthetic simulation of vision tasks in MIS+RAMIS, increasing the availability of openly available 3D meshes in the literature by three orders of magnitude.We supplement our dataset with a series of GPU-enabled rendering environments, which can be used to generate datasets for realistic MIS/RAMIS tasks.Finally, we present an example of the use of \texttt{SARAMIS} assets for an autonomous navigation task in colonoscopy from CT abdomen-pelvis scans for the first time in the literature.\texttt{SARAMIS} is publically made available at https://github.com/NMontanaBrown/saramis/, with assets released under a CC-BY-NC-SA license.

Augmented Reality (AR) is considered to be a promising technology for the guidance of laparoscopic liver surgery. By overlaying pre-operative 3D information of the liver and internal blood vessels on the laparoscopic view, surgeons can better understand the location of critical structures. In an effort to enable AR, several authors have focused on the development of methods to obtain an accurate alignment between the laparoscopic video image and the pre-operative 3D data of the liver, without assessing the benefit that the resulting overlay can provide during surgery. In this paper, we present a study that aims to assess quantitatively and qualitatively the value of an AR overlay in laparoscopic surgery during a simulated surgical task on a phantom setup. We design a study where participants are asked to physically localise pre-operative tumours in a liver phantom using three image guidance conditions - a baseline condition without any image guidance, a condition where the 3D surfaces of the liver are aligned to the video and displayed on a black background, and a condition where video see-through AR is displayed on the laparoscopic video. Using data collected from a cohort of 24 participants which include 12 surgeons, we observe that compared to the baseline, AR decreases the median localisation error of surgeons on non-peripheral targets from 25.8 mm to 9.2 mm. Using subjective feedback, we also identify that AR introduces usability improvements in the surgical task and increases the perceived confidence of the users. Between the two tested displays, the majority of participants preferred to use the AR overlay instead of navigated view of the 3D surfaces on a separate screen. We conclude that AR has the potential to improve performance and decision making in laparoscopic surgery, and that improvements in overlay alignment accuracy and depth perception should be pursued in the future.

A compact implementation of a dynamic charge transfer synapse cell, capable of implementing synaptic depression, is presented. The cell is combined with a simple current mirror summing node to produce biologically plausible postsynaptic potentials (PSPs). A single charge packet is effectively transferred from the synapse to the summing node, whenever a presynaptic pulse is applied to one of its terminals. The charge packet is "weighted" by a voltage applied to the second terminal of the synapse. A voltage applied to the third terminal determines the charge recovery time in the synapse, which can be adjusted over several orders of magnitude. This voltage determines the paired pulse ratio for the synapse. The fall time of the PSP is also adjustable and is set by the gate voltage of a metal-oxide-semiconductor field-effect transistor operating in subthreshold. Results extracted from chips fabricated in a 0.35-μm complementary metal-oxide-semiconductor process, alongside theoretical and simulation results, confirm the ability of the cell to produce PSPs that are characteristic of real synapses. The concept addresses a key requirement for scalable hardware neural networks.

Recent work by the authors proposed compact low power synapses in hardware, based on the charge-coupling principle, that can be configured to yield a static or dynamic response. The focus of this work is to investigate the training dynamics of these synapses. Empirical models of the Post Synaptic Response (PSP), derived from hardware simulations, were developed and subsequently embedded into the MATLAB environment. A network of these synapses was then used to solve a benchmark problem using a well established training algorithm where the performance metric was convergence time, accuracy and weight range; the Spike Response Model (SRM) was used to implement point neurons. Results are presented and compared with standard synaptic responses.

FPGA devices have witnessed popularity in their use for the rapid prototyping of biological Spiking Neural Network (SNNs) applications, as they offer the key requirement of reconfigurability. However, FPGAs do not efficiently realise the biological neuron/synaptic models. Also their routing structures cannot accommodate the high levels of neuron inter-connectivity inherent in complex SNNs. This paper highlights and discusses the current challenges of implementing large scale SNNs on reconfigurable FPGAs. The paper presents a novel Field Programmable Neural Network (FPNN) architecture incorporating low power analogue synapse and a network on chip architecture for SNN routing and configuration. Initial results are presented.

A neuron circuit is presented which can mimic the operation of a spiking neuron cell. A current mirror configuration allows the temporal summing of synaptic inputs, which are subsequently stored as a charge packet on the gate of a CMOS inverter: the inverter is coupled to a second inverter and feedback is used to facilitate re-setting the cell after firing. Charge leakage from the gate of the inverter, via a reverse biased p-n junction, provides a membrane decay time constant comparable with what is observed in biological neurons. Breadboard experiments and simulation results are presented to demonstrate the functionality of the neuron circuit.