Image-guided surgery demands adaptive, real-time decision support, yet static AI models struggle with structured task planning and providing interactive guidance. Large language models (LLMs)-powered agents offer a promising solution by enabling dynamic task planning and predictive decision support. Despite recent advances, the absence of surgical agent datasets and robust parameter-efficient fine-tuning techniques limits the development of LLM agents capable of complex intraoperative reasoning. In this paper, we introduce Surgical AI Copilot, an LLM agent for image-guided pituitary surgery, capable of conversation, planning, and task execution in response to queries involving tasks such as MRI tumor segmentation, endoscope anatomy segmentation, overlaying preoperative imaging with intraoperative views, instrument tracking, and surgical visual question answering (VQA). To enable structured agent planning, we develop the PitAgent dataset, a surgical context-aware planning dataset covering surgical tasks like workflow analysis, instrument localization, anatomical segmentation, and query-based reasoning. Additionally, we propose DEFT-GaLore, a Deterministic Energy-based Fourier Transform (DEFT) gradient projection technique for efficient low-rank adaptation of recent LLMs (e.g., LLaMA 3.2, Qwen 2.5), enabling their use as surgical agent planners. We extensively validate our agent's performance and the proposed adaptation technique against other state-of-the-art low-rank adaptation methods on agent planning and prompt generation tasks, including a zero-shot surgical VQA benchmark, demonstrating the significant potential for truly efficient and scalable surgical LLM agents in real-time operative settings.

Vision-Language Models (VLMs) in visual question answering (VQA) offer a unique opportunity to enhance intra-operative decision-making, promote intuitive interactions, and significantly advance surgical education. However, the development of VLMs for surgical VQA is challenging due to limited datasets and the risk of overfitting and catastrophic forgetting during full fine-tuning of pretrained weights. While parameter-efficient techniques like Low-Rank Adaptation (LoRA) and Matrix of Rank Adaptation (MoRA) address adaptation challenges, their uniform parameter distribution overlooks the feature hierarchy in deep networks, where earlier layers, that learn general features, require more parameters than later ones. This work introduces PitVQA++ with an Open-ended PitVQA dataset and vector matrix-low-rank adaptation (Vector-MoLoRA), an innovative VLM fine-tuning approach for adapting GPT-2 to pituitary surgery. Open-Ended PitVQA comprises 109,173 frames from 25 procedural videos with 795,270 question-answer sentence pairs, covering key surgical elements such as phase and step recognition, context understanding, tool detection, localization, and interactions recognition. Vector-MoLoRA incorporates the principles of LoRA and MoRA to develop a matrix-low-rank adaptation strategy that employs rank vectors to allocate more parameters to earlier layers, gradually reducing them in the later layers. Our approach, validated on the Open-Ended PitVQA and EndoVis18-VQA datasets, effectively mitigates catastrophic forgetting while significantly enhancing performance over recent baselines. Performance-rejection analysis further highlights Vector-MoLoRA's enhanced reliability and trustworthiness in handling uncertain predictions. Our source code and dataset is available at https://github.com/HRL-Mike/PitVQA-Plus.

Rapid advancements in diffusion models have enabled synthesis of realistic and anonymized imagery in radiography. However, due to their complexity, these models typically require large training volumes, often exceeding 10,000 images. Pre-training on natural images can partly mitigate this issue, but often fails to generate anatomically accurate shapes due to the significant domain gap. This prohibits applications in specialized medical conditions with limited data. We propose AnatoDiff, a diffusion model synthesizing high-quality X-Ray images with accurate anatomical shapes using only 500 to 1,000 training samples. AnatoDiff incorporates a Shape Prototype Module and Anatomical Fidelity loss, allowing for smaller training volumes through targeted supervision. We extensively validate AnatoDiff across three open-source datasets from distinct anatomical regions: Neonatal Abdomen (1,000 images); Adult Chest (500 images); and Humerus (500 images). Results demonstrate significant benefits, with an average improvement of 14.9% in Fréchet Inception Distance, 9.7% in Improved Precision, and 2.3% in Improved Recall compared to state-of-the-art (SOTA) few-shot and data-limited natural image synthesis methods. Unlike other models, AnatoDiff consistently generates anatomically correct images with accurate shapes. Additionally, a ResNet-50 classifier trained on AnatoDiff-generated images shows a 2.1% to 5.3% increase in F1-score, compared to being trained on SOTA diffusion images, across 500 to 10,000 samples. A survey with 10 medical professionals reveals that images generated by AnatoDiff are challenging to distinguish from real ones, with a Matthews correlation coefficient of 0.277 and Fleiss' Kappa of 0.126, highlighting the effectiveness of AnatoDiff in generating high-quality, anatomically accurate radiographs. Our code is available at https://github.com/KawaiYung/AnatoDiff.

Accurate depth and camera pose estimation is essential for achieving high-quality 3D visualisations in robotic-assisted surgery. Despite recent advancements in foundation model adaptation to monocular depth estimation of endoscopic scenes via self-supervised learning (SSL), no prior work has explored their use for pose estimation. These methods rely on low rank-based adaptation approaches, which constrain model updates to a low-rank space. We propose Endo-FASt3r, the first monocular SSL depth and pose estimation framework that uses foundation models for both tasks. We extend the Reloc3r relative pose estimation foundation model by designing Reloc3rX, introducing modifications necessary for convergence in SSL. We also present DoMoRA, a novel adaptation technique that enables higher-rank updates and faster convergence. Experiments on the SCARED dataset show that Endo-FASt3r achieves a substantial $$10\%$$ improvement in pose estimation and a $$2\%$$ improvement in depth estimation over prior work. Similar performance gains on the Hamlyn and StereoMIS datasets reinforce the generalisability of Endo-FASt3r across different datasets. Our code is available at: https://github.com/Mona-ShZeinoddin/Endo_FASt3r.git .

Accurate and reliable selection of the appropriate acetabular cup size is crucial for restoring joint biomechanics in total hip arthroplasty (THA). This paper proposes a novel framework integrating square-root velocity function (SRVF)-based elastic shape registration technique with an embedded deformation (ED) graph approach to reconstruct the 3D articular surface of the acetabulum by fusing multiple views of 2D pre-operative pelvic X-ray images and a hemispherical surface model. The SRVF-based elastic registration establishes 2D-3D correspondences between the parametric hemispherical model and X-ray images, and the ED framework incorporates the SRVF-derived correspondences as constraints to optimize the 3D acetabular surface reconstruction using nonlinear least-squares optimization. Validations using both simulation and real patient datasets are performed to demonstrate the robustness and the potential clinical value of the proposed algorithm. The reconstruction result can assist surgeons in selecting the correct acetabular cup on the first attempt in THA, minimising the need for revision surgery. Code and data are available at: https://github.com/zsustc/3D-ASR .

Visual Question Answering (VQA) has advanced in recent years, inspiring adaptations to radiology for medical diagnosis. Longitudinal VQA, which requires an understanding of changes in images over time, can further support patient monitoring and treatment decision-making. This work introduces RegioMix, a retrieval augmented paradigm for longitudinal VQA, formulating a novel approach that generates retrieval objects through a mix-and-match technique, utilizing different regions from various retrieved images. Furthermore, this process generates a pseudo-difference description based on the retrieved pair, by leveraging available reports from each retrieved region. To align such statements to both the posed question and input image pair, we introduce a Dual Alignment module. Experiments on the MIMIC-Diff-VQA X-ray dataset demonstrate our method’s superiority, outperforming the state-of-the-art by 77.7 in CIDEr score and $$8.3\%$$ in BLEU-4, while relying solely on the training dataset for retrieval, showcasing the effectiveness of our approach. Code is available at https://github.com/KawaiYung/RegioMix .

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network's disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process ( 1280×1024 ) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis.

In robotic surgery the motion of instruments and the laparoscopic camera is constrained by their insertion ports, i. e. a remote centre of motion (RCM). We propose a Simultaneous Localisation and Mapping (SLAM) approach that estimates laparoscopic camera motion under RCM constraints. To achieve this we derive a minimal solver for the absolute camera pose given two 2D-3D point correspondences (RCM-PnP) and also a bundle adjustment optimiser that refines camera poses within an RCM-constrained parameterisation. These two methods are used together with previous work on relative pose estimation under RCM [1] to assemble a SLAM pipeline suitable for robotic surgery. Our simulations show that RCM-PnP outperforms conventional PnP for a wide noise range in the RCM position. Results with video footage from a robotic prostatectomy show that RCM constraints significantly improve camera pose estimation.

Computational stereo is one of the classical problems in computer vision. Numerous algorithms and solutions have been reported in recent years focusing on developing methods for computing similarity, aggregating it to obtain spatial support and finally optimizing an energy function to find the final disparity. In this paper, we focus on the feature extraction component of stereo matching architecture and we show standard CNNs operation can be used to improve the quality of the features used to find point correspondences. Furthermore, we use a simple space aggregation that hugely simplifies the correlation learning problem, allowing us to better evaluate the quality of the features extracted. Our results on benchmark data are compelling and show promising potential even without refining the solution.

We present a novel architecture for arm movement classification based on kinematic properties (joint angle and position), computed from MARG sensors, using a quaternion-based gradient-descent method and a 2-link model of the upper limb. The design based on Coordinate Rotation Digital Computer framework was validated on stroke survivors and healthy subjects performing three elementary arm movements (reach and retrieve, lift arm, rotate arm), involved in `making-a-cup-of-tea' an archetypal daily activity, achieved an overall accuracy of 78% and 85% respectively. The design coded in System Verilog, was synthesized using STMicroelectronics 130 nm technology, occupies 340K NAND2 equivalent area and consumes 292 nW @ 150 Hz, besides being functionally verified up to 25 MHz making it suitable for real-time high speed operations. The orientation, arm position and the joint angle, are computed on-the-fly, with the classification performed at the end of movement duration.

We present a low-complexity framework for classifying elementary arm movements (reach retrieve, lift cup to mouth, and rotate arm) using wrist-worn inertial sensors. We propose that this methodology could be used as a clinical tool to assess rehabilitation progress in neurodegenerative pathologies tracking occurrence of specific movements performed by patients with their paretic arm. Movements performed in a controlled training phase are processed to form unique clusters in a multidimensional feature space. Subsequent movements performed in an uncontrolled testing phase are associated with the proximal cluster using a minimum distance classifier (MDC). The framework involves performing the compute-intensive clustering on the training data set offline (MATLAB), whereas the computation of selected features on the testing data set and the minimum distance (Euclidean) from precomputed cluster centroids are done in hardware with an aim of low-power execution on sensor nodes. The architecture for feature extraction and MDC are realized using coordinate rotation digital computer-based design that classifies a movement in (9n + 31) clock cycles, n being number of data samples. The design synthesized in STMicroelectronics 130-nm technology consumed 5.3 nW at 50 Hz, besides being functionally verified up to 20 MHz, making it applicable for real-time high-speed operations. Our experimental results show that the system can recognize all three arm movements with average accuracies of 86% and 72% for four healthy subjects using accelerometer and gyroscope data, respectively, whereas for stroke survivors, the average accuracies were 67% and 60%. The framework was further demonstrated as a field-programmable gate array-based real-time system, interfacing with a streaming sensor unit.

This paper reports an algorithm for the detection of three elementary upper limb movements, i.e., reach and retrieve, bend the arm at the elbow and rotation of the arm about the long axis. We employ two MARG sensors, attached at the elbow and wrist, from which the kinematic properties (joint angles, position) of the upper arm and forearm are calculated through data fusion using a quaternion-based gradient-descent method and a two-link model of the upper limb. By studying the kinematic patterns of the three movements on a small dataset, we derive discriminative features that are indicative of each movement; these are then used to formulate the proposed detection algorithm. Our novel approach of employing the joint angles and position to discriminate the three fundamental movements was evaluated in a series of experiments with 22 volunteers who participated in the study: 18 healthy subjects and four stroke survivors. In a controlled experiment, each volunteer was instructed to perform each movement a number of times. This was complimented by a seminaturalistic experiment where the volunteers performed the same movements as subtasks of an activity that emulated the preparation of a cup of tea. In the stroke survivors group, the overall detection accuracy for all three movements was 93.75% and 83.00%, for the controlled and seminaturalistic experiment, respectively. The performance was higher in the healthy group where 96.85% of the tasks in the controlled experiment and 89.69% in the seminaturalistic were detected correctly. Finally, the detection ratio remains close ( ±6%) to the average value, for different task durations further attesting to the algorithms robustness.

The Selvester score is an effective means for estimating the extent of myocardial scar in a patient from low-cost ECG recordings. Automation of such a system is deemed to help implementing low-cost high-volume screening mechanisms of scar in the primary care. This paper describes, for the first time to the best of our knowledge, an automated implementation of the updated Selvester scoring system for that purpose, where fractionated QRS morphologies and patterns are identified and classified using a novel stationary wavelet transform (SWT)-based fractionation detection algorithm. This stage informs the two principal steps of the updated Selvester scoring scheme--the confounder classification and the point awarding rules. The complete system is validated on 51 ECG records of patients detected with ischemic heart disease. Validation has been carried out using manually detected confounder classes and computation of the actual score by expert cardiologists as the ground truth. Our results show that as a stand-alone system it is able to classify different confounders with 94.1% accuracy whereas it exhibits 94% accuracy in computing the actual score. When coupled with our previously proposed automated ECG delineation algorithm, that provides the input ECG parameters, the overall system shows 90% accuracy in confounder classification and 92% accuracy in computing the actual score and thereby showing comparable performance to the stand-alone system proposed here, with the added advantage of complete automated analysis without any human intervention.

This paper introduces a low-complexity algorithm for the extraction of the fiducial points from the Electrocardiogram (ECG). The application area we consider is that of remote cardiovascular monitoring, where continuous sensing and processing takes place in low-power, computationally constrained devices, thus the power consumption and complexity of the processing algorithms should remain at a minimum level. Under this context, we choose to employ the Discrete Wavelet Transform (DWT) with the Haar function being the mother wavelet, as our principal analysis method. From the modulus-maxima analysis on the DWT coefficients, an approximation of the ECG fiducial points is extracted. These initial findings are complimented with a refinement stage, based on the time-domain morphological properties of the ECG, which alleviates the decreased temporal resolution of the DWT. The resulting algorithm is a hybrid scheme of time and frequency domain signal processing. Feature extraction results from 27 ECG signals from QTDB, were tested against manual annotations and used to compare our approach against the state-of-the art ECG delineators. In addition, 450 signals from the 15-lead PTBDB are used to evaluate the obtained performance against the CSE tolerance limits. Our findings indicate that all but one CSE limits are satisfied. This level of performance combined with a complexity analysis, where the upper bound of the proposed algorithm, in terms of arithmetic operations, is calculated as 2:423N + 214 additions and 1:093N + 12 multiplications for N 861 or 2:553N + 102 additions and 1:093N +10 multiplications for N > 861 (N being the number of input samples), reveals that the proposed method achieves an ideal trade-off between computational complexity and performance, a key requirement in remote CVD monitoring systems.

In this work, we present the design, development, and evaluation of a real-time target tracking system for wireless embedded nodes, capable of effectively tracking manoeuvring targets. The proposed tracking system is designed to operate solely on range measurements obtained with the use of a two-way time-of-flight method without the need for additional hardware being incorporated in the nodes. To address the challenge of coping with manoeuvring targets, the tracking problem is formulated as a dynamical estimation problem where an adaptive multiple-model approach is employed to represent the motion pattern of manoeuvring targets. The ranging observations are produced in real time and used as inputs to a particle filter algorithm that produces the estimates of the target's kinematic variables. Simulations are provided to assess the effect of several factors on the system's performance. Ultimately, the entire system is implemented on commercially available hardware and tested in an outdoor deployment. A total of 25 experiments demonstrate an average RMS accuracy of 2.6 m for position and 1.9 m/s for velocity, in a 15 m × 15 m area. Such performance, which is additionally confirmed from simulation results, reveals the potential of the proposed range-only system in application scenarios where real-time tracking of mobile targets is needed.

In this paper first we present an overview of the state-of-the-art remote patient monitoring systems in the backdrop of real clinical needs. The paper establishes a clear guideline in terms of clinical expectations from such a system from the viewpoint of practicing clinicians. It provides in-depth analysis of the shortcomings of the existing architectures and paves a way towards developing a practical “patient-centric” architecture that could be useful in the day-to-day clinical practice for providing “continuum of care”. Subsequently, the restrictions imposed by the resource constrained nature of such a system on development of appropriate hardware for supporting information processing on the data acquired by body-worn sensors are analyzed.

In this paper we propose a tracking system for wireless sensor networks, which operates on accumulated ranging data from a number of anchor sensor nodes in order to infer the trace and other kinematic characteristics of a mobile target. The nonlinear nature of the “tracking with range-only measureme