Training large language models (LLMs) with increasingly long and varying sequence lengths introduces severe load imbalance challenges in large-scale data-parallel training. Recent frameworks attempt to mitigate these issues through data reorganization or hybrid parallel strategies. However, they often overlook how computational and communication costs scale with sequence length, resulting in suboptimal performance. We identify three critical challenges: (1) varying computation-to-communication ratios across sequences of different lengths in distributed attention, (2) mismatch between static NIC-GPU affinity and dynamic parallel workloads, and (3) distinct optimal partitioning strategies required for quadratic attention versus linear components.

Cloud platforms and open-source model repositories offer advanced AI services. These services are becoming essential components for building AI -enabled software. The opacity and lack of explanation have become new challenges to address within the software service lifecycle. Recent studies demonstrate that augmenting explainability requires the integration of diverse al-gorithms, models, and data pipelines. To address this, we present XAIpipeline, a service that interfaces with cloud AI services and open-source models to provide detailed model explanations. XAIpipeline automates structured approaches to apply multiple explainable AI (XAI) techniques, enhancing the explainability and quality assurance of AI-based software service systems. This work implements the XAIpipeline's design and technology stack, demonstrating its integration of XAI algorithms, cloud AI services, and open-source models with a DevOps-style toolkit. The service executes parallel pipelines and produces end-to-end explanation visualizations from data samples. XAIpipeline offers APIs, CLIs, and web portals, enabling users to configure tasks to their specific requirements. We showcase three XAI service scenarios where AI models are applied to support decision-making: (1) Tabular classification model, (2) Image vision model, and (3) Video action recognition model. The source code and supplementary materials are available on GitHub (https://ithub.com/ZeruiWIXAlpipeline).

Building upon our previous work on trustworthy explanation of cloud AI services published in IEEE Transactions on Cloud Computing (doi: 10.1109/TCC.2024.3398609), this extension proposes a spatio-temporal explanation framework to enhance the adversarial awareness of cloud vision services. Along with the increasing adoption of vision models for learning tasks on video streams, adversarial attack on video becomes a severe source of degrading the cloud vision service's efficacy. The explanation of the spatial features of local image frames and temporal properties along the timeline of frames enables transparency and awareness of the impact source under adversarial attacks. This extension addresses two critical challenges, namely (1) the development of unified spatiotemporal explanations that can handle both image and video models; and (2) the assessment of the vulnerability of cloud vision models to adversarial attacks and their impact on explanation trustworthiness. The proposed extension research will integrate adversarial robustness assessment with spatio-temporal feature analysis, enabling unified explanation pipelines for the multi-tasks of cloud vision services.

The opacity of AI models necessitates both validation and evaluation before their integration into services. To investigate these models, explainable AI (XAI) employs methods that elucidate the relationship between input features and output predictions. The operations of XAI extend beyond the execution of a single algorithm, involving a series of activities that include preprocessing data, adjusting XAI to align with model parameters, invoking the model to generate predictions, and summarizing the XAI results. Adversarial attacks are well-known threats that aim to mislead AI models. The assessment complexity, especially for XAI, increases when open-source AI models are subject to adversarial attacks due to various combinations. To automate the numerous entities and tasks involved in XAI - based assessments, we propose a cloud-based service framework that encapsulates computing components as microser-vices and organizes assessment tasks into pipelines. The current XAI tools are not inherently service-oriented. This framework also integrates open XAI tool libraries as part of the pipeline composition. We demonstrate the application of XAI services for assessing five quality attributes of AI models: (1) computational cost, (2) performance, (3) robustness, (4) explanation deviation, and (5) explanation resilience across computer vision and tabular cases. The service framework generates aggregated analysis that showcases the quality attributes for more than a hundred combination scenarios.

This paper presents the design of an open-API-based explainable AI (XAI) service to provide feature contribution explanations for cloud AI services. Cloud AI services are widely used to develop domain-specific applications with precise learning metrics. However, the underlying cloud AI services remain opaque on how the model produces the prediction. We argue that XAI operations are accessible as open APIs to enable the consolidation of the XAI operations into the cloud AI services assessment. We propose a design using a microservice architecture that offers feature contribution explanations for cloud AI services without unfolding the network structure of the cloud models. We can also utilize this architecture to evaluate the model performance and XAI consistency metrics showing cloud AI services' trustworthiness. We collect provenance data from operational pipelines to enable reproducibility within the XAI service. Furthermore, we present the discovery scenarios for the experimental tests regarding model performance and XAI consistency metrics for the leading cloud vision AI services. The results confirm that the architecture, based on open APIs, is cloud-agnostic. Additionally, data augmentations result in measurable improvements in XAI consistency metrics for cloud AI services.

The ability to recover tissue deformation from surgical video is fundamental for many downstream applications in robotic surgery. Despite noticeable advancements, this task remains under-explored due to the complex dynamics of soft tissues manipulated by surgical instruments. Achieving dense and accurate tissue tracking is further complicated by ambiguous pixel correspondence in regions with homogeneous texture. In this paper, we introduce a novel self-supervised framework to recover tissue deformations from stereo surgical videos. Our approach integrates semantics, cross-frame motion flow, and long-range temporal dependencies to accurately represent tissue dynamics for deformation recovery. Moreover, we incorporate diffeomorphic mapping to regularize the warping field to be physically more realistic. To comprehensively evaluate our method, we collected stereo surgical video clips containing three types of tissue manipulation (i.e., pushing, dissection and retraction) from two surgical procedures (i.e., hemicolectomy and mesorectal excision). Our method demonstrates promising results in capturing tissue 3D deformation, and generalizes well across different actions and procedures. It also outperforms current state-of-the-art approaches based on non-rigid registration and optical flow estimation. To the best of our knowledge, this is the first work on self-supervised learning for dense tissue deformation modeling from stereo surgical videos. The paper's code is available at: https://github.com/ med-air/RecoverTissueDeform.

Accurate segmentation of surgical instrument tip is an important task for enabling downstream applications in robotic surgery, such as surgical skill assessment, tool-tissue interaction and deformation modeling, as well as surgical autonomy. However, this task is very challenging due to the small sizes of surgical instrument tips, and significant variance of surgical scenes across different procedures. Although much effort has been made on visual-based methods, existing segmentation models still suffer from low robustness thus not usable in practice. Fortunately, kinematics data from the robotic system can provide reliable prior for instrument location, which is consistent regardless of different surgery types. To make use of such multi-modal information, we propose a novel visual-kinematics graph learning framework to accurately segment the instrument tip given various surgical procedures. Specifically, a graph learning framework is proposed to encode relational features of instrument parts from both image and kinematics. Next, a cross-modal contrastive loss is designed to incorporate robust geometric prior from kinematics to image for tip segmentation. We have conducted experiments on a private paired visual-kinematics dataset including multiple procedures, i.e., prostatectomy, total mesorectal excision, fundoplication and distal gastrectomy on cadaver, and distal gastrectomy on porcine. The leave-one-procedure-out cross validation demon-strated that our proposed multi-modal segmentation method significantly outperformed current image-based state-of-the-art approaches, exceeding averagely 11.2% on Dice.

Explainable Artificial Intelligence (XAI) research focuses on effective explanation techniques to understand and build AI models with trust, reliability, safety, and fairness. Feature importance explanation summarizes feature contributions for end-users to make model decisions. However, XAI methods may produce varied summaries that lead to further analysis to evaluate the consistency across multiple XAI methods on the same model and data set. This paper defines metrics to measure the consistency of feature contribution explanation summaries under feature importance order and saliency map. Driven by these consistency metrics, we develop an XAI process oriented on the XAI criterion of feature importance, which performs a systematical selection of XAI techniques and evaluation of explanation consistency. We demonstrate the process development involving twelve XAI methods on three topics, including a search ranking system, code vulnerability detection and image classification. Our contribution is a practical and systematic process with defined consistency metrics to produce rigorous feature contribution explanations.

Three-dimensional reconstruction of dynamic objects is important for robotic applications, for example, the robotic recognition and manipulation. In this paper, we present a novel 3D surface reconstruction method for moving objects. The proposed method combines the spatial-multiplexing and time-multiplexing structured-light techniques that have advantages of less image acquisition time and accurate 3D reconstruction, respectively. A set of spatial-temporal encoded patterns are designed, where a spatial-encoded texture map is embedded into the temporal-encoded three-step phase-shifting fringes. The specifically designed spatial-coded texture assigns high-uniqueness codeword to any window on the image which helps to eliminate the phase ambiguity. In addition, the texture is robust to noise and image blur. Combining this texture with high-frequency phase-shifting fringes, high reconstruction accuracy would be ensured. This method only requires 3 patterns to uniquely encode a surface, which facilitates the fast image acquisition for each reconstruction step. A filtering stereo matching algorithm is proposed for the spatial-temporal multiplexing method to improve the matching reliability. Moreover, the reconstruction precision is further enhanced by a correspondence refinement algorithm. Experiments validate the performance of the proposed method including the high accuracy, the robustness to noise and the ability to reconstruct moving objects.

Precise 3-D surface reconstruction plays an important role in automated manipulation, industrial inspection, robotics, and so on. In this article, we present a novel 3-D surface reconstruction framework for stereo vision systems assisted with structured light projection. In the framework, a multistep matching scheme is proposed to establish a reliable correspondence between image pairs with high computation efficiency and accuracy. The successive matching steps can find the most precise correspondence through a step-by-step filtering procedure. To further enhance the precision, a correspondence refinement algorithm is presented. Phase maps with different frequencies are utilized as the code words for the multistep matching due to their high encoding accuracy and robustness to noise. This method does not require phase unwrapping or projector calibration, which improves the reconstruction precision and simplifies the operation. Selection strategies for the number of matching steps, the pattern frequencies, and the matching threshold are proposed. Furthermore, various 3-D reconstruction experiments are conducted using the proposed framework. Comparative experiments verify the advantages of the proposed framework compared with existing 3-D reconstruction methods regarding the accuracy and precision. The adaptability to scenarios with different motion speeds is demonstrated. Robustness and limitations of the framework are also revealed by conducting experiments in challenging scenarios. Note to Practitioners-This article is motivated by the precise 3-D surface reconstruction problem in automated robotic systems. In different scenarios, such as the reconstruction of the static objects or moving objects, the errors induced by sensor noise and motion should be taken into consideration. To enhance the measurement precision under these occasions, selection of pattern number and fringe frequencies has been a problem. To overcome these problems, this article proposes a novel framework for active stereo 3-D surface reconstruction. The framework utilizes multifrequency phase-shifting fringes to encode the reconstructed target. Then, a multistep matching method filters the candidates step by step to obtain the most precise corresponding pixel and avoid noise error accumulation. A refinement method is introduced to further improve the precision. Selection strategies of the number of matching steps, the fringe frequencies, and matching thresholds enable the 3-D reconstruction framework to be utilized on different occasions. In applications, limitations of the proposed method should be noted.

Inherent compliance plays an enabling role in soft robots, which rely on it to mechanically conform to the environment. However, it also limits the payload of the robots. Various variable stiffness approaches have been adopted to limit compliance and provide structural stability, but most of them can only achieve stiffening of discrete fixed regions which means compliance cannot be precisely adjusted for different needs. This paper offers an approach to enhance the payload with finely adjusted compliance for different needs. We have developed a manipulator that incorporates a novel variable stiffness mechanism and a sliding layer mechanism. The variable stiffness mechanism can achieve a 6.4 stiffness changing ratio with a miniaturized size (10 mm diameter for the testing prototype) through interlocking jamming layers with a honeycomb core. The sliding layer mechanism can actively shift the position of the stiffening regions through sliding of jamming layers. A model to predict the robot shape is derived with verifications via an experiment. The stiffening capacity of the variable stiffness mechanism is also empirically evaluated. A case study of a potential application in laparoscopic surgeries is showcased. The payload of the manipulator is investigated, and the prototype shows up to 57.8 percentage decrease of the vertical deflection due to an external load after reconfigurations.

In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the FA include navigating new instruments into the surgeon's field-of-view and passing in or retracting materials from the body using hand-held tools. We previously developed ARssist, an augmented reality application based on an optical see-through head-mounted display, to aid the FA. In this paper, we refine the system and first perform a pilot study with three experienced surgeons for two specific tasks: instrument insertion and tool manipulation. The results suggest that ARssist would be especially useful for less experienced assistants and for difficult hand-eye configurations. We then perform a multi-user study with inexperienced subjects. The results show that ARssist can reduce navigation time by 34.57%, enhance insertion path consistency by 41.74%, reduce root-mean-square path deviation by 40.04%, and reduce tool manipulation time by 72.25%. Thus, ARssist has the potential to improve efficiency, safety and hand-eye coordination, especially for novice assistants.

Three-dimensional vision plays an important role in robotics. In this paper, we present a 3D surface reconstruction scheme based on combination of stereo matching and pattern projection. A two-step matching scheme is proposed to establish reliable correspondence between stereo images with high computation efficiency and accuracy. The first step (coarse matching) can quickly find the correlation candidates, and the second step (precise matching) is responsible for determining the most precise correspondence within the candidates. Two phase maps serve as codewords and are utilized in the two-step stereo matching, respectively. The phase maps are derived from phase-shifting patterns to provide robustness to the background noises. Only five patterns are required, which reduces the image acquisition time. Moreover, the precision is further enhanced by applying a correspondence refinement algorithm. The precision and accuracy are validated by experiments on standard objects. Furthermore, various experiments are conducted to verify the capability of the proposed method, which includes the complex object reconstruction, the high-resolution reconstruction, and the occlusion avoidance. The real-time experimental results are also provided.

In this paper, we propose a novel 3D laparoscopic imaging system based on stereo-photogrammetry which is assisted by projecting patterns on the tissue surface. The proposed laparoscopic imaging system has three optic channels, two of which are responsible for stereo vision feedback and the other one is used for coded structured patterns projection. The projected patterns provide the robustness to homogeneous tissue surface since they add more features that can be relied on in the stereo matching. Image fiber bundles (100k pixels) and Gradient-index (GRIN) lenses are utilized to facilitate the remote image acquisition and miniaturization of the laparoscopic probe. Moreover, we adopt a digital micromirror device (DMD) and high-speed cameras to achieve fast pattern switching (up to 4 kHz) and high frame rate image acquisition. The system configuration allows for implementation of the time multiplexing pattern codification strategy in the 3D laparoscopic imaging system to enhance the reliability and resolution of the 3D surface reconstruction. A prototype is established, and various experiments are conducted. Comparative experimental results prove the advantages of our system design. The static and dynamic 3D reconstruction results validate the performance of the proposed 3D laparoscopic imaging system quantitatively and qualitatively.

Emerging applications of robotic manipulation of deformable objects have opened up new challenges in robot control. While several control techniques have been developed to manipulate deformable objects, the performance of existing methods is commonly limited by two issues: 1) implicit assumption that the physical contact between the end-effector and the object is always maintained, and 2) requirements of exact parameters of deformation model, which are difficult to obtain. This paper presents a new control scheme for robotic manipulation of deformable objects, which allows the robot to automatically contact then actively deform the deformable object by assessing the status of deformation in real time. Instead of designing multiple controllers and switching among them, the proposed method smoothly and stably integrates two control phases (i.e. region reaching and active deforming) into a single controller. The stability of the closed-loop system is rigorously proved with the consideration of the uncertain deformation model and uncalibrated cameras. Hence, the proposed control scheme enhances the autonomous capability of active deformable object manipulation. Experimental studies are conducted with different initial conditions to demonstrate the performance of the proposed controller.

In this paper, we address the 3D position control of RCM-constrained instruments with monocular cameras. To compute the instrument's position from a single 2D image, we develop an innovative gradient descent algorithm which rotates and translates a line segment (over the plane spanned by the imaged instrument and the optical centre) until it best aligns with the manipulated tool. In contrast with other approaches in the literature, our algorithm only requires to simultaneously observe two feature points; the proposed iterative algorithm is not based on the exact solution, therefore it can still work with noisy image measurements. We derive a kinematic controller that uses the proposed position estimator to guide the 3D motion of a robotic instrument with a monocular camera. We evaluate the performance of our approach with numerical simulations and experiments.

In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth information. The modelling solely relies on the simultaneous observation of the needle's apparent tip and junction. No external markers are needed for extra constraints. An adaptive controller combining gradient descent and vector-flow method is introduced to iteratively guide the needle's initial guessing pose to its real pose by minimizing image-based position errors. Experiments have been conducted using both numerical simulations and simulated laparoscopic scenarios to evaluate the performance of the algorithm.

JavaScript frameworks, such as jQuery, are widely used for developing web applications. To facilitate using these JavaScript frameworks to implement a feature (e.g., functionality), a large number of programmers often search for code snippets that implement the same or similar feature. However, existing code search approaches tend to be ineffective, without taking into account the fact that JavaScript code snippets often implement a feature based on various relationships (e.g., sequencing, condition, and callback relationships) among the invoked framework API methods. To address this issue, we present a novel Relationship-Aware Code Search (RACS) approach for finding code snippets that use JavaScript frameworks to implement a specific feature. In advance, RACS collects a large number of code snippets that use some JavaScript frameworks, mines API usage patterns from the collected code snippets, and represents the mined patterns with method call relationship (MCR) graphs, which capture framework API methods’ signatures and their relationships. Given a natural language (NL) search query issued by a programmer, RACS conducts NL processing to automatically extract an action relationship (AR) graph, which consists of actions and their relationships inferred from the query. In this way, RACS reduces code search to the problem of graph search: finding similar MCR graphs for a given AR graph. We conduct evaluations against representative real-world jQuery questions posted on Stack Overflow, based on 308,294 code snippets collected from over 81,540 files on the Internet. The evaluation results show the effectiveness of RACS: the top 1 snippet produced by RACS matches the target code snippet for 46% questions, compared to only 4% achieved by a relationship-oblivious approach.

In this paper, we present a new feedback method to automatically servo-control the 3-D shape of soft objects with robotic manipulators. The soft object manipulation problem has recently received a great deal of attention from robotics researchers because of its potential applications in, e.g., food industry, home robots, medical robotics, and manufacturing. A major complication to automatically control the shape of an object is the estimation of its deformation properties, which determines how the manipulator's motion actively transforms into deformations. Note that these properties are rarely known beforehand, and its offline parametric identification is difficult and/or impractical to conduct in many applications. To cope with this issue, we developed a new algorithm that computes in real time the unknown deformation parameters of a soft object; this algorithm provides a valuable adaptive behavior to the deformation controller, something we cannot achieve with traditional fixed-model approaches. In contrast with most controllers in the literature, our new method can explicitly servo-control 3-D deformations (and not just 2-D image projections) in an entirely model-free way. To validate the proposed adaptive controller, we present a detailed experimental study with robotic manipulators.

In this paper, we propose a novel design of compliant safe joint, which has flexibility when the work load exceeds a predefined threshold. The compliance is generated by a spring. We design a special transmission mechanism to convert axial motion into circumferential motion such that the linear compliance can be converted into circular one. When the end-effector of a surgical robot actuated by the compliant safe joints collides with patient's body, the compliance of the joints will protect the patient by absorbing part of the collision energy. Because of the system's special mechanical structure, the control methods should be different when it works under different states. We propose a simple algorithm to choose control methods so that the system can work both under rigid and flexible states with different controllers. We have built a prototype to validate the design and the controller.

This paper presents the development of an endoscope manipulator with passive and active structures for functional endoscopic sinus surgery (FESS). The 5-DoF passive structure has three translations and two rotations (T3R2) that allows the surgeon to manually place the endoscope near to the entry point during. The 4-DoF motorized structure (T2R2) actively controls the endoscope's position based on the surgeon's input commands. We analyze the reciprocal screw of the passive and active structures. The motion control system is based on a real-time Linux kernel that processes the commands from the surgeon and controls the manipulator's active joints. A user control interface based on an IMU fastened on the surgeon's foot is developed; this interface measures the foot's posture and through a series of gestures, it provides the desired pan/tilt/zoom motions of the camera. The developed endoscope manipulator allows the surgeon to conduct ‘two-hand’ operations while retaining direct control of the camera. We present an experimental study to validate the performance of the robotic prototype.

In this paper, we address the positioning problem of remote centre of motion (RCM) mechanisms with uncalibrated image feedback from a monocular camera. Nowadays, RCM mechanisms are widely used in minimally invasive robotic surgery due to their ability to distally rotate a tool around a fixed entry port; note that in most surgical applications, the tools are typically controlled by manual/teleoperated motion commands given by a human user. In this paper, we depart from the traditional manual control scheme and derive sensor-based methods to automatically position the manipulated tool using real-time image feedback. To this end, we first characterise the mechanism's 3-DOF configuration with the angle of the image projected tool and scalar distances between feature points. To cope with uncertainty in the camera's calibration parameters, we propose two gradient descent estimators that adaptively compute the unknown Jacobian matrix; the stability of these algorithms is proved with Lyapunov theory. Finally, we derive a kinematic image-based controller and evaluate its performance with several positioning experiments.

In this paper, we present a new robotic uterine positioner for total laparoscopic hysterectomy. The robot is designed to actively position the patient's uterus during surgery, a lengthy and tedious task that is traditionally performed by a human assistant. Safety is simply the most important concern when developing robots for surgical purposes; we address this concern in the design of our robot from a mechanical perspective. To this end, we develop a 3-DOF robotic uterine positioner with an in-body remote center of motion (RCM); this key feature allows to prevent injuries to the patient when large motions occur at the cervix. A linearly-actuated arc-guided RCM mechanism is introduced to guarantee the rigidity and stability of the robot; The system's design allows to manipulate the uterus in a decoupled manner, thus control complexity can be reduced. Passive safety mechanisms are also implemented in all DOF of the robot in order to limit the interaction forces with the patient. Experiments, including an ex-vivo test conducted with cadaver, are conducted to verify the robot's performance.