In laparoscopic surgical training and evaluation, real-time recognition of surgical actions with transparency outputs is crucial for automated, objective, and immediate instructional feedback to support skills improvement. However, we face challenges due to limited dataset sizes and variability in surgical environments. This study presents AutoMedTS , an end-to-end automated machine learning framework customized for medical time-series data, enabling rapid deployment using surgical suturing trajectories collected from both expert and novice surgeons. The proposed method features key improvements including: (i) a novel temperature-scaled Softmax resampling technique effectively addressing severe class imbalance, and (ii) an uncertainty-aware ensemble selection mechanism ensuring robust predictions across surgeons with varying skill levels. Additionally, the approach emphasizes model transparency to meet the high standards of reliability and transparency required in medical applications. Compared to deep learning methods, traditional machine learning models not only facilitate efficient rapid deployment but also offer significant transparency advantages. Experimental results demonstrate that our method provides fast, stable, and reliable real-time surgical action recognition in clinical training environments. Code and data are publicly available at https://github.com/baobingzhang/AutoMedTS .

• First beam image dataset capturing diverse optical-alignment patterns and parameters • Optical resonator alignment cast as a pairwise beam-pattern regression task • Two-stage model with both feature interaction and refinement for coarse-to-fine alignment • Trained on one device, the model generalizes to another without re-training • Achieves high accuracy with real-time inference on embedded edge hardware Accurate alignment of a laser resonator is essential for upscaling industrial laser manufacturing and precision processing. However, traditional manual or semi-automatic methods depend heavily on operator expertise, and struggle with the interdependence among multiple alignment parameters. To tackle this, we introduce the first real-world image dataset for automatic laser resonator alignment, collected on a laboratory-built resonator setup. It comprises over 6,000 beam profiler images annotated with four key alignment parameters (intracavity iris aperture diameter, output coupler pitch and yaw actuator displacements, and axial position of the output coupler), with over 500,000 paired samples for data‐driven alignment. Given a pair of beam profiler images exhibiting distinct beam patterns under different configurations, the system predicts the control-parameter changes required to realign the resonator. Leveraging this dataset, we propose a novel two-stage deep learning framework for automatic resonator alignment. In Stage 1, a multi-scale CNN augmented with cross-attention and correlation-difference modules, extracts features and outputs an initial coarse prediction of alignment parameters. In Stage 2, a feature-difference map is computed by subtracting the paired feature representations and fed into an iterative refinement module to correct residual misalignments. The final prediction combines coarse and refined estimates, integrating global context with fine-grained corrections for accurate inference. Experiments on our dataset and a different instance of the same physical system from which the CNN was trained suggest superior accuracy and practicality to manual alignment.

Despite the potential of mobile manipulators and applications where robots require a force-controlled physical interaction with the environment, the majority of robot automation nowadays is still based on fixed manipulators for free-motion tasks (e.g. welding, pick and place, or painting). In this work, we propose a control solution for omnidirectional mobile manipulators in force-tracking tasks, interacting with unknown surface geometries and with a human teleoperator in the control loop. Keeping a teleoperator in the loop makes the system widely applicable to unstructured environments. With little effort, a human can take care of the mobile base navigation, self-collisions, and collisions with the environment, as well as selecting the area of the asset surface to process. The teleoperator interfaces with the robot platform by commanding motion in the mobile base to increase the arm’s workspace and manoeuvrability. The operator can also command the movement of the end-effector, sliding on the surface geometry to process a specific area. Alternatively, he can let the controller execute a parametric trajectory (spiral or raster) for an autonomous area coverage and meanwhile telecommand the base in order to keep the arm in configurations with good dexterity. The autonomous controller, on the other hand, takes responsibility for following the unknown contour on the manipulated surface by only taking observations from a force/torque sensor attached to the arm’s wrist, exerting a prescribed force, and handling the motion control in the base and the arm so that both can follow their respective task requests. Overall, we have developed a user-friendly control scheme, where an operator with little training and using a joystick, can guide the robot system to perform a physically interactive task on the surface of an asset.

This study presents a pioneering effort to replicate human neuromechanics experiments within a virtual environment utilising a digital human model. By employing MyoSuite, a state-of-the-art human motion simulation platform enhanced by Reinforcement Learning (RL), multiple types of impedance identification experiments of human elbows were replicated on a digital musculoskeletal model. We compared the motor control capability of an RL agent with that of an actual human elbow in terms of the impedance identified through torque perturbation. The findings reveal that the RL agent exhibits higher elbow impedance to stabilise the target elbow motion under perturbation than a human does. It is likely due to the shorter reaction time and superior sensory capabilities of the RL agent. This study serves as a preliminary exploration into the potential of human digital twins for neuromechanics experiments. An RL-controlled digital twin with the musculoskeletal structure of the human body is expected to be useful in validating rehabilitation techniques before experiments on real human subjects.

The estimation of the 6D pose of industrial components is essential for smart manufacturing. Especially for complex units that require intensive manual operations, such as a concentrator photovoltaics solar panel, accurate spatial localization provides visual aids for industrial automation. In this paper, we propose an accurate and scalable framework to address the dimensional variability of industrial components and tackle practical implementation issues. First, we use the scalable architecture EfficientNet as the backbone coupled with an enhanced feature pyramid network to estimate the object’s pose. By introducing vertical and horizontal connections of shallow layers, the feature extraction of small objects is optimized for better detection accuracy. Second, leveraging the reliable 2D detection results and geometry information, an adaptive pose refinement module is designed to adjust the estimated 6D pose. The scaling of the backbone network and the computational complexity of refined modules are uniformly adjusted via a shared hyperparameter, resulting in a globally scalable framework. In terms of the pose estimation accuracy, the effectiveness of the refinement module and the real-time performance, validations are conducted both on the LINEMOD dataset and our customized datasets comprising of objects from the industrial photovoltaic system. Additionally, to further illustrate the effectiveness of the proposed method, a precision parallel robot is employed to validate the accuracy of real-time object pose tracking.

Spasticity is a common neuromuscular abnormality following upper motor neuron lesions. Conventionally, spasticity is assessed through manual clinical scales, which have limitations due to the subjectivity involved. The development of rehabilitation robotics introduced new solutions to this problem, producing novel robot-assisted spasticity assessment approaches. In this article, we present the current state and challenges of robot-assisted hand spasticity assessment (RAHSA) based on a review of instrumented clinical scales, biomechanical and neurophysiological measures, and medical imaging methods for upper extremity spasticity assessment between January 2000 and February 2023. The characteristics of hand anatomy and spasticity symptoms make it challenging to develop RAHSA approaches and corresponding robotic systems. Although the combination of hand robots and instrumented assessment methods has evoked studies on RAHSA, more research is needed on the new assessment approaches fusing neurological and nonneurological measures and novel robotic systems specifically designed for hand spasticity assessment.

Unmanned underwater vehicles are crucial for deep-sea exploration and inspection without imposing any danger to human life due to extreme environmental conditions. But, designing a robust controller that can cope with model uncertainties, external disturbances, and time delays for such vehicles is a challenge. This paper implements a sliding mode position control algorithm with a time-delay estimation term to a remotely operated underwater vehicle to deal with disturbances, such as waves, and time delays. The controller is implemented on an underwater vehicle (BlueRov) and compared with a proportional-integral-derivative (PID) controller in a wave tank with different disturbances and when there exist delays within the communication channel. The experimental results show that the proposed control method provides better performance than the conventional PID in the presence of extreme disturbances with less control efforts.

The assembly of 3D products with complex geometry and material, such as a concentrator photovoltaics solar panel unit, is typically conducted manually. This results in low efficiency, precision and throughput. This study is motivated by an actual industrial need and targeted towards automation of the currently manual assembly process. By replacing the manual assembly with robotic assembly systems, the efficiency and throughput could be improved. Prior to assembly, it is essential to estimate the pose of the objects to be assembled with high precision. The choice of the machine vision is important and plays a critical role in the overall accuracy of such a complex task. Therefore, this work focuses on the 6D pose estimation for precision assembly utilizing a 3D vision sensor. The sensor we use is a 3D structured light scanner which can generate high quality point cloud data in addition to 2D images. A 6D pose estimation method is developed for an actual industrial solar-cell object, which is one of the four objects of an assembly unit of concentrator photovoltaics solar panel. The proposed approach is a hybrid approach where a mask R-CNN network is trained on our custom dataset and the trained model is utilized such that the predicted 2D bounding boxes are used for point cloud segmentation. Then, the iterative closest point algorithm is used to estimate the object's pose by matching the CAD model to the segmented object in point cloud.

Assessment of minimally invasive surgical skills is a non-trivial task, usually requiring the presence and time of expert observers, including subjectivity and requiring special and expensive equipment and software. This study develops an algorithm for tracking laparoscopy instruments in the video cues of a standard laparoscopy training box with a single webcam camera and proposes new criteria to assess skill level using the extracted tool trajectories. Instrument tracking and assessment criteria together constitute a significant step towards developing a low cost, automated, and widely applicable laparoscopy training and assessment system using a standard physical training box equipped with a webcam. The developed visual tracking algorithm recovers the 3D positions of the laparoscopic instruments tips to which simple colored tapes (markers) are attached. The new assessment criteria are based on frequency analysis and linear discriminant analysis of the 3D reconstructed trajectories of the instruments. The performance of these proposed criteria are compared to the conventional criteria for laparoscopy training and demonstrated to be superior on the data we have recorded from six professional laparoscopy surgeons and ten novice subjects.

This paper presents measurements of human hand-impedance during a laparoscopic training program with physically interactive robotic manipulators. The knowledge of how the hand-impedance changes due to training might be useful to inform better training programs and to introduce co-manipulated robotic assistants for effective trainings. Ten novice subjects participated in a three weeks training program for a suturing activity in laparoscopy. The subjects have been instructed to set the needle, enter the skin, and tie knots by using laparoscopic tools within a Minimally Invasive Surgery training box. Variable admittance controlled robots, attached to the tools with force sensors, applied step vice velocity disturbances while subjects were trying to set the needle. Based on the interaction force and end-effector position information, impedances of the left and right hands were computed in four different directions. The computed results were compared with respect to the participants skill progression.

In this paper, we present a robotic assistance scheme which allows for impedance compensation with stiffness, damping, and mass parameters for hand manipulation tasks and we apply it to manual welding. The impedance compensation does not assume a preprogrammed hand trajectory. Rather, the intention of the human for the hand movement is estimated in real time using a smooth Kalman filter. The movement is restricted by compensatory virtual impedance in the directions perpendicular to the estimated direction of movement. With airbrush painting experiments, we test three sets of values for the impedance parameters as inspired from impedance measurements with manual welding. We apply the best of the tested sets for assistance in manual welding and perform welding experiments with professional and novice welders. We contrast three conditions: 1) welding with the robot's assistance; 2) with the robot when the robot is passive; and 3) welding without the robot. We demonstrate the effectiveness of the assistance through quantitative measures of both task performance and perceived user's satisfaction. The performance of both the novice and professional welders improves significantly with robotic assistance compared to welding with a passive robot. The assessment of user satisfaction shows that all novice and most professional welders appreciate the robotic assistance as it suppresses the tremors in the directions perpendicular to the movement for welding.

The goal of this paper is to perform end-point impedance measurements across dominant and nondominant hands while doing airbrush painting and to use the results for developing a robotic assistance scheme. We study airbrush painting because it resembles in many ways manual welding, a standard industrial task. The experiments are performed with the 7 degrees of freedom KUKA lightweight robot arm. The robot is controlled in admittance using a force sensor attached at the end-point, so as to act as a free-mass and be passively guided by the human. For impedance measurements, a set of nine subjects perform 12 repetitions of airbrush painting, drawing a straight-line on a cartoon horizontally placed on a table, while passively moving the airbrush mounted on the robot's end-point. We measure hand impedance during the painting task by generating sudden and brief external forces with the robot. The results show that on average the dominant hand displays larger impedance than the nondominant in the directions perpendicular to the painting line. We find the most significant difference in the damping values in these directions. Based on this observation, we develop a "directional damping" scheme for robotic assistance and conduct a pilot study with 12 subjects to contrast airbrush painting with and without robotic assistance. Results show significant improvement in precision with both dominant and nondominant hands when using robotic assistance.

This paper presents a study of hand impedance measurements comparatively across ten professional and 14 novice manual welders, when they are performing tungsten inert gas (TIG) welding interactively with the KUKA lightweight robot arm (LWR). The results show that hand impedance differs across professional and novice welders. The welding torch is attached to the KUKA LWR, which is admittance controlled via a force sensor to give the feeling of a free floating mass at its end-effector. The subjects perform TIG welding on 1.5-mm-thick stainless steel plates by manipulating the torch. Impedance is measured by introducing external force disturbances and fitting a mass-damper-spring model to human hand reactions. The quality of welding is measured using the variance of the position signals above 0.1 Hz. Professional welders demonstrate less variance and, in general, apply larger hand impedance (larger damping and stiffness) than the novice welders. The variance of position during nominal welding is minimal for both professional and novice welders in the direction perpendicular to the welding line in the plane of the plate, which is the most important direction for the quality of the weld. For both professional and novice welders, the mass and damping values are largest in this direction compared with the other two directions. Professional welders demonstrate larger damping than the novice welders in this direction.

This paper presents a study of end-point impedance measurement at human hand, with professional and novice manual welders when they are performing Tungsten Inert Gas (TIG) welding interactively with the KUKA Light Weight Robot Arm (LWR). The welding torch is attached to the KUKA LWR, which is admittance controlled via a force sensor to give the feeling of a free floating mass at its end-effector. The subjects perform TIG welding on 1.5 mm thick stainless steel plates by manipulating the torch attached to the robot. The end-point impedance values are measured by introducing external force disturbances and by fitting a mass-damper-spring model to human hand reactions. Results show that, for professionals and novices, the mass, damping and stiffness values in the direction perpendicular to the welding line are the largest compared to the other two directions. The novices demonstrate less resistance to disturbances in this direction. Two of the professionals present larger stiffness and one of them presents larger damping. This study supports the hypothesis that impedance measurements could be used as a partial indicator, if not direct, of skill level to differentiate across different levels of manual welding performances. This work contributes towards identifying tacit knowledge of manual welding skills by means of impedance measurements.

This paper presents the mechanical design of a distal scanner to perform a spiral scan for mosaic-imaging with a confocal microlaparoscope. First, it is demonstrated with ex vivo experiments that a spiral scan performs better than a raster scan on soft tissue. Then a mechanical design is developed in order to perform the spiral scan. The design in this paper is based on a conic structure with a particular curved surface. The mechanism is simple to implement and to drive; therefore, it is a low-cost solution. A 5:1 scale prototype is implemented by rapid prototyping and the requirements are validated by experiments. The experiments include manual and motor drive of the system. The manual drive demonstrates the resulting spiral motion by drawing the tip trajectory with an attached pencil. The motor drive demonstrates the speed control of the system with an analysis of video thread capturing the trajectory of a laser beam emitted from the tip.

This paper presents an approach for understanding the soft tissue behavior in surface contact with a hard object scanning the tissue. The application domain is confocal microlaparoscope imaging, mostly used for imaging the outer surface of the organs in the abdominal cavity. The probe (optic-head) is swept over the tissue to collect sequential images to obtain a large field of view with mosaicing. The problem we address is that the tissue also moves with the probe due to its softness; therefore the resulting mosaic is not in the same shape and dimension as traversed by the probe. Our approach inspires from the finger slip studies and adapts the idea of load-and-slip that explains the movement of the finger when dragged on a hard surface. We propose the concept of loading-distance and perform measurements with in total 84 experiments on beef liver and chicken breast tissues. Our results indicate that the loading-distance can be measured prior to a scan and be used during the scan in order to compensate the movement of the probe. In this way we can have an image-mosaic of the tissue surface in a desired shape.

Probe-based confocal laser endomicroscopy is a recent tissue imaging technology that requires placing a probe in contact with the tissue to be imaged and provides real time images with a microscopic resolution. Additionally, generating adequate probe movements to sweep the tissue surface can be used to reconstruct a wide mosaic of the scanned region while increasing the resolution which is appropriate for anatomico-pathological cancer diagnosis. However, properly controlling the motion along the scanning trajectory is a major problem. Indeed, the tissue exhibits deformations under friction forces exerted by the probe leading to deformed mosaics. In this paper we propose a visual servoing approach for controlling the probe movements relative to the tissue while rejecting the tissue deformation disturbance. The probe displacement with respect to the tissue is firstly estimated using the confocal images and an image registration real-time algorithm. Secondly, from this real-time image-based position measurement, the probe motion is controlled thanks to a simple proportional-integral compensator and a feedforward term. Ex vivo experiments using a Stäubli TX40 robot and a Mauna Kea Technologies Cellvizio imaging device demonstrate the effectiveness of the approach on liver and muscle tissue.

In this paper, a physically interactive control scheme is developed for a manipulator robot arm. The human touches the robot and applies force in order to make it behave as he/she likes. The communication between the robot and the human is maintained by a physical contact with no sensors. The intent of the human is estimated by observing the change in control effort. The robot receives the estimated human intent and updates its position reference accordingly. The developed method uses the principle of conservation of zero momentum for position-controlled systems. A switching scheme is developed that goes between the modes of pure impedance control with a fixed-position reference and interactive control under human intent. The switching mechanism uses neither a physical switch nor a sensor; it observes the human intent and puts the robot into interactive mode, if there is any. When the human intent disappears, the robot goes into the pure-impedance-control mode, thus stabilizing in the left position.

In this paper, distribution of required forces and moments to the supporting legs of a six-legged robot is handled as a torque-distribution problem. This approach is comparatively contrasted to the conventional approach of tip-point force distribution. The formulation of dynamics is performed by using the joint torques as the primary variables. The sum of the squares of the joint torques on the supporting legs is considered to be proportional to the dissipated power. The objective function is constructed as this sum, and the problem is formulated as to minimize this quadratic objective function with respect to linear equality and inequality constraints. It is demonstrated that the torque-distribution scheme results in a much more efficient distribution compared with the conventional scheme of force distribution. In contrast to the force distribution, the torque-distribution scheme makes good use of interaction forces and friction in order to minimize the required joint torques