Inspecting large-scale industrial surfaces like aircraft fuselages for quality control requires precise, high-resolution 3D geometry. Vision-based tactile sensors (VBTSs) offer high local resolution but require slow 'press-and-lift' measurements for large areas. Sliding or roller/belt VBTS designs provide continuous measurement but face significant challenges: sliding suffers from friction/wear, while both are speed-limited by camera frame rates and motion blur. Thus, a rapid, continuous, high-resolution method is needed. We introduce a novel neuromorphic tactile roller sensor. It uses a modified event-based multi-view stereo algorithm for 3D reconstruction, leveraging high temporal resolution and motion blur robustness. This reconstruction is most effective for surfaces with distinct edges or sharp features, which are often the most critical for defect detection in industrial inspection tasks. We demonstrate 0.5 m/s scanning speeds with MAE below 100$\mu$m (11x faster than prior methods). A multi-reference Bayesian fusion strategy reduces MAE by 25.2% (vs. EMVS) and mitigates curvature errors. We also validate high-speed Braille reading 2.6x faster than previous approaches.

Moving object segmentation is critical to interpret scene dynamics for robotic navigation systems in challenging environments. Neuromorphic vision sensors are tailored for motion perception due to their asynchronous nature, high temporal resolution, and reduced power consumption. However, their unconventional output requires novel perception paradigms to leverage their spatially sparse and temporally dense nature. In this work, we propose a novel event-based motion segmentation algorithm using a Graph Transformer Neural Network, dubbed GTNN. Our proposed algorithm processes event streams as 3D graphs by a series of nonlinear transformations to unveil local and global spatiotemporal correlations between events. Based on these correlations, events belonging to moving objects are segmented from the background without prior knowledge of the dynamic scene geometry. The algorithm is trained on publicly available datasets including MOD, EV-IMO, and EV-IMO2 using the proposed training scheme to facilitate efficient training on extensive datasets. Moreover, we introduce the Dynamic Object Mask-aware Event Labeling (DOMEL) approach for generating approximate ground-truth labels for event-based motion segmentation datasets. We use DOMEL to label our own recorded Event dataset for Motion Segmentation (EMS-DOMEL), which we release to the public for further research and benchmarking. Rigorous experiments are conducted on several unseen publicly-available datasets where the results revealed that GTNN outperforms state-of-the-art methods in the presence of dynamic background variations, motion patterns, and multiple dynamic objects with varying sizes and velocities. GTNN achieves significant performance gains with an average increase of 9.4% and 4.5% in terms of motion segmentation accuracy (IoU%) and detection rate (DR%), respectively.

The difficulties posed by terrain-induced slippage for wheeled rovers traversing soft terrains are critical to ensuring safe and precise mobility. While bilateral teleoperation systems offer a promising solution to this issue, the inherent network-induced delays hinder the fidelity of the closed-loop integration, potentially compromising teleoperator system controls, and resulting in poor command-tracking performance. This work introduces a new model-free predictor framework based on deep learning designed to improve prediction performance and effectively compensate for large network delays in teleoperated wheeled rovers. Our approach employs the Recurrent Neural Network (RNN) to achieve a significant improvement in modeling complexity and prediction accuracy. Particularly, our framework consists of two distinct predictors, each tailored to the forward and backward coupling variables of the teleoperated wheeled rover. Human-in-the-loop experiments were conducted to validate the effectiveness of the developed framework in compensating for the delays encountered by teleoperated wheeled rovers coupled with terrain-induced slippage. The results confirm the improved prediction accuracy of the framework. This improvement is evidenced by improved performance and transparency metrics, which lead to better command-tracking performance. A supplementary video is available at https://youtu.be/-06UGumQ0tA.

Unsupervised anomaly detection techniques, which operate without prior knowledge of anomalies, have garnered significant attention in industrial inspection due to their adaptability and generalization. Therefore, knowledge-based computer vision techniques have been broadly applied to identify unusual image patterns. However, real-time industrial applications present challenges such as limited anomalous samples, inadequate defect knowledge, and complex background textures. These factors lead to difficulties in accurately identifying defect regions, and conventional auto-encoder networks often struggle to overcome these issues.To address these limitations, we propose a multi-scale feature reconstruction (MSFR) network specifically designed for domain shift scenarios. Our approach employs a pyramidal vision transformer network (PVTN) to reconstruct multi-scale feature maps, capturing discriminative features at various scales. Additionally, a pre-trained module extracts multi-level features at the same scale, and a dedicated feature matching module enhances accuracy by improving the alignment probability between features. The MSFR strategy surpasses conventional auto-encoders by filtering pixel-level information at multiple depths. Empirical evaluations were conducted using benchmark datasets such as MVTec AD and AeBAD-S. Furthermore, an extensive ablation study demonstrates the effectiveness and viability of the proposed MSFR approach for industrial anomaly detection tasks. The experimental results show that the proposed model significantly outperforms recent approaches, making it highly suitable for real-world industrial applications, particularly in manufacturing. • MSFR is introduced for industrial anomaly detection to handle domain shifts generalization. • A multi-scale features network is developed to efficiently detect defects. • A pyramidal network is proposed to cover the limitations of traditional autoencoders. • Comprehensive experiments are generated over AeBAD-S and MVTec IAD benchmark datasets.

Object segmentation enhances robotic grasping by aiding object identification. Complex environments and dynamic conditions pose challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Bimodal SegNet that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders — one for RGB signal input and another for Event signal input, in addition to an Atrous Pyramidal Feature Amplification module. Encoders capture and fuse the rich contextual information from different resolutions via a Cross-Domain Contextual Attention layer while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The results show a 4%–6% MIOU score improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The source code, dataset and model are publicly available at: https://github.com/sanket0707/Bimodal-SegNet.

Event cameras triggered a paradigm shift in the computer vision community delineated by their asynchronous nature, low latency, and high dynamic range. Calibration of event cameras is always essential to account for the sensor intrinsic parameters and for 3D perception. However, conventional image-based calibration techniques are not applicable due to the asynchronous, binary output of the sensor. The current standard for calibrating event cameras relies on either blinking patterns or event-based image reconstruction algorithms. These approaches are difficult to deploy in factory settings and are affected by noise and artifacts degrading the calibration performance. To bridge these limitations, we present E-Calib, a novel, fast, robust, and accurate calibration toolbox for event cameras utilizing the asymmetric circle grid, for its robustness to out-of-focus scenes. E-Calib introduces an efficient reweighted least squares (eRWLS) method for feature extraction of the calibration pattern circles with sub-pixel accuracy and robustness to noise. In addition, a modified hierarchical clustering algorithm is devised to detect the calibration grid apart from the background clutter. The proposed method is tested in a variety of rigorous experiments for different event camera models, on circle grids with different geometric properties, on varying calibration trajectories and speeds, and under challenging illumination conditions. The results show that our approach outperforms the state-of-the-art in detection success rate, reprojection error, and pose estimation accuracy.

Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer vision community and is serving as a key enabler for a wide range of applications. This technology has offered significant advantages, including reduced power consumption, reduced processing needs, and communication speedups. However, neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms. In this article, we propose a novel noise filtration algorithm to eliminate events that do not represent real log-intensity variations in the observed scene. We employ a graph neural network (GNN)-driven transformer algorithm, called GNN-Transformer, to classify every active event pixel in the raw stream into real log-intensity variation or noise. Within the GNN, a message-passing framework, referred to as EventConv, is carried out to reflect the spatiotemporal correlation among the events while preserving their asynchronous nature. We also introduce the known-object ground-truth labeling (KoGTL) approach for generating approximate ground-truth labels of event streams under various illumination conditions. KoGTL is used to generate labeled datasets, from experiments recorded in challenging lighting conditions, including moon light. These datasets are used to train and extensively test our proposed algorithm. When tested on unseen datasets, the proposed algorithm outperforms state-of-the-art methods by at least 8.8% in terms of filtration accuracy. Additional tests are also conducted on publicly available datasets (ETH Zürich Color-DAVIS346 datasets) to demonstrate the generalization capabilities of the proposed algorithm in the presence of illumination variations and different motion dynamics. Compared to state-of-the-art solutions, qualitative results verified the superior capability of the proposed algorithm to eliminate noise while preserving meaningful events in the scene.

Vision-based control of unmanned aerial vehicles (UAVs) has been adopted in a wide range of applications due to the availability of low-cost onboard sensors and computers. Tuning such systems to work properly requires extensive domain-specific experience, which limits the growth of emerging applications. Moreover, obtaining performance limits for UAVs performing visual servoing is difficult due to the complexity of the models used. In this article, we propose a novel noise-tolerant approach for real-time identification and tuning of visual servoing systems. This is based on the deep neural networks (DNNs) classification of the system response generated by the modified relay feedback test (MRFT). The proposed method, called DNN with noise-protected MRFT (DNN-NP-MRFT), can be used with a multitude of vision sensors and estimation algorithms despite high levels of sensor noise. The response of DNN-NP-MRFT to noise perturbations is investigated and its effect on identification and tuning performance is analyzed. The proposed DNN-NP-MRFT is able to detect performance changes induced by the use of high latency vision sensors or by integrating an inertial measurement unit sensor into the UAV states estimation pipeline. Experimental identification closely matches simulation results, which can be used to explain the system behavior and predict the closed-loop performance limits for a given hardware and software setup. We also demonstrate the ability of DNN-NP-MRFT-tuned UAVs to reject external disturbances like wind or human push and pull. Finally, we discuss the advantages of the proposed DNN-NP-MRFT visual servoing design approach compared with other approaches in the literature.

In this paper, we introduce a coverage path planning algorithm with adaptive viewpoint sampling to construct accurate 3D models of complex large structures using Unmanned Aerial Vehicle (UAV). The developed algorithm, Adaptive Search Space Coverage Path Planner (ASSCPP), utilizes an existing 3D reference model of the complex structure and the onboard sensors' noise models to generate paths that are evaluated based on the traveling distance and the quality of the model. The algorithm generates a set of viewpoints by performing adaptive sampling that directs the search towards areas with low accuracy and low coverage. The algorithm predicts the coverage percentage obtained by following the generated coverage path using the reference model. A set of experiments were conducted in real and simulated environments with structures of different complexities to test the validity of the proposed algorithm.

This paper presents an artificial neural networks approach to estimate the indicated torque of a single- cylinder diesel engine from crank shaft angular position and velocity measurements. Since these variables can be measured using low-cost sensors, the estimator may be useful in the implementation of the control or diagnostics strategies that require cylinder indicated torque, a variables that are not easily measured and need expensive sensors. The approach is to design indicated torque estimators using feedback and an artificial neural networks model as feedforward. Such an approach can offer the advantage of being amenable to real-time implementation. The estimated results of the engine indicated torque are presented, which compared with experimental data indicate a good agreement.

This paper proposes a novel approach for the identification of tissue properties in-vivo using a force sensitive wheeled probe. The purpose of such a device is to compensate a surgeon for a portion of the loss of haptic and tactile feedback experienced during robotic-assisted minimally invasive surgery. Initially, a testing facility for validating the concept ex-vivo was developed and used to characterize two different testing modalities - static (1-DOF) tissue indentation and rolling (2-DOF) tissue indentation. As part of the static indentation experiments a mathematical model was developed to classify tissue condition based on changes in mechanical response. The purpose of the rolling indentation tests was to detect tissue abnormalities, such as tumors, which are difficult to isolate under static testing conditions. During such tests, the test-rig was capable of detecting simulated miniature buried masses at depths of 12mm. Based on these experiments a portable device capable of carrying out similar tests in-vivo was developed. The device was designed to be operated through a trocar port and its key feature is the ability to transition between static indentation and rolling indentation modalities without retracting and changing the tool.

This paper considers a tracked vehicle traversing unknown terrain, and proposes an approach based on the Generalized Newton Raphson (GNR) method for identifying all the unknown soil parameters required for tractive force prediction. For the first time, the methodology, based on measurements of track slip, i, and tractive force, F, to find unknown soil parameters is developed. The tractive force is the force generated by a tracked vehicle to drive itself forwards. This tractive force depends to a large extent on certain soil parameters, namely soil cohesion (c), soil internal friction angle (phi), and soil shear deformation modulus (K). Accurately identifying parameters of the soil on which a tracked vehicle is moving will potentially lead to accurate traversability prediction, effective traction control, and precise trajectory tracking. The soil parameter identification algorithm is validated with the experimental data from Wong [3] and from in-house track- terrain interaction test rig showing good identification accuracy and fast execution speed. It is also shown to be relatively robust to initial condition. The identified soil parameters are, in turn, used to predict the tractive forces showing good agreement with all the experimental data. The technique presented in this paper is general and can be applied to any tracked vehicle.

This paper proposes a novel nonlinear viscoelastic soft tissue model generated from ex vivo experimental results on ovine liver using a force sensitive probe. In order to study the biomechanics of soft tissue, static indentation tests were applied on ovine liver. An empirical constitutive equation was extracted from the examined data. A mechanical model combining linear viscoelasticity with a nonlinear function of strain-stress is proposed. The developed model has been evaluated both statically and dynamically with different strain rates - i.e. where the velocity of indentation is varied. By comparing simulation results and measured experimental data, it has been concluded that the proposed model is robust for modelling both static and dynamic indentation conditions. The effect of changing boundary conditions on the parameters in the proposed model has been studied by choosing test sites with different underlying tissue thicknesses. The results indicate that for small strain, the effect of the thickness condition is reasonable to be neglected.

This paper presents a novel technique for identifying soil parameters for a wheeled vehicle travelling on an unknown terrain. The identified soil parameters are required for predicting vehicle drawbar pull and wheel drive torque which can be employed for traversability prediction, traction control, and performance optimization of a wheeled vehicle on unknown terrain. The Newton Raphson method is used as the identification technique applied on the modified form of the wheel-soil interaction dynamics model using the composite Simpson's rule. This work focuses on identifying the internal friction angle, the shear deformation modulus, and the lumped pressure-sinkage coefficient. The fourth parameter, cohesion, does not influence the vehicle drawbar pull and is assigned an average value during the identification process. In an experimental study, the identified parameters are compared with known values, and shown to be in good agreement. Soil parameter identification can be carried out on-line and thus our approach is suitable for real-time applications. The robustness of the method is also shown to be relatively good. The identified soil parameters can be used to predict drawbar pull and wheel drive torque with good accuracy

Accurate estimation of slip is essential in developing autonomous navigation strategies for mobile vehicles operating in unstructured terrain. In this paper, a sliding mode observer is firstly constructed to estimate slip parameters based on the kinematics model of a skid-steering vehicle and trajectory measurement. The stability of the sliding mode observer is given in a mathematical context. Slip estimation schemes using an extended Kalman filter and direct mathematical inversion of the kinematic equations are also presented for comparison purposes. It is shown that the non-linear sliding mode observer is more accurate than the other two methods. The robustness and superior performance of the sliding mode observer is demonstrated using both simulation and experimental results. A camera based system is used to measure the vehicle trajectory during experimental validation

Online identification of soil-bucket interaction is significant for the development of an autonomous excavation strategy. This paper presents a method for identifying multiple unknown soil parameters in real-time using the novel Modified Newton Raphson Method. The new approach involving a model selection strategy based on Parallel Execution Model, Flexible Parallel Execution Model and Hybrid Execution Model consisting of the Mohr-Coulomb soil model and the Chen and Liu Upper Bound soil model is examined. The identification of unknown soil parameters is achieved by minimizing the error between the measured forces and the modeled forces computed by the soil model. The results demonstrate that the proposed estimation scheme is accurate when comparing to the measured soil parameters. In addition, the reduced processing time of convergence and high robustness with respect to initial conditions show that the proposed method has real time estimation capability. The proposed method is very promising and highly suitable for soil-bucket interaction identification for an autonomous excavation system in an unpredictable, dynamic environment.

Real time estimation for soil-tool interaction is a key method for the development of an autonomous excavation strategy. This paper presents a method for identifying the unknown soil parameters in real-time using a novel hybrid soil model. The hybrid models consist of the Mohr-Coulomb soil model and the Chen and Liu Upper Bound soil model. A switching mode is utilized to select a more accurate soil model to compute the failure force depending on the position of the excavator bucket. The Newton Raphson method is proposed to identify the soil parameters by minimizing the error between the experimental forces and the forces computed by the hybrid soil model. The results demonstrate that the proposed estimation scheme is accurate when comparing to the measured soil parameters and the experimental data. In addition, the high speed estimation time shows that the proposed method has a real time estimation capability. A very high robustness is achieved when compared to the least square method. The proposed method is very promising and highly suitable for the soil-tool interaction identification of an autonomous excavator in an unpredictable, dynamical and potentially hazardous environment.

A robust, fast, and simple technique for the experimental identification of the link parameters (mass, inertia, and length) and friction coefficients of a full-scale excavator arm is presented. This new technique, based on the generalized Newton method (GNM), estimates unknown individual parameters of the excavator arm dynamic equations. The technique can be used when the number of equations is different from the number of estimated variables. Using experimental data from a full-scale field Combat Engineer Excavator (CEE), the values of link parameters and friction coefficients are successfully identified. The identified parameters are compared with known values, and shown to be in agreement. The method is compared with the least square method, and shows that the GNM is better in terms of prediction accuracy and robustness to noise. Further, the joint positions predicted by the analytical model using the identified parameters are validated against different experimental trajectories, showing very good agreement. The experimental data was obtained in collaboration with QinetiQ Ltd. (Hampshire, U.K.). The technique presented in this paper is general and can be applied to any manipulator.

This paper presents a novel method for estimating soil properties on-line during excavation tasks such as ground leveling, digging and sheet pilling. The proposed method computes key soil parameters by measuring the forces acting on the excavator bucket whilst being in contact with the soil and minimizing the error between measured forces and estimated forces produced by a real-time capable soil model. Two soil models, the Mohr-Coulomb soil model and the Chen and Liu upper bound soil model, are implemented and researched in the context of this estimation scheme. Parameter optimization is carried out employing the Newton Raphson method. The method is evaluated using experimental data and through comparison with an approach that makes use of graphical intersection for model optimization. The results demonstrate that the proposed Newton Raphson-based method is as accurate as the graphical intersection-based approach, but up to 2000 times faster, and thus, most suitable for on-line soil parameter estimation in an automated system which provides optimized digging trajectories for a given excavation task.

A robust and fast yet simple approach for experimental identification of the link (mass, inertia and length) parameters and friction coefficients for a full-scale closed-chain excavator arm is developed. The approach is based on a generalized Newton method, an excavation arm dynamic model and measured data. The new approach can be used where the number of equations is different from the number of variables, or if the Jacobian cannot be assumed nonsingular. The parameters are needed for improving the control actions in autonomous solution for excavation tasks and contributing to excavator design evaluation. Using experimental data obtained while moving the links of an instrumented full-scale combat engineer excavator, the values of the link parameters and friction coefficients for various links were successfully identified. The identified parameters are compared with physical values, and they are in agreement. Further, the joint torques and positions computed by the developed model using the identified parameters are validated against measured data, also showing excellent agreement. The experimental data was obtained in collaboration with QinetiQ. The novel technique presented in this paper is general and can be applied to a wide range of heavy-duty closed chain hydraulic manipulators.

The backpropagation (BP) algorithm is commonly used in many applications, including robotics, automation and weight changes of artificial neural networks (ANNs). This paper proposes the addition of an extra term, a proportional factor (PF), to the standard BP algorithm to speed up the weight adjusting process. The proposed algorithm is tested and the results show that the proposed algorithm outperforms the conventional BP algorithm in convergence speed and the ability to escape from learning stalls. The paper presents a convergence analysis of the three-term BP algorithm. It is shown that if the learning parameters of the three-term BP algorithm satisfies certain conditions given in this paper, then it is guaranteed that the system is stable and will converge to a local minimum. The paper shows that all the local minima of the cost function are stable for the three-term backpropagation algorithm.

This paper presents an integrated physics based model for a front end mobile excavator. The model describes the dynamic relationship between the operator input commands and the position, orientation, speed and forces of the vehicle and the excavation arm. The dynamic model has the potential to be used in advanced controller design for automated excavation systems. The dynamic model for the excavation system is validated against measured data. The validation of the model is conducted in collaboration with QinetiQ, UK. A PID controller for trajectory tracking is implemented and tested using a computer simulation study. The graphical machine model is developed in the Zed3D graphical environment and all the inputs to the graphical model are taken from the simulation model.