Training supervised deep neural networks that perform defect detection and segmentation requires large-scale fully-annotated datasets, which can be hard or even impossible to obtain in industrial environments. Generative AI offers opportunities to enlarge small industrial datasets artificially, thus enabling the usage of state-of-the-art supervised approaches in the industry. Unfortunately, also good generative models need a lot of data to train, while industrial datasets are often tiny. Here, we propose a new approach for reusing general-purpose pre-trained generative models on industrial data, ultimately allowing the generation of self-labelled defective images. First, we let the model learn the new concept, entailing the novel data distribution. Then, we force it to learn to condition the generative process, producing industrial images that satisfy well-defined topological characteristics and show defects with a given geometry and location. To highlight the advantage of our approach, we use the synthetic dataset to optimise a crack segmentor for a real industrial use case. When the available data is small, we observe considerable performance increase under several metrics, showing the method’s potential in production environments.

Cooperation among multiple autonomous surface and underwater vehicles is an important capability for detection and tracking of underwater objects. Cooperative autonomy in the underwater environment, however, is challenged by the communication bandwidth. In this work, we propose a selective communication scheme that underpins collaborative surveillance under communication constraints. This scheme classifies signal reflections of sonar pulses that are detected by on-board sensor processing as contacts with the object of interest or background using a convolutional neural network. This network is trained using previously labelled contact spectrograms obtained during three sea trials carried out between 2016-2018. The classification scores at the CNN output are ordered to select the few contacts that the underwater modem bandwidth allows for transmission to the network. First, we evaluate the accuracy of the data-driven information selection scheme using recall scores and similar performance measures. Then, we find the accuracy in Bayesian recursive filtering (tracking) of these contacts for different communication rates using established error metrics. The results suggest that the selective scheme yields a favourable surveillance performance communication cost trade-off.

The necessity of long-range underwater surveillance has strongly increased in the last decades, and low-frequency active sonar (LFAS) systems seem to fulfill this need. However, in littoral environments with shallow water LFAS may suffer from an elevate number of false alarms. In this context, the capability to distinguish between object-generated and clutter-generated detections is crucial. This paper describes a multiobject tracking framework based on the sum-product algorithm that exploits the information provided by a convolutional neural network that classifies the LFAS detections. The effectiveness of the proposed approach is demonstrated both in a simulated and in a real underwater scenario.

Cooperative autonomy and data sharing can largely improve the mission performance of robotic networks in underwater surveillance applications. In this paper, we describe the cooperative autonomy used to control the Autonomous Underwater Vehicles (AUVs) acting as sonar receiver nodes in the CMRE Anti-Submarine Warfare (ASW) network. The paper focuses on a track management module that was integrated in the robot autonomy software for enabling the share of information. Track to track (T2T) associations are used for improving track classification and for creating a common tactical picture, necessary for AUV cooperative strategies. We also present a new cooperative data-driven AUV behaviour that exploits the spatial diversity of multiple robots for improving target tracking and for facilitating T2T associations. We report results with real data collected at sea that validate the approach. The reported results are one of the first examples that show the potential of cooperative autonomy and data fusion in realistic underwater surveillance scenarios characterised by limited communications.

Multitarget tracking (MTT) is a challenging task that aims at estimating the number of targets and their states from measurements provided by one or multiple sensors. Additional information, such as imperfect estimates of target classes provided by a classifier, can facilitate the target-measurement association and thus improve MTT performance. In this letter, we describe how a recently proposed MTT framework based on the sum-product algorithm can be extended to efficiently exploit class information. The effectiveness of the proposed approach is demonstrated by simulation results.

Capturing and labeling camera images in the real world is an expensive task, whereas synthesizing labeled images in a simulation environment is easy for collecting large-scale image data. However, learning from only synthetic images may not achieve the desired performance in the real world due to a gap between synthetic and real images. We propose a method that transfers learned detection of an object position from a simulation environment to the real world. This method uses only a significantly limited dataset of real images while leveraging a large dataset of synthetic images using variational autoen-coders. Additionally, the proposed method consistently performed well in different lighting conditions, in the presence of other distractor objects, and on different backgrounds. Experimental results showed that it achieved accuracy of 1.5 mm to 3.5 mm on average. Furthermore, we showed how the method can be used in a real-world scenario like a “pick-and-place” robotic task.

This paper is a contribution towards interpretability of the deep learning models in different applications of time-series. We propose a temporal attention layer that is capable of selecting the relevant information to perform various tasks, including data completion, key-frame detection and classification. The method uses the whole input sequence to calculate an attention value for each time step. This results in more focused attention values and more plausible visualisation than previous methods. We apply the proposed method to three different tasks. Experimental results show that the proposed network produces comparable results to a state of the art. In addition, the network provides better interpretability of the decision, that is, it generates more significant attention weight to related frames compared to similar techniques attempted in the past.

This paper describes a framework called MaestROBe It is designed to make the robots perform complex tasks with high precision by simple high-level instructions given by natural language or demonstration. To realize this, it handles a hierarchical structure by using the knowledge stored in the forms of ontology and rules for bridging among different levels of instructions. Accordingly, the framework has multiple layers of processing components; perception and actuation control at the low level, symbolic planner and Watson APIs for cognitive capabilities and semantic understanding, and orchestration of these components by a new open source robot middleware called Project Intu at its core. We show how this framework can be used in a complex scenario where multiple actors (human, a communication robot, and an industrial robot) collaborate to perform a common industrial task. Human teaches an assembly task to Pepper (a humanoid robot from SoftBank Robotics) using natural language conversation and demonstration. Our framework helps Pepper perceive the human demonstration and generate a sequence of actions for UR5 (collaborative robot arm from Universal Robots), which ultimately performs the assembly (e.g. insertion) task.

While deep reinforcement learning techniques have recently produced considerable achievements on many decision-making problems, their use in robotics has largely been limited to simulated worlds or restricted motions, since unconstrained trial-and-error interactions in the real world can have undesirable consequences for the robot or its environment. To overcome such limitations, we propose a novel reinforcement learning architecture, OptLayer, that takes as inputs possibly unsafe actions predicted by a neural network and outputs the closest actions that satisfy chosen constraints. While learning control policies often requires carefully crafted rewards and penalties while exploring the range of possible actions, OptLayer ensures that only safe actions are actually executed and unsafe predictions are penalized during training. We demonstrate the effectiveness of our approach on robot reaching tasks, both simulated and in the real world.

The high precision assembly of mechanical parts requires precision that exceeds that of robots. Conventional part-mating methods used in the current manufacturing require numerous parameters to be tediously tuned before deployment. We show how a robot can successfully perform a peg-in-hole task with a tight clearance through training a recurrent neural network with reinforcement learning. In addition to reducing manual effort, the proposed method also shows a better fitting performance with a tighter clearance and robustness against positional and angular errors for the peg-in-hole task. The neural network learns to take the optimal action by observing the sensors of a robot to estimate the system state. The advantages of our proposed method are validated experimentally on a 7-axis articulated robot arm.

Spatio-temporal anomaly detection by unsupervised learning have applications in a wide range of practical settings. In this paper we present a surveillance system for industrial robots using a monocular camera. We propose a new unsupervised learning method to train a deep feature extractor from unlabeled images. Without any data augmentation, the algorithm co-learns the network parameters on different pseudo-classes simultaneously to create unbiased feature representation. Combining the learned features with a prediction system, we can detect irregularities in high dimensional data feed (e.g. video of a robot performing pick and place task). The results show how the proposed approach can detect previously unseen anomalies in the robot surveillance video. Although the technique is not designed for classification, we show the use of the learned features in a more traditional classification application for CIFAR-10 dataset.

We study the effect of soft (i.e. compliant) soles in humanoid walking. Adding compliance at the foot sole is important for shock absorption during touchdown and to better cast the ground roughness. But, subsequent deformation needs to be considered in the attitude stabilization. We propose a sole deformation estimator used in the controller design, to ensure that the desired ZMP for stability requirement is fulfilled. We performed two comparison experiments on the HRP-4 humanoid robot with/without the estimator. Experiments showed how the estimator enhances the ZMP stabilization during walking and prevents the robot from falling down.