The offshore energy industry faces challenges in maintaining ageing infrastructure, with over half of North Sea platforms past their 25-year design life. This creates a need for scalable inspection methods beyond traditional manual reviews. We present a collaborative Human-AI framework that assists in detecting structural defects whilst preserving expert oversight under challenging marine conditions. Our two-stage system first employs a lightweight classifier to filter video frames by risk level. High-risk frames are then analysed by a modified Pyramid Attention Network that performs precise defect localisation. Experts validate the results at both stages, ensuring the system's continuous improvement. To better identify rare but critical flaws, we design Enhanced Tversky, a composite loss function to mitigate severe class imbalance by explicitly prioritising rare yet safety-critical defects like cracks. Evaluation on 5,525 classified frames and 1,013 segmented images demonstrates F2 score of 87.59% and a mean IoU of 78.73%, with crack detection reaching a crucial 89.62% F2 score. The framework reduces expert review time by 3.88 x whilst maintaining safety standards, offering a practical approach to scaling offshore inspection capabilities through Human-AI collaboration.

Abstract Construction drawings are frequently stored in undigitised formats and consequently, their analysis requires substantial manual effort. This is true for many crucial tasks, including material takeoff where the purpose is to obtain a list of the equipment and respective amounts required for a project. Engineering drawing digitisation has recently attracted increased attention, however construction drawings have received considerably less interest compared to other types. To address these issues, this paper presents a novel framework for the automatic processing of construction drawings. Extensive experiments were performed using two state-of-the-art deep learning models for object detection in challenging high-resolution drawings sourced from industry. The results show a significant reduction in the time required for drawing analysis. Promising performance was achieved for symbol detection across various classes, with a mean average precision of 79% for the YOLO-based method and 83% for the Faster R-CNN-based method. This framework enables the digital transformation of construction drawings, improving tasks such as material takeoff and many others.

Regina Ofori-Boateng, Magaly Aceves-Martins, Nirmalie Wiratunga, Carlos Moreno-Garcia. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Systematic Review (SR) are foundational to influencing policies and decision-making in healthcare and beyond. SRs thoroughly synthesise primary research on a specific topic while maintaining reproducibility and transparency. However, the rigorous nature of SRs introduces two main challenges: significant time involved and the continuously growing literature, resulting in potential data omission, making most SRs become outmoded even before they are published. As a solution, AI techniques have been leveraged to simplify the SR process, especially the abstract screening phase. Active learning (AL) has emerged as a preferred method among these AI techniques, allowing interactive learning through human input. Several AL software have been proposed for abstract screening. Despite its prowess, how the various parameters involved in AL influence the software’s efficacy is still unclear. This research seeks to demystify this by exploring how different AL strategies, such as initial training set, query strategies etc. impact SR automation. Experimental evaluations were conducted on five complex medical SR datasets, and the GLM model was used to interpret the findings statistically. Some AL variables, such as the feature extractor, initial training size, and classifiers, showed notable observations and practical conclusions were drawn within the context of SR and beyond where AL is deployed. • This study explores optimal Active Learning (AL) combinations for systematic reviews (SRs). • Smaller initial training samples improve performance metrics in datasets. • TF-IDF consistently outperformed Doc2Vec and S-BERT. • Certainty and Uncertainty strategies gave comparative results and effectively interacted with the TF-IDF. • The impact of AL variables in SR automation varies according to the specific dataset.

This paper presents an experience report of online attendance and associated behavioural patterns during a module in the first complete semester undertaken fully online in the autumn of 2020, and the corresponding module deliveries in 2021 and 2022. The COVID-19 pandemic of 2020 resulted in a sudden move of most university teaching online, at a global and large-scale level. This, combined with the need to maintain “business as usual” resulted in new levels of student engagement data for largely unchanged pedagogical processes. Engagement data continued to be gathered throughout the subsequent, phased return to face-to-face and hybrid learning, although at a lesser level of granularity. The wealth of student engagement data gathered during this time allows quantitative insights into how student behaviour continued to adapt during and after the enforced online learning during the COVID-19 pandemic. The anonymous subjects of this case study are computing science students in their final year of undergraduate study. We examine their engagement with the virtual learning environment, including engagement with recorded lecture material, attendance in online sessions and engagement during in-person labs. We relate this to both the students' final grades and the content of the module itself. A number of conclusions are drawn based on this empirical data, relating to observations made by staff and pedagogical theory. There was a moderate, but significant, correlation between engagement in synchronous online lecture sessions and grades during thelockdown phase, but the strength of this correlation has reduced in subsequent years as normality has returned. From monitoring behaviour in online sessions down to minute-by-minute accuracy, it can also be seen that some students strategised their engagement based on sessions they perceived to be most directly contributory to their assessment, placing little value on live guest lecturer sessions. During enforced online learning, the most successful students, on average, engaged with less repeat content than less successful students, instead apparently utilising lecture recordings to “catch up” with missed live lectures.

In recent years, deep learning has rapidly become a method of choice for segmentation of medical images. Deep neural architectures such as UNet and FPN have achieved high performances on many medical datasets. However, medical image analysis algorithms are required to be reliable, robust, and accurate for clinical applications which can be difficult to achieve for some single deep learning methods. In this study, we introduce an ensemble of classifiers for semantic segmentation of medical images. The ensemble of classifiers here is a set of various deep learning-based classifiers, aiming to achieve better performance than using a single classifier. We propose a weighted ensemble method in which the weighted sum of segmentation outputs by classifiers is used to choose the final segmentation decision. We use a swarm intelligence algorithm namely Comprehensive Learning Particle Swarm Optimization to optimize the combining weights. Dice coefficient, a popular performance metric for image segmentation, is used as the fitness criteria. Experiments conducted on some medical datasets of the CAMUS competition on cardiographic image segmentation show that our method achieves better results than both the constituent segmentation models and the reported model of the CAMUS competition.

Significant progress has been made in the area of text classification and natural language processing. However, like many other datasets from across different domains, text-based datasets may suffer from class-imbalance. This problem leads to model's bias toward the majority class instances. In this paper, we present a new approach to handle class-imbalance in text data by means of unsupervised learning algorithms. We present class-decomposition using two different unsupervised methods, namely k-means and Density-Based Spatial Clustering of Applications with Noise, applied to two different sentiment analysis data sets. The experimental results show that utilizing clustering to find within-class similarities can lead to significant improvement in learning algorithm's performances as well as reducing the dominance of the majority class instances without causing information loss.

Abstract Class-imbalanced datasets are common across several domains such as health, banking, security, and others. The dominance of majority class instances (negative class) often results in biased learning models, and therefore, classifying such datasets requires employing some methods to compact the problem. In this paper, we propose a new hybrid approach aiming at reducing the dominance of the majority class instances using class decomposition and increasing the minority class instances using an oversampling method. Unlike other undersampling methods, which suffer data loss, our method preserves the majority class instances, yet significantly reduces its dominance, resulting in a more balanced dataset and hence improving the results. A large-scale experiment using 60 public datasets was carried out to validate the proposed methods. The results across three standard evaluation metrics show the comparable and superior results with other common and state-of-the-art techniques.

Maze navigation is a recurring challenge in robotics competitions, where the aim is to design a strategy for one or several entities to traverse the optimal path in a fast and efficient way. To do so, numerous alternatives exist, relying on different sensing systems. Recently, camera-based approaches are becoming increasingly popular to address this scenario due to their reliability and given the possibility of migrating the resulting technologies to other application areas, mostly related to human-robot interaction. The aim of this paper is to present a pipeline methodology towards enabling a robot solving maze autonomously, by means of computer vision and path planning. Afterwards, the robot is capable of communicating the learned experience to a second robot, which then will solve the same challenge considering its own mechanical characteristics which may differ from the first robot. The pipeline is divided into four steps: (1) camera calibration (2) maze mapping (3) path planning and (4) communication. Experimental validation shows the efficiency of each step towards building this pipeline.

Engineering drawings such as Piping and Instrumentation Diagrams contain a vast amount of text data which is essential to identify shapes, pipeline activities, tags, amongst others. These diagrams are often stored in undigitised format, such as paper copy, meaning the information contained within the diagrams is not readily accessible to inspect and use for further data analytics. In this paper, we make use of the benefits of recent deep learning advances by selecting models for both text detection and text recognition, and apply them to the digitisation of text from within real world complex engineering diagrams. Results show that 90% of text strings were detected including vertical text strings, however certain non text diagram elements were detected as text. Text strings were obtained by the text recognition method for 86% of detected text instances. The findings show that whilst the chosen Deep Learning methods were able to detect and recognise text which occurred in simple scenarios, more complex representations of text including those text strings located in close proximity to other drawing elements were highlighted as a remaining challenge.

One of the key features of most document image digitisation systems is the capability of discerning between the main components of the printed representation at hand. In the case of engineering drawings, such as circuit diagrams, telephone exchanges or process diagrams, the three main shapes to be localised are the symbols, text and connectors. While most of the state of the art devotes to top-down recognition approaches which attempt to recognise these shapes based on their features and attributes, less work has been devoted to localising the actual pixels that constitute each shape, mostly because of the difficulty in obtaining a reliable source of training samples to classify each pixel individually. In this work, we present a convolutional neural network (CNN) capable of classifying each pixel, using a type of complex engineering drawings known as Piping and Instrumentation Diagram (P&ID) as a case study. To obtain the training patches, we have used a semi-automated heuristics-based tool which is capable of accurately detecting and producing the symbol, text and connector layers of a particular P&ID standard in a considerable amount of time (given the need of human interaction). Experimental validation shows that the CNN is capable of obtaining these three layers in a reduced time, with the pixel window size used to generate the training samples having a strong influence on the recognition rate achieved for the different shapes. Furthermore, we compare the average run time that both the heuristics-tool and the CNN need in order to produce the three layers for a single diagram, indicating future directions to increase accuracy for the CNN without compromising the speed.

Engineering drawings are commonly used across different industries such as oil and gas, mechanical engineering and others. Digitising these drawings is becoming increasingly important. This is mainly due to the legacy of drawings and documents that may provide rich source of information for industries. Analysing these drawings often requires applying a set of digital image processing methods to detect and classify symbols and other components. Despite the recent significant advances in image processing, and in particular in deep neural networks, automatic analysis and processing of these engineering drawings is still far from being complete. This paper presents a general framework for complex engineering drawing digitisation. A thorough and critical review of relevant literature, methods and algorithms in machine learning and machine vision is presented. Real-life industrial scenario on how to contextualise the digitised information from specific type of these drawings, namely piping and instrumentation diagrams, is discussed in details. A discussion of how new trends on machine vision such as deep learning could be applied to this domain is presented with conclusions and suggestions for future research directions.

Technical drawings are commonly used across different industries such as Oil and Gas, construction, mechanical and other types of engineering. In recent years, the digitization of these drawings is becoming increasingly important. In this paper, we present a semi-automatic and heuristic-based approach to detect and localise symbols within these drawings. This includes generating a labeled dataset from real world engineering drawings and investigating the classification performance of three different state-of the art supervised machine learning algorithms. In order to improve the classification accuracy the dataset was pre-processed using unsupervised learning algorithms to identify hidden patterns within classes. Testing and evaluating the proposed methods on a dataset of symbols representing one standard of drawings, namely Process and Instrumentation (P&ID) showed very competitive results.

Given a fleet of robots, automatic estimation of the relative poses between them could be inaccurate in specific environments. We propose a framework composed by the fleet of robots with embedded stereoscopic cameras providing 2D and 3D images of the scene, a human coordinator and a Human-Machine Interface. We suppose auto localising each robot through GPS or landmarks is not possible. 3D-images are used to automatically align them and deduce the relative position between robots. 2Dimages are used to reduce the alignment error in an interactive manner. A human visualises both 2D-images and the current automatic alignment, and imposes a new alignment through the Human-Machine Interface. Since the information is shared through the whole fleet, robots can deduce the position of other ones that do not visualise the same scene. Practical evaluation shows that in situations where there is a large difference between images, the interactive processes are crucial to achieve an acceptable result.