Road pavement detection and segmentation are critical for developing autonomous road repair systems. However, developing an instance segmentation method that simultaneously performs multi-class defect detection and segmentation is challenging due to the textural simplicity of road pavement image, the diversity of defect geometries, and the morphological ambiguity between classes. We propose a novel end-to-end method for multi-class road defect detection and segmentation. The proposed method comprises multiple spatial and channel-wise attention blocks available to learn global representations across spatial and channel-wise dimensions. Through these attention blocks, more globally generalised representations of morphological information (spatial characteristics) of road defects and colour and depth information of images can be learned. To demonstrate the effectiveness of our framework, we conducted various ablation studies and comparisons with prior methods on a newly collected dataset annotated with nine road defect classes. The experiments show that our proposed method outperforms existing state-of-the-art methods for multi-class road defect detection and segmentation methods.

Ensuring traffic safety is crucial, which necessitates the detection and prevention of road surface defects. As a result, there has been a growing interest in the literature on the subject, leading to the development of various road surface defect detection methods. The methods for detecting road defects can be categorised in various ways depending on the input data types or training methodologies. The predominant approach involves image-based methods, which analyse pixel intensities and surface textures to identify defects. Despite popularity, image-based methods share the distinct limitation of vulnerability to weather and lighting changes. To address this issue, researchers have explored the use of additional sensors, such as laser scanners or LiDARs, providing explicit depth information to enable the detection of defects in terms of scale and volume. However, the exploration of data beyond images has not been sufficiently investigated. In this survey paper, we provide a comprehensive review of road surface defect detection studies, categorising them based on input data types and methodologies used. Additionally, we review recently proposed non-image-based methods and discuss several challenges and open problems associated with these techniques.

Most deep anomaly detection models are based on learning normality from datasets due to the difficulty of defining abnormality by its diverse and inconsistent nature. Therefore, it has been a common practice to learn normality under the assumption that anomalous data are absent in a training dataset, which we call normality assumption. However, in practice, the normality assumption is often violated due to the nature of real data distributions that includes anomalous tails, i.e., a contaminated dataset. Thereby, the gap between the assumption and actual training data affects detrimentally in learning of an anomaly detection model. In this work, we propose a learning framework to reduce this gap and achieve better normality representation. Our key idea is to identify sample-wise normality and utilize it as an importance weight, which is updated iteratively during the training. Our framework is designed to be model-agnostic and hyperparameter insensitive so that it applies to a wide range of existing methods without careful parameter tuning. We apply our framework to three different representative approaches of deep anomaly detection that are classified into one-class classification-, probabilistic model-, and reconstruction-based approaches. In addition, we address the importance of a termination condition for iterative methods and propose a termination criterion inspired by the anomaly detection objective. We validate that our framework improves the robustness of the anomaly detection models under different levels of contamination ratios on five anomaly detection benchmark datasets and two image datasets. On various contaminated datasets, our framework improves the performance of three representative anomaly detection methods, measured by area under the ROC curve.

Reidentification (Re-id) of vehicles in a multicamera system is an essential process for traffic control automation. Previously, there have been efforts to reidentify vehicles based on shots of images with identity (id) labels, where the model training relies on the quality and quantity of the labels. However, labeling vehicle ids is a labor-intensive procedure. Instead of relying on expensive labels, we propose to exploit camera and tracklet ids that are automatically obtainable during a Re-id dataset construction. In this article, we present weakly supervised contrastive learning (WSCL) and domain adaptation (DA) techniques using camera and tracklet ids for unsupervised vehicle Re-id. We define each camera id as a subdomain and tracklet id as a label of a vehicle within each subdomain, i.e., weak label in the Re-id scenario. Within each subdomain, contrastive learning using tracklet ids is applied to learn a representation of vehicles. Then, DA is performed to match vehicle ids across the subdomains. We demonstrate the effectiveness of our method for unsupervised vehicle Re-id using various benchmarks. Experimental results show that the proposed method outperforms the recent state-of-the-art unsupervised Re-id methods. The source code is publicly available on https://github.com/andreYoo/WSCL_VeReid.

The performance of road defect segmentation (a.k.a. pixel-level road defect detection) has been improved alongside with remarkable achievement of deep learning. Those improvements need a large-scale and well-constructed dataset. However, road surface materials or designs vary from country to country, and the patterns of defects are hard to pre-define. In this paper, we propose a novel multi-source domain adaptation method to boost the performance of road defect segmentation on an unlabelled dataset. The proposed method generates multi-source ensembled labels using transferred information from models trained with multiple labelled source domains, which are utilised as supervisory signals for the unlabelled target domain. Furthermore, to reduce the domain gap between each source domain and a target domain, these domains are re-aligned with outlier repositioning to improve the defect segmentation performance. We demonstrate the effectiveness of our proposed method on Cracktree200, CRACK500, CFD, and Crack360 datasets. Experimental results show that the proposed method outperforms the existing unsupervised road defect segmentation methods and achieves competitive performance compared with recent supervised methods. The source code is publicly available on https://github.com/andreYoo/MSDA_RDS.git.

Recently, vehicle re-identification methods based on deep learning constitute remarkable achievement. However, this achievement requires large-scale and well-annotated datasets. In constructing the dataset, assigning globally available identities (Ids) to vehicles captured from a great number of cameras is labour-intensive, because it needs to consider their subtle appearance differences or viewpoint variations. In this paper, we propose camera-tracklet-aware contrastive learning (CTACL) using the multi-camera tracklet information without vehicle identity labels. The proposed CTACL divides an unlabelled domain, i.e., entire vehicle images, into multiple camera-level subdomains and conducts contrastive learning within and beyond the subdomains. The positive and negative samples for contrastive learning are defined using tracklet Ids of each camera. Additionally, the domain adaptation across camera networks is introduced to improve the generalisation performance of learnt representations and alleviate the performance degradation resulted from the domain gap between the subdomains. We demonstrate the effectiveness of our approach on video-based and image-based vehicle Re-ID datasets. Experimental results show that the proposed method outperforms the recent state-of-the-art unsupervised vehicle Re-ID methods. The source code for this paper is publicly available on https://github.com/andreYoo/CTAM-CTACL-VVReID.git.

Detecting unusual behaviors of insiders on enterprise resource planning (ERP) systems is one of the essential parts to reduce the risks of threatening and abusing enterprise resources by insiders. Many approaches to detect the behaviors based on rule-based systems and stochastic processes are currently limited to empirical monitoring using manually established algorithms or probabilistic boundaries. Those approaches need prior knowledge such as user permission guideline and process data characteristics. Unfortunately, obtaining prior knowledge is hard in practice, and these are not appropriate to detect atypical unusual behavior which can not be clearly defined using heuristic rules. Therefore, in this article, we propose a novel framework for unusual insider behavior detection (UIBD) for ERP systems. The proposed framework initially derives a discriminative model for normal behavior samples, and UIBD is conducted by computing an error using the model. Since the model is compiled using normal samples only, the error of unusual samples would be larger than normal ones. To derive a robust normal behavior model, we present adversarial recurrent autoencoder (ARAE). To demonstrate the efficiency of the proposed framework based on ARAE, we conduct experiments using a dataset composed of insider behaviors defined by sequences of security audit logs of ERP systems operating in real-world enterprises. The experimental results show that the proposed framework with ARAE can successfully detect unusual insider behaviors and outperform other methods to detect unusual insider behavior or threatening.

We present adversarial event prediction (AEP), a novel approach to detecting abnormal events through an event prediction setting. Given normal event samples, AEP derives the prediction model, which can discover the correlation between the present and future of events in the training step. In obtaining the prediction model, we propose adversarial learning for the past and future of events. The proposed adversarial learning enforces AEP to learn the representation for predicting future events and restricts the representation learning for the past of events. By exploiting the proposed adversarial learning, AEP can produce the discriminative model to detect an anomaly of events without complementary information, such as optical flow and explicit abnormal event samples in the training step. We demonstrate the efficiency of AEP for detecting anomalies of events using the UCSD-Ped, CUHK Avenue, Subway, and UCF-Crime data sets. Experiments include the performance analysis depending on hyperparameter settings and the comparison with existing state-of-the-art methods. The experimental results show that the proposed adversarial learning can assist in deriving a better model for normal events on AEP, and AEP trained by the proposed adversarial learning can surpass the existing state-of-the-art methods.

The key challenge of unsupervised vehicle re-identification (Re-ID) is learning discriminative features from unlabelled vehicle images. Numerous methods using domain adaptation have achieved outstanding performance, but those methods still need a labelled dataset as a source domain. This paper addresses an unsupervised vehicle Re-ID method, which no need any types of a labelled dataset, through a Self-supervised Metric Learning (SSML) based on a feature dictionary. Our method initially extracts features from vehicle images and stores them in a dictionary. Thereafter, based on the dictionary, the proposed method conducts dictionary-based positive label mining (DPLM) to search for positive labels. Pair-wise similarity, relative-rank consistency, and adjacent feature distribution similarity are jointly considered to find images that may belong to the same vehicle of a given probe image. The results of DPLM are applied to dictionary-based triplet loss (DTL) to improve the discriminativeness of learnt features and to refine the quality of the results of DPLM progressively. The iterative process with DPLM and DTL boosts the performance of unsupervised vehicle Re-ID. Experimental results demonstrate the effectiveness of the proposed method by producing promising vehicle Re-ID performance without a pre-labelled dataset. The source code for this paper is publicly available on https://github.com/andreYoo/VeRI_SSML_FD.git.

Traffic scene recognition, which requires various visual classification tasks, is a critical ingredient in autonomous vehicles. However, most existing approaches treat each relevant task independently from one another, never considering the entire system as a whole. Because of this, they are limited to utilizing a task-specific set of features for all possible tasks of inference-time, which ignores the capability to leverage common task-invariant contextual knowledge for the task at hand. To address this problem, we propose an algorithm to jointly learn the task-specific and shared representations by adopting a multi-task learning network. Specifically, we present a lower bound for the mutual information constraint between shared feature embedding and input that is considered to be able to extract common contextual information across tasks while preserving essential information of each task jointly. The learned representations capture richer contextual information without additional task-specific network. Extensive experiments on the large-scale dataset HSD demonstrate the effectiveness and superiority of our network over state-of-the-art methods.

In the past few years, the performance of road defect detection has been remarkably improved thanks to advancements in various studies on computer vision and deep learning. Although large-scale and well-annotated datasets enhance the performance of detecting road defects to some extent, it is still challengeable to derive a model which can perform reliably for various road conditions in practice, because it is intractable to construct a dataset considering diverse road conditions and defect patterns. To end this, we propose an unsupervised approach to detect road defects, using Adversarial Image-to-Frequency Transform (AIFT). AIFT adopts the unsupervised manner and adversarial learning in deriving the defect detection model, so AIFT does not require annotations for road defects. We evaluate the efficiency of AIFT using GAPs384 dataset, Cracktree200 dataset, CRACK500 dataset, and CFD dataset. The experimental results demonstrate that the proposed approach detects various road detects, and it outperforms existing state-of-the-art approaches.

In this work, we proposed a framework for zero-shot action recognition with spatio-temporal feature (ST-features) in order to address the problem of unconstrained action recognition. It is more challenging than the constrained action recognition problem, since a model has to recognize actions which do not appear in the training step. The proposed framework consists of two models: 1) ST-feature extraction model and 2) verification model. The ST-feature extraction model extracts discriminative ST-features from a given video clip. With these features, the verification model computes the similarity between them to examine class-identity whether their classes are identical or not. The experimental results show that the proposed framework can outperform other action recognition methods under the unconstrained condition.

We propose a condition-adaptive representation learning framework for driver drowsiness detection based on a 3D-deep convolutional neural network. The proposed framework consists of four models: spatio-temporal representation learning, scene condition understanding, feature fusion, and drowsiness detection. Spatio-temporal representation learning extracts features that can describe motions and appearances in video simultaneously. Scene condition understanding classifies the scene conditions related to various conditions about the drivers and driving situations, such as statuses of wearing glasses, illumination condition of driving, and motion of facial elements, such as head, eye, and mouth. Feature fusion generates a condition-adaptive representation using two features extracted from the above models. The drowsiness detection model recognizes driver drowsiness status using the condition-adaptive representation. The condition-adaptive representation learning framework can extract more discriminative features focusing on each scene condition than the general representation so that the drowsiness detection method can provide more accurate results for the various driving situations. The proposed framework is evaluated with the NTHU drowsy driver detection video dataset. The experimental results show that our framework outperforms the existing drowsiness detection methods based on visual analysis.

In this paper, we address face verification and person re-identification tasks under the unconstrained condition. Both tasks under the unconstrained condition are difficult since the testing dataset can contain the identities not appeared in training dataset. To overcome the difficulty, we propose deep discriminative representation learning (DDRL) to learn a discriminative representation which can cover not only trained representation but the appearance of images which are not trained. DDRL can be viewed as imposing discriminative constraints on the learnt representation via joint optimization for verification and identification objectives. The experimental results for face verification and person re-identification shows the superiority of DDRL in both tasks.

Previous models based on Deep Convolutional Neural Networks (DCNN) for face verification focused on learning face representations. The face features extracted from the models are applied to additional metric learning to improve a verification accuracy. The models extract high-dimensional face features to solve a multi-class classification. This results in a dependency of a model on specific training sets since a dimension of the feature should be equal to the number of subjects in a training set. In this paper, we propose a method for learning feature representations which directly determine whether two input images are identical using a single model based on DCNN and residual learning. It is possible to remove the dependency since the model doesn't learn face representations based on multi-class classification. We show that the proposed method achieves the competitive performance for face verification. We demonstrate the face verification performance of the proposed method using the test dataset of Labeled Face in the Wild dataset.