Vision transformers (ViTs), which have demonstrated a state-of-the-art performance in image classification, can also visualize global interpretations through attention-based contributions. However, the complexity of the model makes it difficult to interpret the decision-making process, and the ambiguity of the attention maps can cause incorrect correlations between image patches. In this study, we propose a new ViT neural tree decoder (ViT-NeT). A ViT acts as a backbone, and to solve its limitations, the output contextual image patches are applied to the proposed NeT. The NeT aims to accurately classify fine-grained objects with similar inter-class correlations and different intra-class correlations. In addition, it describes the decision-making process through a tree structure and prototype and enables a visual interpretation of the results. The proposed ViT-NeT is designed to not only improve the classification performance but also provide a human-friendly interpretation, which is effective in resolving the trade-off between performance and interpretability. We compared the performance of ViT-NeT with other state-of-art methods using widely used fine-grained visual categorization benchmark datasets and experimentally proved that the proposed method is superior in terms of the classification performance and interpretability. The code and models are publicly available at https://github.com/jumpsnack/ViT-NeT.

This paper proposes a novel facial landmark detection (FLD) algorithm for use in real driving situations. The proposed algorithm is based on an ensemble of local weighted random forest regressor (WRFR) with random sampling consensus (RANSAC) and explicit global shape models, and considers the dynamic and irregular characteristics of driving. In this paper, to estimate the offset distance from a landmark and a reference point, we first detect the nose region as a reference point. Next, we propose a local WRFR to maintain the generality with a small number of regression trees. With the WRFR, we adopt RANSAC instead of the averaging or median of offsets to handle the problem of sensitivity to outlier offsets, and we estimate the accurate 2D offset vector. To identify the erroneous positions of local landmarks and rearrange the overall landmark layout, we adopt the global face models based on the spatial relation between landmarks. Using the unified framework of the proposed FLD, our proposed algorithm is robust to large head poses and partial occlusions caused by a driver's hair or sunglasses. For a benchmark data set considering real driving situations, we construct a data set called a face alignment data set used in driving (FADID) using a near-infrared camera for FLD under real driving situations. We apply the proposed algorithm to various driving sequences in FADID, and the results show that its FLD performance is better than that of other state-of-the-art methods, while the computational speed is high for real-time applications such as driver-state monitoring systems.

Sudden pedestrian crossing (SPC) is the major reason for pedestrian-vehicle crashes. In this paper, we focus on detecting SPCs at night for supporting an advanced driver assistance system using a far-infrared (FIR) camera mounted on the front roof of a vehicle. Although the thermal temperature of the road is similar to or higher than that of the pedestrians during summer nights, many previous researches have focused on pedestrian detection during the winter, spring, or autumn seasons. However, our research concentrates on SPC during the hot summer season because the number of collisions between pedestrians and vehicles in Korea is higher at that time than during the other seasons. For real-time processing, we first decide the optimal levels of image scaling and search area. We then use our proposed method for detecting virtual reference lines that are associated with road segmentation without using color information and change these lines according to the turning direction of the vehicle. Pedestrian detection is conducted using a cascade random forest with low-dimensional Haar-like features and oriented center-symmetric local binary patterns. The SPC is predicted based on the likelihood and the spatiotemporal features of the pedestrians, such as their overlapping ratio with virtual reference lines, as well as the direction and magnitude of each pedestrian's movement. The proposed algorithm was successfully applied to various pedestrian data sets captured by an FIR camera, and the results show that its SPC detection performance is better than those of other methods.

Pedestrian-vehicle accidents that occur at night are a major social problem worldwide. Advanced driver assistance systems that are equipped with cameras have been designed to automatically prevent such accidents. Among the various types of cameras used in such systems, far-infrared (FIR) cameras are favorable because they are invariant to illumination changes. Therefore, this paper focuses on a pedestrian nighttime tracking system with an FIR camera that is able to discern thermal energy and is mounted on the forward roof part of a vehicle. Since the temperature difference between the pedestrian and background depends on the season and the weather, we therefore propose two models to detect pedestrians according to the season and the weather, which are determined using Weber-Fechner's law. For tracking pedestrians, we perform real-time online learning to track pedestrians using boosted random ferns and update the trackers at each frame. In particular, we link detection responses to trajectories based on similarities in position, size, and appearance. There is no standard data set for evaluating the tracking performance using an FIR camera; thus, we created the Keimyung University tracking data set (KMUTD) by combining the KMU sudden pedestrian crossing (SPC) data set [21] for summer nights with additional tracking data for winter nights. The KMUTD contains video sequences involving a moving camera, moving pedestrians, sudden shape deformations, unexpected motion changes, and partial or full occlusions between pedestrians at night. The proposed algorithm is successfully applied to various pedestrian video sequences of the KMUTD; specifically, the proposed algorithm yields more accurate tracking performance than other existing methods.

In this study, we propose a novel method for facial landmark detection (FLD) based on an ensemble of local weighted regressors and a global face shape model under real driving situations. Unlike other FLD approaches, the method proposed in this study first detects the nose region instead of a face-bounding box as a reference point for estimating the offset from a landmark and a reference point. Next, a weighted random forest regressor (WRFR) is used for designing a regressor that maintains the generality while utilizing a small number of decision trees. During the training period, some of the trees having low accuracy are removed and the remaining trees of the WRFR have different weights according to their regression accuracy. As a global face shape model, we use the spatial relationship between three landmarks to identify erroneous estimates of the local regressors and provide valid alternatives. Using the unified framework of the proposed FLD, our algorithm is robust to facial expressions and partial occlusions caused by a subject's hair or sunglasses. For our experiment, using a near-infrared camera, we constructed a benchmark dataset for FLD under real driving situations, which we call the Face Alignment Dataset used In Driving (FADID). The proposed algorithm was successfully applied to various driving sequences in FADID, and the results show that its FLD detection performance is better than that of other state-of-the-art methods.

According to a report, night time and poor illumination driving is overall 2-3 times more dangerous then day time. For example, young people aged 18-24 were killed between 21:00 and 05:59 (the night-time and early morning) on week-days in the EU-23 countries because of road accident in 2010. As the similar pattern with EU, most pedestrian-vehicle accidents occur between 6 p.m. and 8 a.m., and the rate of pedestrian fatalities is highest between 4 a.m. and 6 a.m. in South Korea. Among several factors such as inebriated drivers and pedestrian, drowsiness, decreased visibility is the major cause of pedestrian-vehicle accident at night. To reduce the accident owing to driver's inattention at night, recent advanced driver assistance system (ADAS) has been researching on automatic pedestrian detection and tracking using night vision camera. Therefore, this tutorial focuses on introducing a multiple pedestrians tracking system using a thermal camera that is able to discern thermal energy at night-time. In a pedestrian tracking-by-detection system, multi-pedestrian detection accuracy is essential for post tracking process. Since the temperature difference between the pedestrian and background depends on the season and weather, we therefore first introduce two models for detecting pedestrians according to the season and weather, which are determined using Weber-Fechner's law. Two detection models use the optimal levels of the image scaling and search area instead of image pyramid to reduce the computational cost of image scaling for detecting multiple pedestrians of various sizes. Online learning is appropriate in the case that image frames is obtained sequentially. Theoretically offline learning could obtain global optimal solution while it is not as practical as online learning. Therefore, we introduce some state-of-the-art real-time online learning algorithms with our online learning based on boosted random ferns (BRFs) in detail based on the references as the second topic. Third, for association checking of multiple pedestrians, we explain the advantages and disadvantages of feed-forward system and global association system. Feed-forward system uses only current and past observations, which is called tracklet to estimate the current tracker's state. On contrary, global association system uses future and global information to estimate the current tracker's state in an offline step, for example, bipartite graph matching, Hungarian algorithm, dynamic programming, and min-cost max-flow network flow. Because maintaining the tracker's ID in successive frames is a challenging task owing to overlapping pedestrians in the multiple pedestrians tracking system, we introduce a few association checking algorithms to maintain the tracker's ID. As the feature for association checking, we also explain popular features in computer vision, such as the spatial proximity, velocity orientation and context (shape, size, color). Fourth, we introduce a few evaluation video sequences for pedestrian tracking such as OSli thermal pedestrian database, CVC-09 sequences, KAIST benchmark dataset, and KMliTD dataset. In particular, we are focusing on the KMliTD which contains video sequences involving a moving camera, moving multiple pedestrians, sudden shape deformations, unexpected motion changes, and partial or full occlusions between pedestrians at summer and winter night. Finally, we introduce the evaluation methods to measure the performance of the pedestrian tracking system such as Multiple Object Tracking Precision (MOTP) and Multiple Object Tracking Accuracy (MOTA), and Tracking Distance Error (TDE). The performance comparison among different tracking approaches is also presented when the proposed online tracking method is applied to benchmark data. As the further research in multiple pedestrians tracking, we will guide the fusion of sensors such as Radio Detection and Ranging (RADAR) sensor or Light Detection and Ranging (LIDAR) sensor with a camera for overcome the limitations occurred in a standalone sensor. In addition, with the increasing sensor resolutions, we mention the plan how to develop computationally feasible multi-extended object tracking algorithm.

This paper presents a novel algorithm for tracking multiple persons with thermal imaging. The algorithm uses online random ferns (RF) learning to update the model of the person and particle filters to approximate the person's location. To estimate the observational likelihood for particle weighting, we perform online training for the initial ferns using boosted random ferns (BRF) in the first frame in regions where persons are detected. Then, RF for the tracker model is re-trained based on the observed distribution of selected ferns in consecutive frames. To design a robust tracking model impervious to occlusion, we divide person regions into 4 x 4 sub-blocks and then train the RF using concatenated feature vectors from 16 sub-blocks. In addition, we propose an occlusion-check algorithm to distinguish normal object-tracking from long and short-term occlusion. The proposed algorithm is compared with similar existing algorithms to show that its tracking performance is superior to those of other classifiers and tracking methods.

This paper presents a wildfire smoke detection method based on a spatiotemporal bag-of-features (BoF) and a random forest classifier. First, candidate blocks are detected using key-frame differences and non-parametric color models to reduce the computation time. Subsequently, spatiotemporal three-dimensional (3D) volumes are built by combining the candidate blocks in the current key-frame and the corresponding blocks in previous frames. A histogram of gradient (HOG) is extracted as a spatial feature, and a histogram of optical flow (HOF) is extracted as a temporal feature based on the fact that the diffusion direction of smoke is upward owing to thermal convection. Using these spatiotemporal features, a codebook and a BoF histogram are generated from training data. For smoke verification, a random forest classifier is built during the training phase by using the BoF histogram. The random forest with BoF histogram can increase the detection accuracy and allow smoke detection to be carried out in near real-time.

In this paper, we propose a novel human detection approach combining wavelet-based center symmetric LBP (WCS-LBP) with a cascade of random forests. To detect human regions, we first extract three types of WCS-LBP features from a scanning window of wavelet transformed sub-images to reduce the feature dimension. Then, the extracted WCS-LBP descriptors are applied to a cascade of random forests, which are ensembles of random decision trees. Using a cascade of random forests with WCS-LBP, human detection is performed in near real-time, and the detection accuracy is also increased, as compared to combinations of other features and classifiers. The proposed algorithm is successfully applied to various human and non-human images from the INRIA dataset, and it performs better than other related algorithms.

Fire-flame detection using a video camera is difficult because a flame has irregular characteristics, i.e., vague shapes and color patterns. Therefore, in this paper, we propose a novel fire-flame detection method using fuzzy finite automata (FFA) with probability density functions based on visual features, thereby providing a systemic approach to handling irregularity in computational systems and the ability to handle continuous spaces by combining the capabilities of automata with fuzzy logic. First, moving regions are detected via background subtraction, and the candidate flame regions are then identified by applying flame color models. In general, flame regions have a continuous irregular pattern; therefore, probability density functions are generated for the variation in intensity, wavelet energy, and motion orientation and applied to the FFA. The proposed algorithm is successfully applied to various fire/non-fire videos, and its detection performance is better than that of other methods.

This paper proposes a new fire-flame detection method using probabilistic membership function of visual features and Fuzzy Finite Automata (FFA). First, moving regions are detected by analyzing the background subtraction and candidate flame regions then identified by applying flame color models. Since flame regions generally have an irregular pattern continuously, membership functions of variance of intensity, wavelet energy and motion orientation are generate and applied to FFA. Since FFA combines the capabilities of automata with fuzzy logic, it not only provides a systemic approach to handle uncertainty in computational systems, but also can handle continuous spaces. The proposed algorithm is successfully applied to various fire videos and shows a better detection performance when compared with other methods.