Detecting dangerous traffic agents in videos captured by a dashboard camera (dashcam) mounted on vehicles is essential to ensure safe navigation in complex driving environments. Crash-related videos are corner cases in driving-related big data, and pre-crash processes are transient and complex. Besides, risky and non-risky traffic agents can be similar in their appearance. These make the localization of risky traffic agents in driving videos particularly challenging. In addressing the challenges, this paper proposes an attention-guided multistream feature fusion network (AM-Net) to localize dangerous traffic agents from dashcam videos ahead of potential accidents. Two Gated Recurrent Unit (GRU) networks use object bounding box and optical flow features extracted from consecutive video frames to capture spatio-temporal cues for distinguishing risky traffic agents. An attention module, coupled with the GRUs, learns to identify traffic agents that are relevant to a crash. Fusing the two streams of global and object-level features, AM-Net predicts the riskiness scores of traffic agents in the video. This paper also introduces a new benchmark dataset called Risky Object Localization (ROL), which contains spatial, temporal, and categorical annotations of the crash, object, and scene-level attributes. The proposed AM-Net achieves a promising performance of 85.59% AUC on the ROL dataset. Additionally, the AM-Net outperforms the current state-of-the-art for video anomaly detection by 3.5% AUC on the public DoTA dataset. A thorough ablation study further reveals AM-Net’s merits by assessing the contributions of its functional constituents.

Future action anticipation aims to infer future actions from the observation of a small set of past video frames. In this paper, we propose a novel Jointly-learnt Action Anticipation Network (J-AAN) via Self-Knowledge Distillation (Self-KD) and cycle consistency for future action anticipation. In contrast to the current state-of-the-art methods which anticipate the future actions either directly or recursively, our proposed J-AAN anticipates the future actions jointly in both direct and recursive ways. However, when dealing with future action anticipation, one important challenge to address is the future’s uncertainty since multiple action sequences may come from or be followed by the same action. Training an action anticipation model with one-hot-encoded hard labels that assign zero probabilities to incorrect yet semantically similar actions may not handle the uncertain future. To address this challenge, we design a Self-KD mechanism to train our J-AAN, where the J-AAN gradually distills its own knowledge during the training to soften the hard labels to model the uncertainty on future action anticipation. Furthermore, we design a forward and backward action anticipation framework with our proposed J-AAN based on a cyclic consistency constraint. The forward J-AAN anticipates the future actions from the observed past actions, and the backward J-AAN verifies the anticipation of the forward J-AAN by anticipating the past actions from the anticipated future actions. The proposed method outperforms all the latest state-of-the-art action anticipation methods on the Breakfast, 50Salads, and EPIC-Kitchens-55 datasets. This project will be publicly available onhttps://github.com/MoniruzzamanMd/J-AAN.

Over 600 000 bridges in the U.S. must be inspected every two years to identify flaws, defects, or potential problems that may need follow-up maintenance. Bridge inspection has adopted unmanned aerial vehicles (or drones) for improving safety, efficiency, and cost-effectiveness. Although drones can operate in an autonomous mode, keeping inspectors in the loop is critical for complex tasks in bridge inspection. Therefore, inspectors need to develop the skill and confidence to operate drones in their jobs. This article presents the design and development of a virtual-reality-based training and assessment system for inspectors assisted by a drone in bridge inspection. The system is composed of four integrated modules: a simulated bridge inspection developed in Unity, an interface that allows a trainee to operate the drone in simulation using a remote controller, data monitoring and analysis to provide real-time in-task feedback to trainees to assist their learning, and a post-study assessment supporting personalized training. This article also conducts a proof-of-concept pilot study to illustrate the functionality of this system. The study demonstrated that the training and assessment system for bridge inspection with an assistant drone, as a tool for the early-stage training, can objectively identify the training needs of individuals in detail and, further, help them develop the skill and confidence in collaborating with a drone in bridge inspection. The system has built a modeling and analysis platform for exploring advanced solutions to the human–drone cooperative inspection of civil infrastructure.

The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.

We Introduce a simple yet effective network that embeds a novel Discriminative Feature Pooling (DFP) mechanism and a novel Video Segment Attention Model (VSAM), for video-based human action recognition from both trimmed and untrimmed videos. Our DFP module introduces an attentional pooling mechanism for 3D Convolutional Neural Networks that attentionally pools 3D convolutional feature maps to emphasize the most critical spatial, temporal, and channel-wise features related to the actions within a video segment, while our VSAM ensembles these most critical features from all video segments and learns (1) class-specific attention weights to classify the video segments into the corresponding action categories, and (2) class-agnostic attention weights to rank the video segments based on their relevance to the action class. Our action recognition network can be trained from both trimmed videos in a fully-supervised way and untrimmed videos in a weakly-supervised way. For untrimmed videos with weak labels, our network learns attention weights without the requirement of precise temporal annotations of action occurrences in videos. Evaluated on the untrimmed video datasets of THUMOS14 and ActivityNet1.2, and trimmed video datasets of HMDB51, UCF101, and HOLLYWOOD2, our network achieves promising performance, compared to the latest state-of-the-art method. The implementation code is available athttps://github.com/MoniruzzamanMd/DFP-VSAM-Networks.

Manufacturing activities dominate the energy consumption and greenhouse emissions of the industrial sector. With the increasing concerns of greenhouse gas (GHG) emissions and climate change in recent years, the significance of the performance in terms of sustainability of manufacturing has been gradually recognized by both academia and industry. Various researches have been implemented to analyze, model, and reduce the energy consumption of manufacturing activities toward sustainable manufacturing. In a typical manufacturing system with multiple machines and buffers, the state of a certain machine is not only determined by the machine itself, but also the states of the adjacent machines and buffers. Therefore, machines may be in idle states due to nonincoming part from the upstream section of the manufacturing system or noncapacity to hold the delivered part to the downstream section of the manufacturing system. Those idle machines consume energy without production if there is no appropriate energy control strategy. In this article, we focus on the reduction of the energy waste for those idle machines in a typical multi-machine and multi-buffer manufacturing system. A Gaussian mixture model (GMM) integrated neural network is proposed to predict the duration of the idle periods for the idle machines, during which optimal energy control action can be identified and implemented under the constraint of production throughput of the manufacturing system. A manufacturing system simulator is built to provide the training dataset including the information, such as production throughput, energy consumption, buffer content, and failure rate, to the proposed neural network. A numerical case study for a five-machine-and-four-buffer manufacturing system is conducted to validate the effectiveness of the proposed prediction model in terms of the energy waste reduction for the idle machines.

The state-of-the-art of fully-supervised methods for temporal action localization from untrimmed videos has achieved impressive results. Yet, it remains unsatisfactory for the weakly-supervised temporal action localization, where only video-level action labels are given without the timestamp annotation on when the actions occur. The main reason comes from that, the weakly-supervised networks only focus on the highly discriminative frames, but there are some ambiguous frames in both background and action classes. The ambiguous frames in background class are very similar to the real actions, which may be treated as target actions and result in false positives. On the other hand, the ambiguous frames in action class which possibly contain action instances, are prone to be false negatives by the weakly-supervised networks and result in a coarse localization. To solve these problems, we introduce a novel weakly-supervised Action Completeness Modeling with Background Aware Networks (ACM-BANets). Our Background Aware Network (BANet) contains a weight-sharing two-branch architecture, with an action guided Background aware Temporal Attention Module (B-TAM) and an asymmetrical training strategy, to suppress both highly discriminative and ambiguous background frames to remove the false positives. Our action completeness modeling contains multiple BANets, and the BANets are forced to discover different but complementary action instances to completely localize the action instances in both highly discriminative and ambiguous action frames. In the i-th iteration, the i-th BANet discovers the discriminative features, which are then erased from the feature map. The partially-erased feature map is fed into the (i+1)-th BANet of the next iteration to force this BANet to discover discriminative features different from the i-th BANet. Evaluated on two challenging untrimmed video datasets, THUMOS14 and ActivityNet1.3, our approach outperforms all the current weakly-supervised methods for temporal action localization.

During the past ten years, on average there were near 16.5 thousands of hazardous materials (hazmat) transport incidents per year resulting in $82 millions of damages. Prompt, accurate, objective assessment on hazmat incidents is important for the first-responders to take appropriate actions timely, which will reduce the damage of hazmat incidents and protect the safety of people and the environment. Therefore, one of the most important steps is to automatically detect transport incidents, such as fire and traffic accidents. In this paper, we introduce a simple and yet effective framework that integrates the convolutional feature maps of deep Convolutional Neural Network with a spatial attention mechanism for fire and traffic accident scene classification. Our spatial attention model learns to highlight the most discriminative convolutional features, which is related to the regions of interest in the input image. We train our network in a weakly supervised way. In other words, without the requirement of precise bounding box annotating the exact location of fire or traffic accidents in the image, our network can be learned from the only image-level label. In addition to the image-based traffic scene classification, the model is also applied on a set of collected videos for real-world applications. The proposed model, a simple end-to-end architecture, achieves promising performance on fire scene classification from images, and traffic accident scene classification from both images and videos.

In the acquisition of acknowledged system of systems (SoSs), the SoS manager attempts to reach a coalition agreement with each of selected constituent systems. When the systems have self-interests that are incompatible with the SoS's goal, contract negotiation is a way of resolving conflicts. Complexity of the acknowledged SoS acquisition makes the negotiation a hard decision process for constituent systems. To the SoS manager, the negotiation with consistent systems is difficult too, considering that the behavior and strategies of constituent systems in negotiation impact the ability and timeliness of a group to agree on an SoS architecture, and so the overall mission effectiveness of the SoS. Motivated by these, this paper develops a contract negotiation model for constituent systems in the acknowledged SoS acquisition. The model consists of: 1) a protocol for negotiating multiple interdependent issues of multiple items over multiple stages and 2) a decision framework of constituent systems. The model is not only a decision support for constituent systems but also a tool with which the SoS manager can better understand constituent systems.

Studying the behavior of fruit flies that mimic normal animal motivations can inform us about the molecular mechanisms and biochemical pathways. We build a glass chamber to house flies and record their behaviors in video frame sequences. Due to the challenges of low image contrast, small object size and fast object motion, we propose an adaptive Local Binary Pattern (LBP) feature to detect flies and develop a cascaded data association approach with fine-to-coarse gating region control to track flies in the spatio-temporal domain. Our approach is validated on two long video sequences with very good performance, showing its potential to enable automated characterization of biological processes.