Cities worldwide use camera systems that collect and store large amounts of images, which are used to study vehicle traffic conditions, facilitating traffic management authorities’ decision-making. Typically, the inspection of those images is performed manually, which prevents extracting relevant information in a timely manner. There is a lack of platforms to collect and analyze key data from traffic videos in an automatic and speedy way. Computer vision can be used in combination with parallel distributed systems to provide city authorities tools for automatic and fast processing of stored videos to determine the most significant driving patterns that cause traffic accidents while allowing to measure the traffic density. We use a Convolutional Neural Network (CNN) to detect vehicles captured by traffic cameras, which are then tracked using an algorithm that we designed, based on multi-tracking Kalman filters. To speed up analysis, we propose a low-cost distributed infrastructure based on Hadoop and Spark frameworks for data processing: videos are equally divided and distributed to multicore CPU nodes for analysis. However, splitting up videos could generate inaccuracies in vehicle counting, which were avoided through the use of an algorithm that we present in this work. We found that it is possible to rapidly determine traffic densities, identify dangerous driving maneuvers, and detect accidents with high accuracy by using low-cost commodity cluster computing. There is a lack of computing platforms to collect and analyze key data from traffic videos in an automatic and speedy way. Computer vision can be used in combination with parallel distributed systems to provide city authorities tools for automatic and fast processing of stored videos to determine the most significant driving patterns that cause traffic accidents while allowing to measure the traffic density. This study explores the integration of different tools such as parallel data processing, deep learning, and probabilistic models. We present an approach based on Convolutional Neural Network (CNN) and Kalman filters to detect and track vehicles captured by traffic cameras. To speed up analysis, we propose and evaluate a low-cost distributed infrastructure based on Hadoop and Spark frameworks and comprised of multicore CPU nodes for data processing. Finally, we present an algorithm to allow vehicle counting while avoiding inaccuracies generated when videos are split to be distributed for analysis. We found that it is possible to rapidly determine traffic densities, identify dangerous driving maneuvers, and detect accidents with high accuracy by using low-cost commodity cluster computing.

Summary We present a detailed examination of time agreement characteristics for nodes within extreme‐scale parallel computers. Using a software tool we introduce in this paper, we quantify attributes of clock skew among nodes in three representative high‐performance computers sited at three national laboratories. Our measurements detail the statistical properties of time agreement among nodes and how time agreement drifts over typical application execution durations. We discuss the implications of our measurements, why the current state of the field is inadequate, and propose strategies to address observed shortcomings.

Next-generation applications increasingly rely on in situ analytics to guide computation, reduce the amount of I/O performed, and perform other important tasks. Scheduling where and when to run analytics is challenging, however. This paper quantifies the costs and benefits of different approaches to scheduling applications and analytics on nodes in large-scale applications, including space sharing, uncoordinated time sharing, and gang scheduled time sharing.

Scientific workloads running on current extreme-scale systems routinely generate tremendous volumes of data for postprocessing. This data movement has become a serious issue due to its energy cost and the fact that I/O bandwidths have not kept pace with data generation rates. In situ analytics is an increasingly popular alternative in which post-simulation processing is embedded into an application, running as part of the same MPI job. This can reduce data movement costs but introduces a new potential source of interference for the application. Using a validated simulation-based approach, we investigate how best to mitigate the interference from time-shared in situ tasks for a number of key extreme-scale workloads. This paper makes a number of contributions. First, we show that the independent scheduling of in situ analytics tasks can significantly degradation application performance, with slowdowns exceeding 1000%. Second, we demonstrate that the degree of synchronization found in many modern collective algorithms is sufficient to significantly reduce the overheads of this interference to less than 10% in most cases. Finally, we show that many applications already frequently invoke collective operations that use these synchronizing MPI algorithms. Therefore, the syncronization introduced by these MPI collective algorithms can be leveraged to efficiently schedule analytics tasks with minimal changes to existing applications. This paper provides critical analysis and guidance for MPI users and developers on the importance of scheduling in situ analytics tasks. It shows the degree of synchronization needed to mitigate the performance impacts of these time-shared coupled codes and demonstrates how that synchronization can be realized in an extreme-scale environment using modern collective algorithms.

Next-generation systems face a wide range of new potential sources of application interference, including resilience actions, system software adaptation, and in situ analytics programs. In this paper, we present a new model for analyzing the performance of bulk-synchronous HPC applications based on the use of extreme value theory. After validating this model against both synthetic and real applications, the paper then uses both simulation and modeling techniques to profile next-generation interference sources and characterize their behavior and performance impact on a selection of HPC benchmarks, mini-applications, and applications. Lastly, this work shows how the model can be used to understand how current interference mitigation techniques in multi-processors work.