Effective monitoring of essential road infrastructure is critical to ensure economic growth and community connectivity, as deterioration from climate change and intensive use can compromise safety and hamper emergency services. To address these critical issues, we propose optimized approaches for road damage detection using YOLOv5, YOLOv8, and YOLOv10 models, with fine-tuning and hyperparameter optimization to improve detection accuracy. Our approaches achieve good overall accuracy (in terms of F1-score) by combining models of different sizes and frameworks using ensemble methods. Moreover, to enhance the speed of inference, parallel processing techniques are used. In order to obtain fast detection performance, small models, such as YOLOv5-small and YOLOv8-nano were given priority in modeling to balance between detection accuracy and inference speed. In the context of the 2024 IEEE BigData Optimized Road Damage Detection Challenge (ORDDC’2024), our approach achieved an F1 score greater than 0.7 with an inference speed of 0.0432 seconds per image, meeting the ORDDC’s accuracy and inference speed goal.

Monitoring public facilities is essential for governments to establish safer, cleaner, and more resilient cities, ultimately benefiting all citizens. One promising approach for public facility surveillance is to employ visual machine-learning methods on street scenes. The potential development and advancements in such state-of-the-art methods require the availability of visual datasets annotated with several public facility defects (e.g., illegal dumping, graffiti, street potholes, and damaged traffic signs). Towards this, we introduce "StreetLens", a new dataset of videos capturing urban streets with plentiful annotations for vision-based public facility monitoring. This dataset includes four-and-a-half hours of videos recorded by smartphone cameras placed in moving vehicles in the suburbs of three different cities.

Wide area urban street monitoring is highly demanding in various smart city applications. Manual monitoring is both laborious and time-consuming, hence automatic vision-based monitoring is a more feasible alternative. An essential part of vision-based street monitoring is detecting and counting objects of interest. However, these tasks are not straightforward due to various challenges, i.e., noisy conditions in a real environment, such as occlusion and high illumination. This study investigates the impact of these challenges on object detection and counting accuracy, then provides an empirical study to address the challenges with respect to video-based street monitoring. The selected case study demonstrates detecting and counting of homeless encampments in Los Angeles streets using street-level videos collected from a moving vehicle.

Automated monitoring systems have been utilized for effective road maintenance in order to eliminate time-consuming and manual inspection by road administration employees. Image-based technology has grown as an important option since a variety of images, such as from surveillance cameras or on-dash cameras, are widely obtainable. Towards that solution, this paper introduces a deep learning-based approach for detecting and classifying road damages in diverse images collected from different countries. Our approach integrates individual models trained per country and a general model trained for all countries. Our approach were evaluated thoroughly using the 2022 IEEE BigData Crowdsensing-based Road Damage Detection Challenge (CRDDC) Datasets. Experimental results show that our approach achieved an F1 score of up to 0.73.

With the availability of massive amounts of visual data covering wide geographical regions, various image learning applications have emerged, including classifying the street cleanliness level, detecting forest fires or road hazards. Such applications share similar characteristics as they need to 1) detect specific objects or events (what), 2) associate the detected object with a location (where), and 3) know the time that the event happened (when). Advancements in image-based machine learning (ML) benefit these applications as they can automate the detection of objects of interest. Along with the edge computing (EC) paradigm, the processing cost is offloaded to the devices, hence reducing latency and communication cost. Moreover, sensors on the edge devices (e.g., GPS) enrich the collected data with metadata. However, a shortcoming of existing approaches is that they rely on pre-trained "static" models. Nonetheless, crowdsourced data at diverse locations can be leveraged to iteratively improve the robustness of a model. We refer to the aforementioned strategy as "spatial crowd-based learning".To showcase this class of applications, we present FloraVision, an end-to-end system that integrates ML, crowdsourcing, and EC to automate the detection, mapping, and exploration of California Native Plants. FloraVision implements a pipeline to collect and clean publicly available image data, train a lightweight MobileNet-based classification model, and then deploy the model on mobile devices. It leverages spatial crowd-based learning to iteratively evolve the initial model from crowdsourced data. Its mobile application facilitates detecting plants and mapping their geolocations. Finally, it allows end-users to submit ad hoc spatio-temporal nearest neighbor queries and visualizes the results in an augmented reality user interface. Although our application focuses on plants, several other applications follow similar architectural patterns.

For efficient road maintenance, an automated monitoring system is required to avoid laboriously and time-consuming manual inspection by road administration crews. One potential solution is to utilize image processing-based technologies, especially, as various sources of images have readily been available, e.g., surveillance cameras, in-vehicle cameras, or smartphones. Such image-based solutions enable detecting and classifying road damages. This paper introduces deep learning-based image analysis for road damage detection and classification. Our ensemble learning approaches with test time augmentation were thoroughly evaluated using the 2020 IEEE Big Data Global Road Damage Detection Challenge Dataset. Experimental results show that our approaches achieved an F1 score of up to 0.67, allowing us to win the Challenge.

Due to the prevalence of GPS-equipped cameras (e.g., smartphones and surveillance cameras), massive amounts of geo-tagged images capturing urban streets are increasingly being collected. Consequently, many smart city applications have emerged, relying on efficient image search. Such searches include spatial-visual queries in which spatial and visual properties are used in tandem to retrieve similar images to a given query image within a given geographical region. Towards this end, new index structures that organize images based on both spatial and visual properties are needed to efficiently execute such queries. Based on our observation that street images are typically similar in the same spatial locality, index structures for spatial-visual queries can be effectively built on a spatial index (i.e., R*-tree). Therefore, we propose a class of R*-tree indexes, particularly, by associating each node with two separate minimum bounding rectangles (MBR), one for spatial and the other for (dimension-reduced) visual properties of their contained images, and adapting the R*-tree optimization criteria to both property types.

Recognizing materials using image analysis is a classic problem. However, little research has been done with the images which have visual impediments such as noise, obstacle, or painting. This paper introduces the problem of recognizing covered materials which are distorted visually (e.g., materials covered by graffiti). We propose a set of approaches to solve this problem using a class of deep learning and transfer learning models, and evaluate our approaches empirically using a large-scale real world dataset that displays street scenes containing various materials which are covered with graffiti. Our experiments show that recognizing covered materials using the state-of-the-art approach for material recognition produced an mAP of 19%, while our proposed approach achieved an mAP of 60%. This evidently demonstrates that an approach for plain material recognition is not suitable for recognizing covered materials; hence this problem should be treated differently as in our proposed approaches.

Due to the ubiquity of GPS-equipped cameras such as smartphones, more photos are getting automatically tagged with camera locations (referred to as geo-tagged images) so large-scale geo-tagged image datasets are available on the Web. And a significant portion of online images such as travel and surveillance may not be meaningful without their location information. Thus, image localization for untagged images has been studied. However, the point camera location of an image might be quite different from the location of the scene depicted in the image (referred to as scene location) rendering image localization inaccurate. To address this problem, we propose a data-centric framework for image scene localization using a CNN-based classification in three steps. First, the framework provides two mechanisms for constructing a reference image dataset tagged with scene locations. Second, a spatial-visual classification approach organizes a dataset spatially using R-tree to generate a set of geographical regions tightly bounding the image scene locations. Then, we train a classifier based on the classes of images corresponding to the generated regions. Finally, to enhance the classification accuracy, we train a set of hierarchical classification models utilizing the spatial hierarchical structure of the R-tree where the trained models enable learning the visual features of images at different geographical granularities. We evaluate our framework using a geo-tagged image dataset obtained from Google Street View and demonstrate that the utilization of scene locations enables localizing images far more accurately as compared with camera location based localization.

With recent advances in technology, it is feasible to conveniently monitor urban roads using various cameras, such as surveillance cameras, in-vehicle cameras, or smartphones, and recognize their conditions by detecting specific types of road damages in order to plan maintenance resources efficiently based on the identified spots. This paper describes a road damage type detection and classification solution submitted to the IEEE BigData Cup Challenge 2018. Our solution is based on the state-of-the-art deep learning methods for an object detection task. In particular, our approach utilizes an object detection algorithm to detect various types of road damages by training the detector on different image examples categorized into a set of damages defined by Japan Road Association. We evaluated our approach thoroughly using different versions of trained models. Our experiments show that our approach was able to achieve an F1 score up to 0.62.

Sentiment analysis of disaster-related posts in social media can contribute to the situation awareness and better understanding of the dynamics of disaster events by identifying the polarity of sentiments expressed by the public. However, Even though many sentiment analysis techniques have been developed and available, there are still limitations in reliably using sentiment analysis since there is no dominantly accepted technique in disasters. Taking advantage of existing state-of-the-art sentiment classifiers, this paper proposes a novel framework for geo-spatial sentiment analysis of disaster-related social media data objects. Our framework addresses three types of challenges: the inaccuracy and discrepancy associated with various text and image sentiment classifiers, the geo-sentiment discrepancy among data objects in a local geographical area, and observing diverse sentiments from multimedia data objects (i.e., text and image). The extracted sentiments are aggregated geographically for the purpose of extracting more accurate local regional insights. For the evaluation of the framework, we explored Twitter and Flickr datasets at the time of Hurricane Sandy and Napa Earthquake and showed how our approach can provide a better understanding of disaster events.

When analyzing and processing videos, it has become increasingly important in many applications to also consider contextual information, in addition to the content. With the ubiquity of sensor-rich smartphones, acquiring a continuous stream of geo-spatial metadata that includes the location and orientation of a camera together with the video frames has become practical. However, no such detailed dataset is publicly available. In this paper we present an extensive geo-tagged video dataset named GeoUGV that has been collected as part of the MediaQ [3] and GeoVid [1] projects. The key features of the dataset are that each video file is accompanied by a metadata sequence of geo-tags consisting of GPS locations, compass directions, and spatial keywords at fine-grained intervals. The GeoUGV dataset has been collected by volunteer users and its statistics can be summarized as follows: 2,397 videos containing 208,976 video frames that are geo-tagged, collected by 289 users in more than 20 cities across the world over a period of 10 years (2007-2016). We hope that this dataset will be useful for researchers, scientists and practitioners alike in their work.

Recently spatial crowd sourcing was introduced as a natural extension to traditional crowd sourcing allowing for tasks to have a geospatial component, i.e., A task can only be performed if a worker is physically present at the location of the task. The problem of assigning spatial tasks to workers in a spatial crowd sourcing system can be formulated as a weighted bipartite b-matching graph problem that can be solved optimally by existing methods for the minimum cost maximum flow problem. However, these methods are still too complex to run repeatedly for an online system, especially when the number of incoming workers and tasks increases. Hence, we propose a class of approaches that utilizes an online partitioning method to reduce the problem space across a set of cloud servers to construct independent bipartite graphs and solve the assignment problem in parallel. Our approaches solve the spatial task assignment approximately but competitive to the exact solution. We experimentally verify that our approximate approaches outperform the centralized and Map Reduce version of the exact approach with acceptable accuracy and thus suitable for online spatial crowd sourcing at scale.

Automatic reconstruction of 3D models is attracting increasing attention in the multimedia community. Scene recovery from video sequences requires a selection of representative video frames. Most prior work adopted content-based techniques to automate key frame extraction. However, these methods take no frame geo-information into consideration and are still compute-intensive. Here we propose a new approach for key frame selection based on the geographic properties of videos. Currently, an increasing number of user-generated videos (UGVs) are collected — a trend that is driven by the ubiquitous availability of smartphones. Additionally, it has become easy to continuously acquire and fuse various sensor data (e.g., geo-spatial metadata) with video to create geo-tagged mobile videos. Our novel technique utilizes these underlying geo-metadata to select the most representative frames. Specifically, a key frame subset with minimal spatial coverage gain difference is extracted by incorporating a manifold structure into reproducing a kernel Hilbert space to analyze the spatial relationship among the frames. Our experimental results illustrate that the execution time of the 3D reconstruction is shortened while the model quality is preserved.