Source-free domain adaptive semantic segmentation aims at adapting a model trained on the source domain to the target domain without requiring access to the source data. Self-training has emerged as a leading approach to address this challenging problem. However, without robust denoising mechanisms to reduce the noise in pseudo labels, it still easily fall into biased estimates. Most existing methods address this issue by introducing novel architectures, but often at the cost of increased model complexity or reliance on additional input modalities. Different from previous studies, this article introduces UniSFDA , a unified multi-stage self-training framework that integrates cross-model transfer learning, uncertainty-aware pseudo label fusion, and intra-domain style augmentation, thereby enhancing both segmentation accuracy and test-time efficiency. Our proposed framework offers exceptional flexibility, with each component being independent and ready to be integrated into any existing self-training framework. Additionally, we investigate the performance of various representative segmentation models, including DeepLabv2, SegFormer, DFormer, and ViT-Adapter, within our framework. It is worth noting that UniSFDA is model-agnostic, allowing both source and target networks to be instantiated with arbitrary segmentation architectures, and thus readily benefiting from future advances in segmentation models. Experiments on the GTA5 \(\rightarrow\) Cityscapes and SYNTHIA \(\rightarrow\) Cityscapes benchmarks demonstrate the effectiveness of our framework. With DeepLabv2 (SegFormer) as the source model, UniSFDA establishes new state-of-the-art performance, achieving mIoU scores of 61.8% (65.4%) and 57.9% (59.6%) on the two benchmarks, respectively.

This paper proposes a two-stage multi-lane unsignalized intersection cooperative control strategy based on mixed platoons (MICS-PF) in the mixed traffic environment of connected automated vehicles (CAVs) and connected human-driven vehicles (CHVs). In the first stage, we propose a 0–1 integer programming model based on vehicles’ intersection movements and potential car-following costs to organize vehicles into mixed platoons led by CAVs. Meanwhile, we design a rule-based algorithm to coordinate the relative distribution of platoons. Then, the lane-changing and car-following models are integrated to ensure that the platoons organization plan can be completed in the limited functional zone. In the second stage, we design an improved minimum covering clique algorithm to generate a conflict-free passing order of platoons by considering the platoon-level conflict relation and the potential yield delay. The proposed strategy couples the process of organizing vehicles into mixed platoons to pass through the intersection. The optimal platoon organization enables CHVs to receive accurate guidance when entering the intersection and avoids the disorderly lane change at the upstream road sections. The mixed platoon control also allows vehicles to avoid stop-and-go, improve the capacity of approach roads, and prevent the intersection capacity from being restricted by the inefficient traffic at the approach road. The conflict-free coordination method of mixed platoons optimizes the throughput of intersections and broadens the usage conditions of unsignalized intersections. A series of simulation experiments verify the effectiveness of the strategy, and the applicable boundary conditions are obtained.

Source-free domain adaptive semantic segmentation has gained increasing attention recently. It eases the requirement of full access to the source domain by transferring knowledge only from a well-trained source model. However, reducing the uncertainty of the target pseudo labels becomes inevitably more challenging without the supervision of the labeled source data. In this work, we propose a novel asymmetric two-stream architecture that learns more robustly from noisy pseudo labels. Our approach simultaneously conducts dual-head pseudo label denoising and cross-modal consistency regularization. Towards the former, we introduce a multimodal auxiliary network during training (and discard it during inference), which effectively enhances the pseudo labels' correctness by leveraging the guidance from the depth information. Towards the latter, we enforce a new cross-modal pixel-wise consistency between the predictions of the two streams, encouraging our model to behave smoothly for both modality variance and image perturbations. It serves as an effective regularization to further reduce the impact of the inaccurate pseudo labels in source-free unsupervised domain adaptation. Experiments on GTA5 → Cityscapes and SYNTHIA → Cityscapes benchmarks demonstrate the superiority of our proposed method, obtaining the new state-of-the-art mIoU of 57.7% and 57.5%, respectively.

When traffic density on urban roads is relatively large, it is difficult for a large number of O-vehicles (O-vehicles) in front of emergency vehicles (E-vehicles) to give way in a short time, resulting in a delay of E-vehicles. This study aimed to design an automatic way-giving system based on vehicle-to-vehicle communication (V2V) in the internet of vehicles (IoV). Also, a control algorithm was proposed for the platoon to give way to the E-vehicle in the minimum time, the factors such as the traffic density, number of vehicles in the platoon, the initial state, and speed limit were considered in this paper. The algorithm firstly calculated the dynamic adjustment trajectory for the platoon to complete the lane change, the adjustment time, and the minimum distance between the E-vehicle and the platoon at the beginning of lane change, to guide the platoon while changing the lane. A simulator utility was developed to interface with the microscopic traffic simulation package Simulation of Urban Mobility (SUMO) by using the Python program. Lastly, a two-lane road segment was simulated to evaluate the effectiveness of the control algorithm of automatic way-giving system, and its performance is compared with conventional vehicle operation. The results demonstrated that the operating efficiency of E-vehicles can be improved by 12.65% and the operating efficiency of O-vehicles can only be decreased by 3.25% by the automatic way-giving system. Besides, the notification distance is a key factor in affecting the operating efficiency of E-vehicles and O-vehicles.

Automatic inference of missing road attributes (e.g., road type and speed limit) for enriching digital maps has attracted significant research attention in recent years. A number of machine learning based approaches have been proposed to detect road attributes from GPS traces, dash-cam videos, or satellite images. However, existing solutions mostly focus on a single modality without modeling the correlations among multiple data sources. To bridge the gap, we present a multimodal road attribute detection method, which improves the robustness by performing pixel-level fusion of crowdsourced GPS traces and satellite images. A GPS trace is usually given by a sequence of location, bearing, and speed. To align it with satellite imagery in the spatial domain, we render GPS traces into a sequence of multi-channel images that simultaneously capture the global distribution of the GPS points, the local distribution of vehicles' moving directions and speeds, and their temporal changes over time, at each pixel. Unlike previous GPS based road feature extraction methods, our proposed GPS rendering does not require map matching in the data preprocessing step. Moreover, our multimodal solution addresses single-modal challenges such as occlusions in satellite images and data sparsity in GPS traces by learning the pixel-wise correspondences among different data sources. Extensive experiments have been conducted on two real-world datasets in Singapore and Jakarta. Compared with previous work, our method is able to improve the detection accuracy on road attributes by a large margin.

Map matching has long been a fundamental yet challenging problem. However, there are currently only a few public small-scale map matching benchmark datasets. Both the GPS trajectories and the road network in the existing map matching datasets are represented by location only, which cannot support the development of data-driven and semantic-enriched map matching algorithms that have increasingly emerged in recent years. To bridge the gap, we present the first large-scale attribute-rich map matching benchmark dataset covering two cities in Southeast Asia (i.e., Singapore and Jakarta). Our GPS trajectories contain rich contextual information including the accuracy level, bearing, speed, and transport mode in addition to the latitude and longitude geo-coordinates. The underlying road network is a snapshot of the OpenStreetMap where roads are associated with rich attributes such as road type, speed limit, etc. To ensure the quality of our dataset, the annotation of the map-matched routes has been conducted by a team of professional map operators. Analysis on our dataset provides new insights into the challenges and opportunities in map matching algorithms.

The quality of a digital map is of utmost importance for geo-aware services. However, maintaining an accurate and up-to-date map is a highly challenging task that usually involves a substantial amount of manual work. To reduce the manual efforts, methods have been proposed to automatically derive road attributes by mining GPS traces. However, previous methods always modeled each road attribute separately based on intuitive hand-crafted features extracted from GPS traces. This observation motivates us to propose a machine learning based method to learn joint features not only from GPS traces but also from map data. To model the relations among the target road attributes, we extract low-level shared feature embeddings via multi-task learning, while still being able to generate task-specific fused representations by applying attention-based feature fusion. To model the relations between the target road attributes and other contextual information that is available from a digital map, we propose to leverage map tiles at road centers as visual features that capture the information of the surrounding geographic objects around the roads. We perform extensive experiments on the OpenStreetMap where state-of-the-art classification accuracy has been obtained compared to existing road attribute detection approaches.

Locating nearest moving objects in real-time is a vital problem that the ride-hailing industry needs to address. For instance, when a passenger makes a booking, the service provider, such as Grab or Uber, needs to locate the K nearest drivers for the given pickup location in case the closest driver is not optimal for this booking request. This poses two main challenges: firstly, massive frequent write operations are needed to track the objects' current locations. As drivers can move as fast as 25 meters per second in developed countries like Singapore, it is therefore important to update drivers' locations at a second, if not millisecond, granularity. Secondly, a K-nearest neighbour (kNN) query poses tremendous challenges, compared to a simple Get query, in a key-value data store such as Redis. This paper presents Sextant, a scalable in-memory spatial data store tailored for kNN searches. Sextant is decentralized, scalable, reliable, efficient and highly available. It has been supporting Grab's daily flow with no downtime for more than one year, with write QPS (query per second) and kNN query QPS approaching millions.

Videos recorded with current mobile devices are increasingly geotagged at fine granularity and used in various location- based applications and services. However, raw sensor data collected is often noisy, resulting in subsequent inaccurate geospatial analysis. In this study, we focus on the challenging correction of compass readings and present an automatic approach to reduce these metadata errors. Given the small geo-distance between consecutive video frames, image-based localization does not work due to the high ambiguity in the depth reconstruction of the scene. As an alternative, we collect geographic context from OpenStreetMap and estimate the absolute viewing direction by comparing the image scene to world projections obtained with different external camera parameters. To design a comprehensive model, we further incorporate smooth approximation and feature-based rotation estimation when formulating the error terms. Experimental results show that our proposed pyramid-based method outperforms its competitors and reduces orientation errors by an average of 58.8%. Hence, for downstream applications, improved results can be obtained with these more accurate geo-metadata. To illustrate, we present the performance gain in landmark retrieval and tag suggestion by utilizing the accuracy-enhanced geo-metadata.

Real-time map matching is a fundamental but challenging problem with various applications in Geographic Information Systems (GIS), Intelligent Transportation Systems (ITS) and beyond. It aims to align a sequence of measured latitude/longitude positions with the road network on a digital map in real-time. There exist a number of statistical matching approaches that unfortunately either process trajectory data offline or provide an online solution without an infimum analysis. Here we propose a novel statistics-based online map matching algorithm called Eddy with a solid error-and delay-bound analysis. More specifically, Eddy employs a Hidden Markov Model (HMM) to represent the spatio-temporal data as state chains, which elucidates the road network's topology, observation noises and their underlying relations. After modeling, we shape the decoding phase as a ski-rental problem, and an improved online-version Viterbi decoding algorithm is proposed to find the most likely sequence of hidden states (road routes) in real-time. We reduce the candidate routes search range during the decoding for efficiency reasons. Moreover, our deterministic decoder trades off latency for expected accuracy dynamically, without having to choose a fixed window size beforehand. We also provide the competitive analysis and the proof that our online algorithm is error-bounded (with a competitive ratio of 2) and latency-bounded. Our experimental results show that the proposed algorithm outperforms widely used existing approaches on both accuracy and latency.

Automatic reconstruction of 3D models is attracting increasing attention in the multimedia community. Scene recovery from video sequences requires a selection of representative video frames. Most prior work adopted content-based techniques to automate key frame extraction. However, these methods take no frame geo-information into consideration and are still compute-intensive. Here we propose a new approach for key frame selection based on the geographic properties of videos. Currently, an increasing number of user-generated videos (UGVs) are collected — a trend that is driven by the ubiquitous availability of smartphones. Additionally, it has become easy to continuously acquire and fuse various sensor data (e.g., geo-spatial metadata) with video to create geo-tagged mobile videos. Our novel technique utilizes these underlying geo-metadata to select the most representative frames. Specifically, a key frame subset with minimal spatial coverage gain difference is extracted by incorporating a manifold structure into reproducing a kernel Hilbert space to analyze the spatial relationship among the frames. Our experimental results illustrate that the execution time of the 3D reconstruction is shortened while the model quality is preserved.

MediaQ is a novel online media management system to collect, organize, share, and search mobile multimedia contents using automatically tagged geospatial metadata. User-generated-videos can be uploaded to the MediaQ from users' smartphones, iPhone and Android, and displayed accurately on a map interface according to their automatically sensed geospatial and other metadata. The MediaQ system provides the following distinct features. First, individual frames of videos (or any meaningful video segments) are automatically annotated by objective metadata which capture four dimensions in the real world: the capture time (when), the camera location and viewing direction (where), several key-words (what) and people (who). We term this data W4-metadata and they are obtained by utilizing camera sensors, geospatial and computer vision techniques. Second, a new approach of collecting multimedia data from the public has been implemented using spatial crowdsourcing, which allows media content to be collected in a coordinated manner for a specific purpose. Lastly, flexible video search features are implemented using W4 metadata, such as directional queries for selecting multimedia with a specific viewing direction.

Due to technological advances and the popularity of camera sensors, it is now straightforward for users to capture and share videos. A large number of geo-tagged photos and videos have been accumulating continuously on the web, posing a challenging problem for mining this type of media data. In one application scenario, users might desire to know what the Points of Interest (POI) are which contain important objects or places in a video. Existing solutions attempt to examine the content of the videos and recognize objects and events. This is typically time-consuming and computationally expensive and the results can be uneven. Therefore these methods face challenges when applied to large video repositories. We propose a novel technique that leverages sensor-generated meta-data (camera locations and viewing directions) which are automatically acquired as continuous streams together with the video frames. Existing smartphones can easily accommodate such integrated recording tasks. By considering a collective set of videos and leveraging the acquired auxiliary meta-data, our approach is able to detect interesting regions and objects (POIs) and their distances from the camera positions in a fully automated way. Because of its computational efficiency, the proposed method scales well and our experiments show very promising results.

Similar to positioning data, camera orientation information has become a powerful contextual feature utilized by a number of GIS and social media applications. Such auxiliary information facilitates higher-level semantic analysis and management of video assets in such applications, e.g., video summarization and video indexing systems. However, it is problematic that raw sensor data collected from current mobile devices is often not accurate enough for subsequent geospatial analysis. To date, an effective orientation data correction system for mobile video content has been lacking. Here we present a content-based approach that improves the accuracy of noisy orientation sensor measurements generated by mobile devices in conjunction with video acquisition. Our preliminary experimental results demonstrate significant accuracy enhancements which benefit upstream sensor-aided GIS applications to access video content more precisely.

In addition to positioning data, other sensor information -- such as orientation data, have become a useful and powerful contextual feature. Such auxiliary information can facilitate higher-level semantic description inferences in many multimedia applications, e.g., video tagging and video summarization. However, sensor data collected from current mobile devices is often not accurate enough for upstream multimedia analysis. An effective orientation data correction system for mobile multimedia content has been an elusive goal so far. Here we present a system, termed Oscor, which aims to improve the accuracy of noisy orientation sensor measurements generated by mobile devices during image and video recording. We provide a user-friendly camera interface to facilitate the gathering of additional information, which enables the correction process on the server-side. Geographic field-of-view (FOV) visualizations based on the original and corrected sensor data help users understand the corrected contextual information and how the erroneous data possibly may affect further processes.

Video associated positioning data has become a useful contextual feature to facilitate analysis and management of media assets in GIS and social media applications. Moreover, with today's sensor-equipped mobile devices, the location of a camera can be continuously acquired in conjunction with the captured video stream without much difficulty. However, most sensor information collected from mobile devices is not highly accurate due to two main reasons: (a) the varying surrounding environmental conditions during data acquisition, and (b) the use of low-cost, consumer-grade sensors in current mobile devices. In this paper, we enhance the noisy positioning data generated by smartphones during video recording by analyzing typical error patterns for real collected data and introducing two robust algorithms, based on Kalman filtering and weighted linear least square regression, respectively. Our experimental results demonstrate significant benefits of our methods, which help upstream sensor-aided applications to access media content precisely.

Camera motion information facilitates higher-level semantic description inference in many video applications, e.g., video retrieval. However, an efficient and accurate methodology for annotating videos with camera motion information is still an elusive goal. In our recent work we have investigated the fusion of captured video with a continuous stream of sensor meta-data. For these so-called sensor-rich videos we present a system, called Motch, which precisely partitions a video document into subshots, automatically characterizes the camera motions and provides video subshot browsing based on an interactive, map-based interface. Moreover, the system computes and presents motion type statistics for each video in real time and renders different subshots distinctively on the map synchronously with the video playback.

User-generated videos have become increasingly popular in recent years. Due to advances in camera technology it is now very easy and convenient to record videos with mobile devices, such as smartphones. Here we consider an application where users collect and share a large set of videos that are related to a geographic area, say a city. Such a repository can be a great source of information for prospective tourists when they plan to visit a city and would like to get a preview of its main areas. The challenge that we address is how to automatically create a preview video summary from a large set of source videos.

Camera motion information is one aspect that helps to infer higher-level semantic descriptions in many video applications, e.g., in video retrieval. However, an efficient methodology for annotating camera motion information is still an elusive goal. Here we propose and present a novel and efficient approach for the task of partitioning a video document into sub-shots and characterizing their camera motion. By leveraging location (GPS) and digital compass data, which are available from most current smartphone handsets, we exploit the geographical sensor information to detect transitions between two sub-shots based on the variations of both the camera location and the shooting direction. The advantage of our method lies in its considerable accuracy. Additionally, the computational efficiency of our scheme enables it to be deployed on mobile devices and to process videos while recording. We utilize this capability to show how the HEX motion estimation algorithm in the H.264/AVC encoder can be simplified with the aid of our camera motion information. Our experimental results show that we can reduce the computation of the HEX algorithm by up to 50% while achieving comparable video quality.

To present user-generated videos that relate to geographic areas for easy access and browsing it is often natural to use maps as interfaces. A common approach is to place thumbnail images of video keyframes in appropriate locations. Here we consider the challenge of determining which keyframes to select and where to place them on the map. Our proposed technique leverages sensor-collected meta-data which are automatically acquired as a continuous stream together with the video. Our approach is able to detect interesting regions and objects (hotspots) and their distances from the camera in a fully automated way. Meaningful keyframes are adaptively selected based on the popularity of the hotspots. Our experiments show very promising results and demonstrate excellent utility for the users.

Result presentations from searches into video repositories is still a challenging problem. Current systems usually display a ranked list that shows the first frame of each video. Users then explore the videos one-by-one. In our recent work we have investigated the fusion of captured video with a continuous stream of sensor meta-data. These so-called sensor-rich videos can conveniently be captured with today's smartphones. Importantly, the recorded sensor-data streams enable processing and result resentation in novel and useful ways.