The increasing deployment of deep neural networks (DNNs) in cyber-physical systems (CPS) enhances perception fidelity, but imposes substantial computational demands on execution platforms, posing challenges to real-time control deadlines. Traditional distributed CPS architectures typically favor on-device inference to avoid network variability and contention-induced delays on remote platforms. However, this design choice places significant energy and computational demands on the local hardware. In this work, we revisit the assumption that cloud-based inference is intrinsically unsuitable for latency-sensitive control tasks. We demonstrate that, when provisioned with high-throughput compute resources, cloud platforms can effectively amortize network and queueing delays, enabling them to match or surpass on-device performance for real-time decision-making. Specifically, we develop a formal analytical model that characterizes distributed inference latency as a function of the sensing frequency, platform throughput, network delay, and task-specific safety constraints. We instantiate this model in the context of emergency braking for autonomous driving and validate it through extensive simulations using real-time vehicular dynamics. Our empirical results identify concrete conditions under which cloud-based inference adheres to safety margins more reliably than its on-device counterpart. These findings challenge prevailing design strategies and suggest that the cloud is not merely a feasible option, but often the preferred inference location for distributed CPS architectures. In this light, the cloud is not as distant as traditionally perceived; in fact, it is closer than it appears.

Cellular-vehicle-to-everything (C-V2X) technology has been increasingly adopted by the research community, automotive industry, and government agencies as the next key technology to enhance transportation safety and efficiency. Recent years have also witnessed emerging C-V2X-based applications, connecting sensors on vehicle (V2V), from infrastructure (V2I), and carried by pedestrians (V2P), to enable novel capabilities such as cooperative perception, sustainable transportation, and remote operations. While researchers have made successful strides in simulating these promising applications, the disconnect with real-world C-V2X performance often renders ungrounded assumptions, resulting in huge barriers towards deployment. In this poster, we conduct an application driven C-V2X network measurement using commercial off-the-shelf standard compliant C-V2X radios. Emulating the traffic patterns of popular C-V2X applications, we investigate the gap between the demand and reality.

Cellular-vehicle-to-everything (C-V2X) technology has been increasingly adopted by the research community, automotive industry, and government agencies as the next key technology to enhance transportation safety and efficiency. Recent years have also witnessed emerging C-V2X-based applications, connecting sensors on vehicles (V2V), from the infrastructure (V2I), and carried by pedestrians (V2P), to enable novel capabilities such as cooperative perception, sustainable transportation, and remote operations. While researchers have made successful strides in simulating these promising applications, the disconnect with real-world C-V2X performance often renders ungrounded assumptions, resulting in huge barriers towards deployment. In this paper, we aim to build and release a real-world C-V2X dataset, SEE-V2X, using commercial off-the-shelf standard compliant C-V2X radios. Emulating the traffic patterns of popular C-V2X applications, we investigate the gap between the demand and reality. Beyond throughput and latency, SEE-V2X contains cross-layer details, offers insight into the resource scheduling and allocation mechanism in various situations, and reveals the impact of nuanced configuration. Our preliminary analysis shows that severe packet collision and jitter can easily happen, indicating opportunities to avoid performance degradation with careful and subtle configuration. SEE-V2X dataset and the analysis tools are available at https://cisl.ucr.edu/SEE-V2X/.

Practitioners face substantial challenges in building multi-modal platforms that are essential for autonomous systems' safe decision-making. Those complications, including synchronization, calibration, and tedious sensor validation, hinder user adoption for real-world applications. We present CMS, a Cooperative Multi-modal Sensing Platform. CMS provides one consistent interface, integrating LiDAR, camera, RaDAR, and GNSS/IMU, streamlines these processes and makes the intricacies transparent to users and applications. Our demonstration shows that CMS can obtain high-quality multi-modal sensor data, paving the way toward real-world prototypes of cooperative autonomous systems.

Widely adopted motion forecasting datasets sub-stitute the observed sensory inputs with higher-level abstractions such as 3D boxes and polylines. These sparse shapes are inferred through annotating the original scenes with perception systems’ predictions. Such intermediate representations tie the quality of the motion forecasting models to the performance of computer vision models. Moreover, the human-designed explicit interfaces between perception and motion forecasting typically pass only a subset of the semantic information present in the original sensory input. To study the effect of these modular approaches, design new paradigms that mitigate these limitations, and accelerate the development of end-to-end motion forecasting models, we augment the Waymo Open Motion Dataset (WOMD) with large-scale, high-quality, diverse LiDAR data for the motion forecasting task.The new augmented dataset (WOMD-LiDAR)1consists of over 100,000 scenes that each spans 20 seconds, consisting of well-synchronized and calibrated high quality LiDAR point clouds captured across a range of urban and suburban geographies. Compared to Waymo Open Dataset (WOD), WOMDLiDAR dataset contains 100× more scenes. Furthermore, we integrate the LiDAR data into the motion forecasting model training and provide a strong baseline. Experiments show that the LiDAR data brings improvement in the motion forecasting task. We hope that WOMD-LiDAR will provide new opportunities for boosting end-to-end motion forecasting models.

In world-locked mixed reality (MR), virtual content is locked in place with respect to the real world. Pose estimation is a key component to create world-locked MR experiences by estimating the device's position and orientation in order to render the virtual content accordingly. Current methods of evaluating world-locked MR include user studies, which are time consuming, and absolute trajectory error (ATE), which does not directly represent what is shown on the user's display. In this work, we propose ReplayAR, a tool that can replay user movement traces and output the corresponding visualizations (renderings) of the MR display. ReplayAR can be used to compare renderings from different MR pose estimation methods side by side, using our proposed Visual Difference metric. We implemented ReplayAR on a Hololens 2 MR headset and used it to evaluate open and closed-source pose estimation methods on standard datasets and our own collected traces. The results suggest that Visual Difference better reflects what is shown on the MR display compared to ATE. We hope that ReplayAR can encourage reproducible evaluation of world-locked MR, and towards this, we release the open-source code.

Collaborative Vehicular Perception (CVP) enables connected and autonomous vehicles (CAVs) to cooperatively extend their views through wirelessly sharing their sensor data. Existing CVP systems employ either a vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) view exchange paradigm. In this paper, we advocate a hybrid CVP design: our developed system, Harbor, employs V2I as its fundamental underlying framework, and opportunistically employs V2V to boost the performance. In Harbor, vehicles (helpers) may serve as relays to assist other vehicles (helpees) in reaching an edge node, which performs sensor data merging to produce the extended view. We judiciously partition the workload between the edge and vehicles, develop a robust helper-helpee assignment model, and solve it efficiently at runtime. We conduct both real-world tests and large-scale emulation experiments using two prevailing CAV applications: drivable space detection and object detection. Our real-world evaluation conducted at one of the world's first purpose-built autonomous driving testbeds demonstrates that Harbor outperforms state-of-the-art V2V- or V2I-only CVP schemes by up to 36% in detection accuracy, resulting in significantly fewer collisions under dangerous driving scenarios.

Interest in cooperative perception is growing quickly due to its remarkable performance in improving perception capabilities for connected and automated vehicles. This improvement is crucial, especially for automated driving scenarios in which perception performance is one of the main bottlenecks to the development of safety and efficiency. However, current cooperative perception methods typically assume that all collaborating vehicles have enough communication bandwidth to share all features with an identical spatial size, which is impractical for real-world scenarios. In this paper, we propose Adaptive Cooperative Perception, a new cooperative perception framework that is not limited by the aforementioned assumptions, aiming to enable cooperative perception under more realistic and challenging conditions. To support this, a novel feature encoder is proposed and named Pillar Attention Encoder. A pillar attention mechanism is designed to extract the feature data while considering its significance for the perception task. An adaptive feature filter is proposed to adjust the size of the feature data for sharing by considering the importance value of the feature. Experiments are conducted for cooperative object detection from multiple vehicle-based and infrastructure-based LiDAR sensors under various communication conditions. Results demonstrate that our method can successfully handle dynamic communication conditions and improve the mean Average Precision by 10.18% when compared with the state-of-the-art feature encoder.

Crowdsourced image processing has the potential to vastly impact response timeliness in various emergency situations. Because images can provide extremely important information regarding an event of interest (hits), sending the right images to an analyzer as soon as possible is of crucial importance. In this paper, we consider the problem of optimally assigning resources, both local (CPUs in phones) and remote (network-based GPUs) to mobile devices for processing images, ultimately sending those of interest to a centralized entity while also accounting for the energy consumption at the distributed nodes. To that end, we use the dual-path Network Utility Maximization (NUM) framework, coupled with a hit-ratio estimator and energy costs, to enable a distributed implementation of the system. We include analysis of different hit-ratio estimators using realistic trace data, first considering immediate and then delayed feedback. We address accuracy concerns when estimating the likelihood of future imagehitsand provide a window-based heuristic for scenarios when hit-ratio feedback is severely delayed. Our TCP-inspired window-method predicts both imagehitlikelihood and current wireless network congestion with great effectiveness. Results are validated using both synthetic simulations and real-life traces.

Optical sensors and learning algorithms for autonomous vehicles have dramatically advanced in the past few years. Nonetheless, the reliability of today's autonomous vehicles is hindered by the limited line-of-sight sensing capability and the brittleness of data-driven methods in handling extreme situations. With recent developments of telecommunication technologies, cooperative perception with vehicle-to-vehicle communications has become a promising paradigm to enhance autonomous driving in dangerous or emergency situations. We introduce Coopernaut,an end-to-end learning model that uses cross-vehicle perception for vision-based cooperative driving. Our model encodes Li-DAR information into compact point-based representations that can be transmitted as messages between vehicles via realistic wireless channels. To evaluate our model, we develop Autocastsim,a network-augmented driving simulation framework with example accident-prone scenarios. Our experiments on Autocastsim suggest that our cooperative perception driving models lead to a 40% improvement in average success rate over egocentric driving mod-els in these challenging driving situations and a$5\times$smaller bandwidth requirement than prior work V2VNet. Cooper-nautand Autocastsim are available at https://ut-austin-rpl.github.io/Coopernaut/.

Autonomous vehicles use 3D sensors for perception. Cooperative perception enables vehicles to share sensor readings with each other to improve safety. Prior work in cooperative perception scales poorly even with infrastructure support. AUTOCAST1 enables scalable infrastructure-less cooperative perception using direct vehicle-to-vehicle communication. It carefully determines which objects to share based on positional relationships between traffic participants, and the time evolution of their trajectories. It coordinates vehicles and optimally schedules transmissions in a distributed fashion. Extensive evaluation results under different scenarios show that, unlike competing approaches, AUTOCAST can avoid crashes and near-misses which occur frequently without cooperative perception, its performance scales gracefully in dense traffic scenarios providing 2-4x visibility into safety critical objects compared to existing cooperative perception schemes, its transmission schedules can be completed on the real radio testbed, and its scheduling algorithm is near-optimal with negligible computation overhead.

Public outdoor surveillance cameras often have limited metadata describing their properties. Frequently, a public camera’s precise position, orientation, focal length, and image center are unknown; these attributes are necessary to precisely pinpoint the location of events seen in the camera. In this article, we ask: what is the minimal information needed to accurately estimate these properties for public cameras? We show, using a judicious combination of projective geometry, neural networks, and crowd-sourced annotations from human workers, that it is possible to, for example, localize 95% of the cameras in our test data set to within 12 m using a single image taken from the camera. This performance is an order of magnitude better than PoseNet, a state-of-the-art neural network that needs significantly more information than our approach, and can only estimate position and orientation (and not other properties). Finally, we show that the camera’s inferred pose and properties can help design a number of virtual sensors , all of which have good accuracy.

Crowdsourced image processing has the potential to vastly impact response timeliness in various emergency situations. Because images can provide extremely important information regarding an event of interest, sending the right images to an analyzer as soon as possible is of crucial importance. In this paper, we consider the problem of optimally assigning resources, both local (CPUs in phones) and remote (network-based GPUs) to mobile devices for processing images, ultimately sending those of interest to a centralized entity while also accounting for the energy consumption. To that end, we use the Network Utility Maximization (NUM) framework, coupled with a hit-ratio estimator and energy costs, to enable a distributed implementation of the system. Our results are validated using both synthetic simulations and real-life traces.

Disaster and emergency response operations require rapid situational assessment of the affected area for timely and efficient rescue operations. A 3D map, collected after a disaster, can provide such awareness, but constructing this map quickly is a significant challenge. In this paper, we explore the design of a capability called QuickSketch that rapidly builds 3D representations of an unknown environment using crowdsourcing. QuickSketch employs multiple vehicles equipped with 3D sensors (stereo cameras) to explore different areas of an unknown territory and then combines 3D data from all the vehicles to build a single 3D map. QuickSketch annotates the 3D map with important landmarks and enables rapid contextualization of visual intelligence (photos) received from first responders and disaster victims to guarantee timely backup and rescue operations. Our evaluation results show that QuickSketch can stitch a 3D map for a large campus with sub-meter mapping accuracy under certain conditions, position landmarks an order of magnitude more accurately than other image matching techniques, and contextualize visual intelligence accurately.

Autonomous vehicle prototypes today come with line-of-sight depth perception sensors like 3D cameras. These 3D sensors are used for improving vehicular safety in autonomous driving, but have fundamentally limited visibility due to occlusions, sensing range, and extreme weather and lighting conditions. To improve visibility and performance, not just for autonomous vehicles but for other Advanced Driving Assistance Systems (ADAS), we explore a capability called Augmented Vehicular Reality (AVR). AVR broadens the vehicle's visual horizon by enabling it to wirelessly share visual information with other nearby vehicles, but requires the design of novel relative positioning techniques, new perspective transformation methods, approaches to isolate and predict the motion of dynamic objects in order to hide latency, and adaptive transmission strategies to cope with wireless bandwidth variability. We show that AVR is feasible using off-the-shelf wireless technologies, and it can qualitatively change the decisions made by autonomous vehicle path planning algorithms. Our AVR prototype achieves positioning accuracies that are within a few percent of car lengths and lane widths, and is optimized to process frames at 30fps.

The enormous success of advanced wireless devices is pushing the demand for higher wireless data rates. The industry is satisfying this increasing demand by densely deploying large numbers of access points (APs). Unfortunately, unicast rates, especially in crowded scenarios, remain very low due to severe interference and time-sharing. However, one may take advantage of the broadcasting nature of wireless transmissions to offer high multicast rates. Motivated by this, we present coordinated broadcasting (Co-BCast), a system which coordinates multiple APs to provide participants of big events with high multicast rates that can support multiple high definition video streams.

Precise positioning of an automobile to within lane-level precision can enable better navigation and context-awareness. However, GPS by itself cannot provide such precision in obstructed urban environments. In this paper, we present a system called CARLOC for lane-level positioning of automobiles. CARLOC uses three key ideas in concert to improve positioning accuracy: it uses digital maps to match the vehicle to known road segments; it uses vehicular sensors to obtain odometry and bearing information; and it uses crowd-sourced location of estimates of roadway landmarks that can be detected by sensors available in modern vehicles. CARLOC unifies these ideas in a probabilistic position estimation framework, widely used in robotics, called the sequential Monte Carlo method. Through extensive experiments on a real vehicle, we show that CARLOC achieves sub-meter positioning accuracy in an obstructed urban setting, an order-of-magnitude improvement over a high-end GPS device.

Precise positioning of an automobile to within lane-level precision can enable better navigation and context-awareness. However, GPS by itself cannot provide such precision in obstructed urban environments. In this paper, we present a system called CARLOC for lane-level positioning of automobiles. CARLOC uses three key ideas in concert to improve positioning accuracy: it uses digital maps to match the vehicle to known road segments; it uses vehicular sensors to obtain odometry and bearing information; and it uses crowd-sourced location of estimates of roadway landmarks that can be detected by sensors available in modern vehicles. CARLOC unifies these ideas in a probabilistic position estimation framework, widely used in robotics, called the sequential Monte Carlo method. Through extensive experiments on a real vehicle, we show that CARLOC achieves sub-meter positioning accuracy in an obstructed urban setting, an order-of-magnitude improvement over a high-end GPS device.

Automotive apps can improve efficiency, safety, comfort, and longevity of vehicular use. These apps achieve their goals by continuously monitoring sensors in a vehicle, and combining them with information from cloud databases in order to detect events that are used to trigger actions (e.g., alerting a driver, turning on fog lights, screening calls). However, modern vehicles have several hundred sensors that describe the low level dynamics of vehicular subsystems, these sensors can be combined in complex ways together with cloud information. Moreover, these sensor processing algorithms may incur significant costs in acquiring sensor and cloud information. In this paper, we propose a programming framework called CARLOG to simplify the task of programming these event detection algorithms. CARLOG uses Datalog to express sensor processing algorithms, but incorporates novel query optimization methods that can be used to minimize bandwidth usage, energy or latency, without sacrificing correctness of query execution. Experimental results on a prototype show that CARLOG can reduce latency by nearly two orders of magnitude relative to an unoptimized Datalog engine.