Point cloud data in the form of LiDAR is often utilized for its spatial qualities, especially in smart city projects for tasks involving vehicles and pedestrians. However, the process in which LiDAR data is acquired can be cumbersome to setup and automate. In this paper, we introduce a streaming and an on-demand pipeline for capturing LiDAR data from Velodyne Ultra Pucks placed along northern Nevada intersections known as the Living Lab as part of a smart city project for the city of Reno. The data coming from these intersections consist of the following formats: ROS 2 bag file, PCD, LAZ, Google Draco, and PCAP. A streaming point cloud service with PCD, LAZ, and Draco was implemented to stream any of these formats, as well as to allow the user to capture the current monitored point cloud. Additionally, two on-demand web services were implemented for both the PCAP and ROS 2 bag file to enable a user to start and stop the acquisition of LiDAR data in these formats. Through our analysis, it was discovered that Draco provided the best processing time and had a wider range of options that affected the quality of the point cloud. To evaluate this pipeline, the features of existing software were compared and a discussion was provided with an analysis of the point cloud formats.

In many smart city projects, a common choice to capture spatial information is the inclusion of LiDAR data, but this decision will often invoke severe growing pains within the existing infrastructure. In this paper, we introduce a data pipeline that orchestrates Apache NiFi (NiFi), Apache MiNiFi (MiNiFi), and several other tools as an automated solution in order to relay and archive LiDAR data captured by deployed edge devices. The LiDAR sensors utilized within this workflow are Velodyne Ultra Pucks sensors that capture at a rate of 10 frames per second and produces 6-7 GB packet capture (PCAP) files per hour. By both compressing the file after capturing it and compressing the file in real-time, we discovered that gzip produced a file of 5 GB and saved about 5 minutes in transmission time to NiFi, as well as saving considerable CPU time when compressing the file in real-time. Alternatively, we chose XZ as the compression algorithm for the ingestion of LiDAR data onto an institution compute cluster due to its high compression ratio. In order to evaluate the capabilities of our system design, the features of this data pipeline were compared against existing third-party services, namely Globus and RSync.

In this paper, we present a novel and accessible approach to time-series data validation: the Near-Real Time Autonomous Quality Control (NRAQC) system. The design, implementation, and impacts of this software are explored in detail within this paper. This software system, created in close conference with environmental scientists, leverages microservice design patterns employed for high volume web applications to develop a contemporary solution to the problem of data quality control with streaming sensor data. Through a comparative analysis between NRAQC and the GCE Toolbox, we argue that the web based deployment of QC software enhances accessibility to crucial tools required to make a robust and useful data product from raw measurements. Additionally, a key innovation of the NRAQC platform is its positive impact on modern data management practices and quality data dissemination.

The NSF EPSCOR funded Solar Nexus Project is a collaborative effort between scientists, engineers, educators, and technicians to increase the amount of renewable solar energy in Nevada while eliminating its adverse effects on the surrounding environment and wildlife, and minimizing water consumption. The project seeks to research multiple areas, including water usage at power plants, the effect of power plant construction on the surrounding ecology, alternative wastewater methods to maintain solar panels, and interdisciplinary solutions to improve solar energy in Nevada. In order to organize and analyze this data to produce effective change, Nexus needs a centralized database to store collected data. To this end the Nevada Research Data Center is designed to collect, format, and store data for scientists to view and consider. This paper presents a new architecture solution for the NRDC. Based in microservices, the solution aims to ensure scalability, reliability, and maintainability of this data center. Background on NRDC is provided in the paper, together with details on the proposed solution's software specification, design, and prototype implementation. A discussion of the microservice-based architecture's benefits and an outline of planned directions of future work are also included.