Stream processing enables real-time data analysis. Recent stream processing engines (SPEs) execute stream processing in a distributed manner for real-time analysis of massive amounts of data produced by IoT devices and sensors. It has been widely adopted in various applications that support critical decision making. To explain the results of stream processing, ensuring provenance is indispensable. Provenance clarifies the relationship between input data and output data in the processing. With provenance, we can understand what input data contributed to the output. Existing frameworks for providing provenance for stream processing generate provenance or additional information to construct provenance at runtime. However, these approaches impose substantial overhead in ordinary stream processing. In this paper, we propose a new framework, named LPStream, for fine-grained lazy provenance. LPStream is the first framework to support lazy provenance for stream processing. In the ordinary execution mode, LPStream executes stream processing with checkpointing but without provenance generation. If provenance is necessary for some target output tuples, it replays the processing from an appropriate checkpoint and generates the provenance for the target tuple. We explain the design and implementation of LPStream and evaluate its performance by comparing LPStream with stream processing without provenance and with eager provenance. The experimental results demonstrate the effectiveness of our proposal.

Abstract Data lineage allows information to be traced to its origin in data analysis by showing how the results were derived. Although many methods have been proposed to identify the source data from which the analysis results are derived, analysis is becoming increasingly complex both with regard to the target (e.g., images, videos, and texts) and technology (e.g., AI and machine learning (ML)). In such complex data analysis, simply showing the source data may not ensure traceability. For example, ML analysts building image classifier models often need to know which parts of images are relevant to the output and why the classifier made a decision. Recent studies have intensively investigated interpretability and explainability in the AI/ML domain. Integrating these techniques into the lineage framework will help analysts understand more precisely how the analysis results were derived and how the results are trustful. In this paper, we propose the concept of augmented lineage for this purpose, which is an extended lineage, and an efficient method to derive the augmented lineage for complex data analysis. We express complex data analysis flows using relational operators by combining user-defined functions (UDFs). UDFs can represent invocations of AI/ML models within the data analysis. Then, we present a method taking UDFs into consideration to derive the augmented lineage for arbitrarily chosen tuples among the analysis results. We also experimentally demonstrate the efficiency of the proposed method.

This paper presents a strategy to choose WiFi channels in urban areas. We consider (i) inter-channel interference where adjacent channels interfere with each other in WiFi systems and (ii) urban situations where many APs in different systems are deployed in an uncoordinated way. As it is often hard to identify the channel with less interference in such a situation, we present a channel scoring function that estimates the performance level of each channel. To build the scoring function, we have conducted exhaustive simulations with a large number of scenarios, and multiple regression analysis has been applied where channel occupancy patterns, traffic volumes and RSS in those channels are used as explanatory variables. To evaluate our method, this scoring function was examined in a realistic scenario where several APs interfere with the AP of interest. We have confirmed that the scores and the actual performance are well-matched where the Spearman's rank correlation coefficient was sufficiently high and can identify the top-ranked channel as well.

This paper presents a scheduling algorithm for a set of wireless stations such as road-side access points for vehicular networks and outdoor WiFi stations, which are deployed in wide urban areas and may compete with each other for limited wireless resources. Different from a number of conventional approaches most of which consider detailed information on individual stations and signal interference among them, we focus more on geography of the areas of interest, and provide a novel algorithm that pursues the best balance among (i) optimality of resource utilization, (ii) robustness to new station installation and traffic demand changes, and (iii) scalability to the population of stations and area size. We have confirmed the performance by experimental simulations with several scenarios, and the applicability of approach has been testified by a case study on a scheduling problem for roadside access points of vehicular networks in cooperation with a manufacturing corporation.