The rapid emergence of cloud data warehouses like Google BigQuery has redefined the landscape of data analytics. With the growth of data volumes, such systems need to scale to hundreds of EiB of data in the near future. This growth is accompanied by an increase in the number of objects stored and the amount of metadata such systems must manage. Traditionally, Big Data systems have tried to reduce the amount of metadata in order to scale the system, often compromising query performance. In Google BigQuery, we built a metadata management system that demonstrates that massive scale can be achieved without such tradeoffs. We recognized the benefits that fine grained metadata provides for query processing and we built a metadata system to manage it effectively. We use the same distributed query processing and data management techniques that we use for managing data to handle Big metadata. Today, BigQuery uses these techniques to support queries over billions of objects and their metadata.

Given a sensor network and aggregate queries over the values sensed by subsets of nodes in the network, how do we ensure that high quality results are served for the maximum possible time? The issues underlying this question relate to the fidelity of query results and lifetime of the network. To maximize both, we propose a novel technique called asynchronous in-network prediction incorporating two computationally efficient methods for in-network prediction of partial aggregate values. These values are propagated via a tree whose construction is cognizant of (a) the coherency requirements associated with the queries, (b) the remaining energy at the sensors, and (c) the communication and message processing delays. Finally, we exploit in-network filtering and in-network aggregation to reduce the energy consumption of the nodes in the network. Experimental results over real world data support our claim that, for aggregate queries with associated coherency requirements, a prediction-based, asynchronous scheme provides higher quality results for a longer amount of time than a synchronous scheme. Also, whereas aggregate dissemination techniques proposed so far for sensor networks appear to have to trade-off quality of data for energy efficiency, we demonstrate that this is not always necessary.