Sensors provide a critical link between digital and physical systems in the Internet of Things (IoT). However, as they age, their accuracy degrades due to drift. This reduces data trustworthiness and requires significant maintenance investment to mitigate, especially in large-scale sensor deployments typical of IoT systems. Previous approaches to drift correction typically require large volumes of ground truth data and do not consider measurement or prediction uncertainty. In this paper, we propose a probabilistic sensor drift correction method that takes a fundamental approach to modelling the sensor response using Gaussian Process Regression. Tested using dissolved oxygen sensors, our method delivers mean squared error (MSE) reductions of up to 90% and more than 20% on average. We also propose a novel uncertainty-driven calibration schedule optimisation approach that builds on top of drift correction and further reduces MSE by up to 15.7%.

Exponential growth in data collection is creating significant challenges for data storage and analytics latency. Approximate Query Processing (AQP) has long been touted as a solution for accelerating analytics on large datasets, however, there is still room for improvement across all key performance criteria. In this paper, we propose a novel histogram-based data synopsis called PairwiseHist that uses recursive hypothesis testing to ensure accurate histograms and can be built on top of data compressed using Generalized Deduplication (GD). We thus show that GD data compression can contribute to AQP. Compared to state-of-the-art AQP approaches, Pairwise-Hist achieves better performance across all key metrics, including 2.6× higher accuracy, 3.5× lower latency, 24× smaller synopses and 1.5--4× faster construction time.

The exponential growth of data generated by the Internet of Things presents significant challenges for data communication, storage, and analytics. Consequently, organizations often face high costs when attempting to leverage their own data. Novel techniques that holistically optimize data storage and analytics in IoT systems are therefore required. One promising approach is generalized deduplication (GD), which is a lossless compression technique that delivers high compression while also enabling low-cost random access directly on compressed data. In this article, we introduce GreedyGD, a novel GD data compression algorithm that offers reliable, efficient data analytics, along with more compression and faster runtime compared to previous GD compressors. Evaluating GreedyGD on 18 real-world datasets revealed excellent performance: a 11.2× speed-up, 1.6× more compression, and more accurate and reliable analytics while using 4× less data compared to previous GD compressors.

The Internet of Things (IoT) has brought about exponential growth in sensor data. This has led to increasing demands for efficient and novel data transmission, storage, and analytics solutions for sustainable IoT ecosystems. It has been shown that the generalized deduplication (GD) compression algorithm offers not only competitive compression ratio and throughput but also random access properties that enable direct analytics of compressed data. In this article, we thoroughly stress test existing methods for direct analytics of GD compressed data with a diverse collection of 103 data sets, identify the need to optimize GD for analytics, and develop a new version of GD to this end. We also propose the generalized deduplication-enabled approximate edge analytics (GLEAN) framework. This framework applies the aforementioned analytics techniques at the Edge server to deliver end-to-end lossless data compression and high-quality Edge analytics in the IoT, thereby addressing challenges related to data transmission, storage, and analytics. Impressive analytics performance was achieved using this framework, with a median increase in$k$-means clustering error of just 2% relative to analytics performed on uncompressed data, while running$7.5\times $faster and requiring$3.9\times $less storage at the Edge server compared to universal compressors.

Given the ever increasing volume of data generated by the Internet of Things, data compression plays an essential role in reducing the cost of data transmission and storage. However, it also introduces a barrier, namely decompression, between users and the data-driven insights they require. We propose methods for direct analytics of compressed data based on the Generalized Deduplication compression algorithm. When applied to data clustering, the accuracy of the best performing method differs by merely 1-5% when compared to analytics performed upon the uncompressed data. However, it runs four times faster, accesses only 14% as much data and requires significantly less storage since the data is always compressed. These results show that it is possible to simultaneously reap the benefits of compression and accurate, high-speed analytics in many applications.