High-performance computing (HPC) systems that run scientific simulations of significance produce a large amount of data during runtime. Transferring or storing such big datasets causes a severe I/O bottleneck and a considerable storage burden. Applying compression techniques, particularly lossy compressors, can reduce the size of the data and mitigate such overheads. Unlike lossless compression algorithms, error-controlled lossy compressors could significantly reduce the data size while respecting the user-defined error bound. DCTZ is one of the transform-based lossy compressors with a highly efficient encoding and purpose-built error control mechanism that accomplishes high compression ratios with high data fidelity. However, since DCTZ quantizes the DCT coefficients in the frequency domain, it may only partially control the relative error bound defined by the user. In this paper, we aim to improve the compression quality of DCTZ. Specifically, we propose a preconditioning method based on level offsetting and scaling to control the magnitude of input of the DCTZ framework, thereby enforcing stricter error bounds. We evaluate the performance of our method in terms of compression ratio and rate distortion with real-world HPC datasets. Our experimental result shows that our method can achieve a higher compression ratio than other state-of-the-art lossy compressors with a tighter error bound while precisely guaranteeing the user-defined error bound.

Recent years have witnessed an upsurge of interest in lossy compression due to its potential to significantly reduce data volume with adequate exploitation of the spatiotemporal properties of IoT datasets. However, striking a balance between compression ratios and data fidelity is challenging, particularly when losing data fidelity impacts downstream data analytics noticeably. In this paper, we propose a lossy prediction model dealing with binary classification analytics tasks to minimize the impact of the error introduced due to lossy compression. We specifically focus on five classification algorithms for frost prediction in agricultural fields allowing preparation by the predictive advisories to provide helpful information for timely services. While our experimental evaluations reaffirm the nature of lossy compressions where allowing higher errors offers higher compression ratios, we also observe that the classification performance in terms of accuracy and F-1 score differs among all the algorithms we evaluated. Specifically, random forest is the best lossy prediction model for classifying frost. Lastly, we show the robustness of the lossy prediction model based on the data fidelity in prediction performance.

Recent years have witnessed the growth of the size and dimensionality of data from various applications at an unprecedented rate. Detecting anomalies in these high-dimensional data has a great significance yet remains challenging due to the sparsity, noise, and irrelevant features in the high-dimensional data. Principle Component Analysis (PCA) and AutoEncoder (AE) are the two most widely used dimension reduction (DR) methods where it reduces the number of features while capturing the most necessary information of the original data. PCA alone is, however, less effective for complex data though it is fast and has explained variance to measure the preserved information of reduced data. While combining PCA with AE can be more effective, determining optimal latent representation is a challenging problem. In this work, we propose a cascaded dimension reduction method (CDR) for effective anomaly detection. In CDR, we combine AE and PCA to reduce dimension size significantly while incorporating a knee point detection algorithm to automatically select optimal dimension size k that maximizes the anomaly detection accuracy. Our extensive evaluation of various datasets and anomaly detection models demonstrate that our proposed CDR can significantly reduce dimension size while preserving the most necessary information for effective anomaly detection. As a result, CDR achieves 80%~98% of reduction ratios and 4~21x of speedup and outperforms state-of-the-art anomaly detection methods.

The growing demand for recording longer ECG signals to improve the effectiveness of IoT-enabled remote clinical healthcare is contributing large amounts of ECG data. While lossy compression techniques have shown potential in significantly lowering the amount of data, investigation on how to trade-off between data reduction and data fidelity on ECG data received relatively less attention. This paper gives insight into the power of lossy compression to ECG signals by balancing between data quality and compression ratio. We evaluate the performance of transformed-based lossy compressions on the ECG datasets collected from the Biosemi ActiveTwo devices. Our experimental results indicate that ECG data exhibit high energy compaction property through transformations like DCT and DWT, thus could improve compression ratios significantly without hurting data fidelity much. More importantly, we evaluate the effect of lossy compression on ECG signals by validating the R-peak in the QRS complex. Our method can obtain low error rates measured in PRD (as low as 0.3) and PSNR (up to 67) using only 5% of the transform coefficients. Therefore, R-peaks in the reconstructed ECG signals are almost identical to ones in the original signals, thus facilitating extended ECG monitoring.

Edge devices with attentive sensors enable various intelligent services by exploring streams of sensor data. However, anomalies, which are inevitable due to faults or failures in the sensor and network, can result in incorrect or unwanted operational decisions. While promptly ensuring the accuracy of IoT data is critical, lack of labels for live sensor data and limited storage resources necessitates efficient and reliable detection of anomalies at edge nodes. Motivated by the existence of unique sparsity profiles that express original signals as a combination of a few coefficients between normal and abnormal sensing periods, we propose a novel anomaly detection approach, called ADSP (Anomaly Detection with Sparsity Profile). The key idea is to apply a transformation on the raw data, identify top-K dominant components that represent normal data behaviors, and detect data anomalies based on the disparity from K values approximating the periods of normal data in an unsupervised manner. Our evaluation using a set of synthetic datasets demonstrates that ADSP can achieve 92%-100% of detection accuracy. To validate our anomaly detection approach on real-world cases, we label potential anomalies using a range of error boundary conditions using sensors exhibiting a straight line in Q-Q plot and strong Pearson correlation and conduct a controlled comparison of the detection accuracy. Our experimental evaluation using real-world datasets demonstrates that ADSP can detect 83%- 92% of anomalies using only 1.7% of the original data, which is comparable to the accuracy achieved by using the entire datasets.

With recent technological advances in sensor nodes, IoT enabled applications have great potential in many domains. However, sensing data may be inaccurate due to not only faults or failures in the sensor and network but also the limited resources and transmission capability available in sensor nodes. In this paper, we first model streams of IoT data as a handful of sampled data in the transformed domain while assuming the information attained by those sampled data reveal different sparsity profiles between normal and abnormal. We then present a novel approach called AD2(Anomaly Detection using Approximated Data) that applies a transformation on the original data, samples top k-dominant components, and detects data anomalies based on the disparity in k values. To demonstrate the effectiveness of AD2, we use IoT datasets (temperature, humidity, and CO) collected from real-world wireless sensor nodes. Our experimental evaluation demonstrates that AD2can approximate and successfully detect 64%-94% of anomalies using only 1.9% of the original data and minimize false positive rates, which would otherwise require the entire dataset to achieve the same level of accuracy.

As the volume of data collected by various IoT stations increases, Big Data management and analytics becomes a huge challenge for IoT applications. Although Big Data can potentially benefit from data compression techniques, the chances are that compression will reduce a negligible amount of data such that it would not worth the effort. The insight of this paper is that only lossy compression can unleash the power of compression to IoT because, compared with its counterpart (lossless one), it can significantly reduce the data volume by taking advantages of spatiotemporal patterns. However, lossy compression faces the challenge of compressing too much data thus losing the data fidelity, which might affect the quality of analytics outcomes. To understand the impact of lossy compression on IoT data management and analytics, we evaluate several classification algorithms on agricultural sensor data reconstructed based on energy concentration. Specifically, we applied three transformation based lossy compression mechanisms to five real-world sensor data from IoT weather stations. Our experimental results indicate that there is a distinctive relationship between energy concentration on the transformed coefficients and compression ratio as well as the amount of error introduced. While we observe a general trend where the higher energy concentration the lower compression and error rates, we also observe that the impact on classification accuracy varies among data sets and algorithms we evaluated.

Word embeddings, learning syntactic and semantic relationships between words from the raw text, are known to achieve superior performance in many prediction and classification tasks. In this paper, we aim to show how word embedding techniques can work on a high number of nontextual data, thus has new applicability in cyber security. Specifically, we build neural embeddings with a large amount of network log data (KDD CUP'99 dataset) and train several classification models using the learned neural embeddings. Our experiment results demonstrate that the vector representations successfully learn useful features from non-textual data, similar to how classic word embeddings do with natural language and can achieve a high F1 score for both binary (normal vs. abnormal) and multi-class (DOS, Probe, U2R, R2L, and normal) classifications. We also demonstrate that the classification algorithms using the embedded vectors can maintain fairly high accuracy even when training data is very small.