Recent years have witnessed the growth of the size and dimensionality of data from various applications at an unprecedented rate. Detecting anomalies in these high-dimensional data has a great significance yet remains challenging due to the sparsity, noise, and irrelevant features in the high-dimensional data. Principle Component Analysis (PCA) and AutoEncoder (AE) are the two most widely used dimension reduction (DR) methods where it reduces the number of features while capturing the most necessary information of the original data. PCA alone is, however, less effective for complex data though it is fast and has explained variance to measure the preserved information of reduced data. While combining PCA with AE can be more effective, determining optimal latent representation is a challenging problem. In this work, we propose a cascaded dimension reduction method (CDR) for effective anomaly detection. In CDR, we combine AE and PCA to reduce dimension size significantly while incorporating a knee point detection algorithm to automatically select optimal dimension size k that maximizes the anomaly detection accuracy. Our extensive evaluation of various datasets and anomaly detection models demonstrate that our proposed CDR can significantly reduce dimension size while preserving the most necessary information for effective anomaly detection. As a result, CDR achieves 80%~98% of reduction ratios and 4~21x of speedup and outperforms state-of-the-art anomaly detection methods.

Edge devices with attentive sensors enable various intelligent services by exploring streams of sensor data. However, anomalies, which are inevitable due to faults or failures in the sensor and network, can result in incorrect or unwanted operational decisions. While promptly ensuring the accuracy of IoT data is critical, lack of labels for live sensor data and limited storage resources necessitates efficient and reliable detection of anomalies at edge nodes. Motivated by the existence of unique sparsity profiles that express original signals as a combination of a few coefficients between normal and abnormal sensing periods, we propose a novel anomaly detection approach, called ADSP (Anomaly Detection with Sparsity Profile). The key idea is to apply a transformation on the raw data, identify top-K dominant components that represent normal data behaviors, and detect data anomalies based on the disparity from K values approximating the periods of normal data in an unsupervised manner. Our evaluation using a set of synthetic datasets demonstrates that ADSP can achieve 92%-100% of detection accuracy. To validate our anomaly detection approach on real-world cases, we label potential anomalies using a range of error boundary conditions using sensors exhibiting a straight line in Q-Q plot and strong Pearson correlation and conduct a controlled comparison of the detection accuracy. Our experimental evaluation using real-world datasets demonstrates that ADSP can detect 83%- 92% of anomalies using only 1.7% of the original data, which is comparable to the accuracy achieved by using the entire datasets.

With recent technological advances in sensor nodes, IoT enabled applications have great potential in many domains. However, sensing data may be inaccurate due to not only faults or failures in the sensor and network but also the limited resources and transmission capability available in sensor nodes. In this paper, we first model streams of IoT data as a handful of sampled data in the transformed domain while assuming the information attained by those sampled data reveal different sparsity profiles between normal and abnormal. We then present a novel approach called AD2(Anomaly Detection using Approximated Data) that applies a transformation on the original data, samples top k-dominant components, and detects data anomalies based on the disparity in k values. To demonstrate the effectiveness of AD2, we use IoT datasets (temperature, humidity, and CO) collected from real-world wireless sensor nodes. Our experimental evaluation demonstrates that AD2can approximate and successfully detect 64%-94% of anomalies using only 1.9% of the original data and minimize false positive rates, which would otherwise require the entire dataset to achieve the same level of accuracy.

Word embeddings, learning syntactic and semantic relationships between words from the raw text, are known to achieve superior performance in many prediction and classification tasks. In this paper, we aim to show how word embedding techniques can work on a high number of nontextual data, thus has new applicability in cyber security. Specifically, we build neural embeddings with a large amount of network log data (KDD CUP'99 dataset) and train several classification models using the learned neural embeddings. Our experiment results demonstrate that the vector representations successfully learn useful features from non-textual data, similar to how classic word embeddings do with natural language and can achieve a high F1 score for both binary (normal vs. abnormal) and multi-class (DOS, Probe, U2R, R2L, and normal) classifications. We also demonstrate that the classification algorithms using the embedded vectors can maintain fairly high accuracy even when training data is very small.