We discuss our progress towards solving a challenging biomedical problem: identifying similar patterns among multiple physiological nerve signals hidden in high throughput data, collected from micro electrical sensors implanted in several animal organs. The problem is difficult because patterns come as spikes within millisecond time-windows, data sets have high dimensionality and there is background electrical noise. A previous analytic system discovers patterns combining PCA dimensionality reduction and K-means clustering, which is slow and misses important patterns hidden by noise. Moreover, it requires reading the data set several times and it requires multiple languages and tools. With such limitations in mind, we present an improved, integrated system that effectively allows the discovery of more accurate patterns, with automated algorithm parameter tuning, by learning model parameters incrementally exploiting summarization. Our integrated solution combines signal filtering, variable construction (feature engineering) and multidimensional data summarization, for a tighter and more effective integration of PCA and K-means clustering. We present preliminary experiments on signals collected from key nerves in a rat. We show our method discovers more patterns in larger time-windows, with better noise filtering, taking less time. In the future, we plan to link signal patterns to specific physiological functions, paving the way for innovative medical treatment via nerve stimulation.

Incremental learning is used to continuously update and tune an existing model, as more data points come in. However, there is a tradeoff between speed and accuracy as the model becomes stable. From a big data angle, computing machine learning models is challenging when data sets cannot fit in main memory or when they exceed CPU capacity. On the other hand, data summarization is a fundamental technique that has promise to accelerate data science computations and compress a data set. Keeping these motivations in mind, we present an innovative system, VIM, that computes machine learning models in an incremental manner, visualizing continuous learning of model parameters as the data set is scanned. Our system is fast, it works for a wide spectrum of machine learning models and it can handle data sets larger than main memory. We developed an intuitive GUI which: (1) guides the user to upload data sets and choose a machine learning model, (2) offers interactive visualization of model parameters, and (3) helps getting an approximate model, stopping early, without reading the whole data set.

Machine learning algorithms must be able to handle large volume in big data. Nowadays, data science languages such as Python and R, are widely popular to compute machine learning models. Unfortunately, model computation can be slow, especially when the data set does not fit in the main memory or it needs to be iteratively analyzed. With these motivations in mind, we present theory and algorithms to produce a multidimensional data set summary. We show our data summaries preserves essential statistical properties of the data set and it can be computed with an accelerated Gramian matrix multiplication. That is, our data summaries represents a lossless compression and we accelerate the expensive Gramian matrix multiplication in Python with C++ code. Our solution also works for a subset of the original data set obtained by variable selection without much loss on accuracy and without recomputing all intermediate matrices. We also consider parallel processing aspects leveraging our recently introduced low-cost parallel architecture. Our experimental evaluation shows that our Gramian matrix multiplication mechanism is superior to Python and it can work beyond RAM limitations. On the other hand, our computation of the machine learning model is competitive with the Python and R on a single machine, but it outperforms Spark in parallel machines.

ER diagrams have a proven track record to rep-resent data structure and relationships, in many CS problems, beyond relational databases. The ER diagram strengths are abstraction, generality, flexibility, and intuitive visual representation, with few weaknesses; hence its popularity. The main con is the old box-diamond-ellipse-line notation, which has been subsumed by the more modern and simpler UML box-line notation. Given the broad, varied, and dynamic nature of big data ER diagrams are mostly ignored, except when the data sources are databases. It is common wisdom raw big data needs significant pre-processing before computing any analytics, resulting in a long chain of data transformations computed in SQL, Python, or R languages, for instance. On the other hand, flow diagrams remain the main mechanism to visualize major components of a software system or main processing steps of an algorithm, showing rectangles (verbs) connected by arrows (processing order, dependence). In this work, we propose to combine both diagrams into one. We propose a hybrid diagram, which we call ER-Flow, based on modern UML notation, to assist analysts in data pre-processing and exploration. Aiming to introduce a minimal change to the ER diagram, we extend relationships lines with an arrow, indicating processing flow and we annotate entities coming from pre-processing with numbers and transformation labels. We illustrate how our novel ER-Flow diagram can help the user navigate big data at the metadata level, providing an integrated view of data and source code, with many practical benefits.

Big Data Systems (Hadoop, DBMSs) require a complicated setup and tuning to store and process big data on a parallel cluster. This is mainly due to static partitioning when data sets are loaded or copied into the file system. Parallel processing thereafter works in a distributed manner, aiming for balanced parallel execution across nodes. Node synchronization, data redistribution and distributed caching in main memory are difficult to tune in the system. On the other hand, there exist analytical problems and algorithms, which can be computed in parallel, with minimal synchronization and fully independent computation. Moreover, some problems can be solved in one pass or few passes. In this paper, we introduce a low cost, yet useful, processing architecture in which data sets are dynamically partitioned at run-time and storage is transient. Each node processes one partition independently and partial results are gathered at the master processing node. Surprisingly, we show this architecture works well for some popular machine learning models as well as some graph algorithms. We attempt to identify which problem characteristics enable such efficient processing, and we also show the main bottleneck is the initial data set partitioning and distribution across nodes. We anticipate our architecture can benefit parallel processing in the cloud, where a dynamic number of virtual processors is decided at runtime or when the data set is analyzed for a short time.

Analysts prefer simpler interpreted languages to program their computations. Prominent languages include R, Python, and Matlab. On the other hand, analysts aim to compute mathematical models as fast as possible, especially with large data sets. Data summarization remains a fundamental technique to accelerate machine learning computations. Based on this motivation, we propose a novel summarization mechanism computed via a single matrix multiplication in the statistical R language. We show our summarization benefits a large family of linear models, including Linear Regression, PCA, and Naive Bayes. We present a subsystem that enables exploiting summarization by detecting Gramian matrix products in R. We optimize the existing R source code by overriding the internal R matrix multiplication algorithm using ours. Our solution can be plugged into R and help solving where a similar matrix multiplication appears, much faster and without RAM limitations. Moreover, our solution can be benefited from the parallel processing ability of the summarization matrix. We present an experimental validation showing our subsystem incurs little overhead since it works on source code while providing much faster speeds compared to the R language built-in functions. To round up our comparisons, we also compare our subsystem with Spark in parallel machines. For our solution, we assume that data can be in the HDFS, disk, or already partitioned. Our solution triumphs Spark in most cases proving we can also compete in the big data space.