We introduce nOS, a “nano-sized” fully distributed operating system aimed at large-scale, many-core embedded systems. nOS enables dynamic runtime optimisation of energy and execution time through lightweight and scalable distributed protocols. nOS implements new dynamic resource optimisation algorithms, and provides an intuitive and easy-to-use programmer API that supports runtime task energy optimisation through dynamic frequency scaling, transparent task communication tracking, and automatic task mapping. Critically, nOS has a completely distributed implementation, providing excellent scalability. Contrary to other approaches, the dynamic runtime optimisations require no a priori knowledge of workload or communication patterns. By generating runtime measurements of thread performance, core load, and process communication, we show that nOS can deliver improvements that would not be possible with only static analysis. Using a many-core system called Swallow, we show a <;3kB fullstack implementation of nOS together with application, OS and hardware. Using two applications with different communication patterns, we illustrate the power and flexibility of our approach, as well as various tradeoffs in energy and performance from making better mapping choices than would be available offline.

Enterprise customers increasingly require greater flexibility in the way they access and process their Big Data while at the same time they continue to request advanced analytics and access to diverse data sources. Yet customers also still require the robustness of enterprise class analytics for their mission-critical data. In this paper, we present our initial efforts toward a solution that satisfies the above requirements by integrating the HPE Vertica enterprise database with Apache Spark's open source big data computation engine. In particular, it enables fast, reliable transferring of data between Vertica and Spark; and deploying Machine Learning models created by Spark into Vertica for predictive analytics on Vertica data. This integration provides a fabric on which our customers get the best of both worlds: it extends Vertica's extensive SQL analytics capabilities with Spark's machine learning library (MLlib), giving Vertica users access to a wide range of ML functions; it also enables customers to leverage Spark as an advanced ETL engine for all data that require the guarantees offered by Vertica.

Due to R's popularity as a data-mining tool, many distributed systems expose an R-based API to users who need to build a distributed application in R. As a result, data scientists have to learn to use different interfaces such as RHadoop, SparkR, Revolution R's ScaleR, and HPE's Distributed R. Unfortunately, these interfaces are custom, non-standard, and difficult to learn. Not surprisingly, R applications written in one framework do not work in another, and each backend infrastructure has spent redundant effort in implementing distributed machine learning algorithms. Working with the members of R-core, we have created ddR (Distributed Data structures in R), a unified system that works across different distributed frameworks. In ddR, we introduce a novel programming primitive called dmapply that executes functions on distributed data structures. The dmapply primitive encapsulates different computation patterns: from function and data broadcast to pair-wise communication. We show that dmapply is powerful enough to express algorithms that fit the statistical query model, which includes many popular machine learning algorithms, as well as applications written in MapReduce. We have integrated ddR with many backends, such as R's single-node parallel framework, multi-node SNOW framework, Spark, and HPE Distributed R, with few or no modifications to any of these systems. We have also implemented multiple machine learning algorithms which are not only portable across different distributed systems, but also have performance comparable to the "native" implementations on the backends. We believe that ddR will standardize distributed computing in R, just like the SQL interface has standardized how relational data is manipulated.