First-order optimization techniques, such as stochastic gradient descent (SGD) and its variants, are widely used in machine learning applications due to their simplicity and low per-iteration costs. However, they often require larger numbers of iterations, with associated communication costs in distributed environments. In contrast, Newton-type methods, while having higher per-iteration computation costs, typically require a significantly smaller number of iterations, which directly translates to reduced communication costs. We present a novel distributed optimizer for classification problems, which integrates a GPU-accelerated Newton-type solver with the global consensus formulation of Alternating Direction of Method Multipliers (ADMM). By leveraging the communication efficiency of ADMM, a highly efficient GPUaccelerated inexact-Newton solver, and an effective spectral penalty parameter selection strategy, we show that our proposed method (i) yields better generalization performance on several classification problems; (ii) significantly outperforms state-of-the-art methods in distributed time to solution; and (iii) offers better scaling on large distributed platforms.

First order optimization methods, which rely only on gradient information, are commonly used in diverse machine learning (ML) applications, owing to their simplicity of implementations and low per-iteration computational/storage costs. However, they suffer from significant disadvantages; most notably, their performance degrades with increasing problem ill-conditioning. Furthermore, they often involve a large number of hyperparameters, and are notoriously sensitive to parameters such as the step-size. By incorporating additional information from the Hessian, second-order methods, have been shown to be resilient to many such adversarial effects. However, these advantages come at the expense of higher per-iteration costs, which in “big data” regimes, can be computationally prohibitive. In this paper, we show that, contrary to conventional belief, second-order methods, when designed suitably, can be much more efficient than first-order alternatives for large-scale ML applications. In convex settings, we show that variants of classical Newton's method in which the Hessian and/or gradient are randomly subsampled, coupled with efficient GPU implementations, far outperform state of the art implementations of existing techniques in popular ML software packages such as TensorFlow. We show that our proposed methods (i) achieve better generalization errors in significantly lower wall-clock time – orders of magnitude faster, compared to first-order alternatives (in TensorFlow) and, (ii) offers significantly smaller (and easily parameterized) hyperparameter space making our methods highly robust.

We present a parallel implementation of the ReaxFF force field on massively parallel heterogeneous architectures, called PuReMD-Hybrid. PuReMD, on which this work is based, along with its integration into LAMMPS, is currently used by a large number of research groups worldwide. Accelerating this important community codebase that implements a complex reactive force field poses a number of algorithmic, design, and optimization challenges, as we discuss in detail. In particular, different computational kernels are best suited to different computing substrates-CPUs or GPUs. Scheduling these computations requires complex resource management, as well as minimizing data movement across CPUs and GPUs. Integrating powerful nodes, each with multiple CPUs and GPUs, into clusters and utilizing the immense compute power of these clusters requires significant optimizations for minimizing communication and, potentially, redundant computations. From a programming model perspective, PuReMD-Hybrid relies on MPI across nodes, pthreads across cores, and CUDA on the GPUs to address these challenges. Using a variety of innovative algorithms and optimizations, we demonstrate that our code can achieve over 565-fold speedup compared to a single core implementation on a cluster of 36 state-of-the-art GPUs for complex systems. In terms of application performance, our code enables simulations of over 1.8M atoms in under 0.68 seconds per simulation time step.

Location Based Social Networks (LBSNs) integrate location-based facilities with social connectivity for delivering a variety of services, enhancing user experience, emergency/disaster management, and streamlining business processes. A number of recent research efforts have studied relationships between geolocation and social connectivity, social connectivity and preferences, and node attributes and strength of social ties. These efforts have successfully demonstrated prediction of various attributes based on social connectivity, mobility, dynamic checkin information etc., including prediction of user location as well as future checkin locations.