Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents HAP, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. HAP jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, HAP explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that HAP achieves up to 2.41x speed-up on heterogeneous clusters.

It is a challenging task to train large DNN models on sophisticated GPU platforms with diversified interconnect capabilities. Recently, pipelined training has been proposed as an effective approach for improving device utilization. However, there are still several tricky issues to address: improving computing efficiency while ensuring convergence, and reducing memory usage without incurring additional computing costs. We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategies of data and pipeline parallelism. We also propose a new runtime scheduling algorithm to reduce device memory usage, which is orthogonal to re-computation approach and does not come at the expense of training throughput. Experiments show that DAPPLE planner consistently outperforms strategies generated by PipeDream's planner by up to 3.23× speedup under synchronous training scenarios, and DAPPLE runtime outperforms GPipe by 1.6× speedup of training throughput and saves 12% of memory consumption at the same time.

Summary American option pricing is a high‐dimensional problem, and its computational challenges have attracted significant attention. We examine this problem using a stochastic mesh method enhanced with bias reduction within the classic Black–Scholes framework. We present Many Integrated Core (MIC) parallelization and acceleration techniques, which result in significant numerical acceleration for large‐scale simulations. In particular, we observe speed‐ups of 21‐fold and 28‐fold for CPU and MIC, respectively, over conventional means. Convergence performance is also examined. Copyright © 2014 John Wiley & Sons, Ltd.

A grid portal is one of the most important ways to access grid systems. The disadvantages existing in current grid portals are that they are designed for specific applications, difficult to add new applications. Besides, the security issues are not considered fully in these portals. In this paper, we proposed a user-centric and secure grid portal architecture-USGPA based on portlet. In USGPA, a security portlet model is proposed in which security issues for sensitive data and the portal server are solved. Based on the model, the application portlet model is proposed with which new applications can be encapsulated into portal quickly. Then all portlets for high-performance computing-HPC are designed based on these two models. With these portlets, users can not only manage jobs via portal but also can custom the portal by selecting and managing the applications they required. Last but important, a prototype is implemented to evaluate the scalability, security and efficiency of the architecture.