Deep learning recommendation models (DLRMs) have been used across many business-critical services at Meta and are the single largest AI application in terms of infrastructure demand in its data-centers. In this paper, we present Neo, a software-hardware co-designed system for high-performance distributed training of large-scale DLRMs. Neo employs a novel 4D parallelism strategy that combines table-wise, row-wise, column-wise, and data parallelism for training massive embedding operators in DLRMs. In addition, Neo enables extremely high-performance and memory-efficient embedding computations using a variety of critical systems optimizations, including hybrid kernel fusion, software-managed caching, and quality-preserving compression. Finally, Neo is paired with ZionEX, a new hardware platform co-designed with Neo's 4D parallelism for optimizing communications for large-scale DLRM training. Our evaluation on 128 GPUs using 16 ZionEX nodes shows that Neo outperforms existing systems by up to 40× for training 12-trillion-parameter DLRM models deployed in production.

The combined effect of Moore's law and the failure of Den nard scaling have led to multi-core mobile devices with immense computation capabilities. The biggest limitation of the computation capability for any mobile device is its battery. Mobile cloud computing is used to offload compute intensive tasks that affect a mobile device's battery. Mobile ad-hoc computing can be used as an alternative to mobile cloud computing in cases where cloud access is not available or is inhibitive to application performance, although battery drain remains a critical argument against mobile ad-hoc computing. In this paper, we present Synergy, a middleware that increases the battery life for a system of mobile devices connected in a peer-to-peer ad-hoc network. Synergy conserves energy by scaling core frequencies and by intelligently distributing the computation among peer devices. The middleware is not restricted to mobile phones and in no way restricts the mobility of the devices. Synergy considers the mobile devices connected in a peer-to-peer fashion as a single multicore device with Wifi as the interconnect. With Synergy running on Google Nexus phones we were able to conserve up to 30.6% of the system battery while incurring a latency penalty of less than 5%.

We report our effort in engineering a high performance remote method invocation (RMI) mechanism for the Common Component Architecture (CCA). This mechanism provides a highly efficient and easy-to-use mechanism for distributed computing in CCA, enabling CCA applications to effectively leverage parallel systems to accelerate computations. This work is built on the previous work of Babel RMI. Babel is a high performance language interoperability tool that is used in CCA for scientific application writers to share, reuse, and compose applications from software components written in different programming languages. Babel provides a transparent and flexible RMI framework for distributed computing. However, the existing Babel RMI implementation is built on top of TCP and does not provide the level of performance required to distribute fine-grained tasks. We observed that the main reason the TCP based RMI does not perform well is because it does not utilize the high performance interconnect hardware on a cluster efficiently. We have implemented a high performance RMI protocol, HPCRMI. HPCRMI achieves low latency by building on top of a low-level portable communication library, Aggregated Remote Message Copy Interface (ARMCI), and minimizing communication for each RMI call. Our design allows a RMI operation to be completed by only two RDMA operations. We also aggressively optimize our system to reduce copying. In this paper, we discuss the design and our experimental evaluation of this protocol. Our experimental results show that our protocol can improve RMI performance by an order of magnitude.