Getting large language models (LLMs) to perform well on the downstream tasks requires pre-training over trillions of tokens. This typically demands a large number of powerful computational devices in addition to a stable distributed training framework to accelerate the training. The growing number of applications leveraging AI/ML led to a scarcity of the expensive conventional accelerators (such as GPUs), which emphasizes the need for the alternative specialized-accelerators that are scalable and cost-efficient. AWS Trainium is the second-generation machine learning accelerator purposely built for training large deep learning models. However, training LLMs with billions of parameters on AWS Trainium is challenging due to its relatively nascent software ecosystem. In this paper, we showcase HLAT: a family of 7B and 70B decoder-only LLMs pre-trained using 4096 AWS Trainium accelerators over 1.8 trillion tokens. The performance of HLAT is benchmarked against popular open source models including LLaMA and OpenLLaMA, which have been trained on NVIDIA GPUs and Google TPUs, respectively. On various evaluation tasks, we show that HLAT achieves model quality on par with the baselines of similar model size. We also open-source all the training scripts and configurations of HLAT1and share the best practice of using the NeuronX Distributed Training (NxDT), a customized distributed training library for AWS Trainium. Our work demonstrates that AWS Trainium powered by NxDT is able to successfully pre-train state-of-the-art LLM models with high performance and cost-effectiveness.

Large models training is plagued by the intense compute cost and limited hardware memory. A practical solution is low-precision representation but is troubled by loss in numerical accuracy and unstable training rendering the model less useful. We argue that low-precision floating points can perform well provided the error is properly compensated at the critical locations in the training process. We propose Collage which utilizes multi-component float representation in low-precision to accurately perform operations with numerical errors accounted. To understand the impact of imprecision to training, we propose a simple and novel metric which tracks the lost information during training as well as differentiates various precision strategies. Our method works with commonly used low-precision such as half-precision ($16$-bit floating points) and can be naturally extended to work with even lower precision such as $8$-bit. Experimental results show that pre-training using Collage removes the requirement of using $32$-bit floating-point copies of the model and attains similar/better training performance compared to $(16, 32)$-bit mixed-precision strategy, with up to $3.7\times$ speedup and $\sim 15\%$ to $23\%$ less memory usage in practice. The code is available at https://github.com/amazon-science/collage.

Large Language Models (LLMs) are revolutionizing the field of code development by leveraging their deep understanding of code patterns, syntax, and semantics to assist developers in various tasks, from code generation and testing to code understanding and documentation. In this survey, accompanying our proposed lecture-style tutorial for KDD 2024, we explore the multifaceted impact of LLMs on the code development, delving into techniques for generating a high-quality code, creating comprehensive test cases, automatically generating documentation, and engaging in an interactive code reasoning. Throughout the survey, we highlight some crucial components surrounding LLMs, including pre-training, fine-tuning, prompt engineering, iterative refinement, agent planning, and hallucination mitigation. We put forward that such ingredients are essential to harness the full potential of these powerful AI models in revolutionizing software engineering and paving the way for a more efficient, effective, and innovative future in code development.

We study distributed algorithms for joint adaptation of precoding and combining filters in frequency division duplex (FDD) multiple-input multiple-output (MIMO) cellular systems. Our approach extends bi-directional training (BiT), designed for time division duplex (TDD) systems, to FDD systems where uplink/downlink reciprocity may not apply. An analysis of the performance loss due to different uplink-downlink frequencies in a point-to-point scenario, shows that a direct application of BiT gives mismatched precoders with substantial performance degradation. We first propose an algorithm assuming each base transceiver station (BTS) knows the uplink and downlink channels of mobiles within the cell. We then consider the scenario where neither the BTSs nor the mobiles have a priori channel state information. Our proposed approaches assume angular reciprocity characterized by angles of arrival/departure that vary predictably with frequency. Hence spatial beams corresponding to angles of arrival can be turned around to the corresponding angles of departure in the paired band. We present three methods, differing in how the angular decomposition is applied and used, namely, to reconstruct the combiner directly, or to reconstruct the received signal. Simulation results indicate that when the multipath is sufficiently sparse, most of achievable gain with channel reciprocity and TDD can be recovered.

We study distributed rank selection in a multi-cell wideband cellular network with multiple antennas per node. The rank for a particular mobile is the number of independent data streams across all sub-carriers and antennas. Rank selection then involves both spatial multiplexing and sub-carrier allocations. We propose assigning frequency-space signatures across the data streams to unify the rank assignment over those two dimensions. A sum rate maximization problem is formulated for allocating the signatures and is solved using distributed bi-directional training. Starting with full-rank transmission, the precoders are iteratively updated, and the optimized rank for a mobile is the number of signatures assigned significant power after convergence. For systems with a large number of antennas and sub-carriers, it may take many iterations to converge. To reduce the training period, we propose using deep learning for estimating the ranks across mobiles before the precoder optimization phase. With a trained Deep Neural Network (DNN), each mobile estimates its own transmission rank directly from its local channel information and locations of neighboring mobiles. Simulation results show that the DNNs can significantly reduce the training period for precoder adaptation while achieving performance close to that with jointly optimized precoders and ranks. This is illustrated with different system parameters and mobile distributions.

We study distributed algorithms for joint adaptation of precoding and combining filters in frequency-division duplex (FDD) multiple-input multiple-output (MIMO) cellular systems. Neither the base stations nor the mobiles have a priori channel state information, and the transmit/receive filters are directly adapted through training. We propose extensions of bi- directional training (BiT), designed for time- division duplex (TDD) systems, to FDD systems where uplink-downlink reciprocity may not apply. A direct application of BiT can give mismatched precoders with substantial performance degradation. Our approach assumes angular reciprocity, that is, the channel multipath, which causes the frequency selectivity, is characterized by angles of arrival/departure that vary predictably with frequency. Hence spatial beams corresponding to angles of arrival can be turned around to point towards the corresponding angles of departure in the paired band. We present three methods based on this general approach with different complexity for angular estimation and selection. Simulation results indicate that when the multipath is sufficiently sparse, most of the achievable gain with channel reciprocity and TDD can be recovered.

This paper studies initial beam acquisition in a millimeter wave network consisting of multiple access points (APs) and mobile devices. A training protocol for joint estimation of transmit and receive beams is presented with a general frame structure consisting of an initial access sub-frame followed by data transmission sub-frames. During the initial subframe, APs and mobiles sweep through a set of beams and determine the best transmit and receive beams via a handshake. All pilot signals are narrowband (tones), and the mobiles are distinguished by their assigned pilot frequencies. Both non-coherent and coherent beam estimation methods based on, respectively, power detection and maximum likelihood (ML) are presented. To avoid exchanging information about beamforming vectors between APs and mobiles, a local maximum likelihood (LML) algorithm is also presented. An efficient fast Fourier transform implementation is proposed for ML and LML to achieve high-resolution. A system-level optimization is performed in which the frame length, training time, and training bandwidth are selected to maximize a rate objective taking into account blockage and mobility. Simulation results based on a realistic network topology are presented to compare the performance of different estimation methods and training codebooks, and demonstrate the effectiveness of the proposed protocol.

This paper presents a content-centric transmission design in a cloud radio access network by incorporating multicasting and caching. Users requesting the same content form a multicast group and are served by a same cluster of base stations (BSs) cooperatively. Each BS has a local cache, and it acquires the requested contents either from its local cache or from the central processor via backhaul links. We investigate the dynamic content-centric BS clustering and multicast beamforming with respect to both channel condition and caching status. We first formulate a mixed-integer nonlinear programming problem of minimizing the weighted sum of backhaul cost and transmit power under the quality-of-service constraint for each multicast group. Theoretical analysis reveals that all the BSs caching a requested content can be included in the BS cluster of this content, regardless of the channel conditions. Then, we reformulate an equivalent sparse multicast beamforming (SBF) problem. By adopting smoothed ℓ0-norm approximation and other techniques, the SBF problem is transformed into the difference of convex programs and effectively solved using the convex-concave procedure algorithms. Simulation results demonstrate significant advantage of the proposed content-centric transmission. The effects of heuristic caching strategies are also evaluated.

Multicast transmission and wireless caching are effective ways of reducing air and backhaul traffic load in wireless networks. This paper proposes to incorporate these two key ideas for content-centric transmission in a cloud radio access network (RAN) where multiple base stations (BSs) are connected to a central processor (CP) via finite-capacity backhaul links. Each BS has a cache with finite storage size and is equipped with multiple antennas. The BSs cooperatively transmit contents, either stored in the local cache or fetched from the CP, to multiple users in the network. Users requesting a same content form a multicast group and are served by a same cluster of BSs cooperatively using multicast beamforming. Assuming fixed cache placement, this paper investigates the joint design of multicast beamforming and content-centric BS clustering by formulating an optimization problem of minimizing the total network cost under the quality-of-service (QoS) constraints for each multicast group. The network cost involves both the transmission power and the backhaul cost. We model the backhaul cost using the mixed ℓ0/ℓ2-norm of beamforming vectors. To solve this non-convex problem, we first approximate it using the semidefinite relaxation (SDR) method and concave smooth functions. We then propose a difference of convex functions (DC) programming algorithm to obtain suboptimal solutions and show the connection of three smooth functions. Simulation results validate the advantage of multicasting and show the effects of different cache size and caching policies in cloud RAN.

In this paper we consider the multicast transmission in massive MIMO systems where the base station (BS) serves multiple multicast groups simultaneously. Each single-antenna user takes interest in multiple groups but can be only assigned to one interested group during each transmission interval. We study the joint optimization of beamformer design and user grouping under the max-min fairness objective. First, when the BS has the perfect channel state information (CSI) of all users, we show that the asymptotic signal to interference plus noise ratio (SINR) is independent of user grouping and derive the optimal beamformers in closed-form. Second, we consider the imperfect CSI case in time-division duplex (TDD) mode where the BS obtains the downlink channel as the reciprocity of the estimated uplink channel. Instead of estimating every individual user channels, the BS estimates the composite channel of each multicast group. With pilot power control and beamformer power allocation, we propose an optimal user grouping algorithm and a low-complexity suboptimal user grouping algorithm. Numerical and simulation results are presented to demonstrate the performances of proposed algorithms.