Training large language models (LLMs) increasingly relies on geographically distributed accelerators, causing prohibitive communication costs across regions and uneven utilization of heterogeneous hardware. We propose HALoS, a hierarchical asynchronous optimization framework that tackles these issues by introducing local parameter servers (LPSs) within each region and a global parameter server (GPS) that merges updates across regions. This hierarchical design minimizes expensive inter-region communication, reduces straggler effects, and leverages fast intra-region links. We provide a rigorous convergence analysis for HALoS under non-convex objectives, including theoretical guarantees on the role of hierarchical momentum in asynchronous training. Empirically, HALoS attains up to 7.5× faster convergence than synchronous baselines in geo-distributed LLM training and improves upon existing asynchronous methods by up to 2.1×. Crucially, HALoS preserves the model quality of fully synchronous SGD—matching or exceeding accuracy on standard language modeling and downstream benchmarks—while substantially lowering total training time. These results demonstrate that hierarchical, server-side update accumulation and global model merging are powerful tools for scalable, efficient training of new-era LLMs in heterogeneous, geo-distributed environments.

The rapid growth in both computational density and scale in AI models in recent years motivates the construction of an efficient and reliable dedicated network infrastructure. This paper presents the design, implementation, and operation of Meta's Remote Direct Memory Access over Converged Ethernet (RoCE) networks for distributed AI training.

Many applications of sensor networks require the base station to collect all the data generated by sensor nodes. As a consequence many-to-one communication pattern, referred to as convergecast, is prevalent in sensor networks. In this paper, we address the challenge of fast and reliable convergecast on top of the collision-prone CSMA MAC layer. More specifically, we extend previous work by considering the following two situations: (1) the length of the packets generated by nodes is much smaller than the maximum length of a data frame that can be transmitted in one time slot and (2) not every node in the network has data to transmit and for those that have, may have lots of data that require more than one packet. The first situation leads to the possibility of improvement by data piggybacking/aggregation; the second scenario arises in networks where nodes locally store the data and serves query request on-demand. We present distributed minimal time scheduling algorithms for both the cases. Simulation results have shown significant performance improvements of our new approaches over existing solutions.

We consider applications of sensor networks wherein data packets generated by every node have to reach the base station. This results in a many-to-one communication paradigm referred to as convergecast. We are interested in determining a TDMA schedule that minimizes the total time required to complete the convergecast. We consider a simple version of the problem wherein every node generates exactly one packet. We propose a distributed convergecast scheduling algorithm that requires at most 3N timeslots, where N represents the number of nodes in the network. Through extensive simulations, we demonstrate that actual number of timeslots needed is around 1.5N. In addition to time efficiency, we prove that our convergecast scheduling algorithm requires the nodes to buffer no more than two packets at any instance. We propose a sleep schedule that conserves more than 50% of the energy. We present simulation results for a real application scenario to show that our convergecast scheduling algorithm performs significantly better than existing convergecast algorithms.

Efficient use of the limited energy available with sensor nodes is an important consideration for routing protocols. In an attempt to improve the lifetime of a sensor network using energy-aware routing, existing protocols might route a packet along a longer path - thus increasing its latency. Many applications of sensor networks, e.g., surveillance and security, require that the maximum latency in routing a packet be bounded. We consider the problem of energy-aware routing with a bound on the maximum number of hops any packet can traverse to reach a base station. We formulate a linear program (LP) to minimize the maximum energy spent by any node in the network subject to a limit on the number of hops any packet may traverse. A solution to the LP yields routing information in terms of the number of packets to be forwarded along each edge; however, non-integral values of the flow variables might require splitting a packet across multiple routes. As packet splitting incurs additional overhead in tracking various fragments, it is desirable to have integral routing information. We, therefore, propose a rounding algorithm based on the minimum cost flow problem and prove that the energy spent by a node in the resulting integral solution is at most a constant more than the optimal LP solution

We consider the issue of designing a generic transport layer protocol for energy-constrained sensor networks. We present the requirements for such a transport protocol and propose sensor transmission control protocol (STCP). STCP is a generic, scalable and reliable transport layer protocol where a majority of the functionalities are implemented at the base station. STCP offers controlled variable reliability, congestion detection and avoidance, and supports multiple applications in the same network. We present the design and implementation of STCP and evaluate the protocol with different scenarios and network characteristics.

We consider the problem of link scheduling in a sensor network employing a TDMA MAC protocol. Our link scheduling algorithm involves two phases. In the first phase, we assign a color to each edge in the network such that no two edges incident on the same node are assigned the same color. We propose a distributed edge coloring algorithm that needs at most (/spl delta/+1) colors, where /spl delta/ is the maximum degree of the graph. To the best of our knowledge, this is the first distributed algorithm that can edge color a graph with at most (/spl delta/+1) colors. In the second phase, we map each color to a unique timeslot and attempt to identify a direction of transmission along each edge such that the hidden terminal and the exposed terminal problems are avoided. Next, considering topologies for which a feasible solution does not exist, we obtain a direction of transmission for each edge using additional timeslots, if necessary. Finally, we show that reversing the direction of transmission along every edge leads to another feasible direction of transmission. Using both the transmission assignments we obtain a TDMA MAC schedule, which enables two-way communication between every pair of neighbors. For acyclic topologies, we show that at most 2(/spl delta/+1) timeslots are required. Through simulations we show that for sparse graphs with cycles the number of timeslots assigned is close to 2(/spl delta/+1).

Sensor networks consist of a large number of sensor nodes performing distributed sensing and event detection. As sensor nodes are energy-constrained, energy-efficient routing is essential for increasing the lifetime of a sensor network. In Gandham, SR et al. (2003), we proposed an ILP-based method for routing in sensor networks with multiple mobile base stations. The ILP-based method does not guarantee integral routes and bounds on running time. In this paper, we consider static base stations and propose an algorithmic approach to obtain integral energy-efficient routes. We propose to split the lifetime of a sensor network into equal periods of time referred to as rounds and model the energy constrained routing during a round as polynomial-time solvable flow problems. The flow information from an optimum solution to a flow problem is then used as a basis for an energy-efficient routing protocol. Through simulations, we demonstrate that our routing algorithm performs significantly better than the shortest path based algorithms and consumes less energy than the ILP-based method. In addition, we present an algorithm to determine, a priori, lower bound on the lifetime of the sensor network.

One of the main design issues for a sensor network is conservation of the energy available at each sensor node. We propose to deploy multiple, mobile base stations to prolong the lifetime of the sensor network. We split the lifetime of the sensor network into equal periods of time known as rounds. Base stations are relocated at the start of a round. Our method uses an integer linear program to determine new locations for the base stations and a flow-based routing protocol to ensure energy efficient routing during each round. We propose four metrics and evaluate our solution using these metrics. Based on the simulation results we show that employing multiple, mobile base stations in accordance with the solution given by our schemes would significantly increase the lifetime of the sensor network.