The flourishing of Large Language Models (LLMs) calls for increasingly ultra-scale training. In this paper, we share our experience in designing, deploying, and operating our novel Astral datacenter infrastructure, along with operational lessons and evolutionary insights gained from its production use. Astral has three important innovations: (i) a same-rail interconnection network architecture on tier-2, which enables the scaling of LLM training. To physically deploy this high-density infrastructure, we introduce a distributed high-voltage direct current power system and a new air-liquid integrated cooling system. (ii) a full-stack monitoring system featuring cross-host and hierarchical logging correlation, which diagnoses failures at scale and precisely localizes root causes. (iii) an operator-granular forecasting component Seer that efficiently generates operator execution timelines with acceptable accuracy, aiding in fault diagnosis, model tuning, and network architecture upgrading. Astral infrastructure has been gradually deployed over 18 months, supporting LLM training and inference for multiple customers.

Human mobility prediction is of great importance for various applications such as smart transportation and personalized recommender systems. Although many traditional pattern-based methods and deep models ($e.g.,$ recurrent neural networks) based methods have been developed for this task, they essentially do not well cope with the sparsity and inaccuracy of trajectory data and the complicated high-order nature of the sequential dependency, which are typical challenges in mobility prediction. To solve the problems, this paper proposes a novel framework named G raph C onvolutional D ual-a ttentive N etworks (GCDAN), which consists of two modules: spatio-temporal embedding and trajectory encoder-decoder. The first module employs a bidirectional diffusion graph convolution to preserve the spatial dependency in the location embedding. The second module employs a dual-attentive mechanism based on a Sequence to Sequence architecture to effectively extract the long-range sequential dependency within a trajectory and the correlation between different trajectories for predictions. Extensive experiments on three real-world datasets show that GCDAN achieves significant performance gain compared with state-of-the-art baselines.

In recent years, more and more wireless networks support both 2.4GHz and 5GHz bands. However, in large-scale dual-band wireless networks, lack of understanding on the behavior and performance makes the network diagnosis and optimization extremely challenging. In this paper, we conduct a comprehensive measurement to characterize the behavior and performance in a large-scale dual-band wireless network (TD WLAN). We make several meaningful observations. (1) Although the 5GHz band outperforms the 2.4GHz band, 60% of devices tend to be associated with the 2.4GHz band. The device association behavior has a large impact on the performance. (2) Rogue and non-WiFi devices are prevalent, wherein hidden terminal interference increases the average loss rate by 8%, carrier sense interference increases the average WiFi latency by 45%, and RF interference further aggravates both packet loss and channel contention. (3) The dynamic channel assignment strategy is not always effective. On this basis, we propose a novel and easy-to-implement strategy to improve the wireless performance by intelligent band navigation and heuristic channel optimization. The actual deployment in TD WLAN shows the packet loss reduces by 40% on average and the WiFi latency for more than 60% of devices is below 5ms.

Dynamic Host Configuration Protocol (DHCP) is widely used to dynamically assign IP addresses to users. However, due to little knowledge on the behavior and performance of DHCP, it is challenging to configure lease time and divide IP addresses for address pools properly in large-scale wireless networks. In this paper, we conduct the largest known measurement on the behavior and performance of DHCP in the wireless network of T University (TWLAN). We find the performance of DHCP is far from satisfactory: (1) The non-authenticated devices lead to a waste of 25% of addresses at the rush hour. (2) Address pool utilization varies greatly under the current address division strategy. (3) A device does not generate traffic for 67% of the lease time on average. Meanwhile, we observe devices of different locations and operating systems show diverse online patterns. A unified lease time setting could result in an inefficient usage of addresses. To address the problems, taking account of authentication information and online patterns, we propose a new leasing strategy. The results show it outperforms three state-of-the-art baselines and reduces the number of assigned addresses by 24% and the average total lease time by 17% without significantly increasing the DHCP server load. Besides, we further propose an adaptive address division strategy to balance the address utilization of pools, which can be deployed in parallel with the new leasing strategy and reduce the risk of address exhaustion.

Anycast has been increasingly deployed for content delivery networks to map clients to their nearby replicas, which relies on the underlying routing. However, the simplicity of operation comes at cost of less precise client-mapping control. Although many works have measured anycast DNS, anycast CDNs, with different service goals and engineering, are still not fully understood. In this paper, we design novel methods and combine large-scale traceroute and HTTP measurement to evaluate the overall client-proximity and inefficient routing of the largest anycast CDN, Cloudflare. We find that 90% paths traverse only 2-4 ASes, which highlights its direct networks providers. By further identifying and characterizing direct providers at finer granularity of facilities, we quantitatively shows that Cloudflare unevenly uses few large transit providers to delivery the majority of contents. Inspired by the observations, we propose an anycast routing pathology and diagnosis methodology. Investigation reveals that few huge providers have outsized impact in that they are not only related to many inter-domain inflations, but also have path inflation inside their own networks, thus deserving priority focus when troubleshooting.

Fraud detection is of great importance because fraudulent behaviors may mislead consumers or bring huge losses to enterprises. Due to the lockstep feature of fraudulent behaviors, fraud detection problem can be viewed as finding suspicious dense blocks in the attributed bipartite graph. In reality, existing attribute-based methods are not adversarially robust, because fraudsters can take some camouflage actions to cover their behavior attributes as normal. More importantly, existing structural information based methods only consider shallow topology structure, making their effectiveness sensitive to the density of suspicious blocks. In this paper, we propose a novel deep structure learning model named DeepFD to differentiate normal users and suspicious users. DeepFD can preserve the non-linear graph structure and user behavior information simultaneously. Experimental results on different types of datasets demonstrate that DeepFD outperforms the state-of-the-art baselines.

Dynamic Host Configuration Protocol (DHCP) is widely used to dynamically assign IP addresses. However, due to little knowledge on the behavior and performance of DHCP, it is challenging to configure a proper lease time in complicated wireless network. In this paper, we conduct the largest known measurement on the behavior and performance of DHCP based on the wireless network of T University (TWLAN). TWLAN has more than 59,000 users, 10,000 wireless access points and 130,000 IP addresses. We find the performance of DHCP is far from satisfactory: (1) The non-authenticated devices lead to a waste of 25% of IP addresses at the rush hour. (2) A device does not generate traffic for 67 % of the lease time on average. Meanwhile, we find devices of different locations and operating systems show diverse online patterns. A unified lease time setting could result in an inefficient utilization of addresses. To address the problems, taking account of authentication information and device online patterns, we propose a new leasing strategy. The results show it reduces the number of assigned addresses by 24 % and reduces the time during which an IP address is occupied by 17 % without sianificantly Increasing the DHCP server load.