Large Language Model (LLM) training is increasingly concentrated in homogeneous datacenters, while private data and underutilized GPUs across universities, laboratories, and edge sites remain difficult to use. This extended abstract presents preliminary results from a geo-distributed LLM training prototype that treats networking constraints as first-order design concerns. The prototype connects three heterogeneous GPU sites via cloud-hosted parameter servers, outbound-only gRPC streams, two-stage delta compression (INT8 quantization + Huffman coding, achieving up to 4× payload reduction), and fault-tolerant rejoin. In real deployments, GPT-2 Medium pretraining achieves stable loss reduction and reaches the target loss 15.2% faster in wall-clock time than the best tested baseline; Llama3-1B pretraining remains stable under larger communication pressure; and cross-site latency traces reveal site-dependent WAN spikes of up to 200s. These results motivate adaptive networking support for synchronization, compression, placement, telemetry, and recovery in geo-distributed LLM training.

We study fully decentralized learning in which agents learn collaborative, yet personalized prediction models. Specifically, when learners’ local datasets are non-IID, a collaboratively trained global model (such as those learned through most federated learning algorithms to minimize the sum of losses across all agents) may sacrifice the local performance on agents’ private datasets. To address this issue and enable personalized learning, we propose DiPLe : an algorithm for Directed Personalized Learning. Through our algorithm, each agent identifies "relevant" agents with whom to exchange model information. This leads to a weighted and directed collaboration graph. Agents repeatedly update this graph, and then exchange information with neighboring agents on this learned graph, to collaboratively train their personalized models. We provide analytical results on the generalization error bounds and convergence of our proposed learning method. We verify the performance of DiPLe through numerical experiments, and show its advantages in terms of personalization compared to a number of existing federated learning and personalized learning algorithms.