Relying on ground-based infrastructure for model training in Low Earth Orbit (LEO) constellations introduces significant challenges, including intermittent and costly spaceground communication. To address these challenges, this paper proposes a decentralized learning framework that enables model training directly within satellite constellations, utilizing intra-and inter-plane inter-satellite links (ISLs) for information exchange. The paper develops a geo-spatial filtering mechanism that selects the most relevant satellites to form a smaller, more efficient constellation. It then introduces a novel Adaptive Halving-Doubling (AHD) algorithm that enables collective communication within the emerging partial ring topology. Experimental results validate the framework's effectiveness, efficiency, and scalability. Notably, the transmission energy was observed to constitute less than 10% of traditional full constellation (FC) setups, even with increasing model sizes. Moreover, communication overhead relative to FC setups decreased with increasing constellation size, highlighting the operational efficiency of the framework.

Deep learning (DL), despite its success in various fields, remains expensive and inaccessible to many due to its need for powerful supercomputing and high-end GPUs. This study explores alternative computing infrastructure and methods for distributed DL on low-energy, low-cost devices. We experiment on Raspberry Pi 4 devices with ARM Cortex-A72 processors and train a ResNet-18 model on the CIFAR-10 dataset. Our findings reveal limitations and opportunities for future optimizations, paving the way for a DL toolset for low-energy edge devices.

The OPTIMA project aims to port and optimize industrial applications and a set of open-source libraries into two novel FPGA-populated HPC systems. Target applications are from the domains of robotics simulation, underground analysis and computational fluid dynamics (CFD), where data processing is based on differential equations, matrix-matrix and matrix-vector operations. Moreover, the OPTIMA OPen Source (OOPS) library will support basic linear algebraic operations, sparse matrix-vector arithmetic, as well as computer-aided engineering (CAE) solvers. The OPTIMA target platforms are JUMAX, an HPC system that couples an AMD Epyc Server with Maxeler FPGA-based Dataflow Engines (DFEs), and server class machines with Alveo FPGA cards installed. Experimental results show that performance on robotic simulation can be enhanced up to 1.2x, and CFD calculations up to 4.7x. Finally, BLAS L1 routines are improved up to 7x, with a performance-per-Watt ratio boost of more than 40x compared to multi-threaded software routines from the Intel Math Kernel Library (MKL) suite when executed on an Intel Xeon server-class machine.

OPTIMA is an SME-driven project (intermediate stage) that aims to port and optimize industrial applications and a set of open-source libraries into two novel FPGA-populated HPC systems. Target applications are from the domain of robotics simulation, underground analysis and computational fluid dy-namics (CFD), where data processing is based on differential equations, matrix-matrix and matrix-vector operations. Moreover, the OPTIMA OPen Source (OOPS) library will support basic linear algebraic operations, sparse matrix-vector arithmetic, as well as computer-aided engineering (CAE) solvers. The OPTIMA target platforms are JUMAX, an HPC system that couples an AMD Epyc Server with Maxeler FPGA-based Dataflow Engines (DFEs), and server-class machines with Alveo FPGA cards in-stalled. Experimental results on applications up to now, show that performance on robotic simulation can be enhanced up to 1.2x, CFD calculations up to 4.7x, and BLAS routines up to 7x compared to optimized software implementations from OpenBLAS.

Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism.