Brain simulation is one of the most important measures to understand how information is represented and processed in the brain, which usually needs to be realized in supercomputers with a large number of interconnected graphical processing units (GPUs). For the whole human brain simulation, tens of thousands of GPUs are utilized to simulate tens of billions of neurons and tens of trillions of synapses for the living brain to reveal functional connectivity patterns. However, as an application of the irregular spares communication problem on a large-scale system, the sparse and imbalanced communication patterns of the human brain make it particularly challenging to design a communication system for supporting large-scale brain simulations. To face this challenge, this paper proposes a hierarchical regularized communication mechanism, HRCM. The HRCM maintains a hierarchical virtual communication topology (HVCT) with a merge-forward algorithm that exploits the sparsity of neuron interactions to regularize inter-process communications in brain simulations. HRCM also provides a neuron-level partition scheme for assigning neurons to simulation processes to balance the communication load while improving resource utilization. In HRCM, neuron partition is formulated as a k-way graph partition problem and solved efficiently by the proposed hybrid multi-constraint greedy (HMCG) algorithm. HRCM performs finer-grained neuron-level communication control while leveraging voxel-level control as the basis, thus being more effective in balancing inter-process traffic in large-scale simulations. The hierarchical characteristics of the finer-grained communication control are considered by the problem formulation and algorithm design in HRCM. HRCM has been implemented in human brain simulations at the scale of up to 86 billion neurons running on 10000 GPUs. Results obtained from extensive simulation experiments verify the effectiveness of HRCM in significantly reducing communication delay, increasing resource usage, and shortening simulation time for large-scale human brain models.

Inter-process communications form a performance bottleneck for large-scale brain simulations. The sparse and imbalanced communication patterns of human brain make it particularly challenging to design a communication system for supporting large-scale brain simulations. In this paper, we tackle the communication challenges posed by large-scale brain simulations with sparse and imbalanced communication patterns. We design a virtual communication topology with a merge and forward algorithm that exploits the sparsity to regularize inter-process communications. To balance the communication loads of different processes, we formulate voxel partition in brain simulations as a k-way graph partition problem and propose a constrained deterministic greedy algorithm to solve the problem effectively. We conducted extensive simulation experiments for evaluating the performance of the proposed communication scheme and found that the proposed method may significantly reduce communication overheads and shorten simulation time for large-scale brain models.