The Mixture-of-Experts (MoE) structure scales the Transformer-based large language models (LLMs) and improves their performance with only the sub-linear increase in computation resources. Recently, a fine-grained DeepSeekMoE structure is proposed, which can further improve the computing efficiency of MoE without performance degradation. However, the All-to-All communication introduced by MoE has become a bottleneck, especially for the fine-grained structure, which typically involves and activates more experts, hence contributing to heavier communication overhead. In this paper, we propose a novel MoE structure named BigMac, which is also fine-grained but with high communication efficiency. The innovation of BigMac is mainly due to that we abandon the Communicate-Descend-Ascend-Communicate (CDAC) manner used by fine-grained MoE, which leads to the All-to-All communication always taking place at the highest dimension. Instead, BigMac designs an efficient Descend-Communicate-Communicate-Ascend (DCCA) manner. Specifically, we add a descending and ascending projection at the entrance and exit of the expert, respectively, which enables the communication to perform at a very low dimension. Furthermore, to adapt to DCCA, we re-design the structure of small experts, ensuring that the expert in BigMac has enough complexity to address tokens. Experimental results show that BigMac achieves comparable or even better model quality than fine-grained MoEs with the same number of experts and a similar number of total parameters. Equally importantly, BigMac reduces the end-to-end latency by up to 3.09 x for training and increases the throughput by up to 3.11 x for inference on state-of-the-art AI computing frameworks including Megatron, Tutel, and DeepSpeed-Inference.

Dropout [1], a standard regularization technique in Large Language Models, incurs extra computational overhead, particularly due to its repeated application during the training process. To address this issue, we propose Accelerated Dropout, an algorithm that revolutionizes the traditional dropout method by employing a bitmask, rather than a float mask, to significantly expedite the training process. In the theoretical proof, we demonstrate that accelerated dropout is exponentially convergent to the probability of retaining neurons. Our extensive experimental analysis, conducted on 13 benchmark datasets and 9 deep learning models, confirms that accelerated dropout outperforms traditional dropout in terms of training efficiency and generalization performance. The experimental results indicate that, compared to torch dropout, our accelerated dropout achieves 8x speedup on a single dropout operator. With the use of accelerated dropout, the average training speed of each model increased by 1.0624x. The training speed improvements ranged from a 1.046x increase for ProphetNet on the CNN/DailyMail dataset to a 1.077x increase for ViT-B on the ImageNet dataset.

Symbolic regression aims to discover concise, interpretable mathematical expressions that satisfy desired objectives, such as fitting data, posing a highly combinatorial optimization problem. While genetic programming has been the dominant approach, recent efforts have explored reinforcement learning methods for improving search efficiency. Monte Carlo Tree Search (MCTS), with its ability to balance exploration and exploitation through guided search, has emerged as a promising technique for symbolic expression discovery. However, its traditional bandit strategies and sequential symbol construction often limit performance. In this work, we propose an improved MCTS framework for symbolic regression that addresses these limitations through two key innovations: (1) an extreme bandit allocation strategy tailored for identifying globally optimal expressions, with finite-time performance guarantees under polynomial reward decay assumptions; and (2) evolution-inspired state-jumping actions such as mutation and crossover, which enable non-local transitions to promising regions of the search space. These state-jumping actions also reshape the reward landscape during the search process, improving both robustness and efficiency. We conduct a thorough numerical study to the impact of these improvements and benchmark our approach against existing symbolic regression methods on a variety of datasets, including both ground-truth and black-box datasets. Our approach achieves competitive performance with state-of-the-art libraries in terms of recovery rate, attains favorable positions on the Pareto frontier of accuracy versus model complexity.

To meet the prevalence of cloud services, it is of great significance for operators to design a cost-effective tenant placement strategy on physical machines while the services are still guaranteed. Moreover, workload balancing between physical machines is critical for high performance. We characterize the problem of tenant placement as a mixed-integer programming problem that efficiently minimizes the number of active physical machines and balances the workloads among them. To address the NP-hardness of the optimization problem, we first investigate the global lower bound and other inherent properties of the proposed model, and then design two efficient heuristic algorithms. Through numerous simulations at different scales and settings, we demonstrate the superiority of our proposed algorithms over state-of-the-art works in terms of the objective function values and computational time.