In multi-speaker scenarios, leveraging spatial features is essential for enhancing target speech. While with limited microphone arrays, developing a compact multi-channel speech enhancement system remains challenging, especially in extremely low signal-to-noise ratio (SNR) conditions. To tackle this issue, we propose a triple-steering spatial selection method, a flexible framework that uses three steering vectors to guide enhancement and determine the enhancement range. Specifically, we introduce a causal-directed U-Net (CDUNet) model, which takes raw multi-channel speech and the desired enhancement width as inputs. This enables dynamic adjustment of steering vectors based on the target direction and fine-tuning of the enhancement region according to the angular separation between the target and interference signals. Our model with only a dual microphone array, excels in both speech quality and downstream task performance. It operates in real-time with minimal parameters, making it ideal for low-latency, on-device streaming applications.

Mixture-of-Experts (MoE) architectures have emerged as a scalable and efficient alternative to dense Transformer models by activating only a subset of experts per layer. However, deploying MoE models in multi-GPU environments faces severe challenges due to expert load imbalance and the resulting inefficient GPU utilization. Existing static replication strategies fail to adapt to dynamic token distributions, while dynamic rebalancing methods incur excessive communication and memory-access overheads, often outweighing the benefits of load balancing. This paper presents MoEoM, an inference system that enhances the efficiency of MoE models by innovatively taking memory-access costs into consideration. MoEoM integrates two complementary techniques: (i) a load-aware offline expert deployer, which symmetrically groups experts across GPUs and selectively replicates high-load experts, and (ii) an I/O-aware online token reallocator, which dynamically redistributes tokens among original and backup experts to minimize the maximum latency across GPUs. Experimental evaluations on state-of-theart MoE models demonstrate that MoEoM reduces end-toend inference latency by up to 16.6 %, improves throughput by 11–20 % in the prefill stage and 11–23 % in the decode stage, and decreases cross-GPU imbalance by 65–80 % (about 75 % on average), compared to prior MoE inference baselines. These results highlight the importance of incorporating both computation and memory-access overhead into expert placement and token scheduling for efficient large-scale MoE deployment.

Advances in deep neural networks (DNNs) have significantly contributed to the development of real-time video processing applications. Efficient scheduling of DNN workloads in cloud-hosted inference systems is crucial to minimizing serving costs while meeting application latency constraints. However, existing systems suffer from excessive module latency during request dispatching, low execution throughput during module scheduling, and wasted latency budget during latency splitting for multi-DNN applications, which undermines their capability to minimize the serving cost. In this paper, we design a DNN inference system called Harpagon, which minimizes the serving cost under latency constraints with a three-level design. It first maximizes the batch collection rate with a batch-aware request dispatch policy to minimize the module latency. It then maximizes the module throughput with multi-tuple configurations and proper amount of dummy requests. It also carefully splits the end-to-end latency into per-module latency budget to minimize the total serving cost for multi-DNN applications. Evaluation shows that Harpagon outperforms the state of the art by 1.49 to 2.37 times in serving cost while satisfying the latency objectives. Additionally, compared to the optimal solution using brute force search, Harpagon derives the lower bound of serving cost for 91.5% workloads with millisecond level runtime.

The rise of Machine Learning as a Service (MLaaS) has driven the demand for complex and customized real-time inference tasks, often requiring cascading multiple deep neural network (DNN) models into inference pipelines. However, these pipelines pose significant challenges due to scheduling complexity, particularly in maintaining strict latency service level objectives (SLOs). Existing systems serve pipelines with model-independent scheduling policies, which ignore the unique workload characteristics introduced by model cascading in the inference pipeline, leading to SLO violations and resource inefficiencies. In this paper, we propose that the serving system should exploit the model-cascading nature and inter-model workload dependency of the inference pipeline to ensure strict latency SLO cost-effectively. Based on this, we design and implementSLOpt, a serving system optimized for real-time inference pipelines with a three-stage co-design of workload estimation, resource provisioning, and request execution.SLOptproposes cascade workload estimation and ahead-of-time tuning, which together address the challenge of cascade blocking and head-of-line blocking in workload estimation and resource provisioning.SLOptfurther implements an adaptive batch drop policy to mitigate latency amplification issues within the pipeline. These innovations enableSLOptto reduce the 99th percentile latency (P99 latency) by 1.4 to 2.5 times compared to the state of the arts while lowering serving costs by up to 29%. Moreover, to achieve comparable P99 latency,SLOptrequires up to 70% less cost than existing systems. Extensive evaluations on a 64-GPU cluster demonstrateSLOpt’s effectiveness in meeting strict P99 latency SLOs under diverse real-world workloads.