Goal-conditioned humanoid locomotion in reinforcement learning (RL) remains challenging due to sparse reward signals and the single-goal overfitting problem. Although contrastive reinforcement learning (CRL) has achieved considerable success in this setting, it can suffer from pronounced estimation variance, since epistemic uncertainty is difficult to reduce given limited task-specific information and model capacity. Ensemble-based critics can partially alleviate this issue. However, sufficient ensemble diversity and accurate individual estimates are not necessarily guaranteed during training, resulting in unstructured exploration. To address these challenges, we propose Conditional Redundancy-Guided Evolutionary Contrastive Ensemble with Direct Preference Optimization weighting (CRECE-DPO), which augments CRL with a vectorized critic ensemble and refines the ensemble via an evolutionary algorithm guided by a tailored fitness metric. Specifically, we design a DPO-weighted conditional redundancy fitness score, to prune redundant representations while promoting effective exploration of the parameter space. Simulation results on challenging goal-conditioned benchmarks, including humanoid locomotion, demonstrate consistent improvements over CRL and other baselines.

Existing congestion control schemes for RDMA resolve the dilemma of guaranteeing high throughput and ultra-low latency to some extent from a variety of perspectives. However, they are inefficient in addressing transient large queue build-up and under-utilized bandwidth caused by frequent traffic bursts. In this paper, we argue that traffic uncertainty is the fundamental challenge that limits these schemes from addressing the aforementioned dilemma. Inspired by the investigation that aggregated flows within the same rack are relatively long-lived, we propose HierCC, which aggregates flows destined to the same IP in a rack to ease traffic uncertainty and further provides hierarchically control within the first-hop ToR and between racks. Specifically, the inter-rack rates of aggregate flows are controlled by a credit-based mechanism. Then the bandwidth obtained by the aggregated flow is allocated to the corresponding intra-rack individual flows promptly and accurately. We implement HierCC in a testbed that consists of DPDK-based end-hosts and P4-based Tofino switches. The performance of HierCC is evaluated by comprehensive testbed experiments and SystemC/NS3 simulations. Results indicate that, compared with state-of-the-art, HierCC can mitigate buffer usage by up to$10\times $and reduce the average and 99th percentile FCT by up to 84% and 80%, respectively.

Sustainability is becoming increasingly challenging in today's data centers with limited space, power and connectivity. Large cloud service providers interconnect geographically distributed datacenters for better scalability and availability. Applications running on cross-datacenter network impose great challenges in transport design. In this paper, we identify two inherent limitations of extending the existing transport technology, RDMA, and its Ethernet derivative, RoCE, to long-haul transmission. First, the on-chip resources of commodity RDMA NICs are insufficient for long-haul transmission. Second, applying existing traffic control schemes to inter-datacenter environment exhibits poor performance. Motivated by this, we propose Torrent, a switch-driven transport framework which partitions end-to-end control into three sub-control loops. To achieve the combined goals of fairness and high performance in cross-datacenter scenarios, Torrent employs fast acknowledgment and near-end congestion control on datacenter interconnection (DCI) switches. We implement Torrent prototypes on commodity programmable switches and evaluate it through real-world testbed experiments. Our results show that Torrent can achieve high link utilization over ultra-long distances and quickly converge congested flows to steady rates.

RDMA has been widely deployed in production datacenters. The conventional wisdom believes that the intra-host network delivers stable and high performance. However, intra-host resources witness a relative stagnation in technology trends compared to the evolving RDMA NIC (RNIC). Thus, the RNIC traffic may not get sufficient intra-host resources when it contends with CPU-to-memory traffic. A line of recent works from large-scale production datacenter operators demonstrates the emergence of intra-host congestion and associated performance collapse, which forces us to revisit the practice of intra-host congestion control. However, the ability to efficiently control RDMA intra-host networks is far less mature than inter-host networks, which brings challenges in congestion monitoring, intra-host resource allocation and RNIC traffic adjustment. In this paper, we propose RDMA intra-Host Congestion Control (RHCC), which combines CPU-to-memory traffic congestion avoidance with sub-RTT granularity and proactive RNIC traffic adjustment. RHCC ensures fast congestion avoidance and can work with different inter-host congestion control methods. We implement RHCC on commodity servers and RNICs and conduct experiments to evaluate the performance. The results show that RHCC can increase/decrease the network throughput/latency by up to 2$\times$and 1.4$\times$, respectively.

The network-intensive applications, like machine learning and cloud storage, are increasingly driving two critical trends:1)RDMA has been widely deployed to provide high-speed networks;2)applications are distributively deployed across multiple regional datacenters to satisfy demands for content providers and customers. To fully utilize the benefits of RDMA, we desire to extend it to support cross-datacenter networks. However, the long-haul transport suffers a considerably long control loop, and thus the hybrid of long-haul and intra-datacenter traffic can easily cause severe congestion. We revisit existing traffic control methods and find they are insufficient to resolve this hybrid traffic congestion. Generally, regional datacenters are connected using dedicated long-haul optical fiber and datacenter interconnection (DCI) switches. In this paper, we propose Approach Traffic Control (ATC), a novel solution focusing on two-side DCI-switches (i.e., the approach point for datacenters) to separately alleviate the hybrid traffic congestion in the local and distal datacenters, as a building block for host-driven control methods. This design principle helps ATC shorten the control loop to a single datacenter scale while aggregating congestion information of the whole datacenter range with minor deployment complexity. We implement ATC on P4-based switches and conduct evaluations using real-world testbeds and large-scale NS3 simulations. The results show that ATC ensures fast congestion avoidance and delivers significant performance. For example, ATC reduces the FCT of intra-datacenter and long-haul traffic by up to 88% and 52%, respectively.

RDMA has been widely deployed in production datacenters. The conventional wisdom believes that the intra-host network delivers stable and high performance. However, intra-host resources witness a relative stagnation in technology trends compared to the evolving RDMA NIC (RNIC). Thus, the RNIC traffic may not get sufficient intra-host resources when it contends with intra-host traffic. A line of recent works from large-scale production datacenter operators demonstrates the emergence of intra-host congestion and associated performance collapse, which forces us to rethink the practice of intra-host congestion control. However, the ability to efficiently control RDMA intra-host networks is far less mature than inter-host networks, which brings challenges in congestion monitoring, intra-host resource allocation and RNIC traffic adjustment. In this paper, we propose RDMA intra-Host Congestion Control (RHCC), which combines sub-RTT granularity intra-host traffic congestion avoidance and proactive RNIC traffic adjustment. We implement RHCC on commodity servers and RNICs and conduct experiments to evaluate the performance. The results show that RHCC can increase/decrease the network throughput/latency by up to 2 × and 1.4 ×, respectively.

The rapid upgrade of link speed and the prosperity of new applications in data center networks (DCNs) lead to a rigorous demand for ultra-low latency and high throughput. To mitigate the overhead of traditional software-based packet processing at end-hosts, RDMA (Remote Direct Memory Access) has been widely adopted in DCNs. Particularly, congestion control (CC) mechanisms designed for RDMA have attracted much attention to avoid performance deterioration when packets lose. However, through comprehensive analysis, we found that existing RDMA CC schemes have limitations of a sluggish response to congestion and unawareness of tiny microbursts due to the long end-to-end control loop. In this paper, we propose PACC, a proactive and accurate switch-driven RDMA CC algorithm with easy deployability. PACC is driven by PI controller-based computation, threshold-based flow discrimination and weight-based allocation at the switch. It leverages real-time queue length to generate accurate congestion feedback proactively and piggybacks it to the corresponding source without modification to end-hosts. We theoretically analyze the stability, convergence and key parameter settings of PACC. Then, we implement PACC in a testbed consisting of DPDK-based end-hosts and Tofino P4 switches. In our evaluation, PACC achieves better fairness, fast reaction, high throughput, and 6$\sim$69% lower FCT (Flow Completion Time) than DCQCN, TIMELY, HPCC and RoCC.