Remote direct memory access (RDMA) technology has significantly enhanced network bandwidth and decreased transmission latency through kernel bypass and protocol offloading, overcoming obstacles in distributed computing systems. However, with the deployment of more intricate services in RDMA networks, current RDMA network interface cards (RNICs) have experienced a notable performance decline as the number of queue pair (QP) connections increases, substantially constraining the broad acceptance of RDMA networks. To address this challenge, this article proposes a novel RNIC architecture with high connection scalability. This architecture incorporates a multitiered cache structure to handle diverse communication contexts, enabling RNIC to support ultrahigh QP connection numbers while minimizing on-chip memory usage. In addition, the architecture facilitates chain prefetching, allowing on-chip caches to manage multiple concurrent requests; thus, averting latency resulting from cache misses and access conflicts during communication under concurrent multiple QP scenarios. This ensures transmission performance in multi-QPs connection scenarios. This article implements and validates the performance of a 100G RNIC based on this architecture on Xilinx’s U280 FPGA. With approximately 1 M memory usage on-chip for context, it can support 64 K performant QP connections ($25\times $than CX-6) and can be extended if necessary. Experimental results confirm the high connection scalability of the RNIC, achieving approximately 92 Gb/s network throughput for data packet transmission with concurrent execution of 1–64 K QPs.