Translation Lookaside Buffers (TLBs) significantly speed up CPU memory accesses by caching recent address translations. The operating system, unaware of the specific core using a mapping, flushes TLB entries on all cores the application is running on when addresses are unmapped to ensure security and consistency. These TLB shootdowns are expensive and bottleneck system performance and scalability. A primary cause of TLB shootdowns is memory-mapped I/O, especially during mmap-munmap cycles and page cache evictions. After deallocations, the same physical pages are often reassigned to the same process, providing an opportunity to minimize TLB shootdowns. We show that avoiding TLB shootdowns for these “recycled pages” becomes possible by modestly extending the mmap function. Therefore, we implement a “fast page recycling” (FPR) option to the mmap system call. FPR-mmaps ensure security by only sending TLB shootdowns, when a page leaves its circles of recycling and when it is assigned to another process. To guarantee consistency when FPR-mmap pointers are accessed, we adapted the virtual memory management to circumvent the ABA-problem. In contrast to previous attempts to overcome the effects of shootdowns, we do not require any hardware modifications and are able to run transparently within the Linux memory system. We evaluated various CPU, memory, and storage configurations, including persistent memory and Optane SSDs and show that FPR achieves significant performance improvements of up to $28 \%$ for real-world applications and ${9 2 \%}$ for micro-benchmarks. In addition, we show that shootdowns are responsible for bottlenecks previously attributed to other parts of the Linux kernel.

With evolving techniques to parallelize Deep Learning (DL) and the growing amount of training data and model complexity, High-Performance Computing (HPC) has become increasingly important for machine learning engineers. Although many compute clusters already use learning accelerators or GPUs, HPC storage systems are not suitable for the I/O requirements of DL workflows. Therefore, users typically copy the whole training data to the worker nodes or distribute partitions. Because DL depends on randomized input data, prior work stated that partitioning impacts DL accuracy. Their solutions focused mainly on training I/O performance on a high-speed network but did not cover the data stage-in process, for example. We show in this paper that, in practice, (unbiased) partitioning is not harmful for distributed DL accuracy. Nevertheless, manual partitioning can be error prone and inefficient. Typically, data must be unpacked and shuffled before it is distributed to nodes. We propose a solution that features both: efficient stage-in and fast access to a global namespace to prevent biases. Our architecture is based around an ad hoc storage system relying on a high-speed interconnect allowing an efficient stage-in of DL data sets into a single global namespace. Our proposed solution does not limit access to parts of the data set or relies on data duplication, also relieving the HPC storage system. We obtain high I/O performance during training and ensure minimal interference with communication of the learning workers. The optimizations are transparent to DL applications and their accuracy is not affected by our architecture.

The availability of non-volatile main memory (NVMM) has started a new era for storage systems and NVMM specific file systems can support extremely high data and metadata rates, which are required by many HPC and data-intensive applications. Scaling metadata performance within NVMM file systems is nevertheless often restricted by the Linux kernel storage stack, while simply moving metadata management to the user space can compromise security or flexibility.