Various scientific applications with different I/O characteristics are executed in HPC systems. However, underlying parallel file systems are unaware of these characteristics of applications, and using a single fixed file layout for all applications can degrade the performance of HPC systems. In this paper, we propose A2FL, an autonomous and adaptive file layout adjustment scheme that optimizes parallel file system configurations by analyzing the access pattern of the applications. The key steps of A2FL are as follows: (1) A2FL initially intercepts the I/O operations of the application, recording their access patterns in real-time. (2) The access patterns are then transformed into a graphical representation used for predicting I/O performance and providing adjustment recommendations. (3) A2FL autonomously adjusts the file layout based on the prediction results, delivering an optimal file layout within the parallel file system. Moreover, we propose A2FL-Compound which analyzes an access pattern by dividing it into smaller components to optimize the file layout in a fine-grained manner. Our evaluations demonstrate that A2FL significantly enhances I/O performance, with improvements of up to 65.9× compared to the default file layout.

Distributed file systems (DFSs) are widely used in large scale computing environments such as cloud computing and high performance computing systems (HPC) where thousands of applications are executed simultaneously. To support applications that produce and process a large amount of data, DFSs manage limited storage resources in the system and allocate the resources to the applications. To efficiently utilize limited storage resources, it is important to consider the I/O characteristics of applications as well as the effect of resource sharing among multiple applications in the system. In this paper, we first perform empirical performance analysis and investigate the effect of I/O interference caused by storage resource allocation. Based on our empirical performance analysis, we propose IFLustre towards interference-free and efficient storage allocation in Lustre file system. IFLustre first utilizes previous execution logs to determine the necessary storage resources for the application and predict the throughput and runtime of the application. Then, it allocates storage resources via file system configurations based on the prediction results and the DFS allocation status. This allows IFLustre to allocate sufficient resources to the application while mitigating interference from resource sharing. The experimental results show that IFLustre can improve the performance by up to 47% compared with Lustre.

Parallel file systems (PFS) are used to distribute data processing and establish shared access to large-scale data. Despite being able to provide high I/O bandwidth on each node, PFS has difficulty utilizing the I/O bandwidth due to a single connection between the client and server nodes. To mitigate the performance bottleneck, users increase the number of connections between the nodes by modifying PFS or applications. However, it is difficult to modify PFS itself due to its complicated internal structure. Thus, PFS users manually increase the number of connections between the nodes by employing several methods. In this paper, we propose a user-transparent I/O subsystem, MulConn, to make users exploit high I/O bandwidth between nodes. To avoid the modifications of PFS and user applications, we have developed a horizontal mount procedure and two I/O scheduling policies, TtoS and TtoM, in the virtual file system (VFS) layer. We expose a single mount point that has multiple connections by modifying the mount path of VFS from vertical hierarchy to horizontal hierarchy. We also introduce two I/O scheduling policies to distribute I/O requests evenly to multiple connections. The experimental results show that MulConn improves write and read performance by up to 2.6x and 2.8x, respectively, compared with those of PFS using the existing kernel. In addition, we provide the best I/O performance that PFS can provide in the given experimental environments.