Vector data management systems (VDMSs) have become an indispensable cornerstone in large-scale information retrieval and machine learning systems like large language models. To enhance the efficiency and flexibility of similarity search, VDMS exposes many tunable index parameters and system parameters for users to specify. However, due to the inherent characteristics of VDMS, automatic performance tuning for VDMS faces several critical challenges, which cannot be well addressed by the existing auto-tuning methods. In this paper, we introduce VDTuner, a learning-based automatic performance tuning framework for VDMS, leveraging multi-objective Bayesian optimization. VDTuner overcomes the challenges associated with VDMS by efficiently exploring a complex multi-dimensional parameter space without requiring any prior knowledge. Moreover, it is able to achieve a good balance between search speed and recall rate, delivering an optimal configuration. Extensive evaluations demonstrate that VDTuner can markedly improve VDMS performance (14.12% in search speed and 186.38 % in recall rate) compared with default setting, and is more efficient compared with state-of-the-art baselines (up to 3.57 x faster in terms of tuning time). In addition, VDTuner is scalable to specific user preference and cost-aware optimization objective. VDTuner is available online at https://github.com/tiannuo-yanWVDTuner.

Workload consolidation is a widely used approach to enhance resource utilization in modern data centers. However, the concurrent execution of multiple jobs on a shared server introduces contention for essential shared resources such as CPU cores, Last Level Cache, and memory bandwidth. This contention negatively impacts job performance, leading to significant degradation in throughput. To mitigate resource contention, effective resource isolation techniques at the software or hardware level can be employed to partition the shared resources among colocated jobs. However, existing solutions for resource partitioning often assume a limited number of jobs that can be colocated, making them unsuitable for scenarios with a large-scale job colocation due to several critical challenges. In this study, we propose Lavender, a framework specifically designed for addressing large-scale resource partitioning problems. Lavender incorporates several key techniques to tackle the challenges associated with large-scale resource partitioning, ensuring efficiency, adaptivity, and optimality. We conducted comprehensive evaluations of Lavender to validate its performance and analyze the reasons for its advantages. The experimental results demonstrate that Lavender significantly outperforms state-of-the-art baselines. Lavender is publicly available at https://github.com/yanxiaoqi932/OpenSourceLavender.

Colocating multiple throughput-oriented jobs on the same server is a commonly used approach for improving system throughput in modern datacenters. The shared resources of the server are usually partitioned among the colocated jobs in order to prevent performance interference caused by resource contention. However, how to properly partition the shared resources among the colocated jobs is nontrival, because it usually has to trade off between two conflict objectives, as datacenter manager wants to maximize server throughput while job owners hope to experience a fair slowdown. Moreover, a desirable resource partitioning strategy should also be efficient, autonomous and adaptive. So far, this problem is not well-addressed in the literature due to several critical challenges. In this paper, we propose an online learning based framework, namedOrchid, to address the multi-objective resource partitioning problem. Orchid leverages contextual multi-armed bandit (CMAB) to model the resource partitioning problem and uses a light-weight online learning algorithm to learn the optimal partitioning configuration according to some easy-to-collect runtime system status. Orchid has two distinguished properties compared to the existing solutions: first, it has the ability to trade off between the two objectives flexibly according to the aspiration of decision maker; second, it has the awareness about runtime system status, which can help to improve the efficiency of finding the optimal partitioning configuration and the adaptivity to dynamic environment changes. Moreover, Orchid does not require any prior knowledge of jobs, and incurs a very small computational overhead. Our evaluations show that Orchid achieves all the desired properties of a good resource partitioning strategy, which outperforms the state-of-the-art baselines with significant margins.Orchidis publicly available athttps://github.com/OpenSourceOrchid/Orchid.