Time series forecasting (TSF) is a prominent chal-lenge in data analytics, relevant to both scientific research and real-world industrial applications. The rapid increase in high-dimensional time series data has led researchers to develop numerous models capable of handling complex forecasting tasks across diverse scenarios. Nevertheless, selecting an appropriate model and optimizing its parameters-an issue known as the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem-remains a significant challenge. It is worth investigating how to satisfy both accuracy and efficiency in selecting an optimal algorithm and its hyperparameter configu-ration for a given time series with minimal human intervention. Unfortunately, there is no such work in the field of TSF, which has been developed for more than a decade. Existing methods suffer low selection rate of optimal algorithms. Meanwhile, the TSF task is extremely algorithm-sensitive, and the prediction performance of different algorithms and hyperparameter settings on the same data varies greatly. In this paper, we propose a Proxy-Model-based meta-learning TSF-CASH approach named Auto- Tsf. In the offline training phase, Auto- Tsfextracts the historical experience based on the proxy models, which is used to guide the automatic algorithm selection in the online working phase. The historical experience extracted in the offline phase not only significantly reduces the time consumption for algorithm selection, but also the introduction of the proxy model enhances the optimal algorithm selection rate. Moreover, we propose an asynchronous parallel HPO method in the most time-consuming HPO stage, which further improves the efficiency of the whole TSF -CASH. The experimental results demonstrate that Auto- Tsfachieves SOTA in terms of performance and efficiency compared to existing CASH methods.

Entity Matching (EM) is a crucial task in data integration. Compared to deep learning-based EM solutions, tree-based machine learning models are more computationally effective and explainable, making them more applicable in real-world EM scenarios. However, Random Forest-based EM methods select features with a static feature engineering rule set for all attributes. Consequently, they suffer model retraining cost to select features, and can hardly customize to different EM tasks. To tackle this problem, we propose Description-Similarity Rules (DSR) for EM feature engineering. DSR introduces diverse attribute value distribution metrics and data-driven thresholds to traditional EM feature engineering rules. Unfortunately, both the DSR search space and its online model retraining costs are exponential. By pushing the model retraining to the offline stage, our DSR set mining algorithm is one order of magnitude faster than the baseline algorithms, taking only seconds for online selection. Empirically, DSR outperforms the feature engineering methods by 3.75 % on average F1 score, while reaching the state-of-the-art EM performance on several datasets.

The existing Click-Through Rate (CTR) algorithms have their own advantages and are sensitive to hyperparameters. Quickly obtaining a high-performance CTR model for the new task can bring good application effects. However, ordinary users fail to do so due to the lack of domain knowledge. In this paper, we remedy this deficiency by proposing AutoCTR, an efficient meta-learning based Combined Algorithm Selection and Hyperparameter Optimization (CASH) algorithm, to help non-expert users quickly find the best CTR model. In AutoCTR, we introduce the meta-learning technique to make full use of the meta-information w.r.t. CTR to guide for the new CTR task. Specifically, we utilize the meta-information to learn characteristics and representations of CTR algorithms with different settings. We use these meta experiences combined with few evaluation information on the target CTR dataset to efficiently exploring the huge CTR CASH search space for the new task. The CTR model representation method has significant influence on the quality of the learned meta experiences. To further enhance the experiences quality, we also design a Graph Neural Network (GNN) based embedding learning method. This method can link different CTR models through their components, and thus quickly learning higher-quality model representations. Extensive experimental results show that AutoCTR can quickly select suitable CTR models for different CTR tasks. Compared with the existing CASH algorithms, which ignore meta-information or rely on a huge amount of meta-information, AutoCTR is more reasonable and efficient.

Neural predictors have shown great potential in the evaluation process of neural architecture search (NAS). However, current predictor-based approaches overlook the fact that training a predictor necessitates a considerable number of trained neural networks as the labeled training set, which is costly to obtain. Therefore, the critical issue in utilizing predictors for NAS is to train a high-performance predictor using as few trained neural networks as possible. Although some methods attempt to address this problem through unsupervised learning, they often result in inaccurate predictions. We argue that the unsupervised tasks intended for the common graph data are too challenging for neural networks, causing unsupervised training to be susceptible to performance crashes in NAS. To address this issue, we propose a CurricuLum-guided Contrastive Learning framework for neural Predictor (DCLP). Our method simplifies the contrastive task by designing a novel curriculum to enhance the stability of unlabeled training data distribution during contrastive training. Specifically, we propose a scheduler that ranks the training data according to the contrastive difficulty of each data and then inputs them to the contrastive learner in order. This approach concentrates the training data distribution and makes contrastive training more efficient. By using our method, the contrastive learner incrementally learns feature representations via unsupervised data on a smooth learning curve, avoiding performance crashes that may occur with excessively variable training data distributions. We experimentally demonstrate that DCLP has high accuracy and efficiency compared with existing predictors, and shows promising potential to discover superior architectures in various search spaces when combined with search strategies. Our code is available at: https://github.com/Zhengsh123/DCLP.

In this era of exploding data volumes, more and more complex data analysis tasks are now accomplished by machine learning (ML) or deep learning (DL). Despite the powerful and flexible task processing capability, it also brings challenges such as how to reasonably design the optimal hyperparameters configuration and the huge consumption of time for a single validation. Existing Hyperrarameter Optimization (HPO) methods are gradually facing performance bottlenecks. However, in an era where progressively data-centric big data analytics methods are prevalent, such efficiency issues need to be urgently addressed in the design of intelligent DBMS. In this paper, we propose ShrinkHPO, an efficient and explainable-designed HPO approach with a major focus on ($a$) efficient hyperparameter configuration search strategy, (b) asynchronous executing intervention, and (c) XAI (eXplainable AI) design. ShrinkHPO employs a hyperparameter weight estimation strategy named Shrink-search together with an asynchronous execution design to improve HPO efficiency. To the best of our knowledge, ShrinkHPO is the first HPO method that introduces explainable analysis and verification to ensure the judgment of “high significance” hyperparameters. We also conduct a series of experiments on data analysis tasks in the database domain (classification, regression, time series classification, CASH [1], etc.), collecting HPO results on the usual ML or DL models in each task. Compared to SOTA asynchronous and sequential HPO baselines, ShrinkHPO achieves top performance on both accuracy (RMSE for regression tasks) and time cost, accelerating from 20% to a maximum of 2.2 x.

Data management system configuration optimization has been the focus of database-related research. It can improve the adaptability of DBMS to various business scenarios by selecting the right parameter combination. However, this has been a daunting task because of the vast number of configuration “knobs” involved, and the information about the effects of each knob usually comes from valuable experience, which is very labor-intensive. While machine learning techniques show promise in certain database tasks, they encounter challenges when applied to configuration optimization. Traditional machine learning models excel in prediction tasks through data fitting, but they lack direct applicability in configuration tuning. Moreover, the lack of explainability in machine learning poses a significant obstacle in quantifying the impact of individual knobs on the database performance. Affected by the above factors, this paper proposes CFTune, a method that can accurately evaluate the performance of a DBMS under each configuration using experience and achieve automatic DBMS configuration tuning with counterfactual techniques. Based on various optimization requirements, the approach can offer configuration tuning advice with minimal modification knobs for DBMS that perform badly under specified workloads. Furthermore, to address model shifts resulting from training data, we offer additional strategies to enhance the robustness of the tuning advice. The experimental results demonstrate that the CFTune effectively optimizes the configuration and enhances the performance.

With years of development, a significant number of Time Series Classification (TSC) algorithms have been proposed and applied to various fields such as scientific research and industry scenarios, including traditional statistical methods, machine learning methods, and recently deep learning models. However, choosing a suitable model along with good parameter values that perform well on a given task, which is also known as Combined Algorithm Selection and Hyperparameter optimization problem (CASH), is still challenging. How to automatically select the appropriate algorithm according to the task during analyzing is a topic worthy of further research. Nevertheless, for TSC, a field that has been developed for decades, there is no effective and efficient approach for automatic algorithm selection. To the best of our knowledge, the current approach is based on genetic search, which is very computationally intensive and time-consuming. Therefore, in this paper, we propose TSC-AutoML, a zero-configuration and meta-learning-based approach for the automatic Time Series Classification algorithm CASH (also known as TSC-CASH). TSC-AutoML extracts knowledge from historical tasks and performs automatic feature selection and knowledge filtering with a reinforcement learning policy. The experience extracted is filtered and transformed into metadata. The meta-learner trained on the metadata together with our proposed warm start strategy will select an optimal algorithm for tasks uploaded by users, and then our proposed Hyperparameter Optimization method based on the Fast Warm Start strategy searches for hyperparameter combinations of the selected algorithm and adjusts parameter configuration to achieve top performance. The entire process is pre-trained, automated for the new task, and parameter-free for the user to decide, making it easy for users with the little domain experience to get started easily. Experimental results illustrate that TSC-AutoML outperforms existing methods in terms of both time and accuracy of optimum algorithm selection.

The increasing complexity of data analysis tasks makes it dependent on human expertise and challenging for non-experts. One of the major challenges faced in data analysis is the selection of the proper algorithm for given tasks and data sets. Motivated by this, we develop Assassin, aiming at helping users without enough expertise to automatically select optimal algorithms for classification tasks. By embedding meta-learning techniques and reinforced policy, our system can automatically extract experiences from previous tasks and train a meta-classifier to implement algorithm recommendations. Then we apply genetic search to explore hyperparameter configuration for the selected algorithm. We demonstrate Assassin with classification tasks from OpenML. The system chooses an appropriate algorithm and optimal hyperparameter configuration for them to achieve a high-level performance target. The Assassin has a user-friendly interface that allows users to customize the parameters during the search process.

In many fields, a mass of algorithms with completely different hyperparameters have been developed to address the same type of problems. Choosing the algorithm and hyperparameter setting correctly can promote the overall performance greatly, but users often fail to do so due to the absence of knowledge. How to help users to effectively and quickly select the suitable algorithm and hyperparameter settings for the given task instance is an important research topic nowadays, which is known as the CASH problem. In this paper, we design the Auto-Model approach, which makes full use of known information in the related research paper and introduces hyperparameter optimization techniques, to solve the CASH problem effectively. Auto-Model tremendously reduces the cost of algorithm implementations and hyperparameter configuration space, and thus capable of dealing with the CASH problem efficiently and easily. To demonstrate the benefit of Auto-Model, we compare it with classical Auto-Weka approach. The experimental results show that our proposed approach can provide superior results and achieves better performance in a short time.