Feature importance scores (FIS) estimation is an important problem in many data-intensive applications. Traditional approaches can be divided into two types; model-specific methods and model-agnostic methods. In this work, we present FeatureLTE, a novel learning-based approach to FIS estimation. For the first time, as we demonstrate through extensive experiments, it is possible to build general-purpose pre-trained models for FIS estimation. Therefore, FIS estimation reduces to prediction outputs from a pre-trained FeatureLTE model. Pre-trained FeatureLTE models enjoy several desired advantages, including accuracy, robustness, efficiency, and evolvability, and FeatureLTE models really begin to shine on large datasets where traditional methods often find themselves unable to scale. We build our pre-trained models for binary classification and regression problems using observations from nearly 1,000 public datasets. We systematically evaluate various design choices of FeatureLTE model construction and carefully design meta features to make sure that they are computationally lightweight. Based on our evaluation, FeatureLTE is on par with the best existing FIS estimators in terms of FIS quality, and achieves up to 339.48x speedup without sacrificing the quality of FIS estimates on large-scale datasets. Finally, we release two pre-trained FeatureLTE models for binary classification and regression problems that are ready to use on almost all tabular datasets, along with the repository of 701 binary classification datasets and 256 regression datasets with pre-computed feature importance scores to promote future research along this direction.

Recently, the topic of table pre-training has attracted considerable research interest.However, how to employ table pre-training to boost the performance of tabular prediction remains an open challenge.In this paper, we propose TAPTAP, the first attempt that leverages table pre-training to empower models for tabular prediction.After pre-training on a large corpus of real-world tabular data, TAPTAP can generate high-quality synthetic tables to support various applications on tabular data, including privacy protection, low resource regime, missing value imputation, and imbalanced classification.Extensive experiments on 12 datasets demonstrate that TAPTAP outperforms a total of 16 baselines in different scenarios.Meanwhile, it can be easily combined with various backbone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer.Moreover, with the aid of table pre-training, models trained using synthetic data generated by TAPTAP can even compete with models using the original dataset on half of the experimental datasets, marking a milestone in the development of synthetic tabular data generation.The code and datasets are available at https: //github.com/ZhangTP1996/TapTap.

The goal of automated feature generation is to liberate machine learning experts from the laborious task of manual feature generation, which is crucial for improving the learning performance of tabular data. The major challenge in automated feature generation is to efficiently and accurately identify effective features from a vast pool of candidate features. In this paper, we present OpenFE, an automated feature generation tool that provides competitive results against machine learning experts. OpenFE achieves high efficiency and accuracy with two components: 1) a novel feature boosting method for accurately evaluating the incremental performance of candidate features and 2) a two-stage pruning algorithm that performs feature pruning in a coarse-to-fine manner. Extensive experiments on ten benchmark datasets show that OpenFE outperforms existing baseline methods by a large margin. We further evaluate OpenFE in two Kaggle competitions with thousands of data science teams participating. In the two competitions, features generated by OpenFE with a simple baseline model can beat 99.3% and 99.6% data science teams respectively. In addition to the empirical results, we provide a theoretical perspective to show that feature generation can be beneficial in a simple yet representative setting.

Gradient Boosting Decision Tree (GBDT) has achieved remarkable success in a wide variety of applications. The split finding algorithm, which determines the tree construction process, is one of the most crucial components of GBDT. However, the split finding algorithm has long been criticized for its bias towards features with a large number of potential splits. This bias introduces severe interpretability and overfitting issues in GBDT. To this end, we provide a fine-grained analysis of bias in GBDT and demonstrate that the bias originates from 1) the systematic bias in the gain estimation of each split and 2) the bias in the split finding algorithm resulting from the use of the same data to evaluate the split improvement and determine the best split. Based on the analysis, we propose unbiased gain, a new unbiased measurement of gain importance using out-of-bag samples. Moreover, we incorporate the unbiased property into the split finding algorithm and develop UnbiasedGBM to solve the overfitting issue of GBDT. We assess the performance of UnbiasedGBM and unbiased gain in a large-scale empirical study comprising 60 datasets and show that: 1) UnbiasedGBM exhibits better performance than popular GBDT implementations such as LightGBM, XGBoost, and Catboost on average on the 60 datasets and 2) unbiased gain achieves better average performance in feature selection than popular feature importance methods.

In this brief, we define a self-limiting control term, which has the function of guaranteeing the boundedness of variables. Then, we apply it to a finite-time stability control problem. For nonstrict feedback nonlinear systems, a finite-time adaptive control scheme, which contains a piecewise differentiable function, is proposed. This scheme can eliminate the singularity of derivative of a fractional exponential function. By adding a self-limiting term to the controller and the virtual control law of each subsystem, the boundedness of the overall system state is guaranteed. Then the unknown continuous functions are estimated by neural networks (NNs). The output of the closed-loop system tracks the desired trajectory, and the tracking error converges to a small neighborhood of the equilibrium point in finite time. The theoretical results are illustrated by a simulation example.

In this note, we investigate the adaptive quantized or event-triggered control designs for strict-feedback nonlinear systems with time-varying input delay. Because the control signal is discontinuous in the quantized control and event-triggered control, many existing methods to deal with the input delay are no longer applicable. Through constructing an auxiliary tracking error and an auxiliary system, the input-quantized control is implemented for nonlinear systems with unknown control gain and unknown input delay. With the well-designed Lyapunov–Krasovskii functionals and the linear growth condition of input delay, the stability analysis is achieved. The studied method is also applicable to the event-triggered control for systems possessing unknown input delay. The stability analysis shows that all the signals are semiglobally uniformly ultimately bounded (SGUUB). The use of dynamic surface control (DSC) effectively simplifies the controller structure. The quantized and event-triggered control simulations illustrate that the proposed schemes are effective.

In this article, an adaptive quantized neural backstepping strategy is investigated for a class of nonlinear systems with input unmodeled dynamics and output constraints based on the small-gain method. A challenge lies in the considered input-quantized actuator possessing both unknown control gain and input unmodeled dynamics, and the application of the small-gain theorem when the system possesses input unmodeled dynamics and the output constraints. By the coordinate transformation of the state variables, the input unmodeled dynamics subsystem is transformed into a suitable form for applying the small-gain theorem. By a logarithmic one to one mapping, the time-varying output constraints are tackled. With these methods, the stability proof based on the small-gain theorem is completed. It is shown that all the signals are bounded, and the output signal is constrained within the preset range.

In this brief, sufficient conditions are proposed for the existence of the compact sets in the neural network controls. First, we point out that the existence of the compact set in a classical neural network control scheme is unsolved and its result is incomplete. Next, as a simple case, we derive the sufficient condition of the existence of the compact set for the neural network control of first-order systems. Finally, we propose the sufficient condition of the existence of the compact set for the neural-network-based backstepping control of high-order nonlinear systems. The theoretic result is illustrated through a simulation example.

In this paper, adaptive neural dynamic surface control (DSC) is developed using radial basis function neural networks (NNs) for a class of pure-feedback nonlinear systems with full state constraints and dynamic uncertainties. Based on a one-to-one nonlinear mapping, the pure-feedback system with full state constraints is transformed into a novel pure-feedback system without state constraints. The dynamic uncertainties are dealt with using a dynamic signal. Using modified DSC and mean value theorem as well as Nussbaum function, two adaptive NN control schemes are proposed based on the transformed system. The designed control strategy removes the conditions that the upper bound of the control gain is known, and the lower bounds and upper bounds of the virtual control coefficients are known. It is shown that all the signals in the closed-loop system are semi-globally uniformly ultimately bounded, and the full state constraints are not violated. Two numerical examples are provided to illustrate the effectiveness of the proposed approach.

This paper is concerned with the problem of adaptive neural tracking control for a class of uncertain pure-feedback nonlinear systems. Using the implicit function theorem and backstepping technique, a practical robust adaptive neural control scheme is proposed to guarantee that the tracking error converges to an adjusted neighborhood of the origin by choosing appropriate design parameters. In contrast to conventional Lyapunov-based design techniques, an alternative Lyapunov function is constructed for the development of control law and learning algorithms. Differing from the existing results in the literature, the control scheme does not need to compute the derivatives of virtual control signals at each step in backstepping design procedures. Furthermore, the scheme requires the desired trajectory and its first derivative rather than its first n derivatives. In addition, the useful property of the basis function of the radial basis function, which will be used in control design, is explored. Simulation results illustrate the effectiveness of the proposed techniques.

Using mean value theorem and backstepping technique, a robust adaptive fuzzy control scheme is proposed for a class of pure-feedback nonlinear systems with unknown dead zone and disturbances via input-to-state stability. Takagi–Sugeno (T--S) type fuzzy logic systems are used to approximate the uncertain nonlinear functions and fewer learning parameters need to be adjusted online. Based on small gain theorem, the closed-loop control system is proven to be semiglobally uniformly ultimately bounded, and the tracking error converges to a neighborhood of zero by choosing appropriate parameters. Simulation results demonstrate the effectiveness of the control scheme.

In this paper, adaptive neural network (NN) tracking control is investigated for a class of uncertain multiple-input-multiple-output (MIMO) nonlinear systems in triangular control structure with unknown nonsymmetric dead zones and control directions. The design is based on the principle of sliding mode control and the use of Nussbaum-type functions in solving the problem of the completely unknown control directions. It is shown that the dead-zone output can be represented as a simple linear system with a static time-varying gain and bounded disturbance by introducing characteristic function. By utilizing the integral-type Lyapunov function and introducing an adaptive compensation term for the upper bound of the optimal approximation error and the dead-zone disturbance, the closed-loop control system is proved to be semiglobally uniformly ultimately bounded, with tracking errors converging to zero under the condition that the slopes of unknown dead zones are equal. Simulation results demonstrate the effectiveness of the approach.

A new design scheme of direct adaptive neural network controller for a class of nonlinear systems with unknown function control gain is proposed in this paper. The design is based on the principle of sliding mode control and the approximation capability of multilayer neural networks (MNNs). By adopting the adaptive compensation term of the upper bound function of the sum of residual and approximation error, the closed-loop control system is shown to be globally stable, with tracking error converging to zero. Simulation results demonstrate the effectiveness of the approach.