Decision trees are widely used in interpretable expert systems because their if-then rules can be inspected by domain experts. However, training accurate decision trees on large continuous-feature datasets remains difficult. Greedy methods such as CART scale well but optimize each split locally, while exact optimal-tree methods can produce high-quality trees only for relatively small datasets or shallow depths. Existing evolutionary approaches are more flexible, but full-tree search becomes increasingly difficult as tree depth grows. This creates a need for an optimization method that improves decision-tree accuracy for deep trees and large datasets. This paper presents the Moving-Horizon Differential Evolution algorithm for Optimal Classification Trees ( MH-DEOCT ). The proposed method combines three components: a moving-horizon decomposition that replaces full-tree optimization with a sequence of tractable subtree problems, a discrete tree decoding strategy that removes redundant threshold searches, and a GPU-accelerated fitness evaluation exploiting Single Instruction Multiple Data parallelism. Extensive experiments on 65 benchmark datasets show that MH-DEOCT trains depth-8 trees on datasets with up to 60 million samples, improves average test accuracy over CART by 2.73%, and exceeds DL8.5 by 2.27% on medium and large datasets. On a 60-million-sample wearable stress detection dataset, it improves test accuracy over CART by 5.64% at depth 8. Overall, these results show that MH-DEOCT improves predictive accuracy for deep decision trees on large datasets while retaining the transparency of the final tree.

Current methods for document key information extraction (DKIE) rely heavily on labeled data with high annotation costs. To mitigate this issue, the semi-supervised learning (SSL) paradigm, which utilizes unlabeled document samples, has gained broad attention in DKIE. However, existing SSL methods require labeled and unlabeled data to share an identical label space, which is impractical in many DKIE tasks (i.e., some unlabeled samples do not belong to any known classes in the labeled set). In this paper, we formulate this problem as Class-Missing Semi-supervised (CMSS) DKIE. In DKIE, unknown classes usually belong to minority and fine-grained categories, intensifying the misconnections between known and unknown classes and making CMSS more challenging. To address this issue, we propose Synergistic Refinement Estimation (SRE), a progressive prototype estimation scheme that alleviates the unknown classes bias to the majority known classes on long-tailed unlabeled data. Furthermore, dynamic threshold hash rectification and structural calibration mechanisms are proposed to correct connections between fine-grained classes. Extensive experimental results demonstrate that SRE surpasses existing state-of-the-art methods on several DKIE benchmarks. Code is available at https://github.com/anonymoulink/SRE_DKIE .

Decision tree, despite its unmatched interpretability and lightweight structure, faces two key issues that limit its broader applicability: non-differentiability and low testing accuracy. This study addresses these issues by developing a differentiable oblique tree that optimizes the entire tree using gradient-based optimization. We propose an exact reformulation of hard-split trees based on "ReLU+Argmin" mechanism, and then cast the reformulated tree training as an unconstrained optimization task. The ReLU-based sample branching, expressed as exact-zero or non-zero values, preserve a unique decision path, in contrast to soft decision trees with probabilistic routing. The subsequent Argmin operation identifies the unique zero-violation path, enabling deterministic predictions. For effective gradient flow, we approximate Argmin behaviors by scaling softmin function. To ameliorate numerical instability, we propose a warm-start annealing scheme that solves multiple optimization tasks with increasingly accurate approximations. This reformulation alongside distributed GPU parallelism offers strong scalability, supporting 12-depth tree even on million-scale datasets where most baselines fail. Extensive experiments demonstrate that our optimized tree achieves a superior testing accuracy against 14 baselines, including an average improvement of 7.54\% over CART.

In recent years, researchers have proposed the approximation of model predictive control (MPC) using deep neural networks (DNNs). However, a limitation arises as DNNs inherently offer one-to-one mappings, posing a challenge when multiple optimal control inputs correspond to each system state, thereby leading to one-to-many mappings. Therefore, we propose an alternative scheme using mixture networks (MNs) with components of probability (density) distributions in the output layer. This method uses conditional probabilities offered by combining several estimated probability distributions, enabling generating multiple control inputs with the highest probabilities. Notably, this approach is applicable to several problems by choosing a suitable probability distribution, such as using a Gaussian distribution for nonlinear problems and a Bernoulli distribution for mixed-integer linear programming (MILP) problems. We investigate two case studies illustrating that the mixture network-based approximation outperforms the DNN-based approximation.Note to Practitioners—This article proposes using mixture (density) networks, a type of machine learning technique, to enable the online implementation of model predictive control. The prohibitive computation cost associated with model predictive control poses a significant challenge when implementing it in complex nonlinear and MILP problems. While approximation methods of control laws using deep neural networks have been studied to address this issue, it is unsuitable for problems where each state has multiple optimal control inputs. In contrast, our proposed approach can accurately approximate the control laws for these problems. This notable feature can facilitate the online implementation of model predictive control in a wide range of nonlinear and MILP problems.

In the context of proxy modeling for process systems, traditional data-driven deep learning approaches frequently encounter significant challenges, such as substantial training costs induced by large amounts of data, and limited generalization capabilities. As a promising alternative, physics-aware models incorporate partial physics knowledge to ameliorate these challenges. Although demonstrating efficacy, they fall short in terms of exploration depth and universality. To address these shortcomings, we introduce a **p**hysics-**a**ware **p**roxy **m**odel (**PAPM**) that fully incorporates partial prior physics of process systems, which includes multiple input conditions and the general form of conservation relations, resulting in better out-of-sample generalization. Additionally, PAPM contains a holistic temporal-spatial stepping module for flexible adaptation across various process systems. Through systematic comparisons with state-of-the-art pure data-driven and physics-aware models across five two-dimensional benchmarks in nine generalization tasks, PAPM notably achieves an average performance improvement of 6.7%, while requiring fewer FLOPs, and just 1% of the parameters compared to the prior leading method.

$k$-center problem is a well-known clustering method and can be formulated as a mixed-integer nonlinear programming problem. This work provides a practical global optimization algorithm for this task based on a reduced-space spatial branch and bound scheme. This algorithm can guarantee convergence to the global optimum by only branching on the centers of clusters, which is independent of the dataset’s cardinality. In addition, a set of feasibility-based bounds tightening techniques are proposed to narrow down the domain of centers and significantly accelerate the convergence. To demonstrate the capacity of this algorithm, we present computational results on 32 datasets. Notably, for the dataset with 14 million samples and 3 features, the serial implementation of the algorithm can converge to an optimality gap of 0.1% within 2 hours. Compared with a heuristic method, the global optimum obtained by our algorithm can reduce the objective function on average by 30.4%.

The training of optimal decision tree via mixed-integer programming (MIP) has attracted much attention in recent literature. However, for large datasets, state-of-the-art approaches struggle to solve the optimal decision tree training problems to a provable global optimal solution within a reasonable time. In this paper, we reformulate the optimal decision tree training problem as a two-stage optimization problem and propose a tailored reduced-space branch and bound algorithm to train optimal decision tree for the classification tasks with continuous features. We present several structure-exploiting lower and upper bounding methods. The computation of bounds can be decomposed into the solution of many small-scale subproblems and can be naturally parallelized. With these bounding methods, we prove that our algorithm can converge by branching only on variables representing the optimal decision tree structure, which is invariant to the size of datasets. Moreover, we propose a novel sample reduction method that can predetermine the cost of part of samples at each BB node. Combining the sample reduction method with the parallelized bounding strategies, our algorithm can be extremely scalable. Our algorithm can find global optimal solutions on dataset with over 245,000 samples (1000 cores, less than 1% optimality gap, within 2 hours). We test 21 real-world datasets from UCI Repository. The results reveal that for datasets with over 7,000 samples, our algorithm can, on average, improve the training accuracy by 3.6% and testing accuracy by 2.8%, compared to the current state-of-the-art.

We study the deterministic global optimization of the K-Medoids clustering problem. This work proposes a branch and bound (BB) scheme, in which a tailored Lagrangian relaxation method proposed in the 1970s is used to provide a lower bound at each BB node. The lower bounding method already guarantees the maximum gap at the root node. A closed-form solution to the lower bound can be derived analytically without explicitly solving any optimization problems, and its computation can be easily parallelized. Moreover, with this lower bounding method, finite convergence to the global optimal solution can be guaranteed by branching only on the regions of medoids. We also present several tailored bound tightening techniques to reduce the search space and computational cost. Extensive computational studies on 28 machine learning datasets demonstrate that our algorithm can provide a provable global optimal solution with an optimality gap of 0.1\% within 4 hours on datasets with up to one million samples. Besides, our algorithm can obtain better or equal objective values than the heuristic method. A theoretical proof of global convergence for our algorithm is also presented.