ABSTRACT Balancing convergence, diversity and feasibility remains a fundamental challenge in constrained optimization. This paper proposes MOEA/D‐PS, which introduces the Pascoletti‐Serafini (PS) scalarization framework into constrained single‐objective optimization for the first time. The algorithm reformulates constrained single‐objective problems into a bi‐objective optimization by treating constraint violation as a secondary objective, then integrates two synergistic mechanisms to achieve a satisfactory balance of convergence, diversity and feasibility: (1) direction vectors sampled from a normal distribution to guide the population search around the elite solution while preserving exploration capability; (2) an adaptive truncation parameter that dynamically adjusts the search region based on the current elite solution. Unlike standard multi‐objective transformations that waste resources approximating the entire Pareto front, the PS‐guided mechanism geometrically restricts exploration to the trajectory toward the feasible optimum, achieving a unified balance among convergence, diversity and feasibility. Comprehensive experiments on 24 benchmark functions from the CEC 2006 test suite show that MOEA/D‐PS attains the best overall ranking among seven state‐of‐the‐art algorithms.

Differentiable Neural Architecture Search (NAS) and traditional evolutionary approaches frequently struggle with premature convergence to local optima. To overcome this limitation, we propose LLMENAS, a hierarchical framework that introduces trajectory-aware fitness design as the upper-level optimizer. By analyzing the convergence state of historical optimization trajectories, the Large Language Model (LLM) acts as a fitness designer to dynamically design fitness functions. This mechanism enables the search to navigate complex landscapes and escape local optima. Furthermore, we introduce a closed-loop self-improving mechanism, enabling the LLM to iteratively enhance its design strategies through self-reflection and self-refinement based on feedback. Extensive experiments show that LLMENAS achieves competitive results, with top-1 accuracies of 97.58% on CIFAR-10, 83.52% on CIFAR-100, and 75.6% on ImageNet-1k. Furthermore, it is achieved with remarkable efficiency, costing only 0.15 GPU days on the CIFAR 10 datasets and 2 GPU days on ImageNet. The source code is publicly available at: https://github.com/LLMENAS/LLMENAS.

Evolutionary algorithms (EAs) for optimization have received wide attention due to their robustness and practicality. However, the traditional way of asynchronously handling bilevel optimization problems (BLOPs) ignores the benefits brought by effective upper- and lower-level collaboration. To address this issue, this article proposes a collaborative multiobjective transformation (MOT)-based EA (MOTEA-II). In MOTEA-II, the BLOP is handled within a decomposition-based multiobjective optimization paradigm using a two-stage collaborative MOT strategy. The stage-1 MOT focuses on multiple lower-level optimizations and collaboration, while stage-2 collaborates the upper-level optimization with lower-level optimization, which makes simultaneously horizontal and vertical optimization information sharing in bilevel optimization possible. In addition, a dynamic decomposition strategy is further proposed to reconstruct the hierarchy relationship in collaborative multiobjective optimization, facilitating the adaptive and flexible importance control of the upper-level objective optimization and lower-level optimality satisfaction for better-bilevel search efficiency. Empirical studies are conducted on two groups of commonly used BLOP benchmark suites and four practical applications. Experimental results show that the proposed collaborative MOTEA-II can achieve performance comparable to that of the previous MOTEA and three other representative EA-based bilevel optimization approaches, but using much fewer computational resources.

Differentiable Architecture Search (DARTS) has achieved a rapid search for excellent architectures by optimizing architecture parameters through gradient descent. However, this efficiency comes with a significant challenge: the risk of premature convergence to local optima, resulting in subpar performance that falls short of expectations. To address this issue, we propose a novel and effective method called Evolutionary Gradient-Based Neural Architecture Search (EG-NAS). Our approach combines the strengths of both gradient descent and evolutionary strategy, allowing for the exploration of various optimization directions during the architecture search process. To begin with, we continue to employ gradient descent for updating network parameters to ensure efficiency. Subsequently, to mitigate the risk of premature convergence, we introduce an evolutionary strategy with global search capabilities to optimize the architecture parameters. By leveraging the best of both worlds, our method strikes a balance between efficient exploration and exploitation of the search space. Moreover, we have redefined the fitness function to not only consider accuracy but also account for individual similarity. This inclusion enhances the diversity and accuracy of the optimized directions identified by the evolutionary strategy. Extensive experiments on various datasets and search spaces demonstrate that EG-NAS achieves highly competitive performance at significantly low search costs compared to state-of-the-art methods. The code is available at https://github.com/caicaicheng/EG-NAS.

The bilevel optimization problem is a kind of commonly existing optimization problem, which includes a nested lower-level optimization problem as a constraint condition. The nested lower-level optimization problem should be solved with every upper-level decision fixed as a parameter. Consequently, it is usually computationally very expensive to solve bilevel optimization problems. In this paper, we propose a bilevel evolutionary algorithm based on upper-level-driven lower-level search (BLEA-UDLS). Driven by the upper-level optimization, the lower-level search in BLEA-UDLS is carried out on some upper-level superior solutions rather than equally and indiscriminately on all solutions, which makes sure that the front solutions of the population have more accurate lower-level decisions and saves lots of evaluation budgets on less important solutions. In the lower-level search, the lower-level decisions of different solutions are optimized cooperatively with the computation resources dynamically adjusted, where more computation resources are assigned for less explored solutions. Compared with some other bilevel evolutionary algorithms, the experimental results have confirmed the effectiveness of the proposed BLEA-UDLS for solving BLOPs and meanwhile saving evaluation budgets.

Face recognition (FR) has encountered great difficulties due to the label noise and low-quality samples in the face recognition database. Previous studies addressed these issues through hard sample mining, focusing on preventing the overfitting of label noise and low-quality samples. However, existing loss functions struggle to handle both label noise and low-quality samples simultaneously. This paper proposes a novel loss function, NLMFace, to address these challenges. By considering a sample’s ground truth class center and its nearest negative class center, NLMFace incorporates a new mining framework with the margin-based loss function. This method adaptively corrects label noise and identifies high-quality samples through data mining. Extensive experiments on CASIA WebFace datasets, along with evaluations on benchmarks like LFW, CPLFW, and CALFW, demonstrate the superior performance of NLMFace.

In clinical dictation, utterances after automatic speech recognition (ASR) without explicit punctuation marks may lead to the misunderstanding of dictated reports. To provide a precise and understandable clinical report with ASR, automatic punctuation restoration (APR) is required. Considering a practical scenario, we propose a fast and lightweight pre-trained model for Chinese medical punctuation restoration based on the 'pre-training and fine-tuning' paradigm. In this work, we distill pre-trained models by incorporating supervised contrastive learning and a novel auxiliary pre-training task (Punctuation Mark Prediction) to make it well-suited for punctuation restoration. We then reformulate APR as a slot tagging problem in the fine-tuning stage to bridge the gap between pre-training and fine-tuning. Our experiments on various distilled models reveal that our model can achieve 95% performance with a 10% model size relative to the state-of-the-art Chinese RoBERTa. © 2024 International Speech Communication Association. All rights reserved.

Nested evolutionary algorithms (EAs) have been regarded as very promising tools for bi-level optimization. Due to the nested structure, the upper level population evaluation requires a set of complete lower level optimizations, thereby reducing the efficiency and practicability of EA methods. In this paper, a multi-objective transformation-based evolutionary algorithm (MOTEA) is proposed to perform multiple lower level optimizations in a parallel and collaborative manner. Specifically, the corresponding multiple lower level optimizations for each generation of the upper level population evaluation are transformed into locating a set of Pareto optimal solutions of a constructed multi-objective optimization problem. By utilizing the built-in implicit parallelism of evolutionary multi-objective optimization, multiple lower level problems can thus be optimized in parallel. Within one multi-objective search population, the collaboration among the parallel lower level optimization can be realized by exploiting and utilizing the implicit similarities among them for better efficiency. The effectiveness and efficiency of the proposed MOTEA are verified by comparing it with four state-of-the-art evolutionary bi-level optimization algorithms on two sets of popular bi-level optimization benchmark test problems and three application problems.

In this paper, we propose a stochastic bilevel optimization approach based on a hypergradient estimator, called BHE- DARTS, as a remedy for this issue that it is easy to search for locally optimal structures rather than globally optimal ones in Differentiable Architecture Search (DARTS) bilevel optimization model. To be specific, we apply a stochastic gradient for updating the lower level variable ω and design a hypergradient estimator, which is built by the Jacobian- and Hessianvector product, to assist in updating the upper level variable α. This operation can more fully apply the gradient information to escape the trap of local optimal in the NAS bilevel model. Compared to state-of-the-art DARTS methods, experimental studies have shown the competitive performance of the proposed BHE-DARTS in the DARTS search space (CIFAR-100: a test accuracy rate of 82.69 % ) and NAS-Bench-201 search space (ImageNet16-120: a test accuracy rate of 42.44%).

Evolutionary algorithms (EAs) have been recognized as a promising approach for bilevel optimization. However, the population-based characteristic of EAs largely influences their efficiency and effectiveness due to the nested structure of the two levels of optimization problems. In this article, we propose a transfer learning-based parallel EA (TLEA) framework for bilevel optimization. In this framework, the task of optimizing a set of lower level problems parameterized by upper level variables is conducted in a parallel manner. In the meanwhile, a transfer learning strategy is developed to improve the effectiveness of each lower level search (LLS) process. In practice, we implement two versions of the TLEA: the first version uses the covariance matrix adaptation evolutionary strategy and the second version uses the differential evolution as the evolutionary operator in lower level optimization. The experimental studies on two sets of widely used bilevel optimization benchmark problems are conducted, and the performance of the two TLEA implementations is compared to that of four well-established evolutionary bilevel optimization algorithms to verify the effectiveness and efficiency of the proposed algorithm framework.

Evolutionary algorithms for bi-level optimization suffer from the low efficiency in dealing with the multiple lower level optimization tasks required by the population based upper level search. In this paper, we design a transfer learning based covariance matrix adaptation evolution strategy (TL-CMA-ES) for bi-level optimization problems, where the tasks of searching for multiple lower level optimal solutions are conducted by a set of CMA-ES optimizers in a parallel manner. Furthermore, a transfer learning strategy is introduced in the parallel lower level CMA-ES search such that each CMA-ES optimizer can learn and utilize useful features gained by its neighbours. Experimental comparison and analysis are carried out to verify the effectiveness and efficiency of the proposed method.

An objective normalization strategy is essential in any evolutionary multiobjective or many-objective optimization (EMO or EMaO) algorithm, due to the distance calculations between objective vectors required to compute diversity and convergence of population members. For the decomposition-based EMO/EMaO algorithms involving the Penalty Boundary Intersection (PBI) metric, normalization is an important matter due to the computation of two distance metrics. In this article, we make a theoretical analysis of the effect of instabilities in the normalization process on the performance of PBI-based MOEA/D and a proposed PBI-based NSGA-III procedure. Although the effect is well recognized in the literature, few theoretical studies have been done so far to understand its true nature and the choice of a suitable penalty parameter value for an arbitrary problem. The developed theoretical results have been corroborated with extensive experimental results on three to 15-objective convex and non-convex instances of DTLZ and WFG problems. The article, makes important theoretical conclusions on PBI-based decomposition algorithms derived from the study.

Decomposition-based evolutionary algorithms have shown great potential in many-objective optimization. However, the lack of theoretical studies on decomposition methods has hindered their further development and application. In this paper, we first theoretically prove that weight sum, Tchebycheff, and penalty boundary intersection decomposition methods are essentially interconnected. Inspired by this, we further show that highly customized dominance relationship can be derived from decomposition for any given decomposition vector. A new evolutionary algorithm is then proposed by applying the customized dominance relationship with adaptive strategy to each subpopulation of multiobjective to multiobjective framework. Experiments are conducted to compare the proposed algorithm with five state-of-the-art decomposition-based evolutionary algorithms on a set of well-known scaled many-objective test problems with 5 to 15 objectives. Simulation results have shown that the proposed algorithm can make better use of the decomposition vectors to achieve better performance. Further investigations on unscaled many-objective test problems verify the robust and generality of the proposed algorithm.

An effective allocation of search effort is important in multiobjective optimization, particularly in many-objective optimization problems (MaOPs). This paper presents a new adaptive search effort allocation strategy for multiobjective evolutionary algorithm based on decomposition MOEA/D-M2M, a recent MOEA/D algorithm for challenging MaOPs. This proposed method adaptively adjusts the subregions of its subproblems by detecting the importance of different objectives in an adaptive manner. More specifically, it periodically resets the subregion setting based on the distribution of the current solutions in the objective space such that the search effort is not wasted on unpromising regions. The basic idea is that the current population can be regarded as an approximation to the Pareto front (PF) and thus one can implicitly estimate the shape of the PF and such estimation can be used for adjusting the search focus. The performance of proposed algorithm has been verified by comparing it with eight representative and competitive algorithms on a set of degenerated MaOPs with disconnected and connected PFs. Performances of the proposed algorithm on a number of nondegenerated test instances with connected and disconnected PFs are also studied.

There are two main tasks involved in addressing a multiobjective optimization problem (MOP) by evolutionary multiobjective (EMO) algorithms: 1) make the population converge close to the Pareto-optimal front and 2) maintain adequate population diversity. However, most state-of-the-art EMO algorithms are designed based on the “convergence first and diversity second” principle. It has been observed that although these EMO algorithms have been successful in optimizing many real-world MOPs, they fail to solve certain problems that feature a severe imbalance between diversity preservation and achieving convergence. This paper characterizes an imbalanced MOP by clearly defining properties and indicating the reasons for the existing EMO algorithms' difficulties in solving them. We then present 14 imbalanced problems, with and without constraints. Computational results using four existing EMO algorithms-elitist non-dominated sorting genetic algorithm (NSGA-II), multiobjective evolutionary algorithm based on decomposition (MOEA/D), strength Pareto evolutionary algorithm 2 (SPEA2), and S metric selection EMO algorithm (SMS-EMOA) and a proposed generalized vector-evaluated genetic algorithm are then presented. It is seen that these EMO algorithms cannot solve these imbalanced problems, but they are able to solve the problems when augmented by multiobjective to multiobjective (M2M), an approach that decomposes the population into several interacting subpopulations. These results and the successful application of the EMO methods with the M2M approach even on standard so-called balanced problems indicate the usefulness of using the M2M approach.

When optimizing an multiobjective optimization problem, the evolution of population can be regarded as a approximation to the Pareto Front (PF). Motivated by this idea, we propose an adaptive region decomposition framework: MOEA/D-AM2M for the degenerated Many-Objective optimization problem (MaOP), where degenerated MaOP refers to the optimization problem with a degenerated PF in a subspace of the objective space. In this framework, a complex MaOP can be adaptively decomposed into a number of many-objective optimization subproblems, which is realized by the adaptively direction vectors design according to the present population's distribution. A new adaptive weight vectors design method based on this adaptive region decomposition is also proposed for selection in MOEA/D-AM2M. This strategy can timely adjust the regions and weights according to the population's tendency in the evolutionary process, which serves as a remedy for the inefficiency of fixed and evenly distributed weights when solving MaOP with a degenerated PF. Five degenerated MaOPs with disconnected PFs are generated to identify the effectiveness of proposed MOEA/D-AM2M. Contrast experiments are conducted by optimizing those MaOPs using MOEA/D-AM2M, MOEA/D-DE and MOEA/D-M2M. Simulation results have shown that the proposed MOEA/D-AM2M outperforms MOEA/D-DE and MOEA/D-M2M.

This paper proposes an improved version of the single objective optimization evolutionary algorithm based on Covariance Matrix Learning and Searching Preference (CMLSP), named ICMLSP. ICMLSP uses the same way with CMLSP to generate high quality solutions by sampling a multivariate Gauss distribution, which uses the best solutions found so far as its mean value. However, unlike the previous one, ICMLSP uses different covariance matrix learning philosophy, that is, the principal component analysis (PCA) method is used to estimate the covariance matrix. Furthermore, ICMLSP tends to use smaller population size than CMLSP to achieve a faster search. In order to get enough information for a reliable estimation, a new cumulation strategy is designed in ICMLSP. Solutions are selected from an archive set which stores the best λ individuals in present population and last t(t >= 1) populations to estimate the covariance matrix. A new adaptive rule, which makes use of the history successful information to generate different searching step from two different Cauchy distributions, is designed for ICMLSP to balance global exploration and local exploitation. Finally, the performance of the ICMLSP has been tested on 15 noiseless optimization problems designed for the CEC 2015 Competition on Learning-based Real-Parameter Single Objective Optimization. The results are reported at the end of this paper.

In this paper, we propose a single objective optimization evolutionary algorithm (EA) based on Covariance Matrix Learning and Searching Preference (CMLSP) and design a switching method which is used to combine CMLSP and Covariance Matrix Adaptation Evolution Strategy (CMAES). Then we investigate the performance of the switch method on a set of 30 noiseless optimization problems designed for the special session on real-parameter optimization of CEC 2014. The basic idea of the proposed CMLSP is that it is more likely to find a better individual around a good individual. That is to say, the better an individual is, the more resources should be invested to search the region around the individual. To achieve it, we discard the traditional crossover and mutation and design a novel method based on the covariance matrix leaning to generate high quality solutions. The best individual found so far is used as the mean of a Gaussian distribution and the covariance of the best λ individuals in the population are used as the evaluation of its covariance matrix and we sample the next generation individual from the Gaussian distribution other than using crossover and mutation. In the process of generating new individuals, the best individual is changed if ever a better one is found. This search strategy emphasizes the region around the best individual so that a faster convergence can be achieved. The use of switch method is to make best use of the proposed CMLSP and existing CMAES. At last, we report the results.

In this paper, a novel multi-objective particle swarm optimization algorithm based on MOEA/D-M2M decomposition strategy (MOPSO-M2M) is proposed. MOPSO-M2M can decompose the objective space into a number of subregions and then search all the subregions using respective sub-swarms simultaneously. The M2M decomposition strategy has two very desirable properties with regard to MOPSO. First, it facilitates the determination of the global best (gbest) for each sub-swarm. A new global attraction strategy based on M2M decomposition framework is proposed to guide the flight of particles by setting an archive set which is used to store the historical best solutions found by the swarm. When we determine the gbest for each particle, the archive set is decomposed and associated with each sub-swarm. Therefore, every sub-swarm has its own archive subset and the gbest of the particle in a sub-swarm is selected randomly in its archive subset. The new global attraction strategy yields a more reasonable gbest selection mechanism, which can be more effective to guide the particles to the Pareto Front (PF). This strategy can ensure that each sub-swarm searches its own subregion so as to improve the search efficiency. Second, it has a good ability to maintain the diversity of the population which is desirable in multi-objective optimization. Additionally, MOPSO-M2M applies the Tchebycheff approach to determine the personal best position (pbest) and no additional clustering or niching technique is needed in this algorithm. In order to demonstrate the performance of the proposed algorithm, we compare it with two other algorithms: MOPSO and DMS-MO-PSO. The experimental results indicate the validity of this method.