Mixed precision quantization (MPQ) is an effective quantization approach to achieve accuracy-complexity trade-off of neural network, through assigning different bit-widths to network activations and weights in each layer. The typical way of existing MPQ methods is to optimize quantization policies (i.e., bit-width allocation) in a gradient descent manner, termed as Differentiable MPQ (DMPQ). At the end of the search, the bit-width associated to the quantization parameters which has the largest value will be selected to form the final mixed precision quantization policy, with the implicit assumption that the values of quantization parameters reflect the operation contribution to the accuracy improvement. While much has been discussed about the MPQ’s improvement, the bit-width selection process has received little attention. We study this problem and argue that the magnitude of quantization parameters does not necessarily reflect the actual contribution of the bit-width to the task performance. Then, we propose a Shapley-based MPQ (SMPQ) method, which measures the bit-width operation’s direct contribution on the MPQ task. To reduce computation cost, a Monte Carlo sampling-based approximation strategy is proposed for Shapley computation. Extensive experiments on mainstream benchmarks demonstrate that our SMPQ consistently achieves state-of-the-art performance than gradient-based competitors.

One-shot neural architecture search (NAS) trains an over-parameterized network (termed as supernet) that assembles all the architectures as its subnets by using weight sharing for computational budget reduction. However, there is an issue of multi-model forgetting during supernet training that some weights of the previously well-trained architecture will be overwritten by that of the newly sampled architecture which has overlapped structures with the old one. To overcome the issue, we propose an orthogonal gradient learning (OGL) guided supernet training paradigm, where the novelty lies in the fact that the weights of the overlapped structures of current architecture are updated in the orthogonal direction to the gradient space of these overlapped structures of all previously trained architectures. Moreover, a new approach of calculating the projection is designed to effectively find the base vectors of the gradient space to acquire the orthogonal direction. We have theoretically and experimentally proved the effectiveness of the proposed paradigm in overcoming the multi-model forgetting. Besides, we apply the proposed paradigm to two one-shot NAS baselines, and experimental results demonstrate that our approach is able to mitigate the multi-model forgetting and enhance the predictive ability of the supernet with remarkable efficiency on popular test datasets.

With advances in the edge computing (EC) and federated learning (FL) technologies in jointcloud, the edge FL service market has emerged recently and it requires trading edge resources between model requesters and data owners to complete FL tasks, which needs to incentivize sufficient data owners to participate in model training tasks. However, the limitations of resource trading and incentive design for edge FL service market have not been well addressed. In this paper, we propose a two-level blockchain-aided resource trading mechanism for encouraging appropriate edge servers to compete for dynamic FL tasks from the market while incentivizing data owners to participate in the FL tasks. At the upper level, we apply the deep learning-based reverse auction to model the dynamics of the task server selection process, with the aim of maximizing the total social welfare of the edge FL service market, where the edge server, as a seller, considers not only the data contribution of edge devices but also the cost of using blockchain when bidding. At the lower level, the edge servers offer rewards in exchange for the data owners’ participation, while the parameter aggregation is completed through the blockchain in a decentralized manner, which improves the FL’s robustness. Then, we utilize the Stackelberg game to model the dynamic process that the data owners compete for the servers’ revenue. We conduct extensive simulation experiments and the experimental results show that the proposed mechanism is able to get maximized social welfare and provide effective insights and strategies for the resource trading in the edge FL market to complete the federated training.

Weight quantization is an effective technique to compress deep neural networks for their deployment on edge devices with limited resources. Traditional loss-aware quantization methods commonly use the quantized gradient to replace the full-precision gradient. However, we discover that the gradient error will lead to an unexpected zig-zagging-like issue in the gradient descent learning procedures, where the gradient directions rapidly oscillate or zig-zag, and such issue seriously slows down the model convergence. Accordingly, this paper proposes a one-step forward and backtrack way for loss-aware quantization to get more accurate and stable gradient direction to defy this issue. During the gradient descent learning, a one-step forward search is designed to find the trial gradient of the next-step, which is adopted to adjust the gradient of current step towards the direction of fast convergence. After that, we backtrack the current step to update the full-precision and quantized weights through the current-step gradient and the trial gradient. A series of theoretical analysis and experiments on benchmark deep models have demonstrated the effectiveness and competitiveness of the proposed method, and our method especially outperforms others on the convergence performance.

Performance predictors are used to reduce architecture evaluation costs in neural architecture search, which however suffers from a large amount of budget consumption in annotating substantial architectures trained from scratch. Hence, how to leverage existing annotated architectures to train a generalized predictor to find the optimal architecture on unseen target search spaces becomes a new research topic. To solve this issue, we propose a Single-Domain Generalized Predictor (SDGP), which aims to make the predictor only trained on a single source search space but perform well on target search spaces. In meta-learning, we firstly adopt feature extractor in learning the domain-invariant features of the architectures. Then, a neural predictor is trained to map the architectures to the accuracy of the candidate architectures over the target domain simulated on the source search space. Moreover, a novel multi-head attention driven regularizer is designed to regulate the predictor to further improve the generalization ability of the predictor for the feature extractor. A series of experimental results have shown that the proposed predictor outperforms the state-of-the-art predictors in generalization and achieves significant performance gains in finding the optimal architectures with test error 2.40% on CIFAR-10 and 23.20% on ImageNet1k within 0.01 GPU days.

Low-rank matrix recovery is a major challenge in machine learning and computer vision, particularly for large-scale data matrices, as popular methods involving nuclear norm and singular value decomposition (SVD) are associated with high computational costs and biased estimators. To overcome this challenge, we propose a novel approach to learning low-rank matrices based on the matrix volume and a nonconvex logarithmic function. The matrix volume is the product of all the nonzero singular values of a matrix and has unique geometric properties and connections with other convex and nonconvex functions. We establish a generalized nonconvex regularization problem using the penalty function strategy and introduce an accelerated proximal alternating linearized minimization (AccPALM) algorithm with double acceleration, which combines Nesterov’s acceleration and power strategy. The algorithm reduces computational costs and has provable convergence results under the Kurdyka-Łojasiewicz (KŁ) inequality with mild conditions. Our approach shows superior accuracy, efficiency, and convergence behavior compared to other low-rank matrix learning methods on robust matrix completion (RMC) and low-rank representation (LRR) tasks. We analyze the impact of algorithm parameters on convergence and performance and present visually appealing results to further demonstrate the effectiveness of our approach. The proposed methodology represents a promising advance in the field of low-rank matrix recovery, and its effectiveness has been validated via extensive numerical experiments. The source code for the proposed algorithms is accessible at https://github.com/ZhangHengMin/AccPALMcodes.

In multi-objective evolutionary neural architecture search (NAS), existing predictor-based methods commonly suffer from the rank disorder issue that a candidate high-performance architecture may have a poor ranking compared with the worse architecture in terms of the trained predictor.To alleviate the above issue, we aim to train a Pareto-wise end-to-end ranking classifier to simplify the architecture search process by transforming the complex multi-objective NAS task into a simple classification task. To this end, a classifier-based Pareto evolution approach is proposed, where an online classifier is trained to directly predict the dominance relationship between the candidate and reference architectures. Besides, an adaptive clustering method is designed to select reference architectures for the classifier, and an α-domination assisted approach is developed to address the imbalance issue of positive and negative samples. The proposed approach is compared with a number of state-of-the-art NAS methods on widely-used test datasets, and computation results show that the proposed approach is able to alleviate the rank disorder issue and outperforms other methods. Especially, the proposed method is able to find a set of promising network architectures with different model sizes ranging from 2M to 5M under diverse objectives and constraints.

Defect recognition is an important task in intelligent manufacturing. Due to the subjectivity of human annotation, the collected defect data usually contains a lot of noise and unpredictable uncertainties, which have a great negative influence on defect recognition. It is a significant challenge to discover an effective defect recognition model with satisfactory uncertainty processing ability. A natural way is to automatically search for an efficient deep model, which can be realized by neural architecture search (NAS). To achieve this, we propose an efficient fuzzy NAS framework for defect recognition, where the searched architecture can effectively handle uncertain information from the given datasets. Specifically, we first design a fuzzy search space and the related encoding strategy for fuzzy NAS. Then, we propose a comparator-based evolutionary search approach, where an online end-to-end comparator is learned to directly determine the selection of candidate architectures from the evolutionary population. The comparator works in an end-to-end way and it transforms the complex ranking problem of evaluating architectures into a simple classification task, which overcomes the rank disorder issue suffered from traditional performance predictors. A series of experimental results demonstrate that the architecture with fewer #Params (1.22 M) search by fuzzy neural architecture search framework for defect recognition method achieves higher accuracy (92.26%) compared to the state-of-the-art results (i.e., DARTS-PV) on the ELPV dataset, as well as competitive results (accuracy = 76.4%, #Params = 1.04 M) on the CODEBRIM dataset. Experimental results show the effectiveness and efficiency of our proposed method in handling uncertain problems.

In multiobjective decision making, most knee identification algorithms implicitly assume that the given solutions are well distributed and can provide sufficient information for identifying knee solutions. However, this assumption may fail to hold when the number of objectives is large or when the shape of the Pareto front is complex. To address the above issues, we propose a knee-oriented solution augmentation (KSA) framework that converts the Pareto front into a multimodal auxiliary function whose basins correspond to the knee regions of the Pareto front. The auxiliary function is then approximated using a surrogate and its basins are identified by a peak detection method. Additional solutions are then generated in the detected basins in the objective space and mapped to the decision space with the help of an inverse model. These solutions are evaluated by the original objective functions and added to the given solution set. To assess the quality of the augmented solution set, a measurement is proposed for the verification of knee solutions when the true Pareto front is unknown. The effectiveness of KSA is verified on widely used benchmark problems and successfully applied to a hybrid electric vehicle controller design problem.

In real-world applications, a specific class of multiobjective optimization problems, such as the cloud service allocation problem (CSAOPs), possess the characteristic of variable-length and mixed variables, termed as variable multiobjective optimization problems (VMMOPs). Unfortunately, little research has been reported to solve them. To fill the gap, we propose a tailored enhanced decomposition-based algorithm to handle the VMMOPs. Specifically, a variable-length coding structure is designed to flexibly represent the solutions of VMMOPs. In order to facilitate the solution generation, a simple dimensionality incremental learning strategy is developed to choose representative solutions for the training of two learning models. The one is the fast-clustering-based histogram model, which is built for the sampling of solutions in the continuous decision space, while the other one is the incremental learning-based histogram model, designed to sample solutions in discrete decision space. Following the traditional constructor of the DTLZ test suite and the features of CSAOPs, we present a test suite of VMMOPs for the verification of the performance of the methods in handling VMMOPs. Experimental results on a number of benchmark problems and two real CSAOPs have shown the effectiveness and competitiveness of the proposed method in handling VMMOPs.

In large refineries, the resource-oriented plan of crude oil is commonly required to be tractable and decomposable for practical operation scheduling, especially for large-scale scheduling. To this end, a framework of hybrid optimization based on plan decomposition (FHO/PD) is proposed, which mainly depends on evolutionary algorithms to realize the flexible decomposition from large-scale planning to scheduling and takes advantage of mathematical programming to improve the solving efficiency synchronously. Finally, the experimental results on a practical case suggest that the proposed method has shown great flexibility and applicability in crude oil scheduling.

Most reference vector-based decomposition algorithms for solving multiobjective optimization problems may not be well suited for solving problems with irregular Pareto fronts (PFs) because the distribution of predefined reference vectors may not match well with the distribution of the Pareto-optimal solutions. Thus, the adaptation of the reference vectors is an intuitive way for decomposition-based algorithms to deal with irregular PFs. However, most existing methods frequently change the reference vectors based on the activeness of the reference vectors within specific generations, slowing down the convergence of the search process. To address this issue, we propose a new method to learn the distribution of the reference vectors using the growing neural gas (GNG) network to achieve automatic yet stable adaptation. To this end, an improved GNG is designed for learning the topology of the PFs with the solutions generated during a period of the search process as the training data. We use the individuals in the current population as well as those in previous generations to train the GNG to strike a balance between exploration and exploitation. Comparative studies conducted on popular benchmark problems and a real-world hybrid vehicle controller design problem with complex and irregular PFs show that the proposed method is very competitive.

Conventional multiobjective optimization algorithms (MOEAs) with or without preferences are successful in solving multi- and many-objective optimization problems. However, a strong hypothesis underlying their performance is that MOEAs are able to find a representative solution set to cover the entire Pareto-optimal front (PF) and decision makers are able to conveniently and precisely articulate their preference, which is not always easy to fulfill in practice. Accordingly, it is suggested that representative solutions in the naturally interesting regions of the PF rather than the whole PF should be targeted. A large body of research has been proposed to search or identify the knees or knee regions over the past decades. Therefore, this article aims to provide a comprehensive survey of the research on knee-oriented optimization. We start with a discussion of the importance and basic concepts of the knees, followed by a summary of knee-oriented benchmarks and indicators. After that, knee-oriented frameworks and techniques, and real-world applications are presented. Finally, potential challenges are pointed out and a few promising future lines of research are suggested. The survey offers a new perspective to develop MOEAs for solving multi- and many-objective optimization problems.

In preference-based optimization, knee points are considered the naturally preferred tradeoff solutions, especially when the decision maker has little a priori knowledge about the problem to be solved. However, identifying all convex knee regions of a Pareto front remains extremely challenging, in particular in a high-dimensional objective space. This article presents a new evolutionary multiobjective algorithm for locating knee regions using two localized dominance relationships. In the environmental selection, the α-dominance is applied to each subpopulation partitioned by a set of predefined reference vectors, thereby guiding the search toward different potential knee regions while removing possible dominance resistant solutions. A knee-oriented-dominance measure making use of the extreme points is then proposed to detect knee solutions in convex knee regions and discard solutions in concave knee regions. Our experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art knee identification algorithms on a majority of multiobjective optimization test problems having up to eight objectives and a hybrid electric vehicle controller design problem with seven objectives.

In multiobjective optimization, it is nontrivial for decision makers to articulate preferences without a priori knowledge, which is particularly true when the number of objectives becomes large. Depending on the shape of the Pareto front, optimal solutions such as knee points may be of interest. Although several multi- and many-objective optimization test suites have been proposed, little work has been reported focusing on designing multiobjective problems whose Pareto front contains complex knee regions. Likewise, few performance indicators dedicated to evaluate an algorithm's ability of accurately locating all knee points in high-dimensional objective space have been suggested. This paper proposes a set of multiobjective optimization test problems whose Pareto front consists of complex knee regions, aiming to assess the capability of evolutionary algorithms to accurately identify all knee points. Various features related to knee points have been taken into account in designing the test problems, including symmetry, differentiability, and degeneration. These features are also combined with other challenges in solving the optimization problems, such as multimodality, linkage between decision variables, nonuniformity, and scalability of the Pareto front. The proposed test problems are scalable to both decision and objective spaces. Accordingly, new performance indicators are suggested for evaluating the capability of optimization algorithms in locating the knee points. The proposed test problems, together with the performance indicators, offer a new means to develop and assess preference-based evolutionary algorithms for solving multi- and many-objective optimization problems.

Past decades have witnessed a rapid development in research on multi- and many-objective evolutionary optimization. Reference-based and preference-based strategies are both widely used in dealing with the multi- and many-objective optimization problems. However, little effort has been devoted to a critical analysis of similarities and differences between the two approaches. This paper revisits the methodologies, compares the similarities and differences, and discusses the limitations of reference-based and preference-based many-objective evolutionary algorithms. Our analyses reveal that preference information may be embedded into reference-based methods in dealing with irregular problems so that the objective space can be better explored and a solution set of interest to the user will be obtained. Meanwhile, it is far from trivial for a decision-maker to provide informed preferences without sufficient a priori knowledge of the problem in the preference-based optimization. Therefore, this paper suggests a new approach to many-objective optimization problems that integrates preference-based and reference-based methodologies, where the solutions of natural interest such as the knee regions are identified at first and then the acquired knowledge of the knee regions can be used in reference-based methods. This way, accurate, diverse and preferred solutions can be obtained, and a deeper insight into the problem can be gained.

Many evolutionary algorithms have been proposed and demonstrated to have excellent performance in striking a balance between convergence and diversity in dealing with multiobjective optimization problems. However, little attention has been paid to the decision making stage where a small number of solutions are selected to be presented to the user. It is believed that knee points are considered to be the naturally preferred solutions when no specific preferences are available, because knee solutions incur a large loss in at least one objective to gain a small amount in other objectives. One common issue in the identification of knee points is that some knee points are easily ignored and knees in concave regions are hard to be identified. To resolve these issues, this paper proposes a novel method for knee identification, which first maps the non-dominated solutions to a constructed hyperplane and then divides them into groups, each representing a candidate knee region, based on the density of the solutions projected on the hyperplane. Finally, the convexity and curvature of the candidate knee groups are determined and only those having a strong curvature are kept. The proposed method is empirically demonstrated to be effective in identifying knee points located in both convex and concave regions on three existing test problems and one newly proposed test problem.

This paper proposes an improved performance metric for multiobjective evolutionary algorithms with user preferences. This metric uses the idea of decomposition to transform the preference information into m+1 points on a constructed preference-based hyperplane, then calculates the Euclidean distances and the angles between the obtained solutions by algorithms and those obtained m+1 points, respectively. By means of these distances and angles, the proposed metric can evaluate effectively both the convergence and diversity of the obtained solution set, with consideration of the preference information. This makes easier and allows meaningful comparisons between different multiobjective evolutionary algorithms using preference information.