Binary neural networks (BNNs) have been applied in limited resources and mobile devices because of their extreme model compression ability. However, manually designing suitable architectures is challenging given the specialized structure of binarized operations. Neural architecture search (NAS) provides a promising approach for designing high-performance BNN architectures. In practice, various situations require networks with different parameter sizes and performance levels. Therefore, this article proposes a multiobjective evolutionary NAS algorithm for BNNs based on a two-stage training strategy (MO-TS-BNAS) to solve these problems. First, the ApproxSign function is used to approximate the gradient error in the training of BNNs. To avoid the small model trap problem, two auxiliary objectives are introduced in nondominated sorting to retain larger models with similar errors. Then, a two-stage training strategy with flexible use of auxiliary objectives is proposed, forming the selection mechanism in environmental selection. The path dropout method is used in the second stage to prevent hypernetwork overfitting. In addition, the mini-batch gradient descent strategy is improved to speed up individual architecture evaluation and reduce time cost in the search process. Finally, the full precision baseline search space is binarized for general experimental comparison. Our MO-TS-BNAS algorithm balances the two different objective functions of the model size and error. A large number of experiments are carried out on the CIFAR10 and ImageNet datasets, and the results show the effectiveness of the proposed method.

As the foundation of next-generation wireless networks, distributed learning (DL) is expected to be integrated into 6 G communication networks, profoundly advancing the transformation of intelligent connectivity. However, network-induced delays critically degrade the performance of DL algorithms in practical deployments. Beyond the communication limitation, many emerging applications involve antagonistic interactions among agents, introducing additional complexity. To overcome these challenges, we propose a decentralized distributed cooperation–competition learning (DCCL) algorithm based on neuro-fuzzy networks, designed for latency communication networks. The algorithm innovatively employs the signed graph to naturally encode the coupling coopetition relationships among agents, and incorporates the delay model into its design. It demonstrates superior adaptability for DL problems with bimodal coalitional adversarial interactions in latency-prone mission-critical services. Moreover, we extend the neuro-fuzzy network into a distributed version, and the resulting distributed neuro-fuzzy model inherently preserves the interpretability characteristic and superior learning capability. Based on structural balance theory and discrete Lyapunov stability theory, we rigorously prove the convergence of the DCCL algorithm and derive an explicit sufficient condition in the form of a maximum allowable latency tolerance. The proposed algorithm benefits privacy protection by transmitting only model parameters. Experiments are conducted to validate the performance of the DCCL algorithm on several datasets for regression and classification. Furthermore, we discuss the limitations of the DCCL algorithm, providing a balanced perspective for future research.

Federated learning (FL) is a distributed learning paradigm that leverages local updates and parameter sharing to address privacy concerns in Industrial Internet of Things (IIoT) environments. The presence of statistical heterogeneity among IIoT devices poses significant challenges for FL, impacting convergence and model performance. Although previous studies have attempted to mitigate this issue, a fundamental solution remains elusive. To address this, we propose a novel approach called federated multidiscriminators multigenerators generative adversarial network (FedMDMG-GAN), which employs a distributed generative adversarial network (GAN). IIoT devices concurrently train local generators and discriminators to generate data with global information. The server then aggregates parameters and redistributes them to the devices to refine local GANs. In addition, we introduce a proximal term for global aggregation to enhance convergence. Theoretical analysis suggests that the FedMDMG-GAN algorithm can asymptotically converge to a stable point. Qualitative assessments demonstrate that our method can generate images closely resembling real data with comprehensive global information. Quantitative results indicate that FedMDMG-GAN outperforms vanilla FL and state-of-the-art methods.

Federated learning (FL) encounters two major obstacles, namely, the heterogeneity among Internet of Things (IoT) devices hampers both personalized and generalized performance, and the limited communication resources impede the learning efficiency. This article proposes a decentralized FL approach based on prototype representation learning (called DeProFL) and time-varying communication topology. The heterogeneity among devices can arise from varying data distributions and model architectures, leading to local or global gradient drift. To mitigate the effects of heterogeneity, we introduce prototype learning to establish consistent representations of samples and models across devices. Rather than directly propagating the large model weights, which require significant transmission volumes, we propagate prototype representations among local devices to reduce transmission. Given the limited communication resources of IoT and the dynamic wireless environments, we propose a time-varying decentralized FL approach to address these practical constraints. During each global iteration, every device shares its prototype only with adjacent devices for the collaborative learning, aiming to improve system stability and reduce network bandwidth usage. We have theoretically analyzed and verified the convergence of the DeProFL under nonconvex conditions. Extensive experiments have been applied to the benchmark data sets, and the results show that DeProFL outperforms state-of-the-art methods.

We consider a federated semi-supervised learning (FSSL) scenario in heterogeneous Internet of Things (IoT) healthcare systems where only a tiny fraction of IoT healthcare devices possess labels. The challenge is that most IoT devices need label information and exhibit heterogeneous data distributions. This article proposes a unified FSSL framework (FSSL-HD) tailored for IoT healthcare systems with heterogeneous distributions. The FSSL-HD framework introduces a novel objective function for unlabeled healthcare devices to enhance generalization performance and reduce intermodel discrepancies. A straightforward loss-based aggregation mechanism is designed to reinforce distillation from labeled healthcare devices to supervise unlabeled samples. Multiple iterations for labeled devices and dynamically adjusted confidence thresholds are proposed to improve the model performance. Our theoretical analysis of the generalization error in FSSL suggests directions for enhancing performance by improving the self-training capabilities of unlabeled devices and reinforcing the distillation and transfer of supervision signals from labeled devices. Our empirical experiments on the federated benchmark data sets and the medical image data sets show that FSSL-HD surpasses the performance of state-of-the-art methods. The code is available athttps://github.com/baoshengli96/FSSL-HD.

Binary Neural Networks (BNNs) using 1-bit weights and activations are emerging as a promising approach for mobile devices and edge computing platforms. Concurrently, traditional Neural Architecture Search (NAS) has gained widespread usage in automatically designing network architectures. However, the computation involved in binary NAS is more complex than in NAS due to the substantial information loss incurred by binary modules, and different binary spaces are required for different tasks. To address these challenges, a universal binary neural architecture search (UBNAS) algorithm is proposed. In this paper, the ApproxSign function is used to reduce the gradient error and accelerate the convergence in binary network searching and training. Moreover, UBNAS adopts a novel search space consisting of operations appropriate for the binary methods. To improve the original space operation module, we explore the effect of diverse structures for various modules and ultimately obtain a universal binary network structure. Additionally, the channel sampling ratio is adjusted to balance the advantages of different operations and an early stopping strategy is implemented to significantly reduce the computational burden associated with searching. We perform extensive experiments on CIFAR10, and ImageNet datasets and the results demonstrate the effectiveness of the proposed method.

When deep neural networks are used to process remote sensing data, complex network structures, and parameters limit their real-time processing deployment in satellites. Binary Neural Networks (BNNs) can overcome this obstacle by reducing network complexity. However, the binarization process naturally leads to information loss, reducing the performance of the original neural networks. In this article, three modules are introduced to improve the performance of BNNs. First, we propose an improved channel shuffle (CS) module to enhance communication between different channels in feature maps, in which the grouped pointwise convolution is used to reduce the original calculation scale. Second, the squeeze and excitation (SE) module called the attention mechanism is inserted before each layer of the binary convolution operation to recognize the importance of different feature channels. Third, the adaptive linear scaling factor (ALSF) module is proposed to make the weights of the convolution kernel more suitable for the structure of BNNs, which further improves the performance of our network. With these three modules, our CSA-Net reaches or even outperforms the other state-of-the-art (SOTA) methods with a lower computational cost. It can achieve 65.35% Top-1 accuracy on the ImageNet, 90.11% accuracy on the CIFAR10, 89.58% accuracy on the WHU-RS19, and 90.86% accuracy on the UC-Merced, respectively. Meanwhile, compared with other mainstream remote sensing classification models, the number of our parameters is reduced by two or even three orders of magnitude, and the results confirm the effectiveness of the proposed method.

A two-phase constraint-handling technique is integrated into the evolutionary algorithms to solve constrained optimization problems (called TPDE) in this article. In phase one, denoted as the exploration phase, an exterior penalty function method with the dynamic penalty coefficients is developed to compare any two candidate solutions, which aims to push the population into the feasible region. To reduce the computational burden, in phase two, denoted as the exploitation phase, an interior penalty function method with the dynamic penalty coefficients is developed, which enhances the search ability by using the information of constraints in feasible solutions. During the optimization process, differential evolution is adopted as the search algorithm to produce the offspring population. Experiment results on four benchmark test suites, namely, IEEE CEC 2006, IEEE CEC 2010, IEEE CEC 2017, and IEEE CEC 2020, indicate that TPDE is competitive with other popular algorithms.

Conventional artificial neural network (ANN) learning algorithms for classification tasks, either derivative-based optimization algorithms or derivative-free optimization algorithms work by training ANN first (or training and validating ANN) and then testing ANN, which are a two-stage and one-pass learning mechanism. Thus, this learning mechanism may not guarantee the generalization ability of a trained ANN. In this article, a novel bilevel learning model is constructed for self-organizing feed-forward neural network (FFNN), in which the training and testing processes are integrated into a unified framework. In this bilevel model, the upper level optimization problem is built for testing error on testing data set and network architecture based on network complexity, whereas the lower level optimization problem is constructed for network weights based on training error on training data set. For the bilevel framework, an interactive learning algorithm is proposed to optimize the architecture and weights of an FFNN with consideration of both training error and testing error. In this interactive learning algorithm, a hybrid binary particle swarm optimization (BPSO) taken as an upper level optimizer is used to self-organize network architecture, whereas the Levenberg-Marquardt (LM) algorithm as a lower level optimizer is utilized to optimize the connection weights of an FFNN. The bilevel learning model and algorithm have been tested on 20 benchmark classification problems. Experimental results demonstrate that the bilevel learning algorithm can significantly produce more compact FFNNs with more excellent generalization ability when compared with conventional learning algorithms.

This paper presents a two-stage approach, denoted as CBRDE-LM, to evolve the architecture and weights of feedforward artificial neural networks. In the first stage, a collaborative binary-real differential evolution (CBRDE) is used to optimize simultaneously network architecture and connection weights of an ANN by a specific individual representation and evolutionary scheme, in which the structure is indirectly represented as binary coding and connection weights are directly encoded by real-valued coding. In the second stage, based on the resulting architecture and weights of an ANN, Levenberg-Marquardt (LM) backpropagation algorithm is adopted for fine-tuning ANN weights. The performance of the two-stage approach has been evaluated on several benchmarks. The results demonstrate that the two-stage approach can fast produce compact ANNs with good generalization ability at low computational cost.

When the leader's objective function of a nonlinear bilevel programming problem is nondifferentiable and the follower's problem of it is nonconvex, the existing algorithms cannot solve the problem. In this paper, a new effective evolutionary algorithm is proposed for this class of nonlinear bilevel programming problems. First, based on the leader's objective function, a new fitness function is proposed that can be easily used to evaluate the quality of different types of potential solutions. Then, based on Latin squares, an efficient crossover operator is constructed that has the ability of local search. Furthermore, a new mutation operator is designed by using some good search directions so that the offspring can approach a global optimal solution quickly. To solve the follower's problem efficiently, we apply some efficient deterministic optimization algorithms in the MATLAB Toolbox to search for its solutions. The asymptotically global convergence of the algorithm is proved. Numerical experiments on 25 test problems show that the proposed algorithm has a better performance than the compared algorithms on most of the test problems and is effective and efficient.

In this paper, a special nonlinear bilevel programming problem (nonlinear BLPP) is transformed into an equivalent single objective nonlinear programming problem. To solve the equivalent problem effectively, we first construct a specific optimization problem with two objectives. By solving the specific problem, we can decrease the leader's objective value, identify the quality of any feasible solution from infeasible solutions and the quality of two feasible solutions for the equivalent single objective optimization problem, force the infeasible solutions moving toward the feasible region, and improve the feasible solutions gradually. We then propose a new constraint-handling scheme and a specific-design crossover operator. The new constraint-handling scheme can make the individuals satisfy all linear constraints exactly and the nonlinear constraints approximately. The crossover operator can generate high quality potential offspring. Based on the constraint-handling scheme and the crossover operator, we propose a new evolutionary algorithm and prove its global convergence. A distinguishing feature of the algorithm is that it can be used to handle nonlinear BLPPs with nondifferentiable leader's objective functions. Finally, simulations on 31 benchmark problems, 12 of which have nondifferentiable leader's objective functions, are made and the results demonstrate the effectiveness of the proposed algorithm.