Federated learning (FL) faces significant challenges in scenarios with nonindependent and identically distributed (non-IID) data distributions across participating clients. Traditional aggregation-based approaches often struggle with the inherent misalignment between local and global optimization objectives, which leads to gradient divergence and suboptimal generalization performance. This article proposes a novel FL framework that replaces conventional aggregation with a biologically inspired model symbiosis approach called FedSym, which employs a dual-level symbiotic mechanism. Ectosymbiosis performs coarse-grained hierarchical parameter recombinations through random layer-wise model combination, while endosymbiosis enables fine-grained intralayer parameter fusion through weighted averaging, collectively steering model updates toward flatter loss landscapes. Our theoretical analysis demonstrates that FedSym’s convergence rate is$O({}{1}/{T})$under non-IID conditions, which matches the convergence properties of FedAvg. Extensive evaluations across multiple datasets and model architectures show that FedSym achieves substantial improvements over state-of-the-art FL methods, particularly in challenging scenarios with high data heterogeneity, and demonstrates robust performance across varying numbers of participating clients and federation scales.

IoT devices have been widely deployed in medical industry, in the objective of improving diagnostic accuracy and increasing the efficiency of healthcare systems. However, traditional centralized learning approaches often fall short in meeting strict privacy requirements and adapting to emerging diseases in clinical environment. To address this, we propose a novel federated continual learning (CL) framework for disease diagnosis (FCL4DD), designed to enable distributed and incremental learning of new disease classes while safeguarding data privacy. To combat catastrophic forgetting in CL, FCL4DD integrates a replay strategy powered by a weakly supervised diffusion model (WSDM) to generate historical data for diagnosis model training. The WSDM leverages weak supervision into diffusion model to capture the diverse characteristics of the real data, enabling the generation of high-quality synthetic samples that maintain the data’s inherent variability. To overcome the challenges of nonindependent and identically distributed (non-IID) data in federated learning, WSDM is deployed at the central server to generate synthetic disease data that conforms to the global distribution. This synthetic data is then used to retrain client models, reducing discrepancies and enhancing performance consistency across clients. Evaluations on various datasets demonstrates that our method outperforms other state-of-the-art approaches, such as FedEWC, FedLwF, FedWeIT, TARGET, and DDDR, achieving up to a 4.85% accuracy improvement over the second-best method. Code are available athttps://github.com/hysshy/FCL4DD.

Choreography refers to creation of dance motions according to both music and dance knowledge, where the created dances should be style-specific and consistent. However, most of the existing methods generate dances using the given music as the only reference, lacking the stylized dancing knowledge, namely, the flag motion patterns contained in different styles. Without the stylized prior knowledge, these approaches are not promising to generate controllable style or diverse moves for each dance style, nor new dances complying with stylized knowledge. To address this issue, we propose a novel music-to-dance generation framework guided by style embedding, considering both input music and stylized dancing knowledge. These style embeddings are learnt representations of style-consistent kinematic abstraction of reference dance videos, which can act as controllable factors to impose style constraints on dance generation in a latent manner. Hence, we can make the style embedding fit into any given style while allowing the flexibility to generate new compatible dance moves by modifying the style embedding according to the learnt representations of a certain style. We are the first to achieve knowledge-driven style control in dance generation tasks. To support this study, we build a large multi-style music-to-dance dataset referred to as I-Dance. The qualitative and quantitative evaluations demonstrate the advantage of the proposed framework, as well as the ability to synthesize diverse moves under a dance style directed by style embedding.

Video Compression Artifact Reduction aims to reduce the artifacts caused by video compression algorithms and improve the quality of compressed video frames. The critical challenge in this task is to make use of the redundant high-quality information in compressed frames for compensation as much as possible. Two important possible compensations: Motion compensation and global context, are not comprehensively considered in previous works, leading to inferior results. The key idea of this paper is to fuse the motion compensation and global context together to gain more compensation information to improve the quality of compressed videos. Here, we propose a novel Spatio-Temporal Compensation Fusion (STCF) framework with the Parallel Swin-CNN Fusion (PSCF) block, which can simultaneously learn and merge the motion compensation and global context to reduce the video compression artifacts. Specifically, a temporal self-attention strategy based on shifted windows is developed to capture the global context in an efficient way, for which we use the Swin transformer layer in the PSCF block. Moreover, an additional Ada-CNN layer is applied in the PSCF block to extract the motion compensation. Experimental results demonstrate that our proposed STCF framework outperforms the state-of-the-art methods up to 0.23dB (27% improvement) on the MFQEv2 dataset.

Semantic image editing provides users with a flexible tool to modify a given image guided by a corresponding segmentation map. In this task, the features of the foreground objects and the backgrounds are quite different. However, all previous methods handle backgrounds and objects as a whole using a monolithic model. Consequently, they remain limited in processing content-rich images and suffer from generating unrealistic objects and texture-inconsistent backgrounds. To address this issue, we propose a novel paradigm, Semantic Image Editing by Disentangling Object and Background (SIEDOB), the core idea of which is to explicitly leverages several heterogeneous subnetworks for objects and backgrounds. First, SIEDOB disassembles the edited input into background regions and instance-level objects. Then, we feed them into the dedicated generators. Finally, all synthesized parts are embedded in their original locations and utilize a fusion network to obtain a harmonized result. Moreover, to produce high-quality edited images, we propose some innovative designs, including Semantic-Aware Self-Propagation Module, Boundary-Anchored Patch Discriminator, and Style-Diversity Object Generator, and integrate them into SIEDOB. We conduct extensive experiments on Cityscapes and ADE20K-Room datasets and exhibit that our method remarkably outperforms the baselines, especially in synthesizing realistic and diverse objects and texture-consistent backgrounds. Code is available at https://github.com/WuyangLuo/SIEDOB.

Federated learning can collaboratively train AI models while protecting data privacy. In practical industry environment, non-independent and identically distributed (Non-IID) characteristics of data affect the effectiveness of federated learning. Personalized federated learning can help resolve this, but it cannot adapt to unknown data. In addition, practical applications also call for trusted training environment and remain stable when there are security threats. In this article, we propose a credible federated self-learning (CFSL), based on the idea of hypernetwork supported by blockchain to achieve secured, credible, personalized federated self-learning, especially, for unknown data in Non-IID environment. Extensive experiments on three Non-IID data sets demonstrate the capabilities on adaptive resilience for security attacks and on accuracy of recognizing unknown objects, with good performance at the same time. CFSL outperforms the existing personalized federated learning methods, with an increase in average accuracy by 4.11%.

Face inpainting aims at plausibly predicting missing pixels of face images within a corrupted region. Most existing methods rely on generative models learning a face image distribution from a big dataset, which produces uncontrollable results, especially with large-scale missing regions. To introduce strong control for face inpainting, we propose a novel reference-guided face inpainting method that fills the large-scale missing region with identity and texture control guided by a reference face image. However, generating high-quality results under imposing two control signals is challenging. To tackle such difficulty, we propose a dual control one-stage framework that decouples the reference image into two levels for flexible control: High-level identity information and low-level texture information, where the identity information figures out the shape of the face and the texture information depicts the component-aware texture. To synthesize high-quality results, we design two novel modules referred to as Half-AdaIN and Component-Wise Style Injector (CWSI) to inject the two kinds of control information into the inpainting processing. Our method produces realistic results with identity and texture control faithful to reference images. To the best of our knowledge, it is the first work to concurrently apply identity and component-level controls in face inpainting to promise more precise and controllable results.

Customer volume prediction is crucial for a variety of urban applications, such as store location selection. So far, the key challenge lies in how to fuse multiple modalities from different data sources, on account of the massive amount of data accessible, for example, spatio-temporal data and satellite images. In this article, we investigate three dynamic weighting ensemble learning models to fuse spatio-temporal features and visual features for predicting customer volume in the urban commercial district of interest. Specifically, we propose the shared-private dynamic weighting model by incorporating graph neural networks, which is proposed to capture geographic dependencies (i.e., competitiveness or dependencies) between urban commercial districts in an end-to-end manner. To the best of our knowledge, it is the first work to utilize graph neural networks to model such geographic relationships. We conduct a series of experiments to demonstrate the effectiveness of the proposed models based on two real datasets. Furthermore, an elaborated visualization method is performed for knowledge discovery.

Regular safety inspection is critical to reduce safety risk in industry. Applying the consortium blockchain technology to safety inspection can ensure the effectiveness of the inspection process and tracing of problems. However, there are two major issues when using conventional consortium blockchain. It is challenging to guarantee the authenticity of the retrieved data source, and meanwhile, achieving a balance between performance and security is not easy. Hence, this article proposes a blockchain-based performance-security balanced safety inspection framework (PSB-SIF), in which a safety inspection box is designed to ensure the authenticity of the inspector’s identity while inspection logic is executed automatically via smart contracts. In addition, this article also proposes a novel credit scoring-based Byzantine fault-tolerant (BFT) consensus algorithm, named safety inspection BFT consensus algorithm (SIBFT), which is used to balance the performance and security of consensus network in a safety inspection. We evaluate the proposed approach by comparing with the solutions using RAFT, Practical BFT (PBFT), and SIBFT consensus algorithms in terms of throughput, transaction latency, scalability, and security of PSB-SIF. The evaluation results show that PSB-SIF is efficient for all these quality metrics.

Real-time intelligent video processing on embedded devices with low power consumption can be useful for applications like drone surveillance, smart cars, and more. However, the limited resources of embedded devices is a challenging issue for effective embedded computing. Most of the existing work on this topic focuses on single device based solutions, without the use of cloud computing mechanisms for parallel processing to boost performance. In this paper, we propose a cloud platform for real-time video processing based on embedded devices. Eight NVIDIA Jetson TX1 and three Jetson TX2 GPUs are used to construct a streaming embedded cloud platform (SECP), on which Apache Storm is deployed as the cloud computing environment for deep learning algorithms (Convolutional Neural Networks - CNNs) to process video streams. Additionally, self-managing services are designed to ensure that this platform can run smoothly and stably, in the form of a metric sensor, a bottleneck detector and a scheduler. This platform is evaluated in terms of processing speed, power consumption, and network throughput by running various deep learning algorithms for object detection. The results show the proposed platform can run deep learning algorithms on embedded devices while meeting the high scalability and fault tolerance required for real-time video processing.

Recently, there is a rising interest in perceiving image aesthetics. The existing works deal with image aesthetics as a classification or regression problem. To extend the cognition from rating to reasoning, a deeper understanding of aesthetics should be based on revealing why a high‐ or low‐aesthetic score should be assigned to an image. From such a point of view, the authors propose a model referred to as Neural Aesthetic Image Reviewer, which can not only give an aesthetic score for an image, but also generate a textual description explaining why the image leads to a plausible rating score. Specifically, they propose three models based on shared aesthetically semantic layers and task‐specific embedding layers at a high level for performance improvement on different tasks. To facilitate researches on this problem, they collect the AVA‐Reviews dataset, which contains 52,118 images and 312,708 comments in total. Through multi‐task learning, the proposed models can rate aesthetic images as well as produce comments in an end‐to‐end manner. It is confirmed that the proposed models outperform the baselines according to the performance evaluation on the AVA‐Reviews dataset. Moreover, they demonstrate experimentally that the authors’ model can generate textual reviews related to aesthetics, which are consistent with human perception.

Seeking effective measures to characterize the chaotic patterns of EEG signals for seizure diagnosis is a long-term endeavor in the literature. We propose to count the number of zero-crossing (ZC) points on Poincaré surface as a feature when the time series of interest is embedded into the reconstructed state space. The experiments show that Poincaré surface can act as a platform to observe the chaotic patterns of EEG signals and the ZC feature on Poincaré surface is a promising pattern descriptor to discriminate different categories of EEG signals. When used alone for EEG classification, the ZC feature achieves 100%, 99.27%, and 94.68% accuracy in 2-class, 3-class, and 5-class classification on a widely used benchmark.

Summary Accurate resource requests prediction is essential to achieve optimal job scheduling and load balancing for cloud Computing. Existing prediction approaches fall short in providing satisfactory accuracy because of high variances of cloud metrics. We propose a deep belief network (DBN)‐based approach to predict cloud resource requests. We design a set of experiments to find the most influential factors for prediction accuracy and the best DBN parameter set to achieve optimal performance. The innovative points of the proposed approach is that it introduces analysis of variance and orthogonal experimental design techniques into the parameter learning of DBN. The proposed approach achieves high accuracy with mean square error of [10−6,10−5], approximately 72%reduction compared with the traditional autoregressive integrated moving average predictor, and has better prediction accuracy compared with the state‐of‐art fractal modeling approach. Copyright © 2016 John Wiley & Sons, Ltd.

Summary Emotion recognition is challenging for understanding people and enhances human–computer interaction experiences, which contributes to the harmonious running of smart health care and other smart services. In this paper, several kinds of speech features such as Mel frequency cepstrum coefficient, pitch, and formant were extracted and combined in different ways to reflect the relationship between feature fusions and emotion recognition performance. In addition, we explored two methods, namely, support vector machine (SVM) and deep belief networks (DBNs), to classify six emotion status: anger, fear, joy, neutral status, sadness, and surprise. In the SVM‐based method, we used SVM multi‐classification algorithm to optimize the parameters of penalty factor and kernel function. With DBN, we adjusted different parameters to achieve the best performance when solving different emotions. Both gender‐dependent and gender‐independent experiments were conducted on the Chinese Academy of Sciences emotional speech database. The mean accuracy of SVM is 84.54%, and the mean accuracy of DBN is 94.6%. The experiments show that the DBN‐based approach has good potential for practical usage, and suitable feature fusions will further improve the performance of speech emotion recognition. Copyright © 2017 John Wiley & Sons, Ltd.

A large number of video applications require real-time response. The high-speed video processing then requires a distributed and parallelized framework utilizing all possible computing resources, i.e., both Central Processing Unit (CPU) and Graphics Processing Unit (GPU) at their best. The CPU-GPU collaboration may cause resource imbalance where GPU-based jobs consume less computing resources while occupying more memory compared with CPU-based jobs. In this paper, we propose a load-aware pluggable cloud framework for real-time video processing where CPU-GPU switching based on workload status can be performed at runtime. Furthermore, we design aspect-oriented monitors to collect framework metrics and propose a distance coverage algorithm to detect performance degradation in order to make sure that the framework runs optimally to achieve good performance when a load-aware task switching is made. We have comprehensively evaluated the framework and the evaluation results show that the proposed framework has good performance, reusability, pluggability, and scalability.

Crowd motion in surveillance videos is comparable to heat motion of basic particles. Inspired by that, we introduce Boltzmann Entropy to measure crowd motion in optical flow field so as to detect abnormal collective behaviors. As a result, the collective crowd moving pattern can be represented as a time series. We found that when most people behave anomaly, the entropy value will increase drastically. Thus, a threshold can be applied to the time series to identify abnormal crowd commotion in a simple and efficient manner without machine learning. The experimental results show promising performance compared with the state of the art methods. The system works in real time with high precision.

The goal of feature selection is to search the optimal feature subset with respect to the evaluation function. Exhaustively searching all possible feature subsets requires high computational cost. The alternative suboptimal methods are more efficient and practical but they cannot promise globally optimal results. We propose a new feature selection algorithm based on distance discriminant and distribution overlapping (HFSDD) for continuous features, which overcomes the drawbacks of the exhaustive search approaches and those of the suboptimal methods. The proposed method is able to find the optimal feature subset without exhaustive search or Branch and Bound algorithm. The most difficult problem for optimal feature selection, the search problem, is converted into a feature ranking problem following rigorous theoretical proof such that the computational complexity can be greatly reduced. Since the distribution of overlapping degrees between every two classes can provide useful information for feature selection, HFSDD also takes them into account by using a new approach to estimate the overlapping degrees. In this sense, HFSDD is a distance discriminant and distribution overlapping based solution. HFSDD was compared with ReliefF and mrmrMID on ten data sets. The experimental results show that HFSDD outperforms the other methods.