Federated Learning (FL) has become a state-of-the-art technique for addressing data isolation and privacy problems. However, the traditional FL framework has limitations on lack of labeled data, adaptation to evolving environments and tasks, and insufficient generalization of the global model due to non-independent and identically distributed (non-IID) data. These issues suggest incorporating human knowledge and interaction into the FL workflow can be beneficial. Human-machine hybrid intelligence is an area that human abilities are considered to prompt the usability and robustness of the system by providing human domain knowledge. Combining FL and human-machine hybrid intelligence can fully utilize their benefits and complement each other perfectly. This article presents our vision of the next generation of FL, Human-Machine Hybrid Federated Learning, namely HIFL. And this work first defines the concept of HIFL and proposes three patterns of collaboration for HIFL: (1) local hybrid intelligence (LocalHIFL), (2) separate hybrid intelligence (SeparateHIFL), and (3) cross hybrid intelligence (CrossHIFL). In each pattern, we survey methodologies and techniques that are utilized to address specific problems that occurred after adding human-machine collaboration in FL process. Besides, we exhibit some potential application scenarios, and provide several open challenges and opportunities contained in HIFL. This survey intends to provide a high-level summarization for improving FL by combining human-machine hybrid intelligence, and to motivate interested readers to consider approaches for designing effective FL approaches and ways of intelligence fusion according to their requirements.

Cloud-based quantization is a key technique for deploying deep neural networks on resource-constrained devices. However, the growing number of heterogeneous devices has placed an increasing burden on the cloud server. It is necessary to handle the high costs of quantizing original models to multiple bitwidths on the cloud server. Therefore, we propose an efficient hierarchical quantization method (HQAQ), which transforms classical cloud-based quantization to two-phase quantization: cloud-edge post-training quantization and edge-device hybrid re-quantization. HQAQ first quantizes original models on the cloud server and distributes the quantized model to edge servers. Then the edge servers re-quantize the quantized model to respond to heterogeneous devices’ requests. To minimize the computational costs on edge servers, edge-device hybrid re-quantization integrates top-bitwidth quantization-aware training with cross-bitwidth quantization points transfer. Quantization points transfer employs clustering to discover the distribution of the quantization points with higher bit-width, which helps quantize the models of lower-bitwidth models. The experimental results on image classification and time series prediction tasks demonstrate that the proposed method improves communication efficiency while maintaining model performance.

As device-free identity recognition (IR) gains popularity and the demand for the Internet of Things (IoT) continues to grow, a new-era IR system featuring multiple distributed recognition devices and edge servers faces two main challenges: model adaptability and balancing the personalization of devices with the generalization of the system. This research introducesFedHMIR, a federated framework designed to simultaneously address these challenges by harmonizing human-machine collaboration with personalization-generalization trade-offs. The proposed framework features a human-machine cooperative online internal update mechanism, leveraging reinforcement learning to maintain the adaptability of personalized local IR models. To counter overfitting and enhance the generalization of the overall IR system, an external update process incorporating a confidence index is introduced. Additionally, the framework employs asynchronous internal and external update procedures to effectively balance personalization and generalization between local and global models. Finally, extensive experiments on three diverse real-world datasets demonstrate the effectiveness and advantages ofFedHMIRcompared to state-of-the-art baselines.

Federated learning (FL) learns an optimal global model by aggregating local models trained on distributed data from different devices. Due to heterogeneous data distributions across devices, local models will be divergent, resulting in the global model’s performance degradation. Recent studies attempt to balance local models to obtain a global model that can adapt to each device. But they ignore a more challenging problem that redundant local models from devices will break the balance, resulting in the global model overfitting redundant local models. Therefore, we propose FedSmooth, a novel global aggregation algorithm. FedSmooth first identifies the redundant local models without sensitive local information (e.g., label distribution), then designs a smooth global aggregation to strengthen the effect of local models that can accelerate finding the optimal global model. Experimental results show that our method outperforms 4 SOTA baseline methods even if there is more redundancy.

Identity recognition plays an important role in ensuring security in our daily life. Biometric-based (especially activity-based) approaches are favored due to their fidelity, universality, and resilience. However, most existing machine learning-based approaches rely on a traditional workflow where models are usually trained once for all, with limited involvement from end users in the process and neglecting the dynamic nature of the learning process. This makes the models static and cannot be updated in time, which usually leads to high false positive or false negative. Thus, in practice, an expert is desired to assist with providing high-quality observations and interpretation of model outputs. It is expedient to combine both advantages of human experts and the computational capability of computers to create a tight-coupling incremental learning process for better performance. In this study, we develop RLTIR, an interactive identity recognition approach based on reinforcement learning, to adjust the identification model by human guidance. We first build a base tree-structured identity recognition model, and an expert is introduced in the model for giving feedback upon model outputs. Then, the model is updated according to strategies that are automatically learned under a designated reinforcement learning framework. To the best of our knowledge, it is the very first attempt to combine human expert knowledge with model learning in the area of identity recognition. The experimental results show that the reinforced interactive identity recognition framework outperforms baseline methods with regard to recognition accuracy and robustness.

It is still a challenge to detect anomalous events in video sequences in the field of computer vision due to heavy object occlusions, varying crowded densities and complex situations. To address this, we propose a novel human-machine cooperative approach which uses human feedback on anomaly confirmation to inform and enhance video anomaly detection. Specifically, we analyze the spatio-temporal characteristics of sequential frames of a video from the appearance and motion perspective from which spatial and temporal features are identified and extracted. We then develop a convolutional autoencoder neural network to compute an abnormal score based on reconstruction errors. In this process, a group of experts will provide human feedback to a certain proportion of classified frames to be incorporated into the model, and also the final judgment for the event anomalies for training and classification. The proposed approach is evaluated on 3 publicly available surveillance datasets, showing improved accuracy and competitive performance (93.7% AUC) with respect to the best performance (90.6% AUC) of the state-of-the-art approaches. The approach has not been previously seen to the best of our knowledge.

Housing demand estimation is an important topic in the field of economic research. It is beneficial and helpful for various applications including real estate market regulation and urban planning, and therefore is crucial for both real estate investors and government administrators. Meanwhile, given the rapid development of the express industry, abundant useful information is embedded in express delivery records, which is helpful for researchers in profiling urban life patterns. The express delivery behaviors of the residents in a residential community can reflect the housing demand to some extent. Although housing demand has been analyzed in previous studies, its estimation has not been very good, and the subject remains under explored. To this end, in this article, we propose a systematic housing demand estimation method based on express delivery data. First, the express delivery records are aggregated on the community scale with the use of clustering methods, and the missing values in the records are completed. Then, various features are extracted from a less sparse dataset considering both the probability of residential mobility and the attractiveness of residential communities. In addition, given that the correlations between different districts can influence the performances of the inference model, the commonalities and differences of different districts are considered. After obtaining the features and correlations between different districts being obtained, the housing demand is estimated by using a multi-task learning method based on neural networks. The experimental results for real-world data show that the proposed model is effective at estimating the housing demand at the residential community level.

Estimation of housing requirement is beneficial for many applications such as guidance of house trading and real estate market regulation. Although there have been studies focusing on the demand analysis of urban resources, estimation of housing requirement is still under explored. To this end, in this paper we propose a systematic housing demand inference method, named Housing Demand Inference Model (HDIM), to estimate housing demand by exploiting the residential mobility of communities based on express delivery data. In this work, we first aggregate the express delivery records at community scale with clustering methods. Then, we propose a useful method to infer residential mobility by extracting express delivery related features and community related features. Since the features extracted are sparse for some residents, we utilize Regularized Singular Value Decomposition Model (RSVD) to construct missing values of features. After that, we infer residential mobility probability of each community by taking advantage of the less sparse features. We also consider community attractiveness as one of the factors influencing housing demand with the help of community profiles and geographical data. With the residential mobility probability and community attractiveness being obtained, we estimate housing demand with a regression model. Finally, experimental results on real-world data show that our model is effective to infer housing demand for communities in urban areas.