By allowing each client to refer to the knowledge from other clients while retaining their specific characteristics, partial knowledge sharing has become one of the main approaches to realizing personalized federated learning (pFL). Representative techniques of partial knowledge sharing propose sharing the feature extractor while customizing the classifier head of the neural network. Although such methods achieve great success, the underlying principle behind them remains yet to be comprehensively understood. A fundamental problem is whether it is really appropriate to fully share the feature extractor. Based on the theory of neural collapse, in this paper, we demonstrate both theoretically and empirically that the feature extractor should be partially shared rather than fully shared. More specifically, we identify a substantial inconsistency between the fused global feature representations and expected local feature representations, and thus it is necessary to preserve partially customized layers of the feature extractor for enhancing personalized representations. Based on this discovery, we further propose a novel method called pFedDKS which detaches the shared global knowledge and customized local knowledge by providing detached feature prototypes. Extensive experiments on various datasets and models show that pFedDKS outperforms state-of-the-arts.

Recommendation systems based on deep learning and graph neural networks have effectively captured complex user-item relationships but struggle to fully leverage textual data for personalized recommendations. This is primarily due to three key challenges: (1) shallow semantic integration in language model representations, (2) semantic degradation during the fusion of large language model (LLM) embeddings with collaborative filtering signals, and (3) scalability bottlenecks in sparse data and large-scale interaction scenarios. To address these challenges, we propose SIP-Rec, a Semantic Integrity Preserving Recommendation Framework based on LLM and contrastive learning. First, SIP-Rec utilizes LLM to generate semantically rich profiles, accurately capturing fine-grained differences in user preferences and item characteristics. Second, it introduces an innovative cross-modal fusion mechanism that employs a dynamic alignment strategy to bridge the representation space gap between high-dimensional LLM embeddings and collaborative filtering signals, effectively mitigating contextual information loss. Finally, by integrating knowledge distillation with contrastive learning objectives, SIP-Rec optimizes the consistency between semantically rich representations and interaction patterns.

Inferring user status at the edge is essential for delivering personalized services, such as detecting emotional states. However, deploying large-scale models directly on user devices is impractical due to substantial computational overhead and the scarcity of labeled data. Conversely, uploading raw data to the cloud for processing raises significant privacy concerns and incurs prohibitive communication costs. To address this challenge, we propose a privacy-preserving multimodal inference framework that leverages large-scale public data while safeguarding sensitive information and optimizing computational efficiency. Specifically, we first train a teacher model in the cloud using publicly available data. Through a feature distillation process, the knowledge from this teacher model is transferred to a lightweight encoder deployed at the user end. This transfer is tailored to the user's data, ensuring that only relevant knowledge is distilled. To accommodate varying communication constraints, we introduce a feature compression mechanism that significantly reduces communication overhead without compromising inference accuracy. Extensive experiments on emotion recognition tasks demonstrate that the proposed framework effectively balances privacy preservation, resource efficiency, and inference accuracy, facilitating seamless collaboration between cloud and edge devices.

This paper investigates a new, practical, but challenging problem named Non-exemplar Online Class-incremental continual Learning (NO-CL), which aims to preserve the discernibility of base classes without buffering data examples and efficiently learn novel classes continuously in a single-pass (i.e., online) data stream. The challenges of this task are mainly two-fold: (1) Both base and novel classes suffer from severe catastrophic forgetting as no previous samples are available for replay. (2) As the online data can only be observed once, there is no way to fully re-train the whole model, e.g., re-calibrate the decision boundaries via prototype alignment or feature distillation. In this paper, we propose a novel Dual-prototype Self-augment and Refinement method (DSR) for NO-CL problem, which consists of two strategies: 1) Dual class prototypes: vanilla and high-dimensional prototypes are exploited to utilize the pre-trained information and obtain robust quasi-orthogonal representations rather than example buffers for both privacy preservation and memory reduction. 2) Self-augment and refinement: Instead of updating the whole network, we optimize high-dimensional prototypes alternatively with the extra projection module based on self-augment vanilla prototypes, through a bi-level optimization problem. Extensive experiments demonstrate the effectiveness and superiority of the proposed DSR in NO-CL.

Recently, Multimodal Learning (MML) has gained significant interest as it compensates for single-modality limitations through comprehensive complementary information within multimodal data. However, traditional MML methods generally use the joint learning framework with a uniform learning objective that can lead to the modality competition issue, where feedback predominantly comes from certain modalities, limiting the full potential of others. In response to this challenge, this paper introduces DI-MML, a novel detached MML framework designed to learn complementary information across modalities under the premise of avoiding modality competition. Specifically, DI-MML addresses competition by separately training each modality encoder with isolated learning objectives. It further encourages cross-modal interaction via a shared classifier that defines a common feature space and employing a dimension-decoupled unidirectional contrastive (DUC) loss to facilitate modality-level knowledge transfer. Additionally, to account for varying reliability in sample pairs, we devise a certainty-aware logit weighting strategy to effectively leverage complementary information at the instance level during inference. Extensive experiments conducted on audio-visual, flow-image, and front-rear view datasets show the superior performance of our proposed method. The code is released at https://github.com/fanyunfeng-bit/DI-MML.

Multi-modal domain generalization (MMDG) requires that models trained on multi-modal source domains can generalize to unseen target distributions with the same modality set. Sharpness-aware minimization (SAM) is an effective technique for traditional uni-modal domain generalization (DG), however, with limited improvement in MMDG. In this paper, we identify that modality competition and discrepant uni-modal flatness are two main factors that restrict multi-modal generalization. To overcome these challenges, we propose to construct consistent flat loss regions and enhance knowledge exploitation for each modality via cross-modal knowledge transfer. Firstly, we turn to the optimization on representation-space loss landscapes instead of traditional parameter space, which allows us to build connections between modalities directly. Then, we introduce a novel method to flatten the high-loss region between minima from different modalities by interpolating mixed multi-modal representations. We implement this method by distilling and optimizing generalizable interpolated representations and assigning distinct weights for each modality considering their divergent generalization capabilities. Extensive experiments are performed on two benchmark datasets, EPIC-Kitchens and Human-Animal-Cartoon (HAC), with various modality combinations, demonstrating the effectiveness of our method under multi-source and single-source settings. Our code is open-sourced.

Multimodal learning (MML) aims to jointly exploit the common priors of different modalities to compensate for their inherent limitations. However, existing MML methods often optimize a uniform objective for different modalities, leading to the notorious “modality imbalance” problem and counterproductive MML performance. To address the problem, some existing methods modulate the learning pace based on the fused modality, which is dominated by the better modality and eventually results in a limited improvement on the worse modal. To better exploit the features of multimodal, we propose Prototypical Modality Rebalance (PMR) to perform stimulation on the particular slow-learning modality without interference from other modalities. Specifically, we introduce the prototypes that represent general features for each class, to build the non-parametric classifiers for uni-modal performance evaluation. Then, we try to accelerate the slow-learning modality by enhancing its clustering toward prototypes. Furthermore, to alleviate the suppression from the dominant modality, we introduce a prototype-based entropy regularization term during the early training stage to prevent premature convergence. Besides, our method only relies on the representations of each modality and without restrictions from model structures and fusion methods, making it with great application potential for various scenarios. The source code is available here11https://github.com/fanyunfeng-bit/Modal-Imbalance-PMR.

Federated Learning enables collaboratively model training among a number of distributed devices with the coordination of a centralized server, where each device alternatively performs local gradient computation and communication to the server. FL suffers from significant performance degradation due to the excessive communication delay between the server and devices, especially when the network bandwidth of these devices is limited, which is common in edge environments. Existing methods overlap the gradient computation and communication to hide the communication latency to accelerate the FL training. However, the overlapping can also lead to an inevitable gap between the local model in each device and the global model in the server that seriously restricts the convergence rate of learning process. To address this problem, we propose a new overlapping method for FL, AOCC-FL, which aligns the local model with the global model via calibrated compensation such that the communication delay can be hidden without deteriorating the convergence performance. Theoretically, we prove that AOCC-FL admits the same convergence rate as the non-overlapping method. On both simulated and testbed experiments, we show that AOCC-FL achieves a comparable convergence rate relative to the non-overlapping method while outperforming the state-of-the-art overlapping methods.

The problem of order and rack assignment to picking stations (ORAPS) is a key joint optimization problem in the order picking process of the robotic mobile fulfillment system (RMFS). Given a set of customer orders and a set of mobile racks, the goal of this problem is to simultaneously assign orders and racks to multiple picking stations so that the racks can provide products to meet the demand of orders by the minimal number of visits to all picking stations. In this article, we build a mathematical model for the ORAPS, which considers all orders assignment with the allocable capacity interval of the picking station and the rack product capacity limitation. To solve the ORAPS problem, we propose a two-stage hybrid heuristic algorithm (TS-HHA) consisting of the reducing stage and the assigning stage. In the reducing stage, a scheme of dynamic programming (DP) is introduced to find a critical rack set, which can focus attention on the most promising racks and increase the speed of problem-solving. In the assigning stage, we propose an optimization strategy that combines a constructive heuristic algorithm and adaptive neighborhood search (CH-ANS). It can generate a high-quality simultaneous assignment scheme and further improve its quality effectively. The computational results show that our proposed algorithm performs better than its competitors on both simulation and practical instances of the ORAPS problem. Note to Practitioners—This article investigates the order and rack assignment to picking stations (ORAPS) problem in the robotic mobile fulfillment system (RMFS) originated from a renowned Chinese logistics platform. We build a more comprehensive and reasonable mathematical programming model for this ORAPS problem and propose a two-stage hybrid heuristic algorithm (TS-HHA) strategy, which can simultaneously assign orders and racks to multiple picking stations so that all customer orders can be processed. The experimental results show that our TS-HHA performs at least 35% better than its competitors on all simulation and practical instances set. We believe that this research work can serve as a kind of generic framework for practical ORAPS problems and be very helpful for operating the RMFS efficiently.

Place recognition is essential for SLAM system since it is critical for loop closure and can help to correct the accumulated drift and result in a globally consistent map. Unlike the visual slam which can use diverse feature detection methods to describe the scene, there are limited works reported to represent a place using single LiDAR scan. In this paper, we propose a segmentation-based egocentric descriptor termed Seed by using a single LiDAR scan to describe the scene. Through the segmentation approach, we first obtain different segmented objects, which can reduce the noise and resolution effect, making it more robust. Then, the topological information of the segmented objects is encoded into the descriptor. Unlike other reported approaches, the proposed method is rotation invariant and insensitive to translation variation. The feasibility of proposed method is evaluated through the KITTI dataset and the results show that the proposed method outperforms the state-of-the-art method in terms of accuracy.

This paper presents a novel sliding mode controller for trajectory tracking of the piezo-driven stage. The tracking performance of piezoelectric actuator is mainly affected by the hysteresis nonlinearity. Sliding mode control is a possible solution to achieve better tracking performance. However, conventional sliding mode control generates discontinuous control signal which results in chattering. Hence, the hysteresis nonlinearity is first compensated with a hysteresis model, and an uncertainty and disturbance estimator is designed and included to devise a smooth control action. The stability of the proposed method is demonstrated via Lyapunov analysis. Both simulation and experiment are also conducted to verify the effectiveness of the proposed approach. The results are compared with a conventional sliding mode controller and a proportional-integral control with notch filter (PIC-NF).