Multi-turn conversational systems based on large language models are increasingly being integrated into web platforms and applied across a wide range of domains. However, these systems typically combine the userߣs current query with contextual information from previous interactions, resulting in continuously expanding input prompts. This leads to a significant increase in time-to-first-token (TTFT), causing intolerable delays in web response times. To address this issue, we introduce TransformKV, which maximizes the reuse of the KV cache from previous conversations rather than recomputing, thereby reducing TTFT latency. TransformKV first identifies the specific locations that require transformation to maximize the reuse of the KV cache with minimal operations. It then efficiently transforms the KV cache for a subset of tokens by recomputing only the KV cache that impact semantics. Additionally, TransformKV further reduces TTFT latency by performing only QKV computations in certain layers while skipping other computations that contribute less to overall performance. Experimental results demonstrate that in multi-turn conversation tasks, TransformKV can reduce inference latency by up to 30%, achieving up to a 1.8× improvement in performance compared to similar approaches. Notably, as the context window size increases, the performance gains become even more pronounced.

Federated prototype learning is in the spotlight as global prototypes are effective in enhancing the learning of local representation spaces, facilitating the ability to generalize the global model. However, when encountering domain-skewed data, conventional federated prototype learning is susceptible to two dilemmas: 1) Local prototypes obtained by averaging intra-class embedding carry domain-specific markers, the margins among aggregated global prototypes could be attenuated and detrimental to inter-class separation. 2) Local domain-skewed embedding may not exhibit a uniform distribution in Euclidean space, which is not conductive to the prototype-induced intra-class compactness. To address the two drawbacks, we go beyond conventional paradigm of federated prototype learning, and propose learnable semantic anchors with hyperspherical contrast (FedLSA) for domain-skewed data. Specifically, we eschew the pattern of yielding prototypes via averaging intra-class embedding and directly learn a set of semantic anchors aided by the global semantic-aware classifier. Meanwhile, the margins between anchors are augmented via pulling apart them, ensuring decent inter-class separation. To guarantee that local domain-skewed representations can be uniformly distributed, local data is projected into the hyperspherical space, and the intra-class compactness is achieved by optimizing the contrastive loss derived from the von Mises-Fisher distribution. Finally, extensive experimental results on three multi-domain datasets show the superiority of the proposed FedLSA compared to existing typical and state-of-the-state methods.

Multi-view graph clustering (MVGC) has been of widespread interest owing to the ability of capturing the complementary information among views, thereby enhancing the performance of node clustering. Despite the impressive achievements of existing methods, they are limited by a common deficiency, namely, the curse of local manifold while failing to perceive the global manifold structure. In light of this drawback, we propose a Consistent Context-Aware Representation Learning (CCARL) method for MVGC, aiming to learn node representations from global space rather than just local topology. Concretely, we define a set of anchors to establish the global coordinate, which are optimally mapped to multi-view graphs with minimal cost via fused Gromov-Wasserstein optimal transport. To fuse the complementary information in various views, the attention mechanism is employed to integrate multiple graph embeddings into a consistent representation. By transforming to the global coordinate connecting with anchors, the consistent representation captures the contextual information, and its clustering-friendliness is further enhanced through a self-training strategy. Finally, extensive experiments on four multi-view graph datasets demonstrate the effectiveness of the proposed CCARL over existing MVGC methods.

Federated graph learning is focused on aggregating knowledge from multi-source graph data and training graph neural networks. Unlike the data that traditional federated learning needs to deal with, federated graph learning also needs to face additional topological information. Further, there are also biases in features and topologies among clients, increasing the difficulty of training models. Previous methods usually seek global calibration information, however, this approach may suffer from information bias caused by data skews, and it is also difficult to naturally combine feature and topology information. Therefore, adjusting the bias before it occurs will hopefully address the learning difficulties caused by the skew. In view of this, we employ background graph data, which works as reference information for local training, to proactively correct bias before it occurs. As a kind of graph data, background graphs are naturally capable of combining feature and topology information to accomplish bias correction among clients in a comprehensive way. Mixing strategy is employed on the background graph to additionally provide privacy-preserving capabilities. Graph generation methods are employed to restore the diversity of background graphs that are blurred by the mixing strategy. Extensive experiments on two real-world datasets demonstrate the sufficient motivation and effectiveness of the proposed method.

As large language models (LLMs) are widely applied across various domains, improving the quality of LLM inference services is essential. In this paper, we find that optimizing configuration parameters of LLM inference engines can significantly improve LLM inference performance in terms of latency and throughput. Therefore, we propose LLMConf, an automated performance tuning system that optimizes multiple LLM inference performance metrics by searching for the optimal configuration parameters of the LLM inference engine. We first introduce a knowledge-enhanced approach to identify the set of configuration parameters (LLMConfigs) that most significantly impact LLM performance from the adjustable parameters provided by the LLM inference engine. We then perform automated data collection to build functional relationships between LLMConfigs and each performance metric. Additionally, LLMConf employs a multi-objective optimization module to obtain optimal LLMConfigs for simultaneously optimizing multiple performance metrics. The experimental results show that LLMConf significantly outperforms existing methods. Compared to the default configuration parameters of the LLM inference engine, LLMConf achieves an average improvement of 20.1% across 7 key performance metrics. Moreover, experiments demonstrate that LLMConf has strong transferability across diverse datasets, varying concurrency levels and different LLM base models.

Federated graph learning (FGL) is a privacy-preserving paradigm for modeling distributed graph data, designed to train a powerful global graph neural network. Existing FGL methods predominantly rely on label information during training, effective FGL in an unsupervised setting remains largely unexplored territory. In this paper, we address two key challenges in unsupervised FGL: 1) Local models tend to converge in divergent directions due to the lack of shared semantic information across clients. Then, how to align representation spaces among multiple clients is the first challenge. 2) Conventional federated weighted aggregation easily results in degrading the performance of the global model, then which raises another challenge, namely how to adaptively learn the global model parameters. In response to the two questions, we propose a tailored framework named FedPAM, which is composed of two modules: Representation Space Alignment (RSA) and Adaptive Global Parameter Learning (AGPL). RSA leverages a set of learnable anchors to define the global representation space, then local subgraphs are aligned with them through the fused Gromov-Wasserstein optimal transport, achieving the representation space alignment across clients. AGPL stacks local model parameters into third-order tensors, and adaptively integrates the global model parameters in a low-rank tensor space, which facilitates to fuse the high-order knowledge among clients. Extensive experiments on eight graph datasets are conducted, the results demonstrate that the proposed FedPAM is superior over classical and SOTA compared methods.

Federated learning enables collaborative training while preserving the privacy of all participants. However, the heterogeneity in data distribution across multiple training nodes poses significant challenges to the construction of federated models. Prior studies were dedicated to mitigating the effects of data heterogeneity by using global information as a blueprint and restricting the local update of the model for reaching a "hard consensus". But this practice makes it difficult to balance local and global information, and it neglects to negotiate amicably between local and global models to reach mutually agreeable results, called ``soft consensus". In this paper, a multiple-path solving method is proposed to balance global and local features and combine these two feature preference paths to reach a soft consensus. Rather than relying on global information as the sole criterion, a negotiation process is employed to address the same objective by accommodating diverse feature preferences, thereby facilitating the discovery of a more plausible solution through multiple distinct pathways. Considering the overwhelming power of local features during local training, a swapping strategy is applied to weaken them to balance the solution paths. Moreover, to minimize the additional communication cost caused by the introduction of multiple paths, the solution of the task network is converted into data adaptation to reduce the amount of parameter transmission. Extensive experiments are conducted to demonstrate the advantages of the proposed method.

Federated learning provides a privacy-preserving modeling schema for distributed data, which coordinates multiple clients to collaboratively train a global model. However, data stored in different clients may be collected from diverse domains, and the resulting feature shift is prone to the degraded performance of global model. In this paper, we propose a Federated Domain-Independent Prototype Learning (FedDP) method with Alignments of Representation and Parameter Spaces for Feature Shift. Concretely, FedDP aims to eliminate the domain-specific information and explore the pure representations via information bottleneck, thus integrating the local and global domain-independent prototypes, respectively. To align the cross-domain representation spaces, the global domain-independent prototypes serve as the supervised signals to enable local intra-class representations to approach them. Further, to mitigate the divergences of optimization directions between multiple clients induced by the feature shift, the global representations are yielded by the global model on the client-side and guide the learning of local representations, thus unifying the parameter spaces of multiple local models. We derive the theoretical lower bound of the optimization objective based on mutual information, which is transformed into a computable loss. The proposed FedDP can be applied in the scenarios of homogeneous and heterogeneous models. Extensive experiments are conducted on three challenging multi-domain datasets. The experimental results illustrate the superiority of FedDP compared with state-of-the-art federated learning methods.

Skill-based deep reinforcement learning (DRL) typically embeds skills extracted from expert demonstrations into a large latent space. However, it is difficult for agents to explore the latent skill space, because only a small portion of the space is relevant to a given robot state. To this end, we propose a three-layer algorithm to realize that the agent can adaptively select skills and directly explore the skill region associated with the state. The high-level policy outputs skill categories with a given state; then, a conditional probability model is constructed using the state and skill categories to guide the agent’s exploration. We first learn high- and mid-level skill prior models and a low-level controller with offline data. Next, we learn high-level and mid-level policies to construct two different action spaces. The high-level action is used to guide the mid-level policy to explore skill-related regions; the mid-level is used to guide low-level models to generate skill-related robot actions. Finally, we validate our approach in three challenging manipulation tasks, and the experimental results show that our approach can significantly speed up exploration and cross-task learning.

Path planning is a crucial process for unmanned aerial vehicles (UAVs) and involves finding a path that is both short and safe. However, with the ever-increasing complexity of the environment, solving the UAV path-planning problem is challenging. Traditional path-planning methods cannot handle conflicting goals effectively, and existing objective methods lack targeted exploration mechanisms, resulting in unsatisfactory outcomes. By modeling the UAV path-planning problem via multi-objective optimization, this study designed a reasonable objective function composition for the model and considered obstacle avoidance as a hard constraint to satisfy the actual situation . A multi-objective evolutionary algorithm based on dimensional exploration and discrepancy evolution (MOEA-2DE) is presented. In particular, MOEA-2DE utilizes dimensional perturbation to identify key dimensions to facilitate prior exploration and enhance the targeted search. An adaptive evolution strategy based on population discrepancy was employed to assess the evolution process, and various methods were adopted to balance convergence and diversity. The effectiveness of the MOEA-2DE was demonstrated through the design of two intricate terrain sets and comparisons with various classic and state-of-the-art multi-objective evolutionary algorithms (MOEAs), including those designed for UAV path planning across multiple metrics. The results verify the superiority of MOEA-2DE in terms of both convergence speed and final effect.

Most multi-view clustering methods based on shallow models are limited in sound nonlinear information perception capability, or fail to effectively exploit complementary information hidden in different views. To tackle these issues, we propose a novel Subspace-Contrastive Multi-View Clustering (SCMC) approach. Specifically, SCMC utilizes a set of view-specific auto-encoders to map the original multi-view data into compact features capturing its nonlinear structures. Considering the large semantic gap of data from different modalities, we project multiple heterogeneous features into a joint semantic space, namely the embedded compact features are passed through the self-expression layers to learn the subspace representations, respectively. In order to enhance the discriminability and efficiently excavate the complementarity of various subspace representations, we use the contrastive strategy to maximize the similarity between positive pairs while differentiate negative pairs. Thus, the graph regularization is employed to encode the local geometric structure within varying subspaces for optimizing the consistent affinity matrix. Furthermore, to endow the proposed SCMC with the ability of handling the multi-view out-of-samples, we develop a consistent sparse representation (CSR) learning mechanism over the in-samples. To demonstrate the effectiveness of the proposed model, we conduct a large number of comparative experiments on ten challenging datasets, and the experimental results show that SCMC outperforms existing shallow and deep multi-view clustering methods. In addition, the experimental results on out-of-samples illustrate the effectiveness of the proposed CSR.

In deep multi-view clustering, three intractable problems are posed ahead of researchers, namely, the complementarity exploration problem, the information preservation problem, and the cluster structure discovery problem. In this paper, we consider the deep multi-view clustering from the perspective of mutual information (MI), and attempt to address the three important concerns with a Mutual Information-Driven Multi-View Clustering (MIMC) method, which extracts the common and view-specific information hidden in multi-view data and constructs a clustering-oriented comprehensive representation. Specifically, three constraints based on MI are devised in response to three issues. Correspondingly, we minimize the MI between the common representation and view-specific representations to exploit the inter-view complementary information. Further, we maximize the MI between the refined data representations and original data representations to preserve the principal information. Moreover, to learn a clustering-friendly comprehensive representation, the MI between the comprehensive embedding space and cluster structure is maximized. Finally, we conduct extensive experiments on six benchmark datasets, and the experimental results indicate that the proposed MIMC outperforms other clustering methods.