Cross-tokenizer knowledge distillation, where the teacher and student employ different tokenizers, is becoming increasingly prevalent, yet it poses underexplored challenges: existing methods fail to capture the rich knowledge encoded in teacher logits, as evidenced by the neglect of semantic information, inaccurate and biased logit alignment, and discarding distributional structure—ultimately leading to unfavorable distillation. To address these issues, we propose SeDi, a semantics and distribution-aware knowledge transfer framework tailored for cross-tokenizer distillation. To preserve factual knowledge, SeDi employs bipartite graph-based alignment at the tokenization level and a sliding window re-encoding strategy at the vocabulary level, enabling unbiased transfer of the teacher’s next-token predictions into the student’s vocabulary space. To further retain distributional information, we align the student’s entropy with that of the teacher by incorporating the student’s own logits during training, which helps to mitigate the exposure bias problem. Experiments on ten datasets across three task domains and five different teacher-student model pairs with varying vocabulary sizes demonstrate that SeDi delivers substantial improvements, with gains of up to 19.8%.

Yixiao He, Menghao Zhang, Haifeng Sun, Jing Wang, Kangheng Lin, Jinghan Wang, Chenye Xu, Pengfei Ren, Qi Qi, Jingyu Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Yuhao Li, Haifeng Sun, Xuesong Zhang, Shu Yao, Haoyu Zheng, Yvchuan Wang, Huazheng Wang, Zirui Zhuang, Qi Qi, Jianxin Liao, Jingyu Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Huazheng Wang, Yongcheng Jing, Haifeng Sun, Yingjie Wang, Jingyu Wang, Jianxin Liao, Dacheng Tao. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

In open-plan offices, visual noise from surrounding people and objects can negatively impact both concentration and mood. Mixed Reality (MR) offers a promising approach to address this challenge by reshaping the workspace. In this paper, we first conducted a survey with 50 office workers to examine the impact of visual noise, identifying common sources of distraction and potential mitigation strategies. Considering the necessity of face-to-face communication in office environments, we designed adaptive user interfaces to strike a balance between deep focus and seamless in-situ collaboration. We utilized Virtual Reality (VR) and Diminished Reality (DR) methods to eliminate visual noise and leveraged face orientation along with a distance threshold to determine collaborative intentions. We developed a prototype system and conducted a user study for evaluation. The results indicate that our system can create a tranquil workspace to foster concentration and workplace well-being, while maintaining necessary in-situ collaboration. These findings provide valuable insights for designing future MR-integrated office environments.

Tangible Augmented Reality (AR) enhances user immersion in virtual world by providing haptic feedback through physical proxy objects. However, existing approaches primarily focus on selecting proxy objects based on their global physical properties, neglecting the utilization of local features. Besides, the prevailing strategy of mapping one virtual object to a single dedicated physical proxy creates an inherent switching cost, limiting flexibility and efficiency. Additionally, due to challenges such as real-time performance, generalization and occlusion, the vision-based hand-object tracking remains a difficult task. In this paper, we propose Ubi Grip, an universal hand-object interaction framework for creating grip-based tangible AR applications based on the local graspable feature and a comprehensive hand-object interaction attributes methodology. We employ a lightweight object tracking method to perform tracking, utilizing a hand mask filter and transformation strategy to optimize object pose based on the hand-held properties. Moreover, we design a user-defined workflow for grasping tangible objects, allowing users to switch grips and map interactions. We evaluated our system through comprehensive algorithmic benchmarks and a user study. The benchmarks demonstrate our SOTA performance in object pose estimation and generalization, while the user study validates system usability, providing deeper insights.

Next Basket Recommendation (NBR) predicts unordered item sets for users' next purchases, crucial for grocery shopping and online retail scenarios. However, existing NBR methods face two critical challenges: (1) They neglect the threshold-meeting phenomenon, such as common add-to-cart behaviors where users intentionally include seemingly irrelevant items to meet promotional or free shipping price thresholds, which is a widespread phenomenon in reality but has not been a dedicated research focus in the recommendation domain; (2) Even though many works study repetitive and exploratory recommendations, they still lack a theoretically analyzable mechanism to directionally decouple these patterns. This may limit the model's performance. To address these issues, we propose DiffNBR, the first diffusion-model-based framework for NBR. DiffNBR employs two denoising diffusion probabilistic models (DDPMs) to jointly learn users' purchasing behaviors in in spatial and temporal dimensions, modeling the latent compositional strategies the dynamic evolution behind phenomena like threshold-meeting. Moreover, we integrate the information bottleneck theory to enforce directional decoupling between repetition and exploration by explicitly regulating information flows. Specifically, our framework constrains the model's generative representation to focus on learning the exploratory patterns, while the habitual repurchase representation is responsible for the repetitive patterns. Extensive experiments on four real-world datasets show that DiffNBR outperforms the state-of-the-art (SOTA) methods.

With the growing adoption of time-series anomaly detection (TAD) technology, numerous studies have employed deep learning-based detectors to analyze time-series data in the fields of Internet services, industrial systems, and sensors. The selection and optimization of anomaly detectors strongly rely on the availability of an effective evaluation for TAD performance. Since anomalies in time-series data often manifest as a sequence of points, conventional metrics that solely consider the detection of individual points are inadequate. Existing TAD evaluators typically employ point-based or event-based metrics to capture the temporal context. However, point-based evaluators tend to overestimate detectors that excel only in detecting long anomalies, while event-based evaluators are susceptible to being misled by fragmented detection results. To address these limitations, we propose OIPR1, a novel TAD evaluator with area-based metrics. It models the process of operators receiving detector alarms and handling anomalies, utilizing area under the operator interest curve to evaluate TAD performance. Furthermore, we build a special scenario dataset to compare the characteristics of different evaluators. Through experiments conducted on the special scenario dataset and five real-world datasets, we demon-strate the remarkable performance of OIPR in extreme and complex scenarios. It achieves a balance between point and event perspectives, overcoming their primary limitations and offering applicability to broader situations.

Online video understanding is pivotal for emerging video streaming services. However, existing Multimodal Large Language Models (MLLMs) encounter significant challenges in this domain, specifically in maintaining holistic visual perception under strict token budgets and attaining grounded semantic reasoning amidst dynamic context changes. To mitigate these issues, we propose OV-DPO, a systematic multimodal preference optimization approach. To ensure holistic visual perception, we introduce a visual preference objective that compels the model to strictly ground its reasoning in clear visual evidence. This is supported by the QGDFR strategy, which dynamically reallocates resolution budgets to preserve critical visual details for chosen samples while intentionally degrading them for rejected ones to construct valid visual contrasts. Concurrently, to promote grounded semantic reasoning, we employ a textual preference objective. We design the SROVA framework to construct high-quality preference pairs, utilizing self-refinement to generate factual chosen responses and visual degradation to induce language-prior-driven hallucinations as rejected samples. By jointly optimizing these objectives alongside anchored and supervised fine-tuning terms, OV-DPO ensures training stability and robust alignment. Experimental results demonstrate that our approach significantly outperforms baselines such as SFT and DPO. Notably, utilizing only 29K high-quality samples, our 3B-scale model achieves results competitive with significantly larger-scale models on both online and offline benchmarks, offering an efficient solution for online video understanding.

Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime.

Sparse attention can effectively alleviate the significant demands on memory when large language models (LLMs) process long contexts. Existing methods typically apply the same sparse pattern across different attention heads and inputs. However, this uniform approach fails to capture the inherent diversity of attention patterns within LLMs — the intrinsic attention clustering. To address this, we propose ClusterAttn, a training-free sparse attention method that provides an efficient prompt cache compression scheme under intrinsic attention clustering for efficient LLM inference.Our findings show that attention heads consistently focus on specific clusters of the prompt during decoding, a pattern detectable from an observation window at the prompt’s end. ClusterAttn adaptively fits these clusters utilizing a density-based attention clustering algorithm, thus compressing the KV cache of the prompt. Evaluations on different models across various benchmarks demonstrate ClusterAttn’s superior compression rates and efficiency. By utilizing only 1024 tokens, it can reduce memory usage by 10%–65%, resulting in a latency reduction of 12%–23% and a throughput increase of 2.6–4.8 times, all with nearly no accuracy loss. Additionally, ClusterAttn can handle up to 128k context on a single A100-80GB GPU, outperforming existing methods.

3D hand reconstruction is essential in non-contact human-computer interaction applications, but existing methods struggle with low-resolution images, which occur in slightly distant interactive scenes. Leveraging temporal information can mitigate the limitations of individual low-resolution images that lack detailed appearance information, enhancing the robustness and accuracy of hand reconstruction. Existing temporal methods typically use joint features to represent temporal information, avoiding interference from redundant background information. However, joint features excessively disregard the spatial context of visual features, limiting hand reconstruction accuracy. We propose to integrate temporal joint features with visual features to construct a robust low-resolution visual representation. We adopt Triplane Features, a dense representation with 3D spatial awareness, to bridge the gap between the joint features and visual features that are misaligned in terms of representation form and semantics. Triplane Features are obtained by orthogonally projecting joint features, embedding hand structure information into the 3D spatial context. Furthermore, we compress the spatial information of the three planes into a 2D dense feature thourgh Spatial-Aware Fusion to enhance the visual features. By using enhanced visual features enriched with temporal information for hand reconstruction, our method achieves competitive performance at much lower resolutions compared to state-of-the-art methods operating at high resolution on DexYCB, HanCo and H2O. Code is available at https://github.com/NewbieFan/Temp-LowRes-hand.

Yiran Yang, Haifeng Sun, Jingyu Wang, Qi Qi, Zirui Zhuang, Huazheng Wang, Pengfei Ren, Jing Wang, Jianxin Liao. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Data plane verification (DPV) can be critical in ensuring the network operates correctly. To be useful in practice, they need to be: (1) fast so as to prevent significant packet loss or security violations; (2) scalable so as to accommodate today's large-scale network architecture. Current DPV tools struggle to meet these requirements due to their centralized architecture. To be concrete, there is a bottleneck for a single-point server to perform real-time DPV tasks. Furthermore, a single-point server makes it hard to collect real-time data plane updates from every device in large-scale networks.

Large language models (LLMs) have demonstrated outstanding performance in various tasks. However, the memory footprint of the key-value (KV) cache generated during model inference poses significant challenges for efficient model deployment. This paper presents a detailed analysis of the KV cache and identifies potential sources of quantization errors. We find that the attention scores exhibit a strong power-law distribution while showing an additional attention to the initial and recent tokens, and that only a small subset of outlier channels in KV cache span a wide numerical range. Based on these findings, we propose QoKV, an efficient post-training quantization (PTQ) method designed for the KV cache, which facilitates accurate low-bit quantization through a novel group evaluation strategy and numerical scaling. Extensive experimental results across various tasks demonstrate that our proposed method outperforms existing methods, achieving near-floating-point performance while saving 73.75% of the cache footprint.

Large Vision-Language Models (LVLMs) suffer from severe object hallucinations, leading them to frequently generate outputs that do not correspond to the image content, significantly reducing the credibility and reliability of their responses. Recent research has attempted to enhance LVLMs by employing Direct Preference Optimization (DPO) to reduce hallucinations and improve response quality. However, these approaches typically utilize text-only preference response pairs, neglecting the influence of visual input in optimizing LVLMs. In this paper, we propose VA-DPO, a multimodal optimization objective. VA-DPO leverages the LVLMs' attention to select and corrupt critical parts of images, thereby constructing visual preference image pairs. This approach integrates both text and visual preference optimization objectives to achieve effective alignment optimization of LVLMs. We conduct extensive experiments on LVLMs of different sizes, and the results demonstrate that VA-DPO effectively reduces hallucinations across various tasks. Compared to other hallucination mitigation approaches, VA-DPO achieves more competitive results.

Recent progress in video anomaly detection assumes that anomalies cannot be effectively reconstructed because they remain unseen during training. However, we observe that most existing methods excessively rely on appearance features, resulting in the accurate reconstruction of anomalies with subtle short-term appearance variations, which we refer to as appearance confusion. Meanwhile, many approaches fail to exploit sufficient semantic distinction, resulting in motion confusion for anomalies with motion patterns similar to normal ones. In this paper, we propose a masked self-supervised learning-based framework, which effectively addresses the two confusions by exploring context-aware motion patterns and discriminative semantic normality representations. First, we introduce reconstructing multi-pattern masked spatiotemporal information to motivate the model to capture motion patterns that focus on long-term context. Then, we design a semantic noise separation network to address motion confusion, facilitating the construction of semantic normality boundaries through semantic-aware separation. Extensive experiments on the Avenue and ShanghaiTech datasets validate the effectiveness of our proposed method.

Precise 3D hand posture is essential for learning musical instruments. Reconstructing highly precise 3D hand gestures enables learners to correct and master proper techniques through 3D simulation and Extended Reality. However, exsiting methods typically rely on precisely calibrated multi-camera systems, which are not easily deployable in everyday environments. In this paper, we focus on calibration-free multi-view 3D hand reconstruction in unconstrained scenarios. Establishing correspondences between multi-view images is particularly challenging without camera extrinsics. To address this, we propose A^3-Net, a multi-level alignment framework that utilizes 3D structural representations with hierarchical geometric and explicit semantic information as alignment proxies, facilitating multi-view feature interaction in both 3D geometric space and 2D visual space. Specifically, we first perfrom global geometric alignment to map multi-view features into a canonical space. Subsequently, we aggregate information into predefined sparse and dense proxies to further integrate cross-view semantics through mutual interaction. Finnaly, we perfrom 2D alignment to align projected 2D visual features with 2D observations. Our method achieves state-of-the-art results in the multi-view 3D hand reconstruction task, demonstrating the effectiveness of our proposed framework.

Deep learning-based recommendation systems are increasingly important in the industry. To meet strict SLA requirements, serving frameworks must efficiently handle concurrent queries. However, current serving systems fail to serve concurrent queries due to the following problems: (1) inefficient operator (op) scheduling due to the query-wise op launching mechanism, and (2) heavy contention caused by the mutable nature of recommendation model inference. This paper presents RecOS, a system designed to optimize concurrent recommendation model inference on GPUs. RecOS efficiently schedules ops from different queries by monitoring GPU workloads and assigning ops to the most suitable streams. This approach reduces contention and enhances inference efficiency by leveraging inter-op parallelism and op characteristics. To maintain correctness across multiple CUDA streams, RecOS introduces a unified asynchronous tensor management mechanism. Evaluations demonstrate that RecOS improves online service performance, reducing latency by up to 68%.

Accurate forecasting of time series is crucial for many applications in the real world. Conventional methods primarily rely on statistical analysis of historical data, often leading to overfitting and failing to account for background information and constraints imposed by external events. Therefore, introducing large language models (LLMs) with robust textual capabilities holds significant potential. However, due to the inherent limitations of LLMs in handling numerical data, they do not exhibit advantages in precise numerical prediction tasks. Therefore, we propose a framework to integrate LLMs with conventional methods synergistically. Rather than directly outputting numerical predictions, we leverage the capabilities of the LLMs to generate textual temporal patterns, thereby fully utilizing their inherent knowledge and reasoning abilities. Additionally, we introduce a memory network designed to decode these textual representations into a format that numerical models can effectively interpret. This approach not only capitalizes on the strengths of the LLM in text processing but also bridges the gap between textual and numerical data, enhancing the overall predictive performance of the model. Our experimental results demonstrate the framework's effectiveness, achieving state-of-the-art performance on various benchmark datasets.

Self-supervised 3D hand pose estimation methods can leverage labeled synthetic data along with unlabeled real-world data for model training, thereby alleviating the reliance on large-scale annotated datasets. Multi-view information fusion is a key factor in the success of these methods. Rule-based fixed fusion methods are simple, efficient, and generalizable, but they neglect the rich visual information in each view. Neural network-based learnable fusion methods can effectively model both intra- and inter-view semantic context, but they tend to overfit to the domain-specific feature of synthetic data and susceptible to interference of domain gaps. In this paper, we decompose multi-view fusion into two components: a learnable confidence estimation stage and a fixed confidence fusion stage. This design not only enables effective use of multi-view semantic cues but also ensures strong cross-domain generalization. To achieve accurate and robust confidence estimation, our method jointly exploits both multi-view pose consistency and pose-to-data consistency. Experiments on three public datasets demonstrate that our approach significantly outperforms existing state-of-the-art self-supervised 3D hand pose estimation methods.

Synthetic images serve as a promising alternative to real images in 3D hand pose estimation, providing accurate annotations at a lower cost. However, the domain gap between real and synthetic images constrains the generalization ability of hand pose estimation trained on synthetic data. Previous methods rely on Generative Adversarial Networks (GANs) for domain translation; however, they fail to achieve realistic depth synthesis due to instability and limited image quality. Diffusion models provide high-quality synthesis due to their stability and controllability. However, existing methods often ignore the 3D structure awareness in hand image generation. In this paper, we propose a Dual-Branch 3D Spatial-Aware Latent Diffusion (DSW-LD) for realistic depth image generation. The Global Structure Module (GSM) and the Local Geometry Module (LGM) complement each other, with GSM capturing global spatial structure through coarse-grained 3D joint features and LGM focusing on local geometric details using fine-grained 3D mesh representations. To maintain the global structure consistency, we adopt a layer-aware injection mechanism that enables the model to adaptively learn the optimal representation from fused 2D latent representations and 3D joint features. To explicitly align 3D and 2D features of local regions and enhance the flexibility of feature matching, we design a dynamic depth-aware interpolation to project 3D mesh features into 2D image space. Both quantitative and qualitative experimental results demonstrate the superiority of our method over the state-of-the-arts for realistic depth synthesis. Compared to training only on real depth images, our method enables the hand pose estimator to achieve significantly better performance with our synthetic data and less real data (10%).

Multivariate time-series (MTS) classification tasks play a key role in data-driven applications spanning healthcare, finance, and mobile communication. As MTS data are typically collected from multiple interdependent sensors, the resulting temporal patterns inherently reflect the characteristics of the underlying sensing systems. Despite this connection, conventional MTS classification models predominantly focus on raw time-series data while disregarding valuable sensor-specific prior knowledge, which fundamentally constrains their classification accuracy. The emergence of large language models (LLMs) has encoded extensive sensor-related knowledge within their parameter spaces. However, effectively harnessing such knowledge to enhance MTS classification networks remains an open challenge. To address this, we propose Foresail, a status-guided neural framework that bridges this gap through systematic integration of LLM-derived sensor knowledge via the status relationship matrix and fine-grained status labels. Foresail can be seamlessly integrated with existing MTS networks to optimize performance and generate interpretable intermediate results. Experiments on irregularly and regularly sampled MTS data demonstrate that Foresail outperforms state-of-the-art approaches, achieving a notable improvement in F1-score of up to 10.9% compared to the basic MTS network.

Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, Jing Wang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Multi-view 3D human pose estimation (HPE) leverages complementary information across views to improve accuracy and robustness. Traditional methods rely on camera calibration to establish geometric correspondences, which is sensitive to calibration accuracy and lacks flexibility in dynamic settings. Calibration-free approaches address these limitations by learning adaptive view interactions, typically leveraging expressive and flexible continuous representations. However, as the multiview interaction relationship is learned entirely from data without constraint, they are vulnerable to noisy input, which can propagate, amplify and accumulate errors across all views, severely corrupting the final estimated pose. To mitigate this, we propose a novel framework that integrates a noise-resilient discrete prior into the continuous representation-based model. Specifically, we introduce the \textit{UniCodebook}, a unified, compact, robust, and discrete representation complementary to continuous features, allowing the model to benefit from robustness to noise while preserving regression capability. Furthermore, we further propose an attribute-preserving and complementarity-enhancing Discrete-Continuous Spatial Attention (DCSA) mechanism to facilitate interaction between discrete priors and continuous pose features. Extensive experiments on three representative datasets demonstrate that our approach outperforms both calibration-required and calibration-free methods, achieving state-of-the-art performance.

Recent advances in hand-object interaction modeling have employed implicit representations, such as Signed Distance Functions (SDF) and Neural Radiance Fields (NeRF) to reconstruct hands and objects with arbitrary topology and photo-realistic detail. However, these methods often rely on dense 3D surface annotations, or are tailored to short clips constrained in motion trajectories and scene contexts, limiting their generalization to diverse environments and movement patterns. In this work, we present HOGS, an adaptively perceptive 3D Gaussian Splatting (3DGS) framework for generalizable hand-object modeling from unconstrained monocular RGB images. By integrating photometric cues from the visual modality with the physically grounded structure of 3D Gaussians, HOGS disentangles inherent geometry from transient lighting and motion-induced appearance changes. This endows hand-object assets with the ability to generalize to unseen environments and dynamic motion patterns. Experiments on two challenging datasets demonstrate that HOGS outperforms state-of-the-art methods in monocular hand-object reconstruction and photo-realistic rendering.

Large Vision-Language Models (LVLMs) pretrained on large-scale multimodal data have shown promising capabilities in Video Anomaly Detection (VAD). However, their ability to reason about abnormal events based on scene semantics remains underexplored. In this paper, we investigate LVLMs’ behavior in VAD from a visual-textual co-occurrence perspective, focusing on whether their decisions are driven by statistical shortcuts between visual instances and textual phrases. By analyzing visual-textual co-occurrence in pretraining data and conducting experiments under different data settings, we reveal a hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality or abnormality, leading to incorrect predictions when these high-frequency objects appear in semantically mismatched contexts. To address this issue, we propose VAD-DPO, a direct preference optimization method supervised with counter-example pairs. By constructing visually similar but semantically contrasting video clips, VAD-DPO encourages the model to align its predictions with the semantics of scene rather than relying on co-occurrence patterns. Extensive experiments on six benchmark datasets demonstrate the effectiveness of VAD-DPO in enhancing both anomaly detection and reasoning performance, particularly in scene-dependent scenarios.

Anomaly detection in distributed systems faces critical challenges from feature heterogeneity—where incomplete or divergent feature sets across nodes degrade detection relia-bility—and privacy risks under regulations like GDPR. While federated learning (FL) enables collaborative training without raw data sharing, existing solutions fail to address both challenges simultaneously: traditional FL suffers from performance drops under feature-missing scenarios, and differential privacy techniques introduce utility penalties. This paper proposes FedMI, a vertical federated learning framework that achieves provable privacy preservation and robust anomaly detection in feature-heterogeneous environments. FedMI's key innovations include a novel framework for vertical federated learning in anomaly detection for distributed systems that maintains high detection accuracy, mimicking real-world distributed system conditions, and a mutual information-guided training mechanism that quantifies and minimizes privacy leakage during federated updates. Evaluations on healthcare, financial, and industrial sensor datasets demonstrate FedMI's robustness: it achieves performance comparable to centralized methods in F1-score under data-island scenarios while ensuring compliance with privacy constraints. By unifying privacy quantification and robustness to feature heterogeneity, FedMI advances the development of dependable AI-driven monitoring for distributed systems.

Recent research indicates that adding residual connections in Graph Neural Networks (GNNs) would amplify susceptibility to anomalous nodes, consequently undermining the robustness of deep GNNs in practical settings. However, existing verification methods encounter challenges with the increasing number of parameters and computational overhead in deep GNNs. In this paper, we derive the general form of the residual connections and apply the dual backpropagation network to deep GNNs. Considering the heightened computational errors arising from the increased number of layers in deep GNNs, we propose a new method for calculating intermediate activation bounds of GNNs based on linear approximation. Experimental results show that new method can effectively enhance the verification accuracy. Notably, the maximum perturbation value of nodes correctly classified shows an average improvement of 119.5%. To showcase the the efficacy and scalability of our method, we verify robustness of deep GNNs on six different graph datasets, and our method can effectively verify the robustness of deep GNNs even with 32 layers of residual connections, i.e. verify over 87.29% of nodes in the Citeseer dataset. Furthermore, we analyse the influence of the graph structural properties on the robustness of the model.

Whiteboard collaboration in virtual reality (VR) is an important task in collaborative virtual environments. The current research mainly relies on the use of controllers or dedicated pens but additional devices will cause inconvenience to users. Bare-hand writing offers rich collaborative semantics through natural gestures but remains underexplored. This paper addresses challenges and solutions for bare-hand whiteboard collaboration. We analyze the input process and identify key challenges in determining pen-drop, writing, and pen-lift intentions while maintaining user control over their avatar. Our approach addresses two VR scenarios: one without and one with physical planes. The method for the first case is called Air-writing, which dynamically adjusts the distance between the avatar's torso and the virtual whiteboard during the processes of pen-drop and pen-lift to ensure a consistent writing experience in VR. The method for the second case is called Physical-writing, which allows users to write smoothly with passive haptic feedback and physical constraints provided by the real surface by remapping the whiteboard in VR with a plane in reality. A comprehensive user study is conducted to evaluate communication efficiency, input accuracy, collaboration efficiency, and user experience of the two methods. The experimental results indicate that bare-hand interaction improves communication efficiency by 8% over controllers and performs similarly to real-world whiteboard collaboration. The Physical-writing method also demonstrates higher accuracy and user satisfaction compared to the Air-writing method.

With the rapid development of next-generation networks, the highly heterogeneous and dynamic nature of networks poses significant challenges for automated network management. Autonomous Network (AN), as a new network paradigm, aims to provide customers with a zero-wait, zero-touch, and zero-fault experience. AN facilitates network management through intent-driven interactions and provides on-demand resource orchestration and service scheduling. However, accurately translating user intents into commands and allocating resources on demand for services remain significant challenges for AN. Therefore, this paper proposes IAN, an intent-based AN framework guided by the Large Language Model (LLM). In the intent translation phase, IAN introduces RAG to enhance command generation quality by retrieving from manuals. In the resource allocation phase, the method utilizes LLM to analyze service characteristics, thereby guiding the training and inference of the resource allocation model to effectively distribute resources uniformly across emerging services. Experimental results demonstrate that IAN improves performance by 52.66% in intent translation tasks and increases overall gain by 33.57% in resource allocation tasks compared to other models.

Within the Internet of Things (IoT) for traffic monitoring, the employment of autonomous aerial vehicles (AAVs) as relays for collecting and transmitting real-time data from traffic sensors to base stations has proven a promising approach. In UAV-assisted Data Time-Sensitive IoT (DTIoT), the Age of Information is a crucial metric assessing data freshness, measuring the elapsed time from traffic sensors to the base station. Optimizing flight trajectories of multiple UAVs to minimize AoI while adhering to energy constraints poses a significant challenge. Current research often employs deep reinforcement learning for UAV trajectory control. Nevertheless, managing multi-agent continuous trajectories in intricate DTIoT network environments faces obstacles due to sparse rewards, thus impeding the training of deep neural network-based control policies using traditional DRL techniques. In this paper, we propose a network-oriented hierarchical reinforcement learning (NO-HRL) to control the UAVs’ flight trajectory in DTIoT networks for minimizing the AoI. We devise a control policy leveraging a two-tier hierarchical DRL framework, with the upper tier determining the target and the lower tier executing it. We also introduce a decoupled sequential training approach to efficiently train the mutually dependent two-tier DRL network of NO-HRL. Experimental results demonstrate that our method excels in optimizing AoI for DTIoT compared to other baselines.

Weakly Supervised Entity Alignment (EA) is the task of identifying equivalent entities across diverse knowledge graphs (KGs) using only a limited number of seed alignments. Despite substantial advances in aggregation-based weakly supervised EA, the underlying mechanisms in this setting remain unexplored. In this article, we present a propagation perspective to analyze weakly supervised EA and explain the existing aggregation-based EA models. Our theoretical analysis reveals that these models essentially seek propagation operators for pairwise entity similarities. We further prove that, despite the structural heterogeneity across different KGs, the potentially aligned entities within aggregation-based EA models exhibit isomorphic subgraphs, a fundamental yet underexplored premise of EA. Leveraging this insight, we introduce a potential isomorphism propagation operator to enhance the propagation of neighborhood information across KGs. We develop a general EA framework, PipEA, incorporating this operator to improve the accuracy of every type of aggregation-based model without altering the learning process. Extensive experiments substantiate our theoretical findings and demonstrate PipEA’s significant performance gains over state-of-the-art weakly supervised EA methods. Our work advances the field and enhances our comprehension of aggregation-based weakly supervised EA.

Recently, reconstruction-based deep models have gained widespread usage in unsupervised anomaly detection. However, they may overlook some anomalies owing to the over-generalization of neural networks. Several studies have incorporated memory networks to mitigate this problem. Nonetheless, some of them lack an explicit memory updating process, while others rely on data-driven updating methods that are sensitive to initial values and unsuitable for end-to-end training. Additionally, the traditional criterion for detection computed in the high-dimensional input space may collapse as the spike in the deviation score is averaged across numerous dimensions. To address these challenges, we propose MCAKE, a M emory-augmented C ontrastive A utoencoder with K NN-Based E xtraction. It is designed to highlight the deviation score for anomalies by reconstructing input using fixed normal prototypes recorded in the memory. We explicitly encourage the memory to be autonomously learned and effectively allocated through contrastive learning with multiple positive and multiple negative samples. Furthermore, we introduce a bivariate detection criterion that calculates anomaly scores considering both input and latent space to tackle the collapse. Extensive experiments on 50 datasets across various categories demonstrate the superiority of our approach, with a 2% relative improvement over the previous state-of-the-art models.

Intent-Based Networking (IBN) represents an emerging network management concept that is designed to fulfill user service requirements through automation. At its core, IBN is capable of translating user intent into network policies, thereby enabling automated configuration and management. However, the application of IBN has been limited by challenges associated with automation and intelligence. The recent widespread adoption of Large Language Model (LLM) has partially mitigated these issues. Nonetheless, hardware heterogeneity and high dynamic networks remain significant challenges for IBN: (i) Devices from different vendors are challenging to manage uniformly; (ii) Aligning service demands with rapidly changing network status is difficult. To address these challenges, we propose LIT, a framework of LLM-empowered Intent Translation with manual guidance. LIT incorporates Retrieval-Augmented Generation (RAG) to reference hardware manuals and enhance the generation results of LLMs. To reduce noise from retrieval results, we optimized the general RAG process. Additionally, LIT introduces MoE (Mixture of Experts) to adjust parameter values according to network status by synthesizing results from multiple expert models. Experiments demonstrate that LIT alleviates the challenges faced by IBN, achieving a 57.5% improvement in F1 score compared to the baseline.

Federated learning (FL) has recently been applied to the parameter-efficient fine-tuning (PEFT) of large language models (LLMs). While promising, client resource heterogeneity has imposed the challenge of the "bucket effect" to FL, where model configuration must cater to the client with the fewest resources. To tackle this issue, heterogeneous low-rank adaptation (LoRA) has recently emerged in FL, which enables clients to do local fine-tuning with different LoRA ranks. However, existing works in this area typically adopt zero-padding, stacking, or singular value decomposition (SVD) for LoRA aggregation, which often incur precision loss or significant overhead, limiting their practicality. In this article, we propose ECLoRA, a novel method for federated fine-tuning with heterogeneous LoRA settings across clients. ECLoRA employs randomized SVD (RSVD) to dramatically reduce aggregation overhead while introducing an error compensation (EC) mechanism that incorporates the decomposition error from previous rounds to improve aggregation precision. Extensive experiments on four widely used foundation models across six public tasks demonstrate the effectiveness of ECLoRA. Specifically, ECLoRA is: (1) accurate, significantly improving the final model performance; (2) fast, accelerating convergence with an average speedup of $1.54\times $ to $3.01\times $ ; and (3) practical, reducing aggregation time by approximately $40\times $ compared to classical SVD.

There is an increasing interest in data plane verification, which is designed to automatically verify network correctness through directly analyzing the data plane. Recent data plane verifiers have been able to do real-time sub-millisecond per rule verification. However, we observe that in real-world networks, individual data plane updates rarely occur. On the contrary, there are always a certain number of updates generated in a short period of time, called asburst updates, due to high-level user intend or uncertain network events. When it comes to this real-life scenario, yet, the current equivalence class (EC) based methods are unable to solve themodel-wide changesproblem caused by the EC itself, which significantly slows down the verification speed of burst updates. To overcome this limitation, we present EPVerifier, a fast, scalable data plane verifier accelerating burst updates verification with edge-predicate (EP). Instead of classifying packets into ECs according to global forwarding behavior, the EPVerifier uses one EP per edge to represent all packets that can pass through. Furthermore, with EPs that clearly have localized properties, we introduce a rule type extension that does not require a change in the granularity of the network model to support ACLs and NATs that are prevalent in real devices, and obtain better-performing parallelism by dividing the verification task based on switches. Experiments on both dataset simulations and real-life deployments show that EPVerifier achieves 2-$10\times $faster data plane verification than the state-of-the-art and such advantage expand with the data plane’s complexity and update scale growth.

Massive logs contain crucial information about the working status of software systems, which contributes to anomaly detection and troubleshooting. For engineers, it is a laborious task to manually inspect raw logs to know the system running status, and therefore an automated log summarization tool can be helpful. However, due to the specificity of logs in terms of grammar, vocabulary and semantics, existing natural language-based methods cannot perform well in log analysis. To address these issues, we propose LogNotion, a general log summarization framework that highlights the log messages to assist human reading and decision making. We first explore the role played by triplets in log analysis, and propose a triplet extraction method based on sequence tagging and component alignment, in which the specificity of logs is fully taken into account. Then, we propose an unsupervised log summarization method to extract both regular and noteworthy information based on triplets. Comprehensive experiments are conducted on seven real-world log datasets and the results show that LogNotion improves the average ROUGE-1 by 0.26, recall by 0.12, and compression ratio by 2.13%, compared to state-of-the-art log summarization tools. The helpfulness, readability and generalizability are also verified through human evaluation and cross-dataset tests.

Logs record crucial information about runtime status of software system, which can be utilized for anomaly detection and fault diagnosis. However, techniques struggle to perform effectively when dealing with interleaved logs and entities that influence each other. Although manually specifying a grouping field for each dataset can handle the single grouping scenario, the problems of multiple and heterogeneous grouping still remain unsolved. To break through these limitations, we first design a log semantic association mining approach to convert log sequences into Log-Entity Graph, and then propose a novel log anomaly detection model named Lograph. The semantic association can be utilized to implicitly group the logs and sort out complex dependencies between entities, which have been overlooked in existing literature. Also, a Heterogeneous Graph Attention Network is utilized to effectively capture anomalous patterns of both logs and entities, where Log-Entity Graph serves as a data management and feature engineering module. We evaluate our model on real-world log datasets, comparing with nine baseline models. The experimental results demonstrate that Lograph can improve the accuracy of anomaly detection, especially on the datasets where entity relationships are intricate and grouping strategies are not applicable.

Previous 3D hand pose estimation methods primarily rely on a single modality, either RGB or depth, and the comprehensive utilization of the dual modalities has not been extensively explored. RGB and depth data provide complementary information and thus can be fused to enhance the robustness of 3D hand pose estimation. However, there exist two problems for applying existing fusion methods in 3D hand pose estimation: redundancy of dense feature fusion and ambiguity of visual features. First, pixel-wise feature interactions introduce high computational costs and ineffective calculations of invalid pixels. Second, visual features suffer from ambiguity due to color and texture similarities, as well as depth holes and noise caused by frequent hand movements, which interferes with modeling cross-modal correlations. In this paper, we propose Keypoint-Fusion for RGB-D based 3D hand pose estimation, which leverages the unique advantages of dual modalities to mutually eliminate the feature ambiguity, and performs cross-modal feature fusion in a more efficient way. Specifically, we focus cross-modal fusion on sparse yet informative spatial regions (i.e. keypoints). Meanwhile, by explicitly extracting relatively more reliable information as disambiguation evidence, depth modality provides 3D geometric information for RGB feature pixels, and RGB modality complements the precise edge information lost due to the depth noise. Keypoint-Fusion achieves state-of-the-art performance on two challenging hand datasets, significantly decreasing the error compared with previous single-modal methods.

Changes in enterprise networks require updated configurations. However, manual configurations with slow update efficiency, poor performance, and handling limitations, lead to the unavailability of updated networks. Therefore, we propose an efficient network renascence framework, NetRen, which synthesizes OSPF/BGP configurations driven by service and traffic migration. We follow the workflow of sketch extraction, configuration synthesis, and repair. Initially, comprehensive graphs are constructed to represent configuration sketches. We propose a GraphTrans synthesizer with Transformer's benefits of long-range focus and parallel reasoning. Training samples with the optimization relationship enable the synthesizer to achieve a mapping that optimizes performance based on configurations. To overcome the satisfiability barrier, configurations from the synthesizer are input to the stepwise configuration repairer as well-initialized solutions, achieving rapid configuration repair. Experiments demonstrate that the consistency of network configurations output by the GraphTrans synthesizer averages 98%. NetRen achieves a 312.4× increase in synthesis efficiency and a 5.83% improvement in network performance.

Code summarization provides a natural language description for a given piece of code. In this work, we focus on scripting code—programming languages that interact with specific devices through commands. The low-resource nature of scripting languages makes traditional code summarization methods challenging to apply. To address this, we introduce a novel framework: distantly supervised contrastive learning for low-resource scripting language summarization. This framework leverages limited atomic commands and category constraints to enhance code representations. Extensive experiments demonstrate our method’s superiority over competitive baselines.

Category-agnostic pose estimation (CAPE) aims to predict the pose of a query image based on few support images with pose annotations. Existing methods achieve the localization of arbitrary keypoints through similarity matching between support keypoint features and query image features. However, these methods primarily focus on mining information from the query images, neglecting the fact that support samples with keypoint annotations contain rich category-specific fine-grained semantic information and prior structural information. In this paper, we propose a Support-based Dynamic Perception Network (SDP-Net) for the robust and accurate CAPE. On the one hand, SDPNet models complex dependencies between support keypoints, constructing category-specific prior structure to guide the interaction of query keypoints. On the other hand, SDPNet extracts fine-grained semantic information from support samples, dynamically modulating the refinement process of query. Our method outperforms existing methods on MP-100 dataset by a large margin.

Recent progress in video anomaly detection suggests that the features of appearance and motion play crucial roles in distinguishing abnormal patterns from normal ones. However, we note that the effect of spatial scales of anomalies is ignored. The fact that many abnormal events occur in limited localized regions and severe background noise in-terferes with the learning of anomalous changes. Mean-while, most existing methods are limited by coarse-grained modeling approaches, which are inadequate for learning highly discriminative features to discriminate subtle differences between small-scale anomalies and normal patterns. To this end, this paper address multi-scale video anomaly detection by multi-grained spatiotemporal representation learning. We utilize video continuity to design three proxy tasks to perform feature learning at both coarse-grained and fine-grained levels, i.e., continuity judgment, discontinuity localization, and missing frame estimation. In particular, we formulate missing frame estimation as a contrastive learning task in feature space instead of a reconstruction task in RGB space to learn highly discriminative features. Experiments show that our proposed method outperforms state-of-the-art methods on four datasets, especially in scenes with small-scale anomalies.

Multi-Modal Entity Alignment (MMEA) is a pivotal task in Multi-Modal Knowledge Graphs (MMKGs), seeking to identify identical entities by leveraging associated modal attributes. However, real-world MMKGs confront the challenges of semantic inconsistency arising from diverse and incomplete data sources. This inconsistency is predominantly caused by the absence of specific modal attributes, manifesting in two distinct forms: disparities in attribute counts or the absence of certain modalities. Current methods address these issues through attribute interpolation, but their reliance on predefined distributions introduces modality noise, compromising original semantic information. Furthermore, the absence of a generalizable theoretical principle hampers progress towards achieving semantic consistency. In this work, we propose a generalizable theoretical principle by examining semantic consistency from the perspective of Dirichlet energy. Our research reveals that, in the presence of semantic inconsistency, models tend to overfit to modality noise, leading to over-smoothing and performance oscillations or declines, particularly in scenarios with a high rate of missing modality. To overcome these challenges, we propose DESAlign, a robust method addressing the over-smoothing caused by semantic inconsistency and interpolating missing semantics using existing modalities. Specifically, we devise a training strategy for multi-modal knowledge graph learning based on our proposed principle. Then, we introduce a propagation strategy that utilizes existing features to provide interpolation solutions for missing semantic features. DESAlign outperforms existing approaches across 60 benchmark splits, encompassing both monolingual and bilingual scenarios, achieving state-of-the-art performance. Experiments on splits with high missing modal attributes demonstrate its effectiveness, providing a robust MMEA solution to semantic inconsistency in real-world MMKGs.

Anomaly detection in multivariate time series (MTS) is crucial for various applications in data mining and industry. Current industrial methods typically approach anomaly detection as an unsupervised learning task, aiming to identify deviations by estimating the normal distribution in noisy, label-free datasets. These methods increasingly incorporate interdependencies between channels through graph structures to enhance accuracy. However, the role of interdependencies is more critical than previously understood, as shifts in interdependencies between MTS channels from normal to anomalous data are significant. This observation suggests that anomalies could be detected by changes in these interdependency graph series. To capitalize on this insight, we introduce MADGA (MTS Anomaly Detection via Graph Alignment), which redefines anomaly detection as a graph alignment (GA) problem that explicitly utilizes interdependencies for anomaly detection. MADGA dynamically transforms subsequences into graphs to capture the evolving interdependencies, and Graph alignment is performed between these graphs, optimizing an alignment plan that minimizes cost, effectively minimizing the distance for normal data and maximizing it for anomalous data. Uniquely, our GA approach involves explicit alignment of both nodes and edges, employing Wasserstein distance for nodes and Gromov-Wasserstein distance for edges. To our knowledge, this is the first application of GA to MTS anomaly detection that explicitly leverages interdependency for this purpose. Extensive experiments on diverse real-world datasets validate the effectiveness of MADGA, demonstrating its capability to detect anomalies and differentiate interdependencies, consistently achieving state-of-the-art across various scenarios.

Intent-Based Networking (IBN) represents a novel paradigm of network automation and intelligence that has gradually been applied to network management. While the emergence of Large Language Models (LLMs) has improved the current state of IBN, hardware heterogeneity and high network dynamics remain significant challenges. Hardware heterogeneity requires that IBN effectively manage a diverse range of devices. The high network dynamics demands that IBN align service needs with rapidly changing network resources. We propose LIT, a framework of LLM-empowered Intent Translation with manual guidance. Given the outstanding language understanding and generation capabilities of LLM, LIT utilizes it in intent translation task. To further address two prevalent problems encountered in IBN, we introduce manual guidance and Mixture of Experts (MoE). Under the guidance of the manual, LLM improves its ability to generate high-quality policies that comply with syntax. After introducing MoE, it makes fine-grained adjustments to the parameters of policies based on network status and service requirements. The experimental outcomes demonstrate that LIT considerably alleviates numerous current challenges confronted by IBN and excels in intent translation, attaining an F1 score that is$\mathbf{5 6. 7 \%}$higher than the baseline model.

Video anomaly detection plays a crucial role in the field of autonomous driving to ensure driving safety. Most existing video anomaly detection methods exhibit mediocre performance when analyzing video frames captured by dynamic cameras. To enhance their performance on dynamic cameras, we propose a video anomaly detection method based on Swin-Unet to separately predict optical flow and images cropped with ROI in the form of dual GAN. By integrating the predicted results of optical flow and ROI-cropped images, the model’s ability to learn dynamic information is significantly improved. The experimental results indicate that our proposed method outperforms state-of-the-art methods in terms of AUC on the Car Crash dataset and RetroTrucks dataset.

Nowadays, the rise of various network applications makes network traffic become increasingly complex, which brings more stringent requirements to traffic engineering (TE). Although the state-of-the-art TE approaches based on deep reinforcement learning (DRL) or traditional methods can generate optimal solutions for fixed traffic matrices, they cannot converge fast enough to provide real-time optimization in real networks either because of excessive computation times or high communication overheads. Moreover, due to the dynamically changing traffic load on the network, it is also challenging to achieve optimization of maximum link utilization (MLU) and end-to-end delay at the same time since these two optimization objectives may be conflicting, especially when the network is under a low traffic load, which makes the modeling very difficult. To meet these challenges, we present RT-TE, a TE system based on DRL and distributed message-passing between intelligent agents that can achieve real-time optimization for both MLU and end-to-end delay. To reduce the communication time due to link propagation delay during the optimization process, we design a proactive message-passing mechanism that allows agents to use partial messages to compute the routing policy while maintaining the optimization performance. Additionally, to achieve the tradeoff between the two optimization objectives, we model the propagation delay into the DRL model and design a multi-objective training framework with parameter transfer for training. Based on theoretical modeling, we can find the best tradeoff between the two objectives. Moreover, to improve the model's generalization for various traffic flows, we use a GNN model to generate the rewards of the DRL model, which greatly speeds up the training phase and allows us to feed massive amounts of data into the model. Through evaluations of real-world network topologies, our approach shows a 10%-20% improvement in optimizing MLU under short traffic-changing intervals and yields a 9%-13% improvement in optimizing end-to-end delay compared to state-of-the-art approaches.

In the imminent era of 6G, Quality of Service (QoS) emerges as a pivotal concern in wireless communications. The prescribed transmission rates and vast access demands mandated by 6G standards impose heightened requirements on network throughput and delay. However, the highly dynamic and often bursty nature of application demands presents challenges for routing and congestion control. The backpressure-based joint rate and routing control algorithm adaptively adjusts network traffic to achieve optimal throughput. However, varying traffic conditions hinder the algorithm’s convergence to ideal states. Additionally, relying solely on first-order backlog differences for forwarding can lead to poor convergence and high delays. In this study, we propose a Second-Order Smooth Backpressure (SoSBP) algorithm, leveraging second-order backlog metrics and dual-level queue mapping, to address throughput, delay, and jitter issues in dynamic network environments. We validate the efficacy of this novel backlog metric using Lyapunov optimization techniques. Simulation results demonstrate that our approach significantly reduces end-to-end delay and data jitter while preserving throughput and eliminating routing loops.

Learning multiple proxy tasks is a popular training strategy in semi-supervised video anomaly detection. However, the traditional method of learning multiple proxy tasks simultaneously is prone to suboptimal solutions, and simply executing multiple proxy tasks sequentially cannot ensure continuous performance improvement. In this paper, we thoroughly investigate the impact of task composition and training order on performance enhancement. We find that ensuring continuous performance improvement in multi-task learning requires different but continuous optimization objectives in different training phases. To this end, a training strategy based on progressive learning is proposed to enhance the multi-task learning in VAD. The learning objectives of the model in previous phases contribute to the training in subsequent phases. Specifically, we decompose video anomaly detection into three phases: perception, comprehension, and inference, continuously refining the learning objectives to enhance model performance. In the three phases, we perform the visual task, the semantic task and the open-set task in turn to train the model. The model learns different levels of features and focuses on different types of anomalies in different phases. Extensive experiments demonstrate the effectiveness of our method, highlighting that the benefits derived from the progressive learning transcend specific proxy tasks.

The real-time control capability of IoT devices is contingent upon the transmission of packets. However, due to the influence of multiple devices accessing the network, the bandwidth available to IoT devices from access points may decline significantly, which causes a surge in the queuing latency and interrupts the transmission. This phenomenon is referred to as transient congestion. To achieve stable high-quality network service, this paper designs RushWay, a QUIC-enabled framework for alleviating transient congestion. RushWay employs stream multiplexing to compress the original packet into the stream frame, thereby reducing the bandwidth required for transmission. Furthermore, RushWay employs an adaptive decision-making algorithm to assess uplink queue conditions and packet latency requirements, thereby alleviating transient congestion. The simulation results demonstrate that RushWay can improve key performance by 13% to 94%.

Huazheng Wang, Jinming Wu, Haifeng Sun, Zixuan Xia, Daixuan Cheng, Jingyu Wang, Qi Qi, Jianxin Liao. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Pre-trained foundation models, particularly large language models, have achieved remarkable success and led to massive fine-tuned variants. These models are commonly fine-tuned locally and then uploaded by users to cloud platforms such as HuggingFace for secure storage. However, the huge model number and their billion-level parameters impose heavy storage overhead for cloud with limited resources. Our empirical and theoretical analysis reveals that most fine-tuned models in cloud have a small difference (delta) from their pre-trained models. To this end, we propose a novel lossless compression scheme FM-Delta specifically for storing massive fine-tuned models in cloud. FM-Delta maps fine-tuned and pre-trained model parameters into integers with the same bits, and entropy codes their integer delta. In this way, cloud only needs to store one uncompressed pre-trained model and other compressed fine-tuned models. Extensive experiments have demonstrated that FM-Delta efficiently reduces cloud storage consumption for massive fine-tuned models by an average of around 50% with only negligible additional time in most end-to-end cases. For example, on up to 10 fine-tuned models in the GPT-NeoX-20B family, FM-Delta reduces the original storage requirement from 423GB to 205GB, significantly saving cloud storage costs.

Time series forecasting has played a pivotal role across various industries, including finance, transportation, energy, healthcare, and climate. Due to the abundant seasonal information they contain, timestamps possess the potential to offer robust global guidance for forecasting techniques. However, existing works primarily focus on local observations, with timestamps being treated merely as an optional supplement that remains underutilized. When data gathered from the real world is polluted, the absence of global information will damage the robust prediction capability of these algorithms. To address these problems, we propose a novel framework named GLAFF. Within this framework, the timestamps are modeled individually to capture the global dependencies. Working as a plugin, GLAFF adaptively adjusts the combined weights for global and local information, enabling seamless collaboration with any time series forecasting backbone. Extensive experiments conducted on nine real-world datasets demonstrate that GLAFF significantly enhances the average performance of widely used mainstream forecasting models by 12.5\%, surpassing the previous state-of-the-art method by 5.5\%.

3D hand reconstruction is an important technique for human-computer interaction. Interactive experience depends on the accuracy, efficiency, and robustness of the algorithm. Therefore, in this paper, we first propose a balanced framework called spatial-aware regression (SAR) to achieve precise and fast reconstruction. SAR can bridge convolutional networks and graph-structure networks more effectively than existing frameworks to fully exploit extracted spatial information using a novel spatial-aware initial graph building module. In addition, SAR uses adaptive-GCN to make keypoints interact efficiently and effectively; and regresses 2.5D belief maps to characterize uncertainty. SAR is highly flexible because it can predict an arbitrary number of keypoints and apply pose-guided refinement for coarse to fine regression. To produce more rational results for challenging cases and mitigate 3D label reliance, we also propose a more robust model-based framework called spatial-guided model-based regression (SMR) that is based on SAR. There are two critical designs of SMR: 1) it uses SAR to enhance the features with pose information to help the regression of hand model parameters; and 2) it regresses parameters in a spatially aware manner that is similar to SAR. Experiments demonstrate that the proposed frameworks surpass existing fully-supervised approaches on the FreiHAND, HO-3D, RHD, and STB datasets. Also, the performances of the proposed frameworks under weakly/self-supervised settings outperform other competitors. Meanwhile, the proposed frameworks are accurate and efficient.

Distributing the inference of convolutional neural network (CNN) to multiple mobile devices has been studied in recent years to achieve real-time inference without losing accuracy. However, how to map CNN to devices remains a challenge. On the one hand, scheduling the workload of state-of-the-art CNNs with multiple devices is NP-Hard because the structures of CNNs are directed acyclic graphs (DAG) rather than simple chains. On the other hand, distributing the inference workload suffers from expensive communication and unbalanced computation due to the wireless environment and heterogeneous devices. This paper presents PICO, a pipeline cooperation framework to accelerate the inference of versatile CNNs on diverse mobile devices. At its core, PICO features: (1) a generic graph partition algorithm that considers the characteristics of any given CNN and orchestrates it into a list of model pieces with suitable granularity, and (2) a many-to-many mapping algorithm that produces the best pipeline configuration for heterogeneous devices. In our experiment with$2 \sim 8$Raspberry-Pi devices, the throughput can be improved by$1.8 \sim 6.8 \times$under different CPU frequencies.

Edge network slicing promises better utilization of network resources by dynamically allocating resources on demand. However, addressing the imbalance between slice resources and user demands becomes challenging when complex user behaviors lead to bursty traffic within the edge network. Hence, we propose a comprehensive dynamic slice strategy with two coupled sub-strategies (i) bursty-sensitive slice resource coordination and (ii) proactive demand resource matching to find an optimal balance. For obtaining stable strategies, the edge network with bursty traffic is formulated as a bi-level Lyapunov optimization problem. Then we propose a resource allocation and request redirection (RA-RR) algorithm with polynomial complexity by introducing deep reinforcement learning to guarantee real-time. Specifically, two agents are trained to solve two sub-strategies, and the Lyapunov drift-plus-penalty function is used as the reward to keep queues stable. RA-RR is responsive to fluctuations in demand and realizes an efficient interaction of coupled decision-making. Moreover, a training method based on alternating optimization is designed to ensure convergence of the RA-RR algorithm. Experiments demonstrate that the proposal can maximize network revenue while ensuring the stability of slice services when edge traffic bursts, and has an average improvement of 20.4% compared with comparisons.

Network operators often need to modify Access Control List (ACL) policies to align with to network upgrades. An essential part of the ACL update task is reachability satisfaction. Previous studies formalize reachability requirements as a set of constraints and then use Boolean Satisfiability (SAT) or Satisfiability Modulo Theories (SMT) solvers to search for solutions. However, as today’s networks grow in size and complexity, the constraints derived from the requirements become increasingly complex, leading to an unacceptable time cost to obtain a correct policy. The sluggish updating of ACL policies can affect the properties of a network, such as connectivity and security. This paper presents a novel approach for fast and scalable ACL policy synthesis under complex constraints. We utilize Graph Neural Networks (GNNs) to learn the relations between nodes and reason the solution that satisfies the update requirements. We further integrate global position encoding into the GNN architecture, which allows for better differentiation of nodes in ACL update tasks. Additionally, an enhanced stochastic local search solver is introduced to address incorrect predictions made by the GNN. Experiments on real-world topologies show that GNN saves up$278\times $time costs compared to advanced SAT/SMT solvers on a 125-node network, and this advantage expands with the network size. Furthermore, our model extrapolates well when faced with different requirements and topologies, demonstrating its ability to handle frequent network upgrades.

The aim of surveillance services is to detect anomalous events that occur in given surveillance videos. Most existing video anomaly detection methods rely on minimizing reconstruction or prediction errors due to the lack of abnormal data, which results in poor generalization and overfitting. In fact, cognitions for anomalies in surveillance videos mainly relies on crucial relationships, including ones between objects and ones between objects and scenes. Focusing on this property of anomaly detection, aCognitionGuidedVideoAnomalyDetection framework based on prior knowledge is proposed, calledCG-VAD. CG-VAD introduces both explicit and implicit prior knowledge into the frame prediction network to let the model exploit crucial relationships. Explicit knowledge containing crucial relationships related to anomaly is introduced into the anomaly detection model through a proposed embedding network based on multi-layer Graph Convolutional Networks. Implicit knowledge in the form of learnable parameters enhances the ability of the model to learn crucial relationships through prompt tuning. By integrating prior knowledge to focus the model on the relationships associated with the anomaly, we find that CG-VAD is not only quick to adapt to new real-world scenarios, but it is also able to recognize the type of anomaly. We have conducted extensive experiments on four benchmark datasets and the results indicate that the proposed method outperforms previous methods. Specifically, CG-VAD achieves an AUROC score of 87.2$\%$on the ShanghaiTech dataset. Code is available athttps://github.com/zmh0124/CG-VAD.

The sketch-based image retrieval (SBIR) task has long been researched at the instance level, where both query sketches and candidate images are assumed to contain only one dominant object. This strong assumption constrains its application, especially with the increasingly popular intelligent terminals and human-computer interaction technology. In this work, a more general scene-level SBIR task is explored, where sketches and images can both contain multiple object instances. The new general task is extremely challenging due to several factors: (i) scene-level SBIR inherently shares sketch-specific difficulties with instance-level SBIR (e.g., sparsity, abstractness, and diversity), (ii) the cross-modal similarity is measured between two partially aligned domains (i.e., not all objects in images are drawn in scene sketches), and (iii) besides instance-level visual similarity, a more complex multi-dimensional scene-level feature matching problem is imposed (including appearance, semantics, layout, etc.). Addressing these challenges, a novel Conditional Graph Autoencoder model is proposed to deal with scene-level sketch-images retrieval. More importantly, the model can be trained with only pairwise supervision, which distinguishes our study from others in that elaborate instance-level annotations (for example, bounding boxes) are no longer required. Extensive experiments confirm the ability of our model to robustly retrieve multiple related objects at the scene level and exhibit superior performance beyond strong competitors.

Sketch-based image retrieval (SBIR) is an attractive research area where freehand sketches are used as queries to retrieve relevant images. Existing solutions have advanced the task to the challenging zero-shot setting (ZS-SBIR), where the trained models are tested on new classes without seen data. However, they are prone to overfitting under a realistic scenario when the test data includes both seen and unseen classes. In this paper, we study generalized ZS-SBIR (GZS-SBIR) and propose a novel semi-transductive learning paradigm. Transductive learning is performed on the image modality to explore the potential data distribution within unseen classes, and zero-shot learning is performed on the sketch modality sharing the learned knowledge through a semi-heterogeneous architecture. A hybrid metric learning strategy is proposed to establish semantics-aware ranking property and calibrate the joint embedding space. Extensive experiments are conducted on two large-scale benchmarks and four evaluation metrics. The results show that our method is superior over the state-of-the-art competitors in the challenging GZS-SBIR task.

Depth images and point clouds are the two most commonly used data representations for depth-based 3D hand pose estimation. Benefiting from the structuring of image data and the inherent inductive biases of the 2D Convolutional Neural Network (CNN), image-based methods are highly efficient and effective. However, treating the depth data as a 2D image inevitably ignores the 3D nature of depth data. Point cloud-based methods can better mine the 3D geometric structure of depth data. However, these methods suffer from the disorder and non-structure of point cloud data, which is computationally inefficient. In this paper, we propose an Image-Point cloud Network (IPNet) for accurate and robust 3D hand pose estimation. IPNet utilizes 2D CNN to extract visual representations in 2D image space and performs iterative correction in 3D point cloud space to exploit the 3D geometry information of depth data. In particular, we propose a sparse anchor-based "aggregation-interaction-propagation'' paradigm to enhance point cloud features and refine the hand pose, which reduces irregular data access. Furthermore, we introduce a 3D hand model to the iterative correction process, which significantly improves the robustness of IPNet to occlusion and depth holes. Experiments show that IPNet outperforms state-of-the-art methods on three challenging hand datasets.

3D hand reconstruction from RGB image has attracted a lot of attention due to its crucial role in human-computer interaction. Nevertheless, it is still challenging to perform 3D hand reconstruction under conditions of hand-object interaction due to severe mutual occlusion. Previous methods usually adopt fixed convolution kernel to extract features. We argue that simply sharing the static filter for all regions is impertinent, given that the occlusion degree varies across different regions, resulting in inconsistent visual representations. To address this issue, we proposed Region-aware Dynamic Filtering Network (RDFNet), which dynamically generates convolution kernels based on the features of different regions, thereby adaptively extracting region-related information. Furthermore, we introduce a dynamic receptive field selection mechanism to determine the most appropriate scale for the convolution kernel. For the severely occluded regions, larger receptive field is needed to capture semantic-related features, while the visible regions are mainly concerned with their own local pattern to accumulate spatial-related features and avoid the interference of irrelevant information. Our proposed RDFNet outperforms state-of-the-art methods by a large margin on several challenging hand-object interaction datasets.

Novel object captioning task aims at describing objects that are absent from training data. Due to the scarcity of novel objects, it’s challenging to find a way to utilize external data to improve model’s reasoning ability. While previously designed methods all follow a deep learning approach, we boost novel object captioning by incorporating reasoning with traditional deep learning framework. We design a manual from dictionaries that provides our model with sufficient and accurate external information on novel objects. We propose Manual-guided Context-aware Novel Object Captioning model (MC-NOC) that utilizes image and caption context to generate novel object captions. It contains a Manual-Guided Novel Object Reasoning module to reason about novel objects based on other objects of the given image and a Caption Reconstruction module to incorporate novel objects into generated captions according to caption context. We validate MC-NOC with state-of-the-art performance on the challenging Held-out COCO and Nocaps dataset, leading their leaderboard. In particular, we improved the CIDER metric by 6.4 points on the held-out coco dataset. Comprehensive experiments demonstrate our model’s reasoning capability and the quality of generated captions.

Transformer-based pretrained language models (PLMs) have achieved great success in modern NLP. An important advantage of PLMs is good out-of-distribution (OOD) robustness. Recently, diffusion models have attracted a lot of work to apply diffusion to PLMs. It remains under-explored how diffusion influences PLMs on OOD data. The core of diffusion models is a forward diffusion process which gradually applies Gaussian noise to inputs, and a reverse denoising process which removes noise. The noised input reconstruction is a fundamental ability of diffusion models. We directly analyze OOD robustness by measuring the reconstruction loss, including testing the abilities to reconstruct OOD data, and to detect OOD samples. Experiments are conducted by analyzing different training parameters and data statistical features on eight datasets. It shows that finetuning PLMs with diffusion degrades the reconstruction ability on OOD data. The comparison also shows that diffusion models can effectively detect OOD samples, achieving state-of-the-art performance in most of the datasets with an absolute accuracy improvement up to 18%. These results indicate that diffusion reduces OOD robustness of PLMs.

Existing weakly-supervised temporal sentence grounding methods typically regard query reconstruction as the pretext task in place of the absent temporal supervision. However, their approaches suffer from two flaws, i.e. insignificant reconstruction and discrepancy in alignment. Insignificant reconstruction indicates the randomly masked words may not be discriminative enough to distinguish the target event from unrelated events in the video. Discrepancy in alignment indicates the incorrect partial alignment built by query reconstruction task. The flaws undermine the reliability of current reconstruction-based methods. To this end, we propose a novel Self-improving Query ReconstrucTion (SQRT) framework for weakly-supervised temporal sentence grounding. To deal with insignificant reconstruction, we devise a key words mining strategy to determine the important words for language grounding. To attain better moment-query alignment, we introduce inter-sample contrast to tackle the partial alignment built by query reconstruction. The self-improving framework utilizes query reconstruction for language grounding and alleviates the discrepancy in alignment, thus turning on the right track. Experiments on two popular datasets show that SQRT achieves state-of-the-art performance on Charades-STA and comparable performance to the state-of-the-art on ActivityNet Captions.

RGB and depth modalities provide complementary information, which can be effectively utilized to improve the performance of hand detection in the wild. Most existing fusion-based methods model the channel-wise or spatial-wise cross-modal correlation to exploit the complementary RGB-D information, in which the modeling operations are shared across all input samples. However, the input images show various modes due to the high diversity of scenes in the wild. This inter-sample variance cannot be effectively perceived by static modeling operations shared across all samples. To address this problem, we propose a Sample-Adapt Fusion Network (SAFNet) with Channel Dynamic Refinement Module (CDRM) and Spatial Dynamic Aggregation Module (SDAM) to adaptively model the channel-wise and spatial-wise cross-modal correlation. Specifically, we propose a Multi-kernel Attention Module (MAM) to individually generate attention maps for each input sample by applying learnable weighting operations to multiple convolutional kernels. Our method outperforms state-of-the-art methods on CUG Hand dataset.

Fake news causes cognitive misperception among the audience and spreads panic to the public. It is crucial to detect fake news and prevent its spread early. Previous methods focus on excavating distinguishable features from news contents in a single domain with deep models, which are difficult to generalize to other domains. To solve this problem, News Optimal Transport (NOT) is proposed to learn transferable features across domains by aligning the source and target news using Optimal Transport (OT) techniques. To mitigate issues of heavy computation cost and negative transfer brought by OT, we further propose a mini-batching scheme and a dynamical weighted self-labeling mechanism respectively for model training. Encouraging empirical results on two public benchmarks Politifact and Gossipcop demonstrate that our method outperforms the state-of-the-art methods. The codes will be in public at https://github.com/OceanTangWei/NOTsoon.

Video anomaly detection aims to automatically detect abnormal objects or behaviors. Most existing methods tackle the problem by minimizing the reconstruction errors stemming from the lack of anomalous data, which leads to poor interpretability and robustness. Focus on the context-dependent nature of anomaly detection, a robust unsupervised Video Anomaly Detection framework based on Knowledge and Frame Prediction is proposed, called VAD-KFP. Prior knowledge which contains the context of anomaly is introduced into the multi-path frame prediction network through multi-layer Graph Convolutional Networks. By integrating the prior knowledge to accurately define anomalies, VAD-KFP is robust to different scenarios and is able to recognize the type of anomaly. An extensive range of experiments have been conducted on three benchmarks, the results of which indicate that our method outperforms strong baselines. Specifically, VAD-KFP obtains an AUROC score of 91.6% for the Avenue dataset.

Reconstructing interacting hands from a single RGB image is a very challenging task. On the one hand, severe mutual occlusion and similar local appearance between two hands confuse the extraction of visual features, resulting in the misalignment of estimated hand meshes and the image. On the other hand, there are complex spatial relationship between interacting hands, which significantly increases the solution space of hand poses and increases the difficulty of network learning. In this paper, we propose a decoupled iterative refinement framework to achieve pixel-alignment hand reconstruction while efficiently modeling the spatial relationship between hands. Specifically, we define two feature spaces with different characteristics, namely 2D visual feature space and 3D joint feature space. First, we obtain joint-wise features from the visual feature map and utilize a graph convolution network and a transformer to perform intra- and inter-hand information interaction in the 3D joint feature space, respectively. Then, we project the joint features with global information back into the 2D visual feature space in an obfuscation-free manner and utilize the 2D convolution for pixel-wise enhancement. By performing multiple alternate enhancements in the two feature spaces, our method can achieve an accurate and robust reconstruction of interacting hands. Our method outperforms all existing two-hand reconstruction methods by a large margin on the InterHand2.6M dataset.

The command line interface (CLI) is widely used to configure and manage network devices. However, as heterogeneous devices are introduced into the network, the CLI-based method is becoming time-consuming and inefficient because much effort is required to learn proprietary configuration languages of different vendors or consult online documents. In this work, we present CONFPILOT, an assistant system that can accelerate configuration by automatically converting natural language intents into commands. Our solution is based on a retrieval-augmented generation framework that features a unified parser that parses device manuals into a searchable configuration library, a vendor-agnostic retriever that finds the most relevant$k$syntaxes through a two-stage coarse-to-fine process, and a reliable generator that predicts syntactically correct commands via a pointer-generator network and syntax-guided decoding. In a nutshell, CONFPILOT frees engineers from most time-consuming efforts by learning directly from device manuals to generate configuration commands. Our evaluation and user study show, CONFPILOT can speed up the configuration process by 60x compared to manual lookup while maintaining acceptable exact match accuracy. Furthermore, CONFPILOT can quickly adapt to new vendors and devices with little human effort and time cost.

Access Control List (ACL) policies often need to be updated due to upgrades in network architecture and services. A critical part of ACL update tasks is reachability satisfaction, which is typically handled using Boolean Satisfiability (SAT) or Satisfiability Modulo Theories (SMT) solvers. However, as modern networks grow in size and complexity, the constraints derived from reachability requirements become increasingly complex, resulting in a considerable time cost to obtain a satisfying policy. The slow update of ACL policies can endanger network connectivity and security. This paper presents a new approach for fast and scalable ACL policy synthesis under complex constraints. We leverage Graph Neural Networks (GNNs) to learn the relations between nodes and reason the solution that satisfies the update requirements. In addition, an enhanced stochastic local search solver is introduced to deal with erroneous predictions of the GNN. Evaluations show that the proposed method guarantees 100% accuracy on real-world topologies. GNN outperforms modern SAT/SMT solvers in speed, saving up to 278x time costs on a 125-node topology. Furthermore, our method extrapolates well when faced with different requirements and topologies.

Recent graph-based methods achieve significant success in multivariate time series modeling and forecasting due to their ability to handle relationships among time series variables. However, only pairwise relationships are considered in most existing works. They ignore beyond-pairwise relationships and their potential categories in practical scenarios, which leads to incomprehensive relationship learning for multivariate time series forecasting. In this paper, we present ReMo, a Relational Modeling-based method, to promote fine-grained relational learning among multivariate time series data. Firstly, by treating time series variables and complex relationships as nodes and hyperedges, we extract multi-view hypergraphs from data to capture beyond-pairwise relationships. Secondly, a novel hypergraph message passing strategy is designed to characterize both nodes and hyperedges by inferring the potential categories of relationships and further distinguishing their impacts on time series variables. By integrating these two modules into the time series forecasting framework, ReMo effectively improves the performance of multivariate time series forecasting. The experimental results on seven commonly used datasets from different domains demonstrate the superiority of our model.

Many unsupervised methods have recently been proposed for multivariate time series anomaly detection. However, existing works mainly focus on stable data yet often omit the drift generated from non-stationary environments, which may lead to numerous false alarms. We propose **D**ynamic **D**ecomposition with **D**iffusion **R**econstruction (D$^3$R), a novel anomaly detection network for real-world unstable data to fill the gap. D$^3$R tackles the drift via decomposition and reconstruction. In the decomposition procedure, we utilize data-time mix-attention to dynamically decompose long-period multivariate time series, overcoming the limitation of the local sliding window. The information bottleneck is critical yet difficult to determine in the reconstruction procedure. To avoid retraining once the bottleneck changes, we control it externally by noise diffusion and directly reconstruct the polluted data. The whole model can be trained end-to-end. Extensive experiments on various real-world datasets demonstrate that D$^3$R significantly outperforms existing methods, with a 11% average relative improvement over the previous SOTA models.

Large Language Models (LLMs) has fostered the creation of innovative requirements. Locally deployed LLMs for micro-enterprise mitigates potential issues such as privacy infringements and sluggish response. However, they are hampered by the limitations in computing capability and memory space of possessed devices. We introduce PipeLLM, which allocates the model across devices commensurate with their computing capabilities. It enables the parallel execution of layers with slicing input sequence along the token dimension. PipeLLM demonstrates the potential to accelerate LLM inference with heterogeneity devices, offering a solution for LLM deployment in micro-enterprise hardware environment.

In this paper, we study the unsupervised plain graph alignment problem, which aims to find node correspondences across two graphs without any side information. The majority of previous works addressed UPGA based on structural information, which will inevitably lead to subgraph isomorphism issues. That is, unaligned nodes could take similar local structural information. To mitigate this issue, we present the Multi-order Matched Neighborhood Consistent (MMNC) which tries to match nodes by aligning the learned node embeddings with only a small number of pseudo alignment seeds. In particular, we extend matched neighborhood consistency (MNC) to vector space and further develop embedding-based MNC (EMNC). By minimizing the EMNC-based loss function, we can utilize the limited pseudo alignment seeds to approximate the orthogonal transformation matrix between two groups of node embeddings with high efficiency and accuracy. Through extensive experiments on public benchmarks, we show that the proposed methods achieve a good balance between alignment accuracy and speed over multiple datasets compared with existing methods.

The current success of entity alignment (EA) is still mainly based on large-scale labeled anchor links. However, the refined annotation of anchor links still consumes a lot of manpower and material resources. As a result, an increasing number of works based on active learning, few-shot learning, or other deep network learning techniques have been developed to address the performance bottleneck caused by a lack of labeled data. These works focus either on the strategy of choosing more informative labeled data or on the strategy of model training, while it remains opaque why existing popular EA models (e.g., GNN-based models) fail the EA task with limited labeled data. To overcome this issue, this paper analyzes the problem of weakly supervised EA from the perspective of model design and proposes a novel weakly supervised learning framework, Position Enhanced Entity Alignment (PEEA). Besides absorbing structural and relational information, PEEA aims to increase the connections between far-away entities and labeled ones by incorporating positional information into the representation learning with a Position Attention Layer (PAL). To fully utilize the limited anchor links, we further introduce a novel position encoding method that considers both anchor links and relational information from a global view. The proposed position encoding will be fed into PEEA as additional entity features. Extensive experiments on public datasets demonstrate the effectiveness of PEEA.

Web-based AR technology has broadened human-computer interaction scenes from traditional mechanical devices and flat screens to the real world, resulting in unconstrained environmental challenges such as complex backgrounds, extreme illumination, depth range differences, and hand-object interaction. The previous hand detection and 3D hand pose estimation methods are usually based on single modality such as RGB or depth data, which are not available in some scenarios in unconstrained environments due to the differences between the two modalities. To address this problem, we propose a multimodal fusion approach, named Scene-Adapt Fusion (SA-Fusion), which can fully utilize the complementarity of RGB and depth modalities in web-based HCI tasks. SA-Fusion can be applied in existing hand detection and 3D hand pose estimation frameworks to boost their performance, and can be further integrated into the prototyping AR system to construct a web-based interactive AR application for unconstrained environments. To evaluate the proposed multimodal fusion method, we conduct two user studies on CUG Hand and DexYCB dataset, to demonstrate its effectiveness in terms of accurately detecting hand and estimating 3D hand pose in unconstrained environments and hand-object interaction.

Artistic portrait drawing (APDrawing) generation has seen progress in recent years. However, due to the naturally high scarcity and artistry, it is difficult to collect large‐scale labeled and paired data and generally divide drawing styles into several specific recognized categories. Existing works suffer from the limited labeled data and naive manual division of drawing styles according to the corresponding artists. They cannot adapt to the actual situations, for example, a single artist might have multiple drawing styles and APDrawings from different artists might share similar styles. In this paper, we propose to use unlabeled and unpaired data and perform the task in an unsupervised manner. Without manual division of drawing styles, we take each portrait drawing as a unique style and introduce self‐supervised feature learning to learn free styles for unlabeled portrait drawings. Besides, we devise a style bank and a decoupled cycle structure to take over two main considerations in the task: generation quality and style control. Extensive experiments show that our model is more adaptable to different style inputs than state‐of‐the‐art methods.

Estimating 3-D hand pose estimation from a single depth image is important for human-computer interaction. Although depth-based 3-D hand pose estimation has made great progress in recent years, it is still difficult to deal with some complex scenes, especially the issues of serious self-occlusion and high self-similarity of fingers. Inspired by the fact that multipart context is critical to alleviate ambiguity, and constraint relations contained in the hand structure are important for the robust estimation, we attempt to explicitly model the correlations between different hand parts. In this article, we propose a pose-guided hierarchical graph convolution (PHG) module, which is embedded into the pixelwise regression framework to enhance the convolutional feature maps by exploring the complex dependencies between different hand parts. Specifically, the PHG module first extracts hierarchical fine-grained node features under the guidance of hand pose and then uses graph convolution to perform hierarchical message passing between nodes according to the hand structure. Finally, the enhanced node features are used to generate dynamic convolution kernels to generate hierarchical structure-aware feature maps. Our method achieves state-of-the-art performance or comparable performance with the state-of-the-art methods on five 3-D hand pose datasets: 1) HANDS 2019; 2) HANDS 2017; 3) NYU; 4) ICVL; and 5) MSRA.

Fraud detection in telecom network is a crucial problem that threatens users’ privacy and property security. In recent years, fraudsters adopt more advanced camouflage strategies to avoid being detected by traditional algorithms. To deal with these new types of fraud, it is necessary to analyze the integrated spatial-temporal features, which are rarely involved in existing literature. In this article, we propose a novel fraud detection model based on the intertwined spatial-temporal patterns of user behaviors. Specifically, we first introduce the extension of statistical and interactive features to dynamic call patterns, and build a probabilistic model to simulate users’ call behaviors. Then the sequential patterns reflecting users’ own behaviors are obtained by the mixture Hidden Markov Models, and the structural patterns reflecting the collaboration between users in the telecom network are obtained by the attention-based Graph-SAGE model. Finally, our model outputs a fraud score for each user to detect potential fraudsters. We conduct extensive experiments on a real-world telecom dataset. The experimental results demonstrate that our intertwined spatial-temporal call patterns can effectively represent user behavior and improve the accuracy of fraud detection compared with state-of-the-art methods. The results also validate the efficiency and the interpretability of our model.

Graph alignment, also known as network alignment, has many applications in data mining tasks. It aims to find the node correspondence across disjoint graphs. With recent representation learning advancements, embedding-based graph alignment has become a hot topic. Existing embedding-based methods focus either on structural proximity across graphs or on the positional proximity within a single graph. However, only considering the structural similarity will make the position relation of nodes not clear enough, which makes it easy to misalign the nodes close in distance, while only considering the position proximity of a single graph will make the node embeddings from different graphs in different subspaces. To mitigate this issue, we propose a novel model CEGA forCross-graphEmbedding-basedGraphAlignment, which can generate node embeddings to reflect structural proximity and positional proximity simultaneously. Meanwhile, we make the proximity trainable thus it can be learned to best suit the alignment task at hand automatically. We show that CEGA outperforms existing graph alignment methods in accuracy under unsupervised scenarios through extensive experiments on public benchmarks.

Multi-access Edge Computing and ubiquitous smart devices help serve end-users efficiently by providing emerging edge-deployed services. On the other hand, more heavy and time-varying traffic loads are generated in mobile edge networks, so that an efficient traffic forwarding mechanism is highly required to handle the routing problem in complex and highly dynamic edge environments. Thus, Deep Reinforcement Learning (DRL) is introduced since it can work in a model-free approach. However, previous centralized DRL-based methods work in a turn-based way that mismatches the real-time property of routing. In this paper, we propose a real-time and distributed learning approach, RTHop, to adapt to the volatile environment and realize a hop-by-hop routing. The Multi-Agent Deep Reinforcement Learning (MADRL) and the Real-Time Markov Decision Process (RTMDP) are used to alleviate network congestion and maximize the utilization of network resources. By joining with the self-attention mechanism, RTHop obtains the semantics from elements of the network state to help agents learn the importance of each element on routing. Experiment results show that RTHop not only overcomes the weakness of conventional turn-based DRL methods but also achieves the increase of delivered packet ratios and effective throughput compared with other routing methods.

Text-to-video temporal grounding aims to locate a target video moment that semantically corresponds to the given sentence query in an untrimmed video. In this task, fully supervised works require text descriptions for each event along with its temporal segment coordinate for training, which is labor-consuming. Existing weakly supervised works require only video-sentence pairs but cannot achieve satisfactory performance. However, many available annotations in the form of coarse temporal boundaries for sentences are ignored and unexploited. These coarse boundaries are common in streaming media platform and can be collected in a mechanical manner. We propose a novel approach to perform fine-grained text-to-video temporal grounding from these coarse boundaries. We take dense video captioning as base task and leverage the trained captioning model to identify the relevance of each video frame to the sentence query according to the frame participation in event captioning. To quantify the frame participation in event captioning, we proposeevent activation sequence, a simple method that highlights the temporal regions which have high correlations to the text modality in videos. Experiments on modified ActivityNet Captions and a use case demonstrate the promising fine-grained performance of our approach.

Network slicing is a key technology in 6G communication systems to support numerous vertical applications for all scenes while providing resources on demand. Due to more time-varying and dynamic traffic flows, it is difficult for traditional methods to manage complex and highly dynamic 6G networks. Therefore, intelligent method such as Deep Reinforcement Learning (DRL) is employed into network management since DRL is a model-free and experience-driven approach. However, it is difficult to leverage one DRL model to provide customized intra-slice orchestration for various applications because of their diverse flow characteristics and service requirements. Moreover, training a DRL model is notoriously time-consuming so that it is not realistic for network operator to individually orchestrate customized network slice for each application with the DRL algorithm. Additionally, the data privacy of each application should be considered into the training process. In this paper, we propose a Federated Meta Reinforcement Learning (FedMRL) approach to tackle the cold-start problem in network slice orchestration, while reserving the data privacy. The Meta Reinforcement Learning (MRL) is leveraged to train a meta policy for rapidly learn a local policy for a specific slice orchestration task by finding a common initialization that allows for a quick adaptation towards each optimal solution. With the help of federated learning setting, the training process of meta policy is not required to collect raw data of applications to the centralized server. Experimental results show that FedMRL outperforms three baselines in terms of overall costs, end-to-end latency and convergence speed.

The primary concern of numerous online social media network (SMN) platforms is how to provide users with effective and personalized web services. To achieve this goal, SMN platforms typically begin by collecting user preferences based on user behaviors (e.g., browsing history, posts) or user profiles. However, the effective information about a specific user on a single SMN platform is limited and monotonous, preventing a comprehensive reflection of the user's preferences. Therefore, recognizing anonymous but identical users across two SMNs to integrate their information is crucial for enhancing web services. Clearly, cross-platform research has the potential to aid in the resolution of numerous problems in service computing theory and applications. Therefore, in this article, we present theCross-PlatformUserMatcher (CPUM) framework, which attempts to map users into a union vector space and then performs user matching based on distance metrics. In particular, we introduce a GNN-based encoderAdaptiveGraphAttention Network (AdaGAT) for modeling user attributes and topology jointly in the social networks to capture two typical alignment principles: topology consistency and attribute consistency. Moreover, we derive AdaGAT from the heuristic of the spectral network alignment technique FINAL, which theoretically guarantees AdaGAT's efficacy. To the best of our knowledge, AdaGAT is the first representation-based alignment model to integrate these two alignment principles synergistically. In addition, two position encoding schemes are introduced to prevent alignment confusion that commonly arises with GNN-based alignment models. Extensive experiments on real-world datasets validate the superiority of the proposed framework.

Aspect-based sentiment analysis (ABSA) aims to distinguish sentiment polarity of every specific aspect in a given sentence. Previous researches have realized the importance of interactive learning with context and aspects. However, these methods are ill-studied to learn complex sentence with multiple aspects due to overlapped polarity feature. And they do not consider the correlation between aspects to distinguish overlapped feature. In order to solve this problem, we propose a new method called Recurrent Inverse Learning Guided Network (RILGNet). Our RILGNet has two points to improve the modeling of aspect correlation and the selecting of aspect feature. First, we use Recurrent Mechanism to improve the joint representation of aspects, which enhances the aspect correlation modeling iteratively. Second, we propose Inverse Learning Guidance to improve the selection of aspect feature by considering aspect correlation, which provides more useful information to determine polarity. Experimental results on SemEval 2014 Datasets demonstrate the effectiveness of RILGNet, and we further prove that RILGNet is state-of-the-art method in multiaspect scenarios.

In this work, we study the cross-view information fusion problem in the task of self-supervised 3D hand pose estimation from the depth image. Previous methods usually adopt a hand-crafted rule to generate pseudo labels from multi-view estimations in order to supervise the network training in each view. However, these methods ignore the rich semantic information in each view and ignore the complex dependencies between different regions of different views. To solve these problems, we propose a cross-view fusion network to fully exploit and adaptively aggregate multi-view information. We encode diverse semantic information in each view into multiple compact nodes. Then, we introduce the graph convolution to model the complex dependencies between nodes and perform cross-view information interaction. Based on the cross-view fusion network, we propose a strong self-supervised framework for 3D hand pose and hand mesh estimation. Furthermore, we propose a pseudo multi-view training strategy to extend our framework to a more general scenario in which only single-view training data is used. Results on NYU dataset demonstrate that our method outperforms the previous self-supervised methods by 17.5% and 30.3% in multi-view and single-view scenarios. Meanwhile, our framework achieves comparable re-sults to several strongly supervised methods.

Multimedia retrieval models need the ability to extract useful information from large-scale data for clients. As an important part of multimedia retrieval, image classification model directly affects the efficiency and effect of multimedia retrieval. We need a lot of data to train a image classification model applied to multimedia retrieval task. However, with the protection of data privacy, the data used to train the model often needs to be kept on the client side. Federated learning is proposed to use data from all clients to train one model while protecting privacy. When federated learning is applied, the distribution of data across different clients varies greatly. Disregarding this problem yields a final model with unstable performance. To enable federated learning to work dependably in the real world with complex data environments, we propose FedNKD, which utilizes knowledge distillation and random noise. The superior knowledge of each client is distilled into a central server to mitigate the instablity caused by Non-IID data. Importantly, a synthetic dataset is created by some random noise through back propagation of neural networks. The synthetic dataset will contain the abstract features of the real data. Then we will use this synthetic dataset to realize the knowledge distillation while protecting users' privacy. In our experimental scenarios, FedNKD outperforms existing representative algorithms by about 1.5% in accuracy.

On-boarding new devices into an existing SDN network is a pain for network operations (NetOps) teams, because much expert effort is required to bridge the gap between the configuration models of the new devices and the unified data model in the SDN controller. In this work, we present an assistant framework NAssim, to help NetOps accelerate the process of assimilating a new device into a SDN network. Our solution features a unified parser framework to parse diverse device user manuals into preliminary configuration models, a rigorous validator that confirm the correctness of the models via formal syntax analysis, model hierarchy validation and empirical data validation, and a deep-learning-based mapping algorithm that uses state-of-the-art neural language processing techniques to produce human-comprehensible recommended mapping between the validated configuration model and the one in the SDN controller. In all, NAssim liberates the NetOps from most tedious tasks by learning directly from devices' manuals to produce data models which are comprehensible by both the SDN controller and human experts. Our evaluation shows, NAssim can accelerate the assimilation process by 9.1x. In this process, we also identify and correct 243 errors in four mainstream vendors' device manuals, and release a validated and expert-curated dataset of parsed manual corpus for future research.

Distributed edge learning collaborates powerful edge devices to train a shared global model. Since the frequent communication between the server and workers is very expensive, it is desired to accelerate the learning process. The gradient sparsification is an efficient method that only uploads a small subset of gradient elements. However, most existing works neglect the distributed nature of local datasets, and consequently the local gradients uploaded by edge devices cannot follow the global correct optimization direction well, which results in the loss of accuracy. In this paper, we propose a new gradient sparsification with a renovating mechanism, called Global Renovating Stochastic Gradient Descent (GRSGD). GRSGD utilizes the previous-round global gradient to estimate the current global one and renovates the current zero-sparsified local gradients. It mitigates the communication overhead while making the convergence direction of training closer to the global optimization, accelerating the distributed edge learning process. We provide a theoretical convergence guarantee for our algorithm based on the non-convex assumption, which better fits most deep learning problems. With extensive experiments in PyTorch, we show that GRSGD effectively accelerates the learning process with a smaller communication cost and a faster convergence rate on most training tasks. For example, on ImageNet MnasNet, GRSGD cuts down the gradient size from 8.47MB to 2.13MB while achieving 9.6%+ higher accuracy.

Given a long dialogue, the dialogue summarization system aims to obtain a shorter highlight which retains the important information in the original text. For the customer service scenarios, the summaries of most dialogues between an agent and a user focus on several fixed key points, such as users’ question, users’ purpose, the agent’s solution, and so on. Traditional extractive methods are difficult to extract all predefined key points exactly. Furthermore, there is a lack of large-scale and high-quality extractive summarization datasets containing the annotation for key points. Moreover, the speaker’s role information is ignored or not fully utilized in previous work. In order to solve the above challenges, we propose a Distant Supervision based Machine Reading Comprehension model for extractive Summarization (DSMRC-S). DSMRC-S transforms the summarization task into the machine reading comprehension problem, to fetch key points from the original text exactly according to the predefined questions. In addition, a distant supervision method is proposed to alleviate the lack of eligible extractive summarization datasets. What’s more, a speaker’s role token and the solver classification task are proposed to make full use of speaker’s role information. We conduct experiments on a real-world summarization dataset collected in customer service scenarios, and the results show that the proposed method outperforms the strong baseline methods by 6 percentage points on ROUGE$_L$.

Cloud of Things (CoT) is an integration of Internet of Things (IoT) and cloud computing, where Network Function Virtualization (NFV) can dynamically provide Virtualized Network Functions (VNFs) for IoT devices based on service-specific requirements. The provisioning of VNFs in CoT is formulated as an online decision-making problem, but widely used methods mostly focus on characterizing the environment using simple models to obtain the optimal solution. Valuable historical experience on provisioning for the best long-term benefits is ignored and Quality of Service (QoS) requirements for different types of CoT services are also not considered, which leads to inefficient and coarse-gained provisioning. In this article, an intelligent provisioning framework of VNFs is proposed for adaptive CoT resource scheduling according to traffic identification of heterogeneous network services. The framework leverages a Deep Reinforcement Learning (DRL)-based model to make decisions based on the complexity of network environments and traffic variances. In this model, a policy gradient DRL algorithm, namely, Policy Optimization using Kronecker-Factored Trust Region (POKTR) is adopted to obtain the stable performance by a novel surrogate objective function. Experimental results verify that our framework improves the QoS in CoT by real-time VNFs provisioning. The DRL-based model with POKTR algorithm reduces network congestion and achieves higher throughput than other DRL algorithms.

Distributed deep learning has been widely used in training deep neural networks, especially for big models on massive datasets. Parameter Server (PS) architecture is the most popular distributed training framework, which can flexibly design the global parameter update manner. However, when scaling to complex heterogeneous clusters, stragglers make it difficult for existing distributed paradigms on PS framework to balance between synchronous waiting and staleness, which slows down the model training sharply. In this article, we proposeGrouping Stale Synchronous Parallel(GSSP) scheme, which groups workers with similar performance together. Group servers coordinate intra-group workers using Stale Synchronous Parallel while they communicate with each other asynchronously to eliminate stragglers and refine the model weights. We further proposeGrouping Dynamic Tok-K Sparsification(GDTopK), which dynamically adjusts the upload ratio for each group so as to make communication volume differentiated and mitigate inter-group iteration speed gap. We have conducted experiments on LeNet-5 on MNIST, ResNet-18, VGG-19 on Cifar-10, and Seq2Seq on Multi30k. Results show that GSSP accelerates the training by 46%$\sim$120%, with less than 1 percent accuracy drop. And GDTopK can make up for part of the lost accuracy.

Fine-grained sketch-based image retrieval (FG-SBIR) is considered an ideal method of image retrieval due to the rich and easily accessible characteristics of sketches. It aims to find the most similar photo from the photo gallery based on the input sketch. Most previous works follow the paradigm that extracting global feature first and then projecting the features of sketch and photo to unified embedding feature space using triplet loss. However, the global feature is not appropriate for extracting the crucial fine-grained information. Based on this principle, we propose a Dual Local Interaction Network (DLI-Net). DLI-Net explores an effective and efficient way to utilize local features for FG-SBIR. Specifically, we first propose a Local Feature Extractor to extract mid-level local features. Then, in response to the problems brought by local features, we propose a Dual Interaction Module, which contains Self Interaction Module and Cross Interaction Module. Self Interaction Module speeds up retrieval by eliminating the redundant local features of background. Cross Interaction Module solves the spatial misalignment by making the sketches interact with photos. Extensive experiments on six commonly used datasets show that our DLI-Net outperforms state-of-the-art competitors by a significant margin with a reasonable retrieval speed. Moreover, to the best of our knowledge, DLI-Net is the first model that beats humans on all six datasets. Besides, DLI-Net also performs best on cross-category fine-grained sketch-based image retrieval task, which further demonstrates local features are more appropriate for FG-SBIR.

In this paper, we aim to explore the fine-grained perception ability of deep models for the newly proposed scene sketch semantic segmentation task. Scene sketches are abstract drawings containing multiple related objects. It plays a vital role in daily communication and human-computer interaction. The study has only recently started due to a main obstacle of the absence of large-scale datasets. The currently available dataset SketchyScene is composed of clip art-style edge maps, which lacks abstractness and diversity. To drive further research, we contribute two new large-scale datasets based on real hand-drawn object sketches. A general automatic scene sketch synthesis process is developed to assist with new dataset composition. Furthermore, we propose to enhancing local detail perception in deep models to realize accurate stroke-oriented scene sketch segmentation. Due to the inherent differences between hand-drawn sketches and natural images, extreme low-level local features of strokes are incorporated to improve detail discrimination. Stroke masks are also integrated into model training to guide the learning attention. Extensive experiments are conducted on three large-scale scene sketch datasets. Our method achieves state-of-the-art performance under four evaluation metrics and yields meaningful interpretability via visual analytics.

Although 3D hand pose estimation has made significant progress in recent years with the development of the deep neural network, most learning-based methods require a large amount of labeled data that is time-consuming to collect. In this paper, we propose a dual-branch self-boosting framework for self-supervised 3D hand pose estimation from depth images. First, we adopt a simple yet effective image-to-image translation technology to generate realistic depth images from synthetic data for network pre-training. Second, we propose a dual-branch network to perform 3D hand model estimation and pixel-wise pose estimation in a decoupled way. Through a part-aware model-fitting loss, the network can be updated according to the fine-grained differences between the hand model and the unlabeled real image. Through an inter-branch loss, the two complementary branches can boost each other continuously during self-supervised learning. Furthermore, we adopt a refinement stage to better utilize the prior structure information in the estimated hand model for a more accurate and robust estimation. Our method outperforms previous self-supervised methods by a large margin without using paired multi-view images and achieves comparable results to strongly supervised methods. Besides, by adopting our regenerated pose annotations, the performance of the skeleton-based gesture recognition is significantly improved.

In this paper, we focus on unsupervised representation learning for skeleton-based action recognition. The critical issue of this task is extracting discriminative spatial-temporal information from skeleton sequences to form action representation. To better solve this, we propose a novel unsupervised framework named contrastive-pretext spatial-temporal network (CP-STN), aiming to achieve accurate action recognition by better exploiting discriminative spatial-temporal enhanced features from massive unlabeled data. We combine contrastive and pretext tasks learning paradigms in one framework by using asymmetric spatial and temporal augmentations to enable network extracting discriminative representations with spatial-temporal information fully. Furthermore, graph-based convolution is used as the backbone to explore natural spatial-temporal graph information in skeleton data. Extensive experimental results show that our CP-STN significantly boosts the performance of existing skeleton-based action representations learning networks and achieves state-of-the-art accuracy on two challenging benchmarks in both unsupervised and semi-supervised settings.

Response generation is a fundamental function in conversational systems, where controllability of the response is a key problem. In this paper, we consider how to control the response by lexical constraints, namely lexically constrained response generation. The stochastic search-based methods have achieved promising performance in satisfying lexical constraints. The idea of these methods is modifying a sentence through the actions of insertion, deletion and replacement guided by an optimization algorithm. The core of our method is modifying the response by incorporating the lexical constraints and preserving the message-related parts. For this purpose, we propose the novel Parallel Decoders to guide response modification. The first decoder generates responses according to the given message and constraints. The second decoder calculates the relevance between the input and the response. Based on Parallel Decoders, during the modification, we could sample the positions in the response for editing according to the relevance score. Experiments show the proposed framework achieves better performance than the state-of-the-art generation models in terms of constraint relevance, sentence fluency, response diversity and human evaluation.

Logs play an important part in analyzing system behavior and diagnosing system failures. As the basic step of log analysis, log parsing converts raw log messages into structured log templates. However, existing log parsing approaches are not adaptive and versatile enough to ensure their high accuracy on all types of datasets. In particular, it is required to design regular expressions or fine-tune the hyper-parameters manually for the best performance. In this paper, we propose Prefix-Graph, an online versatile log parsing approach. Prefix-Graph is a probabilistic graph structure extended from prefix tree. It iteratively merges together two branches which have high similarity in probability distribution, and represents log templates as the combination of cut-edges in root-to-leaf paths of the graph. Since no domain knowledge is used and all the parameters are fixed, Prefix-Graph can be easily applied to different log datasets without any additional manual work. We evaluate our approach on 10 real-world datasets and 117GB log messages obtained from Huawei. The experimental results demonstrate that Prefix-Graph achieves the highest average accuracy of 0.975 and the smallest standard deviation of 0.037. Our approach is superior to baseline methods in terms of adaptability and versatility.

Deep neural networks (DNN) have become indispensable tools for intelligent applications today. The demand for deploying DNN on the edge devices increases dramatically. Unfortunately, it is challenging because the DNN inference is computation-intensive, but edge devices are always resource-constraint. Prior solutions attempted to address these challenges with collaboration between cloud and edge devices, but they do not take the inference request rate into account. However, the inference delay will increase dramatically while the request rate becomes higher.In this paper, we propose a scheme to dynamic partition DNN into two or three parts and distribute them at the edge and cloud, achieving the lowest delay with the change of request rate. The scheme selects the optimal partition points of DNN with a layer evaluation model (LEM) and a total delay prediction model (DPM) under different request rates. The experiments of distributed deploying AlexNet, VGG, NiN and ResNet DNN models on image classification dataset ImageNet show that the proposed scheme significantly reduces the total end-to-end latency by fully using both the edge and cloud resources. It reduces the inference delay by 1.3 to 1.6 times and improves the throughput 1.2 to 1.7 times compared to the state of art partition approach.

3D hand reconstruction is a popular research topic in recent years, which has great potential for VR/AR applications. However, due to the limited computational resource of VR/AR equipment, the reconstruction algorithm must balance accuracy and efficiency to make the users have a good experience. Nevertheless, current methods are not doing well in balancing accuracy and efficiency. Therefore, this paper proposes a novel framework that can achieve a fast and accurate 3D hand reconstruction. Our framework relies on three essential modules, including spatial-aware initial graph building (SAIGB), graph convolutional network (GCN) based belief maps regression (GBBMR), and pose-guided refinement (PGR). At first, given image feature maps extracted by convolutional neural networks, SAIGB builds a spatial-aware and compact initial feature graph. Each node in this graph represents a vertex of the mesh and has vertex-specific spatial information that is helpful for accurate and efficient regression. After that, GBBMR first utilizes adaptive-GCN to introduce interactions between vertices to capture short-range and long-range dependencies between vertices efficiently and flexibly. Then, it maps vertices’ features to belief maps that can model the uncertainty of predictions for more accurate predictions. Finally, we apply PGR to compress the redundant vertices’ belief maps to compact-joints’ belief maps with the pose guidance and use these joints’ belief maps to refine previous predictions better to obtain more accurate and robust reconstruction results. Our method achieves state-of-the-art performance on four public benchmarks, FreiHAND, HO-3D, RHD, and STB. Moreover, our method can run at a speed of two to three times that of previous state-of-the-art methods. Our code is available at https://github.com/zxz267/SAR.

Fine-grained sketch-based image retrieval is considered as an ideal alternative to keyword-based image retrieval and image search by image due to the rich and easily accessible characteristics of sketches. Previous works always follow a paradigm that first extracting image global feature with convolution neural network and then optimizing the model with triplet loss. Many efforts on narrowing the domain gap and extracting discriminating features are made by these works. However, they ignored that the global feature is not good at capturing fine-grained details. In this paper, we emphasize the local features are more discriminating than global feature in FG-SBIR and explore an effective way to utilize local features. Specifically, Local Aligned Network (LA-Net) is proposed first, which solves FG-SBIR by directly aligning the mid-level local features. Experiment manifests it can beat all previous baselines and is easy to implement. LA-Net is hoped to be a new strong baseline for FG-SBIR. Next, Dynamic Local Aligned Network (DLA-Net) is proposed to enhance LA-Net. The question of spatial misalignment caused by the abstraction of the sketch is not considered by LA-Net. To solve this question, a dynamic alignment mechanism is introduced into LA-Net. This new mechanism makes the sketch interact with the photo and dynamically decide where to align according to the different photos. The Experiment indicates DLA-Net successfully addresses the question of spatial misalignment. It gains a significant performance boost over LA-Net and outperforms the state-of-the-art in FG-SBIR. To the best of our knowledge, DLA-Net is the first model that beats humans on all datasets---QMUL FG-SBIR, QMUL Handbag, and Sketchy.

With the incremental deployment of software defined networking, the routing algorithms have gained more power on observability and controllability. Deep reinforcement learning, as an experience-driven approach, shows considerable potential in routing problem with the help of the centralized controller. It is an adaptive, lightweight, and model-free approach to coping with dynamic runtime status, large-scale traffic, and heterogeneous objective of SDN routing. However, it is still not suitable for the variable and complex emerging networks, because the huge training cost prevents fast convergence in a varying or discrepant environment. In this paper, we propose a transfer reinforcement learning algorithm to improve the training efficiency, and handle the variation in network status and topology. Specifically, we leverage the generative adversarial network to learn domain-invariant features that is suitable for deep reinforcement learning-based routing in different network environments. This mechanism utilizes the previous model and accelerates the training process. We implement our routing algorithm in the production level software switches and controller, while evaluating it comprehensively with many topologies and network status distributions. The experimental results show that our work not only outperforms the state-of-the-art deep reinforcement learning-based routing frameworks, but also has more training efficiency than the naive transfer learning algorithm both on different topologies and network status distributions.

With the development of the Internet of Things (IoT) and 5G, there are ubiquitous smart devices and network functions providing emerging network services efficiently and optimally through building many network connections based on WiFi, LTE/5G, Ethernet, and etc. The Multipath TCP (MPTCP) protocol that enables these devices to establish multiple paths for simultaneous data transmission, has been a widely used extension of standard TCP in smart devices and network functions. On the other hand, more heavy and time-varying traffic loads are generated in an MPTCP network, so that an efficient congestion control mechanism that schedules the traffic between multiple subflows and avoids congestion is highly required. In this paper, we propose a decentralized learning approach, DeepCC, to adapt to the volatile environments and realize the efficient congestion control. The Multi-Agent Deep Reinforcement Learning (MADRL) is used to learn a policy of congestion control for each subflow according to the real-time network states. To deal with the problem of the fixed state space and slow convergence, we adopt two self-attention mechanisms to receive the states and train the policy, respectively. Due to the asynchronous design of DeepCC, the learning process will not introduce extra delay and overhead on the decision-making process. Experiment results show that DeepCC consistently outperforms the well-known heuristic method and DRL-based MPTCP congestion control method in terms of goodput and jitter. Besides, DeepCC with the attention mechanism reduces convergence time by about 50% and increase goodput by about 80% compared with the commonly used structures of neural networks.

In this paper, we propose an adaptive weighting regression (AWR) method to leverage the advantages of both detection-based and regression-based method. Hand joint coordinates are estimated as discrete integration of all pixels in dense representation, guided by adaptive weight maps. This learnable aggregation process introduces both dense and joint supervision that allows end-to-end training and brings adaptability to weight maps, making network more accurate and robust. Comprehensive exploration experiments are conducted to validate the effectiveness and generality of AWR under various experimental settings, especially its usefulness for different types of dense representation and input modality. Our method outperforms other state-of-the-art methods on four publicly available datasets, including NYU, ICVL, MSRA and HANDS 2017 dataset.

Cross-domain sentiment classification aims to address the lack of massive amounts of labeled data.It demands to predict sentiment polarity on a target domain utilizing a classifier learned from a source domain.In this paper, we investigate how to efficiently apply the pre-training language model BERT on the unsupervised domain adaptation.Due to the pre-training task and corpus, BERT is taskagnostic, which lacks domain awareness and can not distinguish the characteristic of source and target domain when transferring knowledge.To tackle these problems, we design a post-training procedure, which contains the target domain masked language model task and a novel domain-distinguish pre-training task.The post-training procedure will encourage BERT to be domain-aware and distill the domain-specific features in a self-supervised way.Based on this, we could then conduct the adversarial training to derive the enhanced domain-invariant features.Extensive experiments on Amazon dataset show that our model outperforms state-of-the-art methods by a large margin.The ablation study demonstrates that the remarkable improvement is not only from BERT but also from our method.

Existing embedding methods for Attributed Network aim to learn low-dimensional embeddings for nodes, which can preserve both consistency and complementarity for network structures and node attributes. The main assumption is that nodes with similar structures and/or similar attributes should be close in the embedding space. In reality, nodes with similar attributes might be far away from each other in topology and vice versa. The conflict is often caused by noisy links or incomplete network structures. Previous methods either independently project embeddings based on the assumption without considering the conflicts, or encode embeddings into a shared space ignoring the complementarity. In this paper, we propose a Dual Attention-based Adversarial Attributed Network Embedding framework (DAANE) to preserve the consistency and complementarity between structures and attributes, and reduce the conflict caused by their discrepancy. DAANE includes an attribute attention mechanism designed to detect and weakening the impact of noisy links and a structure attention mechanism applied to assign weights to network structures of different scales and capture a more complete global context. Furthermore, we develop efficient adversarial learning when combining the two heterogeneous embeddings. The adversarial auto-encoder projects embeddings of attributes and structures into the same space. Meanwhile, it completely circumvents the interference of various types of noise by removing the constraints of embedding space. Extensive experiments on three realworld network datasets indicate that the proposed model achieves state-of-the-art results.

The data sparsity is the main challenge in sentence classification in social scenarios, the recent methods incorporate user information by encoding user node in the user-relation network to alleviate this issue. However, the connection between users is not always available due to privacy protection or other commercial reasons. Thus, in this paper, a concept called user polarity is proposed to quantify the tendency of sentences published by a user which are categorized into the same class. Then a self-training framework based on user polarity is proposed, which incorporates user information without connection between users, to alleviate the data sparsity in sentence classification. A regularization term is used to strengthen the prediction of the model in some special points, and a sample selector is designed to reduce the noise in the pseudo-labeled data generated in self-training process. Besides, some hard samples are selected to improve the retraining process. The experimental results conducted on SemEval 2019 task 8 indicate that our method performs significantly better than other three semi-supervised methods and achieves state-of-the-art performance on this benchmark.

Federated learning is an emerging concept that trains the machine learning models with the distributed datasets, without sending the raw data to the data center. But in an edge computing enviroment where the wireless network resource is constrained, the key problem of federated learning is the communication overhead for parameters synchronization, which wastes bandwidth, increases training time, and even impacts the model accuracy. Gradient sparsification has received increasing attention, which only updates significant gradients and accumulates insignificant gradients locally. However, how to preserve the accuracy after a high ratio sparsification has always been ignored. In this paper, a General Gradient Sparsification (GGS) framework is proposed for adaptive optimizers, to correct the sparse gradient update process. It consists of two important mechanisms: gradient correction and batch normalization update with local gradients (BN-LG). With gradient correction, the optimizer can properly treat the accumulated insignificant gradients, which makes the model converge better. Furthermore, updating the batch normalization layer with local gradients can relieve the impact of delayed gradients without increasing the communication overhead. We have conducted experiments on LeNet-5, CifarNet, DenseNet-121, and AlexNet with adaptive optimizers. Results show that when 99.9% gradients are sparsified, validation datasets are maintained with top-l accuracy.

Multi-access Edge Computing (MEC) and ubiquitous smart devices help serve end-users efficiently and optimally through providing emerging edge-deployed services. Meanwhile, heavy and time-varying traffic loads are produced in the edge network, so that an efficient traffic forwarding mechanism is required. In this paper, we propose a parallel and distributed learning approach, DeepHop, to adapt to the volatile environments and realize hop-by-hop routing. The Multi-Agent Deep Reinforcement Learning (MADRL) is used to alleviate the edge network congestion and maximize the utilization of network resources. DeepHop determines the routing among edge network nodes for heterogeneous types of traffic according to the current workload and capability. By joining with an attention mechanism, DeepHop obtains the semantics from the elements of the network state to help the agents learn the importance of each element on routing. Experiment results show that DeepHop achieves the increase of successfully transmitted packets by 15% compared with the state-of-the-art algorithms. Besides, DeepHop with an attention mechanism reduces convergence time by nearly half compared with the common-used structures of neural networks.

Federated learning is an emerging concept that trains the machine learning models with the local distributed data sets, without sending the raw data to the data center. But, in the Internet of Things (IoT) where the wireless network resource is constrained, the key problem of federated learning is the communication overhead for parameter synchronization, which wastes bandwidth, increases training time, and even impacts the model accuracy. Gradient sparsification has received increasing attention, which only updates significant gradients and accumulates insignificant gradients locally. However, how to preserve the accuracy after a high ratio sparsification has been ignored in the literature. In this article, a general gradient sparsification (GGS) framework is proposed for adaptive optimizers, to correct the sparse gradient update process. It consists of two important mechanisms: 1) gradient correction and 2) batch normalization (BN) update with local gradients. With gradient correction, the optimizer can properly treat the accumulated insignificant gradients, which makes the model converge better. Furthermore, updating the BN layer with local gradients can relieve the impact of delayed gradients without increasing the communication overhead. We have conducted experiments on LeNet-5, CifarNet, DenseNet-121, and AlexNet with adaptive optimizers. Results show that when 99.9% gradients are sparsified, validation data sets are maintained with top-1 accuracy.

While sequence-to-sequence (seq2seq) models on semantic parsing have demonstrated significant performance, the need for large amounts of labeled data still hinders the application of this technology to resource-poor domains. In this work, we work on alleviating data scarcity in semantic parsing. We propose a semi-supervised semantic parsing methods by exploiting unlabeled natural utterances in a novel multi-task learning framework. Two strategies are proposed. The first one takes entity sequences as training targets to improve the representations of encoder and reduce entity-mistakes in prediction. The second one extends Mean Teacher to seq2seq model and generates more target-side data to improve the generalizability of decoder network. Experiments demonstrate that our proposed methods significantly outperform the supervised baseline and achieve more impressive improvement than previous methods.

With the rise of time-critical and interactive scenarios, ultra-low latency has become the most urgent requirement. Adaptive bitrate (ABR) schemes have been widely used in reducing latency for live streaming services. However, the traditional solutions suffer from a key limitation: they only utilize coarse-grained chunk to solve the I-frame misalignment problem in different bitrate switching process at the cost of increasing latency. As a result, existing schemes are difficult to guarantee the timeliness and granularity of control in essence. In this paper, we use a frame-based approach to solve the I-frame misalignment problem and propose a video adaptation bitrate system (Vabis) in units of the frame for time-critical live streaming to obtain the optimal quality of experience (QoE). On the server-side, a Few-Wait ABR algorithm based on Reinforcement Learning (RL) is designed to adaptively select the bitrate of future frames by state information that can be observed, which can subtly solve the problem of I-frame misalignment. A rule-based ABR algorithm is designed to optimize the Vabis system for the weak network. On the client-side, three delay control mechanisms are designed to achieve frame-based fine-grained control. We construct a trace-driven simulator and the real live platform to evaluate the comprehensive live streaming performance. The results show that Vabis is significantly better than the existing methods with decreases in an average delay of 32%-77% and improvements in average QoE of 28-67%.

Word embedding is a basic task in natural language processing area. Unlike English, Chinese subword units, such as characters, radicals, and components, contain rich semantic information which can be used to enhance word embeddings. However, existing methods neglect the semantic contribution of corresponding subword units to the word. In this work, we employ attention mechanism to capture the semantic structure of Chinese words and propose a novel framework, named Attention-based multi-Layer Word Embedding model(ALWE). We also design an asynchronous strategy for updating embedding and attention efficiently. Our model learns to share subword information between distinct words selectively and adaptively. Experimental results on the word similarity, word analogy, and text classification show that the proposed model outperforms all baselines, especially when words don't appear frequently. Qualitative analysis further demonstrates the superiority of ALWE.

The rapid development of artificial intelligence in multiple scenarios, including machine learning, image recognition, and autonomous driving, has led to an explosion of computation jobs. These jobs are often divided into parallel child tasks and executed in distributed clusters with limited computing resources, making parallel task scheduling one of the most important research topics nowadays. Most studies about parallel task scheduling focused on formulating special scenarios and service requirements as optimization problems. However, the complicated and dynamic parallel computing environments are hard to model, predict and control, making those previous methods unscalable and unable to reflect the real scenarios. In this paper, a Multi-task Deep reinforcement learning approach for scalable parallel Task Scheduling (MDTS) is firstly devised. Generally, Deep Reinforcement Learning (DRL) is a model-free optimization algorithm for long-term control by leveraging experience, but it suffers the curse of dimensionality for decision when coping with complex parallel computing environments and jobs with diverse properties. We extend the action selection in DRL to a multi-task decision, where the output branches of multitask learning are fine-matched to parallel scheduling tasks. Child tasks of a job are accordingly assigned to distributed nodes without any human knowledge while the resource competition among parallel tasks is leveraged through shared neural network layers. Extensive experiments show that the MDTS significantly reduces the job execution time compared with least-connection scheduling and particle swarm optimization algorithm by 15.3% and 39.8% respectively. Moreover, MDTS outperforms the raw DRL algorithm on job execution time, load imbalance value, and total cost by 42.8%, 47.5%, and 59.0%.

Channel pruning can significantly accelerate and compress deep neural networks. Many channel pruning works utilize structured sparsity regularization to zero out all the weights in some channels and automatically obtain structure-sparse network in training stage. However, these methods apply structured sparsity regularization on each layer separately where the correlations between consecutive layers are omitted. In this paper, we first combine one out-channel in current layer and the corresponding in-channel in next layer as a regularization group, namely out-in-channel. Our proposed Out-In-Channel Sparsity Regularization (OICSR) considers correlations between successive layers to further retain predictive power of the compact network. Training with OICSR thoroughly transfers discriminative features into a fraction of out-in-channels. Correspondingly, OICSR measures channel importance based on statistics computed from two consecutive layers, not individual layer. Finally, a global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Our method is comprehensively evaluated with various CNN architectures including CifarNet, AlexNet, ResNet, DenseNet and PreActSeNet on CIFAR-10, CIFAR-100 and ImageNet-1K datasets. Notably, on ImageNet-1K, we reduce 37.2% FLOPs on ResNet-50 while outperforming the original model by 0.22% top-1 accuracy.

Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao, Chun Wang, Bing Ma. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao, Tong Xu, Ming Liu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Face detection has achieved great success with the development of convolution neural network. However, it remains a challenging problem to detect small and blurred faces in unconstrained environment. This paper presents a novel cascade single-shot face detector, named Cascade Single Shot Face Detector (CSSD), which introduces novel cascade classification and regression network in an anchor-based face detector to reject false positives and improve location accuracy. We have contributed in the following three aspects: 1) proposing a feature enchanted and scale-invariable face detection architecture to process faces with different scales; 2) regressing bounding boxes of faces in two steps with a cascade method; 3) filtering negative anchors online after anchor refinement and rebalancing puzzle negative anchors and positive anchors with rate of 3:1. As a consequence, our method achieves state-of-the-art detection performance on FDDB and WIDER FACE dataset.

In the IoT era, 5G will enable various IoT services such as broadband access everywhere, high user and devices mobility, and connectivity of massive number of devices. Radio environment map (REM) can be applied to improve the utilization of radio resources for the access control of IoT devices by allocating them reasonable wireless spectrum resources. However, the primary problem of constructing REM is how to collect the large scale of data. Mobile crowd sensing (MCS), leveraging the smart devices carried by ordinary people to collect information, is an effective solution for collecting the radio environment information for building the REM. In this paper, we build a REM collecting prototype system based on MCS to collect the data required by the radio environment information. However, limited by the budget of the platform, it is hard to recruit enough participants to join the sensing task to collect the radio environment information. This will make the radio environment information of the sensing area incomplete, which cannot describe the radio information accuracy. Considering that the Kriging algorithm has been widely used in geostatistics principle for spatial interpolation for Kriging giving the best unbiased estimate with minimized variance, we utilize the Kriging interpolation algorithm to infer complete radio environment information from collected sample radio environment information data. The interpolation performance is analyzed based on the collected sample radio environment information data. We demonstrate experiments to analyze the Kriging interpolation algorithm interpolation results and error and compared them with the nearest neighbor (NN) and the inverse distance weighting (IDW) interpolation algorithms. Experiment results show that the Kriging algorithm can be applied to infer radio environment information data based on the collected sample data and the Kriging interpolation has the least interpolation error.

Edge computing reorganizes edge resources but is unstable and unreliable at present, thus, orchestration may occur occasionally. This paper provides an architecture named Boundless Resource Orchestrator (BRO) combing cloud computing and edge computing based on containers. The proposed architecture leverages container technology to accelerate and optimize the orchestration process. A master-slave paradigm is implemented in the architecture to provide region autonomy abilities rather than the centralized architecture. Considering the ever-changing circumstance of edge cloud, an orchestration strategy named Best Performance at Least Cost (BPLC) is proposed to maximize the performance of computing at minimum cost dynamically and automatically. Experiments are carried out on measuring couples of infrastructures and orchestration strategies that prove the BRO and BPLC as prior choices dealing with massive jobs in edge computing.

The virtual network functions (VNF) enables to reduce reliance on expensive proprietary networking gear and increase network elasticity. In this paper, we propose an Elastic Virtual Networks (EVN) system to realize a complete virtual network layout, which supports resource customization, automated deployment, and flexible migration. Through the port and MAC address mapping, a new physical node can access the Virtual Networks (VNs), and the related VNFs are deployed through Docker. Moreover, we set up a white list function to reserve the capacity of the hosts that will be accessed in the future to facilitate a fast access. The experiment results verify that the VNF migration time in EVN by Docker improves three to four times compared with the traditional VM solution.

Route planning with global optimization objectives in graphs is a challenging task with enormous computational complexity and finding the best solution is NP-complete. In addition, the network's operational performance varies whenever the environment changes. Traditional routing schemes fail to deal with these situations. We propose a case-based decision system for routing in packet switched networks to track the networking status. We also design a graph-aware neural network to suggest and revise the solutions from the past cases. The low-level structure of the neural network is learned by fitting with the features not only from each standalone vertex but also from the neighbors of each vertex. Experiments show that the proposed system outperforms state-of-art traffic-split and traffic-engineered routing schemes.

Deep neural networks have been used for traffic classification, and promising results are obtained. However, most previous work confined to one specific classification task and restricted the classifiers potential performance and applications. As the traffic flows can be labeled from different perspectives, the performance of the classifier might be improved by exploring more meaningful latent features. For this purpose, we adopted a multi-output DNN model that simultaneously learns different traffic classification tasks. The common knowledge of traffic is exploited by the synergy among the tasks and boosts the individual performances of the tasks. Experiments show that this structure has the potential to meet new future demands and achieve the classification with advanced speeds and fair accuracies. Yet, due to the heavy training cost, the neural networks, though achieving good performance, are hard to implement in the real environment. We further show that few-shot learning could be a viable approach.

Software defined networking decouples the control plane and data plane, which grants more computing power for routing computations. Traditional routing methods suffer from the complex dynamics in networking, and they are facing issues such as slow convergence and performance decline. Deep learning techniques have shown preliminary results on solving the routing problem, bring more accuracy and precision compared with traditional modeling techniques. However, the deep learning architecture needs to be specially customized to learn the topological relations between switches in an efficient way. Thus, we propose a deep learning based intelligent routing strategy with revised graph-aware neural networks and we design a set of features suitable for network routing. Then we demonstrate the performance of our works by using a real-world topology and the production level software switch. The simulation result shows our work is more accurate and efficient compared to state-of-art routing strategy.

Application awareness is essential for traffic engineering and Quality of Service (QoS) guarantee, especially in Internet of Things (IoT). Software Defined Network (SDN) with centralized controlling of network resources provides opportunities for fine- grained resource allocation. However, the controller cannot autonomously identify applications effectively, because sampling and recognizing traffic data consumes a lot of IO and computing resources. In this demonstration, we provide an intelligent application-aware Virtualized Network Function (VNF) with deep learning technology to identify the network traffic. The traffic type information will be mapped to specific network requirements and utilized to search appropriate route paths for different applications. The intelligent VNF is deployed on a GPU-equipped standalone server and works on the data plane of SDN. It identifies the traffic and sends the type information to the controller through OpenFlow protocol. The experiments show that by introducing the type information, SDN controller can assign more appropriate route paths for different types of traffic and highly improve the network QoS.

The emergence of cloud datacenters enhances the capability of online data storage. Since massive data is stored in datacenters, it is necessary to effectively locate interest data in such a distributed system. However, traditional search techniques only allow users to search images over exact-match keywords through a centralized index. These techniques cannot satisfy the requirements of content based image retrieval (CBIR). In this paper, we propose the scalable image retrieval framework which can efficiently support content similarity search in the distributed environment. Its key idea is to integrate image fusion features into distributed hash tables (DHTs) by exploiting the property of the locality sensitive hashing (LSH). Thus, the images with similar content are most likely gathered into the same node without the knowledge of any global information. To the best of our knowledge, there is less comprehensive study on large-scale CBIR with fusion features in the distributed environment.

Since the emergence of cloud datacenters provides an enormous amount of resources easily accessible to people, it is challenging to provide an efficient search framework in such a distributed environment. However, traditional search techniques only allow users to search images over exact-match keywords through a centralized index. These methods are insufficient to meet requirements of content based image retrieval (CBIR) and more powerful search frameworks are needed. In this paper, we present LCFIR, an effective image retrieval framework for fast content location in the distributed situation. It adopts the peer-to-peer paradigm and combines color and edge features. The basic idea is to construct multiple replicas of an image's index through exploiting the property of Locality Sensitive Hashing (LSH). Thus, the indexes of similar images are probabilistically gathered into the same node without the knowledge of any global information. The empirical results show that the system is able to yield high accuracy with load balancing, and only contacts a few number of the participating nodes.

Aligning different ontologies from similar (or same) domains is an active field of current research. There are various solutions which process and analyze lexical, structural or semantic information to align ontologies. However, there are few solutions that focus on interpreting the concepts that entities are presented with and using them in relation to the semantics implied in an ontology. In this paper, the prototype (OACLAI) is presented to tackle this by combining lexical analysis with consequences from reasoners which reflect the semantics implied in an ontology. We evaluate OACLAI over the four real ontologies and compare it against the seven solutions. The experiments show that the accuracy of OACLAI is higher than those of others on average.