In multi-agent systems, explicit cognition of teammates' decision logic serves as a critical factor in facilitating coordination. Communication (i.e., "Tell") can assist in the cognitive development process by information dissemination, yet it is inevitably subject to real-world constraints such as noise, latency, and attacks. Therefore, building the understanding of teammates' decisions without communication remains challenging. To address this, we propose a novel non-communication MARL framework that realizes the construction of cognition through local observation-based modeling (i.e., "Think"). Our framework enables agents to model teammates' active inference process. At first, the proposed method produces three teammate portraits: perception-belief-action. Specifically, we model the teammate's decision process as follows: 1) Perception: observing environments; 2) Belief: forming beliefs; 3) Action: making decisions. Then, we selectively integrate the belief portrait into the decision process based on the accuracy and relevance of the perception portrait. This enables the selection of cooperative teammates and facilitates effective collaboration. Extensive experiments on the SMAC, SMACv2, MPE, and GRF benchmarks demonstrate the superior performance of our method.

Shuyuan Zhao, Wei Chen, Weijie Zhang, Xinrui Hou, Junfeng Shen, Boyan Shi, Shengnan Guo, Youfang Lin, Huaiyu Wan. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Reflections pose a significant challenge to the quality of light field images. Existing reflection removal methods typically regard it as pixel classification of 2D images, overlooking the fundamental principle that reflections arise from the 3D spatial superposition of background and reflection spaces. In this work, we propose a hierarchical interactive Multi-Plane Image (MPI) construction approach, which separates the mixed 3D space containing reflections and constructs two independent MPIs. We leverage the layered structure of MPIs to introduce three hierarchical interaction mechanisms: The inter interaction is designed to separate and recover background and reflection components, the intra interaction aims to reduce errors in information distribution within each plane, and the inner interaction focuses on optimizing the MPI structure itself. Ultimately, we successfully separate the mixed images and reconstruct two independent spatial representations. Compared to existing reflection removal methods, our approach not only achieves superior separation performance but also supports novel view synthesis of the separated results. In challenging scenarios with severe overlap between background and reflection, our method demonstrates a remarkable ability to improve separation quality.

Light Field (LF) is extensively utilized for depth estimation tasks due to its rich structural information. However, real-world LF images often encounter reflective and transparent surfaces and the related regions contain depth information from the reflection and background layers, which can be modeled as dual-layer scenes. For the existing depth estimation frameworks, the constructed cost volume shows an aliasing bimodal distribution in dual-layer surfaces and further causes serious wrong depth results. In this paper, we propose a novel decoupling-and-aggregating strategy and develop a dual-layer depth estimation network for LF images with complex reflections. Specifically, we develop an adaptive cost volume decoupling module to separate both the background and reflection features from the aliasing cost volume. Light field angular-spatial information is sufficiently extracted to infer the effort of features in different dimensions to the background or reflection layer. Additionally, we employ an iterative self-guided aggregating module with multi-stage supervision to aggregate two branches of cost volumes. The module applies the self-guided masks to regularize the distribution of cost volumes. Given the challenge of acquiring the ground truth disparity maps for the LF images under reflection scenes, we also construct a synthetic dataset with dual-layer properties. Our model is the first to introduce dual-layer scenes into the LF depth estimation task using an end-to-end deep neural network. It successfully separates the background and reflection layers and achieves accurate depth estimation results in both layers. Quantitative and qualitative experiment results on publicly available datasets demonstrate that our method performs better than other state-of-the-art methods.

In Location-Based Services (LBS), such as food delivery, a fundamental task is segmenting Areas of Interest (AOIs), aiming at partitioning the urban geographical spaces into non-overlapping regions. Traditional AOI segmentation algorithms primarily rely on road networks to partition urban areas. While promising in modeling the geo-semantics, road network-based models overlooked the service-semantic goals (e.g., workload equality) in LBS service. In this article, we point out that the AOI segmentation problem can be naturally formulated as a Markov Decision Process (MDP), which gradually chooses a nearby AOI for each grid in the current AOI’s border. Based on the MDP, we present the first attempt to generalize Deep Reinforcement Learning (DRL) for AOI segmentation, leading to a novel DRL-based framework called DRL4AOI. The DRL4AOI framework introduces different service-semantic goals in a flexible way by treating them as rewards that guide the AOI generation. To evaluate the effectiveness of DRL4AOI, we develop and release an AOI segmentation system. We also present a representative implementation of DRL4AOI—TrajRL4AOI—for AOI segmentation in the logistics service. It introduces a Double Deep Q-learning Network (DDQN) to gradually optimize the AOI generation for two specific semantic goals: (i) trajectory modularity, i.e., maximize tightness of the trajectory connections within an AOI and the sparsity of connections between AOIs, (ii) matchness with the road network, i.e., maximizing the matchness between AOIs and the road network. Quantitative and qualitative experiments conducted on synthetic and real-world data demonstrate the effectiveness and superiority of our method. The code and system is publicly available at https://github.com/Kogler7/AoiOpt .

Unsupervised graph alignment finds the node correspondence between a pair of attributed graphs by only exploiting graph structure and node features. One category of recent studies first computes the node representation and then matches nodes with the largest embedding-based similarity, while the other category reduces the problem to optimal transport (OT) via Gromov-Wasserstein learning. However, it remains largely unexplored in the model expressiveness, as well as how theoretical expressivity impacts prediction accuracy. We investigate the model expressiveness from two aspects. First, we characterize the model'sdiscriminative powerin distinguishing matched and unmatched node pairs across two graphs. Second, we study the model's capability of guaranteeingnode matching propertiessuch as one-to-one matching and mutual alignment. Motivated by our theoretical analysis, we put forward a hybrid approach namedCombAlignwith stronger expressive power. Specifically, we enable cross-dimensional feature interaction for OT-based learning and propose an embedding-based method inspired by the Weisfeiler-Lehman test. We also apply non-uniform marginals obtained from the embedding-based modules to OT as priors for more expressiveness. Based on that, we propose a traditional algorithm-based refinement, which combines our OT and embedding-based predictions using the ensemble learning strategy and reduces the problem to maximum weight matching. With carefully designed edge weights, we ensure these matching properties and further enhance prediction accuracy. By extensive experiments, we demonstrate a significant improvement of 14.5% in alignment accuracy compared to state-of-the-art approaches and confirm the soundness of our theoretical analysis.

Visual Reinforcement Learning (VRL) has achieved considerable success in solving control tasks. However, generalizing learned policies to new environments remains a major challenge, as agents often overfit to task-irrelevant features in the training environment. To solve this problem, we introduce the concept of decoupling observations into task-relevant and task-irrelevant representations. Building on this idea, we propose a self-supervised T ask- R elevant R epresentation D ecoupling (T2RD) algorithm for VRL. This algorithm consists of three components: task-relevant representation consistency , cross-reconstruction , and cross-dynamic prediction . The first two components achieve the decoupling of content and style features, but the resulting content representations are not necessarily task-relevant. To further refine task-relevant features from content representations, we design the third component that introduces dynamic prediction. T2RD achieves State-of-the-Art (SOTA) generalization performance and sample efficiency in the DeepMind Control Suite and Robotic Manipulation tasks.

Vehicle re-identification aims to match vehicles across non-overlapping camera views. Many existing methods extract features from one specific image, and these methods lack view-invariance when comparing vehicles of different orientations. As a result, discriminative parts obscured by viewpoint changes cannot contribute effectively to matching. This work presents a novel keypoint-based framework for vehicle Re-ID. We propose to explicitly model the intrinsic structural relationships between vehicle components via knowledge graph. By establishing connection between keypoints, our approach aims to leverage such prior to match vehicles even when some parts are not directly comparable due to orientation inconsistencies. Specifically, given query and gallery images, we first detect visible keypoints. Then, a transformer-based model infers features for non-overlapped keypoints by conditioning on visible correspondences defined in the knowledge graph. The final representation integrates visible and inferred features. Extensive experiments demonstrate our method outperforms state-of-the-arts on standard benchmarks under cross-view matching scenarios. To our knowledge, this is the first work introducing structural priors via keypoint knowledge graphs for view-invariant vehicle re-identification.

Reasoning future unknowable facts on temporal knowledge graphs (TKGs) is a challenging task, holding significant academic and practical values for various fields. Existing studies exploring explainable reasoning concentrate on modeling comprehensible temporal paths relevant to the query. Yet, these path-based methods primarily focus on local temporal paths appearing in recent times, failing to capture the complex temporal paths in TKG and resulting in the loss of longer historical relations related to the query. Motivated by the Dual Process Theory in cognitive science, we propose a Cognitive Temporal Knowledge Extrapolation framework (CognTKE), which introduces a novel temporal cognitive relation directed graph (TCR-Digraph) and performs interpretable global shallow reasoning and local deep reasoning over the TCR-Digraph. Specifically, the proposed TCR-Digraph is constituted by retrieving significant local and global historical temporal relation paths associated with the query. In addition, CognTKE presents the global shallow reasoner and the local deep reasoner to perform global one-hop temporal relation reasoning (System 1) and local complex multi-hop path reasoning (System 2) over the TCR-Digraph, respectively. The experimental results on four benchmark datasets demonstrate that CognTKE achieves significant improvement in accuracy compared to the state-of-the-art baselines and delivers excellent zero-shot reasoning ability.

Spatial-temporal forecasting and imputation are important for real-world intelligent systems. Most existing methods are tailored for individual forecasting or imputation tasks but are not designed for both. Additionally, they are less effective for zero-shot and few-shot learning. While pre-trained language model (PLM) have exhibited strong pattern recognition and reasoning abilities across various tasks, including few-shot and zero-shot learning, their applications in spatial-temporal data understanding has been constrained by insufficient modeling of complex correlations such as the temporal correlations, spatial connectivity, non-pairwise and high-order spatial-temporal correlations within data. In this paper, we propose STD-PLM for understanding both spatial and temporal properties of Spatial-Temporal Data with PLM, which is capable of implementing both spatial-temporal forecasting and imputation tasks. STD-PLM understands spatial-temporal correlations via explicitly designed spatial and temporal tokenizers. Topology-aware node embeddings are designed for PLM to comprehend and exploit the topology structure of data in inductive manner. Furthermore, to mitigate the efficiency issues introduced by the PLM, we design a sandglass attention module(SGA) combined with a specific constrained loss function, which significantly improves the model's efficiency while ensuring performance. Extensive experiments demonstrate that STD-PLM exhibits competitive performance and generalization capabilities across the forecasting and imputation tasks on various datasets. Moreover, STD-PLM achieves promising results on both few-shot and zero-shot tasks.

Uncertainty quantification in travel time estimation (TTE) aims to estimate the confidence interval for travel time, given the origin (O), destination (D), and departure time (T). Accurately quantifying this uncertainty requires generating the most likely path and assessing travel time uncertainty along the path. This involves two main challenges: 1) Predicting a path that aligns with the ground truth, and 2) modeling the impact of travel time in each segment on overall uncertainty under varying conditions. We propose DutyTTE to address these challenges. For the first challenge, we introduce a deep reinforcement learning method to improve alignment between the predicted path and the ground truth, providing more accurate travel time information from road segments to improve TTE. For the second challenge, we propose a mixture of experts guided uncertainty quantification mechanism to better capture travel time uncertainty for each segment under varying contexts. Extensive experiments on two real-world datasets demonstrate the superiority of our proposed method.

Communication has been widely employed to enhance multi-agent collaboration. Previous research has typically assumed delay-free communication, a strong assumption that is challenging to meet in practice. However, real-world agents suffer from channel delays, receiving messages sent at different time points, termed Asynchronous Communication, leading to cognitive biases and breakdowns in collaboration. This paper first defines two communication delay settings in MARL and emphasizes their harm to collaboration. To handle the above delays, this paper proposes a novel framework, Communication Delay-Tolerant Multi-Agent Collaboration (CoDe). At first, CoDe learns an intent representation as messages through future action inference, reflecting the stable future behavioral trends of the agents. Then, CoDe devises a dual alignment mechanism of intent and timeliness to strengthen the fusion process of asynchronous messages. In this way, agents can extract the long-term intent of others, even from delayed messages, and selectively utilize the most recent messages that are relevant to their intent. Experimental results demonstrate that CoDe outperforms baseline algorithms in three MARL benchmarks without delay and exhibits robustness under fixed and time-varying delays.

Recent Continual Learning (CL)-based Temporal Knowledge Graph Reasoning (TKGR) methods focus on significantly reducing computational cost and mitigating catastrophic forgetting caused by fine-tuning models with new data. However, existing CL-based TKGR methods still face two key limitations: (1) They usually one-sidedly reorganize individual historical facts, while overlooking the historical context essential for accurately understanding the historical semantics of these facts; (2) They preserve historical knowledge by simply replaying historical facts, while ignoring the potential conflicts between historical and emerging facts. In this paper, we propose a \textbf{D}eep \textbf{G}enerative \textbf{A}daptive \textbf{R}eplay (DGAR) method, which can generate and adaptively replay historical entity distribution representations from the whole historical context. To address the first challenge, historical context prompts as sampling units are built to preserve the whole historical context information. To overcome the second challenge, a pre-trained diffusion model is adopted to generate the historical distribution. During the generation process, the common features between the historical and current distributions are enhanced under the guidance of the TKGR model. In addition, a layer-by-layer adaptive replay mechanism is designed to effectively integrate historical and current distributions. Experimental results demonstrate that DGAR significantly outperforms baselines in reasoning and mitigating forgetting.

Mutation analysis is a fundamental technique in software engineering, playing a vital role in software testing and debugging. It introduces small artificial faults (mutants) into a program to simulate realistic defects. These mutants are systematically injected and then leveraged in downstream tasks such as mutation testing, mutation-based fault localization (MBFL), and mutation-based test case prioritization (MBTCP). However, all of these applications require generating and executing a large number of mutants, leading to substantial computational overhead. This scalability issue significantly limits the applicability of mutation analysis to large-scale software systems. To address this challenge, we propose CAMUS, a context-aware neural mutation selection approach. CAMUS models each mutation and its surrounding context as a graph enriched with AST hierarchy and type information. It then employs an encoder with self-attention mechanisms to produce embeddings, from which the selection probability of each mutant is predicted. We evaluate CAMUS on three downstream tasks, i.e., mutation testing, MBFL, and MBTCP, by comparing it with state-of-the-art mutation selection techniques, including the recent large language model DeepSeek-v3. Experiments are conducted on the Defects4J 2.0 and ConDefects benchmarks. Our results show that CAMUS consistently outperforms existing methods across a wide range of mutation retention rates. In mutation testing, CAMUS achieves over 95% accuracy using only 50% of the mutants. In MBTCP, it reaches near-optimal prioritization performance with as little as 5% of the mutants, demonstrating its strong efficiency and effectiveness.

Explaining deep models for time-series data is crucial for identifying key patterns in sensitive domains, such as healthcare and finance. However, due to the lack of unified optimization criterion, existing explanation methods often suffer from redundancy and incompleteness, where irrelevant patterns are included or key patterns are missed in explanations. To address this challenge, we propose the Optimal Information Retention Principle, where conditional mutual information defines minimizing redundancy and maximizing completeness as optimization objectives. We then derive the corresponding objective function theoretically. As a practical framework, we introduce an explanation framework ORTE, learning a binary mask to eliminate redundant information while mining temporal patterns of explanations. We decouple the discrete mapping process to ensure the stability of gradient propagation, while employing contrastive learning to achieve precise filtering of explanatory patterns through the mask, thereby realizing a trade-off between low redundancy and high completeness. Extensive quantitative and qualitative experiments on synthetic and real-world datasets demonstrate that the proposed principle significantly improves the accuracy and completeness of explanations compared to baseline methods. The code is available at https://github.com/moon2yue/ORTE_public.

Spatio-temporal trajectories are crucial for data mining tasks, requiring versatile learning methods that can accurately extract movement patterns and travel purposes. While large language models (LLMs) have shown remarkable versatility through training on extensive datasets, and trajectories share similarities with natural language, standard LLMs cannot directly handle spatio-temporal features or extract trajectory-specific information. We propose TrajCogn, a model that effectively adapts LLMs for trajectory learning. TrajCogn incorporates a novel trajectory semantic embedder to process spatio-temporal features and extract movement patterns and travel purposes, along with a trajectory prompt that integrates this information into LLMs for various downstream tasks. Experiments on three real-world datasets and four representative tasks demonstrate TrajCogn's effectiveness.

Advanced deep spatio-temporal networks have become the mainstream for traffic prediction, but the widespread adoption of these models is impeded by the prevalent scarcity of available data. Despite cross-city transfer learning emerging as a common strategy to address this issue, it overlooks the inherent distribution imbalances within each city, which could potentially hinder the generalization capabilities of pre-trained models. To overcome this limitation, we propose a Spatio-Temporal Balanced Transfer Learning (STBaT) framework to enhance existing spatio-temporal prediction networks, ensuring both universality and precision in predictions for new urban environments. A Regional Imbalance Acquisition Module is designed to model the regional imbalances of source cities. Besides, to promote generalizable knowledge acquisition, a Spatio-Temporal Balanced Learning Module is devised to balance the predictive learning process. Extensive experiments on real-world datasets validate the efficacy of our proposed approach compared with several state-of-the-art methods.

Continual Multi-Agent Reinforcement Learning (Co-MARL) requires agents to address catastrophic forgetting issues while learning new coordination policies with the dynamics team. In this paper, we delve into the core of Co-MARL, namely Relation Patterns, which refer to agents’ general understanding of interactions. In addition to generality, relation patterns exhibit task-specificity when mapped to different action spaces. To this end, we propose a novel method called General Relation Patterns-Guided Task-specific Decision-Maker (RPG). In RPG, agents extract relation patterns from dynamic observation spaces using a relation capturer. These task-agnostic relation patterns are then mapped to different action spaces via a task-specific decision-maker generated by a conditional hypernetwork. To combat forgetting, we further introduce regularization items on both the relation capturer and the conditional hypernetwork. Results on SMAC and LBF demonstrate that RPG effectively prevents catastrophic forgetting when learning new tasks and achieves zero-shot generalization to unseen tasks.

Incomplete time series classification is both practically valuable and challenging as missing values in time series data are prevalent in real-world scenarios. Current approaches suffer from two major limitations. First, they overemphasize the consistency of data reconstruction during missing value imputation while neglecting the task-effectiveness of the imputed results for the classification. Second, they fail to systematically establish a synergistic optimization mechanism between data imputation and feature representation. To address these challenges, we propose a Hierarchical Conditional Information Bottleneck (HCIB) framework, which achieves incomplete time series classification through end-to-end joint optimization. Specifically, at the data imputation level, we re-examine the dual effects of missing data: the loss of critical information (Loss) versus the gain in interference suppression (Gain), elucidating this duality through bias-variance trade-off theory. Building on this analysis, we propose a task-information sufficiency criterion and extend the information bottleneck theory into a task-driven imputation framework by incorporating label information as a conditional constraint. At the feature representation level, we construct a hierarchical information bottleneck architecture to learn compressed yet informative temporal representations from the task-oriented imputed data. Furthermore, we derive the optimizable objective function for HCIB and design specialized neural network architectures for time series. Comprehensive experiments on multivariate and univariate time series datasets across multiple domains consistently demonstrate that the proposed method achieves significant improvements in classification performance compared to SOTA approaches.

Light Field (LF) semantic segmentation relies on leveraging redundant information across multiple views to assign a semantic label to each pixel of the central view. Recent approaches typically feed the views into a pre-trained backbone and utilize an estimated depth map to aggregate semantic representations for label prediction. However, these methods ignore the correlation between encoded structural cues in LF and semantic labels. On one hand, it is challenging to identify matching points for regions that are occluded in some views. This broken view consistency emphasizes object edge localization, facilitating more precise edge labeling. On the other hand, the depth continuity for the same object ensures semantic consistency in adjacent regions. Therefore, effectively extracting structural cues and integrating them into semantic segmentation are key points in LF semantic segmentation.In this paper, we propose an Epipolar Consistency-based network for structure-aware LF semantic segmentation, termed ECNet. First, we explore the epipolar consistency between views to characterize the edges and depth cues of the input. Based on the embedded edges information, we design an edge-semantic correlation transformer to generate fine-grained representations of object edges. Furthermore, the proposed depth-semantic correlation transformer maps semantic features of one object closer together according the depth information.Extensive experiments demonstrate that ECNet achieves state-of-the-art performance, which reduces computational cost by 33.3% (in terms of FLOPs) while maintaining high segmentation accuracy.

Unsupervised pre-training on large-scale datasets has demonstrated significant potential for improving the sample efficiency and performance of Reinforcement Learning (RL). Given the large-scale action-free internet videos, existing methods utilize single-step transition prediction and image reconstruction to learn representations. However, these methods prefer to preserve large-proportion stationary information in the pixel space, neglecting small but crucial information. To preserve enough information in the representation, it is essential to pay equal attention to each element in videos. Specifically, we propose a temporal correlation space to distinguish each element. For implementation, we introduce the Multi-scale Temporal Contrastive Learning (MTCL) method to model multi-scale temporal correlations separately. This approach can balance the attention of different elements and yield more informative representations, effectively supporting policy learning in various downstream tasks. Experimental results demonstrate that our method improves sample efficiency and asymptotic performance across various downstream tasks.

Emotion recognition based on multimodal physiological signals is playing an increasingly important role in areas such as human-computer interaction and disease diagnosis, attracting growing attention from the research community. Current studies primarily focus on emotion recognition under unified data collection paradigms, overlooking the prevalent issue of imperfect modality matching in real-world scenarios. In particular, existing methods fail to effectively utilize these mismatched modalities, leading to incomplete emotional representations. This limits the model's ability to accurately capture the multidimensional semantic features of emotions, thereby constraining its effectiveness and applicability in practical settings. To address this challenge, we propose MoCERNet. At the modality level, it first reduces the domain gap among matched modalities and then aligns mismatched modalities in a semantics-aware manner, guided by the matched ones. At the decision level, it further mitigates the global distribution discrepancies to achieve a more complete emotional representation. In addition, we design a Nervous System Functional Structure Transformer (NFSformer) that enables the model to focus on the correlation between different brain regions and peripheral physiological signals under various emotional states, thereby enhancing its capacity to model complex emotional processes. Experiments on three multimodal emotion datasets demonstrate that MoCERNet outperforms state-of-the-art baselines under imperfect modality matching scenarios.

Medical time-series analysis differs fundamentally from general ones by requiring specialized domain knowledge to interpret complex signals and clinical context. Large language models (LLMs) hold great promise for augmenting medical time-series analysis by complementing raw series with rich contextual knowledge drawn from biomedical literature and clinical guidelines. However, realizing this potential depends on precise and meaningful prompts that guide the LLM to key information. Yet, determining what constitutes effective prompt content remains non-trivial—especially in medical settings where signal interpretation often hinges on subtle, expert-defined decision-making indicators. To this end, we propose InDiGO, a knowledge-aware evolutionary learning framework that integrates clinical signals and decision-making indicators through iterative optimization. Across four medical benchmarks, InDiGO consistently outperforms prior methods. The code is available at: https://github.com/jinxyBJTU/InDiGO.

Vehicle GPS trajectories record how vehicles move over time, storing valuable travel semantics, including movement patterns and travel purposes. Learning travel semantics effectively and efficiently is crucial for real-world applications of trajectory data, which is hindered by two major challenges. First, travel purposes are tied to the functions of the roads and points-of-interest (POIs) involved in a trip. Such information is encoded in textual addresses and descriptions and introduces heavy computational burden to modeling. Second, real-world trajectories often contain redundant points, which harm both computational efficiency and trajectory embedding quality. To address these challenges, we propose TrajMamba, a novel approach for efficient and semantically rich vehicle trajectory learning. TrajMamba introduces a Traj-Mamba Encoder that captures movement patterns by jointly modeling both GPS and road perspectives of trajectories, enabling robust representations of continuous travel behaviors. It also incorporates a Travel Purpose-aware Pre-training procedure to integrate travel purposes into the learned embeddings without introducing extra overhead to embedding calculation. To reduce redundancy in trajectories, TrajMamba features a Knowledge Distillation Pre-training scheme to identify key trajectory points through a learnable mask generator and obtain effective compressed trajectory embeddings. Extensive experiments on two real-world datasets and three downstream tasks show that TrajMamba outperforms state-of-the-art baselines in both efficiency and accuracy.

Affective brain-computer interfaces (aBCIs) play a crucial role in personalized human–computer interaction and neurofeedback modulation. To develop practical and effective aBCI paradigms and to investigate the spatial-temporal dynamics of brain activity under emotional inducement, portable electroencephalography (EEG) signals have been widely adopted. To further enhance spatial-temporal perception, functional near-infrared spectroscopy (fNIRS) has attracted increasing interest in the aBCI field and has been explored in combination with EEG. However, existing datasets typically provide only static fixation labels, overlooking the dynamic changes in subjects' emotions. Notably, some studies have attempted to collect continuously annotated emotional data, but they have recorded only peripheral physiological signals without directly observing brain activity, limiting insight into underlying neural states under different emotions. To address these challenges, we present the Real-time labeled EEG-fNIRS Dataset (REFED). To the best of our knowledge, this is the first EEG-fNIRS dataset with real-time dynamic emotional annotations. REFED simultaneously records brain signals from both EEG and fNIRS modalities while providing continuous, real-time annotations of valence and arousal. The results of the data analysis demonstrate the effectiveness of emotion inducement and the reliability of real-time annotation. This dataset offers the possibility for studying the neurovascular coupling mechanism under emotional evolution and for developing dynamic, robust affective BCIs.

Spatiotemporal trajectory data is crucial for various traffic-related applications. However, issues such as device malfunctions and network instability often result in sparse trajectories that lose detailed movement information compared to their dense counterparts. Recovering missing points in sparse trajectories is thus essential. Despite recent progress, three challenges remain. First, the lack of large-scale dense trajectory datasets hinders the training of a trajectory recovery model. Second, the varying spatiotemporal correlations in sparse trajectories make it hard to generalize across different sampling intervals. Third, extracting road conditions for missing points is non-trivial. To address these challenges, we propose $\textit{PLMTrajRec}$, a novel trajectory recovery model. It leverages the scalability of a pre-trained language model (PLM) and can effectively recover trajectories by fine-tuning with small-scale dense trajectory datasets. To handle different sampling intervals in sparse trajectories, we first convert sampling intervals and movement features into prompts for the PLM to understand. We then introduce a trajectory encoder to unify trajectories of varying intervals into a single interval. To extract road conditions for missing points, we propose an area flow-guided implicit trajectory prompt that represents traffic conditions in each region, and a road condition passing mechanism that infers the road conditions of missing points using the observed ones. Experiments on four public trajectory datasets with three sampling intervals demonstrate the effectiveness, scalability, and generalization ability of PLMTrajRec. Code is available at https://github.com/wtl52656/PLMTrajRec.

Vehicle GPS trajectories provide valuable movement information that supports various downstream tasks and applications. A desirable trajectory learning model should be able to transfer across regions and tasks without retraining, avoiding the need to maintain multiple specialized models and subpar performance with limited training data. However, each region has its unique spatial features and contexts, which are reflected in vehicle movement patterns and are difficult to generalize. Additionally, transferring across different tasks faces technical challenges due to the varying input-output structures required for each task. Existing efforts towards transferability primarily involve learning embedding vectors for trajectories, which perform poorly in region transfer and require retraining of prediction modules for task transfer. To address these challenges, we propose $\textit{TransferTraj}$, a vehicle GPS trajectory learning model that excels in both region and task transferability. For region transferability, we introduce RTTE as the main learnable module within TransferTraj. It integrates spatial, temporal, POI, and road network modalities of trajectories to effectively manage variations in spatial context distribution across regions. It also introduces a TRIE module for incorporating relative information of spatial features and a spatial context MoE module for handling movement patterns in diverse contexts. For task transferability, we propose a task-transferable input-output scheme that unifies the input-output structure of different tasks into the masking and recovery of modalities and trajectory points. This approach allows TransferTraj to be pre-trained once and transferred to different tasks without retraining. We conduct extensive experiments on three real-world vehicle trajectory datasets under various transfer settings, including task transfer, zero-shot region transfer, and few-shot region transfer. Experimental results demonstrate that TransferTraj significantly outperforms state-of-the-art baselines in different scenarios, validating its effectiveness in region and task transfer. Code is available at https://github.com/wtl52656/TransferTraj.

C++ is a system-level programming language for modern software development, which supports multiple programming paradigms, including object-oriented, generic, and functional programming. The intrinsic complexity of these paradigms and their interactions grants C++ powerful expressiveness while posing significant challenges for compilers in correctly implementing its type system. A type system encompasses various aspects such as type inference, type checking, subtyping, type conversions, generics, scoping, and binding. However, systematic testing of the type systems of C++ compilers remains largely underexplored in existing studies. In this work, we present TyMut, the first approach specifically designed to test the C++ type system. TyMut is a mutation-based compiler fuzzer equipped with advanced type-driven mutation operators, carefully crafted to target intricate type-related features such as template generics, type conversions, and inheritance. Beyond differential testing, TyMut introduces enhanced test oracles through a must analysis that partially confirms the validity of generated programs. Specifically, mutation operators are classified into well-formed and not-well-formed : Programs generated by well-formed mutation operators are valid and must be accepted by compilers. Programs generated by not-well-formed operators are validated against a set of well-formedness rules . Any violation indicates the program is invalid and must be rejected. For programs that pass the rules but lack a definitive oracle, TyMut applies differential testing to identify behavioral inconsistencies across compilers. The testing campaign took about 32 hours to generate and test 250584 programs. The must analysis provides definite test oracles for nearly 80% of all generated programs. TyMut uncovered 102 bugs in the recent versions of GCC and Clang, with 56 confirmed as new bugs by compiler developers. Among the confirmed bugs, 26 of them cause compiler crashes, and more than 50% cause miscompilation. Additionally, 7 of them had remained hidden for over 20 years, 22 for over 10 years, and 39 for over 5 years. One long-standing bug discovered by TyMut was later confirmed as the root cause of a real-world issue in TensorFlow. Before submitting this paper, 13 bugs were fixed, most of which were fixed within 60 days. Notably, some unconfirmed bugs have led to in-depth discussions among developers. For instance, one bug led a compiler developer to submit a new issue to the C++ language standard, showing that we uncovered ambiguities in the language specification.

ABSTRACT Fault localization (FL) aims to automatically find the location of bugs in a software program. In the family of FL approaches, spectrum‐based fault localization (SBFL) is the most widely used and has been extensively studied, which computes the suspicious scores of a code element to be buggy via test coverage information. Mutation‐based fault localization (MBFL) further analyses the relationship between mutants and test results. Although MBFL involves more information, it also faces the following limitations. (1) To precisely evaluate suspicious sources, SBFL approaches proposed dozens of formulae based on coverage, while only a few of them have been adopted by MBFL. (2) The current MBFL approaches are based on the assumption that the capability in localizing bugs of each mutant is the same, so they assign the same weight to the mutants that change the test outputs and consider the mutant from the same code element equivalent. In this paper, we intend to enrich the MBFL family by the following two approaches. First, we collect 25 typical SBFL formulae and transform them into MBFL versions. Second, we propose new assumptions that the mutants should have different weights in computing suspicious sources and propose BLMu, a novel MBFL approach that treats mutants differently. We also propose novel metrics for MBFL by considering the high cost of mutation analysis. We perform large‐scale experiments by evaluating all the MBFL approaches against 395 real‐world Java bugs of the Defects4J benchmark. Our evaluation results reveal that BLMu demonstrates a substantial improvement over both MUSE and Metallaxis, the most popular MBFL approaches, at both the statement level and the method level. Specifically, in terms of Top‐1 , BLMu improves by 106% at the statement level and 69% at the method level. When compared with other types of FL approaches, MBFL outperforms typical SBFL approaches while still far behind the state‐of‐the‐art learning‐based FL approaches.

The study of the relationship between emotions and physiological signals of subjects under multimedia stimulation is an emerging field, and many important advances are made. However, there are still some challenges: 1) How to effectively utilize the complementarity among spatial-spectral-temporal domain information. 2) How to employ the heterogeneity and the correlation among multi-modal physiological signals simultaneously. 3) How to improve the robustness of the model dealing with missing channels. 4) How to model the dependency among different emotions. In this paper, we propose a novel two-stream Dynamic Heterogeneous Graph Recurrent Neural Network called DHGRNN. Specifically, DHGRNN consists of a spatial-temporal stream, a spatial-spectral stream, a fusion layer, and a multi-label classifier. Each stream is composed of a graph transformer network, evolved graph convolutional neural network, and gated recurrent units. We propose a graph-based two-stream structure to fuse the information of the spatial-spectral-temporal domain simultaneously. Graph transformer network and evolved graph convolutional neural network are used to model the heterogeneity and correlation of multi-modal physiological signals, respectively. To deal with the problem of robustness in the face of missing channel data, we transform it into the problem of dynamic graphs and use a dynamic graph neural network to improve the robustness. In addition, we propose a multi-label classifier to model the dependency among different emotion dimensions. Experiments on three public datasets demonstrate that our proposed model outperforms existing state-of-the-art methods.

Sleep stage classification is an important step in the diagnosis and treatment of sleep disorders. Despite the high classification performance of previous sleep stage classification work, some challenges remain unresolved: 1) How to effectively capture salient waves in sleep signals to improve sleep stage classification results. 2) How to capture salient waves affected by inter-subject variability. 3) How to adaptively regulate the importance of different modals for different sleep stages. To address these challenges, we propose SleepWaveNet, a multimodal salient wave detection network, which is motivated by the salient object detection task in computer vision. It has a U-Transformer structure to detect salient waves in sleep signals. Meanwhile, the subject-adaptation wave extraction architecture based on transfer learning can adapt to the information of target individuals and extract salient waves with inter-subject variability. In addition, the multimodal attention module can adaptively enhance the importance of specific modal data for sleep stage classification tasks. Experiments on three datasets show that SleepWaveNet has better overall performance than existing baselines. Moreover, visualization experiments show that the model has the ability to capture salient waves with inter-subject variability.

Next POI (Point-of-Interest) recommendation aims to forecast users’ future movements based on their historical check-in trajectories, holding significant value in location-based services. Existing methods address trajectory data sparsity by integrating rich auxiliary information or using spatial-temporal knowledge graphs (STKGs), showing promising results. Yet, they face two main challenges: i) Due to the difficulty of transforming structured trajectory data into trajectory text describing users’ spatial-temporal mobility, the powerful reasoning ability of pre-trained language models is rarely explored to enhance recommendation performance. ii) Methods based on STKG can introduce external knowledge inconsistent with user preferences, leading to the knowledge noise generated hampering the accuracy of recommendations. To this end, we propose a novel approach called STKG-PLM that integratesSTKGcontrastive learning andprompt pre-trainedlanguagemodel (PLM) to enhance the next POI recommendation. Specifically, we design a spatial-temporal trajectory prompt template that transforms structured trajectories into text corpus based on STKG, serving as the input of PLM to understand the movement pattern of users from coarse-grained and fine-grained perspectives. Additionally, we propose an STKG contrastive learning framework to mitigate the introduced knowledge noise. Extensive experiments on three real-world datasets demonstrate that STKG-PLM exhibits notable performance improvements over the state-of-the-art baseline methods.

Analyzing and comprehending check-in sequences is crucial for various applications in smart cities. However, publicly available check-in datasets are often limited in scale due to privacy concerns. This poses a significant obstacle to academic research and downstream applications. Thus, it is urgent to generate realistic check-in datasets. The denoising diffusion probabilistic model (DDPM) as one of the most capable generation methods is a good choice to achieve this goal. However, generating check-in sequences using DDPM is not an easy feat. The difficulties lie in handling check-in sequences of variable lengths and capturing the correlation from check-in sequences' distinct characteristics. This paper addresses the challenges by proposing a Spatio-Temporal Contrastive Diffusion Model (STCDM). This model introduces a novel spatio-temporal lossless encoding method that effectively encodes check-in sequences into a suitable format with equal length. Furthermore, we capture the spatio-temporal correlations with two disentangled diffusion modules to reduce the impact of the difference between spatial and temporal characteristics. Finally, we incorporate contrastive learning to enhance the relationship between diffusion modules. We generate four realistic datasets in different scenarios using STCDM and design four metrics for comparison. Experiments demonstrate that our generated datasets are more realistic and free of privacy leakage.

Vehicle movement is frequently captured in the form of GPS trajectories, i.e., sequences of timestamped GPS locations. Such data is widely used for various tasks such as travel-time estimation, trajectory recovery, and trajectory prediction. A universal vehicle trajectory model could be applied to different tasks, removing the need to maintain multiple specialized models, thereby reducing computational and storage costs. However, creating such a model is challenging when the integrity of trajectory features is compromised, i.e., in scenarios where only partial features are available or the trajectories are sparse. To address these challenges, we propose the Universal Vehicle Trajectory Model (UVTM), which can effectively adapt to different tasks without excessive retraining. UVTM incorporates two specialized designs. First, it divides trajectory features into three distinct domains. Each domain can be masked and generated independently to accommodate tasks with only partially available features. Second, UVTM is pre-trained by reconstructing dense, feature-complete trajectories from sparse, feature-incomplete counterparts, enabling strong performance even when the integrity of trajectory features is compromised. Experiments involving four representative trajectory-related tasks on three real-world vehicle trajectory datasets provide insight into the performance of UVTM and offer evidence that it is capable of meeting its objectives.

Spatiotemporal trajectories are sequences of timestamped locations, which enable a variety of analyses that in turn enable important real-world applications. It is common to map trajectories to vectors, called embeddings, before subsequent analyses. Thus, the qualities of embeddings are very important. Methods for pre-training embeddings, which leverage unlabeled trajectories for training universal embeddings, have shown promising applicability across different tasks, thus attracting considerable interest. However, research progress on this topic faces two key challenges: a lack of a comprehensive overview of existing methods, resulting in several related methods not being well-recognized, and the absence of a unified pipeline, complicating the development of new methods and the analysis of methods. We present UniTE, a survey and a unified pipeline for this domain. In doing so, we present a comprehensive list of existing methods for pre-training trajectory embeddings, which includes methods that either explicitly or implicitly employ pre-training techniques. Further, we present a unified and modular pipeline with publicly available underlying code, simplifying the process of constructing and evaluating methods for pre-training trajectory embeddings. Additionally, we contribute a selection of experimental results using the proposed pipeline on real-world datasets.

Traffic signal control (TSC) plays a crucial role in the intelligent transportation system. Among existing TSC approaches, Proactive Traffic Signal Control (PTSC) predicts future traffic states at intersections and proactively adjusts control policies. It is evident that PTSC methods are highly effective in alleviating both current and future traffic congestion at intersections. However, existing PTSC methods focus on point estimation prediction while neglecting prediction reliability. Additionally, they fail to adaptively coordinate between current and future traffic states for optimal control. To address these limitations, we propose an innovative Proactive-Plugin that can be combined with existing TSC methods to enhance the accuracy and robustness of traffic signal control policies. This plugin enhances two critical aspects: 1) Prediction reliability is achieved through Fine-grained Traffic Uncertainty Quantification. This module generates probabilistic forecasts along with confidence intervals to explicitly indicate the credibility of the predictions. 2) Coordination adaptiveness is enabled by a Current-Future Tradeoff Integration mechanism. This mechanism dynamically adjusts the relative influence of current traffic states and probabilistic forecasts on control policies. To further ensure robustness, we design a multi-task joint optimization to reduce the negative impact of inaccurate predictions during training. Experimental results on six real-world datasets demonstrate consistent improvements in traffic efficiency, validating the effectiveness of our approach.

As densely-sampled Light Field (LF) images are beneficial to many applications, LF reconstruction becomes an important technology in related fields. Recently, neural rendering shows great potential in reconstruction tasks. However, volume rendering in existing methods needs to sample many points on the whole camera ray or epipolar line, which is time-consuming. In this paper, specifically for LF images with regular angular sampling, we propose a novel Structure-Aware Pre-Selected neural rendering framework for LF reconstruction. Instead of sampling on the whole epipolar line, we propose to sample on several specific positions, which are estimated using the color and inherent scene structure information explored in the regular angular sampled LF images. Sampling only a few points that closely match the target pixel, the feature of the target pixel is quickly rendered with high-quality. Finally, we fuse the features and decode them in the view dimension to obtain the final target view. Experiments show that the proposed method outperforms the state-of-the-art LF reconstruction methods in both qualitative and quantitative comparisons across various tasks. Our method also surpasses the most existing methods in terms of speed. Moreover, without any retraining or fine-tuning, the performance of our method with no-per-scene optimization is even better than the methods with per-scene optimization.

Self-supervised contrastive learning (SCL) has driven significant advancements in time-series representation learning. While recent studies built upon the information noise contrastive estimation (InfoNCE) loss framework focus on constructing appropriate positives and negatives, we theoretically analyze and identify two overlooked issues inherent in this approach: false negatives and class imbalance. To address these challenges, we propose a simple yet effective modification based on the SimCLR framework, integrating a multi-instance discrimination task to mitigate false negatives. Additionally, we introduce a graph-based interactive projection head and semantic consistency regularization, which enhances minority-class representations with minimal annotation cost. Extensive experiments on six real-world time-series datasets demonstrate that our approach consistently outperforms state-of-the-art methods, achieving up to 3.96% higher accuracy and 10.73% improvement in $F1$ -score, particularly benefiting imbalanced data scenarios.

Visual reinforcement learning has exhibited efficacy in solving control tasks characterized by high-dimensional observations. However, a central challenge persists in deriving dependable and generalizable representations from vision-based observations. Inspired by the human thought process, when the visual representation extracted from the observation can predict the future and trace history, the representation is reliable and accurate in comprehending the environmental state. Based on this concept, we introduce a B idirectional T ransition (BT) framework for representation learning. This framework employs the bidirectional prediction of both forward and backward environmental transitions as auxiliary tasks to extract reliable representations. Additionally, we introduce an inverse dynamic model to predict the actions causing environmental state transitions, thereby learning the task relevance of state representations. Our method demonstrates competitive generalization performance and sample efficiency in two settings in the DeepMind Control suite. Moreover, we utilize the robotic manipulation simulator, autonomous driving simulator CARLA, and visual navigation simulator Habitat to demonstrate the wide applicability of our method. The results indicate that BT offers more stable and reliable representations and exhibits robust generalization performance for visual reinforcement learning tasks.

Modern C++, a programming language characterized by its extensive use of object-oriented programming (OOP) features, is widely used for system programming. However, C++ compilers often struggle to correctly handle these sophisticated OOP features, resulting in numerous high-profile compiler bugs that can lead to crashes or miscompilation. Despite the significance of OOP-related bugs, existing studies largely overlook OOP features, hindering their ability to discover such bugs. To assist both compiler fuzzer designers and compiler developers, we conduct a comprehensive study of the compiler bugs caused by incorrectly handling C++ OOP-related features. First, we systematically extract 788 OOP-related C++ compiler bugs from GCC and LLVM. Second, derived from the core concepts of OOP and C++, we manually identified a two-level taxonomy of the OOP-related features leading to compiler bugs, which consists of 6 primary categories (e.g.,Abstraction & Encapsulation,Inheritance, andRuntime Polymorphism), along with 17 secondary categories (e.g.,Constructors & DestructorsandMultiple Inheritance). Third, we systematically analyze the root causes, symptoms, fixes, options, and C++ standard versions of these bugs. Our analysis yields 13 key findings, highlighting that features related to the construction and destruction of objects lead to the highest number of bugs, crashes are the most frequent symptom, and while the average time from bug introduction to discovery is 1856 days, fixing the bug once discovered takes only 174 days on average. Additionally, more than half of the bugs can be triggered without any compiler options. These findings offer valuable insights not only for developing new compiler testing approaches but also for improving language design and compiler engineering. Inspired by these findings, we developed a proof-of-concept compiler fuzzer OOPFuzz, specifically targeting OOP-related bugs in C++ compilers. We applied it against the newest release versions of GCC and LLVM. In about 3 hours, it detected 9 bugs, of which 3 have been confirmed by the developers, including a bug of LLVM that had persisted for 13 years. The results indicate our taxonomy and analysis provide valuable insights for future research in compiler testing.

Safe exploration can be regarded as a constrained Markov decision problem (CMDP) where the expected long-term cost is constrained. Previous off-policy algorithms convert the constrained optimization problem into the corresponding unconstrained dual problem by introducing the Lagrangian relaxation technique. However, the cost function of the above algorithms provides inaccurate estimations and causes the instability of the Lagrange multiplier learning. In this article, we present a novel off-policy reinforcement learning (RL) algorithm called conservative distributional maximum a posteriori policy optimization (CDMPO). At first, to accurately judge whether the current situation satisfies the constraints, CDMPO adapts distributional RL method to estimate the Q-function and C-function. Then, CDMPO uses a conservative value function loss to reduce the number of violations of constraints during the exploration process. In addition, we utilize adaptive proportional integral derivative (APID) to update the Lagrange multiplier stably. In our experiments, we select eight representative constrained tasks from two well-known safe RL benchmarks (Safety Gym and Bullet Safety Gym), providing a comprehensive evaluation of our methods across diverse scenarios. Empirical results show that the proposed method has fewer violations of constraints in the early exploration process. The final test results also illustrate that our method has better-risk control capabilities.

Recently, Light Field (LF) shows great potential in removing occlusion since the objects occluded in some views may be visible in other views. However, existing LF-based methods implicitly model each scene and can only remove objects that have positive disparities in one central views. In this article, we propose a novel Progressive Multi-Plane Images (MPI) Construction method specifically designed for LF-based occlusion removal. Different from the previous MPI construction methods, we progressively construct MPIs layer by layer in order from near to far. In order to accurately model the current layer, the positions of foreground occlusions in the nearer layers are taken as occlusion prior. Specifically, we propose an Occlusion-Aware Attention Network to generate each layer of MPIs with reliable information in occluded regions. For each layer, occlusions in the current layer are filtered out so that the background is better recovered just using the visible views instead of the other occluded views. Then, by simply removing the layers containing occlusions and rendering MPIs in kinds of viewpoints, the occlusion removal results for different views are generated. Experiments on synthetic and real-world scenes show that our method outperforms state-of-the-art LF occlusion removal methods in quantitative and visual comparisons. Moreover, we also apply the proposed progressive MPI construction method to the view synthesis task. The occlusion edges in our synthesized views achieve significantly better quality, which also verifies that our method can better model the occluded regions.

In visual Reinforcement Learning (RL), the challenge of generalization to new environments is paramount. This study pioneers a theoretical analysis of visual RL generalization, establishing an upper bound on the generalization objective, encompassing policy divergence and Bellman error components. Motivated by this analysis, we propose maintaining the cross-domain consistency for each policy in the policy space, which can reduce the divergence of the learned policy during the test. In practice, we introduce the Truncated Return Prediction (TRP) task, promoting cross-domain policy consistency by predicting truncated returns of historical trajectories. Moreover, we also propose a Transformer-based predictor for this auxiliary task. Extensive experiments on DeepMind Control Suite and Robotic Manipulation tasks demonstrate that TRP achieves state-of-the-art generalization performance. We further demonstrate that TRP outperforms previous methods in terms of sample efficiency during training.

In the domain of real-world agents, the application of Reinforcement Learning (RL) remains challenging due to the necessity for safety constraints. Previously, Constrained Reinforcement Learning (CRL) has predominantly focused on on-policy algorithms. Although these algorithms exhibit a degree of efficacy, their interactivity efficiency in real-world settings is sub-optimal, highlighting the demand for more efficient off-policy methods. However, off-policy CRL algorithms grapple with challenges in precise estimation of the C-function, particularly due to the fluctuations in the constrained Lagrange multiplier. Addressing this gap, our study focuses on the nuances of C-value estimation in off-policy CRL and introduces the Adaptive Ensemble C-learning (AEC) approach to reduce these inaccuracies. Building on state-of-the-art off-policy algorithms, we propose AEC-based CRL algorithms designed for enhanced task optimization. Extensive experiments on nine constrained robotics tasks reveal the superior interaction efficiency and performance of our algorithms in comparison to preceding methods.

Temporal knowledge graphs (TKGs) have been identified as a promising approach to represent the dynamics of facts along the timeline. The extrapolation of TKG is to predict unknowable facts happening in the future, holding significant practical value across diverse fields. Most extrapolation studies in TKGs focus on modeling global historical fact repeating and cyclic patterns, as well as local historical adjacent fact evolution patterns, showing promising performance in predicting future un-known facts. Yet, existing methods still face two major challenges: (1) They usually neglect the importance of historical information in KG snapshots related to the queries when encoding the local and global historical information; (2) They exhibit weak anti-noise capabilities, which hinders their performance when the inputs are contaminated with noise. To this end, we propose a novel Local-global history-aware Contrastive Learning model (LogCL) for TKG reasoning, which adopts contrastive learning to better guide the fusion of local and global historical information and enhance the ability to resist interference. Specifically, for the first challenge, LogCL proposes an entity-aware attention mechanism applied to the local and global historical facts encoder, which captures the key historical information related to queries. For the latter issue, LogCL designs a local-global query contrast module, effectively improving the robustness of the model. The experimental results on four benchmark datasets demonstrate that LogCL delivers better and more robust performance than the state-of-the-art baselines. The code of LogCL is available at https://eithub.com/WeiChen3690/LoeCL.

Vehicle re-identification (Re-ID) aims to identify the same vehicle images from various views in the cross-camera scenario. There are many challenges with the current vehicle Re-ID task, e.g., illumination, viewpoint, and resolution. One of the most common problems is the large difference in vehicle appearance caused by viewpoint. Vehicle Re-ID networks generally focus on discriminative areas (e.g., vehicle logo and vehicle inspections). However, the specific regions are different on different viewpoints, leading to a large intra-class variation on different viewpoints. In this paper, we propose a viewpoint-aware channel selection method (VCS) to solve this problem by exploring channels with distinguished categories on the network. Specifically, we discover the most active channels in each viewpoint. Based on features processed by these active channels, we calculate the distance of vehicle features by common active channels of two viewpoints. In addition, our method is a simple post-processing method in vehicle Re-ID, making it applicable to existing feature representation methods. Experimental results on several large vehicle Re-ID datasets illustrate the effectiveness of our method, especially on the difficult VERI-Wild dataset.

Real-world last-mile express datasets are crucial for research in logistics, supply chain management, and spatio-temporal data mining. Despite a plethora of algorithms developed to date, no widely accepted, publicly available last-mile express dataset exists to support research in this field. In this paper, we introduce LaDe, the first publicly available last-mile express dataset with millions of packages from the industry. LaDe has three unique characteristics: (1)Large-scale. It involves 10,677k packages of 21k couriers over 6 months of real-world operation. (2)Comprehensive information. It offers original package information, task-event information, as well as couriers' detailed trajecotries and road networks. (3)Diversity. The dataset includes data from various scenarios, including package pick-up and delivery, and from multiple cities, each with its unique spatio-temporal patterns due to their distinct characteristics such as populations. We verify LaDe on three tasks by running several classical baseline models per task. We believe that the large-scale, comprehensive, diverse feature of LaDe can offer unparalleled opportunities to researchers in the supply chain community, data mining community, and beyond. The dataset and code is publicly available at https://huggingface.co/datasets/Cainiao-AI/LaDe.

The application of reinforcement learning in traffic signal control (TSC) has been extensively researched and yielded notable achievements. However, most existing works for TSC assume that traffic data from all surrounding intersections is fully and continuously available through sensors. In real-world applications, this assumption often fails due to sensor malfunctions or data loss, making TSC with missing data a critical challenge. To meet the needs of practical applications, we introduce DiffLight, a novel conditional diffusion model for TSC under data-missing scenarios in the offline setting. Specifically, we integrate two essential sub-tasks, i.e., traffic data imputation and decision-making, by leveraging a Partial Rewards Conditioned Diffusion (PRCD) model to prevent missing rewards from interfering with the learning process. Meanwhile, to effectively capture the spatial-temporal dependencies among intersections, we design a Spatial-Temporal transFormer (STFormer) architecture. In addition, we propose a Diffusion Communication Mechanism (DCM) to promote better communication and control performance under data-missing scenarios. Extensive experiments on five datasets with various data-missing scenarios demonstrate that DiffLight is an effective controller to address TSC with missing data. The code of DiffLight is released at https://github.com/lokol5579/DiffLight-release.

Location-based services (LBS) have accumulated extensive human mobility data on diverse behaviors through check-in sequences. These sequences offer valuable insights into users’ intentions and preferences. Yet, existing models analyzing check-in sequences fail to consider the semantics contained in these sequences, which closely reflect human visiting intentions and travel preferences, leading to an incomplete comprehension. Drawing inspiration from the exceptional semantic understanding and contextual information processing capabilities of large language models (LLMs) across various domains, we present Mobility-LLM, a novel framework that leverages LLMs to analyze check-in sequences for multiple tasks. Since LLMs cannot directly interpret check-ins, we reprogram these sequences to help LLMs comprehensively understand the semantics of human visiting intentions and travel preferences. Specifically, we introduce a visiting intention memory network (VIMN) to capture the visiting intentions at each record, along with a shared pool of human travel preference prompts (HTPP) to guide the LLM in understanding users’ travel preferences. These components enhance the model’s ability to extract and leverage semantic information from human mobility data effectively. Extensive experiments on four benchmark datasets and three downstream tasks demonstrate that our approach significantly outperforms existing models, underscoring the effectiveness of Mobility-LLM in advancing our understanding of human mobility data within LBS contexts.

Due to the recurrent structure of RNN, the long information propagation path poses limitations in capturing long-term dependencies, gradient explosion/vanishing issues, and inefficient sequential execution. Based on this, we propose a novel paradigm called Parallel Gated Network (PGN) as the new successor to RNN. PGN directly captures information from previous time steps through the designed Historical Information Extraction (HIE) layer and leverages gated mechanisms to select and fuse it with the current time step information. This reduces the information propagation path to $\mathcal{O}(1)$, effectively addressing the limitations of RNN. To enhance PGN's performance in long-range time series forecasting tasks, we propose a novel temporal modeling framework called Temporal PGN (TPGN). TPGN incorporates two branches to comprehensively capture the semantic information of time series. One branch utilizes PGN to capture long-term periodic patterns while preserving their local characteristics. The other branch employs patches to capture short-term information and aggregate the global representation of the series. TPGN achieves a theoretical complexity of $\mathcal{O}(\sqrt{L})$, ensuring efficiency in its operations. Experimental results on five benchmark datasets demonstrate the state-of-the-art (SOTA) performance and high efficiency of TPGN, further confirming the effectiveness of PGN as the new successor to RNN in long-range time series forecasting. The code is available in this repository: https://github.com/Water2sea/TPGN.

Summary Bug repair is time consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and proliferated a family of approaches. The performance of evolutionary repair approaches is affected by two main factors: (1) search space, which defines all possible patches, and (2) search algorithms, which navigate the space. Although recent approaches have achieved remarkable progress, the main challenges of the two factors still remain. On one hand, the different kinds of search space are very coarse for containing correct patches. On the other hand, the search process guided by genetic algorithms is inefficient in finding the correct patches in an appropriate time budget. In this paper, we propose MicroRepair, a new evolutionary repair approach to address the two challenges. Rather than finding statement‐level patches like existing genetic repair approaches, MicroRepair enlarges the search space by breaking the statements into finer‐granularity ingredients that consist of AST leaves. As the search space grows exponentially, the former search algorithms may become inefficient in navigating the larger space. We utilize the best multiobjective search algorithm selected from our empirical comparison of a set of search algorithms. At last, we find redundancies search in the existing genetic process, and we further design a history‐aware search strategy to accelerate the process. We evaluated MicroRepair on 224 bugs of real‐world from the benchmark Defects4J and compared it with several state‐of‐the‐art repair approaches. The evaluation results show that MicroRepair correctly repaired 26 bugs with a precision of 62%, which significantly outperforms the state‐of‐the‐art evolutionary APR approaches in terms of precision. Moreover, the history‐aware search boosts the repair execution speed by 4% on average.

Object visual navigation aims to steer an agent toward a target object based on visual observations. It is highly desirable to reasonably perceive the environment and accurately control the agent. In the navigation task, we introduce an Agent-Centric Relation Graph (ACRG) for learning the visual representation based on the relationships in the environment. ACRG is a highly effective structure that consists of two relationships, i.e., the horizontal relationship among objects and the distance relationship between the agent and objects. On the one hand, we design the Object Horizontal Relationship Graph (OHRG) that stores the relative horizontal location among objects. On the other hand, we propose the Agent-Target Distance Relationship Graph (ATDRG) that enables the agent to perceive the distance between the target and objects. For ATDRG, we utilize image depth to obtain the target distance and imply the vertical location to capture the distance relationship among objects in the vertical direction. With the above graphs, the agent can perceive the environment and output navigation actions. Experimental results in the artificial environment AI2-THOR demonstrate that ACRG significantly outperforms other state-of-the-art methods in unseen testing environments.

The rapid growth of location-based services (LBS) has yielded massive amounts of data on human mobility. Effectively extracting meaningful representations for user-generated check-in sequences is pivotal for facilitating various downstream services. However, the user-generated check-in data are simultaneously influenced by the surrounding objective circumstances and the user's subjective intention. Specifically, the temporal uncertainty and spatial diversity exhibited in check-in data make it difficult to capture the macroscopic spatial-temporal patterns of users and to understand the semantics of user mobility activities. Furthermore, the distinct characteristics of the temporal and spatial information in check-in sequences call for an effective fusion method to incorporate these two types of information. In this paper, we propose a novel Spatial-Temporal Cross-view Contrastive Representation (STCCR) framework for check-in sequence representation learning. Specifically, STCCR addresses the above challenges by employing self-supervision from “spatial topic” and “temporal intention” views, facilitating effective fusion of spatial and temporal information at the semantic level. Besides, STCCR leverages contrastive clustering to uncover users’ shared spatial topics from diverse mobility activities, while employing angular momentum contrast to mitigate the impact of temporal uncertainty and noise. We extensively evaluate STCCR on three real-world datasets and demonstrate its superior performance across three downstream tasks.

Spatio-temporal trajectories provide valuable information about movement and travel behavior, enabling various downstream tasks that in turn power real-world applications. Learning trajectory embeddings can improve task performance but may incur high computational costs and face limited training data availability. Pre-training learns generic embeddings by means of specially constructed pretext tasks that enable learning from unlabeled data. Existing pre-training methods face (i) difficulties in learning general embeddings due to biases towards certain downstream tasks incurred by the pretext tasks, (ii) limitations in capturing both travel semantics and spatio-temporal correlations, and (iii) the complexity of long, irregularly sampled trajectories. To tackle these challenges, we propose Maximum Multi-view Trajectory Entropy Coding (MMTEC) for learning general and comprehensive trajectory embeddings. We introduce a pretext task that reduces biases in pre-trained trajectory embeddings, yielding embeddings that are useful for a wide variety of downstream tasks. We also propose an attention-based discrete encoder and a NeuralCDE-based continuous encoder that extract and represent travel behavior and continuous spatio-temporal correlations from trajectories in embeddings, respectively. Extensive experiments on two real-world datasets and three downstream tasks offer insight into the design properties of our proposal and indicate that it is capable of outperforming existing trajectory embedding methods.

Trajectory data is essential for various applications. However, publicly available trajectory datasets remain limited in scale due to privacy concerns, which hinders the development of trajectory mining and applications. Although some trajectory generation methods have been proposed to expand dataset scale, they generate trajectories in the geographical coordinate system, posing two limitations for practical applications: 1) failing to ensure that the generated trajectories are road-constrained. 2) lacking road-related information. In this paper, we propose a new problem, road network-constrained trajectory (RNTraj) generation, which can directly generate trajectories on the road network with road-related information. Specifically, RNTraj is a hybrid type of data, in which each point is represented by a discrete road segment and a continuous moving rate. To generate RNTraj, we design a diffusion model called Diff-RNTraj, which can effectively handle the hybrid RNTraj using a continuous diffusion framework by incorporating a pre-training strategy to embed hybrid RNTraj into continuous representations. During the sampling stage, a RNTraj decoder is designed to map the continuous representation generated by the diffusion model back to the hybrid RNTraj format. Furthermore, Diff-RNTraj introduces a novel loss function to enhance trajectory’s spatial validity. Extensive experiments conducted on two datasets demonstrate the effectiveness of Diff-RNTraj.

Recovering intermediate missing GPS points in a sparse trajectory, while adhering to the constraints of the road network, could offer deep insights into users' moving behaviors in intelligent transportation systems. Although recent studies have demonstrated the advantages of achieving map-constrained trajectory recovery via an end-to-end manner, they still face two significant challenges. Firstly, existing methods are mostly sequence-based models. It is extremely hard for them to comprehensively capture the micro-semantics of individual trajectory, including the information of each GPS point and the movement between two GPS points. Secondly, existing approaches ignore the impact of the macro-semantics,i.e., the road conditions and the people's shared travel preferences reflected by a group of trajectories. To address the above challenges, we propose a Micro-Macro Spatial-Temporal Graph-based Encoder-Decoder (MM-STGED). Specifically, we model each trajectory as a graph to efficiently describe the micro-semantics of trajectory and design a novel message-passing mechanism to learn trajectory representations. Additionally, we extract the macro-semantics of trajectories and further incorporate them into a well-designed graph-based decoder to guide trajectory recovery. Extensive experiments conducted on sparse trajectories with three different sampling intervals that are respectively constructed from two real-world trajectory datasets demonstrate the superiority of our proposed model.

Instant delivery services, such as food delivery and package delivery, have achieved explosive growth in recent years by providing customers with daily-life convenience. An emerging research area within these services is service Route&Time Prediction (RTP), which aims to estimate the future service route as well as the arrival time of a given worker. As one of the most crucial tasks in those service platforms, RTP stands central to enhancing user satisfaction and trimming operational expenditures on these platforms. Despite a plethora of algorithms developed to date, there is no systematic, comprehensive survey to guide researchers in this domain. To fill this gap, our work presents the first comprehensive survey that methodically categorizes recent advances in service route and time prediction. We start by defining the RTP challenge and then delve into the metrics that are often employed. Following that, we scrutinize the existing RTP methodologies, presenting a novel taxonomy of them. We categorize these methods based on three criteria: (i) type of task, subdivided into only-route prediction, only-time prediction, and joint route&time prediction; (ii) model architecture, which encompasses sequence-based and graph-based models; and (iii) learning paradigm, including Supervised Learning (SL) and Deep Reinforcement Learning (DRL). Conclusively, we highlight the limitations of current research and suggest prospective avenues. We believe that the taxonomy, progress, and prospects introduced in this paper can significantly promote the development of this field.

Given an object of interest, visual navigation aims to reach the object’s location based on a sequence of partial observations. To this end, an agent needs to (1) acquire specific knowledge about the relations of object categories in the world during training and (2) locate the target object based on the pre-learned object category relations and its trajectory in the current unseen environment. In this article, we propose a Category Relation Graph (CRG) to learn the knowledge of object category layout relations and a Temporal-Spatial-Region attention (TSR) architecture to perceive the long-term spatial-temporal dependencies of objects, aiding navigation. We establish CRG to learn prior knowledge of object layout and deduce the positions of specific objects. Subsequently, we propose the TSR architecture to capture relationships among objects in temporal, spatial, and regions within observation trajectories. Specifically, we implement a Temporal attention module (T) to model the temporal structure of the observation sequence, implicitly encoding historical moving or trajectory information. Then, a Spatial attention module (S) uncovers the spatial context of the current observation objects based on CRG and past observations. Last, a Region attention module (R) shifts the attention to the target-relevant region. Leveraging the visual representation extracted by our method, the agent accurately perceives the environment and easily learns a superior navigation policy. Experiments on AI2-THOR demonstrate that our CRG-TSR method significantly outperforms existing methods in both effectiveness and efficiency. The supplementary material includes the code and will be publicly available.

A core step of mining human mobility data is to learn accurate representations for user-generated check-in sequences. The learned representations should be able to fully describe the spatial-temporal mobility patterns of users and the high-level semantics of traveling. However, existing check-in sequence representation learning is usually implicitly achieved by end-to-end models designed for specific downstream tasks, resulting in unsatisfactory generalizable abilities and poor performance. Besides, although the sequence representation learning models that follow the contrastive learning pre-training paradigm have achieved breakthroughs in many fields like NLP, they fail to simultaneously consider the unique spatial-temporal characteristics of check-in sequences and need manual adjustments on the data augmentation strategies. So, directly applying them to check-in sequences cannot yield a meaningful pretext task. To this end, in this paper we propose a contrastive pre-training model with adversarial perturbations for check-in sequence representation learning (CACSR). Firstly, we design a novel spatial-temporal augmentation block for disturbing the spatial-temporal features of check-in sequences in the latent space to relieve the stress of designing manual data augmentation strategies. Secondly, to construct an effective contrastive pretext task, we generate “hard” positive and negative pairs for the check-in sequence by adversarial training. These two designs encourage the model to capture the high-level spatial-temporal patterns and semantics of check-in sequences while ignoring the noisy and unimportant details. We demonstrate the effectiveness and versatility of CACSR on two kinds of downstream tasks using three real-world datasets. The results show that our model outperforms both the state-of-the-art pre-training methods and the end-to-end models.

In the logistics network, accurately estimating packages' Travel Time Distribution (TTD) given the routes greatly benefits both consumers and platforms. Although recent works perform well in predicting an expected time or a time distribution in a road network, they could not be well applied to estimate TTD in logistics networks. Because TTD prediction in the logistics network requires modeling packages' multimodal TTD (MTTD, i.e., there can be more than one likely output with a given input) while leveraging the complex correlations in the logistics network. To this end, this work opens appealing research opportunities in studying MTTD learning conditioned on graph-structure data by investigating packages' travel time distribution in the logistics network. We propose a Graph-based Mixture Density Network, named GMDNet, which takes the benefits of both graph neural network and mixture density network for estimating MTTD conditioned on graph-structure data (i.e., the logistics network). Furthermore, we adopt the Expectation-Maximization (EM) framework in the training process to guarantee local convergence and thus obtain more stable results than gradient descent. Extensive experiments on two real-world datasets demonstrate the superiority of our proposed model.

Spatio-temporal graph neural networks (STGNN) have emerged as the dominant model for spatio-temporal graph (STG) forecasting. Despite their success, they fail to model intrinsic uncertainties within STG data, which cripples their practicality in downstream tasks for decision-making. To this end, this paper focuses on probabilistic STG forecasting, which is challenging due to the difficulty in modeling uncertainties and complex ST dependencies. In this study, we present the first attempt to generalize the popular de-noising diffusion probabilistic models to STGs, leading to a novel non-autoregressive framework called DiffSTG, along with the first denoising network UGnet for STG in the framework. Our approach combines the spatio-temporal learning capabilities of STGNNs with the uncertainty measurements of diffusion models. Extensive experiments validate that DiffSTG reduces the Continuous Ranked Probability Score (CRPS) by 4%-14%, and Root Mean Squared Error (RMSE) by 2%-7% over existing methods on three real-world datasets.

Sleep stage classification based on physiological time-series is essential for sleep quality evaluation and the diagnosis of sleep disorders in clinical practice. Existing machine learning studies have achieved adequate results in sleep stage classification. However, those methods neglect the significance of simultaneously capturing the interactivity and heterogeneity of physiological signals. In this paper, we propose a novel Sleep Heterogeneous Graph Neural Network (SleepHGNN) to employ these essential features. The SleepHGNN is a deep graph network consisting of Heterogeneous Graph Transformer layers, which are composed of a Heterogeneous Message Passing module for capturing the heterogeneity and a Target-Specific Aggregation module for capturing the interactivity of physiological signals. The experiments show that the SleepHGNN outperforms the state-of-the-art models on the sleep stage classification task. The source code of SleepHGNN is available at: https://github.com/zhouyh310/SleepHGNN.

In intelligent transportation systems, accurate long-term traffic forecasting is informative for administrators and travelers to make wise decisions in advance. Recently proposed spatial-temporal forecasting models perform well for short-term traffic forecasting, but two challenges hinder their applications for long-term forecasting in practice. Firstly, existing traffic forecasting models do not have satisfactory scalability on effectiveness and efficiency, i.e., as the prediction time spans extend, existing models either cannot capture the long-term spatial-temporal dynamics of traffic data or equip global receptive fields at the cost of quadratic computational complexity. Secondly, the dilemma between the models’ strong appetite for high-quality training data and their generalization ability is also a challenge we have to face. Thus how to improve data utilization efficiency deserves thoughtful thinking. Aiming at solving the long-term traffic forecasting problem and facilitating the deployment of traffic forecasting models in practice, this paper proposes an efficient and effective Self-supervised Spatial-Temporal Bottleneck Attentive Network (SSTBAN). Specifically, SSTBAN follows a multi-task framework by incorporating a self-supervised learner to produce robust latent representations for historical traffic data, so as to improve its generalization performance and robustness for forecasting. Besides, we design a spatial-temporal bottleneck attention mechanism, reducing the computational complexity meanwhile encoding global spatial-temporal dynamics. Extensive experiments on real-world long-term traffic forecasting tasks, including traffic speed forecasting and traffic flow forecasting under nine scenarios, demonstrate that SSTBAN not only achieves the overall best performance but also has good computation efficiency and data utilization efficiency.

Instant-logistics (e.g., food delivery and package pick-up) is increasingly calling for Route and Time Prediction (RTP), which aims to predict both future route and arrival time of a courier’s unvisited locations. Accurate RTP can greatly benefit the platform, such as optimizing order dispatching and improving user experience. Although recent years have witnessed various works for solving the RTP problem, they still suffer from the following three limitations: i) Failing to consider the high-level transfer mode of couriers between AOIs (Areas Of Interest, such as residential quarters or office buildings), which can help to build more accurate RTP. ii) Failing to simultaneously make the route and time prediction. Existing works either separately predict route/time or predict them in a two-step way. However, since route and time are strongly correlated (nearby locations in the route should have similar arrival times), jointly predicting them should be more effective. iii) The widely adopted tree-based or sequence-based architecture fails to fully encode the spatial relationship between different locations. To address the above limitations, we propose a multi-level and multi-task graph model, named M2G4RTP, for instant-logistics route and time joint prediction. Specifically, we propose a multi-level graph encoder equipped with a newly-designed GAT-e encoding module to capture couriers’ both high-level transfer modes between AOIs and low-level transfer modes between locations. Moreover, a multi-task decoder is presented to jointly predict the route and time at different levels. Finally, a loss weighting method based on homoscedastic uncertainty is designed to balance the two tasks adaptively. Extensive experiments on an industry-scale real-world dataset, as well as the online deployment on Cainiao Alibaba, demonstrate the superiority of our proposed model.

With the deepening of neuroscience research, data mining of brain signals is becoming an emerging topic. Among various brain signals, electroencephalography (EEG) has attracted more and more attention due to its advantages of non-invasiveness, portability, and low cost. EEG modeling and analysis play a vital role in human healthcare. Although many machine learning algorithms have been successfully applied to data mining of EEG signals, few of them achieve a win-win in classification performance and interpretability. In this paper, we propose a Bayesian graph neural network named BayesEEGNet. Considering an electrical impulse between two nodes in the brain as a Poisson process, the countless electrical impulses generated by the brain in a period are represented as an infinite number of connection probability graphs. After coupling and transforming these probability graphs, we interpret the brain’s electrical activity state as the brain’s perceptual state. Benefiting from the joint optimization of Bayesian modules and deep neural networks, our model shows superior classification performance in sleep stage classification and emotion recognition tasks. Meanwhile, our model is able to learn interpretable functional connectivity relationships between EEG channels without any prior knowledge.

Pick-up and Delivery Route Prediction (PDRP), which aims to estimate the future service route of a worker given his current task pool, has received rising attention in recent years. Deep neural networks based on supervised learning have emerged as the dominant model for the task because of their powerful ability to capture workers' behavior patterns from massive historical data. Though promising, they fail to introduce the non-differentiable test criteria into the training process, leading to a mismatch in training and test criteria. Which considerably trims down their performance when applied in practical systems. To tackle the above issue, we present the first attempt to generalize Reinforcement Learning (RL) to the route prediction task, leading to a novel RL-based framework called DRL4Route. It combines the behavior-learning abilities of previous deep learning models with the non-differentiable objective optimization ability of reinforcement learning. DRL4Route can serve as a plug-and-play component to boost the existing deep learning models. Based on the framework, we further implement a model named DRL4Route-GAE for PDRP in logistic service. It follows the actor-critic architecture which is equipped with a Generalized Advantage Estimator that can balance the bias and variance of the policy gradient estimates, thus achieving a more optimal policy. Extensive offline experiments and the online deployment show that DRL4Route-GAE improves Location Square Deviation (LSD) by 0.9%-2.7%, and Accuracy@3 (ACC@3) by 2.4%-3.2% over existing methods on the real-world dataset.

Offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets without interacting with the environment. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. Real-world data collection is often expensive and uncontrollable, leading to small and narrowly covered datasets and posing significant challenges for practical deployments of offline RL. In this paper, we provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets. Specifically, we propose a Time-reversal symmetry (T-symmetry) enforced Dynamics Model (TDM), which establishes consistency between a pair of forward and reverse latent dynamics. TDM provides both well-behaved representations for small datasets and a new reliability measure for OOD samples based on compliance with the T-symmetry. These can be readily used to construct a new offline RL algorithm (TSRL) with less conservative policy constraints and a reliable latent space data augmentation procedure. Based on extensive experiments, we find TSRL achieves great performance on small benchmark datasets with as few as 1% of the original samples, which significantly outperforms the recent offline RL algorithms in terms of data efficiency and generalizability. Code is available at: https://github.com/pcheng2/TSRL

Capturing semantic information is crucial for accurate long-range time series forecasting, which involves modeling global and local correlations, as well as discovering long- and short-term repetitive patterns. Previous works have partially addressed these issues separately, but have not been able to address all of them simultaneously. Meanwhile, their time and memory complexities are still not sufficiently low for long-range forecasting. To address the challenge of capturing different types of semantic information, we propose a novel Water-wave Information Transmission (WIT) framework. This framework captures both long- and short-term repetitive patterns through bi-granular information transmission. It also models global and local correlations by recursively fusing and selecting information using Horizontal Vertical Gated Selective Unit (HVGSU). In addition, to improve the computing efficiency, we propose a generic Recurrent Acceleration Network (RAN) which reduces the time complexity to $\mathcal{O}(\sqrt{L})$ while maintaining the memory complexity at $\mathcal{O}(L)$. Our proposed method, called Water-wave Information Transmission and Recurrent Acceleration Network (WITRAN), outperforms the state-of-the-art methods by 5.80% and 14.28% on long-range and ultra-long-range time series forecasting tasks respectively, as demonstrated by experiments on four benchmark datasets. The code is available at: https://github.com/Water2sea/WITRAN.

The autonomous driving algorithm studied in this paper makes a ground vehicle capable of sensing its environment via visual images and moving safely with little or no human input. Due to the limitation of the computing power of end side devices, the autonomous driving algorithm should adopt a lightweight model and have high performance. Conditional imitation learning has been proved an efficient and promising policy for autonomous driving and other applications on end side devices due to its high performance and offline characteristics. In driving scenarios, the images captured in different weathers have different styles, which are influenced by various interference factors, such as illumination, raindrops, etc. These interference factors bring challenges to the perception ability of deep models, thus affecting the decision-making process in autonomous driving. The first contribution of this paper is to investigate the performance gap of driving models under different weather conditions. Following the investigation, we utilise StarGAN-V2 to translate images from source domains into the target clear sunset domain. Based on the images translated by StarGAN-V2, we propose Conditional Imitation Learning with ResNet backbone named Star-CILRS. The proposed method is able to convert an image to multiple styles using only one single model, making our method easier to deploy on end side devices. Visualization results show that Star-CILRS can eliminate some environmental interference factors. Our method outperforms other methods and the success rate values in different tasks are 98%, 74%, and 22%, respectively.

Given an origin (O), a destination (D), and a departure time (T), an Origin-Destination (OD) travel time oracle~(ODT-Oracle) returns an estimate of the time it takes to travel from O to D when departing at T. ODT-Oracles serve important purposes in map-based services. To enable the construction of such oracles, we provide a travel-time estimation (TTE) solution that leverages historical trajectories to estimate time-varying travel times for OD pairs. The problem is complicated by the fact that multiple historical trajectories with different travel times may connect an OD pair, while trajectories may vary from one another. To solve the problem, it is crucial to remove outlier trajectories when doing travel time estimation for future queries. We propose a novel, two-stage framework called Diffusion-based Origin-destination Travel Time Estimation (DOT), that solves the problem. First, DOT employs a conditioned Pixelated Trajectories (PiT) denoiser that enables building a diffusion-based PiT inference process by learning correlations between OD pairs and historical trajectories. Specifically, given an OD pair and a departure time, we aim to infer a PiT. Next, DOT encompasses a Masked Vision Transformer~(MViT) that effectively and efficiently estimates a travel time based on the inferred PiT. We report on extensive experiments on two real-world datasets that offer evidence that DOT is capable of outperforming baseline methods in terms of accuracy, scalability, and explainability.

Contrastive learning has successfully addressed the scarcity of large-scale labeled datasets, especially in the physiological time series field. Existing methods construct easy positive pairs as substitutes for ground truth based on temporal dynamics or instance consistency. Despite the potential of hard positive samples to provide richer gradient information and facilitate the acquisition of more discriminative representations, they are frequently overlooked in sampling strategies, thus constraining the classification capacity of models. In this paper, we focus on multi-view physiological signals and propose a novel hard positive sampling strategy based on the view consistency. Multi-view signals are recorded from sensors attached to different organs of human body. Additionally, we propose a Multi-View Consistency Contrastive (MVCC) learning framework to jointly extract intra-view temporal dynamics and inter-view consistency features. Experiments have been carried out on two public datasets and our method demonstrates state-of-the-art performance, achieving 83.25% and 73.37% accuracy on SleepEDF and ISRUC, respectively.

In intelligent logistics systems, predicting the Estimated Time of Pick-up Arrival (ETPA) of packages is a crucial task, which aims to predict the courier’s arrival time to all the unpicked-up packages at any time. Accurate prediction of ETPA can help systems alleviate customers’ waiting anxiety and improve their experience. We identify three main challenges of this problem. First, unlike the travel time estimation problem in other fields like ride-hailing, the ETPA task is distinctively a multi-destination and path-free prediction problem. Second, an intuitive idea for solving ETPA is to predict the pick-up route and then the time in two stages. However, it is difficult to accurately and efficiently predict couriers’ future routes in the route prediction step since their behaviors are affected by multiple complex factors. Third, furthermore, in the time prediction step, the requirement for providing a courier’s all unpicked-up packages’ ETPA at once in real time makes the problem even more challenging. To tackle the preceding challenges, we propose RankETPA, which integrates the route inference into the ETPA prediction. First, a learning-based pick-up route predictor is designed to learn the route-ranking strategies of couriers from their massive spatial-temporal behaviors. Then, a spatial-temporal attention-based arrival time predictor is designed for real-time ETPA inference via capturing the spatial-temporal correlations between the unpicked-up packages. Extensive experiments on two real-world datasets and a synthetic dataset demonstrate that RankETPA achieves significant performance improvement against the baseline models.

Vehicle re-identification aims to identify vehicles from different cameras and has drawn much attention in the multimedia community. In recent years, significant achievements in vehicle re-identification have been made due to the development of neural networks and deep feature representations. However, existing deep models are regarded as “black box” methods considering the lack of explaining where they focus. With Class Activation Mapping (CAM), we can localize the discriminative image regions by utilizing convolutional feature maps. After answering the “where” question, the question of “how” to generate reasonable features appears to be substantial. In this paper, we propose the novel Spatially-Regularized Features (SRF) that can be extracted from discriminative regions in an explainable way. Specifically, we first provide an evaluation mechanism called Peak-to-Sidelobe Ratio (PSR) to measure the distribution of the convolutional feature maps. PSR outputs the strength of a matrix peak and can be used to indicate the attention intensity of a specific region. Moreover, we propose a spatially regularized loss to make the deep models focus on more reasonable and discriminative image regions. Note that no additional manual annotation data is involved in the training process, making the SRF an efficient and effective approach. Extensive subjective and objective experiments show that the proposed method significantly outperforms the state-of-the-art methods on three large-scale vehicle re-identification datasets.

Couriers’ package pick-up route prediction is a fundamental task in the emerging intelligent logistics systems. It is beneficial for order dispatching and arrival-time estimation by leveraging the predicted routes to improve those downstream tasks. However, the package pick-up route prediction problem is challenging since couriers’ behaviors are affected by both strict Spatial–Temporal Constraints (STC) and personalized spatial-transfer patterns (STP). Specifically, couriers have to consider explicitly spatial-temporal requirements such as the locations of packages and the promised pick-up time when selecting future routes. In addition, couriers have personalized mobility patterns between different locations, which are implicit patterns hidden behind couriers’ historical trajectories and cannot be ignored to precisely depict their behaviors. This paper proposes a novel framework, named CP-Route, to predict a specific courier’s future package pick-up route under strict spatial-temporal constraints, and enhance the prediction performance by learning the spatial-transfer patterns (i.e., the routing patterns) of couriers. A sophisticated encoder is designed to capture the STC and STP, and a mixed-distribution-based decoder is designed to simultaneously consider the influence of spatial-temporal constraints and routing patterns on couriers’ final decisions. Extensive experiments conducted on an industry-scale logistics dataset demonstrate the superiority of our proposed framework against the existing baseline methods. The online A/B test shows our contribution to the improvement of arrival time prediction.

Recent works demonstrate that capturing correlations between road network nodes is crucial to improving traffic flow forecasting accuracy. In general, there are spatial, temporal, and joint spatial-temporal correlations between two nodes, whose strength is related to spatial and temporal position factors. For example, traffic congestion that occurs at a traffic hub has a wider and stronger impact than that at a branch road. Moreover, the above impacts can vary with temporal position. Although spatial-temporal graph convolution networks have become a popular paradigm for modeling those correlations, there are still three problems with existing models: (i) failing to effectively model joint spatial-temporal correlations; (ii) ignoring spatial and temporal position factors when modeling the aforementioned correlations; and (iii) failing to capture distinct spatial-temporal patterns of each node. To cope with the above issues, this paper proposes a novelSpatial-TemporalPosition-awareGraphConvolutionNetwork (STPGCN) for traffic flow forecasting. Specifically, a trainable embedding module is constructed to represent the spatial and temporal positions of the nodes. Subsequently, a spatial-temporal position-aware relation inference module is proposed to adaptively infer the correlation weights of the three important spatial-temporal relations. Based on this, the generated spatial-temporal relations are integrated into a graph convolution layer for aggregating and updating node features. Finally, we design a spatial-temporal position-aware gated activation unit in the graph convolution, to capture the node-specific pattern features under the guidance of position embedding. Extensive experiments on six real-world datasets demonstrate the superiority of our model in terms of prediction performance and computational efficiency.

Target visual navigation aims at controlling the agent to find a target object based on a monocular visual RGB image in each step. It is crucial for the agent to adapt to new environments. As target visual navigation is a complex task, understanding the behavior of the agent is beneficial for analyzing the reasons for failure. This work focuses on improving the readability and success rate of navigation policies. In this paper, we propose a framework named Skill-based Hierarchical Reinforcement Learning (SHRL) for target visual navigation. SHRL contains a high-level policy and three low-level skills. The high-level policy accomplishes the task by utilizing or stopping low-level skills at each step. Low-level skills are designed to separately solve three sub-tasks, i.e.,Search, Adjustment, andExploration. In addition, we propose an Abstract Representation and two penalty items to feed robust features to the high-level policy. Abstract Representation is designed to focus on selecting low-level skills rather than the details of navigation. Experimental results in the artificial environment AI2-Thor indicate that the proposed method outperforms state-of-the-art by a large margin in unseen indoor environments. Moreover, we also provide case studies to illustrate the advantages of SHRL.

Electroencephalogram (EEG) signals for motor imagery (MI) are easily influenced by the environment and the state of the subject, which exhibit temporal and spatial variance. And this variance is more significant across subjects and sessions, which imposes limitations on the cross-domain MI tasks. To address this problem, we propose a Multi-level Spatial-Temporal Adaptation Network (MSTAN), extracting domain-invariant multi-level spatial-temporal features to overcome domain differences. First, stacked spatial-temporal graph convolution (STGCN) layers and an attention-based readout module are designed to extract spatial-temporal patterns of EEGs at multiple levels. An adaptation scheme is then introduced to narrow domain differences: 1) Individual graph parameters for the source and target domains are designed at each STGCN layer to capture the domain-specific brain region dynamic relationships; 2) The differences of spatial-temporal features between the source and target domain are reduced by minimizing the distribution distance. Experiments are conducted to evaluate the proposed method on a public dataset and the results show that our method achieves state-of-the-art performance in cross-domain motor imagery classification.

Although supervised deep learning methods achieve favorable performance in automatic sleep staging, they are limited in clinical situations due to the heavy reliance on massive labeled multi-channel polysomnogram (PSG) recordings. Accordingly, to alleviate the reliance on labeled PSG, we present SleepECL, a self-supervised learning framework based on electroencephalogram (EEG) signals taking advantage of contrastive learning, which learns efficient representations by contrasting semantically consistent and inconsistent samples (a.k.a. positive samples and negative samples). Specifically, SleepECL conducts contrastive learning upon local representations (i.e., intra-epoch EEG decoding) as well as contextual representations (i.e., interepoch dependency) and incorporates sleep expert knowledge to discover more accurate positive samples in contrastive learning, leading to more effective representations. Experimental results on two publicly available datasets demonstrate that our SleepECL outperforms state-of-the-art self-supervised methods. Moreover, the pre-trained model achieves acceptable performance using only a few label single-channel EEG recordings, which contributes to a more convenient application of automatic sleep staging in clinical situations.

Bug repair is time-consuming and tedious, which hampers software maintenance. To alleviate the burden, automated program repair (APR) is proposed and has been fruitful in the last decade. Evolutionary repair is the seminal work of this field and proliferated a family of approaches. The performance of evolutionary repair approaches is affected by two main factors: (1) search space, which defines all possible patches, and (2) search algorithms, which navigates the space. Although recent approaches have achieved remarkable progress, the main challenges of the two factors still remain. On one hand, the different kinds of search space are very coarse for containing correct patches. On the other hand, the search process guided by genetic algorithms is inefficient to find the correct patches in an appropriate time budget.

Pick-up and delivery (P&D) services such as food delivery have achieved explosive growth in recent years by providing customers with daily-life convenience. Though many service providers have invested considerably in routing tools, more and more practitioners realize that significant deviations exist between workers' actual routes and planned ones. So it is not wise to feed "optimal routes" as workers' actual service routes into downstream tasks (e.g., arrival-time prediction and order dispatching), whose performances count on the accuracy of route prediction, i.e., to predict the future service route of a worker's unfinished tasks. Therefore, to meet the rising calling for route prediction models that can capture workers' future routing behaviors, in this paper, we formulate the Pick-up and Delivery Route Prediction task (PDRP task for short) from the graph perspective for the first time, then propose a dynamic spatial-temporal graph-based model, named Graph2Route. Unlike previous sequence-based models, our model leverages the underlying graph structure and features into the encoding and decoding process. Moreover, the dynamic graph-based nature can spontaneously describe the evolving relationship between different problem instances. As a result, abundant decision context information and various spatial-temporal information of node/edge can be fully utilized in Graph2Route to improve the prediction performance. Offline experiments over two real-world industry-scale datasets under different P&D services (i.e., food delivery and package pick-up) and online A/B test demonstrate the superiority of our proposed model.

In the hybrid lenses Light Field (LF) images, a high-resolution (HR) camera is in the center of the multiple low-resolution (LR) cameras, which introduces the beneficial high-frequency information for LF super-resolution. Therefore, how to effectively utilize the high-frequency information of the central view is the key issue for the hybrid lenses LF images super-resolution. In this paper, we propose a novel learning-based framework with Layered Refinement to super-resolve the hybrid lenses LF images. Specifically, we first transform the depth information of the scene into the layered position information, and refine it by complementing the high-frequency information of the HR central view to generate a high-quality representation of the depth information. Then, guided by high-quality depth representation, we propagate the information of the HR central view to the surrounding views accurately, and utilize the layered position information to maintain the occlusion relationship during the propagation. Moreover, as the generation of each layer position information is independent in our method, our trained model can flexibly adapt kinds of scenes with various disparity ranges without additional training. Experiments show that the proposed method outperforms the SOTA methods in kinds of scenes from simulated and real-world datasets with various disparity ranges. The code is available at \urlhttps://github.com/racso10/LFHSR.

Sleep stage classification is critical for diagnosing sleep quality. While deep neural networks are becoming popular for automatic sleep stage classification with supervised learning, large-scale labeled datasets are still hard to acquire. Recently, self-supervised learning (SSL) has become one of the most prominent approaches to alleviate the burden of labeling works. However, existing SSL methods are mainly designed for non-temporally correlated data and only learning representations from an instance level. Hence, the objective of this paper is to learn robust and generalizable representations for physiological signals with self-supervised learning. Specifically, we make the following contributions: (1) we propose a novel co-training scheme by exploiting complementary information from multiple views (time view and frequency view) of physiological signals to mine more positive samples, which overcomes the drawback of popular InfoNCE and achieves semantic-level representation learning; (2) we extend our framework with a memory module, implemented by a queue and a moving-averaged encoder, to enlarge the pool of negative candidates and keep the up-to-date representation; (3) extensive experiments conducted on sleep stage classification demonstrate state-of-the-art performance compared with SSL baselines, achieving 71.6% and 57.9% accuracies on two sleep datasets, SleepEDF and ISRUC respectively. The code is publicly available athttps://github.com/larryshaw0079/CoSleep.

Over 10 billion packages are picked up every day in China. A fundamental task raised in the emerging intelligent logistics systems is the couriers’ package pick-up route prediction, which is beneficial for package dispatching, arrival-time estimation and overdue-risk evaluation, by leveraging the predicted routes to improve those downstream tasks. In the package pick-up scene, the decision-making of a courier is affected by strict spatial-temporal constraints (e.g., package location, promised pick-up time, current time, and courier’s current location). Furthermore, couriers have different decision preferences on various factors (e.g., time factor, distance factor, and balance of both), based on their own perception of the environments and work experience. In this article, we propose a novel model, named DeepRoute+, to predict couriers’ future package pick-up routes according to the couriers’ decision experience and preference learned from the historical behaviors. Specifically, DeepRoute+ consists of three layers: (1) The representation layer produces experience- and preference-aware representations for the unpicked-up packages, in which a decision preference module can dynamically adjust the importance of factors that affects the courier’s decision under the current situation. (2) The transformer encoder layer encodes the representations of packages while considering the spatial-temporal correlations among them. (3) The attention-based decoder layer uses the attention mechanism to generate the whole pick-up route recurrently. Experiments on a real-world logistics dataset demonstrate the state-of-the-art performance of our model.

Forecasting traffic inflows and outflows is crucial for intelligent transportation applications such as traffic management and risk assessment. Recently, deep learning models, which focus on capturing spatio-temporal correlations between stations (locations) by constructing Spatio-Temporal Feature Learners (STFL), have achieved promising performance in traffic inflows and outflows prediction. However, two unresolved issues limit the performance of these models. i) dynamic and heterogeneous intra- and inter-relationships between flows are ignored, and ii) the STFL in these models cannot capture the global information. To address the above issues, we propose a novel deep Spatio-Temporal Network framework based on Multi-Relational learning (MR-STN) for predicting traffic inflows and outflows. Specifically, a multi-relational learning module is designed to comprehensively model three kinds of relationships between flows while extracting diverse spatio-temporal features. In this module, an enhanced STFL is developed to capture both local and global information. Then, a feature fusion module is introduced to extract fused features for inflows and outflows respectively via a gated fusion mechanism. On this basis, the prediction module uses fusion features to generate future inflows and outflows. Finally, we implement the proposed framework with four state-of-the-art graph-based deep spatio-temporal models to demonstrate its generality and superiority. Extensive experiments on three datasets show that the proposed framework can significantly boost the performance of existing models.

Accurate traffic forecasting is critical in improving safety, stability, and efficiency of intelligent transportation systems. Despite years of studies, accurate traffic prediction still faces the following challenges, including modeling the dynamics of traffic data along both temporal and spatial dimensions, and capturing the periodicity and the spatial heterogeneity of traffic data, and the problem is more difficult for long-term forecast. In this paper, we propose an Attention based Spatial-Temporal Graph Neural Network (ASTGNN) for traffic forecasting. Specifically, in the temporal dimension, we design a novel self-attention mechanism that is capable of utilizing the local context, which is specialized for numerical sequence representation transformation. It enables our prediction model to capture the temporal dynamics of traffic data and to enjoy global receptive fields that is beneficial for long-term forecast. In the spatial dimension, we develop a dynamic graph convolution module, employing self-attention to capture the spatial correlations in a dynamic manner. Furthermore, we explicitly model the periodicity and capture the spatial heterogeneity through embedding modules. Experiments on five real-world traffic flow datasets demonstrate that ASTGNN outperforms the state-of-the-art baselines.

With the increasing accumulation of spatial-temporal trajectory data, location-based data mining has recently been extensively studied. A fundamental research topic in this field is learning the embedding vectors of locations through self-supervised pre-training. Pre-trained embedding vectors can utilize the highly available unlabeled trajectory data, and benefit downstream tasks in multiple aspects. However, most existing methods ignore the temporal information hidden in the visited time of locations in trajectories. Considering that human activities are highly regulated by specific periods of a day, temporal information can reflect some intrinsic characteristics of locations, so it is necessary to fuse them into location embedding vectors. In this paper, we propose a Time-Aware Location Embedding (TALE) pre-training method based on the CBOW framework, which is able to incorporate temporal information into the learned embedding vectors of locations. A novel temporal tree structure is designed to extract temporal information during the calculation of Hierarchical Softmax. In order to verify the effectiveness of TALE, we apply the learned embedding vectors into three downstream location-based prediction tasks, i.e., location classification, location visitor flow prediction and user next location prediction. Experiments are conducted on four real-world user trajectory datasets, and the experimental results demonstrate that our TALE model can obviously help downstream tasks gain better performance.

Dynamic networks are widely used in the social, physical, and biological sciences as a concise mathematical representation of the evolving interactions in dynamic complex systems. Measuring distances between network snapshots is important for analyzing and understanding evolution processes of dynamic systems. To the best of our knowledge, however, existing network distance measures are designed for static networks. Therefore, when measuring the distance between any two snapshots in dynamic networks, valuable context structure information existing in other snapshots is ignored. To guide the construction of context-aware distance measures, we propose a context-aware distance paradigm, which introduces context information to enrich the connotation of the general definition of network distance measures. A Context-aware Spectral Distance (CSD) is then given as an instance of the paradigm by constructing a context-aware spectral representation to replace the core component of traditional Spectral Distance (SD). In a node-aligned dynamic network, the context effectively helps CSD gain mainly advantages over SD as follows: (1) CSD is not affected by isospectral problems; (2) CSD satisfies all the requirements of a metric, while SD cannot; and (3) CSD is computationally efficient. In order to process large-scale networks, we develop a kCSD that computes top- k eigenvalues to further reduce the computational complexity of CSD. Although kCSD is a pseudo-metric, it retains most of the advantages of CSD. Experimental results in two practical applications, i.e., event detection and network clustering in dynamic networks, show that our context-aware spectral distance performs better than traditional spectral distance in terms of accuracy, stability, and computational efficiency. In addition, context-aware spectral distance outperforms other baseline methods.

Depth estimation from Light Field (LF) images is a crucial basis for LF related applications. Since multiple views with abundant information are available, how to effectively fuse features of these views is a key point for accurate LF depth estimation. In this paper, we propose a novel attention-based multi-level fusion network. Combined with the four-branch structure, we design intra-branch fusion strategy and inter-branch fusion strategy to hierarchically fuse effective features from different views. By introducing the attention mechanism, features of views with less occlusions and richer textures are selected inside and between these branches to provide more effective information for depth estimation. The depth maps are finally estimated after further aggregation. Experimental results shows the proposed method achieves state-of-the-art performance in both quantitative and qualitative evaluation, which also ranks first in the commonly used HCI 4D Light Field Benchmark.

Pre-training location embeddings from spatial-temporal trajectories is a fundamental procedure and very beneficial for user next location prediction. In the real world, a location usually has variable functionalities under different contextual environments. If the exact functions of a location in the trajectory can be reflected in its embedding, the accuracy of user next location prediction should be improved. Yet, existing location embeddings pre-trained on trajectories are mostly based on distributed word representations, which mix a location's various functionalities into one latent representation vector. To address this problem, we propose a Context and Time aware Location Embedding (CTLE) model, which calculates a location's representation vector with consideration of its specific contextual neighbors in trajectories. In this way, the multi-functional properties of locations can be properly tackled. Furthermore, in order to incorporate temporal information in trajectories into location embeddings, we propose a subtle temporal encoding module and a novel pre-training objective, which further improve the quality of location embeddings. We evaluate our proposed model on two real-world mobile user trajectory datasets. The experimental results demonstrate that, compared with the existing embedding methods, our CTLE model can pre-train higher quality location embeddings and significantly improve the performance of downstream user location prediction models.

Traffic accident forecasting is of great importance to urban public safety, emergency treatment, and construction planning. However, it is very challenging since traffic accidents are affected by multiple factors, and have multi-scale dependencies on both spatial and temporal dimensional features. Meanwhile, traffic accidents are rare events, which leads to the zero-inflated issue. Existing traffic accident forecasting methods cannot deal with all above problems simultaneously. In this paper, we propose a novel model, named GSNet, to learn the spatial-temporal correlations from geographical and semantic aspects for traffic accident risk forecasting. In the model, a Spatial-Temporal Geographical Module is designed to capture the geographical spatial-temporal correlations among regions, while a Spatial-Temporal Semantic Module is proposed to model the semantic spatial-temporal correlations among regions. In addition, a weighted loss function is designed to solve the zero-inflated issue. Extensive experiments on two real-world datasets demonstrate the superiority of GSNet against the state-of-the-art baseline methods.

Over 10 billion packages are picked up every day in China. Accurate prediction of couriers' pick-up routes can help the dispatch system to assign packages to couriers more intelligently, which is able to further increase the pick-up efficiency and reduce the overdue rate. In the package pick-up scene, the decision-making of a courier is quite complex since it's affected by strict spatial-temporal constraints (e.g., package location, promised pick-up time, current time and courier's current location). In this paper, we propose a novel model, named DeepRoute, to predict couriers' future package pick-up routes according to the couriers' decision experience learnt from their historical spatial-temporal behaviors. Specifically, DeepRoute consists of three layers: 1) The representation layer produces experience-aware representations for unpicked-up packages. 2) The transformer encoder layer encodes the representations of packages while considering the spatial-temporal correlations among them. 3) The attention-based decoder layer uses the attention mechanism to generate the whole pick-up route recurrently. Experiments on a real-world logistics dataset demonstrate the state-of-the-art performance of our DeepRoute model.

The motion coordination of multi-robots is the basis of the multi-robot system (MRS). The motion coordination problem of multi-robot satisfies the Markov property, so deep reinforcement learning (DRL) can be used to solve this problem. There are two limitations in the existing researches which apply DRL to solve this problem: the success rate of training is low and the effect of coordination is poor. We believe that in the training process, the agent should learn the collision constraint relationship from the collision. Therefore, we propose a Partially Tolerant Collision (PTC) collision handling strategy. And we believe that the completion time of motion coordination is related to the remaining distance of robots. Therefore, we propose the Dynamic Priority Strategy (DPS), which sets the priority for the robot based on the remaining distance of robots. This strategy is integrated into the reward setting of DRL. We use Path Checkerboard Diagram (PCD) as the basis for training and simulation. By experimenting with algorithms such as DQN, DDQN, and MLDDQN, the results of our proposed model are better than previous studies.

Foreground occlusion removal task aims to automatically detect and remove foreground occlusions and recover background objects. Since for Light Fields (LFs), background objects occluded in some views may be seen in other views, the foreground occlusion removal task for LFs is easy to achieve. In this paper, we propose a learning-based method combining ‘seeking’ and ‘generating’ to recover occluded background. Specifically, the micro-lens dynamic filters are proposed to ‘seek’ occluded background points in shifted micro-lens images and remove occlusions using angular information. The shifted images are then combined to further ‘generate’ background regions to supplement more background details using spatial information. By fully exploring the angular and spatial information in LFs, the dense and complex occlusions can be easily removed. Quantitative and qualitative experimental results show that our method outperforms other state-of-the-arts methods by a large margin.

Sleep staging is fundamental for sleep assessment and disease diagnosis. Although previous attempts to classify sleep stages have achieved high classification performance, several challenges remain open: 1) How to effectively extract salient waves in multimodal sleep data; 2) How to capture the multi-scale transition rules among sleep stages; 3) How to adaptively seize the key role of specific modality for sleep staging. To address these challenges, we propose SalientSleepNet, a multimodal salient wave detection network for sleep staging. Specifically, SalientSleepNet is a temporal fully convolutional network based on the $U^2$-Net architecture that is originally proposed for salient object detection in computer vision. It is mainly composed of two independent $U^2$-like streams to extract the salient features from multimodal data, respectively. Meanwhile, the multi-scale extraction module is designed to capture multi-scale transition rules among sleep stages. Besides, the multimodal attention module is proposed to adaptively capture valuable information from multimodal data for the specific sleep stage. Experiments on the two datasets demonstrate that SalientSleepNet outperforms the state-of-the-art baselines. It is worth noting that this model has the least amount of parameters compared with the existing deep neural network models.

Label distribution learning (LDL) is a framework to solve label ambiguity. To improve the performance of LDL, some existing algorithms exploit global and local label correlations. In reality, local correlations are more reasonable than global ones because few correlations can be globally adapted to all samples. The existing algorithms exploiting local correlations assume that the smaller the Euclidean distance between samples, the more likely the samples are to share the same label correlations. Specifically, they all use K-means to divide samples into several clusters, and then mine local correlations in each cluster based on this assumption. However, this assumption is incorrect in some cases and may lead to inappropriate clustering results and biased correlations. In this paper, we propose a novel LDL algorithm which mines local label correlations in self-regulating clusters independent of sample distance. In particular, we introduce clustering with learnable parameters into the model to realize that the clustering can be optimized jointly with the objective function instead of depending on the distance between samples. In this way, our proposed algorithm can mine more accurate local label correlations in these more appropriate clusters. Experimental results on 15 real-world datasets demonstrate the effectiveness of the proposed algorithm.

The research on human emotion under multimedia stimulation based on physiological signals is an emerging field and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines.

Existing light field based works utilize either views or focal stacks for saliency detection. However, since depth information exists implicitly in adjacent views or different focal slices, it is difficult to exploit scene depth information from both. By comparison, Epipolar Plane Images (EPIs) provide explicit accurate scene depth and occlusion information by projected pixel lines. Due to the fact that the depth of an object is often continuous, the distribution of occlusion edges concentrates more on object boundaries compared with traditional color edges, which is more beneficial for improving accuracy and completeness of saliency detection. In this paper, we propose a learning-based network to exploit occlusion features from EPIs and integrate high-level features from the central view for accurate salient object detection. Specifically, a novel Occlusion Extraction Module is proposed to extract occlusion boundary features from horizontal and vertical EPIs. In order to naturally combine occlusion features in EPIs and high-level features in central view, we design a concise Bi-directional Guiding Flow based on cascaded decoders. The flow leverages generated salient edge predictions and salient object predictions to refine features in mutual encoding processes. Experimental results demonstrate that our approach achieves state-of-the-art performance in both segmentation accuracy and edge clarity.

Accurate depth estimation is an essential part of most light field applications. Previous approach Spinning Parallelogram Operator (SPO) achieves robust depth estimation results in noisy and occluded scenes by fitting lines in Epipolar Plane Images (EPIs). However, only using histogram distances, SPO cannot handle complex occlusions and is sensitive to different parameters. In this letter, we propose enhanced Spinning Parallelogram Operator utilizing color constraint and histogram integration (SPO-CH) to address the above problems. The color consistency between different views is first introduced to exclude occlusions when measuring the histogram distances of occluded areas. We then design a novel Gaussian integration histogram that combines information from other adjacent bins, which improves the performances of noisy and texture-less scenes. Experimental results show that the proposed method effectively improves the depth estimation results. Comparisons with other state-of-the-art approaches show that the proposed method achieves better results for both synthetic and real-world images.

Light Field (LF) cameras are considered to have many potential applications since angular and spatial information is captured simultaneously. However, the limited spatial resolution has brought lots of difficulties in developing related applications and becomes the main bottleneck of LF cameras. In this paper, an end-to-end learning-based method is proposed to simultaneously reconstruct all view images in LFs with higher spatial resolution. Based on the epipolar geometry, view images in one LF are first grouped into several image stacks and fed into different network branches to learn sub-pixel details for each view image. Since LFs have dense sampling in angular domain, sub-pixel details in multiple spatial directions are learned from corresponding angular directions in multiple branches, respectively. Then, sub-pixel details from different directions are further integrated to generate global high-frequency residual details. Combined with the spatially upsampled LF, the final LF with high spatial resolution is obtained. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods in both visual and numerical evaluations. We also implement the proposed method on LFs with different angular resolution and experiments show that the proposed method achieves superior results than others, especially for LFs with small angular resolution. Furthermore, since the epipolar geometry is fully considered, the proposed network shows good performances in preserving the inherent epipolar property in LF images.

Spatial-temporal network data forecasting is of great importance in a huge amount of applications for traffic management and urban planning. However, the underlying complex spatial-temporal correlations and heterogeneities make this problem challenging. Existing methods usually use separate components to capture spatial and temporal correlations and ignore the heterogeneities in spatial-temporal data. In this paper, we propose a novel model, named Spatial-Temporal Synchronous Graph Convolutional Networks (STSGCN), for spatial-temporal network data forecasting. The model is able to effectively capture the complex localized spatial-temporal correlations through an elaborately designed spatial-temporal synchronous modeling mechanism. Meanwhile, multiple modules for different time periods are designed in the model to effectively capture the heterogeneities in localized spatial-temporal graphs. Extensive experiments are conducted on four real-world datasets, which demonstrates that our method achieves the state-of-the-art performance and consistently outperforms other baselines.

Motor imagery (MI), as one of the important applications of brain-computer interface (BCI), has lately received great attention. However, current MI researches have not provided satisfactory representations of electroencephalogram (EEG), taking account of the space-time-frequency features for MI classification. Moreover, those models also lack the exploration of attentive spatial, temporal, and spectral dynamics. In this study, we propose TA3D (Two-stream Attention based 3D network), a novel model for MI classification. It mainly consists of two streams: the space-time stream and the space-frequency stream, representing and learning discriminative features in the space-time-frequency dimension. Specifically, each stream contains three key parts: 1) 3D representations of EEG signals depict the spatial information over temporal/spectral distributions; 2) Attention mechanisms adaptively explore attentive dynamics of EEG signals and focus on the most valuable information in separate dimensions; 3) 3D convolutions learn spatial representation, temporal dependence, and spectral dependence. The outputs of the two streams are concatenated for space-time-frequency feature fusion. Extensive experiments implemented on two BCI datasets demonstrate that our model outperforms state-of-the-art MI classification methods.

Sleep stage classification is essential for sleep assessment and disease diagnosis. However, how to effectively utilize brain spatial features and transition information among sleep stages continues to be challenging. In particular, owing to the limited knowledge of the human brain, predefining a suitable spatial brain connection structure for sleep stage classification remains an open question. In this paper, we propose a novel deep graph neural network, named GraphSleepNet, for automatic sleep stage classification. The main advantage of the GraphSleepNet is to adaptively learn the intrinsic connection among different electroencephalogram (EEG) channels, represented by an adjacency matrix, thereby best serving the spatial-temporal graph convolution network (ST-GCN) for sleep stage classification. Meanwhile, the ST-GCN consists of graph convolutions for extracting spatial features and temporal convolutions for capturing the transition rules among sleep stages. Experiments on the Montreal Archive of Sleep Studies (MASS) dataset demonstrate that the GraphSleepNet outperforms the state-of-the-art baselines.

Multimedia stimulation of brain activities has not only become an emerging field for intensive research, but also achieves important progress in the electroencephalogram (EEG) emotion classification based on brain activities. However, how to make full use of different EEG features and the discriminative local patterns among the features for different emotions is challenging. Existing models ignore the complementarity among the spatial-spectral-temporal features and discriminative local patterns in all features, which limits the classification ability of the models to a certain extent. In this paper, we propose a novel spatial-spectral-temporal based attention 3D dense network, named SST-EmotionNet, for EEG emotion recognition. The main advantage of the SST-EmotionNet is the simultaneous integration of spatial-spectral-temporal features in a unified network framework. Meanwhile, a 3D attention mechanism is designed to adaptively explore discriminative local patterns. Extensive experiments on two real-world datasets demonstrate that the SST-EmotionNet outperforms the state-of-the-art baselines.

Forecasting the traffic flows is a critical issue for researchers and practitioners in the field of transportation. However, it is very challenging since the traffic flows usually show high nonlinearities and complex patterns. Most existing traffic flow prediction methods, lacking abilities of modeling the dynamic spatial-temporal correlations of traffic data, thus cannot yield satisfactory prediction results. In this paper, we propose a novel attention based spatial-temporal graph convolutional network (ASTGCN) model to solve traffic flow forecasting problem. ASTGCN mainly consists of three independent components to respectively model three temporal properties of traffic flows, i.e., recent, daily-periodic and weekly-periodic dependencies. More specifically, each component contains two major parts: 1) the spatial-temporal attention mechanism to effectively capture the dynamic spatialtemporal correlations in traffic data; 2) the spatial-temporal convolution which simultaneously employs graph convolutions to capture the spatial patterns and common standard convolutions to describe the temporal features. The output of the three components are weighted fused to generate the final prediction results. Experiments on two real-world datasets from the Caltrans Performance Measurement System (PeMS) demonstrate that the proposed ASTGCN model outperforms the state-of-the-art baselines.

Relation classification is one of the most important tasks in the field of information extraction, and also a key component of systems that require relational understanding of unstructured text. Existing relation classification approaches mainly rely on exploiting external resources and background knowledge to improve the performance and ignore the correlations between entity pairs which are helpful for relation classification. We present the concept of entity pair graph to represent the correlations between entity pairs and propose a novel entity pair graph based neural network (EPGNN) model, relying on graph convolutional network to capture the topological features of an entity pair graph. EPGNN combines sentence semantic features generated by pre-trained BERT model with graph topological features for relation classification. Our proposed model makes full use of a given corpus and forgoes the need of external resources and background knowledge. The experimental results on two widely used dataset: SemEval 2010 Task 8 and ACE 2005, show that our method outperforms the state-of-the-art approaches.

Event Detection (ED) is one of the most important task in the field of information extraction. The goal of ED is to find triggers in sentences and classify them into different event types. In previous works, the information of entity types are commonly utilized to benefit event detection. However, the sequential features of entity types have not been well utilized yet in the existing ED methods. In this paper, we propose a novel ED approach which learns sequential features from word sequences and entity type sequences separately, and combines these two types of sequential features with the help of a trigger-entity interaction learning module. The experimental results demonstrate that our proposed approach outperforms the state-of-the-art methods.

Light field cameras are considered to have many potential applications since angular and spatial information is captured simultaneously. However, the limited spatial resolution has brought lots of difficulties in developing related applications and becomes the main bottleneck of light field cameras. In this paper, a learning-based method using residual convolutional networks is proposed to reconstruct light fields with higher spatial resolution. The view images in one light field are first grouped into different image stacks with consistent sub-pixel offsets and fed into different network branches to implicitly learn inherent corresponding relations. The residual information in different spatial directions is then calculated from each branch and further integrated to supplement high-frequency details for the view image. Finally, a flexible solution is proposed to super-resolve entire light field images with various angular resolutions. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods by a large margin in both visual and numerical evaluations. Furthermore, the proposed method shows good performances in preserving the inherent epipolar property in light field images.

In deep reinforcement learning, experience replay has been shown an effective solution to handle sample-inefficiency. Prioritized Experience Replay (PER) uses temporal-difference error (TD error) as replay priority in Deep Q-Networks (DQN), so that agent can learn more effectively from important experiences. However, experiences with large TD error may appear near the edge of state space and these experiences do not help agent learn policy quickly. We present a novel technique called High-Value Prioritized Experience Replay (HVPER), which designs a combination of TD error and value (reward or state-action value) in replay priority. Specifically, we first propose prioritizing replay based on reward and TD error in sparse reward environment. Extendedly, we design prioritizing replay based on state-action value and TD error for more ordinary environment. We design experiments in the gym environment to evaluate the proposed HVPER. First, we verify that the combination of TD error and reward improves the training speed in two problems with sparse rewards compared to DQN algorithm and PER algorithm. In addition, HVPER accelerates the network learning and achieves a better performance in two continuous space problems compared to Deep Deterministic Policy Gradient algorithm.

Multi-robot motion coordination is a sequential decision problem that controls each robot's action through a coordination mechanism to avoid collisions and reach their respective destinations. This problem can be viewed as a Markov Decision Process. Therefore, under determinate conditions of the motion path, we make the attempt of applying deep reinforcement learning (DRL) to solve the motion coordination problems. In this paper, in order to make this problem solvable through RL methods, we design a Path Checkerboard Diagram (PCD) to represent the collision relationship among different paths and determine whether collisions occur when the agent takes action. Technically, we design the states, actions and reward function. However, the traditional Deep Q-Networks (DQN) algorithm has the problem of overestimation of state-action values and high approximation error variance of the target values. To fit the motion coordination problem scenario, we further propose a new algorithm Multi-Loss Double DQN (MLDDQN) based on the Double DQN (DDQN), which minimizes the weighted sum of loss of different target networks we have learned before. We evaluate the performance of our algorithm through the different motion coordination tasks. Results suggest that our MLDDQN method yields remarkable improvements in terms of the success rate, success steps and the stability of training process compared with the original DDQN algorithm.

Reliable traffic prediction is critical to improve safety, stability, and efficiency of intelligent transportation systems. However, traffic prediction is a very challenging problem because traffic data are a typical type of spatio-temporal data, which simultaneously shows correlation and heterogeneity both in space and time. Most existing works can only capture the partial properties of traffic data and even assume that the effect of correlation on traffic prediction is globally invariable, resulting in inadequate modeling and unsatisfactory prediction performance. In this paper, we propose a novel end-to-end deep learning model, called ST-3DNet, for traffic raster data prediction. ST-3DNet introduces 3D convolutions to automatically capture the correlations of traffic data in both spatial and temporal dimensions. A novel recalibration (Rc) block is proposed to explicitly quantify the difference of the contributions of the correlations in space. Considering two kinds of temporal properties of traffic data, i.e., local patterns and long-term patterns, ST-3DNet employs two components consisting of 3D convolutions and Rc blocks to, respectively, model the two kinds of patterns and then aggregates them together in a weighted way for the final prediction. The experiments on several real-world traffic datasets, viz., traffic congestion data and crowd flows data, demonstrate that our ST-3DNet outperforms the state-of-the-art baselines.

In this paper, we propose a new automatic sleep stage classification method based on convolutional neural networks (CNN) using single-channel electroencephalogram (EEG). The sleep stages usually consist of five stages: awake, rapid eye movement (REM), and three non-rapid eye movement stages (N1/N2/SWS). In this method, we introduced two novel ways to convert EEG time series to meaningful matrices which CNN can handle with: the dynamical time-frequency spectrum based on Hilbert-Huang transform and the temporal feature matrix, capturing the characteristics of different sleep stages. Thirtynine whole-night sleep EEG signals from Physionet database were used to evaluate the performance of our proposed method. The results show that our proposed method give good classifications for most sleep stages, especially for awake and the SWS stages. Moreover, we obtain an average accuracy of 84.5%, which outperforms other four existing methods. Different from traditional methods based on artificial feature extracting, this method are simple and more applicable to various time series.

This paper presents an efficient approach to simultaneous localization and mapping problems (SLAM) in complex office environment for a service robot without the capability of self-localization but equipped with a 2D laser radar. We propose a model of multi-layer bipartite graph of boundary segments extracted from successive data frames. To eliminate the cumulative errors caused by dead reckoning and radar itself, we introduce an error vector model defined between two successive layers and correct each data frame based on error vectors. The experimental results in a real office environment show that the approach we proposed can effectively reduce the cumulative errors during localization and mapping.

People usually travel to the same destinations and for the same purposes together with other people in groups. Inferring the travel purposes of passenger groups is a very interesting research problem in the field of passenger transport, because it can help us to better understand passengers and should bring about meaningful changes for personalized travel service and decision making of passenger carriers, organizations, and even governments. In this paper, we attempt to solve this problem by utilizing the historical travel records of passengers. To overcome the constraint of the independent and identical distribution assumption of traditional classifiers, we propose a novel iterative classification approach based on the idea of collective inference. First, we construct cotravel networks by extracting social relations between passengers from their historical travel records that are available in carriers' passenger information systems. Then, we generate a series of sophisticated features for each passenger group in the context of cotravel networks to capture the link structure information between passengers and use the overlapping relations between passenger groups to capture the probabilistic dependence relations between their labels. Finally, we collectively infer the labels of all the groups in an iterative way. Experimental results on a real data set of passenger travel records in the field of civil aviation demonstrate that our proposed iterative classification approach can efficiently infer the travel purposes of passenger groups.

This paper presents an efficient reactive approach to mobile robot motion planning in dynamically unknown environments featuring multi-shaped obstacles moving with varying velocities both in direction and in magnitude. In order to endow the mobile robot with real-time response capability, the entire motion planning duration is divided into tiny time slices during each of which a control input is determined for the mobile robot. We present a concept of a conservative collision-free state and mathematically define a collision-free state area as a feasible solution space which guarantees the safety in such a way that, as long as the center of the mobile robot is located in the solution space, a collision with any obstacles will never occur until the current time slice expires. Then, a reasonable oriented point is rapidly selected from the feasible solution space through a greedy principle based upon an equal angle-interval sampling method. In order to improve the escape capability of the mobile robot operating in a crowded environment, we present an urgency strategy by establishing a virtual repulsive force field around the mobile robot and determining the escape orientation via a potential energy function defined in the force field. Simulation results show that this reactive approach is very effective and well-suited for mobile robot motion planning in dynamically unknown environments.

Iterated local search (ILS) is a simple, effective and efficient metaheuristic, displaying strong performance on the permutation flow shop scheduling problem minimizing total flow time. Its perturbation method plays an important role in practice. However, in ILS, current methodology does not use an evaluation of the search status to adjust the perturbation strength. In this work, a method is proposed that evaluates the neighborhoods around the local optimum and adjusts the perturbation strength according to this evaluation using a technique derived from simulated-annealing. Basically, if the neighboring solutions are considerably worse than the best solution found so far, indicating that it is hard to escape from the local optimum, then the perturbation strength is likely to increase. A self-adaptive ILS named SAILS is proposed by incorporating this perturbation strategy. Experimental results on benchmark instances show that the proposed perturbation strategy is effective and SAILS performs better than three state of the art algorithms.