Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, Jincai Huang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-language models are widely used in VLN. However, most of them are trained on web-crawled generalpurpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic Domain-Aware Prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.

Multimodal sentiment analysis (MSA) detects human sentiments by understanding data from multiple modalities, such as text and images. Existing research primarily strives for an effective multimodal fusion framework to derive informative representations. However, these methods neglect the necessity of exploiting external knowledge to aid in analyzing sentiments. As a result, the lack of external commonsense embarrasses these models when the opinion cues come in an implicit and obscure manner. To address the limitation, in this paper, we propose an Auxiliary Rationale Knowledge enhanced framework, namely ARK, which improves MSA models via learning from a multimodal large language model (MLLM). Specifically, based on text-image pairs, we employ Chain-of-Thought prompting to generate image descriptions and rationales from the MLLM as auxiliary knowledge, thus enriching the original samples with commonsense knowledge encoded within the MLLM. By combining the source text with image descriptions, we are able to effectively handle MSA through a Text+Text paradigm. In this paradigm, smaller pre-trained language models (LMs) can be tasked for sentiment classification via prompt-tuning. Besides, rationales are leveraged as additional supervision to facilitate the learning of reasoning abilities by LMs. Experimental results demonstrate that our proposed method outperforms current state-of-the-art approaches across four datasets. Our data and code are available at https://github.com/ningpang/ArkMSA.

Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.

The transition from cyber-physical-system-based (CPS-based) Industry 4.0 to cyber-physical-social-system-based (CPSS-based) Industry 5.0 brings new requirements and opportunities to current sensing approaches, especially in light of recent progress in large language models (LLMs) and retrieval augmented generation (RAG). Therefore, the advancement of parallel intelligence powered crowdsensing intelligence (CSI) is witnessed, which is currently advancing toward linguistic intelligence. In this article, we propose a novel sensing paradigm, namely conversational crowdsensing, for Industry 5.0 (especially for social manufacturing). It can alleviate workload and professional requirements of individuals and promote the organization and operation of diverse workforce, thereby facilitating faster response and wider popularization of crowdsensing systems. Specifically, we design the architecture of conversational crowdsensing to effectively organize three types of participants (biological, robotic, and digital) from diverse communities. Through three levels of effective conversation (i.e., interhuman, human–AI, and inter-AI), complex interactions and service functionalities of different workers can be achieved to accomplish various tasks across three sensing phases (i.e., requesting, scheduling, and executing). Moreover, we explore the foundational technologies for realizing conversational crowdsensing, encompassing LLM-based multiagent systems, scenarios engineering and conversational human–AI cooperation. Finally, we present potential applications of conversational crowdsensing and discuss its implications. We envision that conversations in natural language will become the primary communication channel during crowdsensing process, enabling richer information exchange and cooperative problem-solving among humans, robots, and AI.

The dynamic data driven simulation (DDDS) is a simulation paradigm where the simulation is continuously influenced by fresh data sampled from the real system for better analysis and prediction of the system under study. Traditional DDDS operates in the “computation away from data” mode, which would incur long response time, high network bandwidth requirement, and critical information loss. This paper proposes a novel distributed dynamic data driven simulation paradigm. In this paradigm, each participated simulation models a portion of a system, while all the participants collectively model the whole system. Additionally, each simulation assimilates data collected locally to produce local state estimations, which are aggregated somehow to generate global state estimation. The proposed simulation paradigm is supposed to have properties such as shorter response time, lower network bandwidth requirement, and less information loss. Finally, a case is studied to illustrate the effectiveness of the novel simulation paradigm.

By encapsulating model parameters in blocks when training machine learning models collaboratively, blockchain is recognized as a promising enabling technology to facilitate reliable federated learning under a distributed and untrusted environment. However, storing updated models of each worker in blockchain induces potential privacy risks, such as membership inference attacks. Besides, the volatile network conditions in the distributed environment may cause the deterioration of system robustness. This paper aims at addressing the privacy and robustness issues mentioned above. Specifically, a Credit-based Differential Privacy stochastic model aggregation algorithm combined with SIGN operation (Cre-DPSIGN) is adopted in our peer-to-peer network, which can realize the trade-off between privacy and accuracy. Furthermore, leveraging the transparency and tamper-proofing of blockchain, we design practical and reliable smart contracts for unbiased sampling based on the credit of workers to improve system robustness against Byzantine workers. In addition, we have demonstrated that the use of biased differential privacy mechanisms can lead to performance degradation. Therefore, we have introduced two unbiased differential privacy mechanisms and have proven their convergence and privacy guarantee. Extensive experiments conducted on MNIST datasets show that our algorithm can achieve byzantine fault tolerance rate with a private loss ϵ = 0.4. Compared with the state-of-the-art, aka DP-RSA (IJCAI-22), Cre-DPSIGN shows lower privacy loss consumption and better system robustness.

Drive testing continues to play a key role in mobile network optimization for operators but its high cost is a big concern. Alternative approaches like virtual drive testing (VDT) target device testing in the lab whereas MDT or crowdsourcing based approaches are limited by the incentives users have to participate and contribute measurements. With the aim of augmenting drive testing and significantly reducing its cost, we propose GenDT, a novel deep generative model that synthesizes high-fidelity time series of key radio network key performance indicators (KPIs). The training of GenDT relies on a relatively small amount of real-world measurement data along with corresponding and easily accessible network and environment context data. Through this, GenDT learns the relationship between context and radio network KPIs as they vary over time, and therefore trained GenDT model can subsequently be relied on to generate time series for different KPIs for new drive test routes (trajectories) without having to collect field measurements. GenDT represents an initial attempt at enabling efficient drive testing via generative modeling. Evaluations with real-world mobile network drive testing measurement datasets from two countries demonstrate that GenDT can synthesize significantly more dependable data than a range of baselines. We further show that GenDT has the potential to significantly reduce the drive testing related measurement effort, and that GenDT-generated data yields similar results to that with real data in the context of two downstream use cases - QoE prediction and handover analysis.

Mobile network traffic data offers unprecedented opportunities for innovative studies within and beyond networking. However, progress is hindered by the very limited access that the research community at large has to the real-world mobile network data that is needed to develop and dependably test mobile traffic data-driven solutions. As a contribution to overcome this barrier, we propose CartaGenie, a generator of realistic mobile traffic snapshots at city scale. Taking a deep generative modeling approach and through a tailored conditional generator design, CartaGenie can synthesize high-fidelity and artifact-free spatial traffic snapshots using only contextual information about the target geographical region that is easily found in public repositories. Hence, CartaGenie allows researchers to create their own realistic datasets of spatial traffic from open data about their region of interest. Experiments with real-world mobile traffic measurements collected in multiple metropolitan areas show that CartaGenie can produce dependable network traffic loads for areas where no prior traffic information is available, significantly outperforming a comprehensive set of benchmarks. Moreover, tests with practical case studies demonstrate that the synthetic data generated by CartaGenie is as good as real data in supporting diverse research-oriented mobile traffic data-driven applications.

Service-level mobile traffic data enables research studies and innovative applications with a potential to shape future service-oriented communication systems and beyond. However, real-world datasets reporting measurements at the individual service level are hard to access as such data is deemed commercially sensitive by operators. APPSHOT is a model for generating synthetic high-fidelity city-scale snapshots of service level mobile traffic. It can operate in any geographical region and relies solely on easily available spatial context information such as population density, thus allowing the generation of new and open traffic datasets for the research community. The design of APPSHOT is informed by an original characterization of service-level mobile traffic data. APPSHOT is a novel conditional GAN design instantiated by a convolutional neural network generator and two discriminators. The model features several other innovative mechanisms including multi-channel and overlapping patch based generation to address the unique challenges involved in generating mobile service traffic snapshots. Experiments with ground-truth data collected by a major European operator in multiple metropolitan areas show that APPSHOT can produce realistic network loads at the service level for areas where it has no prior traffic knowledge, and that such data can reliably support service-oriented networking studies.

City-scale spatiotemporal mobile network traffic data can support numerous applications in and beyond networking. However, operators are very reluctant to share their data, which is curbing innovation and research reproducibility. To remedy this status quo, we propose SpectraGAN, a novel deep generative model that, upon training with real-world network traffic measurements, can produce high-fidelity synthetic mobile traffic data for new, arbitrary sized geographical regions over long periods. To this end, the model only requires publicly available context information about the target region, such as population census data. SpectraGAN is an original conditional GAN design with the defining feature of generating spectra of mobile traffic at all locations of the target region based on their contextual features. Evaluations with mobile traffic measurement datasets collected by different operators in 13 cities across two European countries demonstrate that SpectraGAN can synthesize more dependable traffic than a range of representative baselines from the literature. We also show that synthetic data generated with SpectraGAN yield similar results to that with real data when used in applications like radio access network infrastructure power savings and resource allocation, or dynamic population mapping.

Goal recognition for path planning is an important task of intention identification and situation awareness, requiring an observer to predict the goal of an evader given observations of its movements. While existing models based on planning or Markov Decision Process (MDP) show superior performance over traditional library based methods, they require much effort in model design and can hardly provide legible decision rules for their users. To make the system more user-friendly while preserving accuracy of goal inference, this paper proposes a novel self-organizing neural network based inference model, which learns compact rule sets through generalizing the streaming observations of an evader. More critically, the system manifests a high level of interpretability with the linguistic if-then rule base, making it easily comprehensible for human decision makers. We conducted extensive experiments on a large-scale real-world road network. Results show that the proposed model produces accuracy comparable to those of two state-of-the-art methods while uniquely providing legible inference rules and strong robustness against multiple goals with missing data.

Traffic congestion plagues cities around the world. Recent years have witnessed an unprecedented trend in applying reinforcement learning for traffic signal control. However, the primary challenge is to control and coordinate traffic lights in large-scale urban networks. No one has ever tested RL models on a network of more than a thousand traffic lights. In this paper, we tackle the problem of multi-intersection traffic signal control, especially for large-scale networks, based on RL techniques and transportation theories. This problem is quite difficult because there are challenges such as scalability, signal coordination, data feasibility, etc. To address these challenges, we (1) design our RL agents utilizing ‘pressure’ concept to achieve signal coordination in region-level; (2) show that implicit coordination could be achieved by individual control agents with well-crafted reward design thus reducing the dimensionality; and (3) conduct extensive experiments on multiple scenarios, including a real-world scenario with 2510 traffic lights in Manhattan, New York City 1 2.

Cooperation among the traffic signals enables vehicles to move through intersections more quickly. Conventional transportation approaches implement cooperation by pre-calculating the offsets between two intersections. Such pre-calculated offsets are not suitable for dynamic traffic environments. To enable cooperation of traffic signals, in this paper, we propose a model, CoLight, which uses graph attentional networks to facilitate communication. Specifically, for a target intersection in a network, CoLight can not only incorporate the temporal and spatial influences of neighboring intersections to the target intersection, but also build up index-free modeling of neighboring intersections. To the best of our knowledge, we are the first to use graph attentional networks in the setting of reinforcement learning for traffic signal control and to conduct experiments on the large-scale road network with hundreds of traffic signals. In experiments, we demonstrate that by learning the communication, the proposed model can achieve superior performance against the state-of-the-art methods.

Increasingly available city data and advanced learning techniques have empowered people to improve the efficiency of our city functions. Among them, improving urban transportation efficiency is one of the most prominent topics. Recent studies have proposed to use reinforcement learning (RL) for traffic signal control. Different from traditional transportation approaches which rely heavily on prior knowledge, RL can learn directly from the feedback. However, without a careful model design, existing RL methods typically take a long time to converge and the learned models may fail to adapt to new scenarios. For example, a model trained well for morning traffic may not work for the afternoon traffic because the traffic flow could be reversed, resulting in very different state representation. In this paper, we propose a novel design called FRAP, which is based on the intuitive principle of phase competition in traffic signal control: when two traffic signals conflict, priority should be given to one with larger traffic movement (i.e., higher demand). Through the phase competition modeling, our model achieves invariance to symmetrical cases such as flipping and rotation in traffic flow. By conducting comprehensive experiments, we demonstrate that our model finds better solutions than existing RL methods in the complicated all-phase selection problem, converges much faster during training, and achieves superior generalizability for different road structures and traffic conditions.

Traffic signal control is essential for transportation efficiency in road networks. It has been a challenging problem because of the complexity in traffic dynamics. Conventional transportation research suffers from the incompetency to adapt to dynamic traffic situations. Recent studies propose to use reinforcement learning (RL) to search for more efficient traffic signal plans. However, most existing RL-based studies design the key elements - reward and state - in a heuristic way. This results in highly sensitive performances and a long learning process. To avoid the heuristic design of RL elements, we propose to connect RL with recent studies in transportation research. Our method is inspired by the state-of-the-art method max pressure (MP) in the transportation field. The reward design of our method is well supported by the theory in MP, which can be proved to be maximizing the throughput of the traffic network, i.e., minimizing the overall network travel time. We also show that our concise state representation can fully support the optimization of the proposed reward function. Through comprehensive experiments, we demonstrate that our method outperforms both conventional transportation approaches and existing learning-based methods.

Goal recognition, which is the task of inferring an agent’s goals given some or all of the agent’s observed actions, is one of the important approaches in bridging the gap between the observation and decision making within an observe-orient-decide-act cycle. Unfortunately, few researches focus on how to improve the utilization of knowledge produced by a goal recognition system. In this work, we propose a Markov Decision Process-based goal recognition approach tailored to a dynamic shortest-path local network interdiction (DSPLNI) problem. We first introduce a novel DSPLNI model and its solvable dual form so as to incorporate real-time knowledge acquired from goal recognition system. Then a Markov Decision Process-based goal recognition model along with its dynamic Bayesian network representation and the applied goal inference method is proposed to identify the evader’s real goal within the DSPLNI context. Based on that, we further propose an efficient scalable technique in maintaining action utility map used in fast goal inference, and develop a flexible resource assignment mechanism in DSPLNI using knowledge from goal recognition system. Experimental results show the effectiveness and accuracy of our methods both in goal recognition and dynamic network interdiction.

In this research-in-progress paper we present a new real-world domain for studying the aggregation of different opinions: optimal urban tactical position selection (TPS). This is an important and foreseeable real world application, not only because cities have been viewed as centers of gravity by military planners throughout history, but also because the military significance of cities has increased proportionally as the global urbanization does. We first present a mapping between the domain of engineering research and that of the agent models present in the literature and use genetic multi-objective optimization method to generate Pareto TPS plans. Further we study the importance of forming diverse teams when aggregating opinions of different problem solvers for tactical position selection, and also the relationships of the number of problem solvers with time ratio and the solution efficiency. We show that a diverse team of problem solvers is able to provide better force deployment plans for early-stage decision makers to choose from. We also find that opinion aggregation methods, like approval voting, help to allocate a difficult problem solving among several computing resources, and at the same time ensuring the efficiency of solutions. Finally, we present next steps for a deeper exploration of our questions.