Structured data question answering (QA), including table QA, Knowledge Graph (KG) QA, and temporal KG QA, is a pivotal research area. Advances in large language models (LLMs) have driven significant progress in unified structural QA frameworks like TrustUQA. However, these frameworks face challenges when applied to small-scale LLMs since small-scale LLMs are prone to errors in generating structured queries. To improve the structured data QA ability of small-scale LLMs, we propose a self-correction distillation (SCD) method. In SCD, an error prompt mechanism (EPM) is designed to detect errors and provide customized error messages during inference, and a two-stage distillation strategy is designed to transfer large-scale LLMs' query-generation and error-correction capabilities to small-scale LLM. Experiments across 5 benchmarks with 3 structured data types demonstrate that our SCD achieves the best performance and superior generalization on small-scale LLM (8B) compared to other distillation methods, and closely approaches the performance of GPT4 on some datasets. Furthermore, large-scale LLMs equipped with EPM surpass the state-of-the-art results on most datasets.

Siyuan Gan, Jiaheng Liu, Boyan Wang, Tianpei Yang, Runqing Miao, Yuyao Zhang, Fanyu Meng, Junlan Feng, Linjian Meng, Jing Huo, Yang Gao. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Tao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui, Yong Ma, Yingying Gao, Chao Deng, Shilei Zhang, Junlan Feng. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Fengyuan Liu, Yi Huang, Sichun Luo, Yuqi Wang, Yazheng Yang, Xinye Li, Zefa Hu, Junlan Feng, Qi Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Yitong Wang, Xue Han, WenChun Gao, Qian Hu, Jiahui Wang, Ziqing Wang, Lijun Mei, Junlan Feng. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Existing user simulators based on prompting to role-play or SFT are generally confined to imitating users' textual utterances, without adequately considering the multi-faceted cognitive processes that underlie human decision-making during interactions.To facilitate better alignment with real human thinking patterns, we construct the LMSYS-UserThinking dataset, in which we augment 51k human-LLM conversations by reconstructing the user's inner reasoning both during and at the end of each dialogue.Furthermore, to enhance controllability and situational coherence, we introduce scenario settings that describe the global context and user goals throughout multi-turn conversations.Using this dataset, we train user simulators called ThinkingUS on different base models.We evaluate our approach from both offline and online user simulation perspectives, ultimately demonstrating its effectiveness.

The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap by presenting a comprehensive table reasoning benchmark, TReB. Firstly, we propose a taxonomy to systematically measure both shallow table understanding abilities and deep table reasoning abilities, covering a total of 26 sub-tasks. We then construct a high quality dataset through a dedicated data processing and synthesis procedure. Based on these well-constructed samples, we design an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes. Experimental results with our data and framework reveal that existing LLMs still have significant room for improvement in addressing the complex and real world table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on huggingface.co/datasets/JT-LM/JIUTIAN-TReB, and the framework on github.com/JT-LM/jiutian-treb.

Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of high-resource languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts(MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR’s effectiveness in improving expanded languages and preserving original language proficiency with superior scalability.

In a conversational system, dynamically generating follow-up questions based on context can help users explore information and provide a better user experience. Humans are usually able to ask questions that involve some general life knowledge and demonstrate higher order cognitive skills. However, the questions generated by existing methods are often limited to shallow contextual questions that are uninspiring and have a large gap to the human level. In this paper, we propose a three-stage external knowledge-enhanced follow-up question generation method, which generates questions by identifying contextual topics, constructing a knowledge graph (KG) online, and finally combining these with a large language model to generate the final question. The model generates information-rich and exploratory follow-up questions by introducing external common sense knowledge and performing a knowledge fusion operation. Experiments show that compared to baseline models, our method generates questions that are more informative and closer to human questioning levels while maintaining contextual relevance.

Cross-lingual context retrieval (extracting contextual information in one language based on requests in another) is a fundamental aspect of cross-lingual alignment, but the performance and mechanism of it for large language models (LLMs) remains unclear.In this paper, we evaluate the cross-lingual context retrieval of over 40 LLMs across 12 languages, using cross-lingual machine reading comprehension (xMRC) as a representative scenario.Our results show that post-trained open LLMs show strong cross-lingual context retrieval ability, comparable to closed-source LLMs such as GPT-4o, and their estimated oracle performances greatly improve after posttraining.Our mechanism analysis shows that the cross-lingual context retrieval process can be divided into two main phases: question encoding and answer retrieval, which are formed in pre-training and post-training respectively.The phasing stability correlates with xMRC performance, and the xMRC bottleneck lies at the last model layers in the second phase, where the effect of post-training can be evidently observed.Our results also indicate that largerscale pretraining cannot improve the xMRC performance.Instead, larger LLMs need further multilingual post-training to fully unlock their cross-lingual context retrieval potential.

Transformer has become one of the most popular architectures for multivariate time series (MTS) forecasting. However, existing Transformer-based methods still lack consideration of cross-time-and-channel dependency modeling, which is important to MTS forecasting. In addition, existing methods either completely ignore the channel dependency for robustness or model channel dependency at the sacrifice of robustness. How to achieve a balance between robustness and information capacity has not been investigated. To address these problems, a Transformer-based method, InjectTST, is proposed in this paper. Instead of designing a channel-dependent model directly, we retain the channel-independent backbone and restrainedly inject global information into individual channels. A joint temporal-channel attention scheme is proposed to better model the global information and a token dropout mechanism is designed to improve the model robustness and calculation efficiency. Through a Transformer-based injection module, the independent channels get improved by concentrating on useful global information. Experiments indicate that InjectTST can achieve stable improvement compared with state-of-the-art methods.

Self-supervised learning (SSL) has significantly advanced speaker verification, especially in scenarios with limited labeled data. This paper introduces Energy-based Confidence-Aware Distillation (EBCA-DINO), an SSL enhancement for speaker verification that integrates Energy-Based Models (EBMs) into the DINO (Distillation with No Labels) framework. EBMs use energy scores to assess data complexity and uncertainty, guiding label-free self-distillation. The adaptive temperature scaling tailors the learning process to data characteristics, allowing the teacher model to dynamically adjust the student model’s focus based on sample difficulty. This energy-aware distillation optimizes speaker verification performance. Experimental results demonstrate that EBCA-DINO improves speaker verification with relative performance gains of 4.3%, 4.9%, and 8.7% on the Vox1-O, E, and H test trials, respectively.

Using larger scale datasets in the training stage of speaker verification model usually leads to better performance. However, when the speaker number of the training dataset becomes extreme large (e.g., more than 1 million), the training speed and GPU memory demand will become bottlenecks which are mainly brought by the extreme large dimension of last fully-connected(FC) layer’s weight matrix. We propose dynamic active sub FC layers (DAS-FC) to tackle this problem. Firstly, all speakers are dynamically divided into speaker groups by clustering rows of last FC layer’s weight matrix. Then, sub FC layers are generated according to speaker groups for model training. We also introduce Mini-Batch K-means and speaker based dataloader to further reduce time and resource costing. Experiments on an extreme large dataset with 1,068,237 speakers show that compared to traditional FC layer, DAS-FC can save up to 87% training time and save 56% GPU memory occupancy with only a 4.2% drop in model performance.

Few-shot learning has made substantial progress due to the pre-trained cross-modal models, e.g., CLIP, which contrastively align the vision-language pairs. Simply aligning the class token of the image with the associated text suffers from the challenge, that the intrinsic mismatch between the vision and language features in a given pair is ignored. In this study, we propose a Holism-to-Atomism approach, called H2A, to address this mismatch challenge. During training, it first aligns the vision patches with the language tokens by determining the similarity matrix between patches and tokens. Meanwhile, the refined vision patches are scratched according to the similarity matrix, in order to represent the given training image compactly while removing the redundant patches. Then, the alignment between those refined patches and the other image patches is implemented, by calculating the patch-based prediction probability based on the image patch similarity. During inference, the queried images are classified according to the patch-based prediction probability combined with the existing class token-based one. Experiments on 11 datasets show that the H2A significantly improves upon the baseline approaches especially on 8-shot and 16-shot.

Large-scale pre-trained language models (PLMs) require significant computational resources to train from scratch on large volumes of data. But in the real world, emerging data from diverse sources may not be initially available for pre-training. Recent studies on lifelong learning have tried to solve this problem by exploring the use of model growth techniques to effectively incorporate new knowledge without the need for complete re-training. However, model growth approaches utilized have issues with growth operators that do not ensure strict function preservation or growth schedules that only include a few growth dimensions, reducing lifelong learning's effect. Furthermore, existing approaches often assume that emerging data has the same distribution as pre-training data, causing catastrophic forgetting of previously acquired knowledge. To address the aforementioned issues, we introduce LOIRE, a framework for lifelong learning that enables PLMs to effectively grow their capacity using incremental data. LOIRE employs growth operators for all feasible dimensions and a growth schedule to generate the optimal expansion sequence in the field of lifelong learning. Specifically, we present a novel plug-in layer growth operator with residual connections that skip the newly added layer during initial training while ensuring function preservation. We additionally propose an iterative distillation strategy for LOIRE that allows an intermediate model in the growth stages to switch between being a student and a teacher, reducing catastrophic forgetting during growth. Experiments show that LOIRE can reduce computational expenses by an average of 29.22\% while retaining equivalent or better downstream performance.

The recent boom of Transformer-based models have enhanced state-of-the-art results of multivariate time series (MTS) forecasting. However, MTS forecasting remains a challenging problem, primarily because of the intricate temporal patterns and obscured spatial correlations. Existing models are not only computationally expensive in modeling long-term temporal dependencies, but more importantly, fail to adequately capture the interrelations among variables. To address these problems, we propose STNet, a Spatial-Temporal Transformer Network with self-supervised pre-training scheme for MTS forecasting. In STNet, the MTS are formalized as a data-driven graph structure, which is learned through the training process to extract the latent patterns within the spatial dependencies of the data. Then we encode the structural information of the graph into the spatial Transformer encoder to help STNet better model refined spatial dependencies. A patch-level Transformer encoder is implemented to efficiently enhance locality and comprehensively capture semantic information pertaining to temporal dependencies. Moreover, the pre-training model generates rich contextual information, which consistently leads to reliable outcomes in transfer performance for downstream tasks. Extensive experiments conducted on five real-world benchmark datasets show the proposed STNet improves the prediction accuracy by 5.0%-25.2% compared to previous state-of-the-arts.

Large-scale mobile traffic prediction is critical for intelligent telecommunication networks, but remains challenging due to complex spatial-temporal patterns and network heterogeneity. Existing transfer learning-based prediction methods, tailored for grid data, inadequately handle graph-based mobile traffic data and ignore network heterogeneity. In this paper, we propose a Knowledge-enhanced Hierarchical Graph Transfer Learning (KHGTL) framework integrating graph-level and node-level knowledge transfer, for large-scale mobile traffic prediction. We also present a Spatial-Temporal Adaptive Graph Convolutional Network to capture the intricate spatial-temporal correlations. Experiments on real datasets show KHGTL reduces MAE by 44.7% compared to baselines, outperforms full supervision with 80% faster training, addressing scalability and efficiency challenges in large-scale traffic prediction.

Peng Hu, Sizhe Liu, Changjiang Gao, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Zhe Yang, Yi Huang, Yaqin Chen, XiaotingWu XiaotingWu, Junlan Feng, Chao Deng. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

This paper presents a comprehensive exploration of the phenomenon of data redundancy in video understanding, with the aim to improve computational efficiency. Our investigation commences with an examination of spatial redundancy, which refers to the observation that the most informative region in each video frame usually corresponds to a small image patch, whose shape, size and location shift smoothly across frames. Motivated by this phenomenon, we formulate the patch localization problem as a dynamic decision task, and introduce a spatially adaptive video recognition approach, termed AdaFocus. In specific, a lightweight encoder is first employed to quickly process the full video sequence, whose features are then utilized by a policy network to identify the most task-relevant regions. Subsequently, the selected patches are inferred by a high-capacity deep network for the final prediction. The complete model can be trained conveniently in an end-to-end manner. During inference, once the informative patch sequence has been generated, the bulk of computation can be executed in parallel, rendering it efficient on modern GPU devices. Furthermore, we demonstrate that AdaFocus can be easily extended by further considering the temporal and sample- wise redundancies, i.e., allocating the majority of computation to the most task-relevant video frames, and minimizing the computation spent on relatively "easier" videos. Our resulting algorithm, Uni-AdaFocus, establishes a comprehensive framework that seamlessly integrates spatial, temporal, and sample- wise dynamic computation, while it preserves the merits of AdaFocus in terms of efficient end-to-end training and hardware friendliness. In addition, Uni-AdaFocus is general and flexible as it is compatible with off-the-shelf backbone models (e.g., TSM and X3D), which can be readily deployed as our feature extractor, yielding a significantly improved computational efficiency. Empirically, extensive experiments based on seven widely-used benchmark datasets (i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2, Jester, and Kinetics-400) and three real-world application scenarios (i.e., fine-grained diving action classification, Alzheimer's and Parkinson's diseases diagnosis with brain magnetic resonance images (MRI), and violence recognition for online videos) substantiate that Uni-AdaFocus is considerably more efficient than the competitive baselines. Code and pre-trained models are available at https://github.com/blackfeather-wang/AdaFocus and https://github.com/LeapLabTHU/AdaFocusV2.

App usage prediction aims to predict the next app most likely to be used based on historical behaviors, which is beneficial for smartphone system optimization, such as system resource management, battery energy optimization, and user experience enhancement. Existing studies have treated it as a simple time series prediction problem and overlooked the sessionization characteristic of mobile app usage, i.e., neglecting the intent context in which the user interacts with apps. In this paper, we explore the context of user intents and incorporate app sessionization features into prediction models to improve prediction accuracy. Specifically, we first extract the semantic meaning of spatio-temporal contextual information of app usage by constructing an urban knowledge graph. Second, we devise a hypergraph-based embedding model to extract the hyper-relations of intra-session apps. Third, we utilize a self-attention mechanism to fuse intra-session apps’ representations and combine spatio-temporal contextual embedding to form the session representation. We further leverage a transformer for inter-session intent transition modeling to extract users’ dynamic intent (i.e., the semantic meaning of sessions) for app usage. Finally, we jointly fuse dynamic intent and recently used app features using the MLP model for the prediction. The novelty of our method is that we are the first to leverage dynamic hypergraphs for modeling sessionization features, and we model both inter-session and intra-session relations. We evaluate our model based on two real-world datasets collected in Shanghai and Nanchang. In terms of prediction accuracy, mean reciprocal rank, and normalized discounted cumulative gain, our proposed framework outperforms state-of-the-art baselines by more than 30% in the Shanghai dataset and 20% in the Nanchang dataset, respectively.

By integrating machine learning with edge devices, we can augment the capabilities of edge devices, such as IoT devices, household appliances, and wearable technologies. These edge devices generally operate on microcontrollers with inherently limited resources, such as constrained RAM capacity and limited computational power. Nonetheless, they often process data in a high-velocity stream fashion, exemplified by sequences of activities and statuses monitored by advanced industrial sensors. In practical scenarios, models must be interpretable to facilitate troubleshooting and behavior understanding. Implementing machine learning models on edge devices is valuable and challenging, striking a balance between model efficacy and resource constraint. To address this challenge, we introduce our novel Onfesk, which combines online learning algorithms with an innovative interpretable kernel. Specifically, our Onfesk trains an online classifier over the kernel's feature sketches. Benefiting from our specially designed modules, the kernel's feature sketches can be efficiently produced, and the memory requirements of the classifier can be significantly reduced. As a result, Onfesk delivers effective and efficient performance in environments with limited resources without compromising on model interpretability. Extensive experiments with diverse real-world datasets have shown that Onfesk outperforms state-of-the-art methods, achieving up to a 7.4% improvement in accuracy within identical memory constraints.

The integration of multimodal Electronic Health Records (EHR) data has significantly advanced clinical predictive capabilities. Existing models, which utilize clinical notes and multivariate time-series EHR data, often fall short of incorporating the necessary medical context for accurate clinical tasks, while previous approaches with knowledge graphs (KGs) primarily focus on structured knowledge extraction. In response, we propose EMERGE, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR predictive modeling. We extract entities from both time-series data and clinical notes by prompting Large Language Models (LLMs) and align them with professional PrimeKG, ensuring consistency. In addition to triplet relationships, we incorporate entities' definitions and descriptions for richer semantics. The extracted knowledge is then used to generate task-relevant summaries of patients' health statuses. Finally, we fuse the summary with other modalities using an adaptive multimodal fusion network with cross-attention. Extensive experiments on the MIMIC-III and MIMIC-IV datasets' in-hospital mortality and 30-day readmission tasks demonstrate the superior performance of the EMERGE framework over baseline models. Comprehensive ablation studies and analysis highlight the efficacy of each designed module and robustness to data sparsity. EMERGE contributes to refining the utilization of multimodal EHR data in healthcare, bridging the gap with nuanced medical contexts essential for informed clinical predictions. We have publicly released the code at https://github.com/yhzhu99/EMERGE.

Shimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Active learning (AL) aims to reduce labeling costs by selecting the most valuable samples to annotate from a set of unlabeled data. However, recognizing these samples is particularly challenging in multi-label text classification tasks due to the high dimensionality but sparseness of label spaces. Existing AL techniques either fail to sufficiently capture label correlations, resulting in label imbalance in the selected samples, or suffer significant computing costs when analyzing the informative potential of unlabeled samples across all labels. Facing these challenges, we propose an efficient two-stage sample acquisition strategy for multi-label active learning, called ALMuLa-mix. For saving the computational costs, ALMuLa-mix first attempts to identify unlabeled samples with novel features by employing a time-efficient feature-mixing method in conjunction with label correlations. Regarding the label imbalance, ALMuLa-mix then leverages the minority class in the labeled set to select a small batch of candidate unlabeled samples with greater inter-class diversity from the candidates with novel features. Experimental results on publicly available datasets show that ALMuLa-mix is superior to other strong baselines for handling multi-label text classification tasks.

Dialogue generation is a ambitious task that requires generating coherent and natural responses. In this paper, we propose a novel approach that utilizes a Seq2Seq generator enhanced by neural architecture search (NAS) on top of adversarial training to this end. Specifically, we apply the NAS for data native exploration, motivating the intrinsic characteristics for an adaptive structure. Further, we borrow the two-phase paradigm idea of pre-training and fine-tuning, extending the adversarial training process in terms of diversity and alignment. Experiments on the MultiWOZ-series datasets demonstrate our framework achieves superior performance in both automatic and human evaluations, and effectiveness is shown under low-resource settings. Comparative experiments indicate that the searched structure has a significantly faster convergence speed.

To improve the robustness of pre-trained language models (PLMs), previous studies have focused more on how to efficiently obtain adversarial samples with similar semantics, but less attention has been paid to the perturbed samples that change the gold label. Therefore, to fully perceive the effects of these different types of small perturbations on robustness, we propose a RObust Self-supervised leArning (ROSA) method, which incorporates different types of perturbed samples and the robustness improvements into a unified framework. Subsequently, to implement ROSA, a perturbed sample generation strategy supported by the large language models (LLMs) is proposed, which adaptively controls the generation process based on the fine-grained similarity information among the training samples. The experimental results demonstrate the remarkable performance of our ROSA.

The rapid development of Knowledge Graph (KG) technology has led to the emergence of Temporal Knowledge Graphs (TKGs), which hold significant research importance and value. Temporal Knowledge Graph Embedding (TKGE) techniques complement TKGs and predict links within them. The efficacy of TKGE hinges upon its capability to effectively model intrinsic temporal relation patterns. However, existing methodologies often need to capture temporal relation patterns adequately or establish intrinsic connections between evolving relations. We propose a highly robust TKGE framework called HouRP to address this limitation and incorporate a more comprehensive range of relational information. This framework introduces two types of Householder transformations into TKGE. Specifically, the Householder projections enable the generation of temporal-specific representations for each entity. At the same time, the relational Householder rotations facilitate high-dimensional rotations between projected entities, thereby capturing relation-specific properties. Our proposed model’s effectiveness is demonstrated through extensive experimentation on four widely used TKGs datasets.

In the rapidly evolving landscape of telecommunications, Radio Access Network (RAN) optimization is critical for maintaining high network performance and adapting to diverse service requirements. Traditionally, RAN optimization has relied heavily on manual adjustments by human experts, lacking in intelligent decision-making model. However, applying decision-making model to RAN optimization is challenged by high interaction costs and feedback delays. To address these challenges, we introduce a World Model aided Parameter Adjustment Decision and Evaluation System (WMDE), utilizing a World Model framework with offline reinforcement learning to adjust RAN parameters. WMDE, integrating the Transformer-Informed Adjustment Decision Net (TADNet) and the Causal Adjustment Effect Evaluation Net (CAENet). The WMDE system sidesteps real-time network interaction in model training with CAENet's causal estimation, cutting interaction costs. Meanwhile, TADNet employs its Transformer structure and data processing to provide a long-term, global perspective on adjustment effects, reducing feedback delay issues. Utilizing real-world operational RAN parameter adjustment data, our experiments validate the effectiveness of WMDE in decision-making for RAN parameter adjustment.

Computing set difference cardinalities is a critical task in database optimization, network management, and anomaly detection. Due to the limited computational and mem-ory resources, exactly calculating set difference cardinalities becomes impractical in real-world applications. To solve this issue, sketch methods such as Odd sketch, Tug-of-War sketch, and HyperLogLog sketch can be extended to provide approximate estimations of set difference cardinalities. They use a family of hash functions to compress all elements in a set into a compact data structure. Unfortunately, Odd sketch suffers from limited estimation range, while Tug-of-War sketch and HyperLogLog sketch unavoidably face the problems of large estimation errors and high computational costs. In this paper, we design a novel data structure of bit array GXBits to fast and accurately estimate set difference cardinalities in a large range. In GXBits, the prob-ability of each bit recording its corresponding elements follows a variant of geometric distributions and varies across different bits. We conduct extensive experiments on synthetic datasets and real-world datasets. Experimental results demonstrate that our method GXBits is more computationally and memory efficient, and significantly increases the estimation accuracy of existing methods by up to 221.3 times.

Fine-grained action recognition aims to classify complex coarse actions into finer ones. It suffers due to varying temporal scales and subtle differences between categories, which brings significant challenges to the temporal perception capabilities of current models. In this paper, we proposed a novel Temporal Cues Transformer (TCT) to guide the recognition of fine-grained action by exploiting detailed temporal cues in video duration and action sequence. The proposed TCT consists of a duration-aware encoder and a hierarchical sequence aggregation decoder. In the encoder, we extract duration-aware representations from the video and its duration. In the decoder, we reinforce the learning of action sequence by first searching the fine-grained elements of action with a hierarchical element query module, then aggregating the elements with a sequence aggregation module to predict the action category. Extensive experiments on widely used Diving48 and FineGym datasets demonstrate the superiority of the proposed method compared to the state-of-the-art methods.

Recently, increasing research interests have focused on retrieval augmented generation (RAG) to mitigate hallucination for large language models (LLMs). Following this trend, we launch the FutureDial-RAG challenge at SLT 2024, which aims at promoting the study of RAG for dialog systems. The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs containing annotations for knowledge base query and corresponding results. Over the dataset, we define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response. The baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.

The advancements in large language models (LLMs) have significantly propelled the level of artificial intelligence, further enhancing the model’s problem-solving capabilities across a variety of dialogue-oriented tasks. However, the substantial costs associated with training and inference processes for LLMs hinder their deployment across various dialogue scenarios, while small language models (SLMs) tend to perform poorly with limited samples in new settings or domains. Therefore, we propose a collaborative mechanism between LLMs and SLMs, wherein prompts are employed to bridge the gap between them. For the dialogue system, the LLM acts as a source from which SLM derives, facilitating task planning, data generating, training and optimization. Experimental results indicate that our method can significantly reduce inference overhead in new dialogue scenarios and outperforms the original pipeline architecture in terms of inference performance.

Fine-grained image recognition is a longstanding computer vision challenge that focuses on differentiating objects belonging to multiple subordinate categories within the same meta-category. Since images belonging to the same meta-category usually share similar visual appearances, mining discriminative visual cues is the key to distinguishing fine-grained categories. Although commonly used image-level data augmentation techniques have achieved great success in generic image classification problems, they are rarely applied in fine-grained scenarios, because their random editing-region behavior is prone to destroy the discriminative visual cues residing in the subtle regions. In this paper, we propose diversifying the training data at the feature-level to alleviate the discriminative region loss problem. Specifically, we produce diversified augmented samples by translating image features along semantically meaningful directions. The semantic directions are estimated with a covariance prediction network, which predicts a sample-wise covariance matrix to adapt to the large intra-class variation inherent in fine-grained images. Furthermore, the covariance prediction network is jointly optimized with the classification network in a meta-learning manner to alleviate the degenerate solution problem. Experiments on four competitive fine-grained recognition benchmarks (CUB-200-2011, Stanford Cars, FGVC Aircrafts, NABirds) demonstrate that our method significantly improves the generalization performance on several popular classification networks (e.g., ResNets, DenseNets, EfficientNets, RegNets and ViT). Combined with a recently proposed method, our semantic data augmentation approach achieves state-of-the-art performance on the CUB-200-2011 dataset. Source code is available at https://github.com/LeapLabTHU/LearnableISDA.

Understanding and accurately predicting cellular traffic data is vital for communication operators and device users, as it facilitates efficient resource allocation and ensures superior service quality. However, large-scale cellular traffic data forecasting remains challenging due to intricate temporal variations and complex spatial relationships. This article proposes a Knowledge Graph Driven Decomposition Approach (KGDA) for precise cellular traffic prediction. The KGDA breaks down the impact of static environmental factors and dynamic autocorrelations of cellular traffic time series, enabling the capture of overall traffic changes and understanding of traffic dependence on past values. Specifically, we propose an urban knowledge graph to capture the static environmental context of base stations, mapping these entities into the same latent space while retaining static environmental knowledge. The cellular traffic is divided into a regular pattern and fluctuating residual components, with the KGDA comprising four modules: a Knowledge Graph Representation Learning model, a traffic regular pattern prediction module, a traffic residual dynamic prediction module, and an attentional fusion module. The first leverages graph neural networks to extract spatial contexts and predict regular patterns, the second utilizes the Bi-directional Long Short-Term Memory (Bi-LSTM) model to capture autocorrelations of traffic time series, and the final module integrates the patterns and residuals to produce the final prediction result. Comprehensive experiments demonstrate that our proposed model outperforms state-of-the-art models by more than 10% in forecasting cellular traffic.

With the rapid development of location acquisition technologies, massive mobile trajectories have been collected and made available to us, which support a fantastic way of understanding and modeling individuals’ mobility. However, existing data-driven methods either fail to capture the long-range dependency or suffer from a high computational cost. To overcome these issues, we propose a knowledge-driven framework for mobility prediction, which leverages knowledge graphs (KG) to formulate the mobility prediction task into the KG completion problem through integrating the structured “knowledge” from the mobility data. However, most related mobility prediction works only focus on the structured information encoded in existing triples, which ignores the rich semantic information of relation paths composed of multiple relation triples. In this article, we apply a dedicated module to extract the supplementary semantic structure of paths in KG, which contributes to the interpretability and accuracy of our model. Specifically, the extracted rules are applied to capture the dependencies between relational facts. Moreover, by incorporating user information in the entity-relation space with the corresponding hyperplane, our method could capture diverse user mobility patterns and model the personal characteristics of users to improve the accuracy of mobility prediction. Extensive evaluations illustrate that our proposed model beats state-of-the-art mobility prediction algorithms, which verifies the superiority of utilizing logical rules and user hyperplanes. Our implementation code is available at https://github.com/tsinghua-fib-lab/RulekG-MobiPre.git

Multi-action dialog policy (MADP), which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing MADP models usually imitate action combinations from the labeled multi-action dialog samples. Due to data limitations, they generalize poorly toward unseen dialog flows. While reinforcement learning-based methods are proposed to incorporate the service ratings from real users and user simulators as external supervision signals, they suffer from sparse and less credible dialog-level rewards. To cope with this problem, we explore to improve MADPL with explicit and implicit turn-level user feedback received for historical predictions (i.e., logged user feedback) that are cost-efficient to collect and faithful to real-world scenarios. The task is challenging since the logged user feedback provides only partial label feedback limited to the particular historical dialog actions predicted by the agent. To fully exploit such feedback information, we propose BanditMatch, which addresses the task from a feedback-enhanced semi-supervised learning perspective with a hybrid learning objective of SSL and bandit learning. BanditMatch integrates pseudo-labeling methods to better explore the action space through constructing full label feedback. Extensive experiments show that our BanditMatch improves MADPL over the state-of-the-art methods by generating more concise and informative responses. The source code and the appendix of this paper can be obtained from https://github.com/ShuoZhangXJTU/BanditMatch.

Continual learning is crucial for dialog state tracking (DST) in dialog systems, since requirements from users for new functionalities are often encountered. However, most of existing continual learning methods for DST require task identities during testing, which is a severe limit in real-world applications. In this paper, we aim to address continual learning of DST in the class-incremental scenario (namely the task identity is unknown in testing). Inspired by the recently emerging prompt tuning method that performs well on dialog systems, we propose to use the prompt pool method, where we maintain a pool of key-value paired prompts and select prompts from the pool according to the distance between the dialog history and the prompt keys. The proposed method can automatically identify tasks and select appropriate prompts during testing. We conduct experiments on Schema-Guided Dialog dataset (SGD) and another dataset collected from a real-world dialog application. Experiment results show that the prompt pool method achieves much higher joint goal accuracy than the baseline. After combining with a rehearsal buffer, the model performance can be further improved.

With the rapid development of smart TV, TV recommendation is attracting more and more users. TV users usually distribute in multiple regions with different cultures and hence have diverse TV program preferences. From the perspective of engineering practice and performance improvement, it's very essential to model users from multiple regions with one single model. In previous work, Multi-gate Mixture-of-Expert (MMoE) has been widely adopted in multi-task and multi-domain recommendation scenarios. In practice, however, we first observe the embeddings generated by experts tend to be homogeneous which may result in high semantic similarities among embeddings that reduce the capability of Multi-gate Mixture-of-Expert (MMoE) model. Secondly, we also find there are lots of commonalities and differences between multiple regions regarding user preferences. Therefore, it's meaningful to model the complicated relationships between regions. In this paper, we first introduce contrastive learning to overcome the expert representation degeneration problem. The embeddings of two augmented samples generated by the same experts are pushed closer to enhance the alignment, and the embeddings of the same samples generated by different experts are pushed away in vector space to improve uniformity. Then we propose a Graph-based Gating Mechanism to empower typical Multi-gate Mixture-of-Experts. Graph-based MMoE is able to recognize the commonalities and differences among multiple regions by introducing a Graph Neural Network (GNN) with region similarity prior. We name our model Multi-gate Mixture-of-Contrastive-Experts model with Graph-based Gating Mechanism (MMoCEG). Extensive offline experiments and online A/B tests on a commercial TV service provider over 100 million users and 2.3 million items demonstrate the efficacy of MMoCEG compared to the existing models.

The optimization and management of transportation play an important role in the Intelligent Transportation System (ITS), and the prediction of traffic flow data is the basis of it. Recently, graph convolution network (GCN) is widely applied to extract the features of non-Euclidean data, some models usually use the predefined distance-based adjacency matrix for GCN to extract node features. However, the distance graph can only capture part of the spatial correlation and limit the effective learning of those models. In fact, there are close similarities between the remote intersection with similar time sequence, they may be living quarters or industrial zones in the city. Besides, some previous models capture incomplete temporal correlation, which will affect the long-term prediction task to a certain extent. To address these limitations, we propose a novel Spatial-Temporal Residual Multi-Graph Convolution Network (ST-RMGCN) for traffic flow forecasting. Specifically, our model utilizes SimHash to calculate the time sequence similarity between nodes and build a semantic graph which is be instrumental in capturing remote spatial correlation. Moreover, multi-graph convolution network also extracts the features of adjacent nodes through distance graph. In addition, we design a novel time embedding method which contains the exact time information of the time step, and it contributes to the attention mechanism and Gate Recurrent Unit (GRU) to capture the local and global time information. Extensive experiments on six real-world datasets show that our proposed model performs better than the state-of-the-art methods.

Fine-grained address entity recognition (FGAER) from multi-turn spoken dialogues is particularly challenging.The major reason lies in that a full address is often formed through a conversation process.Different parts of an address are distributed through multiple turns of a dialogue with spoken noises.It is nontrivial to extract by turn and combine them.This challenge has not been well emphasized by main-stream entity extraction algorithms.To address this issue, we propose in this paper a logic-guided fine-grained address recognition method (Log-FGAER), where we formulate the address hierarchy relationship as the logic rule and softly apply it in a probabilistic manner to improve the accuracy of FGAER.In addition, we provide an ontology-based data augmentation methodology that employs ChatGPT to augment a spoken dialogue dataset with labeled address entities.Experiments are conducted using datasets generated by the proposed data augmentation technique and derived from real-world scenarios.The results of the experiment demonstrate the efficacy of our proposal.Where are you in Suzhou city, Jiangsu Prov? 您现在在江苏 苏州市 的哪里？ I'm in Wujiang District, Shengze Town in Wujiang District.

Accurately predicting base station traffic volumes and understanding mobile traffic patterns is essential for smart city development, enabling efficient resource allocation and ensuring high-quality communication services. However, existing works have limitations in capturing spatial information, though the surrounding environment plays a critical role in mobile traffic prediction. In this paper, we utilize a spatial knowledge graph to represent spatial information and add important urban components to augment it making it a more effective tool for capturing environmental information. we further propose a multi-relational knowledge graph convolutional network model for mobile traffic prediction, which consists of three parts. The environmental context modelling captures spatial information from the augmented spatial knowledge graph using tucker decomposition and relational graph convolutional network. The semantic relationship modelling extracts semantic relationships between base stations and employs transformer and causal convolution to capture temporal features. The inter-attentional fusion modelling utilizes the self-attention mechanism to further capture base station relationships and predict future traffic volumes. Extensive experiments demonstrate that our proposed model significantly outperforms the state-of-the-art models by over 10% in mobile traffic prediction. The code is available at https://github.com/tsinghua-fiblab/Mobile-Traffic-Prediction-sigspatial23

We tend to assume most available speech corpora we use are either completely clean or completely noised. However, the reality is most of them are a mix of both. In this paper, we propose a semi-supervised speech enhancement framework to enhance such typical speech datasets. This framework includes an estimator to measure the speech purity. Utterances with high speech purity are considered clean, otherwise noised. For clean speech utterances, we follow the supervised learning mechanism to train a deep learning speech enhancement model. For noised speech, we update the model in an unsupervised manner. Hence, we design our training loss as a combination of the supervised loss and unsupervised loss. We refer to this framework as SemiEnhance. Experimental results show that SemiEnhance substantially improves the speech quality, and achieves new state-of-the-art results on benchmark datasets: 2022 DNS Challenge and NoiseX-92.

Previous contrastive learning methods for sentence representations often focus on insensitive transformations to produce positive pairs, but neglect the role of sensitive transformations that are harmful to semantic representations. Therefore, we propose an Equivariant Self-Contrastive Learning (ESCL) method to make full use of sensitive transformations, which encourages the learned representations to be sensitive to certain types of transformations with an additional equivariant learning task. Meanwhile, in order to improve practicability and generality, ESCL simplifies the implementations of traditional equivariant contrastive methods to share model parameters from the perspective of multi-task learning. We evaluate our ESCL on semantic textual similarity tasks. The proposed method achieves better results while using fewer learning parameters compared to previous methods.

The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves a 2.79% false rejection rate and a 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.

Session-based recommendation (SBR) aims to predict the user’s action at the next timestamp according to an anonymous yet short interaction sequence (i.e., session). Almost all the existing SBR solutions for user preference are only based on the current session without exploiting the high-order relations among other sessions, which may restrict the SBR representation ability and even deteriorate the performance. To this end, we propose a Hyper-relation alignment hyperGraph Convolutional Network, called Hyra-GCN, for better inferring the user preference of the current session. Specifically, we first model session-based data as a hyper-graph capable of representing high-order relationships to exploit item transitions over sessions in a more subtle manner. Subsequently, we explore self-supervised learning on item-session hypergraphs, so as to alleviate the problem of data sparsity. Experimental results on real-world datasets demonstrate the effectiveness of our proposed Hyra-GCN against state-of-the-art baselines.

Due to the growth of densification and multi-band services in emerging cellular networks, the next generation NodeB will confront with a plethora of handover-related Configuration and Optimization Parameters (COPs). Adjusting the handover-related COPs will directly impact the networks Key Performance Indicators (KPIs). Thus, accurately estimating the effect of the handover-related COPs adjustment on KPI values and modeling the COP-KPI relationship can provide guidelines for intelligent network management. Existing studies have applied machine learning to model the intricate relationship between COP and KPI, but these methods merely exploit their correlations. In this paper, considering that there exist the causal effects between COPs and KPIs that go beyond the correlation analysis of them, we propose the causal inference aided handover parameter adjusting effect estimation and design the Causal-Stable Decomposition Network (CSDnet). Specifically, we propose the causal-stable deconstructor module based on self-attention mechanism to characterize the complex causality-stability of KPI induced by the COP adjustment. To ensure the training, we propose the gradient matching loss and the consistency constraint loss. Then, we design the stable decoder and the causal decoder that incorporate the causal information into COP-KPI modeling. Extensive experiments on the COP-KPI dataset collected from the real-world cellular networks verify the effectiveness of CSDnet.

Cellular traffic prediction is an indispensable part for intelligent telecommunication networks. Nevertheless, due to the frequent user mobility and complex network scheduling mechanisms, cellular traffic often inherits complicated spatial-temporal patterns, making the prediction incredibly challenging. Although recent advanced algorithms such as graph-based prediction approaches have been proposed, they frequently model spatial dependencies based on static or dynamic graphs and neglect the coexisting multiple spatial correlations induced by traffic generation. Meanwhile, some works lack the consideration of the diverse cellular traffic patterns, result in suboptimal prediction results. In this paper, we propose a novel deep learning network architecture, Adaptive Hybrid Spatial-Temporal Graph Neural Network (AHSTGNN), to tackle the cellular traffic prediction problem. First, we apply adaptive hybrid graph learning to learn the compound spatial correlations among cell towers. Second, we implement a Temporal Convolution Module with multi-periodic temporal data input to capture the nonlinear temporal dependencies. In addition, we introduce an extra Spatial-Temporal Adaptive Module to conquer the heterogeneity lying in cell towers. Our experiments on two real-world cellular traffic datasets show AHSTGNN outperforms the state-of-the-art by a significant margin, illustrating the superior scalability of our method for spatial-temporal cellular traffic prediction.

Early exiting has become a promising approach to improving the inference efficiency of deep networks. By structuring models with multiple classifiers (exits), predictions for "easy" samples can be generated at earlier exits, negating the need for executing deeper layers. Current multi-exit networks typically implement linear classifiers at intermediate layers, compelling low-level features to encapsulate high-level semantics. This sub-optimal design invariably undermines the performance of later exits. In this paper, we propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task with a novel dual-branch architecture. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Bi-directional cross-attention layers are established to progressively fuse the information of both branches. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features. Dyn-Perceiver constitutes a versatile and adaptable framework that can be built upon various architectures. Experiments on image classification, action recognition, and object detection demonstrate that our method significantly improves the inference efficiency of different backbones, outperforming numerous competitive approaches across a broad range of computational budgets. Evaluation on both CPU and GPU platforms substantiate the superior practical efficiency of Dyn-Perceiver. Code is available at https://www.github.com/LeapLabTHU/Dynamic_Perceiver.

Item representation learning (IRL) plays an essential role in recommender systems, especially for sequential recommendation. Traditional sequential recommendation models usually utilize ID embeddings to represent items, which are not shared across different domains and lack the transferable ability. Recent studies use pre-trained language models (PLM) for item text embeddings (text-based IRL) that are universally applicable across domains. However, the existing text-based IRL is unaware of the important collaborative filtering (CF) information. In this paper, we propose CoWPiRec, an approach of Collaborative Word-based Pre-trained item representation for Recommendation. To effectively incorporate CF information into text-based IRL, we convert the item-level interaction data to a word graph containing word-level collaborations. Subsequently, we design a novel pre-training task to align the word-level semantic-and CF-related item representation. Extensive experimental results on multiple public datasets demonstrate that compared to state-of-the-art transferable sequential recommenders, CoWPiRec achieves significantly better performances in both fine-tuning and zero-shot settings for cross-scenario recommendation and effectively alleviates the cold-start issue. The code is available at: https://github.com/ysh-1998/CoWPiRec.

Traffic forecasting has numerous application scenarios, but it is also a challenging task considering complex spatial and temporal correlations. Deep learning has shown great potential in recent years. Graph convolutional networks, recurrent neural networks and other deep learning architectures have been used in traffic forecasting. However, most existing methods capture spatial and temporal correlations sequentially or synchronously and couple spatial and temporal representations, which may introduce spurious dependencies. The heterogeneity of spatio-temporal data also increases the difficulty of modeling and makes it harder to obtain accurate predictions. In this paper, we propose a novel multi-branch spatial-temporal decoupling neural network (MBSTDN) for traffic forecasting. Specifically, we independently map hidden features into spatial and temporal representations and process them through corresponding branches to produce partial predictions. The partial predictions are then aggregated into final predictions by residual links. At the same time, we reconstruct input signals through a separate branch, introducing layer-by-layer decomposition based on residual learning to deal with the intricate dependencies. Furthermore, we design an embedding layer utilizing multi-dimensional information to reduce the influence of spatio-temporal heterogeneity on predictions. Experiments on four public traffic datasets demonstrate the effectiveness of MBSTDN which performs better than the state-of-the-art baselines.

With the rapid development of the cellular network, network planning is increasingly important. Generating large-scale urban cellular traffic contributes to network planning via simulating the behaviors of the planned network. Existing methods fail in simulating the long-term temporal behaviors of cellular traffic while cannot model the influences of the urban environment on the cellular networks. We propose a knowledge-enhanced GAN with multi-periodic patterns to generate large-scale cellular traffic based on the urban environment. First, we design a GAN model to simulate the multi-periodic patterns and long-term aperiodic temporal dynamics of cellular traffic via learning the daily patterns, weekly patterns, and residual traffic between long-term traffic and periodic patterns step by step. Then, we leverage urban knowledge to enhance traffic generation via constructing a knowledge graph containing multiple factors affecting cellular traffic in the surrounding urban environment. Finally, we evaluate our model on a real cellular traffic dataset. Our proposed model outperforms three state-of-art generation models by over 32.77%, and the urban knowledge enhancement improves the performance of our model by 4.71%. Moreover, our model achieves good generalization and robustness in generating traffic for urban cellular networks without training data in the surrounding areas.

Calibration in recommender systems ensures that the user’s interests distribution over groups of items is reflected with their corresponding proportions in the recommendation, which has gained increasing attention recently. For example, a user who watched 80 entertainment videos and 20 knowledge videos is expected to receive recommendations comprising about 80% entertainment and 20% knowledge videos as well. However, with the increasing calls for responsible recommendation, it has become inadequate to just match users’ historical behaviors especially when items are grouped by their qualities, which could result in undesired effects at the system level (e.g., overwhelming clickbaits). In this paper, we envision the two-sided calibration task that not only matches the users’ past interests distribution (user-level calibration) but also guarantees an overall target exposure distribution of different item groups (system-level calibration). The target group exposure distribution can be explicitly pursued by users, platform owners, and even the law (e.g., the platform owners expect about 50% knowledge video recommendation on the whole). To support this scenario, we propose a post-processing method named PCT. PCT first solves personalized calibration targets that minimize the changes in users’ historical interest distributions while ensuring the overall target group exposure distribution. Then, PCT reranks the original recommendation lists according to personalized calibration targets to generate both relevant and two-sided calibrated recommendations. Extensive experiments demonstrate the superior performance of the proposed method compared to calibrated and fairness-aware recommendation approaches.

This paper is concerned with TV recommendation, where one major challenge is the coupling behavior issue that the behaviors of multiple users are coupled together and not directly distinguishable because the users share the same account. Unable to identify the current watching user and use the coupling behaviors directly could lead to sub-optimal recommendation results due to the noise introduced by the behaviors of other users. Most existing methods deal with this issue either by unsupervised clustering algorithms or depending on latent user representation learning with strong assumptions. However, they neglect to sophisticatedly model the current session behaviors, which carry the information of user identification. Another critical limitation of the existing models is the lack of supervision signal on distinguishing behaviors because they solely depend on the final click label, which is insufficient to provide effective supervision. To address the above problems, we propose the Coupling Sequence Model (COSMO) for TV recommendation. In COSMO, we design a session-aware co-attention mechanism that uses both the candidate item and session behaviors as the query to attend to the historical behaviors in a fine-grained manner. Furthermore, we propose to use the data of accounts with multiple devices (e.g., families with various TV sets), which means the behaviors of one account are generated on different devices. We regard the device information as weak supervision and propose a novel pair-wise attention loss for learning to distinguish the coupling behaviors. Extensive offline experiments and online A/B tests over a commercial TV service provider demonstrate the efficacy of COSMO compared to the existing models.

Recently, two approaches, fine-tuning large pre-trained language models and variational training, have attracted significant interests, separately, for semi-supervised end-to-end task-oriented dialog (TOD) systems. In this paper, we propose Variational Latent-State GPT model (VLS-GPT), which is the first to combine the strengths of the two approaches. Among many options of models, we propose the generative model and the inference model for variational learning of the end-to-end TOD system, both as auto-regressive language models based on GPT-2, which can be further trained over a mix of labeled and unlabeled dialog data in a semi-supervised manner. Variational training of VLS-GPT is both statistically and computationally more challenging than previous variational learning works for sequential latent variable models, which use turn-level first-order Markovian. The inference model in VLS-GPT is non-Markovian due to the use of the Transformer architecture. In this work, we establish Recursive Monte Carlo Approximation (RMCA) to the variational objective with non-Markovian inference model and prove its unbiasedness. Further, we develop the computational strategy of sampling-then-forward-computation to realize RMCA, which successfully overcomes the memory explosion issue of using GPT in variational learning and speeds up training. Semi-supervised TOD experiments are conducted on two benchmark multi-domain datasets of different languages - MultiWOZ2.1 and CrossWOZ. VLS-GPT is shown to significantly outperform both supervised-only and semi-supervised self-training baselines.

To further improve the quality of the enhanced speech, it is appealing that more profound articulatory and auditory knowledge should be introduced into the speech enhancement model. Among these, harmonics seriously affect speech timbre and play a crucial role in speech intelligibility. Especially in the frequency domain, harmonics appear as the local maximum peaks of energy, which could be expected to serve as anchors to recover the distorted speech. In this paper, an explicit modeling method, harmonic attention, is presented, patching the harmonics with the help of residual ones. In order to maintain the spectral structure of speech during the processing and to enable the network to support harmonic modeling, a harmonic attention-based progressive enhancement network (HAPNet) is applied, which gradually approaches clean speech with stacked modules of harmonic attention. In addition, to make enhanced speech more consistent with hearing, a loss function based on the loudness power compression (LC-SNR) is used, which measures both magnitude and phase values with appropriate auditory effects. The experimental visualization indicates that the harmonic attention can capture and recover the harmonics of speech. And the objective evaluations show that the presented HAPNet and LC-SNR outperform the referenced methods. Furthermore, the presented model trained on 100 hours of data achieves competitive results with the referenced models trained on 3000+ hours of data, and one trained on 500 hours of data yields the state-of-the-art performance.

Topology impacts important network performance metrics, including link utilization, throughput and latency, and is of central importance to network operators. However, due to the combinatorial nature of network topology, it is extremely difficult to obtain an optimal solution, especially since topology planning in networks also often comes with management-specific constraints. As a result, local optimization with hand-tuned heuristic methods from human experts is often adopted in practice. Yet, heuristic methods cannot cover the global topology design space while taking into account constraints, and cannot guarantee to find good solutions. In this paper, we propose a novel deep reinforcement learning (DRL) algorithm for graph searching, called DRL-GS, for network topology optimization. DRL-GS consists of three novel components, including a verifier to validate the correctness of a generated network topology, a graph neural network (GNN) to efficiently approximate topology rating, and a DRL agent to conduct a topology search. DRL-GS can efficiently search over relatively large topology space and output topology with satisfactory performance. We conduct a case study based on a real-world network scenario, and our experimental results demonstrate the superior performance of DRL-GS in terms of both efficiency and performance.

Graphs are widely used to represent the relations among entities. When one owns the complete data, an entire graph can be easily built, therefore performing analysis on the graph is straightforward. However, in many scenarios, it is impractical to centralize the data due to data privacy concerns. An organization or party only keeps a part of the whole graph data, i.e., graph data is isolated from different parties. Recently, Federated Learning (FL) has been proposed to solve the data isolation issue, mainly for Euclidean data. It is still a challenge to apply FL on graph data because graphs contain topological information which is notorious for its non-IID nature and is hard to partition. In this work, we propose a novel FL framework for graph data, FedCog, to efficiently handle coupled graphs that are a kind of distributed graph data, but widely exist in a variety of real-world applications such as mobile carriers’ communication networks and banks’ transaction networks. We theoretically prove the correctness and security of FedCog. Experimental results demonstrate that our method FedCog significantly outperforms traditional FL methods on graphs. Remarkably, our FedCog improves the accuracy of node classification tasks by up to 14.7%.

Accurate transmission performance prediction of the balise system is important for reliable ground–train communication in high-speed rails. However, the combined effects of ground–train coupling and on-board demodulation make it difficult to predict long-term and volatile holistic transmission performance under multisource disturbances. To address these issues, this article proposes a gate-steered residual interweave architecture. A new gated double-dilated temporal convolutional network (GDTCN) is introduced by combining large and small receptive fields with gating units to extract multiscale features and filter out useless information. It can learn multisource parameter disturbances induced by transient coupling and intermittent demodulation to reduce the prediction errors of occasional volatilities in holistic transmission performance. A new residual interweave GDTCN structure with linear gated cross connections is built to remit the differences between signal and telegram parameter features in view of the inconsistent data transfer modes during coupling and demodulation. This structure can facilitate multipath feature interaction and fusion to enrich discriminative correlation information, thereby improving the prediction accuracy of holistic transmission performance. An adaptive gated attention mechanism is designed to exploit correlation and dependent information between coupling and demodulation in a weighted fusion manner. It can minimize prediction error accumulation to enhance long-term holistic transmission performance forecasting. Extensive experiments demonstrate that the proposed architecture can perform predictions with high accuracy and efficiency under different rail conditions. The proposed method can provide early warning before ground balise or on-board module failures to improve the reliability and availability of ground–train communication.

Sequential recommendation aims at predicting the next item that the user may be interested in given the historical interaction sequence. Typical neural models derive a single history embedding to represent the user's interests. Moving one step forward, recent studies point out that multiple sequence embeddings can help to better capture multi-faceted user interests. However, when ranking candidate items, these methods usually adopt the greedy inference strategy. This approach uses the best matching interest for each candidate item to calculate the ranking score, neglecting the target interest distribution in different contexts, which might lead to incompatibility with the current user intent. In this paper, we propose to enhance multi-interest recommendation by predicting the target user interest with a separate interest predictor and a specifically designed distillation loss. The proposed framework consists of two modules: the 1) multi-interest extractor to generate multiple embeddings regarding different user interests; and the 2) target-interest predictor to predict the interest distribution in the current context, which will be further utilized to dynamically aggregate multi-interest embeddings. To provide explicit supervision signals to the target-interest predictor, we devise a target-interest distillation loss that uses the similarity between the target item and multi-interest embeddings as the soft label of the target interest. This helps the target-interest predictor to accurately predict the user interest at the inference stage and enhances its generalization ability. Extensive experiments on three real-world datasets show the effectiveness and flexibility of the proposed framework.

Self-supervised entity alignment (EA) aims to link equivalent entities across different knowledge graphs (KGs) without the use of pre-aligned entity pairs. The current state-of-the-art (SOTA) self-supervised EA approach draws inspiration from contrastive learning, originally designed in computer vision based on instance discrimination and contrastive loss, and suffers from two shortcomings. Firstly, it puts unidirectional emphasis on pushing sampled negative entities far away rather than pulling positively aligned pairs close, as is done in the well-established supervised EA. Secondly, it advocates the minimum information requirement for self-supervised EA, while we argue that self-described KG's side information (e.g., entity name, relation name, entity description) shall preferably be explored to the maximum extent for the self-supervised EA task. In this work, we propose an interactive contrastive learning model for self-supervised EA. It conducts bidirectional contrastive learning via building pseudo-aligned entity pairs as pivots to achieve direct cross-KG information interaction. It further exploits the integration of entity textual and structural information and elaborately designs encoders for better utilization in the self-supervised setting. Experimental results show that our approach outperforms the previous best self-supervised method by a large margin (over 9% [email protected] absolute improvement on average) and performs on par with previous SOTA supervised counterparts, demonstrating the effectiveness of the interactive contrastive learning for self-supervised EA. The code and data are available at https://github.com/THU-KEG/ICLEA.

Most existing slot filling models tend to memorize inherent patterns of entities and corresponding contexts from training data. However, these models can lead to system failure or undesirable outputs when being exposed to spoken language perturbation or variation in practice. We propose a perturbed semantic structure awareness transferring method for training perturbation-robust slot filling models. Specifically, we introduce two MLM-based training strategies to respectively learn contextual semantic structure and word distribution from unsupervised language perturbation corpus. Then, we transfer semantic knowledge learned from upstream training procedure into the original samples and filter generated data by consistency processing. These procedures aims to enhance the robustness of slot filling models. Experimental results show that our method consistently outperforms the previous basic methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts.

Traditional intent classification models are based on a pre-defined intent set and only recognize limited in-domain (IND) intent classes. But users may input out-of-domain (OOD) queries in a practical dialogue system. Such OOD queries can provide directions for future improvement. In this paper, we define a new task, Generalized Intent Discovery (GID), which aims to extend an IND intent classifier to an open-world intent set including IND and OOD intents. We hope to simultaneously classify a set of labeled IND intent classes while discovering and recognizing new unlabeled OOD types incrementally. We construct three public datasets for different application scenarios and propose two kinds of frameworks, pipeline-based and end-to-end for future work. Further, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future GID research.

With the widespread use of mobile devices in recent years, accurate prediction of base station traffic is vital for maintaining a good quality of mobile network services. In this paper, we propose an ensemble learning framework to predict the cellular traffic of base stations. Specifically, we introduce the Granger causality test to find the causal relationship in the base stations and model the spatial relationship between them simultaneously. We also employ a temporal convolutional network (TCN) to extract the sequential temporal features of base station traffic. Aiming at modelling the long-tail characteristics of the traffic distribution of base stations, we use the technique of redundant encoding to refine the prediction task to learn the base station traffic of different scales combined with ensemble learning. Extensive experimental results demonstrate that our method can predict base station traffic precisely and outperforms the best baseline by nearly 13% on average in terms of NMSE and NRMSE.

The harmonic structure of speech is resistant to noise, but the harmonics may still be partially masked by noise. Therefore, we previously proposed a harmonic gated compensation network (HGCN) to predict the full harmonic locations based on the unmasked harmonics and process the result of a coarse enhancement module to recover the masked harmonics. In addition, the auditory loudness loss function is used to train the network. For the DNS Challenge, we update HGCN with the following aspects, resulting in HGCN+. First, a high-band module is employed to help the model handle full-band signals. Second, cosine is used to model the harmonic structure more accurately. Then, the dual-path encoder and dual-path rnn (DPRNN) are introduced to take full advantage of the features. Finally, a gated residual linear structure replaces the gated convolution in the compensation module to increase the receptive field of frequency. The experimental results show that each updated module brings performance improvement to the model. HGCN+ also outperforms the referenced models on both wide-band and full-band test sets.

Mask processing in the time-frequency (T-F) domain through the neural network has been one of the mainstreams for single-channel speech enhancement. However, it is hard for most models to handle the situation when harmonics are partially masked by noise. To tackle this challenge, we propose a harmonic gated compensation network (HGCN). We design a high-resolution harmonic integral spectrum to improve the accuracy of harmonic locations prediction. Then we add voice activity detection (VAD) and voiced region detection (VRD) to the convolutional recurrent network (CRN) to filter harmonic locations. Finally, the harmonic gating mechanism is used to guide the compensation model to adjust the coarse results from CRN to obtain the refinedly enhanced results. Our experiments show HGCN achieves substantial gain over a number of advanced approaches in the community.

Graph Neural Networks (GNNs) have been success-fully applied to a variety of graph analysis tasks. Some recent studies have demonstrated that decoupling neighbor aggregation and feature transformation helps to scale GNNs to large graphs. However, very large graphs, with billions of nodes and millions of features, are still beyond the capacity of most existing GNNs. In addition, when we are only interested in a small number of nodes (called target nodes) in a large graph, it is inefficient to use the existing GNNs to infer the labels of these few target nodes. The reason is that they need to propagate and aggregate either node features or predicted labels over the whole graph, which incurs high additional costs relative to the few target nodes. To solve the above challenges, in this paper we propose a novel scalable and effective GNN framework COSAL. In COSAL, we substitute the expensive aggregation with an efficient proximate node selection mechanism, which picks out the most important$K$nodes for each target node according to the graph topology. We further propose a fine-grained neighbor importance quantification strategy to enhance the expressive power of COSAL. Empirical results demonstrate that our COSAL achieves superior performance in accuracy, training speed, and partial inference efficiency. Remarkably, in terms of node classification accuracy, our model COSAL outperforms baselines by significant margins of 2.22%, 2.23%, and 3.95% on large graph datasets Amazon2M, MAG-Scholar-C, and ogbn-papers100M, respectively.11Code available at https://github.com/joyce-x/COSAL.

Despite the success of text-to-text pre-trained models in various natural language generation (NLG) tasks, the generation performance is largely restricted by the number of labeled data in downstream tasks, particularly in data-to-text generation tasks. Existing works mostly utilize abundant unlabeled structured data to conduct unsupervised pre-training for task adaption, which fail to model the complex relationship between source structured data and target texts. Thus, we introduce self-training as a better few-shot learner than task-adaptive pre-training, which explicitly captures this relationship via pseudo-labeled data generated by the pre-trained model. To alleviate the side-effect of low-quality pseudo-labeled data during self-training, we propose a novel method called Curriculum-Based Self-Training (CBST) to effectively leverage unlabeled data in a rearranged order determined by the difficulty of text generation. Experimental results show that our method can outperform fine-tuning and task-adaptive pre-training methods, and achieve state-of-the-art performance in the few-shot setting of data-to-text generation.

Multi-action dialog policy (MADP), which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing MADP models usually imitate action combinations from the labeled multi-action dialog samples. Due to data limitations, they generalize poorly toward unseen dialog flows. While interactive learning and reinforcement learning algorithms can be applied to incorporate external data sources of real users and user simulators, they take significant manual effort to build and suffer from instability. To address these issues, we propose Planning Enhanced Dialog Policy (PEDP), a novel multi-task learning framework that learns single-action dialog dynamics to enhance multi-action prediction. Our PEDP method employs model-based planning for conceiving what to express before deciding the current response through simulating single-action dialogs. Experimental results on the MultiWOZ dataset demonstrate that our fully supervised learning-based method achieves a solid task success rate of 90.6%, improving 3% compared to the state-of-the-art methods. The source code and the appendix of this paper can be obtained from https://github.com/ShuoZhangXJTU/PEDP.

Spoken language understanding (SLU) treats automatic speech recognition (ASR) and natural language understanding (NLU) as a unified task and usually suffers from data scarcity.We exploit an ASR and NLU joint training method based on meta auxiliary learning to improve the performance of low-resource SLU task by only taking advantage of abundant manual transcriptions of speech data.One obvious advantage of such method is that it provides a flexible framework to implement a lowresource SLU training task without requiring access to any further semantic annotations.In particular, a NLU model is taken as label generation network to predict intent and slot tags from texts; a multi-task network trains ASR task and SLU task synchronously from speech; and the predictions of label generation network are delivered to the multi-task network as semantic targets.The efficiency of the proposed algorithm is demonstrated with experiments on the public CATSLU dataset, which produces more suitable ASR hypotheses for the downstream NLU task.

Developing semi-supervised task-oriented dialog (TOD) systems by leveraging unlabeled dialog data has attracted increasing interests.For semi-supervised learning of latent state TOD models, variational learning is often used, but suffers from the annoying high-variance of the gradients propagated through discrete latent variables and the drawback of indirectly optimizing the target log-likelihood.Recently, an alternative algorithm, called joint stochastic approximation (JSA), has emerged for learning discrete latent variable models with impressive performances.In this paper, we propose to apply JSA to semi-supervised learning of the latent state TOD models, which is referred to as JSA-TOD.To our knowledge, JSA-TOD represents the first work in developing JSA based semi-supervised learning of discrete latent variable conditional models for such long sequential generation problems like in TOD systems.Extensive experiments show that JSA-TOD significantly outperforms its variational learning counterpart.Remarkably, semi-supervised JSA-TOD using 20% labels performs close to the full-supervised baseline on MultiWOZ2.1.

Recently, Transformer based pretrained language models (PLMs), such as GPT2 and T5, have been leveraged to build generative task-oriented dialog (TOD) systems. A drawback of existing PLM-based models is their non-Markov architectures across turns, i.e., the whole history is used as the conditioning input at each turn. First, this brings inefficiencies in memory and computation. Furthermore, using the whole history increases model complexity and may hurt the training efficiency, especially when facing small amounts of labeled training data (the low-resource setting). In this paper, motivated by the observation that dialog states could be viewed as Markov states, we propose to build Markovian Generative Architectures (MGA) over PLM backbones for efficient TOD systems. Experiments on MultiWOZ2.1 show that in the rich-resource setting, the proposed Markov models reduce memory and time costs without performance degradation; in the low-resource setting, the training efficiency of the Markov models is more significant.

Contract consistency is important in ensuring the legal validity of the contract. In many scenarios, a contract is written by filling the blanks in a precompiled form. Due to carelessness, two blanks that should be filled with the same (or different) content may be incorrectly filled with different (or same) content. This will result in the issue of contract inconsistencies, which may severely impair the legal validity of the contract. Traditional methods to address this issue mainly rely on manual contract review, which is labor-intensive and costly. In this work, we formulate a novel Contract Inconsistency Checking (CIC) problem, and design an end-to-end framework, called Pair-wise Blank Resolution (PBR), to solve the CIC problem with high accuracy. Our PBR model contains a novel BlankCoder to address the challenge of modeling meaningless blanks. BlankCoder adopts a two-stage attention mechanism that adequately associates a meaningless blank with its relevant descriptions while avoiding the incorporation of irrelevant context words. Experiments conducted on real-world datasets show the promising performance of our method with a balanced accuracy of 94.05% and an F1 score of 90.90% in the CIC problem.

Mobile network traffic forecasting is one of the key functions in daily network operation. A commercial mobile network is large, heterogeneous, complex and dynamic. These intrinsic features make mobile network traffic forecasting far from being solved even with recent advanced algorithms such as graph convolutional network-based prediction approaches and various attention mechanisms, which have been proved successful in vehicle traffic forecasting. In this paper, we cast the problem as a spatial-temporal sequence prediction task. We propose a novel deep learning network architecture, Adaptive Multi-receptive Field Spatial-Temporal Graph Convolutional Networks (AMF-STGCN), to model the traffic dynamics of mobile base stations. AMF-STGCN extends GCN by (1) jointly modeling the complex spatial-temporal dependencies in mobile networks, (2) applying attention mechanisms to capture various Receptive Fields of heterogeneous base stations, and (3) introducing an extra decoder based on a fully connected deep network to conquer the error propagation challenge with multi-step forecasting. Experiments on four real-world datasets from two different domains consistently show AMF-STGCN outperforms the state-of-the-art methods.

A Dialogue State Tracker (DST) is a core component of a modular task-oriented dialogue system.Tremendous progress has been made in recent years.However, the major challenges remain.The state-of-the-art accuracy for DST is below 50% for a multi-domain dialogue task.A learnable DST for any new domain requires a large amount of labeled indomain data and training from scratch.In this paper, we propose a Meta-Reinforced Multi-Domain State Generator (MERET).Our first contribution is to improve the DST accuracy.We enhance a neural model based DST generator with a reward manager, which is built on policy gradient reinforcement learning (R-L) to fine-tune the generator.With this change, we are able to improve the joint accuracy of DST from 48.79% to 50.91% on the Multi-WOZ corpus.Second, we explore to train a DST meta-learning model with a few domains as source domains and a new domain as target domain.We apply the model-agnostic metalearning (MAML) algorithm to DST and the obtained meta-learning model is used for new domain adaptation.Our experimental results show this solution is able to outperform the traditional training approach with extremely less training data in target domain.

Structured belief states are crucial for user goal tracking and database query in task-oriented dialog systems.However, training belief trackers often requires expensive turn-level annotations of every user utterance.In this paper we aim at alleviating the reliance on belief state labels in building end-to-end dialog systems, by leveraging unlabeled dialog data towards semi-supervised learning.We propose a probabilistic dialog model, called the LAtent BElief State (LABES) model, where belief states are represented as discrete latent variables and jointly modeled with system responses given user inputs.Such latent variable modeling enables us to develop semi-supervised learning under the principled variational learning framework.Furthermore, we introduce LABES-S2S, which is a copyaugmented Seq2Seq model instantiation of LABES 1 .In supervised experiments, LABES-S2S obtains strong results on three benchmark datasets of different scales.In utilizing unlabeled dialog data, semi-supervised LABES-S2S significantly outperforms both supervisedonly and semi-supervised baselines.Remarkably, we can reduce the annotation demands to 50% without performance loss on MultiWOZ.

Conditional random fields (CRFs) have been shown to be one of the most successful approaches to sequence labeling. Various linear-chain neural CRFs (NCRFs) are developed to implement the non-linear node potentials in CRFs, but still keeping the linear-chain hidden structure. In this paper, we propose NCRF transducers, which consists of two RNNs, one extracting features from observations and the other capturing (theoretically infinite) long-range dependencies between labels. Different sequence labeling methods are evaluated over POS tagging, chunking and NER (English, Dutch). Experiment results show that NCRF transducers achieve consistent improvements over linear-chain NCRFs and RNN transducers across all the four tasks, and can improve state-of-the-art results.

Mobile voice search provides users an easier way to search for information using voice from mobile devices. Most mobile search applications have access to the latitude/longitude coordinates of the device, which can be exploited to deliver location-specific search results. Geography and queries users submit are inextricably intertwined. In this paper we present the findings of a study on the spatial proximity of queried location and device location in a local search application. We then propose a location-aware query parsing model to parse queries into concepts that are necessary for high precision search.

Mobile search is a fast-growing business. Mobile voice search provides an easier way to search for information from mobile devices using voice. Natural language understanding (NLU) is a key component technology in voice search to assure search effectiveness. This paper describes a general framework for building the NLU modules in voice search applications. The NLU task is defined as segmenting ASR output, including ASR 1-Best and ASR Word Confusion Networks, into several concepts that are necessary for high-precision search. Application data such as raw query logs, annotated queries and source database are used to train the NLU models. We instantiated this framework on a mobile business search application and demonstrated the flexibility of using this framework. We report the experimental results on this application.

Social media sites such as Twitter continue to grow at a fast pace. People of all generations use social media to exchange messages and share experiences of their life in a timely fashion. Most of these sites make their data available. An intriguing question is can we exploit this real-time and massive data-flow to improve business in a measurable way. In this paper, we are particularly interested in tweets (Twitter messages) that are relevant to mobile network performance. We compare tweets with a more traditional source of user experience, i.e., customer care tickets, and correlate both of them with a list of major network incidents. From our study, we have the following observations. First, Twitter users and users who call customer service tend to report different types of performance issues. Second, we observe that tweets typically appear more rapidly in response to network problems than customer tickets. They also appear to respond to a wider range of network issues. Third, significant spikes in the number of tweets appear to indicate short term performance impairments which are not reported in our current list of major network incidents. These observations together indicate that Twitter is an attractive, complementary source for monitoring service performance and its impact on user experience.

We present a machine learning approach to sentiment classification on twitter messages (tweets). We classify each tweet into two categories: polar and non-polar. Tweets with positive or negative sentiment are considered polar. They are considered non-polar otherwise. Sentiment analysis of tweets can potentially benefit different parties, such as consumers and marketing researchers, for obtaining opinions on different products and services. We present methods for text normalization of the noisy tweets and their classification with respect to the polarity. We experiment with a mixture model approach for generation of sentimental words, which are later used as indicator features of the classification model. Based on a gold standard manually annotated ensemble of tweets, with the new approach, we obtain F-scores that are relatively 10% better than a classification baseline that uses raw word n-gram features.

Mobile voice search is a fast-growing business. It provides users an easier way to search for information using voice from mobile devices. In this paper, we describe a statistical approach to query parsing to assure search effectiveness. The task is to segment speech recognition (ASR) output, including ASR 1-Best and ASR word lattices, into segments and associate each segment with needed concepts in the application. We train the models including concept prior probability, query segment generation probability, and query subject probability from application data such as query log and source database. We apply the learned models on a mobile business search application and demonstrate the robustness of query parsing to ASR errors.

With the exponential growth in the number of mobile devices, voice enabled local search is emerging as one of the most popular applications. Although the application is essentially an integration of automatic speech recognition (ASR) and text or database search, the potential usefulness of this application has been widely acknowledged. In this paper, we present a data-driven approach to voice query parsing, that segments the input query into several fields that are necessary for high-precision local search. We also demonstrate the robustness of our approach to ASR errors. We present an approach for exploiting multiple hypotheses from ASR, in the form of word confusion networks, in order to achieve tighter coupling between ASR and query parsing. A confusion-network based query parsing outperforms ASR 1-best based query-parsing by 2.6% absolute.

Web Talk is a system for analyzing unstructured information from company websites to support automtic reation of spoken dialog applications. The goal is to completely automate the process of building, maintaining and deploying dialog applications by leveraging the wealth of information on the World Wide Web. Web Talk employs technologies in web mining, document understanding, question/answering, and speech and language processing. In this paper, we review extensions to these technologies to make them suitable for creating a Web Talk application. e present an evaluation study of a Web Talk spoken dialog system that has been instantiated on a telecom company website. Experiments with 30 different scenarios indicate promising results and provide evidence that such systems can potentially revolutionize the paradigm for creating and scaling spoken dialog services.

This paper proposes a learning approach for discovering the semantic structure of Web pages. The task includes partitioning the text on a Web page into information blocks and identifying their semantic categories. We employed two machine learning techniques, Adaboost and SVMs, to learn from a labeled Web page corpus. We evaluated our approach on general Web pages from the World Wide Web and obtained encouraging results. This work can be beneficial to a number of Web-driven applications such as search engines, Web-based question answering, Web-based data mining as well as voice enabled Web navigation.

This paper investigates whether a customer care dialog system can be built automatically by mining and leveraging the wealth of information on a company’s website. Our objective is to allow the users to ask FAQ-like questions which may request a specific piece of information, an analytical answer, or a transaction such as checking the status of the payment. We developed an infrastructure, referred to as WebTalk, that constructs different application-specific dialog systems by taking different websites as input. In this paper, we overview the involved technologies in WebTalk, address the challenges that will be shown very different from those in traditional dialog systems, and describe our efforts to approach these challenges. We present an evaluation for one WebTalk component: website based question answering. 1.

WebTalk is an intelligent agent that automatically mines company websites to create interactive spoken and chat-based dialog systems. In this paper, we briefly describe the technologies that WebTalk utilizes for extracting task knowledge from websites and answering questions specified in natural language. We elaborate on an evaluation of the question answering component of WebTalk. In particular, we compare this component with a deployed hand-crafted question answering agent. A detailed evaluation analysis is presented.