Chains of Thought (CoT) have shown potential in augmenting the reasoning capabilities of language models, yet their effectiveness is predominantly observed in large language models (LLMs). Recently, several attempts have been made to inject CoT into small language models (SLMs) using distillation and achieved promising results. However, current methods (1) ignore the rationality and hierarchical logic of reasoning when constructing CoT; (2) fail to inject hierarchical reasoning priors into SLMs. In this paper, we design a Hierarchical CoT distillation framework termed HCoTT, whose core component is a hierarchical recursive sampling module and a hierarchical learning module. Specifically, hierarchical recursive sampling utilizes a hierarchical logic process to generate more diverse explanations and a Hierarchical Chain of Thought (HCoT). Furthermore, hierarchical learning encompasses hierarchical supervision and representation learning, which is designed to augment learning and representation of implicit explanatory priors in HCoT for SLMs. Experimental results show that HCoTT can effectively improve the performance of SLMs on Faculty-Reasoning and Multiple-Choice QA tasks. More impressively, our method is model-independent and can consistently improve performance with existing language model fusions of different scales.

Chains of thought (CoTs) have achieved success in enhancing the reasoning capabilities of large language models (LLMs), while their effectiveness is predominantly observed in LLMs. Existing solutions methods adopt distillation to inject chain-of-thought capabilities into small models (SLMs). However, they: (1) can not guarantee the rationality of the generated explanation due to hallucinations; (2) ignore diverse structures of CoT during knowledge transfer. In this paper, we propose a unified CoT distillation framework termed UniCoTT for considering diverse structural CoTs (\emph{i.e.}, chain, tree, and graph). UniCoTT contains two core strategies: iterative construction for structured CoTs and the structural constraint strategy. Specifically, UniCoTT prompts LLMs to iteratively produce accurate explanations with answers and unifies structured explanations as UniCoT which is seen as a bridge for knowledge transfer. Furthermore, UniCoTT utilizes the proposed unified supervised learning and structural consistency learning strategies to transfer knowledge of structured CoT to SLMs. Experimental results show that UniCoTT can significantly improve the performance of SLMs on multiple datasets across different NLP tasks. Our code is available at https://github.com/mengchuang123/UniCoTT.

Xianwei Zhuang, Xuxin Cheng, Liming Liang, Yuxin Xie, Zhichang Wang, Zhiqi Huang, Yuexian Zou. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Recent joint models for multi-intent detection and slot filling (a.k.a multi-intent SLU) have obtained promising results by leveraging the semantic similarities or co-occurrence relationships between intent and slot labels. However, a critical aspect frequently neglected by current models is the significant correlations between label co-occurrences and specific scenarios, such as watching a movie or booking a ticket, which is essential for understanding user utterances in multi-intent SLU. In this paper, we propose a new framework dubbed SALA (short for Scenario-aware Label graph interaction), which effectively captures the dynamic co-occurrence relationships among labels across various scenarios, employing a strategy akin to a divide-and-conquer approach. Concretely, SALA first autonomously classifies the scenario of utterances, and tracks the co-occurring labels by maintaining a unique co-occurrence matrix for each scenario during the training phase. These scenario-independent co-occurrence matrices are further employed to guide the interactions among label representations through graph propagation to conduct accurate prediction. Extensive experiments on two multi-intent SLU benchmark datasets demonstrate the superiority of our SALA. More strikingly, SALA also attains competitive results on four extra single-intent and multi-domain SLU benchmark datasets, demonstrating its strong generalizability.

Zero-shot Spoken Language Understanding (SLU) aims to enable task-oriented dialogue systems to understand user needs without training data. Challenging but worthwhile, zero-shot SLU reduces the time and effort that data labeling takes. Recent advancements in large language models (LLMs), such as GPT3.5 and ChatGPT, have shown promising results in zero-shot settings, which motivates us to explore prompt-based methods. In this study, we investigate whether strong SLU models can be constructed by directly prompting LLMs. Specifically, we propose a simple yet effective two-stage framework dubbed GPT-SLU, which transforms the SLU task into a question-answering problem. Powered by multi-stage mutual guided prompts, GPT-SLU can leverage the correlations between two subtasks in SLU to achieve better predictions, which is greatly explored in the traditional fine-tuning paradigm. Experimental results on three SLU benchmark datasets demonstrate the significant potential of LLMs for zero-shot SLU. Comprehensive analyses validate the effectiveness of our proposed framework and also indicate that there is still room for further improvement of LLMs in SLU scenarios.

Xianwei Zhuang, Zhichang Wang, Xuxin Cheng, Yuxin Xie, Liming Liang, Yuexian Zou. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Urban traffic data plays an important role in urban transportation planning. Due to the scarcity of real-life urban traffic data, many transportation planning applications need to generate synthesized traffic flows based on the real-life trajectory datasets. However, those synthesized traffic flows can only fit the input trajectories, which are static and does not reflect the real traffic distributions. In this paper, we use a stochastic origin-destination (OD) matrix to represent the density of the dynamic traffic flows and then develop a dynamic traffic flow generator. We extract the stochastic OD matrix from the trajectory data, design an efficient neural network to the predict successive stochastic OD matrices, and deploy our model on a real-world road network. The proposed model surpasses the existing generative model in RMSE, MAE, VAR, KL indicators, and is significantly better than the existing model in the MAE indicator. Our traffic generator is able to dynamically adjust urban traffics to generate different simulation environments.