Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.

The growing availability of the Internet provides opportunities for depression screening. In recent years, depression analysis based on social media texts has shown great promise, yet most works have focused on treating it as a binary problem. Meanwhile, existing methods need the perception of fine-grained emotions in historical contexts. Moreover, they do not consider the time interval between posts and thus ignore the decay of emotions over time. In this paper, we propose a Time-aware Depression identification model based on Emotion capturing (TIDE), a novel method to identify the severity of depression in social media users. We define the assessment as an ordinal regression problem to distinguish differences in depression levels. Specifically, TIDE uses Plutchik's wheel of emotions to characterize the emotional historical spectrum, then models the emotion decay process of historical context using a Time-aware LSTM (T-LSTM). We experiment on two Reddit public datasets to demonstrate that our approach outperforms state-of-the-art models. Then, ablation studies and qualitative analysis further demonstrate the validity of the proposed modules. Overall, our work can facilitate social media-based analysis of depression and shows potential for application to mental health-related issues.

Augmenting pre-trained language models (PLMs) with knowledge graphs (KGs) has demonstrated superior performance on commonsense reasoning.Given a commonsense based QA context (question and multiple choices), existing approaches usually estimate the plausibility of candidate choices separately based on their respective retrieved KGs, without considering the interference among different choices.In this paper, we propose an Attention guided Commonsense rEasoning Network (ACENet) 1 to endow the neural network with the capability of integrating hybrid knowledge.Specifically, our model applies the multilayer interaction of answer choices to continually strengthen correct choice information and guide the message passing of GNN.In addition, we also design a mix attention mechanism of nodes and edges to iteratively select supporting evidence on hybrid knowledge graph.Experimental results demonstrate the effectiveness of our proposed model through considerable performance gains across CommonsenseQA and OpenbookQA datasets.

One of the key challenges of knowledge base question answering (KBQA) is the multi-hop reasoning.Since in different hops, one attends to different parts of question, it is important to dynamically represent the question semantics for each hop.Existing methods, however, (i) infer the dynamic question representation only through coarse-grained attention mechanisms, which may bring information loss, (ii) and have not effectively modeled the sequential logic, which is crucial for the multi-hop reasoning process in KBQA.To address these issues, we propose a sequential reasoning selfattention mechanism to capture the crucial reasoning information of each single hop in a more fine-grained way.Based on Gated Recurrent Unit (GRU) which is good at modeling sequential process, we propose a simple but effective GRU-inspired Flow Control (GFC) framework to model sequential logic in the whole multi-hop process.Extensive experiments on three popular benchmark datasets have demonstrated the superior effectiveness of our model.In particular, GFC achieves new state-of-the-art Hits@1 of 76.8% on WebQSP and is also effective when KB is incomplete.Our code and data are available at https: //github.com/