Current neural networks often employ multi-domain-learning or attribute-injecting mechanisms to incorporate non-independent and identically distributed (non-IID) information for text understanding tasks by capturing individual characteristics and the relationships among samples. However, the extent of the impact of non-IID information and how these methods affect pre-trained language models (PLMs) remains unclear. This study revisits the assumption that non-IID information enhances PLMs to achieve performance improvements from a Bayesian perspective, which unearths and integrates non-IID and IID features. Furthermore, we proposed a multi-attribute multi-grained framework for PLM adaptations (M2A), which combines multi-attribute and multi-grained views to mitigate uncertainty in a lightweight manner. We evaluate M2A through prevalent text-understanding datasets and demonstrate its superior performance, mainly when data are implicitly non-IID, and PLMs scale larger.

Effectively and efficiently adapting a pre-trained language model (PLM) for human-centered text understanding (HCTU) is challenging since user tokens are million-level in most personalized applications and do not have concrete explicit semantics. A standard and parameter-efficient approach (e.g., LoRA) necessitates memorizing numerous suits of adapters for each user. In this work, we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically deploying in PLMs. Moreover, a personalized dropout and a mutual information maximizing strategies are adopted and hence the proposed PLoRA can be well adapted to few/zero-shot learning scenarios for the cold-start issue. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios for the HCTU task, even though it has fewer trainable parameters. For reproducibility, the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.

Existing studies on personalized sentiment classification consider a document review as an overall text unit and incorporate backgrounds (i.e., user and product information) to learn sentiment representation. However, it is difficult when these methods meet the current pretrained language models (PLMs) owing to quadratic costs that increase with text length and heterogeneous mixes of randomly initialized background information and textual information initialized from well-pretrained checkpoints during information incorporation. To address these problems, we propose a knowledge-enhanced and parameter-efficient layer normalization (E2LN) for efficient and effective review modeling via leveraging LN in transformer structures. Initially, a knowledge base is introduced that stores well-pretrained checkpoints, structured text information, and background information. Based on such a knowledge base, the ability of LN can be magnified as being a crucial component of transformer structure and then improve the performance of PLMs in downstream tasks. Moreover, the proposed E2LN can make PLMs capable of modeling long document reviews and incorporating background information with parameter-efficient fine-tuning and knowledge injecting. Extensive experimental results were obtained for three document-level sentiment classification benchmark datasets. By comparing the results, the effectiveness and efficiency of the proposed model was demonstrated. Code and Data are released at https://github.com/yoyo-yun/E2LN.

Automatic charge prediction based on deep learning methods is a crucial task in legal judgment prediction, aiming to predict the charges based on the fact description for a criminal case. While existing methods focus on multi-class cases with a single defendant, they fail to account for situations involving multiple defendants and labels, limiting their real-world application. To address these limitations, we propose a multi-defendant charge prediction approach that learns defendant-aware label representations (DLR). To handle complex circumstances for diverse defendants in a case, we extract defendant-specific representation by a machine reading comprehension approach via prompting the defendant’s name. In comparison with traditional text classifications that use discrete one-hot label representations, labels in charge predictions require clear definitions such as textual descriptions for determining the exact classified target. Therefore, we resort to a label description encoder to facilitate the charge predictions via understanding defendant-specific representations. Accordingly, we empower an efficient low-rank adaption module as a feature fuser that incorporates dependent-specific representations into label encoders. The proposed method is evaluated on both multi- and single-dependent-based charge prediction datasets, showing its comparable performances in real-world scenarios. The codes and collected datasets for our study are available at: https://github.com/cy330874054/LDLRMDCP.

The utilization of code-mixed texts allows individuals the opportunity to express sentiments flexibly in international and multilingual contexts. However, the diversity of languages and pragmatic writing styles can lead to semantic shifts at both word and sentence levels, resulting in a degraded comprehension of code-mixed texts by machines. To tackle this issue, we propose a method for language alignments, which aligns both word- and sentence-level semantic representations via a low-rank injection (LoRI) and a data augmentation strategy (DA), dubbed LoRIDA. LoRI integrates linguistic features into textual representations as a feature fusion mechanism. To further bridge the gaps between sentence-level semantics, we augment code-mixed data into individual source languages and apply a knowledge distillation method for joint alignments. We evaluate the performance of the proposed method on four code-mixed sentiment analysis datasets, demonstrating its superiority over existing methods. Our code is publicly available at https://github.com/linsongisgood/LELA.

Compared with conventional word embeddings, sentiment embeddings can distinguish words with similar contexts but opposite sentiment. They can be used to incorporate sentiment information from labeled corpora or lexicons by either end-to-end training or sentiment refinement. However, these methods present two major limitations. First, traditional approaches provide a fixed representation to each word but ignore the alternation of word meaning in different contexts. As a result, the polarity of a certain emotional word may vary with context, but will be assigned with a same representation. Another problem is the handling of out-of-vocabulary (OOV) or informal-writing sentiment words that would be assigned generic vectors (e.g., ). In addition, if affective words are not included in affective corpora or lexicons, they would be treated as neutral. Using such low-quality embeddings for building a neural model will reduce performance. This study proposes a training model of contextual sentiment embeddings. A stacked two-layer GRU model was used as the language model, simultaneously trained to incorporate semantic and sentiment information from labeled corpora and lexicons. To deal with OOV or informal-writing sentiment words, the WordPiece tokenizer was used to divide the text into subwords. The resulting model can be transferred to downstream applications by either feature extractor or fine-tuning. The results show that the proposed model can handle unseen or informal writing sentiment words and thus outperforms previously proposed methods.