The development of large language models (LLMs) raises the importance of assessing the fairness and completeness of various evaluation benchmarks. Regrettably, these benchmarks predominantly utilize uniform manual prompts, which may not fully capture the expansive capabilities of LLMs—potentially leading to an underestimation of their performance. To unlock the potential of LLMs, researchers pay attention to automated prompt search methods, which employ LLMs as optimizers to discover optimal prompts. However, previous methods generate the solutions implicitly, which overlook the underlying thought process and lack explicit feedback. In this paper, we propose a novel prompt introspective search framework, namely PromISe, to better release the capabilities of LLMs. It converts the process of optimizing prompts into an explicit chain of thought, through a step-by-step procedure that integrates self-introspect and self-refine. Extensive experiments, conducted over 73 tasks on two major benchmarks, demonstrate that our proposed PromISe significantly boosts the performance of 12 well-known LLMs compared to the baseline approach. Moreover, our study offers enhanced insights into the interaction between humans and LLMs, potentially serving as a foundation for future designs and implementations. Keywords: large language models, prompt search, self-introspect, self-refine

Adversarial vulnerability remains a major obstacle to the construction of reliable NLP systems. When imperceptible perturbations are added to raw input text, the performance of a deep learning model may drop dramatically under attacks. Recent work has argued that the adversarial vulnerability of a model is caused by non-robust features in supervised training. Thus, in this paper, we tackle the adversarial robustness challenge by means of disentangled representation learning, which is able to explicitly disentangle robust and non-robust features in text. Specifically, inspired by the variation of information (VI) in information theory, we derive a disentangled learning objective composed of mutual information to represent both the semantic representativeness of latent embeddings and the differentiation of robust and non-robust features. On the basis of this, we design a disentangled learning network to estimate the mutual information for realization. Experiments on the typical text-based tasks show that our method significantly outperforms the representative methods under adversarial attacks, indicating that discarding non-robust features is critical for improving model robustness.

Adversarial training is the dominant strategy towards model robustness.Current adversarial training methods typically apply perturbations to embedding representations, whereas actual text-based attacks introduce perturbations as discrete tokens.Thus there exists a gap between the continuous embedding representations and discrete text tokens that hampers the effectiveness of adversarial training.Moreover, the continuous representations of perturbations cannot be further utilized, resulting in the suboptimal performance.To bridge this gap for adversarial robustness, in this paper, we devise a novel generative adversarial training framework that integrates gradient-based learning, adversarial example generation and perturbed token detection.Our proposed framework consists of generative adversarial attack and adversarial training process.Specifically, in generative adversarial attack, the embeddings are shared between the classifier and the generative model, which enables the generative model to leverage the gradients from the classifier for generating perturbed tokens.Then, adversarial training process combines adversarial regularization with perturbed token detection to provide token-level supervision and improve the efficiency of sample utilization.Extensive experiments on five datasets from the AdvGLUE benchmark demonstrate that our framework significantly enhances the model robustness, surpassing the state-of-the-art results of ChatGPT by 10% in average accuracy.

Recent studies show that neural models for natural language processing are usually fragile under adversarial attacks (e.g., character-level insertion and word-level synonym substitution), which exposes the lack of robustness. Most defense techniques are tailored to specific semantic level attacks and do not possess the ability to mitigate multi-level attack simultaneously. Adversarial training has been shown the effectiveness of increasing model robustness. However, it often suffers from degradation on normal data, especially when the proportion of adversarial examples increase. To address this, we propose mixup regularized adversarial training (MRAT) against multi-level attack. Our method can utilize multiple adversarial examples to increase model intrinsic robustness without sacrificing the performance on normal data. We evaluate our method on text classification and entailment tasks. Experimental results on different text encoders (BERT, LSTM and CNN) with multi-level attack show that our method outperforms adversarial training consistently.

There is a great need for effective summarization methods to absorb the key points of large amounts of opinions expressed on the Web. In this paper, we study the problem of opinionated text summarization, which aims to generate a coherent summary for a set of opinionated texts towards a specific topic (e.g., a movie or a controversial issue). The main characteristic of this problem is that the input set contains an arbitrary number of texts, which brings about redundant opinions and useless texts. Further, informative opinions to be summarized are scattered over different opinionated texts, thus it is vital to avoid focusing only on partial opinions. However, previous work can not tackle the above two issues effectively. To address such issues, we propose a two-stage graph-to-sequence learning framework for summarizing opinionated texts. The first stage selects summary-worthy texts from all input opinionated texts, and we construct an opinion relation graph to help estimate salience via exploiting the relationships among the input texts. Given the selected texts, the second stage generates an opinion summary via a maximal marginal relevance guided graph-to-sequence model, which gives consideration to both salient and non-redundant opinions. Experimental results on two benchmark datasets show that our framework outperforms the existing state-of-the-art methods. Human evaluation further verifies that our framework can generate more informative and compact opinion summaries than previous methods.

Emotion-cause pair extraction aims to extract all emotion clauses coupled with their cause clauses from a given document.Previous work employs two-step approaches, in which the first step extracts emotion clauses and cause clauses separately, and the second step trains a classifier to filter out negative pairs.However, such pipeline-style system for emotion-cause pair extraction is suboptimal because it suffers from error propagation and the two steps may not adapt to each other well.In this paper, we tackle emotion-cause pair extraction from a ranking perspective, i.e., ranking clause pair candidates in a document, and propose a onestep neural approach which emphasizes interclause modeling to perform end-to-end extraction.It models the interrelations between the clauses in a document to learn clause representations with graph attention, and enhances clause pair representations with kernel-based relative position embedding for effective ranking.Experimental results show that our approach significantly outperforms the current two-step systems, especially in the condition of extracting multiple pairs in one document.Clause I.