Shenglai Zeng, Tianqi Zheng, Chuan Tian, Dante Everaert, Yau-Shian Wang, Yupin Huang, Michael J. Morais, Rohit Patki, Jinjin Tian, Xinnan Dai, Kai Guo, Monica Xiao Cheng, Hui Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

The remarkable performance of large language models (LLMs) in zero-shot language understanding has garnered significant attention.However, employing LLMs for large-scale inference or domain-specific fine-tuning requires immense computational resources due to their substantial model size.To overcome these limitations, we introduce a novel method, namely GENCO, which leverages the strong generative power of LLMs to assist in training a smaller and more adaptable language model.In our method, an LLM plays an important role in the self-training loop of a smaller model in two important ways.Firstly, the LLM is used to augment each input instance with a variety of possible continuations, enriching its semantic context for better understanding.Secondly, it helps crafting additional high-quality training pairs, by rewriting input texts conditioned on predicted labels.This ensures the generated texts are highly relevant to the predicted labels, alleviating the prediction error during pseudolabeling, while reducing the dependency on large volumes of unlabeled text.In our experiments, GENCO outperforms previous state-ofthe-art methods when only limited (< 5% of original) in-domain text data is available.Notably, our approach surpasses the performance of Alpaca-7B with human prompts, highlighting the potential of leveraging LLM for selftraining.

We present PESCO, a novel contrastive learning framework that substantially improves the performance of zero-shot text classification.We formulate text classification as a neural text matching problem where each document is treated as a query, and the system learns the mapping from each query to the relevant class labels by (1) adding prompts to enhance label matching, and (2) using retrieved labels to enrich the training set in a self-training loop of contrastive learning.PESCO achieves state-of-the-art performance on four benchmark text classification datasets.On DBpedia, we achieve 98.5% accuracy without any labeled data, which is close to the fully-supervised result.Extensive experiments and analyses show all the components of PESCO are necessary for improving the performance of zero-shot text classification.

The semantic matching problem in product search seeks to retrieve all semantically relevant products given a user query. Recent studies have shown that extreme multi-label classification~(XMC) model enjoys both low inference latency and high recall in real-world scenarios. These XMC semantic matching models adopt TF-IDF vectorizers to extract query text features and use mainly sparse matrices for the model weights. However, limited availability of libraries for efficient parallel sparse modules may lead to tediously long model building time when the problem scales to hundreds of millions of labels. This incurs significant hardware cost and renders the semantic model stale even before it is deployed. In this paper, we investigate and accelerate the model building procedures in a tree-based XMC model. On a real-world semantic matching task with 100M labels, our enhancements achieve over 10 times acceleration (from 3.1 days to 6.7 hours) while reducing hardware cost by 25%.

Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space.Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel sentences.In this work, we propose mSimCSE, which extends SimCSE (Gao et al., 2021) to multilingual settings and reveal that contrastive learning on English data can surprisingly learn high-quality universal cross-lingual sentence embeddings without any parallel data.In unsupervised and weakly supervised settings, mSim-CSE significantly improves previous sentence embedding methods on cross-lingual retrieval and multilingual STS tasks.The performance of unsupervised mSimCSE is comparable to fully supervised methods in retrieving lowresource languages and multilingual STS.The performance can be further enhanced when cross-lingual NLI data is available.1

Yaushian Wang, Hung-Yi Lee, Yun-Nung Chen. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Auto-encoders compress input data into a latent-space representation and reconstruct the original data from the representation.This latent representation is not easily interpreted by humans.In this paper, we propose training an auto-encoder that encodes input text into human-readable sentences, and unpaired abstractive summarization is thereby achieved.The auto-encoder is composed of a generator and a reconstructor.The generator encodes the input text into a shorter word sequence, and the reconstructor recovers the generator input from the generator output.To make the generator output human-readable, a discriminator restricts the output of the generator to resemble human-written sentences.By taking the generator output as the summary of the input text, abstractive summarization is achieved without document-summary pairs as training data.Promising results are shown on both English and Chinese corpora.

Conventional seq2seq chatbot models only try to find the sentences with the highest probabilities conditioned on the input sequences, without considering the sentiment of the output sentences. Some research works trying to modify the sentiment of the output sequences were reported. In this paper, we propose five models to scale or adjust the sentiment of the chatbot response: persona-based model, reinforcement learning, plug and play model, sentiment transformation network and cycleGAN, all based on the conventional seq2seq model. We also develop two evaluation metrics to estimate if the responses are reasonable given the input. These metrics together with other two popularly used metrics were used to analyze the performance of the five proposed models on different aspects, and reinforcement learning and cycleGAN were shown to be very attractive. The evaluation metrics were also found to be well correlated with human evaluation.