Large Language Models (LLMs) have achieved oustanding performance across various natural language processing tasks, including those from Discourse and Dialogue traditions. However, these achievements are typically obtained thanks to pretraining on huge datasets. In contrast, humans learn to speak and communicate through dialogue and spontaneous speech with only a fraction of the language exposure. This disparity has spurred interest in evaluating whether smaller, more carefully selected and curated pretraining datasets can support robust performance on specific tasks. Drawing inspiration from the BabyLM initiative, we construct small (10M-token) pretraining datasets from different sources, including conversational transcripts and Wikipedia-style text. To assess the impact of these datasets, we develop evaluation benchmarks focusing on discourse and interactional markers, extracted from high-quality spoken corpora in English, French, and Mandarin. Employing a zero-shot classification framework inspired by the BLiMP benchmark, we design tasks wherein the model must determine, between a genuine utterance extracted from a corpus and its minimally altered counterpart, which one is the authentic instance. Our findings reveal that the nature of pretraining data significantly influences model performance on discourse-related tasks. Models pretrained on conversational data exhibit a clear advantage in handling discourse and interactional markers compared to those trained on written or encyclopedic text. Furthermore, the models, trained on small amount spontaneous speech transcripts, perform comparably to standard LLMs.

It is the most important way for researchers to acquire academic progress via reading scientific papers, most of which are in PDF format. However, existing PDF Readers like Adobe Acrobat Reader and Foxit PDF Reader are usually only for reading by rendering PDF files as a whole, and do not consider the multi-granularity content understanding of a paper itself. Specifically, taking a paper as a basic and separate unit, existing PDF Readers cannot access extended information about the paper, such as corresponding videos, blogs and codes. Meanwhile, they cannot understand the academic content of a paper, such as terms, authors, and citations. To solve these problems, we introduce Hammer PDF, an intelligent PDF Reader for scientific papers. Apart from basic reading functions, Hammer PDF has the following four innovative features: (1) information extraction ability, which can locate and mark spans like terms and other entities; (2) information extension ability, which can present relevant academic content of a paper, such as citations, references, codes, videos, blogs, etc; (3) built-in Hammer Scholar, an academic search engine based on academic information collected from major academic databases; (4) built-in Q&A bot, which can find helpful conference information. The proposed Hammer PDF Reader can help researchers, especially those studying computer science, to improve the efficiency and experience of reading scientific papers. We have released Hammer PDF, available at https://pdf.hammerscholar.net/face.

We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.

We correct wrongly reported results on BLiMP.

Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, Samuel R. Bowman. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Tree-structured neural network architectures for sentence encoding draw inspiration from the approach to semantic composition generally seen in formal linguistics, and have shown empirical improvements over comparable sequence models by doing so.Moreover, adding multiplicative interaction terms to the composition functions in these models can yield significant further improvements.However, existing compositional approaches that adopt such a powerful composition function scale poorly, with parameter counts exploding as model dimension or vocabulary size grows.We introduce the Lifted Matrix-Space model, which uses a global transformation to map vector word embeddings to matrices, which can then be composed via an operation based on matrix-matrix multiplication.Its composition function effectively transmits a larger number of activations across layers with relatively few model parameters.We evaluate our model on the Stanford NLI corpus, the Multi-Genre NLI corpus, and the Stanford Sentiment Treebank and find that it consistently outperforms TreeLSTM (Tai et al., 2015), the previous best known composition function for treestructured models.

OFDM is a promising scheme for multicarrier (MC) transmission, since it has the advantage of high spectral efficiency, robustness to frequency selective fading, and feasibility of low-cost transceiver implementation. The combination of OFDM and CDMA, referred to as OFDM-CDMA, recently attracted most research interests in wireless communications. In this paper, adaptive receivers for OFDM-CDMA communication systems are developed based on the least mean square (LMS) algorithm. We evaluate the down-link performance for two forms of OFDM-CDMA schemes which are MC-CDMA and MC-DS-CDMA in multipath fading channels. A drawback of OFDM schemes is the rate reduction due to the cyclic prefix (CP) overhead, which significantly affects the channel utilization in MC-DS-CDMA systems. Thus we propose adaptive receiver structure that exploits both frequency-domain and time-domain equalization to improve the performances of MC-DS-CDMA systems with shorter or no CP. Simulation results show performance comparisons between MC-CDMA and MC-DS-CDMA systems, which also verify the performance improvement of our proposed receiver for MC-DS-CDMA systems exploiting shorter or no CP.