Authorship models have historically generalized poorly to new domains because of the wide distribution of author-identifying signals across domains. In particular, the effects of topic and genre are highly domain-dependent and impact authorship analysis performance greatly. This paper addresses the existing data gap in authorship for these resources by introducing CROSSNEWS, a novel cross-genre dataset that connects formal journalistic articles and casual social media posts. CROSSNEWS is the largest authorship dataset of its kind for supporting both verification and attribution tasks, with comprehensive topic and genre annotations. We use CROSSNEWS to demonstrate that current models exhibit poor performance in genre transfer scenarios, underscoring the need for authorship models robust to genre-specific effects. We also explore SELMA, a new LLM embedding approach for large-scale authorship setups that outperforms existing models in both same-genre and cross-genre settings.

The surge of LLM studies makes synthesizing their findings challenging. Analysis of experimental results from literature can uncover important trends across studies, but the time-consuming nature of manual data extraction limits its use.Our study presents a semi-automated approach for literature analysis that accelerates data extraction using LLMs.It automatically identifies relevant arXiv papers, extracts experimental results and related attributes, and organizes them into a structured dataset, LLMEvalDB.We then conduct an automated literature analysis of frontier LLMs, reducing the effort of paper surveying and data extraction by more than 93% compared to manual approaches.We validate LLMEvalDB by showing that it reproduces key findings from a recent manual analysis of Chain-of-Thought (CoT) reasoning and also uncovers new insights that go beyond it, showing, for example, that in-context examples benefit coding & multimodal tasks but offer limited gains in math reasoning tasks compared to zero-shot CoT.Our automatically updatable dataset enables continuous tracking of target models by extracting evaluation studies as new data becomes available. Through LLMEvalDB and empirical analysis, we provide insights into LLMs while facilitating ongoing literature analyses of their behavior.

Post-training of Large Language Models often involves a pipeline of Supervised Finetuning (SFT) followed by Preference Finetuning (PFT) using methods like Direct Preference Optimization. Both stages require annotated data that are very different in structure and costs. We study how to optimally allocate a fixed training data budget between the two stages, through extensive experiments spanning four diverse tasks, multiple model sizes and various data annotation costs. Our findings reveal that just SFT on the base model dominates performance in low-data regimes (<1,000 annotated examples). With larger data-budgets, we observe that a combination of SFT and PFT, often with increasing portions allocated towards preference data yields optimal performance. However, completely eliminating SFT and running PFT directly on the base model yields suboptimal performance, described as the cold start problem on tasks like mathematics. We observe that this is due to the distribution shift arising from using DPO directly on the base model to elicit step-by-step reasoning. This limitation can be effectively addressed by allocating even a small portion (<10%) of the budget to SFT first, resulting in performance improvements of 15-20% on analytical benchmarks like GSM8k. These results provide actionable insights for researchers and practitioners optimizing model development under budget constraints, where high-quality data curation often represents a significant portion of the total costs of model development.

We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self-optimized routing. This allows for dynamic and capability-specific handling of various target tasks, enhancing overall capabilities, without extensive human-labeled data and added parameters. Our empirical results reveal that specializing LLMs may exhibit potential trade-offs in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such as knowledge, reasoning, math, and coding. It also consistently outperforms other methods, including instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.

Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions.Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources.Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data.In this paper, we investigate how to most efficiently use a fixed budget to build a compact model.Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small).We further investigate how the optimal budget allocated towards computation varies across scenarios.We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

A graph is a suitable data structure to represent the structural information of text.Recently, multi-hop question answering (MHQA) tasks, which require inter-paragraph/sentence linkages, have come to exploit such properties of a graph.Previous approaches to MHQA relied on leveraging the graph information along with the pre-trained language model (PLM) encoders.However, this trend exhibits the following drawbacks: (i) sample inefficiency while training in a low-resource setting; (ii) lack of reusability due to changes in the model structure or input.Our work proposes the Graph-Induced Transformer (GIT) that applies graphderived attention patterns directly into a PLM, without the need to employ external graph modules.GIT can leverage the useful inductive bias of graphs while retaining the unperturbed Transformer structure and parameters.Our experiments on HotpotQA successfully demonstrate both the sample efficient characteristic of GIT and its capacity to replace the graph modules while preserving model performance.

As DNNs improving state-of-the-art accuracy on many artificial intelligence (AI) applications such as computer vision processing for autonomous driving, the data processing bandwidth and power consumption between neural network accelerator and the off-chip memory are big challenge to enhance the compute performance metric TOPs/watt. To overcome the limited compute and energy resources in automobile environment, inferencing by PIM (Processing in Memory) or AiM (Accelerator in Memory) which deployed MAC(Multiply and Accumulation) units and activation function inside DRAM is one of the key solution by using multi bank parallelism and memory cell architecture. When memory technology equipped with analog logic inside mature in the near future, ultra-low power analog accelerator based neuromorphic computing architecture will lead the future autonomous driving solution.

The semantic matching capabilities of neural information retrieval can ameliorate synonymy and polysemy problems of symbolic approaches.However, neural models' dense representations are more suitable for re-ranking, due to their inefficiency.Sparse representations, either in symbolic or latent form, are more efficient with an inverted index.Taking the merits of the sparse and dense representations, we propose an ultra-high dimensional (UHD) representation scheme equipped with directly controllable sparsity.UHD's large capacity and minimal noise and interference among the dimensions allow for binarized representations, which are highly efficient for storage and search.Also proposed is a bucketing method, where the embeddings from multiple layers of BERT are selected/merged to represent diverse linguistic aspects.We test our models with MS MARCO and TREC CAR, showing that our models outperforms other sparse models.

Tag recommendation relies on either a ranking function for top-k tags or an autoregressive generation method.However, the previous methods neglect one of two seemingly conflicting yet desirable characteristics of a tag set: orderlessness and inter-dependency.While the ranking approach fails to address the interdependency among tags when they are ranked, the autoregressive approach fails to take orderlessness into account because it is designed to utilize sequential relations among tokens.We propose a sequence-oblivious generation method for tag recommendation, in which the next tag to be generated is independent of the order of the generated tags and the order of the ground truth tags occurring in training data.Empirical results on two different domains, Instagram and Stack Overflow, show that our method is significantly superior to the previous approaches.

Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years.While the previous models for numerical MRC are able to interpolate the learned numerical reasoning capabilities, it is not clear whether they can perform just as well on numbers unseen in the training dataset.Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle passages that contain out-of-range numbers.One of the key findings is that the models fail to extrapolate to unseen numbers.Presenting numbers as digit-by-digit input to the model, we also propose the E-digit number form that alleviates the lack of extrapolation in models and reveals the need to treat numbers differently from regular words in the text.Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC.

Advances in Question Answering (QA) research require additional datasets for new domains, languages, and types of questions, as well as for performance increases.Human creation of a QA dataset like SQuAD, however, is expensive.As an alternative, an unsupervised QA approach has been proposed so that QA training data can be generated automatically.However, the performance of unsupervised QA is much lower than that of supervised QA models.We identify two anomalies in the automatically generated questions and propose how they can be mitigated.We show our approach helps improve unsupervised QA significantly across a number of QA tasks. Context Generated Question..