Retrieval-augmented generation (RAG) systems empower large language models (LLMs) to access external knowledge during inference.Recent advances have enabled LLMs to act as search agents via reinforcement learning (RL), improving information acquisition through multi-turn interactions with retrieval engines.However, existing approaches either optimize retrieval using search-only metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM to jointly reason and retrieve-entangling retrieval with generation and limiting the real search utility and compatibility with frozen or proprietary models.In this work, we propose s3, a lightweight, modelagnostic framework that decouples the searcher from the generator and trains the searcher using a Gain Beyond RAG reward: the improvement in generation accuracy over naïve RAG.s3 requires only 2.4k training samples to outperform baselines trained on over 70× more data, consistently delivering stronger downstream performance across six general QA and five medical QA benchmarks. 1

Hierarchical text classification aims to categorize each document into a set of classes in a label taxonomy, which is a fundamental web text mining task with broad applications such as web content analysis and semantic indexing. Most earlier works focus on fully or semi-supervised methods that require a large amount of human annotated data which is costly and time-consuming to acquire. To alleviate human efforts, in this paper, we work on hierarchical text classification with a minimal amount of supervision: using the sole class name of each node as the only supervision. Recently, large language models (LLM) have shown competitive performance on various tasks through zero-shot prompting, but this method performs poorly in the hierarchical setting because it is ineffective to include the large and structured label space in a prompt. On the other hand, previous weakly-supervised hierarchical text classification methods only utilize the raw taxonomy skeleton and ignore the rich information hidden in the text corpus that can serve as additional class-indicative features. To tackle the above challenges, we propose TELEClass, Taxonomy Enrichment and LLM-Enhanced weakly-supervised hierarchical text Classification, which combines the general knowledge of LLMs and task-specific features mined from an unlabeled corpus. TELEClass automatically enriches the raw taxonomy with class-indicative features for better label space understanding and utilizes novel LLM-based data annotation and generation methods specifically tailored for the hierarchical setting. Experiments show that TELEClass can significantly outperform previous baselines while achieving comparable performance to zero-shot prompting of LLMs with drastically less inference cost.

The field of open relation extraction (ORE) has recently observed significant advancement thanks to the growing capability of large language models (LLMs).Nevertheless, challenges persist when ORE is performed on specific topics.Existing methods give suboptimal results in five dimensions: factualness, topic relevance, informativeness, coverage, and uniformity.To improve topic-oriented ORE, we propose a zero-shot approach called Pri-ORE: Open Relation Extraction with a Priori seed generation.PriORE leverages the builtin knowledge of LLMs to maintain a dynamic seed relation dictionary for the topic.The dictionary is initialized by seed relations generated from topic-relevant entity types and expanded during contextualized ORE.PriORE then reduces the randomness in generative ORE by converting it to a more robust relation classification task.Experiments show the approach empowers better topic-oriented control over the generated relations and thus improves ORE performance along the five dimensions, especially on specialized and narrow topics. Dimension Extracted Relation Explanation Factualness gives toWrong given text and topic Relevance is exposed to Correct given text but not directly relevant to topic Informativeness form compound with Correct but conceptually too general given text and topic Coverage form powdery magnesium oxide Correct but too specific, covering too few instances Uniformity {be oxidized by, give electrons to, . . .} Multiple correct expressions extracted for the same relation

Entity set expansion, the task of expanding a small set of similar entities into a much larger set, is a vital step for downstream tasks such as named entity recognition, knowledge base construction and information retrieval. Existing entity set expansion methods were developed by mainly considering entities at coarse-grained levels, which encounter difficulties for entity set expansion at fine-grained levels, due to the subtlety on fine-grained type inference and semantic drifting. In this study, we propose an automated (i.e. without human annotation), fine-grained set expansion framework, FGExpan, which utilizes a taxonomy structure and a pre-trained language model to achieve high performance. To facilitate our testing, a new fine-grained set expansion dataset is also constructed. Experiments on this dataset and those used in previous studies show that FGExpan achieves significantly better performance (MAP up by 0.176) on finegrained types and also the state-of-the-art expansion quality on coarse-grained entity sets.

We propose REACTCLASS that automatically maps the low-level concrete chemical entities into the high-level reactant groups without human effort for training data annotation. REACTCLASS is designed to take two special characteristics of the chemical molecules into consideration. The first characteristic is that each chemical molecule can be represented in two modalities: a chemical name in the text and a molecular structure in the graph. We propose to use cross-modal supervision to automatically create the training data for chemical name classification in the text via molecular structure matching in the graph. The second characteristic is that there is a knowledge-aware subword correlation between the surface names of the chemical entities to be classified and that of the reactant groups as class labels. We propose to train a classification model based on the subword cross-attention map between each chemical name and the corresponding reaction group. Experiments demonstrate that REACTCLASS is highly effective, achieving state-of-the-art performance in classifying the chemical names into human-defined reactant groups without requiring human effort for training data annotation.

Automated knowledge discovery from trending chemical literature is essential for more efficient biomedical research. How to extract detailed knowledge about chemical reactions from the core chemistry literature is a new emerging challenge that has not been well studied. In this paper, we study the new problem of fine-grained chemical entity typing, which poses interesting new challenges especially because of the complex name mentions frequently occurring in chemistry literature and graphic representation of entities. We introduce a new benchmark data set (CHEMET) to facilitate the study of the new task and propose a novel multi-modal representation learning framework to solve the problem of fine-grained chemical entity typing by leveraging external resources with chemical structures and using cross-modal attention to learn effective representation of text in the chemistry domain. Experiment results show that the proposed framework outperforms multiple state-of-the-art methods.1

Scientific literature analysis needs fine-grained named entity recognition (NER) to provide a wide range of information for scientific discovery.For example, chemistry research needs to study dozens to hundreds of distinct, fine-grained entity types, making consistent and accurate annotation difficult even for crowds of domain experts.On the other hand, domain-specific ontologies and knowledge bases (KBs) can be easily accessed, constructed, or integrated, which makes distant supervision realistic for fine-grained chemistry NER.In distant supervision, training labels are generated by matching mentions in a document with the concepts in the knowledge bases (KBs).However, this kind of KB-matching suffers from two major challenges: incomplete annotation and noisy annotation.We propose CHEMNER, an ontologyguided, distantly-supervised method for finegrained chemistry NER to tackle these challenges.It leverages the chemistry type ontology structure to generate distant labels with novel methods of flexible KB-matching and ontology-guided multi-type disambiguation.It significantly improves the distant label generation for the subsequent sequence labeling model training.We also provide an expertlabeled, chemistry NER dataset with 62 finegrained chemistry types (e.g., chemical compounds and chemical reactions).Experimental results show that CHEMNER is highly effective, outperforming substantially the stateof-the-art NER methods (with .25 absolute F1 score improvement).

Large-scale, online Data Science (DS) courses and degree programs are becoming increasingly common due to the global rise in popularity and demand for data scientists. Although project-based learning is integral to gaining hands-on experience in DS education, providing fair, timely, and high-quality feedback on varied projects for a large number of diverse students is challenging. To address those challenges in scaling up the assessment of DS group projects, we integrated multiple techniques, such as rapid feedback, peer grading, graders as meta-reviewers, etc. We present a case study of deploying those strategies for group projects in a large online DS course titled Text Information Systems offered in Fall, 2020. We synthesize our findings from analyzing student and grader survey responses, and share useful lessons and future work.

Given a query, unlike traditional IR that finds relevant documents or entities, in this work, we focus on retrieving both entities and their connections for insightful knowledge summarization. For example, given a query "computer vision'' on a CS literature corpus, rather than returning a list of relevant entities like "cnn'', "imagenet'' and "svm'', we are interested in the connections among them, and furthermore, the evolution patterns of such connections along particular ordinal dimensions such as time. Particularly, we hope to provide structural knowledge relevant to the query, such as "svm'' is related to "imagenet'' but not "cnn''. Moreover, we aim to model the changing trends of the connections, such as "cnn'' becomes highly related to "imagenet'' after 2010, which enables the tracking of knowledge evolutions. In this work, to facilitate such a novel insightful search system, we propose SetEvolve, which is a unified framework based on nonparanomal graphical models for evolutionary network construction from large text corpora. Systematic experiments on synthetic data and insightful case studies on real-world corpora demonstrate the utility of SetEvolve.

Literature search is critical for any scientific research. Different from Web or general domain search, a large portion of queries in scientific literature search are entity-set queries, that is, multiple entities of possibly different types. Entity-set queries reflect user's need for finding documents that contain multiple entities and reveal inter-entity relationships and thus pose non-trivial challenges to existing search algorithms that model each entity separately. However, entity-set queries are usually sparse (i.e., not so repetitive), which makes ineffective many supervised ranking models that rely heavily on associated click history. To address these challenges, we introduce SetRank, an unsupervised ranking framework that models inter-entity relationships and captures entity type information. Furthermore, we develop a novel unsupervised model selection algorithm, based on the technique of weighted rank aggregation, to automatically choose the parameter settings in SetRank without resorting to a labeled validation set. We evaluate our proposed unsupervised approach using datasets from TREC Genomics Tracks and Semantic Scholar's query log. The experiments demonstrate that SetRank significantly outperforms the baseline unsupervised models, especially on entity-set queries, and our model selection algorithm effectively chooses suitable parameter settings.

Motivation: Several large-scale efforts have been made to collect gene expression signatures from a variety of biological conditions, such as response of cell lines to treatment with drugs, or tumor samples with different characteristics. These gene signature collections are utilized through bioinformatics tools for 'signature matching', whereby a researcher studying an expression profile can identify previously cataloged biological conditions most related to their profile. Signature matching tools typically retrieve from the collection the signature that has highest similarity to the user-provided profile. Alternatively, classification models may be applied where each biological condition in the signature collection is a class label; however, such models are trained on the collection of available signatures and may not generalize to the novel cellular context or cell line of the researcher's expression profile. Results: We present an advanced multi-way classification algorithm for signature matching, called SigMat, that is trained on a large signature collection from a well-studied cellular context, but can also classify signatures from other cell types by relying on an additional, small collection of signatures representing the target cell type. It uses these 'tuning data' to learn two additional parameters that help adapt its predictions for other cellular contexts. SigMat outperforms other similarity scores and classification methods in identifying the correct label of a query expression profile from as many as 244 or 500 candidate classes (drug treatments) cataloged by the LINCS L1000 project. SigMat retains its high accuracy in cross-cell line applications even when the amount of tuning data is severely limited. Availability and implementation: SigMat is available on GitHub at https://github.com/JinfengXiao/SigMat. Supplementary information: Supplementary data are available at Bioinformatics online.

In the literature, two series of models have been proposed to address prediction problems including classification and regression. Simple models, such as generalized linear models, have ordinary performance but strong interpretability on a set of simple features. The other series, including tree-based models, organize numerical, categorical and high dimensional features into a comprehensive structure with rich interpretable information in the data. In this paper, we propose a novel Discriminative Pattern-based Prediction framework (DPPred) to accomplish the prediction tasks by taking their advantages of both effectiveness and interpretability. Specifically, DPPred adopts the concise discriminative patterns that are on the prefix paths from the root to leaf nodes in the tree-based models. DPPred selects a limited number of the useful discriminative patterns by searching for the most effective pattern combination to fit generalized linear models. Extensive experiments show that in many scenarios, DPPred provides competitive accuracy with the state-of-the-art as well as the valuable interpretability for developers and experts. In particular, taking a clinical application dataset as a case study, our DPPred outperforms the baselines by using only 40 concise discriminative patterns out of a potentially exponentially large set of patterns.