We tackle Extreme Multi-Label Text Classification (XMTC), which involves assigning relevant labels to texts from a huge label space. Attempting to optimize the underexplored tail-head trade-off, we address the XMTC task through its core challenges of volume, skewness, and quality by proposing xCoRetriev, a novel two-stage retrieving and fusing ranking pipeline. Our pipeline addresses the volume challenge by dynamically slicing the large label space; it also tackles the skewness challenge by favoring the tail labels while fusing sparse and dense retrievers. Finally, xCoRetriev faces the quality challenge by enhancing the label space with Retrieval-Augmented Generated (RAG)-labels. Our experiments with four XMTC benchmarks with hundreds of thousands of text documents and labels against six state-of-the-art XMTC baselines demonstrate xCoRetriev's strengths in terms of: (i)~scalability for large label spaces, being among the most efficient methods at training and prediction; (ii)~effectiveness in the face of high skewness, with gains of up to 48% in propensity-scored metrics against the best state-of-the-art baselines; and (iii)~capability of handling very noisy datasets by exploiting RAG-labels.

Transformer-based deep learning is currently the state-of-the-art in many NLP and IR tasks. However, fine-tuning such Transformers for specific tasks, especially in scenarios of ever-expanding volumes of data with constant re-training requirements and budget constraints, is costly (computationally and financially) and energy-consuming. In this paper, we focus on Instance Selection (IS) - a set of methods focused on selecting the most representative documents for training, aimed at maintaining (or improving) classification effectiveness while reducing total time for training (or fine-tuning). We propose E2SC-IS -- Effective, Efficient, and Scalable Confidence-Based IS -- a two-step framework with a particular focus on Transformers and large datasets. E2SC-IS estimates the probability of each instance being removed from the training set based on scalable, fast, and calibrated weak classifiers. E2SC-IS also exploits iterative heuristics to estimate a near-optimal reduction rate. Our solution can reduce the training sets by 29% on average while maintaining the effectiveness in all datasets, with speedup gains up to 70%, scaling for very large datasets (something that the baselines cannot do).