Generative transformer-based models have reached cutting-edge performance in long document summarization. Nevertheless, this task is witnessing a paradigm shift in developing ever-increasingly computationally-hungry solutions, focusing on effectiveness while ignoring the economic, environmental, and social costs of yielding such results. Accordingly, such extensive resources impact climate change and raise barriers to small and medium organizations distinguished by low-resource regimes of hardware and data. As a result, this unsustainable trend has lifted many concerns in the community, which directs the primary efforts on the proposal of tools to monitor models' energy costs. Despite their importance, no evaluation measure considering models' eco-sustainability exists yet. In this work, we propose Carburacy, the first carbon-aware accuracy measure that captures both model effectiveness and eco-sustainability. We perform a comprehensive benchmark for long document summarization, comparing multiple state-of-the-art quadratic and linear transformers on several datasets under eco-sustainable regimes. Finally, thanks to Carburacy, we found optimal combinations of hyperparameters that let models be competitive in effectiveness with significantly lower costs.

Although current summarization models can process increasingly long text sequences, they still struggle to capture salient related information spread across the lengthy size of inputs with few labeled training instances. Today’s research still relies on standard input truncation without considering graph-based modeling of multiple semantic units to summarize only crucial facets. This paper proposes G-SEEK, a graph-based summarization of extracted essential knowledge. By representing the long source with a heterogeneous graph, our method extracts and provides salient sentences to an abstractive summarization model to generate the summary. Experimental results in low-resource scenarios, distinguished by data scarcity, reveal that G-SEEK consistently improves both the long- and multi-document summarization performance and accuracy across several datasets.

Although current state-of-the-art Transformerbased solutions succeeded in a wide range for single-document NLP tasks, they still struggle to address multi-input tasks such as multidocument summarization.Many solutions truncate the inputs, thus ignoring potential summary-relevant contents, which is unacceptable in the medical domain where each information can be vital.Others leverage linear model approximations to apply multi-input concatenation, worsening the results because all information is considered, even if it is conflicting or noisy with respect to a shared background.Despite the importance and social impact of medicine, there are no ad-hoc solutions for multi-document summarization.For this reason, we propose a novel discriminative marginalized probabilistic method (DAMEN) trained to discriminate critical information from a cluster of topic-related medical documents and generate a multi-document summary via token probability marginalization.Results prove we outperform the previous state-of-theart on a biomedical dataset for multi-document summarization of systematic literature reviews.Moreover, we perform extensive ablation studies to motivate the design choices and prove the importance of each module of our method.1

The latest batch of research has equipped language models with the ability to attend over relevant and factual information from nonparametric external sources, drawing a complementary path to architectural scaling.Besides mastering language, exploiting and contextualizing the latent world knowledge is crucial in complex domains like biomedicine.However, most works in the field rely on general-purpose models supported by databases like Wikipedia and Books.We introduce BIOREADER 1 , the first retrieval-enhanced text-to-text model for biomedical natural language processing.Our domain-specific T5-based solution augments the input prompt by fetching and assembling relevant scientific literature chunks from a neural database with ≈60 million tokens centered on PubMed.We fine-tune and evaluate BIORE-ADER on a broad array of downstream tasks, significantly outperforming several state-of-theart methods despite using up to 3x fewer parameters.In tandem with extensive ablation studies, we show that domain knowledge can be easily altered or supplemented to make the model generate correct predictions bypassing the retraining step and thus addressing the literature overload issue.