Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries.Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens.However, in question answering, language models exhibit uneven utilization of their input context.They tend to favor the initial and final segments, resulting in a U-shaped performance pattern concerning where the answer is located within the input.This bias raises concerns, particularly in summarization where crucial content may be dispersed throughout the source document(s).Besides, in summarization, mapping facts from the source to the summary is not trivial as salient content is usually re-phrased.In this paper, we conduct the first comprehensive study on context utilization and position bias in summarization.Our analysis encompasses 6 LLMs, 10 datasets, and 5 evaluation metrics.We introduce a new evaluation benchmark called MiddleSum on the which we benchmark two alternative inference methods to alleviate position bias: hierarchical summarization and incremental summarization 1 .Metric Model CNN/DM XSum Reddit SAMSum Multi-X AVG Arxiv PubMed GovReport SummScreenFD Multi-N AVG ROUGE-2 Flan-UL2 -0.296 -0.124 0.048 -0.069 -0.201 -0.128 _ _ _ _ _ _ Llama-2-7B -0.160 -0.023 0.063 -0.059 -0.100 -0.056 0.022 -0.113 -0.109 -0.079 -0.210 -0.098 Llama-2-13B -0.166 -0.086 0.031 -0.078 -0.039 -0.068 -0.017 -0.081 -0.166 -0.139 -0.213 -0.123 Xgen-7B -0.228 -0.042 0.066 -0.039 -0.041 -0.056 0.028 -0.091 -0.405 0.063 -0.283 -0.138 Mistral-7B -0.289 -0.031 0.006 -0.024 -0.052 -0.078 -0.270 -0.279 -0.585 -0.132 -0.324 -0.318 GPT-3.5 -0.323 -0.027 -0.031 -0.097 0.088 -0.078 0.026 -0.093 -0.123 -0.061 -0.233 -0.097 BERTScore Flan-UL2 -0.331 -0.185 0.062 -0.144 -0.399 -0.

Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Conversational recommender systems (CRS) aim to recommend relevant items to users by eliciting user preference through natural language conversation.Prior work often utilizes external knowledge graphs for items' semantic information, a language model for dialogue generation, and a recommendation module for ranking relevant items.This combination of multiple components suffers from a cumbersome training process, and leads to semantic misalignment issues between dialogue generation and item recommendation.In this paper, we represent items in natural language and formulate CRS as a natural language processing task.Accordingly, we leverage the power of pre-trained language models to encode items, understand user intent via conversation, perform item recommendation through semantic matching, and generate dialogues.As a unified model, our PECRS (Parameter-Efficient CRS), can be optimized in a single stage, without relying on non-textual metadata such as a knowledge graph.Experiments on two benchmark CRS datasets, ReDial and INSPIRED, demonstrate the effectiveness of PECRS on recommendation and conversation.Our

Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset.These models are typically decoded with beam search to generate a unique summary.However, the search space is very large, and with the exposure bias, such decoding is not optimal.In this paper, we show that it is possible to directly train a secondstage model performing re-ranking on a set of summary candidates.Our mixture-of-experts SummaReranker learns to select a better candidate and consistently improves the performance of the base model.With a base PEGASUS, we push ROUGE scores by 5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-theart.Our code and checkpoints will be available at https://github.com/ntunlp/ SummaReranker.

Sequence-to-sequence deep neural models fine-tuned for abstractive summarization can achieve great performance on datasets with enough human annotations.Yet, it has been shown that they have not reached their full potential, with a wide gap between the top beam search output and the oracle beam.Recently, re-ranking methods have been proposed, to learn to select a better summary candidate.However, such methods are limited by the summary quality aspects captured by the first-stage candidates.To bypass this limitation, we propose a new paradigm in secondstage abstractive summarization called Sum-maFusion that fuses several summary candiates to produce a novel abstractive secondstage summary.Our method works well on several summarization datasets, improving both the ROUGE scores and qualitative properties of fused summaries.It is especially good when the candidates to fuse are worse, such as in the few-shot setup where we set a new state-of-the-art.We will make our code and checkpoints available at https: //github.com/ntunlp/SummaFusion/.