Modern large language models and chatbots based on them show impressive results in text generation and dialog tasks. At the same time, these models are subject to criticism in many aspects, e.g., they can generate hate speech and untrue and biased content. In this work, we show another problematic feature of such chatbots: they are echo chambers in the sense that they tend to agree with the opinions of their users. Social media, such as Facebook, was criticized for a similar problem and called an echo chamber. We experimentally test five LLM-based chatbots, which we feed with opinionated inputs. We annotate the chatbot answers whether they agree or disagree with the input. All chatbots tend to agree. However, the echo chamber effect is not equally strong. We discuss the differences between the chatbots and make the dataset publicly available.

In this paper, we investigate the use of large language models (LLMs) to enhance the editorial process of rewriting customer help pages.We introduce a German-language dataset comprising Frequently Asked Question-Answer pairs, presenting both raw drafts and their revisions by professional editors.On this dataset, we evaluate the performance of four large language models (LLM) through diverse prompts tailored for the rewriting task.We conduct automatic evaluations of content and text quality using ROUGE, BERTScore, and ChatGPT.Furthermore, we let professional editors assess the helpfulness of automatically generated FAQ revisions for editorial enhancement.Our findings indicate that LLMs can produce FAQ reformulations beneficial to the editorial process.We observe minimal performance discrepancies among LLMs for this task, and our survey on helpfulness underscores the subjective nature of editors' perspectives on editorial refinement.

Despite the recent progress, little is known about the features captured by state-of-the-art neural relation extraction (RE) models.Common methods encode the source sentence, conditioned on the entity mentions, before classifying the relation.However, the complexity of the task makes it difficult to understand how encoder architecture and supporting linguistic knowledge affect the features learned by the encoder.We introduce 14 probing tasks targeting linguistic properties relevant to RE, and we use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets, TACRED and SemEval 2010 Task 8. We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance.For example, adding contextualized word representations greatly increases performance on probing tasks with a focus on named entity and part-of-speech information, and yields better results in RE.In contrast, entity masking improves RE, but considerably lowers performance on entity type related probing tasks.

TACRED (Zhang et al., 2017) is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE).But, even with recent advances in unsupervised pretraining and knowledge enhanced neural RE, models still show a high error rate.In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement?And how do crowd annotations, dataset, and models contribute to this error rate?To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators.We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1.After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models.We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

High cost and time consumption are concurrent barriers for research and application of automated summarization. In order to explore options to overcome this barrier, we analyze the feasibility and appropriateness of micro-task crowdsourcing for evaluation of different summary quality characteristics and report an ongoing work on the crowdsourced evaluation of query-based extractive text summaries. To do so, we assess and evaluate a number of linguistic quality factors such as grammaticality, non-redundancy, referential clarity, focus and structure & coherence. Our first results imply that referential clarity, focus and structure & coherence are the main factors effecting the perceived summary quality by crowdworkers. Further, we compare these results using an initial set of expert annotations that is currently being collected, as well as an initial set of automatic quality score ROUGE for summary evaluation. Preliminary results show that ROUGE does not correlate with linguistic quality factors, regardless if assessed by crowd or experts. Further, crowd and expert ratings show highest degree of correlation when assessing low quality summaries. Assessments increasingly divert when attributing high quality judgments.