Nora Hollenstein

VLDB 2026 Research / reviewers in the wild / expert

dblp:154/4482 · DBLP profile ↗

← Back

15ranked-venue papers

5as first author

11since 2021 · last 2026

0000-0001-7936-4170ORCID · verified

Domains — the database's venue-derived domains; a paper can count in several

Artificial intelligence and machine learning · 13 · 5 first-author · 9 since 2021Databases, data management, data science and information retrieval · 1 · 1 since 2021Graphics, computer vision, multimedia, augmented reality and games · 1 · 1 since 2021Human-computer interaction and ubiquitous computing · 1 · 1 since 2021

Year	Publication	Venue	Position
2026	The MultiplEYE Text Corpus: Towards a Diverse and Ever-Expanding Multilingual Text Corpus Ramune Kaspere, Anna Bondar, Sergiu Nisioi, Maja Stegenwallner-Schütz, Hanne B. Søndergaard Knudsen, Ana Matic Skoric, Eva Pavlinusic Vilus, Dorota Klimek-Jankowska, Chiara Tschirner, Not Battesta Soliva, Deborah N. Jakobi, Cui Ding, Dima Abu Romi, Cengiz Acartürk, Matilda Agdler, Anton Marius Alexandru, Mohd Faizan Ansari, Annalisa Arcidiacono, Elizabete Ausma Velta Barisa, Ana Bautista, Lisa Beinborn, Yevgeni Berzak, Nedeljka Bjelanovic, Anna Isabelle Bothmann, Jan Brasser, Caterina Cacioli, Anila Çepani, Ilze Ceple, Adelina Çerpja, Dalí Chirino, Jan Chromý, Alessandro Corona Mendozza, Iria de-Dios-Flores, Nazik Dinçtopal Deniz, Ana Dosen, Kristian Elersic, Inmaculada Fajardo, Zigmunds Freibergs, Angelina Ganebnaya, Jessica Gomes, Annjo Klungervik Greenall, Alba Haveriku, Anamaria Hodivoianu, Yu-Yin Hsu, Amanda Isaksen, Andreia Janeiro, Kristine M. Jensen de López, Aleksandar Jevremovic, Vojislav Jovanovic, Hanna Kedzierska, Nik Kharlamov, Sara Kosutar, Nelda Kote, Vanja Kovic, Izabela Krejtz, Thyra Krosness, Oleksandra Kuvshynova, Eilam Lavy, Ella Lion, Marta Lockiewicz, Kaidi Lõo, Paula Luegi, Mircea Mihai Marin, Clara Martin, Svitlana Matvieieva, Diane C. Mézière, Xavier Mínguez-López, Valeriia Modina, Jurgita Motiejuniene, Marie-Luise Müller, Tolgonai Nasipbek kyzy, Jamal Abdul Nasir, Johanne Sofie Krog Nedergård, Aysegül Özkan, Patrizia Paggio, Marijan Palmovic, Maria Christina Panagiotopoulou, Alberto Parola, Helena Pérez, Klaudia Petersen, Anja Podlesek, Eva Pospísilová, Marta Praulina, Mikulás Preininger, Loredana Punga, Diego Rossini, Spela Rot, Habib Sani Yahaya, Irina A. Sekerina, Anne Gabija Skadina, Jordi Solé i Casals, Lonneke van der Plas, Saara M. Varjopuro, Spyridoula Varlokosta, João Veríssimo, Oskari Juhapekka Virtanen, Nemanja Vracar, Mila Dimitrova-Vulchanova, Ahmad Mustapha Wali, Peizheng Wu, Nilgün Yücel, Stefan Frank, Nora Hollenstein, Lena A. Jäger, Somayeh Bakhtiari	LREC	105
2025	MultiplEYE: Creating a multilingual eye-tracking-while-reading corpus abstract Contains fulltext : 326363.pdf (Publisher’s version ) (Open Access) Deborah N. Jakobi, Maja Stegenwallner-Schütz, Nora Hollenstein, Cui Ding, Ramune Kaspere, Ana Matic Skoric, Eva Pavlinusic Vilus, Stefan Frank, Marie-Luise Müller, Kristine M. Jensen de López, Nik Kharlamov, Hanne B. Søndergaard Knudsen, Yevgeni Berzak, Ella Lion, Irina A. Sekerina, Cengiz Acartürk, Mohd Faizan Ansari, Katarzyna Harezlak, Pawel Kasprowski, Ana Bautista, Lisa Beinborn, Anna Bondar, Antonia Boznou, Leah Bradshaw, Jana Mara Hofmann, Thyra Krosness, Not Battesta Soliva, Anila Çepani, Kristina Cergol, Ana Dosen, Marijan Palmovic, Adelina Çerpja, Dalí Chirino, Jan Chromý, Vera Demberg, Iza Skrjanec, Nazik Dinçtopal Deniz, Inmaculada Fajardo, Mariola Giménez-Salvador, Xavier Mínguez-López, Maros Filip, Zigmunds Freibergs, Jessica Gomes, Andreia Janeiro, Paula Luegi, João Veríssimo, Sasho Gramatikov, Jana Hasenäcker, Alba Haveriku, Nelda Kote, Muhammad Mohsin Kamal, Hanna Kedzierska, Dorota Klimek-Jankowska, Sara Kosutar, Daniel Krakowczyk, Izabela Krejtz, Marta Lockiewicz, Kaidi Lõo, Jurgita Motiejuniene, Jamal Abdul Nasir, Johanne Sofie Krog Nedergård, Aysegül Özkan, Mikulás Preininger, Loredana Punga, David R. Reich, Chiara Tschirner, Spela Rot, Andreas Säuberli, Jordi Solé i Casals, Ekaterina Strati, Igor Svoboda, Evis Trandafili, Spyridoula Varlokosta, Mila Dimitrova-Vulchanova, Lena A. Jäger	ETRA	3
2024	Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations abstract Rationales in the form of manually annotated input spans usually serve as ground truth when evaluating explainability methods in NLP. They are, however, time-consuming and often biased by the annotation process. In this paper, we debate whether human gaze, in the form of webcam-based eye-tracking recordings, poses a valid alternative when evaluating importance scores. We evaluate the additional information provided by gaze data, such as total reading times, gaze entropy, and decoding accuracy with respect to human rationale annotations. We compare WebQAmGaze, a multilingual dataset for information-seeking QA, with attention and explainability-based importance scores for 4 different multilingual Transformer-based language models (mBERT, distil-mBERT, XLMR, and XLMR-L) and 3 languages (English, Spanish, and German). Our pipeline can easily be applied to other tasks and languages. Our findings suggest that gaze data offers valuable linguistic insights that could be leveraged to infer task difficulty and further show a comparable ranking of explainability methods to that of human rationales. Stephanie Brandl, Oliver Eberle, Anders Søgaard, Nora Hollenstein	LREC/COLING	5
2024	Reading Does Not Equal Reading: Comparing, Simulating and Exploiting Reading Behavior across Populations abstract Eye-tracking-while-reading corpora play a crucial role in the study of human language processing, and, more recently, have been leveraged for cognitively enhancing neural language models. A critical limitation of existing corpora is that they often lack diversity, comprising primarily native speakers. In this study, we expand the eye-tracking-while-reading dataset CopCo, which initially included only Danish L1 readers with and without dyslexia, by incorporating a new dataset of L2 readers with diverse L1 backgrounds. Thus, the extended CopCo corpus constitutes the first eye-tracking-while-reading dataset encompassing neurotypical L1 and L1 readers with dyslexia as well as L2 readers, all reading the same materials. We first provide extensive descriptive statistics of the extended CopCo corpus. Second, we investigate how different degrees of diversity of the training data affect a state-of-the-art generative model of eye movements in reading. Finally, we use this scanpath generation model for gaze-augmented language modeling and investigate the impact of diversity in the training data on the model’s performance on a range of NLP downstream tasks. The code can be found here: https://github.com/norahollenstein/copco-processing. David R. Reich, Shuwen Deng, Marina Björnsdóttir, Lena A. Jäger, Nora Hollenstein	LREC/COLING	5
2024	Eye-Tracking Features Masking Transformer Attention in Question-Answering Tasks abstract Eye movement features are considered to be direct signals reflecting human attention distribution with a low cost to obtain, inspiring researchers to augment language models with eye-tracking (ET) data. In this study, we select first fixation duration (FFD) and total reading time (TRT) as the cognitive signals to guide Transformer attention in question-answering (QA) tasks. We design three different ET attention masks based on the two features, either collected from human reading events or generated by a gaze-predicting model. We augment BERT and ALBERT models with attention masks structured based on the ET data. We find that augmenting a model with ET data carries linguistic features complementing the information captured by the model. It improves the models’ performance but compromises the stability. Different Transformer models benefit from different types of ET attention masks, while ALBERT performs better than BERT. Moreover, ET data collected from real-life reading events has better model augmenting ability than the model-predicted data. Leran Zhang, Nora Hollenstein	LREC/COLING	2
2023	Synthesizing Human Gaze Feedback for Improved NLP Performance abstract Integrating human feedback in models can improve the performance of natural language processing (NLP) models.Feedback can be either explicit (e.g.ranking used in training language models) or implicit (e.g. using human cognitive signals in the form of eyetracking).Prior eye tracking and NLP research reveal that cognitive processes, such as human scanpaths, gleaned from human gaze patterns aid in the understanding and performance of NLP models.However, the collection of real eyetracking data for NLP tasks is challenging due to the requirement of expensive and precise equipment coupled with privacy invasion issues.To address this challenge, we propose ScanTextGAN, a novel model for generating human scanpaths over text.We show that ScanTextGAN-generated scanpaths can approximate meaningful cognitive signals in human gaze patterns.We include synthetically generated scanpaths in four popular NLP tasks spanning six different datasets as proof of concept and show that the models augmented with generated scanpaths improve the performance of all downstream NLP tasks. Varun Khurana, Yaman Singla, Nora Hollenstein, Rajesh Kumar 0016, Balaji Krishnamurthy	EACL	3
2022	Interpreting Character Embeddings With Perceptual Representations: The Case of Shape, Sound, and Color abstract Character-level information is included in many NLP models, but evaluating the information encoded in character representations is an open issue.We leverage perceptual representations in the form of shape, sound, and color embeddings and perform a representational similarity analysis to evaluate their correlation with textual representations in five languages.This cross-lingual analysis shows that textual character representations correlate strongly with sound representations for languages using an alphabetic script, while shape correlates with featural scripts.We further develop a set of probing classifiers to intrinsically evaluate what phonological information is encoded in character embeddings.Our results suggest that information on features such as voicing are embedded in both LSTM and transformer-based representations. Sidsel Boldsen, Manex Agirrezabal, Nora Hollenstein	ACL (1)	3
2022	The Copenhagen Corpus of Eye Tracking Recordings from Natural Reading of Danish Texts abstract Eye movement recordings from reading are one of the richest signals of human language processing. Corpora of eye movements during reading of contextualized running text is a way of making such records available for natural language processing purposes. Such corpora already exist in some languages. We present CopCo, the Copenhagen Corpus of eye tracking recordings from natural reading of Danish texts. It is the first eye tracking corpus of its kind for the Danish language. CopCo includes 1,832 sentences with 34,897 tokens of Danish text extracted from a collection of speech manuscripts. This first release of the corpus contains eye tracking data from 22 participants. It will be extended continuously with more participants and texts from other genres. We assess the data quality of the recorded eye movements and find that the extracted features are in line with related research. The dataset available here: https://osf.io/ud8s5/. Nora Hollenstein, Maria Barrett, Marina Björnsdóttir	LREC	1
2022	Dynamic Human Evaluation for Relative Model Comparisons abstract Collecting human judgements is currently the most reliable evaluation method for natural language generation systems. Automatic metrics have reported flaws when applied to measure quality aspects of generated text and have been shown to correlate poorly with human judgements. However, human evaluation is time and cost-intensive, and we lack consensus on designing and conducting human evaluation experiments. Thus there is a need for streamlined approaches for efficient collection of human judgements when evaluating natural language generation systems. Therefore, we present a dynamic approach to measure the required number of human annotations when evaluating generated outputs in relative comparison settings. We propose an agent-based framework of human evaluation to assess multiple labelling strategies and methods to decide the better model in a simulation and a crowdsourcing case study. The main results indicate that a decision about the superior model can be made with high probability across different labelling strategies, where assigning a single random worker per task requires the least overall labelling effort and thus the least cost. Thórhildur Thorleiksdóttir, Cédric Renggli, Nora Hollenstein, Ce Zhang 0001	LREC	3
2021	Ease.ML: A Lifecycle Management System for Machine Learning Leonel Aguilar Melgar, David Dao, Shaoduo Gan, Nezihe Merve Gürel, Nora Hollenstein, Jiawei Jiang 0001, Bojan Karlas, Thomas Lemmin, Tian Li 0005, Yang Li 0106, Susie Xi Rao, Johannes Rausch, Cédric Renggli, Luka Rimanic, Maurice Weber, Shuai Zhang 0007, Zhikuan Zhao, Kevin Schawinski, Wentao Wu 0001, Ce Zhang 0001	CIDR	5
2021	Multilingual Language Models Predict Human Reading Behavior abstract Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, Lisa Beinborn. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021. Nora Hollenstein, Federico Pirovano, Ce Zhang 0001, Lena A. Jäger, Lisa Beinborn	NAACL-HLT	1
2020	ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation abstract We recorded and preprocessed ZuCo 2.0, a new dataset of simultaneous eye-tracking and electroencephalography during natural reading and during annotation. This corpus contains gaze and brain activity data of 739 English sentences, 349 in a normal reading paradigm and 390 in a task-specific paradigm, in which the 18 participants actively search for a semantic relation type in the given sentences as a linguistic annotation task. This new dataset complements ZuCo 1.0 by providing experiments designed to analyze the differences in cognitive processing between natural reading and annotation. The data is freely available here: https://osf.io/2urht/. Nora Hollenstein, Marius Troendle, Ce Zhang 0001, Nicolas Langer	LREC	1
2019	CogniVal: A Framework for Cognitive Word Embedding Evaluation abstract An interesting method of evaluating word representations is by how much they reflect the semantic representations in the human brain.However, most, if not all, previous works only focus on small datasets and a single modality.In this paper, we present the first multimodal framework for evaluating English word representations based on cognitive lexical semantics.Six types of word embeddings are evaluated by fitting them to 15 datasets of eyetracking, EEG and fMRI signals recorded during language processing.To achieve a global score over all evaluation hypotheses, we apply statistical significance testing accounting for the multiple comparisons problem.This framework is easily extensible and available to include other intrinsic and extrinsic evaluation methods.We find strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks. Nora Hollenstein, Antonio de la Torre, Nicolas Langer, Ce Zhang 0001	CoNLL	1
2018	Sequence Classification with Human Attention abstract Learning attention functions requires large volumes of data, but many NLP tasks simulate human behavior, and in this paper, we show that human attention really does provide a good inductive bias on many attention functions in NLP.Specifically, we use estimated human attention derived from eyetracking corpora to regularize attention functions in recurrent neural networks.We show substantial improvements across a range of tasks, including sentiment analysis, grammatical error detection, and detection of abusive language. Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, Anders Søgaard	CoNLL	3
2016	Inconsistency Detection in Semantic Annotation Nora Hollenstein, Nathan Schneider 0001, Bonnie L. Webber	LREC	1