The diversification of information access systems, from RAG to autonomous agents, creates a critical need for comparative user studies. However, the technical overhead to deploy and manage these distinct systems is a major barrier. We present UXLab ,, an open-source system for web-based user studies that addresses this challenge. Its core is a web-based dashboard enabling the complete, no-code configuration of complex experimental designs. Researchers can visually manage the full study, from recruitment to comparing backends like traditional search, vector databases, and LLMs. We demonstrate UXLab’s value via a micro case study comparing user behavior with RAG versus an autonomous agent. UXLab allows researchers to focus on experimental design and analysis, supporting future multi-modal interaction research.

A fundamental tension exists between the demand for sophisticated AI assistance in web search and the need for user data privacy. Current centralized models require users to transmit sensitive browsing data to external services, which limits user control. In this paper, we present a browser extension that provides a viable in-browser alternative. We introduce a hybrid architecture that functions entirely on the client side, combining two components: (1) an adaptive probabilistic model that learns a user’s behavioral policy from direct feedback, and (2) a Small Language Model (SLM), running in the browser, which is grounded by the probabilistic model to generate context-aware suggestions. To evaluate this approach, we conducted a three-week longitudinal user study with 18 participants. Our results show that this privacy-preserving approach is highly effective at adapting to individual user behavior, leading to measurably improved search efficiency. This work demonstrates that sophisticated AI assistance is achievable without compromising user privacy or data control.

User simulators are essential for evaluating search systems, but they primarily copy user actions without understanding the underlying thought process. This gap exists since large-scale interaction logs record what users do, but not what they might be thinking or feeling, such as confusion or satisfaction. To solve this problem, we present a framework to infer cognitive traces from behavior logs. Our method uses a multi-agent system grounded in Information Foraging Theory (IFT) and human expert judgment. These traces improve model performance on tasks like forecasting session outcomes and user struggle recovery. We release a collection of annotations for several public datasets, including AOL and Stack Overflow, and an open-source tool that allows researchers to apply our method to their own data ( https://github.com/searchsim-org/cognitive-traces ). This work provides the tools and data needed to build more human-like user simulators and to assess retrieval systems on user-oriented dimensions of performance.

LLM-based chatbots are increasingly augmented with retrieval mechanisms operating over web-scale corpora. Evaluating the effectiveness of these retrieval components is challenging, as explicit relevance judgments are often unavailable. Query performance prediction (QPP) addresses this limitation by providing unsupervised estimates of retrieval effectiveness. However, existing QPP methods assume a static corpus and do not account for the impact of corpus growth on query performance. In this work, we extend the QPP paradigm by studying query performance degradation under corpus inflation in dense retrieval systems. Using tiered corpora with fixed relevance judgments, we analyze how query effectiveness evolves as the corpus (index) size increases and evaluate the ability of established score-based and embedding-based post-retrieval QPP methods to predict such degradation. Our findings show that the reliability of these predictors is dependent on the dataset. We propose simple adaptations to established QPP measures, most notably a top-k vs background Wasserstein distance measure, which yield more consistent associations with degradation and outperform their original counterparts. These findings highlight limitations of several QPP approaches in large-scale, continuously expanding retrieval environments and motivate the development of corpus-growth-aware QPP measures.

We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss.

Retrieval-augmented generation (RAG) systems are frequently evaluated via fact-based metrics, yet standard implementations retrieve passages or static propositions. This unit mismatch between evaluation and retrieval objects hinders maintenance when corpora evolve and fails to capture superseded facts or source disagreements. We propose NuggetIndex, a retrieval system that stores atomic information units as managed records, so called nuggets. Each record maintains links to evidence, a temporal validity interval, and a lifecycle state. By filtering invalid or deprecated nuggets prior to ranking, the system prevents the inclusion of outdated information. We evaluate the approach using a nuggetized MS MARCO subset, a temporal Wikipedia QA dataset, and a multi-hop QA task. Against passage and unmanaged proposition retrieval baselines, NuggetIndex improves nugget recall by 42%, increases temporal correctness by 9 percentage points without the recall collapse observed in time-filtered baselines, and reduces conflict rates by 55%. The compact nugget format reduces generator input length by 64% while enabling lightweight index structures suitable for browser-based and resource-constrained deployment. We release our implementation, datasets, and evaluation scripts

Training trustworthy agentic LLMs requires data that shows the grounded reasoning process, not just the final answer. Existing datasets fall short: question-answering data is outcome-only, chain-of-thought data is not tied to specific documents, and web-agent datasets track interface actions rather than the core retrieval and synthesis steps of a RAG workflow. We introduce AgentSim, an open-source platform for simulating RAG agents. It generates verifiable, stepwise traces of agent reasoning over any document collection. AgentSim uses a policy to ensure the agent widely explores the document set. It combines a multi-model validation pipeline with an active human-in-the-loop process. This approach focuses human effort on difficult steps where models disagree. Using AgentSim, we construct and release the Agent-Trace Corpus (ATC), a large collection of grounded reasoning trajectories spanning three established IR benchmarks. We make three contributions: (1) the AgentSim platform with two mechanisms, Corpus-Aware Seeding and Active Validation, that improve trace diversity and quality; (2) the Agent-Trace Corpus (ATC), over 103,000 verifiable reasoning steps spanning three IR benchmarks, with 100% grounding rate on substantive answers; and (3) a comparative behavioral analysis revealing systematic differences in how state-of-the-art models approach information seeking. Platform, toolkit, and corpus are publicly available.

User simulation is a valuable methodology for evaluation in Information Retrieval (IR), enabling low-cost experimentation and counterfactual analysis. However, existing simulation frameworks are primarily code-centric libraries that require substantial setup effort, which limits adoption and hinders reproducibility. The bottleneck is not the simulation engines themselves, but the lack of infrastructure connecting experiment design, execution, and sharing into a single verifiable workflow. This paper introduces IIRSim Studio, a web-based workbench that addresses this gap through four contributions: (1) a visual environment for composing simulation pipelines on top of simulation frameworks, serving both novices learning simulation concepts and experts piloting large-scale experiments; (2) a component lifecycle that supports authoring, versioning, and sharing custom simulation components through Git-backed storage and runtime injection; (3) a provenance model based on experiment bundles and environment templates that makes the scope of replication explicit; and (4) a shared-task workflow, demonstrated through the re-deployment of a Sim4IA micro-task. IIRSim Studio is available as a hosted service and as a portable containerized deployment.

The rapid adoption of Large Language Models (LLMs) for search engines and fact-checking platforms necessitates enhancing their output accuracy.Retrieval Augmented Generation (RAG) mitigates hallucinations but requires semantically rich repositories like Wikidata.However, there is a lack of high-quality data to fine-tune LLMs for querying such knowledge bases.To address this gap, we propose a curated dataset with 2,771 unique queries for fine-tuning LLMs to generate accurate and syntactically valid SPARQL queries from natural language instructions.This dataset, customized for interaction with Wikidata, also serves as a robust benchmark for text-to-SPARQL task evaluation.Key findings show that models generally perform better on queries with lower complexity.

Embedding models are central to dense retrieval, semantic search, and recommendation systems, but their size often makes them impractical to deploy in resource-constrained environments such as browsers or edge devices. While smaller embedding models offer practical advantages, they typically underperform compared to their larger counterparts. To bridge this gap, we demonstrate that concatenating the raw embedding vectors of multiple small models can outperform a single larger baseline on standard retrieval benchmarks. To overcome the resulting high dimensionality of naive concatenation, we introduce a lightweight unified decoder trained with a Matryoshka Representation Learning (MRL) loss. This decoder maps the high-dimensional joint representation to a low-dimensional space, preserving most of the original performance without fine-tuning the base models. We also show that while concatenating more base models yields diminishing gains, the robustness of the decoder's representation under compression and quantization improves. Our experiments show that, on a subset of MTEB retrieval tasks, our concat-encode-quantize pipeline recovers 89% of the original performance with a 48× compression factor when the pipeline is applied to a concatenation of four small embedding models.

Simulating nuanced user experiences within complex interactive search systems poses distinct challenge for traditional methodologies, which often rely on static user proxies or, more recently, on standalone large language model (LLM) agents that may lack deep, verifiable grounding. The true dynamism and personalization inherent in human-computer interaction demand a more integrated approach. This work introduces UXSim, a novel framework that integrates both approaches. It leverages grounded data from traditional simulators to inform and constrain the reasoning of an adaptive LLM agent. This synthesis enables more accurate and dynamic simulations of user behavior while also providing a pathway for the explainable validation of the underlying cognitive processes.

Rhetorical figures play an important role in our communication. They are used to convey subtle, implicit meaning, or to emphasize statements. We notice them in hate speech, fake news, and propaganda. By improving the systems for computational detection of rhetorical figures, we can also improve tasks such as hate speech and fake news detection, sentiment analysis, opinion mining, or argument mining. Unfortunately, there is a lack of annotated data, as well as qualified annotators that would help us build large corpora to train machine learning models for the detection of rhetorical figures. The situation is particularly difficult in languages other than English, and for rhetorical figures other than metaphor, sarcasm, and irony. To overcome this issue, we develop a web application called “Find your Figure” that facilitates the identification and annotation of German rhetorical figures. The application is based on the German Rhetorical ontology GRhOOT which we have specially adapted for this purpose. In addition, we improve the user experience with Retrieval Augmented Generation (RAG). In this paper, we present the restructuring of the ontology, the development of the web application, and the built-in RAG pipeline. We also identify the optimal RAG settings for our application. Our approach is one of the first to practically use rhetorical ontologies in combination with RAG and shows promising results.

Retrieval-Augmented Generation (RAG) extends LLMs with external knowledge, typically from continuous text sources. In this work, we explore the application of RAG to ontologies to support ontology engineers and users in querying structured knowledge. To demonstrate the generalizability of our approach, we evaluate RAG on three ontologies from the domains of video games, IoT infrastructure, and rhetorical figures. Two of the ontologies are in English, while one is in German. Our study compares chunking strategies, chunk sizes, and the use of rerankers. We observe that advanced chunking techniques offer limited performance gains due to the inherently modular structure of ontologies. Additionally, we compare RAG on ontological document structure with the same information in a continuous format and reveal that the performance of RAG is not negatively affected by the ontological document structure. An advantage of using ontologies is that they can be evaluated against their competency questions that are typically developed alongside and define the intended use. With the Ragas evaluation framework, we achieve promising results across domains and languages. To address challenges regarding verbose LLM outputs that decrease performance, we propose a new metric Answer Average Match (AAM), which is specifically designed for the evaluation of RAG on ontologies. Our findings highlight the viability of ontology-aware RAG and offer practical insights for its optimization and evaluation.

We propose a lightweight and adaptable model tailored to German argument mining in two dissimilar domains. It offers improvements over the state-of-the-art, as well as over recent LLM approaches. Specifically, we achieve an 11.5 % absolute improvement across all classes of the SPWSLE dataset as well as 3.5% absolute detection performance boost across all datasets of the CIMT collection. We show that for both tasks, the hitherto disjoint steps of argument detection followed by argument classification can be merged with no loss in quality, resulting in a greater than$2 x$training and inference speed-up. For the naive baseline, this joint scheme also significantly improves results. For CIMT we propose realistic evaluation that incorporates actual detection results instead of simulating perfect detection. We also investigate the transferability between and to new datasets and find it to be higher than in previous work and in certain cases even better than within the dataset. A super model trained on all other datasets adapts to unseen data without re-training or performance loss.

Grades deliver important quantitative feedback on student learning, but grading can be time-consuming for large student cohorts. Recent advances in large language models (LLMs) offer promising opportunities for automating this process. Still, a systematic examination of different models and prompting paradigms remains lacking. To fill this gap, we compare the performance of 23 closed-source and open-source LLMs, testing the best-performing six across five different prompt paradigms including zero-shot, pre-written sample solutions with an integrated grading rubric, few-shot examples, and retrievalaugmented generation (RAG) based on textbook excerpts. We hence investigate the extent to which external information is necessary for reliable grading outcomes. As a benchmark, we include four fine-tuned encoder models. Our study focuses on German-language undergraduate economics courses, using an open macroeconomics (Macro) and a novel institutional economics (Insti) dataset graded by three human examiners. LLMhuman agreement reaches Quadratic Weighted Kappas of up to 0.811 for Macro and 0.687 for Insti, with the best LLM-prompt combinations nearing fine-tuned encoder model performance for Macro. Providing a sample solution with a grading rubric emerges as a crucial prompting feature, as it enables near-human performance even in zero-shot setups, particularly for easier-to-grade answers. Adding few-shot examples further improves grading for difficult answers. Few-shot examples without a sample solution, RAG, or relying solely on LLMs' inherent knowledge, on the other hand, perform significantly worse.

We present WebFAQ, a large-scale collection of open-domain question answering datasets derived from FAQ-style schema.org annotations. In total, the data collection consists of 96 million natural question-answer (QA) pairs across 75 languages, including 47 million (49%) non-English samples. WebFAQ further serves as the foundation for 49 monolingual retrieval benchmarks with a total size of 11.2 million QA pairs (5.9 million non-English). These datasets are carefully curated through refined filtering and near-duplicate detection, yielding high-quality resources for training and evaluating multilingual dense retrieval models. To empirically confirm WebFAQ's efficacy, we use the collected QAs to fine-tune an in-domain pretrained XLM-RoBERTa model. Through this process of dataset-specific fine-tuning, the model achieves significant retrieval performance gains, which generalize - beyond WebFAQ - to other multilingual retrieval benchmarks evaluated in zero-shot setting. Last but not least, we utilize WebFAQ to construct a set of QA-aligned bilingual corpora spanning over 1000 language pairs using state-of-the-art bitext mining and automated LLM-assessed translation evaluation. Due to our advanced, automated method of bitext dataset generation, the resulting bilingual corpora demonstrate higher translation quality compared to similar datasets. WebFAQ and all associated resources are publicly available on GitHub and HuggingFace.

Rhetorical figures play an important role in influencing readers and listeners. Some of these word constructs that deviate from the usual language structure are known to be persuasive – antithesis is one of them. This figure combines parallel phrases with opposite ideas or words to highlight a contradiction. By identifying this figure, persuasive actors can be better identified. For this task, we create an annotated German dataset for antithesis detection. The dataset consists of posts from a Telegram channel criticizing the COVID-19 politics in Germany. Furthermore, we propose a three-block pipeline approach to detect the figure antithesis using large language models. Our pipeline splits the text into phrases, identifies phrases with a syntactically parallel structure, and detects if these parallel phrase pairs present opposing ideas by fine-tuning the German ELECTRA model, a state-of-the-art deep learning model for the German language. Furthermore, we compare the results with multilingual BERT and German BERT. Our novel approach outperforms the state-of-the-art methods (F1-score of 50.43 %) for antithesis detection by achieving an F1-score of 65.11 %.

Supervised machine learning often encounters concept drift, where the data distribution changes over time, degrading model performance. Existing drift detection methods focus on identifying these shifts but often overlook the challenge of acquiring labeled data for model retraining after a shift occurs. We present the Strategy for Drift Sampling (SUDS), a novel method that selects homogeneous samples for retraining using existing drift detection algorithms, thereby enhancing model adaptability to evolving data. SUDS seamlessly integrates with current drift detection techniques. We also introduce the Harmonized Annotated Data Accuracy Metric (HADAM), a metric that evaluates classifier performance about the quantity of annotated data required to achieve the stated performance, thereby taking into account the difficulty of acquiring labeled data. Our contributions are twofold: SUDS combines drift detection with strategic sampling to improve the retraining process, and HADAM provides a metric that balances classifier performance with the amount of labeled data, ensuring efficient resource utilization. Empirical results demonstrate the efficacy of SUDS in optimizing labeled data use in dynamic environments, significantly improving the performance of machine learning applications in real-world scenarios. Our code is open source and available at https://github.com/cfellicious/SUDS/

Abstract Web search is a crucial technology for the digital economy. Dominated by a few gatekeepers focused on commercial success, however, web publishers have to optimize their content for these gatekeepers, resulting in a closed ecosystem of search engines as well as the risk of publishers sacrificing quality. To encourage an open search ecosystem and offer users genuine choice among alternative search engines, we propose the development of an Open Web Index (OWI). We outline six core principles for developing and maintaining an open index, based on open data principles, legal compliance, and collaborative technology development. The combination of an open index with what we call declarative search engines will facilitate the development of vertical search engines and innovative web data products (including, e.g., large language models), enabling a fair and open information space. This framework underpins the EU‐funded project OpenWebSearch.EU, marking the first step towards realizing an Open Web Index.

The use of BERT, one of the most popular language models, has led to improvements in many Natural Language Processing (NLP) tasks. One such task is Named Entity Recognition (NER) i.e. automatic identification of named entities such as location, person, organization, etc. from a given text. It is also an important base step for many NLP tasks such as information extraction and argumentation mining. Even though there is much research done on NER using BERT and other popular language models, the same is not explored in detail when it comes to Legal NLP or Legal Tech. Legal NLP applies various NLP techniques such as sentence similarity or NER specifically on legal data. There are only a handful of models for NER tasks using BERT language models, however, none of these are aimed at legal documents in German. In this paper, we fine-tune a popular BERT language model trained on German data (German BERT) on a Legal Entity Recognition (LER) dataset. To make sure our model is not overfitting, we performed a stratified 10-fold cross-validation. The results we achieve by fine-tuning German BERT on the LER dataset outperform the BiLSTM-CRF+ model used by the authors of the same LER dataset. Finally, we make the model openly available via HuggingFace.

The field of legal Natural Language Processing faces a lot of challenges due to the unavailability of properly structured datasets. One such instance is the need for a dataset that not only separates different parts of legal references, such as an article or paragraph number but also provides information about what a particular legal reference dictates. Having access to such a dataset can provide easy access to researchers working on experiments such as context similarity between law texts and legal cases that refer to a particular law. In this paper, we present a dataset of 2944 legal references in German law that are manually annotated by law experts. This dataset has 21 properties for each law reference in the dataset, such as Buch, Teil, Titel, Untertitel, etc. It also provides the complete text of each law reference in the dataset, along with specific paragraph text mentioned in the law reference. Furthermore, using this dataset together with Open Legal Data, we perform a law reference prediction task to compare the performance between predicting full law reference and only the base law reference.

Existing wordnets mainly focus on synonyms, while antonyms have often been neglected, especially in wordnets in languages other than English.In this paper, we show how regular expressions are used to generate an antonym resource for German by using Wiktionary as a source.This resource contains antonyms for 45499 words.The antonyms can be used to extend existing wordnets.We show that this is important by comparing our antonym resource to the antonyms in OdeNet, the only freely available German wordnet that contains antonyms for 3059 words.We demonstrate that antonyms are relevant for the detection of the rhetorical figure antithesis.This figure has been known to influence the audience by creating contradiction and using a parallel sentence structure combined with antonyms.We first detect parallelism with part-of-speech tags and then apply our rule-based antithesis detection algorithm to a dataset of the messenger service Telegram.We evaluate our approach and achieve a precision of 57 % and a recall of 45 % thus overcoming the existing approaches.

Fraud detection systems (FDS) mainly perform two tasks: (i) real-time detection while the payment is being processed and (ii) posterior detection to block the card retrospectively and avoid further frauds. Since human verification is often necessary and the payment processing time is limited, the second task manages the largest volume of transactions. In the literature, fraud detection challenges and algorithms performance are widely studied but the very formulation of the problem is never disrupted: it aims at predicting if a transaction is fraudulent based on its characteristics and the past transactions of the cardholder. Yet, in posterior detection, verification often takes days, so new payments on the card become available before a decision is taken. This is our motivation to propose a new paradigm: posterior fraud detection with "future" information. We start by providing evidence of the on-time availability of subsequent transactions, usable as extra context to improve detection. We then design a Bidirectional LSTM to make use of these transactions. On a real-world dataset with over 30 million transactions, it achieves higher performance than a regular LSTM, which is the state-of-the-art classifier for fraud detection that only uses the past context. We also introduce new metrics to show that the proposal catches more frauds, more compromised cards, and based on their earliest frauds. We believe that future works on this new paradigm will have a significant impact on the detection of compromised cards.

Simulated user retrieval system interactions enable studies with controlled user behavior. To this end, the SimIIR framework offers static, rule-based methods. We present an extended SimIIR 2.0 version with new components for dynamic user type-specific Markov model-based interactions and more realistic query generation. A flexible modularization ensures that the SimIIR 2.0 framework can serve as a platform to implement, combine, and run the growing number of proposed search behavior and query simulation ideas.

GRhOOT, the German RhetOrical OnTology, is a domain ontology of 110 rhetorical figures in the German language. The overall goal of building an ontology of rhetorical figures in German is not only the formal representation of different rhetorical figures, but also allowing for their easier detection, thus improving sentiment analysis, argument mining, detection of hate speech and fake news, machine translation, and many other tasks in which recognition of non-literal language plays an important role. The challenge of building such ontologies lies in classifying the figures and assigning adequate characteristics to group them, while considering their distinctive features. The ontology of rhetorical figures in the Serbian language was used as a basis for our work. Besides transferring and extending the concepts of the Serbian ontology, we ensured completeness and consistency by using description logic and SPARQL queries. Furthermore, we show a decision tree to identify figures and suggest a usage scenario on how the ontology can be utilized to collect and annotate data.

Digital forensics has become an important topic during this digital devices era. One major problem is extracting critical information from a digital device, and one of such data sources from a digital device is the main memory. The main memory holds many data that can be as simple as a text and a number or as complex as a data structure that holds cryptography keys. In line with the growth of the digital devices era, the need for privacy is also becoming an emerging topic. Encryption is one of the ways to achieve privacy during data transmission. As mentioned, the main memory might hold the session keys to encrypt or decrypt such secure data transmissions. This paper proposes a pure brute-force and a machine learning augmented brute force method to extract session keys from the main memory. In addition, we reduce the training data footprint using a novel entropy-based preprocessing method. We choose the most commonly used secure communication protocols: Secure Shell (SSH) and Transport Layer Security (TLS). With the help of machine learning, our method becomes efficient compared to the brute-force method. Our performance evaluation shows that our methods can extract the keys with high precision and considerably fast run-time.

Abstract The state-of-the-art feature-engineering method for fraud classification of electronic payments uses manually engineered feature aggregates, i.e., descriptive statistics of the transaction history. However, this approach has limitations, primarily that of being dependent on expensive human expert knowledge. There have been attempts to replace manual aggregation through automatic feature extraction approaches. They, however, do not consider the specific structure of the manual aggregates. In this paper, we define the novel Neural Aggregate Generator (NAG), a neural network-based feature extraction module that learns feature aggregates end-to-end on the fraud classification task. In contrast to other automatic feature extraction approaches, the network architecture of the NAG closely mimics the structure of feature aggregates. Furthermore, the NAG extends learnable aggregates over traditional ones through soft feature value matching and relative weighting of the importance of different feature constraints. We provide a proof to show the modeling capabilities of the NAG. We compare the performance of the NAG to the state-of-the-art approaches on a real-world dataset with millions of transactions. More precisely, we show that features generated with the NAG lead to improved results over manual aggregates for fraud classification, thus demonstrating its viability to replace them. Moreover, we compare the NAG to other end-to-end approaches such as the LSTM or a generic CNN. Here we also observe improved results. We perform a robust evaluation of the NAG through a parameter budget study, an analysis of the impact of different sequence lengths and also the predictions across days. Unlike the LSTM or the CNN, our approach also provides further interpretability through the inspection of its parameters.

Deep learning models have been shown to be vulnerable to adversarial attacks. This perception led to analyzing deep learning models not only from the perspective of their performance measures but also their robustness to certain types of adversarial attacks. We take another step forward in relating the architectural structure of neural networks from a graph theoretic perspective to their robustness. We aim to investigate any existing correlations between graph theoretic properties and robustness of Sparse Neural Networks. Our hypothesis is, that graph theoretic properties as a prior of neural network structures is related to their robustness. To give answer to this hypothesis, we designed an empirical study with neural network models obtained through random graphs used as sparse structural priors for the networks. We additionally investigated the evaluation of a randomly pruned fully connected network as point of reference. We found that robustness measures are independent to initialization methods but show weak correlations with graph properties: higher graph densities correlate with lower robustness, but higher average path lengths and average node eccentricities show negative correlations with robustness measures. We hope to motivate further empirical and analytical research to tightening an answer to our hypothesis.

Media has a substantial impact on public perception of events, and, accordingly, the way media presents events can potentially alter the beliefs and views of the public. One of the ways in which bias in news articles can be introduced is by altering word choice. Such a form of bias is very challenging to identify automatically due to the high context-dependence and the lack of a large-scale gold-standard data set. In this paper, we present a prototypical yet robust and diverse data set for media bias research. It consists of 1,700 statements representing various media bias instances and contains labels for media bias identification on the word and sentence level. In contrast to existing research, our data incorporate background information on the participants’ demographics, political ideology, and their opinion about media in general. Based on our data, we also present a way to detect bias-inducing words in news articles automatically. Our approach is feature-oriented, which provides a strong descriptive and explanatory power compared to deep learning techniques. We identify and engineer various linguistic, lexical, and syntactic features that can potentially be media bias indicators. Our resource collection is the most complete within the media bias research area to the best of our knowledge. We evaluate all of our features in various combinations and retrieve their possible importance both for future research and for the task in general. We also evaluate various possible Machine Learning approaches with all of our features. XGBoost, a decision tree implementation, yields the best results. Our approach achieves an F1-score of 0.43, a precision of 0.29, a recall of 0.77, and a ROC AUC of 0.79, which outperforms current media bias detection methods based on features. We propose future improvements, discuss the perspectives of the feature-based approach and a combination of neural networks and deep learning with our current system.

Network embedding is to learn low-dimensional representations of nodes which mostly preserve the network topological structure. In real-world networks, however, nodes are often associated with a rich set of attributes and labels which are potentially valuable in seeking more effective vector representations. To properly utilize this information, we propose a Joint Autoencoders framework for Multi-task network Embedding (JAME), which aims to encode a shared representation of local network structure, node attributes, and available node labels. Jointly embedding via multi-task learning is strongly dependent on the relative weighting between each task's loss function. Tuning these weights by hand is an expensive and difficult process, making multi-task learning prohibitive in practice. Therefore, we define an adaptive loss weighting layer capable of learning an optimal combination of loss weights during model training. This allows to dynamically updates task weights for controlling the influence of individual tasks based on task difficulty. Empirical evaluations on three real-world datasets demonstrate effectiveness of JAME compared to the relevant baseline methods.

In order to detect insider threats, anomaly-based intrusion detection must learn profiles of normal user behavior. However this is particularly difficult when historical audit data is scarce. Zero-shot learning can address this limitation by compensating the absence of examples with semantic knowledge, allowing to better estimate behavior of unknown users. In this paper, we address insider threat detection in two use cases where historical user data is unavailable or obsolete. We extend an existing intrusion detection system by adding information describing user positions, roles and projects assignments. These semantic descriptions are encoded via graph embeddings. Experimental results show that providing this additional context improves insider threat detection significantly. This suggests that zero-shot learning is a promising way of improving intrusion detection systems.

The state-of-the-art feature-engineering method for fraud classification of electronic payments uses manually engineered feature aggregates, i.e. descriptive statistics of the transaction history. However, this approach has limitations, primarily that of being dependent on expensive human expert knowledge. There have been attempts to replace manual aggregation through automatic feature extraction approaches. They, however, do not consider the specific structure of the manual aggregates. In this paper, we define the novel Neural Aggregate Generator (NAG), a neural network based feature extraction module that learns feature aggregates end-to-end on the fraud classification task. In contrast to other automatic feature extraction approaches, the network architecture of the NAG closely mimics the structure of feature aggregates. Furthermore, the NAG extends learnable aggregates over traditional ones through soft feature value matching, and relative weighting of the importance of different feature constraints. We compare the performance of the NAG to the state-of-the-art approaches on a real-world dataset with millions of transactions. More precisely, we show that features generated with the NAG lead to improved results over manual aggregates for fraud classification, thus demonstrating its viability to replace them. Moreover, we compare the NAG to other automatic approaches such as the LSTM or a generic CNN. Here we also observe improved results, with the NAG requiring far less parameters. Unlike the LSTM or the CNN, our approach also provides further interpretability through the inspection of its parameters.

The Common European Framework of Reference (CEFR) provides generic guidelines for the evaluation of language proficiency. Nevertheless, for automated proficiency classification systems, different approaches for different languages are proposed. Our paper evaluates and extends the results of an approach to Automatic Essay Scoring proposed as a part of the REPROLANG 2020 challenge. We provide a comparison between our results and the ones from the published paper and we include a new corpus for the English language for further experiments. Our results are lower than the expected ones when using the same approach and the system does not scale well with the added English corpus.

Abusive language detection is an unsolved and challenging problem for the NLP community. Recent literature suggests various approaches to distinguish between different language phenomena (e.g., hate speech vs. cyberbullying vs. offensive language) and factors (degree of explicitness and target) that may help to classify different abusive language phenomena. There are data sets that annotate the target of abusive messages (i.e.OLID/OffensEval (Zampieri et al., 2019a)). However, there is a lack of data sets that take into account the degree of explicitness. In this paper, we propose annotation guidelines to distinguish between explicit and implicit abuse in English and apply them to OLID/OffensEval. The outcome is a newly created resource, AbuseEval v1.0, which aims to address some of the existing issues in the annotation of offensive and abusive language (e.g., explicitness of the message, presence of a target, need of context, and interaction across different phenomena).

Sparse Neural Networks regained attention due to their potential of mathematical and computational advantages. We give motivation to study Artificial Neural Networks (ANNs) from a network science perspective, provide a technique to embed arbitrary Directed Acyclic Graphs into ANNs and report study results on predicting the performance of image classifiers based on the structural properties of the networks’ underlying graph. Results could further progress neuroevolution and add explanations for the success of distinct architectures from a structural perspective.

In the 2015 migration crisis thousands of refugees and migrants crossed the border to Hungary, Austria and Germany. The movements of these people are reflected in social media, especially on Twitter. In this paper we present a dataset of 3275 Tweets from the months September and October 2015. These Tweets are annotated regarding their relevance of containing quantitative movement information of refugees/migrants into Hungary, Austria and Germany. We present this dataset for a posterior analysis of the 2015 migration crisis or as a basis for creating an automated extraction / prediction system.

Computing shortest path distances between nodes lies at the heart of many graph algorithms and applications. Traditional exact methods such as breadth-first-search (BFS) do not scale up to contemporary, rapidly evolving today's massive networks. Therefore, it is required to find approximation methods to enable scalable graph processing with a significant speedup. In this paper, we utilize vector embeddings learnt by deep learning techniques to approximate the shortest paths distances in large graphs. We show that a feedforward neural network fed with embeddings can approximate distances with relatively low distortion error. The suggested method is evaluated on the Facebook, BlogCatalog, Youtube and Flickr social networks.

Search experts use advanced query language and search tactics to formulate their queries. However, the effectiveness of those advanced techniques depends on the search engine internals. We propose QueryCrumbs for Experts, a compact visualization, which facilitates insights to the search engine internals and therefore allows the searcher to determine effective search strategies. Treating the search engine as a black box, QueryCrumbs can be seamlessly integrated into existing search interfaces, guiding the user's exploration and assessment of results. QueryCrumbs for Experts visualize the recent search history alongside with a simple and also a qualitative comparison of the result sets, from which conclusions about the search engine internals can be drawn. The evaluation shows that, by identifying specific patterns in the visualization, expert users can gain valuable insights into search engine internals, empowering them to adapt their search accordingly.

Most unsupervised anomaly ranking approaches are compatible with numeric data only, leading to categorical features often being ignored in practice. Even though some methods address this issue, few support mixed data and the influence of excluding or including categorical attributes has not been studied well yet. In this paper, we take a first step towards considering categorical and numeric attributes jointly for unsupervised anomaly ranking by benchmarking selected methods. We introduce three new approaches: two entropy-based methods based on individual and collective entropy contribution, as well as an extension of Isolation Forest supporting mixed data, and benchmark them against SPAD, a state-of-the-art probabilistic anomaly ranker. We observe that our entropy methods detect very similar anomalies in practice, and these anomalies are mostly globally isolated observations. Both entropy methods are also closely related to SPAD. Our empirical study additionally shows that categorical features can have high impact on anomaly ranking performance and thus should not be blindly ignored.

Artificial Neural Networks have shown impressive success in very different application cases. Choosing a proper network architecture is a critical decision for a network’s success, usually done in a manual manner. As a straightforward strategy, large, mostly fully connected architectures are selected, thereby relying on a good optimization strategy to find proper weights while at the same time avoiding overfitting. However, large parts of the final network are redundant. In the best case, large parts of the network become simply irrelevant for later inferencing. In the worst case, highly parameterized architectures hinder proper optimization and allow the easy creation of adverserial examples fooling the network. A first step in removing irrelevant architectural parts lies in identifying those parts, which requires measuring the contribution of individual components such as neurons. In previous work, heuristics based on using the weight distribution of a neuron as contribution measure have shown some success, but do not provide a proper theoretical understanding. Therefore, in our work we investigate game theoretic measures, namely the Shapley value (SV), in order to separate relevant from irrelevant parts of an artificial neural network. We begin by designing a coalitional game for an artificial neural network, where neurons form coalitions and the average contributions of neurons to coalitions yield to the Shapley value. In order to measure how well the Shapley value measures the contribution of individual neurons, we remove low-contributing neurons and measure its impact on the network performance. In our experiments we show that the Shapley value outperforms other heuristics for measuring the contribution of neurons.

Models of human information seeking reveal that search, in particular ad-hoc retrieval, is non-linear and iterative. Despite these findings, todays search user interfaces do not support non-linear navigation, like for example backtracking in time. In this work, we propose QueryCrumbs, a compact and easy-to-understand visualization for navigating the search query history supporting iterative query refinement. We apply a multi-layered interface design to support novices and firsttime users as well as intermediate users. The formative evaluation with first-time and intermediate users showed that the interactions can be easily performed, and the visual encodings were well understood without instructions. Results indicate that QueryCrumbs can support users when searching for information in an iterative manner.

The inferences of a machine learning algorithm are naturally limited by the available data. In many real-world applications, the provided internal data is domain-specific and we use external background knowledge to derive or add new features. Semantic networks, like linked open data, provide a largely unused treasure trove of background knowledge. This drives a recent surge of interest in unsupervised methods to automatically extract such semantic background knowledge and inject it into machine learning algorithms. In this work, we describe the general process of extracting knowledge from semantic networks through vector space embeddings. The locations in the vector space then reflect relations in the original semantic network. We perform this extraction for geographic background knowledge and inject it into a neural network for the complicated real-world task of credit-card fraud detection. This improves the performance by 11.2%.

Hypervideos, consisting of media enriched and linked video scenes, have proven useful in many scenarios. Software solutions exist that help authors make hypervideos from media files. However, recording and editing video scenes for hypervideos is a tedious and time consuming job. Huge video databases like YouTube exist that can provide rich sources of video material. Yet it is often illegal to download and re-purpose videos from these sites, requiring a solution that links whole videos or parts of videos and plays them in an embedded player. This work presents the SIVA Web Producer, a Chrome extension for the creation of hypervideos consisting of scenes from YouTube videos. After creating a project, the SIVA Web Producer embeds YouTube videos or parts thereof as video clips. These can then be linked in a scene graph and extended with annotations. The plug-in provides a preview space for testing the hypervideo. Finalized videos can be published on the SIVA Web Portal or embedded in a Web page.

Entity disambiguation is the task of mapping ambiguous terms in natural-language text to its entities in a knowledge base. It finds its application in the extraction of structured data in RDF (Resource Description Framework) from textual documents, but equally so in facilitating artificial intelligence applications, such as Semantic Search, Reasoning and Question & Answering. We propose a new collective, graph-based disambiguation algorithm utilizing semantic entity and document embeddings for robust entity disambiguation. Robust thereby refers to the property of achieving better than state-of-the-art results over a wide range of very different data sets. Our approach is also able to abstain if no appropriate entity can be found for a specific surface form. Our evaluation shows, that our approach achieves significantly (>5%) better results than all other publicly available disambiguation algorithms on 7 of 9 datasets without data set specific tuning. Moreover, we discuss the influence of the quality of the knowledge base on the disambiguation accuracy and indicate that our algorithm achieves better results than non-publicly available state-of-the-art algorithms.

The growing number of elderly people combined with financial cuts in the health care sector lead to an increased demand for computer supported medical services. New standards like HTML5 allow the creation of hypervideo training applications that run on a variety of end user devices. In this paper, we evaluate an HTML5 player running an e-health hypervideo for the support of pelvic floor exercises. In an experimental test setting we compared the hypervideo to a primarily linear version regarding usability and utilization for self-controlled training. Our results show the hypervideo version leads to slightly more usability problems but facilitated a more active and individual training.

With the availability of free online enrichment services injection of additional, external resources in existing Web content becomes more and more widespread. For the specific area of just-in-time retrieval of digital resources based on web page content, there are no specific guidelines of how to design and integrate the additional user interface components. In this paper, we conceptualise related user interface issues, investigating the central questions: (i) how can a user be visually notified that additional results are available, and (ii) with which user interface elements should the results be presented. Concretely, we identified four different notification styles and six different result presentation styles. In a survey-based study with 75 participants we elicited the users' preferences, revealing a clear preference for the representation style (split pane) and a strong preference for three notification styles (notification bubble, icon appearance and change of icon's appearance). The latter preferences are related to the preferred browser. The results can serve as guideline for designing web-based user interfaces for just-in-time retrieval.

Despite advancing search technologies, information overload has not yet been solved. Getting an overview of information or explorative access to information becomes increasingly difficult with the exponentially increasing amount of information. Search result visualizations, especially for faceted browsing, aim at supporting users to find their way through large document collections. We propose Facets cape, a novel visualization for navigation and refinement of search results allowing users to visually construct complex boolean search queries for narrowing down the search space. This visualization combines Voronoi subdivision and a tag cloud representation of the search facets. Further it includes a preview of action (query preview) and interactions to allow users to focus on important aspects of the data for the task at hand. In a comparative user study with 15 users we compared the visualization to a standard faceted browsing interface for different types of search tasks. The study revealed that participants used the unfamiliar interface as efficiently and effectively as the familiar tree-like display. Results indicate that the Facets cape is a promising way of supporting users in exploring the faceted search space.

The objective of this special issue is to report on recent trends in digital and media technologies responding to the challenges of managing and accessing multimedia (images, audio, video, 3D/4D material, etc.).In a highly selective review procedure we accepted contributions describing recent work that aims at narrowing the large disparity between the low-level multimedia descriptors and the richness of subjectivity of semantics in user queries and human interpretations of audiovisual data.The articles in this special issue can be grouped into following categories: Multimedia Analysis (Section 1), Multimedia Ontologies and Data Integration (Section 2), and Social Media and Retrieval (Section 3).

Cluster label quality is crucial for browsing topic hierarchies obtained via document clustering. Intuitively, the hierarchical structure should influence the labeling accuracy. However, most labeling algorithms ignore such structural properties and therefore, the impact of hierarchical structures on the labeling accuracy is yet unclear. In our work we integrate hierarchical information, i.e. sibling and parent-child relations, in the cluster labeling process. We adapt standard labeling approaches, namely Maximum Term Frequency, Jensen-Shannon Divergence, Chi Square Test, and Information Gain, to take use of those relationships and evaluate their impact on 4 different datasets, namely the Open Directory Project, Wikipedia, TREC Ohsumed and the CLEF IP European Patent dataset. We show, that hierarchical relationships can be exploited to increase labeling accuracy especially on high-level nodes.

Abstract Social software has received much attention in the academia and industry due to many success stories. However, although social software is used widely for business support, its relationship with Business Process Management has not been analysed. The results of the workshop on Business Process Management and Social Software (BPMS2'08), as part of the International Conference on Business Process Management in Milano, show the manifold possibilities of combining concepts from Business Process Management and social software. Social software provides a better integration of all stakeholders into the business process life cycle and offers new possibilities for a more effective and flexible design of business processes. The modelling of business processes may profit particularly from using social software techniques by alleviating the integration of process knowledge from all stakeholders. In addition, the implementation and deployment phase of the business process life cycle may profit from social software by collecting valuable information for continuous process improvement from a larger set of sources than before. Furthermore, social software environments may be used to provide workflow support. Moreover, the use of social software also requires new considerations about the digital identity and reputation in business processes. Copyright © 2010 John Wiley & Sons, Ltd.

Knowledge discovery involves data driven processes where data is transformed and processed by various algorithms to identify new knowledge. KnowMiner is a service oriented framework providing a rich set of knowledge discovery functionalities with focus on text data sets. Complementing results of automatic machine analysis with the immense processing power of human visual apparatus has the potential of significantly improving the process of acquiring new knowledge. VisTools is a lightweight visual analytics framework based on multiple coordinated views (MCV) paradigm designed for deployment atop the KnowMinerpsilas service architecture. In this paper we briefly present both frameworks and, driven by real-world customer requirements, describe how visual techniques can be synergistically combined with machine processing for effective analysis of dynamically changing, metadata-rich text documents sets.

The task of linear text segmentation is to split a large text document into shorter fragments, usually blocks of consecutive sentences. The algorithms that demonstrated the best performance for this task come at the price of high computational complexity. In our work we present an algorithm that has a computational complexity of O(n) with n being the number of sentences in a document. The performance of our approach is evaluated against algorithms of higher complexity using standard benchmark data sets and we demonstrate that our approach provides comparable accuracy.

Tag clouds are text-based visual representations of a set of tags usually depicting tag importance by font size. Recent trends in social and collaborative software have greatly increased the popularity of this type of visualization. This paper proposes a family of novel algorithms for tag cloud layout and presents evaluation results obtained from an extensive user study and a technical evaluation. The algorithms address issues found in many common approaches, for example large whitespaces, overlapping tags and restriction to specific boundaries. The layouts computed by these algorithms are compact and clear, have small whitespaces and may feature arbitrary convex polygons as boundaries. The results of the user study and the technical evaluation enable designers to devise a combination of algorithm and parameters which produces satisfying tag cloud layouts for many application scenarios.

In many application areas which deal with heterogeneous data sets temporal developments and topical relationships both play an important role. A number of visual representations have been developed which separately address each of these two aspects of the data. However, a simultaneous analysis of both aspects is also often required. In this paper we introduce an approach which utilizes a combination of visual representations, where some are targeting the temporal behaviour while others are focusing on topical relationships. The views are fused in the sense that they are tightly coupled with respect to the visualised data set, considered time interval, and user actions. The resulting user interface enables the user to simultaneously explore and discover patterns in both temporal and topical aspects of a heterogeneous data set. As a proof of concept a demo application was developed and applied for visualising temporal and topical patterns extracted from meeting recordings.

An encyclopedia provides a written compendium of knowledge consisting of articles concisely and exhaustively covering topics of interest. Most general encyclopedias are structured in an alphabetical manner and favor keyword queries and link-based navigation as the primary form of access. On closer examination, many encyclopedia articles can directly or indirectly be associated with geospatial references. This publication explores ways of anchoring encyclopedia articles to geospatial references and presents a Web-based 3D interface which allows navigation of the German Brockhaus encyclopedia through a geospatial metaphor

This publication presents a prototype visualisation system for navigation of encyclopedia articles based on knowledge maps connecting articles to each other. A turning table is used as a metaphor to depict a result set as a whole, while three-dimensional objects of various shapes placed on top of the table represent individual encyclopaedia articles. The system employs a number of depth cues to enhance spatial perception and uses depth-dependent semi-transparency of labels to avoid display cluttering. A concrete implementation for the German language "Brockhaus Multimedial" electronic encyclopaedia is presented and discussed.