Cyber threats are evolving rapidly, making anomaly detection (AD) in system log data increasingly important for detection of known and unknown attacks. The configuration of AD algorithms heavily depends on the data at hand. It often involves a complex feature selection process and the determination of parameters such as thresholds or window sizes. In many cases, configuration requires manual intervention by domain experts, which limits accessibility and effectiveness of AD algorithms. This work introduces a Configuration-Engine (CE), which employs a semi-supervised approach to automate the configuration process or optimize existing configurations. The CE utilizes statistical methods to identify log line properties to recognize meaningful tokens for AD methods to monitor. It categorizes variables by their characteristics and behavior over time, then specifies which log parts a detector should observe, and sets appropriate configuration parameters.The CE was evaluated using four different detectors. Evaluations on different Apache Access and audit datasets containing attack traces showed that the CE achieved an average precision of over 0.94 for Apache and over 0.79 for audit datasets, while maintaining high recall, competing with the performance of expert-crafted configurations. The optimization approach was able to strongly improve the precision of both the CE’s and the experts’ configurations for Apache data in 7 out of 16 cases. Furthermore, the CE’s configurations were significantly dissimilar to each other when generated on audit data, highlighting the importance of automated configuration.

The rapid advancement of Large Language Models (LLMs) has transformed natural language processing, yet comprehensive evaluation methods are necessary to ensure their reliability, particularly in Retrieval-Augmented Generation (RAG) tasks. This study aims to evaluate and compare the performance of open-source LLMs by introducing a rigorous evaluation framework. We benchmark 20 LLMs using a combination of established metrics such as BLEU, ROUGE, BERTScore, along with and a novel metric, RAGAS. The models were tested across two distinct datasets to assess their text generation quality. Our findings reveal that models like nous-hermes-2-solar-10.7b and mistral-7b-instruct-v0.1 consistently excel in tasks requiring strict instruction adherence and effective use of large contexts, while other models show areas for improvement. This research contributes to the field by offering a comprehensive evaluation framework that aids in selecting the most suitable LLMs for complex RAG applications, with implications for future developments in natural language processing and big data analysis.

In the era of big data, research has become increasingly data-driven, with vast amounts of information being generated and analyzed to produce new insights and discoveries. This data deluge requires a combination of methods and technologies to store, process, share and preserve research data. With many of the world’s most valuable data being stored in relational databases where it evolves over time as new knowledge is gained and old knowledge invalidated, current repository systems fail to provide researchers with interfaces to conveniently work with this kind of data within their research environments. For this reason, we have developed DBRepo, an institutional data repository for research data in databases (DBRepo) supporting guidelines of the Working Group on Data Citation of the Research Data Alliance. The system has been in use at TU Wien for almost three years now and provides a variety of data science-related interfaces and can be integrated into many workflows and tools. Further, it assists researchers in depositing their datasets by suggesting the table schema (column names, data types, primary key constraints) and it addresses data interoperability issues by suggesting semantic concepts for dataset columns and units of measurements, where applicable. DBRepo is currently in use by six universities globally who use it as data store for hot and cold research data sets. In the paper, we describe their use-cases and provide lessons learned from the various deployments and workflows. Finally, we show how depositing research data into DBRepo increases the data’s visibility.

Fingerprinting is a method of embedding a traceable mark into digital data to (i) verify the owner and (ii) identify the recipient of a released copy of a data set. This is crucial when releasing data to third parties, especially if it involves a fee, or if the data is of sensitive nature and further sharing and leaks should be discouraged and deterred from. A fingerprint is required to (i) be robust against modifications t o t he d ata to achieve successful ownership protection, while (ii) affecting the quality and utility of the data as little as possible.So far, literature mostly assumes attackers with rather limited capabilities who perform random modification t o t he dataset. With a certain task in mind to perform on the data, the attacker can however perform an adaptive and targeted attack that maximises its chances of removing or invalidating the fingerprint, while reducing the data utility the least. In the same line, the data owner can optimise the robustness of the scheme by anticipating a specific f ocus o f t he a ttacker a nd f ocusing t he fingerprint embedding on the most valuable parts of the data. In this paper, we, therefore, provide an in-depth discussion on threat models, targeted attacks and adaptive defences. We further demonstrate the impact of targeted attacks on classical and, in comparison, adaptive fingerprinting i n a n e mpirical manner.

Argumentative Zoning (AZ) is a tool to obtain informative summaries of scientific articles. Using AZ assumes the definition of the main rhetorical structure in scientific articles, which are, then, used for the summary creation. The unavailability of large AZ annotated benchmark datasets is a bottleneck to training AZ-based summarization algorithms. In this work, we present an annotation platform for an AZ that defines four categories (zones), Claim, Method, Result and Conclusion, that are used to label sentences selected from scientific articles. The proposed tool can be used both for collecting benchmark datasets, and to help the researchers to create their own sub-corpora.

Intrusion Detection Systems (IDS) rely on the availability and correctness of Indicators of Compromise (IoC), i.e., artifacts such as IP addresses that are known to correspond to malicious system activities. However, the simple nature and limited validity of these indicators impairs protection against cyber threats. Tactics, Techniques and Procedures (TTP) provide abstract information on attacker behavior, but are only available in human-readable format that prevents automatic detection using IDSs. In this paper we therefore propose an approach that extracts cyber threat intelligence from raw log data and combines the advantages of IoCs and TTPs by producing detectable patterns of complex system behavior. Other than existing approaches, our approach employs log data anomaly detection to disclose suspicious log events, which are used for iterative clustering, pattern recognition, and refinement. Our evaluations show that automatically extracted threat intelligence corresponding to a multi-step attack is suitable for detection of the same attack on another system.

The growth of Linked Data has created a promising environment for data exploration and a growing number of tools allow users to interactively integrate data from various sources. Eliciting the reliability of the results of such ad-hoc integration processes, consistently recreating those results, and identifying changes upon re-execution, however, can be difficult. Automated process provenance trail creation can provide major benefits in this context, because (i) it enables users to trace the contribution of individual sources and processing steps to the final outcome and judge whether the result can be trusted; (ii) it ensures repeatability and raises the trustworthiness of results; (iii) it ideally enables reconstruction of Linked Data integration processes from the provenance information embedded in the final result. In this paper, we present a provenance model that facilitates automatic generation of semantic provenance information for generic Linked Data integration processes. We implement the generic model in a collaborative mashup environment and evaluate it by means of an example application. We find that the model provides a solid foundation for verifiability and contributes towards making Linked Data integration processes more open, transparent, and reusable, which is crucial in domains where the origin of data is essential, such as, for instance, statistical analyses, scientific research, and data journalism.

Over decades, music labels have shaped easily identifiable genres to improve recognition value and subsequently market sales of new music acts. Referring to print magazines and later to music television as important distribution channels, the visual representation thus played and still plays a significant role in music marketing. Visual stereotypes developed over decades that enable us to quickly identify referenced music only by sight without listening. Despite the richness of music-related visual information provided by music videos and album covers as well as T-shirts, advertisements, and magazines, research towards harnessing this information to advance existing or approach new problems of music retrieval or recommendation is scarce or missing. In this article, we present our research on visual music computing that aims to extract stereotypical music-related visual information from music videos. To provide comprehensive and reproducible results, we present the Music Video Dataset, a thoroughly assembled suite of datasets with dedicated evaluation tasks that are aligned to current Music Information Retrieval tasks. Based on this dataset, we provide evaluations of conventional low-level image processing and affect-related features to provide an overview of the expressiveness of fundamental visual properties such as color, illumination, and contrasts. Further, we introduce a high-level approach based on visual concept detection to facilitate visual stereotypes. This approach decomposes the semantic content of music video frames into concrete concepts such as vehicles, tools, and so on, defined in a wide visual vocabulary. Concepts are detected using convolutional neural networks and their frequency distributions as semantic descriptions for a music video. Evaluations showed that these descriptions show good performance in predicting the music genre of a video and even outperform audio-content descriptors on cross-genre thematic tags. Further, highly significant performance improvements were observed by augmenting audio-based approaches through the introduced visual approach.

Patent text is a mixture of legal terms and domain specific terms. In technical English text, a multi-word unit method is often deployed as a word formation strategy in order to expand the working vocabulary, i.e. introducing a new concept without the invention of an entirely new word. In this paper we explore query generation using natural language processing technologies in order to capture domain specific concepts represented as multi-word units. In this paper we examine a range of query generation methods using both linguistic and statistical information. We also propose a new method to identify domain specific terms from other more general phrases. We apply a machine learning approach using domain knowledge and corpus linguistic information in order to learn domain specific terms in relation to phrases' Termhood values. The experiments are conducted on the English part of the CLEF-IP 2013 test collection. The outcome of the experiments shows that the favoured method in terms of PRES and recall is when a language model is used and search terms are extracted with a part-of-speech tagger and a noun phrase chunker. With our proposed methods we improve each evaluation metric significantly compared to the existing state-of-the-art for the CLEP-IP 2013 test collection: for [email protected] by 26% (0.544 from 0.433), for [email protected] by 17% (0.631 from 0.540) and on document MAP by 57% (0.300 from 0.191).

Sharing research data is becoming increasingly important as it enables peers to validate and reproduce data driven experiments. Without original raw data at hand, serious peer review is impossible. Also exchanging data allows scientists to reuse data in different contexts and gather new knowledge from available sources. But with increasing volume and iteratively enhanced data sets, researchers need to reference exact versions of data sets. Until now access to research data often based on single archives of data files where versioning and subsetting support is limited. In this paper we introduce a mechanism that allows researchers to create versioned subsets of research data which can be cited and shared in a lightweight and secure manner. We demonstrate a prototype that supports researchers in creating subsets based on filtering and sorting source data. These subsets can be cited for later reference and reuse. The system produces evidence that allows users to verify the correctness and completeness of a subset based on cryptographic hashing. We describe a replication scenario for enabling scalable data citation in dynamic contexts.

Many business and scientific processes make extensive use of service-oriented architectures, using distributed services. These are often provided by third parties and are thus not under direct control of process owners. In this paper we discuss the issues of ensuring continuous and faithful execution of processes in distributed environments, focusing specifically on Web Services. Recently, we introduced a specification of Resilient Web Services, that makes current Web Services more robust, and a framework for the monitoring of Web Services, that allows detecting anomalies. In this paper, we describe alternative implementations of the framework for monitoring of Web Services. We also present possible approaches easing the deployment of Resilient Web Services: a framework consisting of tools deployable at the Web Service operator site enabling easy transformation of a regular Web Service into a Resilient Web Service, and a registry with notifications that decorates existing Web Services with resilient methods.

We present a generic model for multimodal information retrieval, leveraging different information sources to improve the effectiveness of a retrieval system. The proposed method is able to take into account both explicit and latent semantics present in the data and can be used to answer complex queries, not currently answerable neither by document retrieval systems, nor by semantic web systems. By providing a hybrid approach combining IR and structured search techniques, we prepare a framework applicable to multimodal data collections. To test its effectiveness, we instantiate the model for an image retrieval task.

Uniquely and precisely identifying and citing arbitrary subsets of data is essential in many settings, e.g. to facilitate experiment validation and data re-use in meta-studies. Current approaches relying on pointers to entire data collections or on explicit copies of data do not scale. We propose a novel approach relying on persistent, timestamped, adapted queries to versioned and timestamped data sources. Result set hashes are used for validation correctness on later re-execution. The proposed method works both for static as well as dynamically growing or changing data. Alternative implementation styles for relational databases are presented and evaluated with regard to performance issues and impact on existing applications while aiming at minimal to no additional effort requirements for data users. The approach is validated in an infrastructure monitoring domain relying on sensor data networks.

Accessible emulation is often the method of choice for maintaining digital objects, specifically complex ones such as applications, business processes, or electronic art. However, validating the emulator’s ability to faithfully reproduce the original behavior of digital objects is complicated. This article presents an evaluation framework and a set of tests that allow assessment of the degree to which system emulation preserves original characteristics and thus significant properties of digital artifacts. The original system, hardware, and software properties are described. Identical environment is then recreated via emulation. Automated user input is used to eliminate potential confounders. The properties of a rendered form of the object are then extracted automatically or manually either in a target state, a series of states, or as a continuous stream. The concepts described in this article enable preservation planners to evaluate how emulation affects the behavior of digital objects compared to their behavior in the original environment. We also review how these principles can and should be applied to the evaluation of migration and other preservation strategies as a general principle of evaluating the invocation and faithful rendering of digital objects and systems. The article concludes with design requirements for emulators developed for digital preservation tasks.

The 4th International Workshop on Patent Information Retrieval builds on the experiences of the first three workshops, to provide its participants an exciting, scientifically challenging and interactive event, where specific issues of patent retrieval may be put into the general context of Information Retrieval and Knowledge Management, in order to explore innovative solutions to new and old problems, but also to evaluate and adapt traditional or classic approaches to new problems. This year, we observe an increase in the use of standardized test collections in the contributions received, and, at the same time, new discussion points on how to make such standardized evaluation exercises more accessible to the larger IP community.

Abstract Bias quantification of retrieval functions with the help of document retrievability scores has recently evolved as an important evaluation measure for recall‐oriented retrieval applications. While numerous studies have evaluated retrieval bias of retrieval functions, solid validation of its impact on realistic types of queries is still limited. This is due to the lack of well‐accepted criteria for query generation for estimating retrievability. Commonly, random queries are used for approximating documents retrievability due to the prohibitively large query space and time involved in processing all queries. Additionally, a cumulative retrievability score of documents over all queries is used for analyzing retrieval functions (retrieval) bias. However, this approach does not consider the difference between different query characteristics (QCs) and their influence on retrieval functions' bias quantification. This article provides an in‐depth study of retrievability over different QCs. It analyzes the correlation of lower/higher retrieval bias with different query characteristics. The presence of strong correlation between retrieval bias and query characteristics in experiments indicates the possibility of determining retrieval bias of retrieval functions without processing an exhaustive query set. Experiments are validated on TREC Chemical Retrieval Track consisting of 1.2 million patent documents.

The enormous amount of valuable information that is produced today and needs to be made available over the long-term has led to increased efforts in scalable, automated solutions for long-term digital preservation. The mission of preservation planning is to define the optimal actions to ensure future access to digital content and react to changes that require adjustments in repository operations. Considerable effort has been spent in the past on defining, implementing, and validating a framework and system for preservation planning. This article sheds light on the actual decision criteria and influence factors to be considered when choosing digital preservation actions. It is based on an extensive evaluation of case studies on preservation planning for a range of different types of objects with partners from different institutional backgrounds. We categorize decision criteria from a number of real-world decision-making instances in a taxonomy. We show that a majority of the criteria can be evaluated by applying automated measurements under realistic conditions, and demonstrate that controlled experimentation and automated measurements can be used to substantially improve repeatability of decisions and reduce the effort needed to evaluate preservation components. The presented measurement framework enables scalable preservation and monitoring and supports trust in preservation decisions because extensive evidence is produced in a reproducible, automated way and documented as the basis of decision making in a standardized form.

High findability of documents within a certain cut-off rank is considered an important factor in recall-oriented application domains such as patent or legal document retrieval. Findability is hindered by two aspects, namely the inherent bias favoring some types of documents over others introduced by the retrieval model, and the failure to correctly capture and interpret the context of conventionally rather short queries. In this paper, we analyze the bias impact of different retrieval models and query expansion strategies. We furthermore propose a novel query expansion strategy based on document clustering to identify dominant relevant documents. This helps to overcome limitations of conventional query expansion strategies that suffer strongly from the noise introduced by imperfect initial query results for pseudo-relevance feedback documents selection. Experiments with different collections of patent documents suggest that clustering based document selection for pseudo-relevance feedback is an effective approach for increasing the findability of individual documents and decreasing the bias of a retrieval system.

While electronic music archives are gaining popularity, access to and navigation within these archives is usually limited to text-based queries or manually predefined genre category browsing. We present a system that automatically organizes a music collection according to the perceived sound similarity resembling genres or styles of music. Audio signals are processed according to psychoacoustic models to obtain a time-invariant representation of its characteristics. Subsequent clustering provides an intuitive interface where similar pieces of music are grouped together on a map display.

Document archives may be regarded as a perfect application arena for unsupervised neural networks because many of the operations computers have to perform on text documents are classification tasks based on noisy patterns. The "noise" originates from the known inaccuracy of mapping free-form natural language to an indexing vocabulary representing the contents of the documents. We describe an approach towards management of distributed document archives based on a system of independent self-organizing maps, each of which represents just a portion of the complete document archive. These individual maps may be integrated in a hierarchical fashion. This leads to enormous flexibility for the user who may define her own personal library that reflects her particular interests.