Assessing the quality of LLM-generated text remains a fundamental challenge in natural language processing. Current evaluation approaches often rely on isolated metrics or simplistic aggregations that fail to capture the nuanced trade-offs between coherence, diversity, fluency, and other relevant indicators of text quality. In this work, we adapt a recently proposed framework for statistical inference based on Generalized Stochastic Dominance (GSD) that addresses three critical limitations in existing benchmarking methodologies: the inadequacy of single-metric evaluation, the incompatibility between cardinal automatic metrics and ordinal human judgments, and the lack of inferential statistical guarantees. The GSD-front approach enables simultaneous evaluation across multiple quality dimensions while respecting their different measurement scales, building upon partial orders of decoding strategies, thus avoiding arbitrary weighting of the involved metrics. By applying this framework to evaluate common decoding strategies against human-generated text, we demonstrate its ability to identify statistically significant performance differences while accounting for potential deviations from the i.i.d. assumption of the sampling design.

Given the vast number of classifiers that have been (and continue to be) proposed, reliable methods for comparing them are becoming increasingly important. The desire for reliability is broken down into three main aspects: (1) Comparisons should allow for different quality metrics simultaneously. (2) Comparisons should take into account the statistical uncertainty induced by the choice of benchmark suite. (3) The robustness of the comparisons under small deviations in the underlying assumptions should be verifiable. To address (1), we propose to compare classifiers using a generalized stochastic dominance ordering (GSD) and present the GSD-front as an information-efficient alternative to the classical Pareto-front. For (2), we propose a consistent statistical estimator for the GSD-front and construct a statistical test for whether a (potentially new) classifier lies in the GSD-front of a set of state-of-the-art classifiers. For (3), we relax our proposed test using techniques from robust statistics and imprecise probabilities. We illustrate our concepts on the benchmark suite PMLB and on the platform OpenML.

We demonstrate that numerous machine learning algorithms are specific instances of one single paradigm: reciprocal learning. These instances range from active learning over multi-armed bandits to self-training. We show that all these algorithms not only learn parameters from data but also vice versa: They iteratively alter training data in a way that depends on the current model fit. We introduce reciprocal learning as a generalization of these algorithms using the language of decision theory. This allows us to study under what conditions they converge. The key is to guarantee that reciprocal learning contracts such that the Banach fixed-point theorem applies. In this way, we find that reciprocal learning converges at linear rates to an approximately optimal model under some assumptions on the loss function, if their predictions are probabilistic and the sample adaption is both non-greedy and either randomized or regularized. We interpret these findings and provide corollaries that relate them to active learning, self-training, and bandits.

We propose a framework for descriptively analyzing sets of partial orders based on the concept of depth functions. Despite intensive studies in linear and metric spaces, there is very little discussion on depth functions for non-standard data types such as partial orders. We introduce an adaptation of the well-known simplicial depth to the set of all partial orders, the union-free generic (ufg) depth. Moreover, we utilize our ufg depth for a comparison of machine learning algorithms based on multidimensional performance measures. Concretely, we provide two examples of classifier comparisons on samples of standard benchmark data sets. Our results demonstrate promisingly the wide variety of different analysis approaches based on ufg methods. Furthermore, the examples outline that our approach differs substantially from existing benchmarking approaches, and thus adds a new perspective to the vivid debate on classifier comparison.1

Spaces with locally varying scale of measurement, like multidimensional structures with differently scaled dimensions, are pretty common in statistics and machine learning. Nevertheless, it is still understood as an open question how to exploit the entire information encoded in them properly. We address this problem by considering an order based on (sets of) expectations of random variables mapping into such non-standard spaces. This order contains stochastic dominance and expectation order as extreme cases when no, or respectively perfect, cardinal structure is given. We derive a (regularized) statistical test for our proposed generalized stochastic dominance (GSD) order, operationalize it by linear optimization, and robustify it by imprecise probability models. Our findings are illustrated with data from multidimensional poverty measurement, finance, and medicine.

209

Although being a crucial question for the development of machine learning algorithms, there is still no consensus on how to compare classifiers over multiple data sets with respect to several criteria. Every comparison framework is confronted with (at least) three fundamental challenges: the multiplicity of quality criteria, the multiplicity of data sets and the randomness of the selection of data sets. In this paper, we add a fresh view to the vivid debate by adopting recent developments in decision theory. Based on so-called preference systems, our framework ranks classifiers by a generalized concept of stochastic dominance, which powerfully circumvents the cumbersome, and often even self-contradictory, reliance on aggregates. Moreover, we show that generalized stochastic dominance can be operationalized by solving easy-to-handle linear programs and moreover statistically tested employing an adapted two-sample observation-randomization test. This yields indeed a powerful framework for the statistical comparison of classifiers over multiple data sets with respect to multiple quality criteria simultaneously. We illustrate and investigate our framework in a simulation study and with a set of standard benchmark data sets.

Automated image analysis and artificial intelligence (AI) are a growing market in digital pathology. While various proprietary pathology systems exist, there are no fully vendor-agnostic integration approaches for AI apps. This makes it difficult for vendors of AI solutions to integrate their products into the multitude of non-standard software systems in pathology. The EMPAIA Consortium (EcosysteM for Pathology Diagnostics with AI Assistance) develops an open and decentralized platform allowing AI-based apps of different vendors to be integrated with existing lab IT infrastructures. This is intended to lower the barriers to entry for AI vendors and provide pathologists with access to advanced AI tools. The EMPAIA platform is based on web technologies that can be deployed both on-premises and in the cloud. There are open-source reference implementations for core platform services that can be integrated with or replaced by proprietary alternatives as long as they conform to open API specifications. Apps can be obtained through a central marketplace so pathologists can use them in their daily workflow. In this paper, we provide an overview of the EMPAIA platform architecture. We identify critical use cases and requirements for AI-based software platforms in pathology and explain how these are fulfilled by the EMPAIA platform. Finally, we evaluate the efficiency of routing image data through the platform.

Convolutional Neural Networks (CNN) are used for automatic cancer detection in pathological images. These data-driven experiments are difficult to reproduce, because the CNNs may require CUDA-enabled Nvidia GPUs for acceleration and training is often performed on a large dataset stored on a researcher's computer, inaccessible to others. We introduce the RED file format for reproducible experiment description, where executable programs are packaged and referenced as Docker container images. Data inputs and outputs are described as network resources using standard transmission and authentication protocols instead of local file paths. Following the FAIR guiding principles, the RED format is based on and compatible with the established Common Workflow Language specification. RED files are interpreted by the accompanying Curious Containers (CC) software. Arbitrarily large datasets are mounted inside containers via FUSE network filesystems like SSHFS. SSHFS is compared to NFS and a local SSD in artificial benchmarks and in the context of a CNN training scenario, where SSHFS introduces a performance decrease by a factor of 1.8. We are convinced that RED can greatly improve the reproducibility of deep learning workloads and data-driven experiments. This is in particular important in clinical scenarios where the result of an analysis may contribute to a patient's treatment.

Summary Applications for biomedical data processing often integrate external libraries and frameworks for common algorithmic tasks. It typically reduces development time and increases overall code quality. With the introduction of lightweight container‐based virtualization, the bundling of applications and their required dependencies has become feasible, and containers can be transferred and executed in distributed environments. However, the incorporation of unreviewed code poses a security threat as it might contain malicious components. In this paper, measures to minimize risks of untrusted application execution are presented. Based on the system calls issued during sample execution of the application, both the container itself and the container runtime configuration are restricted to the set of actions the application requires. It is shown that the employed security measures are suited to counteract different attacks while application runtime is not affected.

Applications for data analysis of biomedical data are complex programs and often consist of multiple components. Re-usage of existing solutions from external code repositories or program libraries is common in algorithm development. To ease reproducibility as well as transfer of algorithms and required components into distributed infrastructures Linux containers are increasingly used in those environments, that are at least partly connected to the internet. However concerns about the untrusted application remain and are of high interest when medical data is processed. Additionally, the portability of the containers needs to be ensured by using only security technologies, that do not require additional kernel modules. In this paper we describe measures and a solution to secure the execution of an example biomedical application for normalization of multidimensional biosignal recordings. This application, the required runtime environment and the security mechanisms are installed in a Docker-based container. A fine-grained restricted environment (sandbox) for the execution of the application and the prevention of unwanted behaviour is created inside the container. The sandbox is based on the filtering of system calls, as they are required to interact with the operating system to access potentially restricted resources e.g. the filesystem or network. Due to the low-level character of system calls, the creation of an adequate rule set for the sandbox is challenging. Therefore the presented solution includes a monitoring component to collect required data for defining the rules for the application sandbox. Performance evaluation of the application execution shows no significant impact of the resulting sandbox, while detailed monitoring may increase runtime up to over 420%.

Clinical Sleep Research is an inherent multidisciplinary field, as many health issues may affect a person's sleep conditions and sleep disorders may cause several health problems. Many patients with chronic sleep disorders suffer from different further medical conditions - called multimorbidity. Due to the high variety of the reasons and the courses of sleep disorders, individual cases are difficult to compare. Therefore there is a high demand for sleep researchers to collaborate with each other to reach necessary participant numbers and multidisciplinary expertise. To date, inter-institutional sleep research is poorly supported by IT systems. In particular the heterogeneity and the quality variations within the acquired bio signal data - caused by different bio signal recorders or different measurement procedures - are impeding common bio signal data processing. In this manuscript we introduce a virtual research platform supporting inter-institutional data sharing and processing. The infrastructure is based on XNAT - a free and open-source neuroimaging research platform - a loosely coupled service oriented architecture and scalable virtualization in the backend. The system is capable of local pseudonymization of bio signal data, mapping to a standardized set of parameters and automatic quality assessment. Terms and quality measures are derived from the "Manual for the Scoring of Sleep and Associated Events" of the American Academy of Sleep Medicine, the de-facto standard for diagnostic bio signal analysis in sleep medicine.

In this paper, we present a novel approach to reconstruct missing areas in facial images by using a series of Restricted Boltzman Machines (RBMs). RBMs created with a low number of hidden neurons generalize well and are able to reconstruct basic structures in the missing areas. On the other hand networks with many hidden neurons tend to emphasize details, when using the reconstruction of the previous, more generalized RBMs, as their input. Since trained RBMs are fast in encoding and decoding data by design, our method is also suitable for processing video streams.

Neurosciencific research is increasingly based on image analysis methods. Large sets of imaging data are processed using complex image analysis tools. While today magnetic resonance imaging (MRI) is widely used for both functional and anatomical analysis of the human brain, new imaging modalities are beginning to prove their capabilities for neurological research. Among them, optical coherence tomography (OCT) allows for noninvasive visualization of anatomical structures on a micrometer scale. Becoming a standard diagnostic tool in ophthalmology, it is of rising interest for neurological research. Crucial to all data analysis methods is the quality of the input data. The platform presented in this paper is designed for automatic quality assessment of retinal OCTs. It extends the image management platform XNAT by services to calculate and store quality measures. It is also extensible regarding new quality measure algorithms, allowing the developer to upload Matlab code, compile it for the infrastructure's hardware architecture and test it in the system. The image processing tools to calculate the quality measures are provided as a cloud-based service employing Open Stack as underlying IT infrastructure. The prototype implementation encompassing security and performance aspects are presented.