Abstract Context Computational materials science (CMS) focuses on in silico experiments to compute the properties of known and novel materials, where many software packages are used in the community. The NOMAD Laboratory (Draxl C, Scheffler) offers to store the input and output files in its FAIR data repository. Since the file formats of these software packages are non‐standardized, parsers are used to provide the results in a normalized format. Objective The main goal of this article is to report experience and findings of using grammar‐based fuzzing on these parsers. Method We have constructed an input grammar for four common software packages in the CMS domain and performed an experimental evaluation on the capabilities of grammar‐based fuzzing to detect failures in the Novel Materials Discovery (NOMAD) parsers. Results With our approach, we were able to identify three unique critical bugs concerning service availability, as well as several additional syntactic, semantic, logical, and downstream bugs in the investigated NOMAD parsers. We reported all issues to the developer team prior to publication. Conclusion Based on the experience gained, we can recommend grammar‐based fuzzing also for other research software packages to improve the trust level in the correctness of the produced results.

Metamorphic testing involves reasoning on necessary properties that a program under test should exhibit regarding multiple input and output variables. A general approach consists of extracting metamorphic relations from auxiliary artifacts such as user manuals or documentation, a strategy particularly fitting to testing scientific software. However, such software typically has large input-output spaces, and the fundamental prerequisite – extracting variables of interest – is an arduous and non-scalable process when performed manually. To this end, we devise a workflow around an autoregressive transformer-based Large Language Model (LLM) towards the extraction of variables from user manuals of scientific software. Our end-to-end approach, besides a prompt specification consisting of few-shot examples by a human user, is fully automated, in contrast to current practice requiring human intervention. We showcase our LLM workflow over a real case, and compare variables extracted to ground truth manually labelled by experts. Our preliminary results show that our LLM-based workflow achieves an accuracy of 0.87, while successfully deriving 61.8% of variables as partial matches and 34.7% as exact matches.

Software testing is an important part of the software life-cycle. Unfortunately, some software systems have the inherent problem, that it is not clear a priori what the expected outcome is. This is known in the literature as the Oracle Problem. Scientific software suffers from this problem in particular. Metamorphic Testing is a testing approach that mitigates the Oracle Problem, as it is based on identified relations between a program's in- and output pairs. In this study, we investigate the feasibility of automatically finding such metamorphic relations on a software package known as the exciting-NOMAD parser which is widely used in computational material science. We are able to show that it is indeed possible to automatically find metamorphic relations within the NOMAD parser for the density functional theory code exciting. We analyse the metamorphic relations found through our tool in terms of both quantity and relation quality. Furthermore, we also publish our developed tool, as well as used data alongside this study through our replication package.