Large Language Models (LLMs) are being explored for applications in scientific research, including their capabilities to synthesize literature, answer research questions, generate research ideas, and even conduct computational experiments.Ultimately, our goal is for these to help scientists derive novel scientific insights. In many areas of science, such insights often arise from processing and visualizing data to understand its patterns. However, evaluating whether an LLM-mediated scientific workflow produces outputs conveying the correct scientific insights is challenging to evaluate and has not been addressed in past work.We introduce AstroVisBench, the first benchmark for both scientific computing and visualization in the astronomy domain.AstroVisBench judges a language model’s ability to both (1) create astronomy-specific workflows to process and analyze data and (2) visualize the results of these workflows through complex plots.Our evaluation of visualizations uses a novel LLM-as-a-judge workflow, which is validated against annotation by five professional astronomers.Using AstroVisBench we present an evaluation of state-of-the-art language models, showing a significant gap in their ability to engage in astronomy research as useful assistants.This evaluation provides a strong end-to-end evaluation for AI scientists that offers a path forward for the development of visualization-based workflows, which are central to a broad range of domains from physics to biology.

Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, Junyi Jessy Li. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Jan Trienes, Sebastian Joseph, Jörg Schlötterer, Christin Seifert, Kyle Lo, Wei Xu, Byron Wallace, Junyi Jessy Li. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Automated text simplification aims to produce simple versions of complex texts.This task is especially useful in the medical domain, where the latest medical findings are typically communicated via complex, technical articles.This creates barriers for laypeople seeking access to up-to-date medical findings, consequently impeding progress on health literacy.Most existing work on medical text simplification has focused on monolingual settings, with the result that such evidence would be available only in just one language (most often, English).This work addresses this limitation via multilingual simplification, i.e., directly simplifying complex texts into simplified texts in multiple languages.We introduce MULTICOCHRANE, the first sentence-aligned multilingual text simplification dataset for the medical domain in four languages: English, Spanish, French, and Farsi.We evaluate fine-tuned and zero-shot models across these languages with extensive human assessments and analyses.Although models can generate viable simplified texts, we identify several outstanding challenges that this dataset might be used to address. MultiCochrane Complex SimplePreclinical studies have suggested that RIC may have beneficial effects in ischaemic stroke patients and those at risk of ischaemic stroke.English: Studies have suggested that RIC may have beneficial effects for preventing and treating ischaemic stroke.Spanish: Los estudios han indicado que el CIR puede tener efectos beneficiosos en la prevención y el tratamiento del accidente cerebrovascular isquémico.French: Des études ont suggéré que le CID pourrait avoir des effets bénéfiques sur la prévention et le traitement de l'AVC ischémique.Farsi: ‫ﮐﮫ‬ ‫اﻧد‬ ‫ﮐرده‬ ‫ﭘﯾﺷﻧﮭﺎد‬ ‫طﺎﻟﻌﺎت‬ RIC ‫درﻣﺎن‬ ‫و‬ ‫ﭘﯾﺷﮕﯾری‬ ‫ﺑرای‬ ‫ﻣﻔﯾدی‬ ‫اﺛرات‬ ‫اﺳت‬ ‫ﻣﻣﮑن‬ ‫ﺑﺎﺷد‬ ‫داﺷﺗﮫ‬ ‫اﯾﺳﮑﻣﯾﮏ‬ ‫ﻣﻐزی‬ ‫.ﺳﮑﺗﮫ‬ Human evaluation of system outputsEnglish: Interventions have suggested that RAP may have beneficial effects in ischaemic stroke patients, those at risk of stroke.Spanish: Las intervenciones pueden ser efectivas para los pacientes que se accidente cerebrovascular isquémico y los que se encuentran en riesgo del accidente cerebrovascular isquémico.Gloss: The interventions can be effective for the patients that accident themselves ischemic stroke and those that find themselves at risk of the ischemic stroke.