Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven Corman, Chitta Baral. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

OBJECTIVE: Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. MATERIALS AND METHODS: A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance. RESULTS: In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. DISCUSSION: Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. CONCLUSION: Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly "living" systematic reviews.

Contribution: This full innovative practice paper presents a series of pedagogical interventions implemented to enhance student engagement, preparedness, and writing skills in a laboratory setting. It integrates Perusall assignments, peer reviews, peer mentoring, and a specification grading system into the curriculum, offering a structured but still flexible approach to laboratory education. Background: Many students struggle with scientific writing and find the lab work process challenging and disengaging. Previous works have utilized peer-assisted learning and digital tools, but there remains a need for the use of new strategies to address these challenges in a laboratory environment. Intended Outcomes: The primary goal is to improve students' readiness for lab sessions, increase engagement through collaborative learning, and enhance their scientific writing skills. The interventions are designed to create a supportive learning environment that encourages active participation. Application Design: The interventions were implemented over three academic quarters. Students were required to prepare for lab sessions by annotating lab manuals via the Perusall platform, which grades engagement automatically. Peer reviews were conducted with structured rubrics. Peer mentoring was incorporated into lab sessions, with initial guidance from the instructor followed by student-led training. To provide opportunities for report revision based on feedback, the specification grading system was used by categorizing lab reports as complete or incomplete. Findings: Student performance and a Likert scale survey were used to measure the outcome of the interventions. Over 80% of students reported that Perusall assignments enhanced their understanding of lab concepts, and more than 70% felt they learned effectively from their peers. Peer review was not received favorably with only 25% finding it helpful. The majority felt that peer mentoring significantly increased lab engagement. The specification grading approach resulted in a higher rate of reports meeting initial submission standards, particularly in later sessions, indicating improved writing quality. Overall, these methods demonstrated a positive impact on student learning and engagement in laboratory courses.

This Research-to-Practice Full Paper describes how Active Learning (AL) was implemented in two discussion sections of an “Introduction to Materials Science” course in a large classroom with 172 students. We studied the effect of group work on students' learning, and whether 50 minutes of AL is sufficient to increase students' performance in this course. To do so, worksheets for the discussion sections were designed. There were four discussion sections, two of which were chosen as the treatment group. In the treatment group the students worked on a worksheet of problems in groups of 3 while a teaching assistant was present to answer their questions. In other sections, the control group, the same problems were presented using PowerPoint slides and solved by the same teaching assistant. To assess students' learning, their performance in two midterm exams and the final exam are considered. Moreover, to evaluate students' interest in engineering, the Jones et al. (2010) survey was used to study students' motivation and interest in engineering. We found implementing AL for a short amount of time in each week did not change students' performance. We hope this paper will encourage educators to spend more time on AL, in their classrooms.

Software modernization has been extensively researched, primarily focusing on observing the associated phenomena, and providing technical solutions to facilitate the modernization process. Software modernization is claimed to be successful when the modernization is completed using those technical solutions. Very limited research, if any, is reported with an aim at documenting the post-modernization impacts, i.e., whether any of the pre-modernization business goals are in fact achieved after modernization. In this research, we attempt to address this relative absence of empirical study through five retrospective software modernization case studies. We use an explanatory case study approach to document the pre-modernization business goals, and to decide whether those goals have been achieved. The intended benefits for each of the five cases we considered were all (partially) met, and in most cases fully. Moreover, many cases exhibited a number of unintended benefits, and some reported detrimental effects of modernization.

Topic modeling has seen a surge in use for software comprehension. Although the models inferred from the source code are a great source of knowledge, they fail to fully capture the conceptual relationships between the topics. Here we investigate the use of interactive topic modeling for source code analysis by feeding-in information from the end-users, including developers and architects, to refine the inferred topic models. We have implemented a web-based toolkit called ITMViz to provide support to interpret the topic models, and use the results to cluster modules together. A medium-sized Java project is used to evaluate our approach in understanding the software system.

Unsupervised software clustering is the problem of automatically decomposing the software system into meaningful units. Some approaches solely rely on the structure of the system, such as the module dependency graph, to decompose the software systems into cohesive groups of modules. Other techniques focus on the informal knowledge hidden within the source code itself to retrieve the modular architecture of the system. However both techniques in the case of large systems fail to produce decompositions that correspond to the actual architecture of the system. To overcome this problem, we propose a novel approach to clustering software systems by incorporating knowledge from different viewpoints of the system, such as the knowledge embedded within the source code as well as the structural dependencies within the system, to produce a clustering. In this setting, we adopt a search-based approach to the encoding of multi-view clustering and investigate two approaches to tackle this problem, one based on a linear combination of objectives into a single objective, the other a multi-objective approach to clustering. We evaluate our approach against a set of substantial software systems. The two approaches are evaluated on a dataset comprising of 10 Java open source projects. Finally, we propose two techniques based on interpolation and hierarchical clustering to combine different results obtained to yield a single result for single-objective and multi-objective encodings, respectively.

Existing research in legacy system modernization has traditionally focused on technical challenges, and takes the standpoint that legacy systems are obsolete, yet crucial for an organization's operation. Nonetheless, it remains unclear whether practitioners in the industry also share this perception. This paper describes the outcome of an exploratory study in which 26 industrial practitioners were interviewed on what makes a software system a legacy system, what the main drivers are that lead to the modernization of such systems, and what challenges are faced during the modernization process. The findings of the interviews have been validated by means of a survey with 198 respondents. The results show that practitioners value their legacy systems highly, the challenges they face are not just technical, but also include business and organizational aspects.

Legacy systems are vitally important for the continuation of business in an enterprise as they support complex core business processes. However, legacy systems have several well-known disadvantages such as being inflexible and hard to maintain, so momentum is growing to evolve those systems into new technology environments. Recently, service-oriented architecture has emerged as a promising architectural style that enables existing legacy systems to expose their functionality as services, without making significant changes to the legacy systems themselves. A significant number of the legacy to service migration approaches address the technical perspective (i.e., supporting technology) to expose the legacy code as services. The other approaches focus on determining the feasibility of the migration that includes economical and technical feasibility, based on the characteristics of existing legacy system and the requirements of the target SOA system. In this paper, a legacy to SOA migration method that does not single out the migration feasibility and technical perspectives, but combines these two perspectives of migration, is proposed. Method engineering is used to develop the migration method by reusing method fragments from existing service-oriented development methods. Then, concept slicing is used to develop the service by extracting the relevant parts of the legacy code. The method is evaluated and enhanced by interviewing experts and further validated with two case studies. The method is found to be appropriate and effective in extracting services from legacy code with the aim of reusing these services in new configurations.