Experimental evaluations of software engineering innovations, e.g., tools and processes, often include human-subject studies as a component of a multi-pronged strategy to obtain greater generalizability of the findings. However, human-subject studies in our field are challenging, due to the cost and difficulty of finding and employing suitable subjects, ideally, professional programmers with varying degrees of experience. Meanwhile, large language models (LLMs) have recently started to demonstrate human-level performance in several areas. This paper explores the possibility of substituting costly human subjects with much cheaper LLM queries in evaluations of code and coderelated artifacts. We study this idea by applying six state-of-theart LLMs to ten annotation tasks from five datasets created by prior work, such as judging the accuracy of a natural language summary of a method or deciding whether a code change fixes a static analysis warning. Our results show that replacing some human annotation effort with LLMs can produce inter-rater agreements equal or close to human-rater agreement. To help decide when and how to use LLMs in human-subject studies, we propose model-model agreement as a predictor of whether a given task is suitable for LLMs at all, and model confidence as a means to select specific samples where LLMs can safely replace human annotators. Overall, our work is the first step toward mixed human-LLM evaluations in software engineering.

One of the central tasks in software maintenance is being able to understand and develop code changes. Thus, given a natural language description of the desired new operation of a function, an agent (human or AI) might be asked to generate the set of edits to that function to implement the desired new operation; likewise, given a set of edits to a function, an agent might be asked to generate a changed description, of that function’s new workings. Thus, there is an incentive to train a neural model for change-related tasks. Motivated by this, we offer a new, “natural”, large dataset of coupled changes to code and documentation mined from actual high-quality GitHub projects, where each sample represents a single commit where the code and the associated docstring were changed together. We present the methodology for gathering the dataset, and some sample, challenging (but realistic) tasks where our dataset provides opportunities for both learning and evaluation. We find that current models (specifically Llama-3.1 405B, Mixtral $8 \times 22 \mathrm{~B}$) do find these maintenance-related tasks challenging.

With the advent of powerful neural language models, AI-based systems to assist developers in coding tasks are becoming widely available; Copilot is one such system. Copilot uses Codex, a large language model (LLM), to complete code conditioned on a preceding "prompt". Codex, however, is trained on public GitHub repositories, viz., on code that may include bugs and vulnerabilities. Previous studies [1], [2] show Codex reproduces vulnerabilities seen in training. In this study, we examine how prone Codex is to generate an interesting bug category, single statement bugs, commonly referred to as simple, stupid bugs or SStuBs in the MSR community. We find that Codex and similar LLMs do help avoid some SStuBs, but do produce known, verbatim SStuBs as much as 2x as likely than known, verbatim correct code. We explore the consequences of the Codex generated SStuBs and propose avoidance strategies that suggest the possibility of reducing the production of known, verbatim SStubs, and increase the possibility of producing known, verbatim fixes.

In this paper, we present a new variant of group trip planning (GTP) queries in spatial databases with weighted point of interests (POIs), called Weighted optimal Sequenced Group Trip Planning (WSGTP) query. Specifically, we focus on the sequenced version of the query where the order of visiting POI types are fixed by the users. Traditional GTP queries consider unweighted POIs which means all POIs are equal, whereas in WSGTP queries, each POI has a weight that determines its utility to the users. Given source and destination locations of a group of users and a set of POI types to be visited by the group in the given order, a WSGTP query retrieves an optimal POI set, consisting of one POI of each type, from a location database. Unlike traditional GTP queries that minimize traveling distance of the group, the optimal POI set of a WSGTP query minimizes a cost function which implies minimizing the aggregate group trip distance as well as maximizing the aggregate utility obtained from the POIs. Thus, WSGTP query returns the best solution for a group trip in weighted POI scenarios. We provide an efficient solution to process WSGTP queries in both Euclidean space and road networks. Experiments show that our approach can compute WSGTP query solutions within reasonable time bounds and significantly outperforms a straight-forward approach.