Multi-agent LLM-driven systems for software development are rapidly gaining traction, offering new opportunities to enhance productivity. At Atlassian, we deployed Human-in-the-Loop Software Development Agents to resolve Jira work items and evaluated the generated code quality using functional correctness testing and GPT-based similarity scoring. This paper highlights two major challenges: the high computational costs of unit testing and the variability in LLM-based evaluations. We also propose future research directions to improve evaluation frameworks for Human-In-The-Loop software development tools.

Agile software development prioritizes the delivery of working software. However, there are challenges in the sprint planning process that could impact the reliability of sprint delivery. In this paper, we list three challenges related to the sprint planning process. We also discuss future work directions to facilitate the sprint planning process and mitigate those challenges for software teams.

In agile iterative development, an agile team needs to analyze documented information for effort estimation and sprint planning. While documentation can be changed, the documentation changes after sprint planning may invalidate the estimated effort and sprint plan. Hence, to help the team be aware of the potential documentation changes, we developed DocWarn to estimate the probability that a work item will have documentation changes. We developed three variations of DocWarn, which are based on the characteristics extracted from the work items (DocWarn-C), the natural language text (DocWarn-T), and both inputs (DocWarn-H).