Application Understanding task aims to help users comprehend an application's capabilities by systematically analyzing its artifacts. Ideally, such summaries should align with how the application is used in practice, highlighting essential workflows and functional modules in a structured manner. However, existing automated approaches often fall short of this expectation. Lack of application-specific background and domain knowledge limits the system's ability to present functionalities meaningfully. To address these challenges, we propose a novel agentic approach leveraging multimodal LLMs that integrate code analysis, textual artifacts, and domain knowledge to identify key business flow entities-such as programs and tables-within a repository and infer application workflows. This work opens new avenues in LLM-guided software comprehension, bridging the gap between code-centric insights and high-level business process understanding.

Kishan Maharaj, Vitobha Munigala, Srikanth G. Tamilselvam, Prince Kumar, Sayandeep Sen, Palani Kodeswaran, Abhijit Mishra, Pushpak Bhattacharyya. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

In a typical introductory programming course, grading of student submitted programs is often done manually by examining the source code prior to assigning the final grade for reasons such as checking for compliance to some criteria (e.g. ‘Use iteration, not recursion’, or ‘do not use additional arrays'), or for allotting partial marks. A rubric is often used by graders to grade according such criteria. However, manual grading of source code can be labor-intensive and impractical for large-scale online courses. Therefore, in this paper, we propose techniques based on Large Language Models (LLM) for code to automatically grade student programs according to instructor-specified rubrics. Leveraging a dataset of 27966 datapoints that we created, we study a total of 44 combinations of different open source LLMs and methodologies including Zero-Shot prompting, Few-Shot prompting, Supervised Fine Tuning, QLoRA, Direct Preference Optimization (DPO), code scrambling and code augmentation. To our knowledge, we are the first to address the generalized source code grading problem and to propose a solution with promising results. We find that among the models we studied, while Codestral 22B achieves a high micro-accuracy of 85% without any fine-tuning, Qwen-2.5-Coder-7B-Instruct with DPO fine-tuning achieves the same micro-accuracy with only 35 % of the GPU memory usage and 10 % of the inference time taken by Codestral 22B.

Many enterprise systems including large-scale deployment platforms like Ansible provide a declarative user interface through programming languages like JavaScript Object Notation (JSON). These systems maintain integrity through validation rules, typically enforced via JSON schemas. However, enterprise tasks in these systems are often complex, involving multiple schemas, which makes it challenging for the developers to select the appropriate ones and write schema-compliant code snippets for each task. Recently, Large Language Models (LLMs) have shown promising performance for many declarative code generation tasks when adopted with constrained generation using a pre-known schema. However, to cater to real-world enterprise tasks, each task often requiring multiple code snippets to generate while ensuring compliance with their respective schemas, we introduce a novel framework that allows LLMs to generate multiple code snippets while choosing an appropriate schema for each of the snippets for constrained generation. To the best of our knowledge, we are the first to study this crucial enterprise problem for declarative systems and preliminary results on two real-world use cases demonstrate substantial improvements in both syntactic and semantic task performance. These findings highlight the potential of the approach to enhance the reliability and scalability of LLMs in declarative enterprise systems, indicating a promising direction for future research and development.

In a typical introductory programming course, grading student-submitted programs involves an autograder which compiles and runs the programs and tests their functionality with predefined test cases, with no attention to the source code. However, in an educational setting, grading based on inspection of the source code is required for two main reasons (1) awarding partial marks to 'partially correct' code that may be failing the testcase check (2) awarding marks (or penalties) based on source code quality or specific criteria that the instructor may have laid out in the problem statement (e.g. 'implement sorting using bubble-sort'). However, grading based on studying the source code can be highly time consuming when the course has a large enrollment. In this paper we present the design and evaluation of an AI Assistant for source code grading, which we have named TA Buddy. TA Buddy is powered by Code Llama, a large language model especially trained for code related tasks, which we fine-tuned using a graded programs dataset. Given a problem statement, student code submissions and a grading rubric, TA Buddy can be asked to generate suggested grades, i.e. ratings for the various rubric criteria, for each submission. The human teaching assistant (TA) can then accept or overrule these grades. We evaluated the TA Buddy-assisted manual grading against 'pure' manual grading and found that the time taken to grade reduced by 24% while maintaining grade agreement in the two cases at 90%.

Sameer Pimparkhede, Mehant Kammakomati, Srikanth G. Tamilselvam, Prince Kumar, Ashok Pon Kumar, Pushpak Bhattacharyya. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models (LLM).Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, like pseudocode.In this paper, we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models.We manually create a dataset 1 of pseudo-code prompts for 132 different tasks spanning classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset (Wang et al., 2022b).Using these prompts along with their counterparts in natural language, we study their performance on two LLM families -BLOOM (Scao et al., 2023), CodeGen (Nijkamp et al., 2023).Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks.We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance.To the best of our knowledge, our work is the first to demonstrate how pseudocode prompts can be helpful in improving the performance of pre-trained LMs.* Equal contribution 1 Code and dataset available at https://github.com/ mayank31398/pseudo-code-instructions Listing 1 An example pseudo-code instruction for the task from Wang et al. (2022b).A successful model is expected to use the provided pseudo-code instructions and output responses to a pool of evaluation instances.1 def generate_sentiment(sentence: str) -> str: 2 """For the given sentence, the task is to 3 predict the sentiment.For positive 4 sentiment return "positive" else return 5 "negative".

Learning effective representations of source code is critical for any Machine Learning for Software Engineering (ML4SE) system. Inspired by natural language processing, large language models (LLMs) like Codex and CodeGen treat code as generic sequences of text and are trained on huge corpora of code data, achieving state of the art performance on several software engineering (SE) tasks. However, valid source code, unlike natural language, follows a strict structure and pattern governed by the underlying grammar of the programming language. Current LLMs do not exploit this property of the source code as they treat code like a sequence of tokens and overlook key structural and semantic properties of code that can be extracted from code-views like the Control Flow Graph (CFG), Data Flow Graph (DFG), Abstract Syntax Tree (AST), etc. Unfortunately, the process of generating and integrating code-views for every programming language is cumbersome and time consuming. To overcome this barrier, we propose our tool COMEX - a framework that allows researchers and developers to create and combine multiple code-views which can be used by machine learning (ML) models for various SE tasks. Some salient features of our tool are: (i) it works directly on source code (which need not be compilable), (ii) it currently supports Java and C#, (iii) it can analyze both method-level snippets and program-level snippets by using both intra-procedural and inter-procedural analysis, and (iv) it is easily extendable to other languages as it is built on tree-sitter - a widely used incremental parser that supports over 40 languages. We believe this easy-to-use code-view generation and customization tool will give impetus to research in source code representation learning methods and ML4SE. The source code and demonstration of our tool can be found at https://github.com/IBM/tree-sitter-codeviews and https://youtu.be/GER6U87FVbU, respectively.

When microservices are written from scratch, APIs are usually made stateless. However, when an existing monolith application is decomposed into microservices, it may not be possible to make all the APIs stateless. Therefore, objects transferred via APIs may contain pointers. Consequently, data transfer via an API i.e., from a client address space to a server address space, reconstruction at the server, and returning to the client become non-trivial operations.Conventionally, data transfer between microservices is done using JSON, which serializes pointers to values that they point to. Once the data in JSON reaches the server, deserialization creates objects of the original types on the server. However, deserialization is unable to return the same objects passed by the client because serialization leads to loss of pointer information. We propose to apply pointer swizzling to solve this problem. Pointer swizzling modifies the definition of the class by introducing ID of the object and by replacing all pointers with IDs of the objects it refers. These IDs help to maintain correct reference in the server. After the server API operates on the objects, the server returns new objects of the same types to the client. These new objects need to be plugged back in the client address space i.e., pointers to the old objects in the client need to now point to the corresponding new objects. This plugging back is non-trivial because we do not know how the old objects map to the new objects. We propose creating memory maps at runtime to overcome this challenge.

Monolithic software encapsulates all functional capabilities into a single deployable unit. But managing it becomes harder as the demand for new functionalities grow. Microservice architecture is seen as an alternative as it advocates building an application through a set of loosely coupled small services wherein each service owns a single functional responsibility. But the challenges associated with the separation of functional modules, slows down the migration of a monolithic code into microservices. In this work, we propose a representation learning based solution to tackle this problem. We use a heterogeneous graph to jointly represent software artifacts (like programs and resources) and the different relationships they share (function calls, inheritance, etc.), and perform a constraint-based clustering through a novel heterogeneous graph neural network. Experimental studies show that our approach is effective on monoliths of different types.

Microservices are becoming the defacto design choice for software architecture. It involves partitioning the software components into finer modules such that the development can happen independently. It also provides natural benefits when deployed on the cloud since resources can be allocated dynamically to necessary components based on demand. Therefore, enterprises as part of their journey to cloud, are increasingly looking to refactor their monolith application into one or more candidate microservices; wherein each service contains a group of software entities (e.g., classes) that are responsible for a common functionality. Graphs are a natural choice to represent a software system. Each software entity can be represented as nodes and its dependencies with other entities as links. Therefore, this problem of refactoring can be viewed as a graph based clustering task. In this work, we propose a novel method to adapt the recent advancements in graph neural networks in the context of code to better understand the software and apply them in the clustering task. In that process, we also identify the outliers in the graph which can be directly mapped to top refactor candidates in the software. Our solution is able to improve state-of-the-art performance compared to works from both software engineering and existing graph representation based techniques.

In this paper, we propose a novel approach for monolith decomposition, that maps the implementation structure of a monolith application to a functional structure that in turn can be mapped to business functionality. First, we infer the classes in the monolith application that are distinctively representative of the business functionality in the application domain. This is done using formal concept analysis on statically determined code flow structures in a completely automated manner. Then, we apply a clustering technique, guided by the inferred representatives, on the classes belonging to the monolith to group them into different types of partitions, mainly: 1) functional groups representing microservice candidates, 2) a utility class group, and 3) a group of classes that require significant refactoring to enable a clean microservice architecture. This results in microservice candidates that are naturally aligned with the different business functions exposed by the application. A detailed evaluation on four publicly available applications show that our approach is able to determine better quality microservice candidates when compared to other existing state of the art techniques. We also conclusively show that clustering quality metrics like modularity are not reliable indicators of microservice candidate goodness.

For document level sentiment analysis (SA), Subjectivity Extraction, ie., extracting the relevant subjective portions of the text that cover the overall sentiment expressed in the document, is an important step. Subjectivity Extraction, however, is a hard problem for systems, as it demands a great deal of world knowledge and reasoning. Humans, on the other hand, are good at extracting relevant subjective summaries from an opinionated document (say, a movie review), while inferring the sentiment expressed in it. This capability is manifested in their eye-movement behavior while reading: words pertaining to the subjective summary of the text attract a lot more attention in the form of gaze-fixations and/or saccadic patterns. We propose a multi-task deep neural framework for document level sentiment analysis that learns to predict the overall sentiment expressed in the given input document, by simultaneously learning to predict human gaze behavior and auxiliary linguistic tasks like part-of-speech and syntactic properties of words in the document. For this, a multi-task learning algorithm based on multi-layer shared LSTM augmented with task specific classifiers is proposed. With this composite multi-task network, we obtain performance competitive with or better than state-of-the-art approaches in SA. Moreover, the availability of gaze predictions as an auxiliary output helps interpret the system better; for instance, gaze predictions reveal that the system indeed performs subjectivity extraction better, which accounts for improvement in document level sentiment analysis performance.

The sentiment aggregation problem accounts for analyzing the sentiment of a user towards various aspects/features of a product, and meaningfully assimilating the pragmatic significance of these features/aspects from an opinionated text. The current paper addresses the sentiment aggregation problem, by assigning weights to each aspect appearing in the user-generated content, that are proportionate to the strategic importance of the aspect in the pragmatic domain. The novelty of this paper is in computing the pragmatic significance (weight) of each aspect, using graph centrality measures (applied on domain specific ontology-graphs extracted from ConceptNet), and deeply ingraining these weights while aggregating the sentiments from opinionated text. We experiment over multiple real-life product review data. Our system consistently outperforms the state of the art - by as much as a F-score of 20.39% in one case.