In a typical introductory programming course, grading of student submitted programs is often done manually by examining the source code prior to assigning the final grade for reasons such as checking for compliance to some criteria (e.g. ‘Use iteration, not recursion’, or ‘do not use additional arrays'), or for allotting partial marks. A rubric is often used by graders to grade according such criteria. However, manual grading of source code can be labor-intensive and impractical for large-scale online courses. Therefore, in this paper, we propose techniques based on Large Language Models (LLM) for code to automatically grade student programs according to instructor-specified rubrics. Leveraging a dataset of 27966 datapoints that we created, we study a total of 44 combinations of different open source LLMs and methodologies including Zero-Shot prompting, Few-Shot prompting, Supervised Fine Tuning, QLoRA, Direct Preference Optimization (DPO), code scrambling and code augmentation. To our knowledge, we are the first to address the generalized source code grading problem and to propose a solution with promising results. We find that among the models we studied, while Codestral 22B achieves a high micro-accuracy of 85% without any fine-tuning, Qwen-2.5-Coder-7B-Instruct with DPO fine-tuning achieves the same micro-accuracy with only 35 % of the GPU memory usage and 10 % of the inference time taken by Codestral 22B.

In introductory programming courses, autograders typically evaluate student programs by running testcases without inspecting the source code. However, educational grading often requires manual code inspection for two key reasons: (1) to award partial marks for code that may fail test cases but is partially correct, and (2) to assign marks based on code quality or specific criteria set by the instructor, such as requiring a particular algorithm, e.g., bubble sort. Rubric-based subjective grading is beneficial for these reasons, but manual grading for large course enrollments is time consuming. This demo introduces TA Buddy, an AI assistant integrated with IIT Bombay's BodhiTree Evalpro platform, which is designed to streamline grading in introductory programming courses. It is powered by a pre-trained code LLM, which was fine-tuned with a dataset created here at IITB Bombay. Its key benefits include speeding up the grading process with AI-generated suggestions for ratings of the criteria of a grading rubric. Furthermore, it provides feedback with justifications for assigned grades, making it useful for large courses where manual grading is time-consuming. Note that TA-Buddy only suggests grades to TAs, TAs are still required to review the grades and accept or reject them. In that sense, TA-Buddy offers an AI-Assisted grading option to TAs. This hybrid approach reduces grading time by up to 45% while maintaining an average match of 90% (on a sample of six problems) with un-assisted manual grades.

In a typical introductory programming course, grading student-submitted programs involves an autograder which compiles and runs the programs and tests their functionality with predefined test cases, with no attention to the source code. However, in an educational setting, grading based on inspection of the source code is required for two main reasons (1) awarding partial marks to 'partially correct' code that may be failing the testcase check (2) awarding marks (or penalties) based on source code quality or specific criteria that the instructor may have laid out in the problem statement (e.g. 'implement sorting using bubble-sort'). However, grading based on studying the source code can be highly time consuming when the course has a large enrollment. In this paper we present the design and evaluation of an AI Assistant for source code grading, which we have named TA Buddy. TA Buddy is powered by Code Llama, a large language model especially trained for code related tasks, which we fine-tuned using a graded programs dataset. Given a problem statement, student code submissions and a grading rubric, TA Buddy can be asked to generate suggested grades, i.e. ratings for the various rubric criteria, for each submission. The human teaching assistant (TA) can then accept or overrule these grades. We evaluated the TA Buddy-assisted manual grading against 'pure' manual grading and found that the time taken to grade reduced by 24% while maintaining grade agreement in the two cases at 90%.

Current autograders of programming assignments are typically program output based; they fall short in many ways: e.g. they do not carry out subjective evaluations such as code quality, or whether the code has followed any instructor specified constraints; this is still done manually by teaching assistants. In this paper, we tackle a specific aspect of such evaluation: to verify whether a program implements a specific algorithm that the instructor specified. An algorithm, e.g. bubble sort, can be coded in myriad different ways, but a human can always understand the code and spot, say a bubble sort, vs. a selection sort. We develop and compare four approaches to do precisely this: given the source code of a program known to implement a certain functionality, identify the algorithm used, among a known set of algorithms. The approaches are based on code similarity, Support Vector Machine (SVM) with tree or graph kernels, and transformer neural architectures based only source code (CodeBERT), and the extension of this that includes code structure (GraphCodeBERT). Furthermore, we use a model for explainability (LIME) to generate insights into why certain programs get certain labels. Results based on our datasets of sorting, searching and shortest path codes, show that GraphCodeBERT, fine-tuned with scrambled source code, i.e., where identifiers are replaced consistently with arbitrary words, gives the best performance in algorithm identification, with accuracy of 96--99% depending on the functionality. Additionally, we add uncalled function source code elimination to our pre-processing pipeline of test programs, to improve the accuracy of classification of obfuscated source code.

A multi-tier Internet server application needs to be analyzed for its performance before it is released. Performance analysis is usually done by (a) load testing of the application on a testbed and (b) building a performance model of the application. While there are a plethora of Web load-generator tools available, there are two problems with these tools: one, the tests have to be configured manually, which can lead to a time-consuming trial-and-error process until the desired performance charts in the appropriate load ranges are obtained; and two, the load generator tools do not produce output that is directly useful for creating a performance model of the application. In this paper, we present AutoPerf, a load generator tool designed to meet two distinct goals, named capacity analysis and profiling. The goal of capacity analysis is to run a comprehensive load test on a Web application, in an appropriately chosen range, at a minimal number of load levels, while still producing an accurate graph of throughput and response time vs load levels. The goal of profiling is to generate a detailed server resource usage profile per request type, without instrumenting the application code. This data (e.g. CPU execution time by Web server for one request) is crucial for parameterizing performance models of the application. AutoPerf intelligently plans and configures its load tests by using analytical results from queuing theory along with some heuristics. Results show that AutoPerf is able to run performance tests very efficiently while still producing an accurate chart of performance metrics.

This paper presents a case study of 278 first year students enrolled in a CS101 course of the department of Computer Science at the Indian Institute of Technology Bombay. The sufficiency of a low cost (Rs.7,500 or $110) laptop for this course is assessed. From the feedback questionnaires, the students' perception on the likability of this device is assessed. It is found that the students like this device. Users of this device used it as much others who had access to more expensive computers, and also performed as well as others in the course. This device is useful for several other applications also, and hence is a candidate for mass deployment in developing countries, such as India.

The performance analysis of a server application and the sizing of the hardware required to host it in a data center continue to be pressing issues today. With most server-grade computers now built with "frequency-scaled CPUs" and other such devices, it has become important to answer performance and sizing questions in the presence of such hardware. PowerPerfCenter is an application performance modeling tool that allows specification of devices whose operating speeds can change dynamically. It also estimates power usage by the machines in presence of such devices. Furthermore, it allows specification of a dynamic workload which is required to understand the impact of power management. We validated the performance metrics predicted by PowerPerfCenter against measured ones of an application deployed on a test-bed consisting of frequency-scaled CPUs, and found the match to be good. We also used PowerPerfCenter to show that power savings may not be significant if a device does not have different idle power consumption when configured with different operating speeds.

The success probability of I/O requests in presence of failures is increased by a combination of failover mechanisms built into the storage server, multiple access paths from I/O clients to the server, and timeout-retry mechanisms at the client itself. We define and evaluate a unified availability metric, request failures per million (RFPM), which quantifies request failure probability while taking into account client-side as well as server-side mechanisms. We calculate this metric using a two-level model of I/O service - a probability tree that captures the I/O driver behaviour, and a set of CTMC (Continuous Time Markov Chain) models that capture failover mechanisms at the server. The I/O driver model captures detailed timeout-retry mechanisms including retries at multiple ports (“multipathing”). The server model captures transient phenomena such as failure detection, takeover and emulation behaviour of a paired storage controller. The model shows that client retry mechanisms provide significant improvement in request success probability. The model is then used to study the sensitivity of RFPMs to parameters such as timeouts, reboot time and failure detection delay. The results show that the model can help in answering several what-if questions related to how system parameters impact request success rate.

The system capacity available to a multi-tier Web based application is often a dynamic quantity. Most static threshold-based overload control mechanisms are best suited to situations where the system's capacity is constant or the bottleneck resource is known. However, with varying capacity, the admission control mechanism needs to adapt dynamically. We propose and implement an adaptive admission control mechanism that adjusts the admitted load to compensate for changes in system capacity. The proposed solution is implemented as a proxy server between clients and front-end Web servers. The proxy monitors dasiablack-boxpsila performance metrics-response time and rate of successfully completed requests (goodput). With these measurements as indicators of system state, we employ a control theory based feedback loop to dynamically determine the rate of admitted requests. The objective is to balance changes in response time and changes in goodput, while preventing overloads due to reduction in available system capacity. We evaluate our mechanism with experiments on a test-bed and find that it is able to maintain higher productivity than a static admission control scheme.

A distributed connection admission control (CAC) scheme where stations independently admit or reject flows based on channel utilization threshold is easy to implement for WLANs. However, owing to variable protocol capacity, the advisable threshold also is variable, limiting the efficacy of fixed threshold based CAC. If the admission threshold is tuned to reflect the current protocol capacity, better throughput can be obtained while avoiding WLAN overload. In this paper, we propose a feedback based scheme to tune the utilization threshold for admission. We select the performance metric used for feedback, derive the system model from empirical data, and use control theory to design a feedback controller. Further, we present an analytical model and heuristics using which the controller can be adapted to work under diverse operating scenarios. Simulation results of proportional and proportional-integral controllers in OPNET suggest that the feedback based CAC is able to avoid WLAN overload and achieve high throughput despite large changes in protocol capacity.

Multimedia applications can be offered to mobile users on the IEEE 802.11 WLAN, if their bandwidth, delay and jitter requirements are met, even in the presence of handoffs. Applications such as VoIP require a delay of less than 150 msecs and packet loss less than 3%. Studies have shown that the latencies achieved by existing handoff implementations can exceed 200 ms and packet loss can exceed 10%. Thus, there is a need for fast handoff solutions, that can meet multimedia traffic requirements. We propose a mechanism for layer-2 fast handoff with the help of background scanning, restricted channel set and preauthentication which does not require any code modification at the access points. The mechanism has been implemented in the MadWiFi wireless open source Linux drivers and tested in an indoor wireless environment. Results have shown that our mechanism achieves latency of less than 10 msec with negligible packet loss even in the presence of handoffs. Thus our mechanism makes mobile multimedia applications possible in IEEE 802.11 WLANs.

Many overload control mechanisms for Web based applications aim to prevent overload by setting limits on factors such as admitted load, number of server threads, buffer size. For this they need online measurements of metrics such as response time, throughput, and resource utilization. This requires instrumentation of the server by modifying server code, which may not be feasible or desirable. An alternate approach is to use a proxy between the clients and servers.We have developed a proxy-based overload control platform called MASTH Proxy--Multi-class Admission-controlled Self-Tuning HTTP Proxy. It records detailed measurements, supports multiple request classes, manages queues of HTTP requests, provides tunable parameters and enables easy implementation of dynamic overload control. This gives designers of overload control schemes a platform where they can concentrate on developing the core control logic, without the need to modify upstream server code.

We present a tool, PerfCenter, that takes as input the deployment, configuration, message flow and workload details of the hardware and software servers in an application hosting center, and predicts the performance of the applications. We allow for a hierarchical specification of the data center, where software is deployed on machines, machines consist of hardware devices and are deployed on LANs. We also explicitly model network links between LANs and model the contention at those links due to messages exchanged between servers. While tools and methodologies for such analysis have been proposed earlier, our approach allows for the most natural specification of a "data center" architecture, and is best suited for aiding in design decisions regarding deployment and configuration of software on various hardware architecture scenarios. The tool takes this high level input and generates the underlying queueing network, which is then solved analytically. Since we allow for synchronous method calls, and model contention at software as well as hardware resources, the generated queueing network is solved using approximate methods. We validate the solution against results obtained from a measurement testbed, and found that the predicted values were reasonably accurate.

Admission control as a mechanism for providing QoS requires an accurate description of the requested flow as well as already admitted flows. Since 802.11 WLAN capacity is shared between flows belonging to all stations, admission control requires knowledge of all flows in the WLAN. Further, estimation of the load-dependent WLAN capacity through analytical model requires inputs about channel data rate, payload size and the number of stations. These factors combined point to a centralized admission control whereas for 802.11 DCF it is ideally performed in a distributed manner. The use of measurements from the channel avoids explicit inputs about the state of the channel described above. BUFFET, a model based measurement-assisted distributed admission control scheme for DCF proposed in this paper relies on measurements to derive model inputs and predict WLAN saturation, thereby maintaining average delay within acceptable limits. Being measurement based, it adapts to heterogeneous flows too, making it completely autonomous and distributed. Performance analysis and comparison with two other schemes using OPNET simulations suggests that BUFFET is able to ensure average delay under 7ms at a near-optimal throughput.

A distributed software system uses a complex system of resources which work together to process a request. A typical web-based request flows through various servers including Web servers, database servers, Java application servers, etc, deployed on various hardware platforms. Such a request encounters various forms of delays at and between these servers: communication delay, processing delay and queueing delay. Queueing delay is incurred at every point where there is any contention for resources, e.g. for acquiring a thread, or the CPU, or the lock to a log file. Queueing delay depends on the rate at which requests arrive for that particular resource, which in turn depends on the user behavior, the flow of the request, and the deployment of the servers. Given the number and type of soft and hard resources that make up a distributed system, it is a non-trivial task to quantify these delays. To address this problem, a number of methodologies and tools have been proposed, which allow a distributed system to be specified at a high level, and which generate and solve an underlying model using queueing theory techniques, to answer questions such as what the response time of a request is, what the bottleneck server is, and so on. In this tutorial we will review the state-of-the art in methods and tools for modeling and analyzing distributed software systems. This includes: Motivating examples of web-based multi-tier server systems Queueing systems primer (M/M/c/K, M/G/1 etc). Simple examples of application of queueing theory to software servers Introductory example of the "layered queueing network" method Overview of generalized software performance modeling methods and tools Real-life applicability of modeling methodologies: comparisons with measured performance.

We present a load generator and performance measurement tool AutoPerf which requires minimal input and configuration from the user, and produces a comprehensive capacity analysis as well as server-side resource usage profile of a Web-based distributed system, in an automated fashion. The tool requires only the workload and deployment description of the distributed system, and automatically sets typical parameters that load generator programs need, such as maximum number of users to be emulated, number of users for each experiment, warm-up time, etc. The tool also does all the co-ordination required to generate a critical type of measure, namely, resource usage per transaction or per user for each software server. This is a necessary input for creating a performance model of a software system.

We present a methodology and tool for performance analysis of distributed server systems, which allows high-level specification of the system, and generates and solves the underlying queueing network model. Our approach is different from the existing ones in that the specification captures the natural manner in which application servers are deployed on machines and machines are deployed on networks. The model does not impose any strict tiers on the server system. Multiple use case scenarios can be specified, and the tool computes measures such as end-to-end response times for each scenario while taking into account queueing delays at the hardware device, software threads and at the network. The development of the tool is ongoing, and will include detailed network protocol models as well as more flexible distributed system behavior, in the future.

Various RF based location determination systems have been proposed that use received signal strength fingerprints to identify locations. We implemented a Bayesian method for location determination in a WLAN testbed and were able to get about 80% accuracy of estimation with a precision of 2.5 meters. We proposed two mechanisms to improve this accuracy: 1) Kalman filtering to remove noise in received signal strength readings and 2) a technique which uses estimates from multiple observers to determine the location. Results from an IEEE 802.11b based implementation of the first method shows that Kalman filtering during the training phase can increase this accuracy to 90%. The multiple observer technique that uses received signal strength readings of the mobile device at the access point, also shows a similar increase in accuracy. Since the multiple observer technique requires more time and resources, we conclude that Kalman filtering is a more efficient and simple way to increase the accuracy of location determination