The dynamic nature of cloud spending and pricing structures pose challenges for practitioners in IT Financial Operations (FinOps). Recent advances in agentic systems enables them to instead rely on agents for complex FinOps tasks such as drawing insights from their data through natural language queries. In this work, we present an IT FinOps Data Insights Agent, that implements “chat with your data” approach to support practitioners in their daily tasks. Our agent achieves up to 90% accuracy across ITBench FinOps scenarios.

Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

We address the contextual bandit problem for high-dimensional combinatorial action spaces involving a class of structured discrete constrained optimization problems. In this setting, key quantities within the optimization formulation, such as performance indicators contributing to the objective or constraints, must be estimated from data during the trial sequence of optimized actions. These problems frequently arise in the many operations management domains including IT resource allocation and retail assortment price optimization. We propose a novel, practical, and transparent approach based on general-purpose regression oracles, leveraging Inverse Gap Weighting (IGW) for seamless integration within an optimization framework. IGW sampling is efficiently managed by: (a) a column generation reformulation of the underlying Mixed Integer Programming (MIP) model which allows for flexible lower-level predictors, causal coherence, and efficient representation of large action spaces;(b) a diverse solution pool generation to balance the exploration-exploitation trade-off in large action spaces. To address non-smoothness in the reward function due to optimization constraints, we incorporate a risk-averse phased learning strategy. We validate our approach on a real-world auto-scaling problem in IT automation, achieving a significant reduction in cumulative regret through skillful exploration, with additional gains from risk-averse methods that effectively manage constraint violations.

This study characterizes GPU resilience in Delta, a large-scale AI system that consists of 1,056 A100 and H100 GPUs, with over 1,300 petaflops of peak throughput. We used 2.5 years of operational data (11.7 million GPU hours) on GPU errors. Our major findings include: (i) H100 GPU memory resilience is worse than A100 GPU memory, with 3.2x lower per-GPU MTBE for memory errors, (ii) The GPU memory error-recovery mechanisms on H100 GPUs are insufficient to handle the increased memory capacity, (iii) H100 GPUs demonstrate significantly improved GPU hardware resilience over A100 GPUs with respect to critical hardware components, (iv) GPU errors on both A100 and H100 GPUs frequently result in job failures due to the lack of robust recovery mechanisms at the application level, and (v) We project the impact of GPU node availability on larger-scales and find that significant overprovisioning of 5% is necessary to handle GPU failures.

In the context of cloud computing, enterprises are increasingly adopting multi-cloud strategies to enhance performance, ensure cost efficiency, and avoid vendor lock-in. This trend presents a significant challenge for the migration of AI for IT operations (AIOps) models across different cloud providers due to variations in architecture, performance, and data distribution. Traditional methods of re-training AIOps models for new cloud environments are labor-intensive and delay deployment. To address this issue, we introduce a novel framework called SAM (Subseries Augmentation-based Meta-learning), which facilitates seamless model migration between clouds without the need for re-training from scratch. SAM leverages data augmentation and meta-learning to efficiently adapt AIOps models to new cloud environments. It has proven effective in adapting anomaly detectors across various config-urations over both public and simulated datasets. We believe that SAM can also be adapted to other AI models used for automating IT tasks such as alerting and resource scaling.

The widespread adoption of public and hybrid clouds, along with elastic resources and various automation tools for dynamic deployment, has accelerated the rapid provisioning of compute resources as needed. Despite these advancements, numerous resources persist unnecessarily due to factors such as poor digital hygiene, risk aversion, or the absence of effective tools, resulting in substantial costs and energy consumption. Existing threshold-based techniques prove inadequate in effectively addressing this challenge. To address this issue, we propose an unsupervised machine learning framework to automatically identify resources that can be de-provisioned completely or summoned on a schedule. Application of this approach to enterprise data has yielded promising initial results, facilitating the segregation of productive workloads with recurring demands from non-productive ones.

Large language model (LLM) serving is becoming an increasingly critical workload for cloud providers. Existing LLM serving systems focus on interactive requests, such as chatbots and coding assistants, with tight latency SLO requirements. However, when such systems execute batch requests that have relaxed SLOs along with interactive requests, it leads to poor multiplexing and inefficient resource utilization. To address these challenges, we propose QLM, a queue management system for LLM serving. QLM maintains batch and interactive requests across different models and SLOs in a request queue. Optimal ordering of the request queue is critical to maintain SLOs while ensuring high resource utilization. To generate this optimal ordering, QLM uses a Request Waiting Time (RWT) Estimator that estimates the waiting times for requests in the request queue. These estimates are used by a global scheduler to orchestrate LLM Serving Operations (LSOs) such as request pulling, request eviction, load balancing, and model swapping. Evaluation on heterogeneous GPU devices and models with real-world LLM serving dataset shows that QLM improves SLO attainment by 40-90% and throughput by 20-400% while maintaining or improving device utilization compared to other state-of-the-art LLM serving systems. QLM's evaluation is based on the production requirements of a cloud provider. QLM is publicly available at https://www.github.com/QLM-project/QLM.

Selecting the right set of hyperparameters is crucial in time series forecasting. The classical temporal cross-validation framework for hyperparameter optimization (HPO) often leads to poor test performance because of a possible mismatch between validation and test periods. To address this test-validation mismatch, we propose a novel technique, H-Pro to drive HPO via test proxies by exploiting data hierarchies often associated with time series datasets. Since higher-level aggregated time series often show less irregularity and better predictability as compared to the lowest-level time series which can be sparse and intermittent, we optimize the hyperparameters of the lowest-level base-forecaster by leveraging the proxy forecasts for the test period generated from the forecasters at higher levels. H-Pro can be applied on any off-the-shelf machine learning model to perform HPO. We validate the efficacy of our technique with extensive empirical evaluation on five publicly available hierarchical forecasting datasets. Our approach outperforms existing state-of-the-art methods in Tourism, Wiki, and Traffic datasets, and achieves competitive result in Tourism-L dataset, without any model-specific enhancements. Moreover, our method outperforms the winning method of the M5 forecast accuracy competition.

As sophisticated enterprise applications move to the Web, some advanced user experiences become difficult to migrate due to prohibitively high computation, memory, and bandwidth requirements. State-dependent visualizations of large-scale data sets are particularly difficult since a change in the client’s context necessitates a change in the displayed results. This paper describes a Web architecture where clients are served a session-specific image of the data, with this image divided into tiles dynamically generated by the server. This set of tiles is supplemented with a corpus of metadata describing the immediate vicinity of interest; additional metadata is delivered as needed in a progressive fashion in support and anticipation of the user’s actions. We discuss how the design of this architecture was motivated by the goal of delivering a highly responsive user experience. As an example of a complete application built upon this architecture, we present OrgMaps, an interactive system for navigating hierarchical data, enabling fluid, low-latency navigation of trees of hundreds of thousands of nodes on standard Web browsers using only HTML and JavaScript.

This paper advocates the use of RESTful data services in mobile commerce information spaces. REST is a restriction of web services to a simple protocol centered on a limited set of access operations against data resources. This paper analyzes how REST may be used as a general design principle for mobile commerce spaces. This analysis is threefold. First, it provides an abstract component model that contains a minimal embodiment of the REST functionality needed to support mobile commerce. Second, it describes and analyzes the operation of a mobile commerce space, resulting in a reference REST-based data decomposition that may be reused across a variety of domains. Third, it provides and analyzes a set of usage cases in the mobile commerce domain, which can form the basis of future REST-oriented programming models and tools. This analysis is based on a substantial mobile commerce framework we have implemented, called Celadon. This framework is currently being used to demonstrate several ambitious mobile commerce applications.

Searching and presenting rich data using mobile devices is hard given their inherent I/O limitations. One approach for alleviating these limitations is device symbiosis, whereby the interaction with one’s personal mobile device is augmented by additionally engaging with more capable infrastructure d

Accessing information on mobile and wearable devices often requires the user's visual attention, and the precise operation of virtual or physical widgets. However, these interactions may sometimes be too time-consuming and socially inappropriate. To address this, we introduce a novel input/output device that is based on the manipulation of a retractable string in a polar coordinate frame. Depending on how the user pulls the string from its enclosure--to a particular length, at a particular angle--various system features may be directly accessed. Furthermore, we present our concept for a 1D pixel array, embedded in the string that may be used as a secondary 1D display. Since it is possible to unwind the display itself and trigger functionality with a single pull, information may be accessed and presented quickly, and perceived at a glance. We present scenarios for how the string input/output device may be used in conjunc-tion with the mobile device's primary 2D display and describe our augmented reality proof-of-concept prototype.

In this paper we introduce the inverted browser, a novel approach to enable mobile users to view content from their personal devices on public displays. The inverted browser is a network service to start and control a browser that is then used to view the content. In contrast to a traditional Web browser, which runs on the client device and pulls content from a server, content is pushed to the inverted browser from a personal data source upon user input. This approach allows a wide variety of personal content to be viewed by facilitating symbiotic relationships between mobile devices and intelligent displays in the environment. Our initial inverted browser prototype is based on a Web services wrapper around a traditional Web browser. Our experiments show that the inverted browser approach is superior to other solutions in terms of user convenience, ease of use, energy consumption, and privacy, but interaction latencies need improvement.

We present PowerNap, an OS power management scheme, which can significantly improve the battery life of mobile devices. The key feature of PowerNap is the skipping of the periodic system timer ticks associated with the operating system. On an idle device, this modification increases the time between successive timer interrupts and enables us to put the processor/system into a more efficient low power state. This saves the energy consumed by workless timer interrupts and the excess energy consumed by the processor in less efficient low power states. PowerNap is tightly integrated with the kernel and is designed for optimal control of the latency and energy associated with transitioning in and out of the low power states. We describe an implementation of PowerNap and its impact on system software. Experiments with IBM's WatchPad verify the ability of PowerNap to extend battery life. An analytical model that quantifies the ability of the scheme to reduce power is also presented. The model is in good agreement with experimental results. We apply the model to small form-factor devices which use processors that have a PowerDown state. In such devices, PowerNap may extend battery life by more than 42 percent for small processor workloads and for background power levels below 10 mW.

Several novel portable and wearable form factors have been prototyped at IBM over the last eight years. Each of the different form factors has some unique advantages and addresses different user needs. This invited paper present our experience building these prototypes and describe how improvements in technology have advanced our prototypes over time. Recently we have been investigating the middleware necessary to use a collection of devices symbiotically to address some of the limitations of each form factor. Drawbacks of some form factors include battery, small screen size, limited bandwidth and limited user interface capability. However form factors with such limitations also typically come with several advantages, i.e., they are always with you and provide a private display and a trusted platform. The paper sketches a few applications that could use devices symbiotically and outline some of the challenges that need to be overcome to enable this symbiosis.

A variety of portable and wearable device form factors such as the VisionPad, WatchPad, WearableData, MetaPad, Personal Mobile Hub, and SoulPad have been prototyped at IBM Research over the last several years. Each of different form factors has some unique advantages and addresses different user needs. This talk will present our experience building these prototypes and describe how we expect this landscape of devices to change. The continuous availability of wearable and mobile devices enables the deployment of many novel applications and scenarios but places heavy demands on energy, infrastructure support, and usability. Recently we have been investigating how some of the inherent limitations of mobile devices can be alleviated by symbiosis with more powerful stationary devices. Effective device symbiosis requires middleware that addresses device heterogeneity, improves usability, provides privacy and ensures security. The availability of a variety of new form factors coupled with device symbiosis will catalyze the way we conduct business, entertain and take care of ourselves

The ability to walk up to any computer, personalize it, and use it as one's own has long been a goal of mobile computing research. We present SoulPad, a new approach based on carrying an auto-configuring operating system along with a suspended virtual machine on a small portable device. With this approach, the computer boots from the device and resumes the virtual machine, thus giving the user access to his personal environment, including previously running computations. SoulPad has minimal infrastructure requirements and is therefore applicable to a wide range of conditions, particularly in developing countries. We report our experience implementing SoulPad and using it on a variety of hardware configurations. We address challenges common to systems similar to SoulPad, and show that the SoulPad model has significant potential as a mobility solution.

The session initiation protocol (SIP) has been widely adopted for instant messaging (IM) and VoIP telephony both by the enterprise and in service provider systems. Till now, SIP functionality has been accessible to end-users primarily via computing platforms such as an IM client on a laptop or on communication platforms such as an IP phone or a cell phone. We show that wearable devices such as the IBM Linux Watch (WatchPad/spl trade/) can play a powerful role as a control device in a SIP infrastructure, given their easily accessible, always available, and visible user-interfaces. We have designed and prototyped several key control applications, using a SIP User Agent on the WatchPad, such as the ability to initiate and route incoming and outgoing phone calls, receive simple queries as instant messages, and call initiation based on address books residing on wearable devices. We discuss other examples such as video conferencing and the authentication layer for workflow. Our investigation and implementation shows that wearable devices can plug an important gap and significantly enhance user experience.

Symbiotic displays are intelligent network devices that can be enlisted on demand by mobile devices, offering higher resolutions and larger viewing areas. Users of small mobile devices can consequently map various kinds of information to the displays that are most appropriate for comfortably viewing them. The availability of these displays in public areas, where passersby may glimpse another's sensitive information, introduces a privacy issue. To address this, we suggest a technique that combines the blurring of sensitive words on the public symbiotic display with a user interface that allows these words to be read on the display of a personal mobile device. We describe three ways in which such interactions can be enabled and discuss the architectural trade-offs associated with these approaches. We also describe a prototype implementation of an email reading application utilizing an IBM WatchPad and an Everywhere Display projector. This prototype was demonstrated at PerCom'04

Abstract A parallel raster algorithm to draw Gouraud shaded triangles is presented. At the heart of the algorithm is a new constrained parallel edge‐traversal technique. This parallel traversal represents an increased level of parallelism compared to the existing solutions. Next, traditional algorithms take different amounts of time to advance from one horizontal span to another for the left edge and the right edge of the triangle when the slope of one of the edges is more than one and that of the other edge is less than one. This causes one processor to wait for another processor. The parallel constrained edge traversal technique removes this problem by directly jumping from one span to the next. It also ensures that adjacent triangles that share an edge do not share any pixels. Moreover, no cracks occur between adjacent polygons. Unlike some existing algorithms whose complexity depends on the size of the bounding box of the triangle, the complexity of our algorithm is solely dependenton the perimeter and area of the triangle. Due to the above features, the algorithm presented here exposes a greater degree of parallelism at considerably lesser cost and achieves better processor utilization, compared to existing algorithms for this problem1, 2, 3, 4, 5, 6. The algorithm is well suited for hardware implementation.