Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

Automated Log Analysis tasks such as root cause analysis and fault prediction play a pivotal role in maintaining the overall application health. These tasks employ log parsers to extract the dynamic (variable) and constant (template) parts of a log line to generate a template. However, our observations indicate that not all templates carry equal significance. Hence, there is a need to prioritize which templates/variables to use for log analysis. In this paper, we introduce LogMId, a Logs-based Metric Identification method, which is designed to extract critical IT metrics from logs. Through LogMId, we aim to en-hance monitoring, observability tools and in turn Site Reliability Engineers to mine better insights from log data. We showcase the effectiveness of LogMId on a popular log analysis task of anomaly detection. Our experiments indicate that integrating previously used benchmark tools with LogMId features lead to improved results. Additionally, LogMId demonstrates effectiveness even with a smaller amount of training data, emphasising its utility.

Creativity, a process that generates novel and meaningful ideas, involves increased association between task-positive (control) and task-negative (default) networks in the human brain. Inspired by this seminal finding, in this study we propose a creative decoder within a deep generative framework, which involves direct modulation of the neuronal activation pattern after sampling from the learned latent space. The proposed approach is fully unsupervised and can be used off- the-shelf. Several novelty metrics and human evaluation were used to evaluate the creative capacity of the deep decoder. Our experiments on different image datasets (MNIST, FMNIST, MNIST+FMNIST, WikiArt and CelebA) reveal that atypical co-activation of highly activated and weakly activated neurons in a deep decoder promotes generation of novel and meaningful artifacts.

Acoustic signals contain rich information of the environment. They can be used for detecting anomalous events such as in automated machine monitoring. In this demonstration, we present our acoustic anomaly detection system that captures acoustic signals and classifies them using machine learning techniques. Our system includes a server for sound management and model training, a mobile client for sound capturing and real-time classification, and a workbench that acts as a user interface. We will show the full operational pipeline of our system in this demonstration.

Open user modeling has been perceived as an important mechanism to enhance the effectiveness of personalization. However, several studies have reported that open and editable user models can harm the performance of personalized search systems. This paper re-examines the value of open and editable user models in the context of personalized search. We implemented a personalized search system with 2D user manipulatable visualization and concept-based user model components. A user study result suggests that the proposed visualization-based open user modeling approach can be beneficial for adaptive search.

Visualization has proven to be a useful tool for understanding network structures. Yet the dynamic nature of social media networks requires powerful visualization techniques that go beyond static network diagrams. To provide strong temporal network visualization tools, designers need to understand what tasks the users have to accomplish. This paper describes a taxonomy of temporal network visualization tasks. We identify the 1) entities, 2) properties, and 3) temporal features, which were extracted by surveying 53 existing temporal network visualization systems. By building and examining the task taxonomy, we report which tasks are well covered by existing systems and make suggestions for designing future visualization tools. The feedback from 12 network analysts helped refine the taxonomy.

Expert Collaborative Filtering is an approach to recommender systems in which recommendations for users are derived from ratings coming from domain experts rather than peers. In this paper we present an implementation of this approach in the music domain. We show the applicability of the model in this setting, and show how it addresses many of the shortcomings in traditional Collaborative Filtering such as possible privacy concerns. We also describe a number of technologies and an architectural solution based on REST and the use of Linked Data that can be used to implement a completely distributed and privacy-preserving recommender system.

Information retrieval is one of the most popular information access methods for overcoming the information overload problem of the Web. However, its interaction model is still utilizing the old text-based ranked lists and static interaction algorithm. In this paper, we introduce our adaptive visualization approach for searching the Web, which we call Adaptive VIBE. It is an extended version of a reference point-based spatial visualization algorithm, and is designed to serve as a user interaction module for a personalized search system. Personalized search can incorporate dynamic user interests and different contexts, improving search results. When it is combined with adaptive visualization, it can encourage users to become involved in the search process more actively by exploring the information space and learning new facts for effective searching. In this paper, we introduce the rationale and functions of our adaptive visualization approach and discuss the approaches' potential to create a better search environment for the Web.

Personalized Web search has emerged as one of the hottest topics for both the Web industry and academic researchers. However, the majority of studies on personalized search focused on a rather simple type of search, which leaves an important research topic - the personalization in exploratory searches - as an under-studied area. In this paper, we present a study of personalization in task-based information exploration using a system called TaskSieve. TaskSieve is a Web search system that utilizes a relevance feedback based profile, called a "task model", for personalization. Its innovations include flexible and user controlled integration of queries and task models, task-infused text snippet generation, and on-screen visualization of task models. Through an empirical study using human subjects conducting task-based exploration searches, we demonstrate that TaskSieve pushes significantly more relevant documents to the top of search result lists as compared to a traditional search system. TaskSieve helps users select significantly more accurate information for their tasks, allows the users to do so with higher productivity, and is viewed more favorably by subjects under several usability related characteristics.

Adaptive information filtering is a promising tool for both casual Web news readers and professional intelligence analysts. Adaptive filtering augments the traditional query- or profile-based rankings provided by search engines. An interesting research challenge in this context is to offer users more control over the rankings by letting them mediate between the two extremes - query- and profile-based rankings. To address this challenge, we developed an adaptive relevance-based visual exploration tool based on the VIBE (visual information browsing environment) visualization approach, which was previously developed at our School. This paper presents the rationale and functionality of this visual exploration tool and reports the results of its preliminary evaluation.

In profile-based or content-based adaptive systems, one of the open research questions is how frequently the user's profile and the list of recommended items should be updated. Different systems tend to choose one of the two extremes. Some systems do it once per session (thus called between-session update strategy), whereas some others update whenever there is feedback (called instant update strategy). This paper presents our attempt to assess the value of keeping the list of recommended items up-to-date in the context of task-based information exploration. We conducted controlled studies involving human users performing realistic tasks using two systems that have the same adaptive filtering engine but with the above two different update strategies. Our results show that the between-session strategy helped to find better quality information, and received better subjects' responses about its usefulness and usability. However, it prolonged the selection of useful passages, whereas the instant update strategy helped subjects to obtain almost all of their selected passages (>98%) within the first 5 minutes. Based on the results, we hypothesize that the best strategy for updating might be a hybrid between the two update strategies, where both adaptability and stability can be achieved.

Over the last five years, a range of projects have focused on progressively more elaborated techniques for adaptive news delivery. However, the adaptation process in these systems has become more complicated and thus less transparent to the users. In this paper, we concentrate on the application of open user models in adding transparency and controllability to adaptive news systems. We present a personalized news system, YourNews, which allows users to view and edit their interest profiles, and report a user study on the system. Our results confirm that users prefer transparency and control in their systems, and generate more trust to such systems. However, similar to previous studies, our study demonstrate that this ability to edit user profiles may also harm the system.s performance and has to be used with caution.

A number of research teams are working to organize personalized access to the modern repositories of educational resources. The goal of personalized access is to help students locate resources that match their individual goals, interests, and current knowledge. The project presented in this paper is focused on the least explored way of personalized access - adaptive visualization. Here, we present the NavEx ADVISE visualization system, which provides personalized access to a repository of educational examples. The system combines spatial, similarity-based visualization with adaptive annotations of resources. The spatial layout and the adaptive annotations are generated using a knowledge-based indexing of examples with domain concepts.