Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.

Regulated compliance has become a business liabil-ity given the constant changes to the IT environments driven by the new Cloud normal enabling daily upgrades, and by the new programs related to cybersecurity. To keep pace with the changes, enterprises and auditor agencies shifted from annual audit to continuous compliance. Critical enablers of continuous compliance are standardization and automation. To achieve this, we need to treat everything as code, from the compliance artifacts to the policies and their results. Currently, Compliance as Code (CaC) is utilized to represent compliance artifacts and processes, while Policy as Code (PaC) is employed to express the logic of validating the actual state of the systems against the desired state. Although there is emerging technology for codifying compliance and policy, there are challenges that hinder the compliance digitization transformation towards continuous compliance. The first challenge is how to migrate from document-based operations to the program-based operations. The second challenge arises from the disconnect between CaC and the varied native interfaces of existing PaC solutions. Lastly, there is a concern regarding the reliability of results from end-to-end automation when replacing human-in-the-loop overseeing and guaranteeing the integrity with fully automated process. To address these challenges, we have developed a GitOps-based secure pipeline framework, seamlessly integrating compliance authoring with CaC while ensuring the secure integration of Compliance as Code and Policy as Code, protecting data integrity and preserving data traceability. In this paper, we present the details of the framework which enables end-to-end automation of compliance processes while supporting integration with various Policy Validation Points (PVPs), and showcase the evaluation of our implemented solution's reliability by testing risks such as unintentional modification of policies.

In this study, we analyze Python program codes and questionnaire results collected in the programming classes for 23 senior high school students on December 19, 2020, and for 48 junior high school students on August 11 and 12, 2021. Each student entered and executed the 168 Python programs described in the text on Google Colaboratory using each personal computer. Students’ programs were collected by sharing in Google Drive. In the analysis, the similarity matrix is obtained from the correct / incorrect relationship of the students’ programs and hierarchical clustering is performed by Ward's method and visualized by the dendrogram. As a result, the cluster is confirmed from the correct/incorrect relationship. For the questionnaire, we use conjoint analysis for the relationship with the number of correct answers, and Quantification Method of the Third Type for the relationship with the questionnaire category. As a result, we confirm that the tendency for the senior high school students is the same as or different from that for the junior high school students.

Integrity of the cloud is the most important requirement for mission-critical enterprise workloads. NIST SP 800–53 states that information systems must prevent the installation of any components that have not been verified digitally with a signed certificate that is recognized and approved by the organization's information system. On a Kubernetes cluster, the admission controller can control requests for application installation, and it would be a powerful protection tool if it could control requests for Kubernetes resources based on signature verification. However, there are various technical challenges when it comes to verifying the signature for a Kubernetes resource at the admission controller because a signed resource is rewritten automatically by internal cluster work and many requests that include internal mutation without a signature are generated. In this work, we propose an approach to protect the integrity of a Kubernetes resource with signature verification at the admission controller. Our approach addresses the issue that the differences between the signed resource in the admission request and the signature message occur automatically in Kubernetes and conducts signature verification properly by using Dry Run. We also propose a profile framework to address the internal mutation request that cannot be attached to the signature. Our experimental results demonstrate that standard applications can be protected by our approach.

Enterprises adopting cloud increasingly use container orchestration systems (e.g., Kubernetes) to manage applications and their configurations at scale. In Kubernetes environment, developers use package managers (e.g., Helm) for bundling, distributing, and deploying applications. These developments in cloud native applications have introduced new challenges. One of the challenges is protecting the integrity of application packages (e.g., Helm chart) deployed in a large-scale enterprise cluster. Existing tools for verifying integrity of Helm charts are limited to verify provenance and integrity of application packages. Therefore, in this work, we propose a mechanism to verify provenance and integrity of Helm charts at the cluster-side by addressing the granularity gap to verify each resource in a chart. We demonstrate how our approach successfully enforces integrity of Helm charts and evaluate the cost of integrity enforcement with a preliminary study.

Container integrity monitoring is defined as key requirements for regulatory compliance, such as PCI-DSS, in which any unexpected changes such as file updates or program runs must be logged for later audit. Syscall monitoring provides comprehensive monitoring of such change events on container, while it suffered from large amount of false alarms unless well-defined allowlist rules are coordinated before deploying container. Defining such comprehensive allowlist is not feasible especially when managing various kinds of application workloads in large-scale enterprise cluster. We propose new approach for identifying real anomaly of syscall events effectively without relying on any predefined allowlist configuration in this paper. Our novel filtering algorithm based on the knowledge acquired autonomously from Kubernetes cluster control plane reduces 99.999 % noise effectively and distilling only abnormal events in real time. Our experiment with real applications on more than 4000 containers demonstrates its effectiveness even on large-scale cluster.

Registering altered domain names with the purpose of confusing users and conducting malicious activities is one of the most widespread types of attacks on the Web, conforming a family of techniques known as domain squatting. Detecting these domains is a difficult task, given the large a mount of combinations and the massive and heterogeneous nature of the Web. In this work, we propose a set of models to firstly learn the distributional regularities from detected squatted domains, and from that, automatically generate realistic modified domains. Our goal is to proactively guide the generation of squatted domains towards malicious domains that exists but have not been detected yet. We conducted an empirical study for both typo-squatting and combo-squatting generation approaches against strong baselines on real world data, showing their feasibility and providing insights to support for proactive defense in the context of cloud security.

In this study, we apply recurrent neural networks and Long Short-Term Memory (LSTM) to 3-axis accelerations of walking acquired by a smartphone for gait identification and authentication. First, the accelerations during walking for 21 subjects are recorded in two holding situations, namely, while placing the smartphone in the pocket or looking at the screen of the smartphone. We then identify and authenticate the user by inputting the accelerations directly to the recurrent neural networks and LSTM. We have confirmed that the performance of this method is better than the conventional method of extracting features from the accelerations and using random forests.

In this study, we record 3-axis accelerations during various measurement tests with a smartphone, extract features from the accelerations, and then identify young people and the elderly people, as well as healthy elderly people and care-requiring preliminary group for the features by random forests of machine learning. We also perform the selection of the features specific to the elderly people. For the measurement tests, each subject conducts normal and fast 10m walking, chair standing and sitting test, alternate step test, and balance test with 3 kinds of standing position holding. The subjects are 23 young people and 25 elderly people who can walk independently. As a result, the accuracies in all the measurement tests are 78% or more using selected features, and 91% or more in the alternate step test in particular. Moreover, the amplitude and the dispersion of the acceleration, the frequency component, and the period time are selected as the characteristic peculiar to the elderly people.

In our previous study, we collected a touch operations history when 40 subjects performed basic operation, text browsing, and web browsing using our Android application. From the touch history, we extracted 8 or 16 features for 6 gestures of swipe and pinch, and then identified subjects using some machine learning algorithms. The results showed that user identification rate reached about 95% for basic operation and text browsing. However, we used only one day touch history for each subject, so that a long-term influence when each subject performs the touch operations many times for a long period has been unclear. In this study, we record 10 touch operations histories of 11 subjects for a half year using the Android application to examine the long-term changes of user identification rate. The results show that the correctly classified rates for pinch gestures and swipe from down to up during simple text browsing are almost constant for a long term while the accuracy for swipe gesture in web browsing drops by about 10% as the number of experiments increases.

Toward an immunity-based gait recognition to integrate the identification results from multiple smart phone sensors, in our previous study, we collected gait data using an iOS application when some subjects walked along a corridor carrying an iPhone in the following 3 holding states: 1) in the pocket, 2) holding the phone to the ear, and 3) looking on the screen. We extracted 43 features from the user-generated acceleration data of all the recorded gait data, and then identified subject using some machine learning algorithms. However, we have not yet carefully examined the necessity of the 43 features. We also have coped with only walking on a flat land. In this study, we first develop a similar gait record application on Android devices. Using the application, we collect gait data for 15 subjects in 5 walking states adding to going upstairs and downstairs. From the 3-axes accelerometer data, we extract 52 features adding 9 features of maximum, minimum, and energy for each axis to the 43 features, and then select the subset of the 52 features with small number and high accuracy. The result shows that the accuracies of going upstairs and downstairs are improved by the feature selection.

In our previous study on gait recognition, we developed the application on iOS for smart phone to record not only the acceleration but also more data such as the rotation of the phone around the 3 axes. And then we performed preliminary experiments using the application to collect the user-generated acceleration data on walking when the phone was held calling and touching as well as in the pocket. However, the acceleration data were recorded for only 4 subjects on one day. In this study, we carry out additional experiments increasing the number of subjects and adding the data recorded on other day. The results show that although the correctly classified rate when the phone is in the pocket is not yet affected by the number of subjects, the performance in the cases of calling and touching is influenced by the small number of subjects. Interestingly, we observe that the additional data on other day have a different effect on identification performance according to phone holding states. We also examine the influence of various walking states for one subject. Furthermore, we discuss the application of an immunity-based diagnosis model to gait recognition to integrate the identification results from multiple smart phone sensors.

In wireless sensor networks, sensor nodes with dead battery make holes in sensing coverage. It is necessary to fill the holes by replacing the nodes of dead battery with fully-charged ones. This is called sensor replacement. In this paper, following the existing study, to clarify the distribution of nodes information to robots, we propose two centralized manager algorithms where repairing robots have or not have the information list of nodes with dead battery. We carry out some simulations under the constant density of nodes per robots and then clarify the tradeoff between the messaging overhead and the motion overhead. We also investigate the case of the limited energy of the robots. Finally, we discuss the analogy between immune system and sensor replacement using mobile robots.

Smart phone stores a lot of important private information, so that user authentication is increasingly necessary to prevent attacks by illegal users who are not the owner of the smart phone. Password authentication or biometrics can be generally applied only on login. After the authentication is passed, not only the legal owner but also illegal users freely use the smart phone. Therefore we are trying to develop a behavior-based user authentication system to continuously check the user activities after login. The developing system can extract many operational and behavioral features characteristic of user by multiple sensors; for example, touch screen, accelerometer, microphone, and GPS sensor. And then it can combine the authentication results from the multiple sensors because a single sensor may produce poor authentication accuracy. In this paper, we report the ongoing results of our system, that is, the experimental results from user authentication using touch operational features, and some features extracted from accelerometer. We also discuss the introduction of immunity-based model to our system to integrate the authentication results from the multiple sensors.

In wireless sensor network, statistical en-route filtering (SEF) schemes can detect and eliminate false data injection attacks. However, SEF does not deal with the detection of the compromised nodes which are injecting false reports. In our previous studies, we have proposed an immunity-based SEF (ImSEF) to identify compromised nodes and achieve earlier detection of false reports. In ImSEF, each node assigns credibility to its neighboring nodes and updates the credibility based on the success or failure of filtering and communication. And then each node can use the updated credibility as the probability of the next communication. Preliminary simulation results against only one compromised node showed that ImSEF outperformed the original SEF. However, we have not yet analyzed the performance of ImSEF in detail. In this paper, we examine various attack scenarios and parameters settings for ImSEF. By the parameters analyses, some phase transitions can be interestingly observed. We also analyze and reconfirm the performance of ImSEF mathematically.

Recently, much attention has been paid to research on distributed coalitions, as a possible mechanism to embody distributed information flow control which can apply security policies to distributed components over nodes by making the components enforce mandatory access controls for resources based on the policies. Some projects have proposed prototype systems of distributed coalitions, but they assume that each component that participates in a domain has domain management functions. This assumption is reasonable when the components are designed for a distributed coalition, but it has been an obstacle when actually building distributed coalitions in existing environments, because it is difficult for existing components in real environments that were not designed for use in distributed coalitions to update their code and add support for domain management functions while considering the influences of their environments, especially for commercial components. In this paper, we propose a Domain Management Agent (DMA) for building practical distributed coalitions, which performs domain management on behalf of a component and emphasizes minimizing the influence on existing environments. We implement a prototype system on Microsoft Windows platform for broad use by many people, evaluate its performance overhead, and show that our approach is feasible.

Information coordination among business planning, MES (manufacturing execution system), and equipment controllers is significant for manufacturers. Because production information systems are often constructed in a multi-vendor environment, a large amount of man-hours has been required to develop connection functions corresponding to various combinations of vendors. In order to solve this problem, we have constructed a standard model by a use case analysis of manufacturing operations management in the Japanese standardization committee MESX (MES on XML) joint project. By this model, we have defined XML communications protocols for production information systems. This paper focuses on the protocols between the MES layer and the equipment control layer, and reports on features and effects.

Recalls and defective parts are a major problem for manufacturers and a major challenge in information management. Increasingly complex supply chains have created a situation where parts' data is distributed across a volatile network of suppliers' databases. The challenge of parts tracking in this environment requires maintaining data views across constantly-restructuring networks of information systems as new suppliers are contracted and subassemblies are outsourced. In this paper we present a method for manufacturers to efficiently and automatically index all component parts in their products, allowing for complex changes to the supply chain, and enabling highly efficient reverse-lookups for recalls. Our method improves parts lookup and search time by subdividing all component part indexes into a distributed, versioned matrix of tables which models the structure of the supply chain. We introduce a method of reverse-indexing component parts using manufacturing time to pinpoint products for selective recalls, running in a fraction of the time of a traditional database search. We evaluate our method for applicability to the automotive industry using statistically re-generated data from a major automobile manufacturer to simulate a supply chain 7 layers deep, comprising 1000 individual component parts, indexing a distributed dataset totaling approximately 1TB in size.

A distributed coalition supports distributed mandatory access controls for resources whose security policies differ for each group of components over nodes, and provides secure information operations and exchanges with nodes that handle information over which conflicts of interest may occur. Many projects have proposed distributed coalitions using a virtual machine monitor, but this approach for strong confinement tends to hinder successful deployments in real world scenarios that involve complicated operations and management for applications because such access control is coarse-grained for the resources.

Article Free Access Share on Reducing the round complexity of a sealed-bid auction protocol with an off-line TTP Authors: Yuji Watanabe Institute of Industrial Science, University of Tokyo, 7-22-1 Roppongi, Minatoku, Tokyo, 106-8558, Japan Institute of Industrial Science, University of Tokyo, 7-22-1 Roppongi, Minatoku, Tokyo, 106-8558, JapanView Profile , Hideki Imai Institute of Industrial Science, University of Tokyo, 7-22-1 Roppongi, Minatoku, Tokyo, 106-8558, Japan Institute of Industrial Science, University of Tokyo, 7-22-1 Roppongi, Minatoku, Tokyo, 106-8558, JapanView Profile Authors Info & Claims CCS '00: Proceedings of the 7th ACM conference on Computer and Communications SecurityNovember 2000 Pages 80–86https://doi.org/10.1145/352600.352612Online:01 November 2000Publication History 21citation458DownloadsMetricsTotal Citations21Total Downloads458Last 12 Months3Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

In recent years much attention has been focused on behavior-based artificial intelligence (AI), which has already demonstrated its robustness and flexibility against dynamically changing world. However, in this approach, the followings have not yet been resolved: how do we construct an appropriate arbitration mechanism, and how do we prepare appropriate competence modules. In this paper, to overcome these problems, we propose a new decentralised consensus-making system inspired by the biological immune system. And we apply our proposed method to behavior arbitration for an autonomous mobile robot, namely garbage collecting problem that takes into account the concept of self-sufficiency. To verify the feasibility of our method, we carry out some simulations. In addition, we investigate two types of adaptation mechanisms, and try to evolve the proposed artificial immune network using reinforcement signals.

Conventional AI technology has been criticized on many grounds such as brittleness under dynamically changing environments. To overcome this problem, other approaches called new AI, emergent computation, animat approach, behavior-based AI and so on were proposed and confirmed their usefulness. On the other hand, biological information processing systems have many interesting functions and are expected to provide various feasible ideas to engineering fields, especially robotics. Immune system plays important roles to maintain its own system against dynamically changing environments. Therefore, we believe that immune system would provide a new paradigm suitable for dynamic problem dealing with unknown environments rather than static problem. In this paper, we propose a new approach to behavior-based AI by paying attention to biological immune system. We confirm the feasibility of our proposed method by applying to a behavior control of an autonomous mobile robot in cluttered environment as a practical example.