While artificial intelligence (AI) models have improved at understanding large-scale data, understanding AI models themselves at any scale is difficult. For example, even two models that implement the same network architecture may differ in frameworks, datasets, or even domains. Furthermore, attempting to use either model often requires much manual effort to understand it. As software engineering and AI development share many of the same languages and tools, techniques in mining software repositories should enable more scalable insights into AI models and AI development. However, much of the relevant metadata around models are not easily extractable. This paper (an extension of our MSR 2020 paper) presents a library called AIMMX for AI Model Metadata eXtraction from software repositories into enhanced metadata that conforms to a flexible metadata schema. We evaluated AIMMX against 7,998 open-source models from three sources: model zoos, arXiv AI papers, and state-of-the-art AI papers. We also explored how AIMMX can enable studies and tools to advance engineering support for AI development. As preliminary examples, we present an exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models. We also demonstrate the flexibility of extracted metadata by using the evaluation dataset in an existing natural language processing (NLP) analysis platform to identify trends in the dataset. Overall, we hope AIMMX fosters research towards better AI development.

Despite all of the power that machine learning and artificial intelligence (AI) models bring to applications, much of AI development is currently a fairly ad hoc process. Software engineering and AI development share many of the same languages and tools, but AI development as an engineering practice is still in early stages. Mining software repositories of AI models enables insight into the current state of AI development. However, much of the relevant metadata around models are not easily extractable directly from repositories and require deduction or domain knowledge. This paper presents a library called AIMMX that enables simplified AI Model Metadata eXtraction from software repositories. The extractors have five modules for extracting AI model-specific metadata: model name, associated datasets, references, AI frameworks used, and model domain. We evaluated AIMMX against 7,998 open-source models from three sources: model zoos, arXiv AI papers, and state-of-the-art AI papers. Our platform extracted metadata with 87% precision and 83% recall. As preliminary examples of how AI model metadata extraction enables studies and tools to advance engineering support for AI development, this paper presents an exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models. Our analysis suggests that while data reproducibility may be relatively poor with 42% of models in our sample citing their datasets, method reproducibility is more common at 72% of models in our sample, particularly state-of-the-art models. Our collected models are searchable in a catalog that uses existing metadata to enable advanced discovery features for efficiently finding models.

Today's cloud service architectures follow a “one size fits all” deployment strategy where the same service version instantiation is provided to the end users. However, consumers are broad and different applications have different accuracy and responsiveness requirements, which as we demonstrate renders the “one size fits all” approach inefficient in practice. We use a production grade speech recognition engine, which serves several thousands of users, and an open source computer vision based system, to explain our point. To overcome the limitations of the “one size fits all” approach, we recommend Tolerance Tiers where each MLaaS tier exposes an accuracy/responsiveness characteristic, and consumers can programmatically select a tier. We evaluate our proposal on the CPU-based automatic speech recognition (ASR) engine and cutting-edge neural networks for image classification deployed on both CPUs and GPUs. The results show that our proposed approach provides a MLaaS cloud service architecture that can be tuned by the end API user or consumer to outperform the conventional “one size fits all” approach.

The dramatic proliferation of virtual machines (VMs) in datacenters and the highly-dynamic and transient nature of VM provisioning has revolutionized datacenter operations. However, the management of these environments is still carried out using re-purposed versions of traditional agents, originally developed for managing physical systems, or most recently via newer virtualization-aware alternatives that require guest cooperation and accessibility. We show that these existing approaches are a poor match for monitoring and managing (virtual) systems in the cloud due to their dependence on guest cooperation and operational health, and their growing lifecycle management overheads in the cloud.

Virtual-machine images are currently distributed as disk-image files, which are files that mirror the content of disks. This format is convenient for the virtual machine monitors that execute these images. However, it is not well-suited for administering images because storing images as disk-image files forces administrators to maintain the software on images with the same tools that they use to maintain the software on machines. Already, these tools cannot cope with physical server sprawl; in the future, because images can be snapshotted and cloned easily, enterprises that migrate from machines to images will need tools that scale to cope with the larger problem of virtual-machine image sprawl.To address this problem, this paper proposes the Mirage image format (MIF), a new storage format that exposes the rich semantic information currently buried in disk-image files. Disk-image files contain a mapping from file name to file content (and file metadata). MIF decouples this mapping into a manifest that maps file names to content descriptors (and file metadata) and a store that holds the content. Each image has its own manifest and a store may contain content for many images. As with disk-image files, images in MIF fully encapsulate application state including all software dependences. In addition, conversion between MIF and traditional disk-image formats is easy.This paper shows, through examples, that MIF makes some typical software management tasks--inventory control, customized deployment, and image update--faster and easier. The general technique is to operate on manifests instead of on content whenever possible. These tasks can be performed without starting images and, because manifests are simpler and orders of magnitude smaller than disk-image files, without accessing large amounts of data.

The Progressive Deployment System (PDS) is a virtual execution environment and infrastructure designed specifically for deploying software, or on demand while enabling management from a central location. PDS intercepts a select subset of system calls on the target machine to provide a partial virtualization at the operating system level. This enables an asset's install-time environment to be reproduced virtually while otherwise not isolating the asset from peer applications on the target machine. Asset components, or shards, are fetched as they are needed (or they may be pre-fetched), enabling the asset to be progressively deployed by overlapping deployment with execution. Cryptographic digests are used to eliminate redundant shards within and among assets, which enables more efficient deployment. A framework is provided for intercepting interfaces above the operating system (e.g., Java class loading), enabling optimizations requiring semantic awareness not present at the OS level. The paper presents the design of PDS, motivates its porous isolation model with respect to the challenges of software deployment, and presents measurements of PDS's execution characteristics.

Commodityworkstations connected by commodity networks are increasingly viewed as an economically viable alternative to tightly coupled multiprocessors.In recent years, many scientific computing applications have been able to make effective use of various types of workstation clusters.The main difference between workstation clusters and the more traditional, tightly coupled distributedmemory systems is communication performance.In this paper we present a host-network interface architecture that supports efficient remote memory writes across standard ATM networks, bringing performance closer to that of special-purpose, tightly coupled systems for a large class of applications.We show that minimal hardware support is required on the adaptor, and that the required features are very similar to those already needed on adaptors for high-speed networks.We also describe an implementation of this architecture, and present measurements of communication performance indicating its effect on the breadth of applications that can use a general purpose network m an effective way.

Among the many benefits of ATM networking are the potential for connections with negotiated quality-of-service (QoS) guarantees and application-specific data management at network endpoints. We describe the architecture of a PCI bus host adapter for OC-3 and OC-12 ATM, focusing on challenges in the areas of buffer management and flow control, since these are vital to realizing the bandwidth and QoS potential of ATM endpoint hosts.

Achieving high-speed network I/O on distributed-memory systems is difficult because their architecture is in general ill-suited for communication processing. Some of the common problems are: inability to do protocol processing, inefficient handling of data distribution, and poor management of the I/O. The authors present an I/O architecture that addresses these problems and supports high-speed network I/O on distributed-memory systems. The key to good performance is to partition the work appropriately between the system and the network interface. The authors perform some communication tasks on the distributed-memory parallel system since it is more powerful, and less likely to become a bottleneck than the network interface. Tasks that do not parallelize well are performed on the network interface and hardware support is provided for the most time-critical operations. They emphasize the use of simple I/O mechanisms that can be used by programming tools that map applications on the distributed-memory system to implement efficient I/O for the class of applications they support. This architecture has been implemented for the iWarp distributed-memory system. The authors describe this implementation and present performance results.>