ABSTRACT Deploying end‐to‐end ML applications on edge resources becomes a viable solution to achieve performance and data regulations. With the microservice architecture, these applications can scale dynamically, improving service availability under dynamic workloads. However, orchestrating multiple end‐to‐end ML applications within heterogeneous edge environments must deal with numerous challenges while sharing computing resources. Prevalent orchestration tools/frameworks supporting edge ML serving are inefficient in provisioning methods due to constrained resources, diverse resource demands and utilization patterns. In this work, we present a provisioning method to optimize resource utilization for end‐to‐end ML applications on a heterogeneous edge. By profiling all microservices within the application, we estimate scales and allocate them on desired hardware platforms with sufficient resources when considering their runtime utilization patterns. We also provide several practical analyses on runtime monitoring metrics to detect and mitigate resource contentions, guaranteeing performance. The experiments with three real‐world ML applications demonstrate the practicality of our method on a heterogeneous edge cluster of Raspberry Pis and Jetson Developer Kits.

A long running data-intensive computational application acquires costly computing resources. With the emerging new architectures, like computing systems with multiple nodes of many-core CPUs and accelerators, while domain-specific tools and libraries employed in such an application leverage high parallelism on accelerators for intensive computations, the remaining resources can potentially be utilized for other application-related data operations. Such data operations, called opportunistic data operations in this work, must usually be carried out for post-processing or follow-up analytics based on results produced during the runtime of the application. These operations are not easily backfilled or preempted under the guidance of the domain scientist or by common task scheduling systems due to their complex dependencies.In this paper, we introduce a framework for domain scientists to identify and execute opportunistic data operation tasks. With a minimal specification or modification of the main application, the scientists can specify, monitor, and execute opportunistic tasks independently from the main application and the framework will detect underutilized resources to execute these tasks, thereby, optimizing utilization efficiency within the allocated resources. We present experiments to demonstrate the applicability of our framework on a magnetic field modeling running on the LUMI computing system.

The growing complexity of end-to-end Machine Learning (ML) serving across the edge-cloud continuum has raised the necessity for runtime explainability to support service optimizations, transparency, and trustworthiness. That involves many challenges in managing ML service quality and engineering runtime explainability based on ML service contracts. Currently, consumers use ML services almost as a black box with insufficient explainability for not only inference decisions but also other contractual aspects, such as data/service quality and costs. The generic explainability for ML models is inadequate to explain the runtime ML usage for individual consumers. Moreover, ML-specific metrics have not been addressed in existing service contracts. In this work, we introduce a novel contract-based runtime explainability framework for end-to-end ensemble ML serving. The framework provides a comprehensive engineering toolset, including explainability constraints in ML contracts, report schemas, and interactions between ML consumers and the components of the ML serving for evaluating service quality with contract-based explanations. We develop new monitoring probes to measure ML-specific metrics on data quality, inference confidence, inference accuracy, and capture runtime ML usage. Finally, we present essential quality analyses via an observation agent. That interprets ML inferences and evaluates contributions of ML inference microservices, assisting ML serving optimization. The agent also integrates ML algorithms for detecting relations among metrics, supporting constraint developments. We demonstrate our work with two real-world applications for malware and object detection.

The Digital Twin (DT) paradigm has been largely adopted for many smart systems in various domains. Due to the heterogeneous and distributed nature of the physical twins, these systems increasingly incorporate disparate security tools, especially those based on service-based AI/ML capabilities. That presents numerous challenges in achieving a comprehensive understanding of security analytics and explainability in security operations carried out by ML-based security services, which require continuous monitoring and optimization to remain effective. This paper aims to support security service integration and automated analyses with enhanced explainability in DTs. We introduce a novel framework that unifies runtime contexts to facilitate security services unification and operation interpretation in security orchestration. We define a workflow and provide necessary services for generating security reports across physical and logical layers. Leveraging a centralized knowledge service, we let security analysts incorporate domain knowledge in automating incident reasoning and security enforcement at the logical layer. We demonstrate our explainability framework on a DT of an Industry 4.0 toy factory with two ML-based security services detecting network anomalies. Our experiments show a significant reduction in manual effort for orchestrating security incident analysis and mitigation.

Important service-level constraints in machine learning (ML) services must be communicated and agreed among relevant stakeholders. Due to the lack of studies and support, it is unclear which and how ML-specific attributes and constraints should be specified and assured in service contracts for ML services. This paper examines service contracts in the three stakeholders engagement model of ML services. We identify key ML-specific attributes that should be specified and monitored for the ML service provider, ML consumer and ML infrastructure provider. Based on that, we propose QoA4ML (Quality of Analytics for ML) as a framework to support ML-specific service contracts. QoA4ML includes an ML-specific service contract specification, monitoring utilities and a contract observability service. To illustrate the usefulness of QoA4ML, we present real-world examples for contract terms and policies, monitoring and contract evaluation with dynamic ML services in predictive maintenance.

With the rapid growth of Internet technology and infrastructure, we have entered the era of data explosion. Following this is the emergence of social networks, which have brought an enormous and ever-growing amount of online content into our digital world. Knowing precisely the popularity of online contents is of great importance for developing advanced caching algorithms as well as content distribution strategies. In this study, we provide some crucial insights into the characteristics of online content popularity over time in different locations and propose a simple predictive model to estimate the popularity of online contents in particular periods. By experiencing with the real datasets of MovieLens and Youtube, our model not only achieves considerable accuracy but also shows an impressive reduction in computation time, from 80 to 250 times faster comparing to some baseline methods. At last, we also provide the potentials and limitations of our model in practice.