Large language models (LLMs), the foundation of generative AI systems like ChatGPT, are transforming many fields and applications, including multimedia, enabling more advanced content generation, analysis, and interaction. However, cloud-based LLM deployments face three key limitations: high computational and energy demands, privacy and reliability risks from remote processing, and recurring API costs. Recent advances in agentic AI, especially in structured reasoning and tool use, offer a better way to exploit open and locally deployed tools and LLMs. This paper presents ELLMPEG, an edge-enabled agentic LLM framework for the automated generation of video-processing commands. ELLMPEG integrates tool-aware Retrieval-Augmented Generation (RAG) with iterative self-reflection to produce and locally verify executable FFmpeg and VVenC commands directly at the edge, eliminating reliance on external cloud APIs. To evaluate ELLMPEG, we collect a dedicated prompt dataset comprising 480 diverse queries covering different categories of FFmpeg and the Versatile Video Codec (VVC) encoder (VVenC) commands. We validate command generation accuracy and evaluate four open-source LLMs based on command validity, tokens generated per second, inference time, and energy efficiency. We also execute the generated commands to assess their runtime correctness and practical applicability. Experimental results show that Qwen2.5, when augmented with the ELLMPEG framework, achieves an average command-generation accuracy of 78 % with zero recurring API cost, outperforming all other open-source models across both the FFmpeg and VVenC datasets.

Serverless cloud computing is increasingly adopted for workflow management, optimizing resource utilization for providers while lowering costs for customers. Integrating edge computing into this paradigm enhances scalability and efficiency, enabling seamless workflow distribution across geographically dispersed resources on the computing continuum. However, existing serverless workflow orchestration methods on the computing continuum often prioritize time and/or cost objectives, neglecting energy consumption and carbon footprint. This paper introduces EnergyLess, a multi-objective concurrent serverless workflow batch orchestration service for the computing continuum. EnergyLess decomposes workflow functions within a batch into finer-grained sub-functions and schedules either the original or sub-function versions to suitable regions and instances on the continuum, improving energy consumption, carbon footprint, economic cost, and completion time while considering individual workflow requirements and resource constraints. We formulate the problem as a mixed-integer nonlinear programming (MINLP) model and propose three lightweight heuristic algorithms to enable scalable function scheduling and execution. Evaluations on a large-scale computing continuum testbed, spanning AWS Lambda, Google Cloud Functions (GCF), and 325 fog and edge instances across six regions, demonstrate that EnergyLess improves cost efficiency by 75%, completion time by 6%, energy consumption by 15%, and CO2emissions by 20% for a batch size of 300, compared to three baseline methods.

Data within a specific context gains deeper significance beyond its isolated interpretation. In distributed systems, interdependent data sources reveal hidden relationships and latent structures, representing valuable information for many applications. This paper introduces Osmotic Learning (OSM-L), a self-supervised distributed learning paradigm designed to uncover higher-level latent knowledge from distributed data. The core of OSM-L is osmosis, a process that synthesizes dense and compact representation by extracting contextual information, eliminating the need for raw data exchange between distributed entities. OSM-L iteratively aligns local data representations, enabling information diffusion and convergence into a dynamic equilibrium that captures contextual patterns. During training, it also identifies correlated data groups, functioning as a decentralized clustering mechanism. Experimental results confirm OSM-L’s convergence and representation capabilities on structured datasets, achieving over 0.99 accuracy in local information alignment while preserving contextual integrity.

The environmental impact of video streaming is gaining more attention due to its growing share in global internet traffic and energy consumption. To support accurate and transparent sustainability assessments, we present SEED (Streaming Energy and Emission Dataset): an open dataset for estimating energy usage and CO2emissions in adaptive video streaming. SEED comprises 500 video segments. It provides segment-level measurements of energy consumption and emissions for two primary stages: provisioning, which encompasses encoding and storage on cloud infrastructure; and end-user consumption, including network interface retrieval, video decoding, and display on end-user devices. The dataset covers multiple codecs (AVC, HEVC), resolutions, bitrates, cloud instance types, and geographic regions, reflecting real-world variations in computing efficiency and regional carbon intensity. By combining empirical benchmarks with component-level energy models, SEED enables detailed analysis and supports the development of energy- and emission-aware adaptive bitrate (ABR) algorithms. The dataset is publicly available at: https://github.com/cd-athena/SEED.

This paper presents a review of state-of-the-art solutions concerning the optimization of computing in the field of parallel and distributed systems. Firstly, we contribute by identifying resources and quality metrics in this context including servers, network interconnects, storage systems, computational devices as well as execution time/performance, energy, security, and error vulnerability, respectively. We subsequently identify commonly used problem formulations and algorithms for integer linear programming, greedy algorithms, dynamic programming, genetic algorithms, particle swarm optimization, ant colony optimization, game theory, and reinforcement learning. Afterward, we characterize frequently considered optimization problems by stating these terms in domains such as data centers, cloud, fog, blockchain, high performance, and volunteer computing. Based on the extensive analysis, we identify how particular resources and corresponding quality metrics are considered in these domains and which problem formulations are used for which system types, either parallel or distributed environments. This allows us to formulate open research problems and challenges in this field and analyze research interest in problem formulations/domains in recent years.

Extending cloud computing towards fog and edge computing yields a heterogeneous computing environment known as computing continuum. In recent years, increasing demands for scalable, cost-effective, and streamlined maintenance services have led application and service providers to prefer serverless models over monolithic and serverful processing. However, orchestrating the computing continuum in complex application workflows of serverless functions, each with distinct requirements, introduces new resource management and scheduling challenges. This paper introduces an orchestration service for concurrent serverless workflow processing across the computing continuum called HEFTLess. HEFTLess uses two deployment modes tailored to serve each workflow function: predeployed and undeployed. We formulate the problem as a Binary Integer Linear Programming (BLP) optimization model, incorporating multiple groups of constraints to minimize the overall completion time and monetary cost of executing workflow batches. Inspired by the Heterogeneous Earliest Finish Time (HEFT) algorithm, we propose a lightweight serverless workflow scheduling heuristic to cope with the high optimization time complexity in polynomial time. We evaluate HEFTLess using two machine learning-based serverless workflows on a real computing continuum testbed, including AWS Lambda and 325 combined on-promise and cloud instances from Exoscale, distributed across five geographic locations. The experimental results confirm that HEFTLess outperforms state-of-the-art methods in terms of both workflow batch completion time and cost.

Video streaming has dominated Internet traffic, pushing network providers to ensure high-quality services to avoid customer churn. However, predicting streaming quality is challenging due to traffic encryption, requiring extensive network monitoring. While several prediction approaches have been studied, they often overlook resource and energy demands. To address this, we analyze existing methods, quantifying monitoring efficiency to predict video quality degradation. Finally, we highlight significant differences in efficiency, driven by data requirements and the prediction approach, offering insights for providers to select a suitable method for their needs.

The increasing size of graph structures in real-world applications, such as distributed computing networks, social media, or bioinformatics, requires appropriate sampling algorithms that simplify them while preserving key properties. Unfortunately, predicting the outcome of graph sampling algorithms is challenging due to their irregular complexity and randomized properties. Therefore, it is essential to identify appropriate graph features and apply suitable models capable of estimating their sampling outcomes. In this paper, we compare three machine learning (ML) models for predicting the divergence of five metrics produced by twelve node, edge, and traversal-based graph sampling algorithms: degree distribution (D3), clustering coefficient distribution (C2D2), hop-plots distribution (HPD2) (including the largest connected component (HPD2C)), and execution time. We use these prediction models to recommend suitable sampling algorithms for each metric and conduct mutual information analysis to extract relevant graph features. Experiments on six large real-world graphs demonstrate a prediction error under 20 % in C2D2and HPD2prediction for most algorithms despite their relatively high dissimilarity with the training data. Sampling algorithm recommendations on ten real-world graphs show higher hits@3 for D3 and C2D2and comparable results for HPD2and HPD2Ccompared to the K-best baseline method. Finally, ML models show superior runtime recommendations compared to baseline methods, with hits@3 over 86 % for synthetic and real graphs and hits@ 1 over 60 % for small graphs. These findings are promising for algorithm recommendation systems, particularly when balancing quality and runtime preferences.

Addressing climate change requires a global decrease in greenhouse gas (GHG) emissions. In today's digital landscape, video streaming significantly influences internet traffic, driven by the widespread use of mobile devices and the rising popularity of streaming platforms. This trend emphasizes the importance of evaluating energy consumption and the development of sustainable and eco-friendly video streaming solutions with a low Carbon Dioxide (CO2) footprint. We developed a specialized tool, released as an open-source library called GREEM, addressing this pressing concern. This tool measures video encoding and decoding energy consumption and facilitates measurement testbeds. It monitors the computational impact on hardware resources and offers various analysis cases. GREEM is helpful for developers, researchers, service providers, and policymakers interested in minimizing the energy consumption of video encoding and streaming.

Video streaming constitutes 65 % of global internet traffic, prompting an investigation into its energy consumption and CO2 emissions. Video encoding, a computationally intensive part of streaming, has moved to cloud computing for its scalability and flexibility. However, cloud data centers' energy consumption, especially video encoding, poses environmental challenges. This paper presents VEED, a FAIR Video Encoding Energy and CO2 Emissions Dataset for Amazon Web Services (AWS) EC2 instances. Additionally, the dataset also contains the duration, CPU utilization, and cost of the encoding. To prepare this dataset, we introduce a model and conduct a benchmark to estimate the energy and CO2 emissions of different Amazon EC2 instances during the encoding of 500 video segments with various complexities and resolutions using Advanced Video Coding (AVC) and High-Efficiency Video Coding (HEVC). VEED and its analysis can provide valuable insights for video researchers and engineers to model energy consumption, manage energy resources, and distribute workloads, contributing to the sustainability of cloud-based video encoding and making them cost-effective. VEED is available at https://github.com/cd-athena/VEED-dataset.

As multimedia dominates Internet traffic, users seek a better Quality of Experience (QoE), often resulting in increased energy consumption and a higher carbon footprint. The increasing focus on sustainability underscores the critical need to balance energy consumption and QoE in video streaming. This paper proposes a modular architecture that refines video encoding parameters by assessing video complexity and encoding settings for the prediction of energy consumption and video quality (based on Video Multimethod Assessment Fusion (VMAF)) using lightweight XGBoost models trained on the multi-dimensional video compression dataset (MVCD). We apply Explainable AI (XAI) techniques to identify the critical encoding parameters that influence the energy consumption and video quality prediction models and then tune them using a weighting strategy between energy consumption and video quality. The experimental results confirm that applying a suitable weighting factor to energy consumption in the x265 encoder results in a 46 % decrease in energy consumption, with a 4-point drop in VMAF, staying below the Just Noticeable Difference (JND) threshold.

Abstract This paper analyzes the timing performance of a persistent storage designed for distributed container-based architectures in industrial control applications. The timing performance analysis is conducted using an in-house simulator, which mirrors our testbed specifications. The storage ensures data availability and consistency even in presence of faults. The analysis considers four aspects: 1. placement strategy, 2. design options, 3. data size, and 4. evaluation under faulty conditions. Experimental results considering the timing constraints in industrial applications indicate that the storage solution can meet critical deadlines, particularly under specific failure patterns. Comparison results also reveal that, while the method may underperform current centralized solutions in fault-free conditions, it outperforms the centralized solutions in failure scenario. Moreover, the used evaluation method is applicable for assessing other container-based critical applications with timing constraints that require persistent storage.

Abstract Cloud service providers offer application providers with virtually infinite storage and computing resources, while providing cost-efficiency and various other quality of service (QoS) properties through a storage-as-a-service (StaaS) approach. Organizations also use multi-cloud or hybrid solutions by combining multiple public and/or private cloud service providers to avoid vendor lock-in, achieve high availability and performance, and optimise cost. Indeed cost is one of the important factors for organizations while adopting cloud storage; however, cloud storage providers offer complex pricing policies, including the actual storage cost and the cost related to additional services (e.g., network usage cost). In this article, we provide a detailed taxonomy of cloud storage cost and a taxonomy of other QoS elements, such as network performance, availability, and reliability. We also discuss various cost trade-offs, including storage and computation, storage and cache, and storage and network. Finally, we provide a cost comparison across different storage providers under different contexts and a set of user scenarios to demonstrate the complexity of cost structure and discuss existing literature for cloud storage selection and cost optimization. We aim that the work presented in this article will provide decision-makers and researchers focusing on cloud storage selection for data placement, cost modelling, and cost optimization with a better understanding and insights regarding the elements contributing to the storage cost and this complex problem domain.

Cloud computing has become an increasingly popular choice for businesses and individuals due to its flexibility, scalability, and convenience; however, the rising cost of cloud resources has become a significant concern for many. The pay-per-use model used in cloud computing means that costs can accumulate quickly, and the lack of visibility and control can result in unexpected expenses. The cost structure becomes even more complicated when dealing with hybrid or multi-cloud environments. For businesses, the cost of cloud computing can be a significant portion of their IT budget, and any savings can lead to better financial stability and competitiveness. In this respect, it is essential to manage cloud costs effectively. This requires a deep understanding of current resource utilization, forecasting future needs, and optimising resource utilization to control costs. To address this challenge, new tools and techniques are being developed to provide more visibility and control over cloud computing costs. In this respect, this paper explores a graph-based solution for modelling cost elements and cloud resources and potential ways to solve the resulting constraint problem of cost optimisation. We primarily consider utilization, cost, performance, and availability in this context. Such an approach will eventually help organizations make informed decisions about cloud resource placement and manage the costs of software applications and data workflows deployed in single, hybrid, or multi-cloud environments.

Modeling and understanding an expensive next-generation data center operating at a sustainable exascale performance remains a challenge yet to solve. The paper presents the approach taken by the Graph-Massivizer project, funded by the European Union, towards a sustainable data center, targeting a massive graph representation and analysis of its digital twin. We introduce five interoperable open-source tools that support this undertaking, creating an automated, sustainable loop of graph creation, analytics, optimization, sustainable resource management, and operation, emphasizing state-of-the-art progress. We plan to employ the tools for designing a massive data center graph, representing a digital twin describing spatial, semantic, and temporal relationships between the monitoring metrics, hardware nodes, cooling equipment, and jobs. The project aims to strengthen Bologna Technopole as a leading European supercomputing and big data hub offering sustainable green computing for improved societally relevant science throughput.

5G and 6G networks are expected to support various novel emerging adaptive video streaming services (e.g., live, VoD, immersive media, and online gaming) with versatile Quality of Experience (QoE) requirements such as high bitrate, low latency, and sufficient reliability. It is widely agreed that these requirements can be satisfied by adopting emerging networking paradigms like Software-Defined Networking (SDN), Network Function Virtualization (NFV), and edge computing. Previous studies have leveraged these paradigms to present network-assisted video streaming frameworks, but mostly in isolation without devising chains of Virtualized Network Functions (VNFs) that consider the QoE requirements of various types of Multime-dia Services (MS). To bridge the aforementioned gaps, we first introduce a set of multimedia VNFs at the edge of an SDN-enabled network, form diverse Service Function Chains (SFCs) based on the QoE requirements of different MS services. We then propose SARENA, an _S_FC-enabled ArchitectuRe for adaptive VidEo StreamiNg Applications. Next, we formulate the problem as a central scheduling optimization model executed at the SDN controller. We also present a lightweight heuristic solution consisting of two phases that run on the SDN controller and edge servers to alleviate the time complexity of the optimization model in large-scale scenarios. Finally, we design a large-scale cloud-based testbed including 250 HTTP Adaptive Streaming (HAS) players requesting two popular MS applications (i.e., live and VoD), conduct various experiments, and compare its effectiveness with baseline systems. Experimental results illustrate that SARENA outperforms baseline schemes in terms of users' QoE by at least 39.6%, latency by 29.3%, and network utilization by 30% in both MS services.

HTTP Adaptive Streaming (HAS) methods divide a video into smaller segments, encoded at multiple pre-defined bitrates to construct a bitrate ladder. Bitrate ladders are usually optimized per title over several dimensions, such as bitrate, resolution, and framerate. This paper adds a new dimension to the bitrate ladder by considering the energy consumption of the encoding process. Video encoders often have multiple pre-defined presets to balance the trade-off between encoding time, energy consumption, and compression efficiency. Faster presets disable certain coding tools defined by the codec to reduce the encoding time at the cost of reduced compression efficiency. Firstly, this paper evaluates the energy consumption and compression efficiency of different x265 presets for 500 video sequences. Secondly, optimized presets are selected for various representations in a bitrate ladder based on the results to guarantee a minimal drop in video quality while saving energy. Finally, a new per title model, which optimizes the trade-off between compression efficiency and energy consumption, is proposed. The experimental results show that decreasing the VMAF score by 0.15 and 0.39 while choosing an optimized preset results in encoding energy savings of 70% and 83%, respectively.

The accelerating growth of modern distributed applications with low delivery deadlines leads to a paradigm shift towards the multi-tier computing continuum. However, the geographical dispersion, heterogeneity, and availability of the continuum resources may result in failures and quality of service degradation, significantly negating its advantages and lowering users’ satisfaction. We propose in this paper a proactive application placement (PROS) method relying on distributed coordination to prevent the quality of service violations through service-level agreements on the computing continuum. PROS employs a sigmoid function with adaptive weights for the different parameters to predict the service level agreement assurance of devices based on their past credentials and current capabilities. We evaluate PROS using two application workloads with different traffic stress levels up to 90 million services on a real testbed with 600 heterogeneous instances deployed over eight geographical locations. The results show that PROS increases the success rate by 7%–33%, reduces the response time by 16%–38%, and increases the deadline satisfaction rate by 19%–42% compared to two related work methods. A comprehensive simulation study with 1000 devices and a workload of up to 670 million services confirm the scalability of the results.

The challenging demands for the next generation of the Internet of Things have led to a massive increase in edge computing and network virtualization technologies. While there is vast potential for research in these areas, managing complex adaptive infrastructure is difficult, and experiments with real hardware are tedious to set up. Furthermore, proposed solutions often require expensive hardware or labor-intensive procedures to replicate and build on these ideas. With our C3-Edge testbed, we address these challenges and propose a novel approach for automated edge testbed setup with a low-cost software-defined network and adaptive infrastructure configuration. We validated the efficiency of our approach on a real-world computing continuum infrastructure. The evaluation results confirm that our flexible approach is suitable for all but the most bandwidth-intensive applications.

Fog computing platforms became essential for deploying low-latency applications at the network's edge. However, placing and managing time-critical applications over a Fog infrastructure with many heterogeneous and resource-constrained devices over a dynamic network is challenging. This paper proposes an incremental multilayer resource-aware partitioning (M-RAP) method that minimizes resource wastage and maximizes service placement and deadline satisfaction in a dynamic Fog with many application requests. M-RAP represents the heterogeneous Fog resources as a multilayer graph, partitions it based on the network structure and resource types, and constantly updates it upon dynamic changes in the underlying Fog infrastructure. Finally, it identifies the device partitions for placing the application services according to their resource requirements, which must overlap in the same low-latency network partition. We evaluated M-RAP through extensive simulation and two applications executed on a real testbed. The results show that M-RAP can place 1.6 times as many services, satisfy deadlines for 43% more applications, lower their response time by up to 58%, and reduce resource wastage by up to 54% compared to three state-of-the-art methods.

Media production and consumption behaviors are changing in response to new technologies and demands, giving birth to a new generation of social applications. Among them, crowd journalism represents a novel way of constructing democratic and trustworthy news relying on ordinary citizens arriving at breaking news locations and capturing relevant videos using their smartphones. The ARTICONF project [1] proposes a trustworthy, resilient, and globally sustainable toolset for developing decentralized applications (DApps). Leveraging the ARTICONF tools, we introduce a new DApp for crowd journalism called MOGPlay. MOGPlay collects and manages audio-visual content generated by citizens and provides a secure blockchain platform that rewards all stakeholders involved in professional news production. Besides live streaming, MOGPlay offers a marketplace for audio-visual content trading among citizens and free journalists with an internal token ecosystem. We discuss the functionality and implementation of the MOGPlay DApp and illustrate three pilot crowd journalism live scenarios that validate the prototype.

Many researchers migrate scientific serverless workflows or function choreographies (FCs) on Function-as-a-Service (FaaS) to benefit from its high scalability and elasticity. Unfortunately, the heterogeneity of federated FaaS hampers decisions on appropriate parameter setup to run FCs. Consequently, scientists must choose between accurate but tedious and expensive experiments or simple but cheap and less accurate simulations. Unfortunately, related works support either simulation models for serverfull workflows running on virtual machines and containers or partial FaaS models for individual serverless functions focused on execution time and neglecting various kinds of federated overheads.

Today's distributed computing infrastructures en-compass complex workflows for real-time data gathering, transferring, storage, and processing, quickly overwhelming centralized cloud centers. Recently, the computing continuum that federates the Cloud services with emerging Fog and Edge devices represents a relevant alternative for supporting the next-generation data processing workflows. However, eminent challenges in automating data processing across the computing continuum still exist, such as scheduling heterogeneous devices across the Cloud, Fog, and Edge layers. We propose a new scheduling algorithm called C3-MATCH, based on matching theory principles, involving two sets of players negotiating different utility functions: 1) workflow microservices that prefer computing devices with lower data processing and queuing times; 2) computing continuum devices that prefer microservices with corresponding resource requirements and less data transmission time. We evaluate$C^{3}$-MATCH using real-world road sign inspection and sentiment analysis workflows on a federated computing continuum across four Cloud, Fog, and Edge providers. Our combined simulation and real execution results reveal that$C^{3}$-MATCH achieves up to 67% lower completion time than three state-of-the-art methods with 10 ms-1000 ms higher transmission time.

The Computing Continuum, covering Cloud, Fog, and Edge systems, promises to provide on-demand resource-as-a-service for Internet applications with diverse requirements, ranging from extremely low latency to high-performance processing. However, eminent challenges in automating the resources man-agement of Big Data pipelines across the Computing Continuum remain. The resource management and adaptation for Big Data pipelines across the Computing Continuum require significant research effort, as the current data processing pipelines are dynamic. In contrast, traditional resource management strategies are static, leading to inefficient pipeline scheduling and overly complex process deployment. To address these needs, we propose in this work a scheduling and adaptation approach implemented as a software tool to lower the technological barriers to the management of Big Data pipelines over the Computing Continuum. The approach separates the static scheduling from the run-time execution, em-powering domain experts with little infrastructure and software knowledge to take an active part in the Big Data pipeline adaptation. We conduct a feasibility study using a digital healthcare use case to validate our approach. We illustrate concrete scenarios supported by demonstrating how the scheduling and adaptation tool and its implementation automate the management of the lifecycle of a remote patient monitoring, treatment, and care pipeline.

Video streaming is the dominating traffic in today’s data-sharing world. Media service providers stream video content for their viewers, while worldwide users create and distribute videos using mobile or video system applications that significantly increase the traffic share. We propose a multilayer and pipeline encoding on the computing continuum (MPEC2) method that addresses the key technical challenge of high-price and computational complexity of video encoding. MPEC2 splits the video encoding into several tasks scheduled on appropriately selected Cloud and Fog computing instance types that satisfy the media service provider and user priorities in terms of time and cost. In the first phase, MPEC2 uses a multilayer resource partitioning method to explore the instance types for encoding a video segment. In the second phase, it distributes the independent segment encoding tasks in a pipeline model on the underlying instances. We evaluate MPEC2 on a federated computing continuum encompassing Amazon Web Services (AWS) EC2 Cloud and Exoscale Fog instances distributed in seven geographical locations. Experimental results show that MPEC2 achieves 24% faster completion time and 60% lower cost for video encoding compared to resource allocation related methods. When compared with baseline methods, MPEC2 yields 40%– 50% lower completion time and 5%–60% reduced total cost.

The emergence of advanced technologies has triggered a sweeping digital transformation in the industrial ecosystem. The cutting-edge technologies (like, Internet of things, big data, artificial intelligence, drones, cyber-physical systems, and augmented reality, and computer vision) are key enablers of this industrial revolution. Industry 4.0 has reshaped the conventional manufacturing and production processes into automated operations and workflows. This industrial transition is fueled by advanced computing (cloud and edge computing), analytic (big data analytics and computational analytics), intelligent (machine and deep learning), and communication (programmable and intelligent networks) infrastructure and technologies. The collection, aggregation, analysis, and processing of big data generated from the industrial periphery (like manufacturing equipment and maintenance systems) enable real-time decision-making and autonomous opportunities. However, the voluminous size, variability, and frequency of this data bring a wide array of disputes and oppositions in the resource-limited Industrial systems. Moreover, the continuous decision-making workflow in production and manufacturing segments increases the sharing of data across different functions, systems, and organizational boundaries. For this reason, cloud computing and big data technologies (Hadoop and Map-Reduce) can improve the anticipated response and reaction times. Industry 4.0 will lead toward more devices enriched with embedded computing platforms which boost the capabilities of the overall workflow. But, this also leads towards an increased communication and interaction between these devices which can end up in various challenges for the underlying network infrastructure. However, the conventional communication protocols may end up in various performance bottlenecks which in turn can increase the threat from different kinds of attacks and security challenges. Concluding the above discussion, the industrial ecosystem would rely on two entities: (1) users or infrastructure (physical world) and (2) cloud-enabled algorithms and autonomous systems (virtual world) that are connected through advanced and autonomous communication technologies. The driving force behind the success of these industrial ecosystems relies on the efficient gathering/collection, analysis, and storage of data generated by smart devices and sensors. Under this domain, big data analytics is set to be driving predictive manufacturing and provide timely detection of anomalies and system failures to predict product quality. In this way, big data is bound to play a prominent role in driving the industrial ecosystem. Even more, the only reason for this concern is not limited to the volume of data but the major concern is the contribution of this data for the design and implementation of efficient industrial processes and policies. The interpretation and understanding of the available data help to the design of efficient processes and policies related to industrial systems. The focus of this special issue is to present novel and seminal contributions around the important issues and challenges related to big data management and analytics for industrial 4.0 ecosystems. It provides ground-breaking research from academia and industry, that emphasizes the novel solutions, applications, tools, software, and algorithms designed to handle the industrial big data. A substantial number of submissions were received for the special issue. The papers were reviewed by at least three reviewers and underwent a rigorous two rounds of reviews. After the completion of the peer review process, we have accepted 10 seminal contributions related to big data analytics for Industry 4.0. All the accepted papers either discuss the recent solutions related to big data analytics or proposes an innovative way of handling big data across diverse infrastructure deployments. The outline of these contributions discussed below. The first paper titled "An Efficient Scheme for Secure Feature Location using Data Fusion and Data Mining in IOT Environment" by Balaji et al.1 proposes a secure feature location approach based on data fusion and data mining to overcome the challenges of the existing textual and dynamic approaches. The first step in this approach involves the removal of repeated test cases followed by the selection of important attributes. The artificial flora optimization algorithm was used to remove the repeated test cases. After this, the Caesar Cipher-RSA algorithm was used to encrypt the selected attributes, and thereafter a score value was assigned to them. This score value acts as an input to the K-mean algorithm to normalize it using the min-max approach. The evaluation results show that the proposed approach is superior in comparison to existing variants. The second paper titled "Data Dimensionality Reduction Techniques for Industry 4.0: Research Results, Challenges, and Future Research Directions" by Chhikara et al.2 provides a comprehensive survey on dimensionality reduction techniques. This survey discussed various data dimensionality techniques, analyzed them, and provided a thorough comparison based on different factors and parameters. The survey provided an understanding of the applicability of dimensionality reduction techniques in group or stand-alone use cases. The third paper titled "Deep-Q Learning-based Heterogeneous Earliest Finish Time Scheduling Algorithm for Scientific Workflows in Cloud" by Kaur et al.3 proposes a workflow scheduling approach wherein a deep-Q learning mechanism is used. This deep-Q mechanism was based on a heterogeneous earliest-finish-time algorithm was designed to amalgamate the deep learning approach with the heuristic approach for task scheduling. The evaluations were performed on a workflow simulator and the results depict the superiority of the proposed approach in contrast to the existing algorithms in terms of makespan and speed. The fourth paper titled "A multi-domain VNE algorithm based on multi-objective optimization for IoD architecture in Industry 4.0" by Zhang et al.4 proposes a multidomain virtual network embedding algorithm to improve the performance and reduce the computational delay. This algorithm is based on centralized hierarchical architecture and avoids local optimum by improvising the particle swarm optimization algorithm to include a genetic variation factor. However, as the problem is composed of multiple objectives, the proposed work simplifies the same by decomposing it into a single-objective problem using a weighted summation method. According to the obtained results, the proposed approach converges to an optimal solution quickly. Further, a candidate selection algorithm was proposed to reduce the cost associated with mapping. In this algorithm, the physical domain calculates the mapping cost for all nodes and selects the one with the lowest mapping cost. The results show the efficiency of the proposed approach in terms of delay, cost, and several other performance indicators. The fifth paper titled "A Community-based Hierarchical User Authentication Scheme for Industry 4.0" by Sinha et al.5 proposes a community-based hierarchical approach that is used to decide the way to provide access rights of the smart end devices to the users in the Industry 4.0 ecosystem. The hierarchical structure helps to ensure that only the legitimate users get access rights after clearing the multilevel authorization process. This approach also ensures identity leakage as the legitimate parties coordinate closely with each other for the authentication process. The validation shows that the proposed approach is susceptible to various types of attacks. The sixth paper titled "PSSCC: Provably Secure Communication Framework for Crowdsourced Industrial Internet of Things Environments" by Dharminder et al.6 proposes an identity-based signcryption method in the provably secure communication framework. During signcryption, the end-user performs pairing-free computation that proves to be computationally efficient. Based on the modified bilinear Diffie-Hellman inversion and strong Diffie-Hellman problems, the framework is proved to be secure in the Industrial Internet of Things environment. The evaluation was performed based on communication and computation cost and the results look very promising. The seventh paper titled "Applying Artificial Bee Colony Algorithm to the Multi-depot Vehicle Routing Problem" by Gu et al.7 utilizes an artificial bee colony algorithm in multidepot Vehicle Routing Problem to manage the vehicular routes among multiple depots in an optimized and time-efficient manner. Initially, the multidepot Vehicle Routing Problem is decomposed single-depot problem using depot clustering. After this, a modified artificial bee colony algorithm is used to generate solutions for each depot. In the end, a coevolution strategy is proposed to realize a complete solution to the multi-depot Vehicle Routing Problem. The proposed algorithm was validated through extensive experiments and the results were compared with greedy and genetic algorithms based on different parameters. The results depict a performance enhancement to the tune 70% over the greedy algorithm and 3% over the genetic algorithm. The eighth paper titled "An IoT-enabled Decision Support System for Circular Economy Business Model" by Mboli et al.8 proposes a decision support system based on the Internet of Things for a circular economy business model. This system is based on an ontological model that effectively allows to predict, track, and monitor the residual value of the product. This allows businesses to utilize circularity decisions complemented by a semantic decision support system to create a first of its kind, semantic ontological model. The proposed model was validated based on real-world use case scenario to understand viability and applicability. The ninth paper titled "Security Analytics for Real-Time Forecasting of Cyberattacks" by Javed et al.9 proposes a pattern identification framework for cyberthreats. After identification of the cyber patterns a forecasting model suggests the pattern of growth in an emerging network threat. This framework predicts the maximum threat intensity and its occurrence over the period thereby suggesting the likelihood of maximum intensity. The framework involves four steps, (1) continuous activity monitoring, (2) behavior forecasting, (3) estimating the intensity of a potential cyberattack, and (4) predicting the potential risk of cyber attacks over a predefined time window. The validation depicts an average lead time of 1.75 h good enough to limit the potential impact of the attack. The tenth paper titled "An Efficient Hadoop based Brain Tumor Detection Framework using Big Data Analytic" by Chahal et al.10 proposes a brain tumor segmentation approach based on a hybrid weighted fuzzy mechanism. This approach works in tandem with the Matlab Distributed Computing Server and Hadoop to fuzify the pixel values to create meaningful clusters of large data. The approach is validated based on huge MR brain data across clusters of varying sized DICOM datasets using hybrid fuzzy clustering in MapReduce on Hadoop. The experiments performed compared the read, write, and processing time on each node. The outcomes show an elevation in the read and write operation time with an increase in the data size to multinode. The processing time comes out to be 35 min and 3.4 min on single and three-node clusters, respectively. Further increasing the data size to 7.3 GB, the proposed approach process the data in 235.4 min and 2085.2 min for the three-node cluster and single node, respectively. We hope that the seminal research contributions and findings presented in this special issue would benefit the readers to enhance their knowledge base and encourage them to work on various aspects of big data analytics. We express our sincere gratitude and thanks to the editor-in-chief for allowing us to organize this special issue. The support from the editorial office staff was excellent and we thank them for the same. We are also thankful to all the authors who submitted their ideas and finding in this special issue and made it possible, and to the reviewers for their thoughtful and critical suggestions to improve the quality of the submitted papers.

The Edge computing extension of the Cloud services towards the network boundaries raises important placement challenges for IoT applications running in a heterogeneous environment with limited computing capacities.Unfortunately, existing works only partially address this challenge by optimizing a single or aggregate objective (e.g., response time), and not considering the edge devices' mobility and resource constraints.To address this gap, we propose a novel mobility-aware multi-objective IoT application placement (mMAPO) method in the Cloud -Edge Continuum that optimizes completion time, energy consumption, and economic cost as conflicting objectives.mMAPO utilizes a Markov model for predictive analysis of the Edge device mobility and constrains the optimization to devices that do not frequently move through the network.We evaluate the quality of the mMAPO placements using simulation and real-world experimentation on two IoT applications.Compared to related work, mMAPO reduces the economic cost by 28 percent and decreases the completion time by 80 percent while maintaining a stable energy consumption.

Latency-sensitive and bandwidth-intensive stream processing applications are dominant traffic generators over the Internet network. A stream consists of a continuous sequence of data elements, which require processing in nearly real-time. To improve communication latency and reduce the network congestion, Fog computing complements the Cloud services by moving the computation towards the edge of the network. Unfortunately, the heterogeneity of the new Cloud – Fog continuum raises important challenges related to deploying and executing data stream applications. We explore in this work a two-sided stable matching model called Cloud – Fog to data stream application matching (CODA) for deploying a distributed application rep-resented as a workflow of stream processing microservices on heterogeneous computing continuum resources. In CODA, the application microservices rank the continuum resources based on their microservice stream processing time, while resources rank the stream processing microservices based on their residual bandwidth. A stable many-to-one matching algorithm assigns microservices to resources based on their mutual preferences, aiming to optimize the complete stream processing time on the application side, and the total streaming traffic on the resource side. We evaluate the CODA algorithm using simulated and real-world Cloud – Fog experimental scenarios. We achieved 11-45% lower stream processing time and 1.3-20% lower streaming traffic compared to related state-of-the-art approaches.

Video streaming became an undivided part of the Internet. To efficiently utilise the limited network bandwidth it is essential to encode the video content. However, encoding is a computationally intensive task, involving high-performance resources provided by private infrastructures or public clouds. Public clouds, such as Amazon EC2, provide a large portfolio of services and instances optimized for specific purposes and budgets. The majority of Amazon’s instances use x86 processors, such as Intel Xeon or AMD EPYC. However, following the recent trends in computer architecture, Amazon introduced Arm-based instances that promise up to 40% better cost performance ratio than comparable x86 instances for specific workloads. We evaluate in this paper the video encoding performance of x86 and Arm instances of four instance families using the latest FFmpeg version and two video codecs. We examine the impact of the encoding parameters, such as different presets and bitrates, on the time and cost for encoding. Our experiments reveal that Arm instances show high time and cost saving potential of up to 33.63% for specific bitrates and presets, especially for the x264 codec. However, the x86 instances are more general and achieve low encoding times, regardless of the codec.

Fog computing emerged as a crucial platform for the deployment of IoT applications. The complexity of such applications require methods that handle the resource diversity and network structure of Fog devices, while maximizing the service placement and reducing the resource wastage. Prior studies in this domain primarily focused on optimizing application-specific requirements and fail to address the network topology combined with the different types of resources encountered in Fog devices. To overcome these problems, we propose a multilayer resource-aware partitioning method to minimize the resource wastage and maximize the service placement and deadline satisfaction rates in a Fog infrastructure with high multi-user application placement requests. Our method represents the heterogeneous Fog resources as a multilayered network graph and partitions them based on network topology and resource features. Afterwards, it identifies the appropriate device partitions for placing an application according to its requirements, which need to overlap in the same network topology partition. Simulation results show that our multilayer resource-aware partitioning method is able to place twice as many services, satisfy deadlines for three times as many application requests, and reduce the resource wastage by up to 15-32 times compared to two availability-aware and resource-aware state-of-the-art methods.

Organisations possess and continuously generate huge amounts of static and stream data, especially with the proliferation of Internet of Things technologies. Collected but unused data, i.e., Dark Data, mean loss in value creation potential. In this respect, the concept of Computing Continuum extends the traditional more centralised Cloud Computing paradigm with Fog and Edge Computing in order to ensure low latency pre-processing and filtering close to the data sources. However, there are still major challenges to be addressed, in particular related to management of various phases of Big Data processing on the Computing Continuum. In this paper, we set forth an ecosystem for Big Data pipelines in the Computing Continuum and introduce five relevant real-life example use cases in the context of the proposed ecosystem.

Peak mitigation is of interest to power companies as peak periods may require the operator to over provision supply in order to meet the peak demand. Flattening the usage curve can result in cost savings, both for the power companies and the end users. Integration of renewable energy into the energy infrastructure presents an opportunity to use excess renewable generation to supplement supply and alleviate peaks. In addition, demand side management can shift the usage from peak to off-peak times and reduce the magnitude of peaks. In this work, we present a data driven approach for incentive-based peak mitigation. Understanding user energy profiles is an essential step in this process. We begin by analysing a popular energy research dataset published by the Ausgrid corporation. Extracting aggregated user energy behavior in temporal contexts and semantic linking and contextual clustering give us insight into consumption and rooftop solar generation patterns. We implement, and performance test a blockchain-based prosumer incentivization system. The smart contract logic is based on our analysis of the Ausgrid dataset. Our implementation is capable of supporting 792,540 customers with a reasonably low infrastructure footprint.

Social media applications are essential for next-generation connectivity. Today, social media are centralized platforms with a single proprietary organization controlling the network and posing critical trust and governance issues over the created and propagated content. The ARTICONF project funded by the European Union's Horizon 2020 program researches a decentralized social media platform based on a novel set of trustworthy, resilient and globally sustainable tools that address privacy, robustness and autonomy-related promises that proprietary social media platforms have failed to deliver so far. This paper presents the ARTICONF approach to a car-sharing decentralized application (DApp) use case, as a new collaborative peer-to-peer model providing an alternative solution to private car ownership. We describe a prototype implementation of the car-sharing social media DApp and illustrate through real snapshots how the different ARTICONF tools support it in a simulated scenario.

Social media is a popular medium for the dissemination of real-time news all over the world. Easy and quick information proliferation is one of the reasons for its popularity. An extensive number of users with different age groups, gender, and societal beliefs are engaged in social media websites. Despite these favorable aspects, a significant disadvantage comes in the form of fake news, as people usually read and share information without caring about its genuineness. Therefore, it is imperative to research methods for the authentication of news. To address this issue, this article proposes a two-phase benchmark model named WELFake based on word embedding (WE) over linguistic features for fake news detection using machine learning classification. The first phase preprocesses the data set and validates the veracity of news content by using linguistic features. The second phase merges the linguistic feature sets with WE and applies voting classification. To validate its approach, this article also carefully designs a novel WELFake data set with approximately 72 000 articles, which incorporates different data sets to generate an unbiased classification output. Experimental results show that the WELFake model categorizes the news in real and fake with a 96.73% which improves the overall accuracy by 1.31% compared to bidirectional encoder representations from transformer (BERT) and 4.25% compared to convolutional neural network (CNN) models. Our frequency-based and focused analyzing writing patterns model outperforms predictive-based related works implemented using the Word2vec WE method by up to 1.73%.

Social media applications are essential for next generation connectivity. Today, social media are centralized platforms with a single proprietary organization controlling the network and posing critical trust and governance issues over the created and propagated content. The ARTICONF project funded by the European Union’s Horizon 2020 program researches a decentralized social media platform based on a novel set of trustworthy, resilient and globally sustainable tools to fulfil the privacy, robustness and autonomy-related promises that proprietary social media platforms have failed to deliver so far. This paper presents the ARTICONF approach to a car-sharing use case application, as a new collaborative peer-to-peer model providing an alternative solution to private car ownership. We describe a prototype implementation of the car-sharing social media application and illustrate through real snapshots how the different ARTICONF tools support it in a simulated scenario.

Nowadays, massive amounts of data are acquired, transferred, and analyzed nearly in real-time by utilizing a large number of computing and storage elements interconnected through high-speed communication networks. However, one issue that still requires research effort is to enable efficient monitoring of applications and infrastructures of such complex systems. In this paper, we introduce an Integer Linear Programming (ILP) model called M3AT for optimized assignment of monitoring agents and aggregators on large-scale computing systems. We identified a set of requirements from three representative data-intensive applications and exploited them to define the model's input parameters. We evaluated the scalability of M3AT using the Constraint Integer Programing (SCIP) solver with default configuration based on synthetic data sets. Preliminary results show that the model provides optimal assignments for subsystems composed of up to 200 monitoring agents with complex I/O policies, while keeping the number of aggregators constant and demonstrates variable sensitivity with respect to the scale of monitoring data aggregators and limitation policies imposed.

Minimizing the resource wastage reduces the energy cost of operating a data center, but may also lead to a considerably high resource overcommitment affecting the Quality of Service (QoS) of the running applications. The effective tradeoff between resource wastage and overcommitment is a challenging task in virtualized Clouds and depends on the allocation of virtual machines (VMs) to physical resources. We propose in this paper a multi-objective method for dynamic VM placement, which exploits live migration mechanisms to simultaneously optimize the resource wastage, overcommitment ratio and migration energy. Our optimization algorithm uses a novel evolutionary meta-heuristic based on an island population model to approximate the Pareto optimal set of VM placements with good accuracy and diversity. Simulation results using traces collected from a real Google cluster demonstrate that our method outperforms related approaches by reducing the migration energy by up to 57% with a QoS increase below 6%.

Virtual machine image retrieval a b s t r a c tInfrastructure-as-a-service (IaaS) Clouds concurrently accommodate diverse sets of user requests, requiring an efficient strategy for storing and retrieving virtual machine images (VMIs) at a large scale.The VMI storage management requires dealing with multiple VMIs, typically in the magnitude of gigabytes, which entails VMI sprawl issues hindering the elastic resource management and provisioning.Unfortunately, existing techniques to facilitate VMI management overlook VMI semantics (i.e at the level of base image and software packages), with either restricted possibility to identify and extract reusable functionalities or with higher VMI publishing and retrieval overheads.In this paper, we propose Expelliarmus, a novel VMI management system that helps to minimize VMI storage, publishing and retrieval overheads.To achieve this goal, Expelliarmus incorporates three complementary features.First, it models VMIs as semantic graphs to facilitate their similarity computation.Second, it provides a semantically-aware VMI decomposition and base image selection to extract and store non-redundant base image and software packages.Third, it assembles VMIs based on the required software packages upon user request.We evaluate Expelliarmus through a representative set of synthetic Cloud VMIs on a real test-bed.Experimental results show that our semantic-centric approach is able to optimize the repository size by 2.3 -22 times compared to state-of-the-art systems (e.g.IBM's Mirage and Hemera) with significant VMI publishing and slight retrieval performance improvement.

Many researchers rely on simulations to analyze and validate their researched methods on Cloud infrastructures. However, determining relevant simulation parameters and correctly instantiating them to match the real Cloud performance is a difficult and costly operation, as minor configuration changes can easily generate an unreliable inaccurate simulation result. Using legacy values experimentally determined by other researchers can reduce the configuration costs, but is still inaccurate as the underlying public Clouds and the number of active tenants are highly different and dynamic in time. To overcome these deficiencies, we propose a novel model that simulates the dynamic Cloud performance by introducing noise in the computation and communication tasks, determined by a small set of runtime execution data. Although the estimating method is apparently costly, a comprehensive sensitivity analysis shows that the configuration parameters determined for a certain simulation setup can be used for other simulations too, thereby reducing the tuning cost by up to 82.46 percent, while declining the simulation accuracy by only 1.98 percent on average. Extensive evaluation also shows that our novel model outperforms other state-of-the-art dynamic Cloud simulation models, leading up to 22 percent lower makespan inaccuracy.

Realistic, relevant, and reproducible experiments often need input traces collected from real-world environments. In this work, we focus on traces of workflows-common in datacenters, clouds, and HPC infrastructures. We show that the state-of-the-art in using workflow-traces raises important issues: (1) the use of realistic traces is infrequent and (2) the use of realistic, open-access traces even more so. Alleviating these issues, we introduce the Workflow Trace Archive (WTA), an open-access archive of workflow traces from diverse computing infrastructures and tooling to parse, validate, and analyze traces. The WTA includes > 48 million workflows captured from > 10 computing infrastructures, representing a broad diversity of trace domains and characteristics. To emphasize the importance of trace diversity, we characterize the WTA contents and analyze in simulation the impact of trace diversity on experiment results. Our results indicate significant differences in characteristics, properties, and workflow structures between workload sources, domains, and fields.

Minimizing the resource wastage reduces the energy cost of operating a data center, but may also lead to a considerably high resource overcommitment affecting the Quality of Service (QoS) of the running applications. Determining the effective tradeoff between resource wastage and overcommitment is a challenging task in virtualized Cloud data centers and depends on how Virtual Machines (VMs) are allocated to physical resources. In this paper, we propose a multi-objective framework for dynamic placement of VMs exploiting live-migration mechanisms which simultaneously optimize the resource wastage, overcommitment ratio and migration cost. The optimization algorithm is based on a novel evolutionary meta-heuristic using an island population model underneath. We implemented and validated our method based on an enhanced version of a well-known simulator. The results demonstrate that our approach outperforms other related approaches by reducing up to 57% migrations energy consumption while achieving different energy and QoS goals.

Infrastructure-as-a-service (IaaS) Clouds concurrently accommodate diverse sets of user requests, requiring an efficient strategy for storing and retrieving virtual machine images (VMIs) at a large scale. The VMI storage management require dealing with multiple VMIs, typically in the magnitude of gigabytes, which entails VMI sprawl issues hindering the elastic resource management and provisioning. Nevertheless, existing techniques to facilitate VMI management overlook VMI semantics (i.e at the level of base image and software packages) with either restricted possibility to identify and extract reusable functionalities or with higher VMI publish and retrieval overheads. In this paper, we design, implement and evaluate Expelliarmus, a novel VMI management system that helps to minimize storage, publish and retrieval overheads. To achieve this goal, Expelliarmus incorporates three complementary features. First, it makes use of VMIs modelled as semantic graphs to expedite the similarity computation between multiple VMIs. Second, Expelliarmus provides a semantic aware VMI decomposition and base image selection to extract and store non-redundant base image and software packages. Third, Expelliarmus can also assemble VMIs based on the required software packages upon user request. We evaluate Expelliarmus through a representative set of synthetic Cloud VMIs on the real test-bed. Experimental results show that our semantic-centric approach is able to optimize repository size by 2.2 - 16 times compared to state-of-the-art systems (e.g. IBM's Mirage and Hemera) with significant VMI publish and slight retrieval performance improvement.

Summary New software engineering technologies facilitate development of applications from reusable software components, such as Virtual Machine and container images (VMI/CIs). Key requirements for the storage of VMI/CIs in public or private repositories are their fast delivery and cloud deployment times. ENTICE is a federated storage facility for VMI/CIs that provides optimisation mechanisms through the use of fragmentation and replication of images and a Pareto Multi‐Objective Optimisation (MO) solver. The operation of the MO solver is, however, time‐consuming due to the size and complexity of the metadata, specifying various non‐functional requirements for the management of VMI/CIs, such as geolocation, operational cost, and delivery time. In this work, we address this problem with a new semantic approach, which uses an ontology of the federated ENTICE repository, knowledge base, and constraint‐based reasoning mechanism. Open Source technologies such as Protégé, Jena Fuseki, and Pellet were used to develop a solution. Two specific use cases, (1) repository optimisation with offline and (2) online redistribution of VMI/CIs, are presented in detail. In both use cases, data from the knowledge base are provided to the MO solver. It is shown that Pellet‐based reasoning can be used to reduce the input metadata size used in the optimisation process by taking into consideration the geographic location of the VMI/CIs and the provenance of the VMI fragments. It is shown that this process leads to reduction of the input metadata size for the MO solver by up to 60% and reduction of the total optimisation time of the MO solver by up to 68%, while fully preserving the quality of the solution, which is significant.

Large scale computing solutions are increasingly used in the context of Big Data platforms, where efficient scheduling algorithms play an important role in providing optimized cluster resource utilization, throughput and fairness. This paper deals with the problem of scheduling a set of jobs across a cluster of machines handling the specific use case of fair scheduling for jobs and machines with heterogeneous characteristics. Although job and cluster diversity is unprecedented, most schedulers do not provide implementations that handle multiple resource type fairness in a heterogeneous system. We propose in this paper a new scheduler called h-Fair that selects jobs for scheduling based on a global dominant resource fairness heterogeneous policy, and dispatches them on machines with similar characteristics to the resource demands using the cosine similarity. We implemented h-Fair in Apache Hadoop YARN and we compare it with the existing Fair Scheduler that uses the dominant resource fairness policy based on the Google workload trace. We show that our implementation provides better cluster resource utilization and allocates more containers when jobs and machines have heterogeneous characteristics.

Demand response management in smart cities is one of the most challenging tasks to be performed due to the continuous changes in the load profile of the home users. The existing proposals in the literature fail to observe the hidden patterns in the load profile of these users. So, to fill these gaps, the concept of deep learning has been used in this paper for smart energy management in a smart city. The consumption data from smart homes (SHs) is gathered and taken as an input to the deep learning model, convolution neural network (CNN). The CNN model learns the hidden patterns in the data and outputs different load curves. These load curves are then used to train a support vector regression (SVR) model, which predicts the overall load consumption of all SHs in the smart city. This prediction is then compared with the power generation from the grid and consequently the demand response (DR) of the connected SHs is managed so as to minimize the gap between predicted demand and supply. The proposed scheme has been evaluated on the dataset collected from PJM and open energy information with respect to load demand prediction and DR management. The results obtained prove the efficacy of the proposed scheme. The prediction errors, i.e., root mean squared error and mean absolute percentage error are observed less in comparison to the cases when CNN and SVR are used individually.

During the last decade, Cloud computing has efficiently exploited the economy of scale by providing low cost computational and storage resources over the Internet, eventually leading to consolidation of computing resources into large data centers. However, the nascent of the highly decentralized Internet of Things (IoT) technologies that cannot effectively utilize the centralized Cloud infrastructures pushes computing towards resource dispersion. Fog computing extends the Cloud paradigm by enabling dispersion of the computational and storage resources at the edge of the network in a close proximity to where the data is generated. In its essence, Fog computing facilitates the operation of the limited compute, storage and networking resources physically located close to the edge devices. However, the shared complexity of the Fog and the influence of the recent IoT trends moving towards deploying and interconnecting extremely large sets of pervasive devices and sensors, requires exploration of adaptive Fog architectural approaches capable of adapting and scaling in response to the unpredictable load patterns of the distributed IoT applications. In this paper we introduce a promising new nature- inspired Fog architecture, named SmartFog, capable of providing low decision making latency and adaptive resource management. By utilizing novel algorithms and techniques from the fields of multi- criteria decision making, graph theory and machine learning we model the Fog as a distributed intelligent processing system, therefore emulating the function of the human brain.

Summary The use of virtual machines (VMs) in Cloud computing provides various benefits in the overall software engineering lifecycle. These include efficient elasticity mechanisms resulting in higher resource utilization and lower operational costs. The VMs as software artifacts are created using provider‐specific templates, called virtual machine images (VMI), and are stored in proprietary or public repositories for further use. However, some technology‐specific choices can limit the interoperability among various Cloud providers and bundle the VMIs with nonessential or redundant software packages, leading to increased storage size, prolonged VMI delivery, stagnant VMI instantiation, and ultimately vendor lock‐in. To address these challenges, we present a set of novel functionalities and design approaches for efficient operation of distributed VMI repositories, specifically tailored for enabling (1) simplified creation of lightweight and size optimized VMIs tuned for specific application requirements; (2) multi‐objective VMI repository optimization; and (3) efficient reasoning mechanism to help optimizing complex VMI operations. The evaluation results confirm that the presented approaches can enable VMI size reduction by up to 55%, while trimming the image creation time by 66%. Furthermore, the repository optimization algorithms can reduce the VMI delivery time by up to 51% and cut down the storage expenses by 3%. Moreover, by implementing replication strategies, the optimization algorithms can increase the system reliability by 74%.

Large Scale

In the recent years, overlay networks have emerged as a crucial platform for deployment of various distributed applications. Many of these applications rely on data redundancy techniques, such as erasure coding, to achieve higher fault tolerance. However, erasure coding applied in large scale overlay networks entails various overheads in terms of storage, latency and data rebuilding costs. These overheads are largely attributed to the selected erasure coding scheme and the encoded chunk placement in the overlay network. This paper explores a multi-objective optimization approach for identifying appropriate erasure coding schemes and encoded chunk placement in overlay networks. The uniqueness of our approach lies in the consideration of multiple erasure coding objectives such as encoding rate and redundancy factor, with overlay network performance characteristics like storage consumption, latency and system reliability. Our approach enables a variety of tradeoff solutions with respect to these objectives to be identified in the form of a Pareto front. To solve this problem, we propose a novel two stage multiobjective evolutionary algorithm, where the first stage determines the optimal set of encoding schemes, while the second stage optimizes placement of the corresponding encoded data chunks in overlay networks of varying sizes. We study the performance of our method by generating and analyzing the Pareto optimal sets of tradeoff solutions. Experimental results demonstrate that the Pareto optimal set produced by our multi-objective approach includes and even dominates the chunk placements delivered by a related state-of-the-art weighted sum method.

Virtualization is a key enabling technology in Cloud computing that allows users to run multiple virtual machines (VMs) with their own application environment on top of physical hardware. It permits scaling up and down of applications by elastic on-demand provisioning of VMs in response to their variable load to achieve increased utilization efficiency at a lower operational cost, while guaranteeing the desired level of Quality of Service (QoS) to the end-users. Typically, VMs are created using provider-specific templates that are stored in proprietary repositories, leading to provider lock-in and hampering portability or simultaneous usage of multiple federated Clouds. In this context, optimization at the level of the virtual machine image is needed both by the applications and by the underlying Cloud providers for improved resource usage, operational costs, elasticity, storage use, and other desired QoS-related features. To overcome those issues, the ENTICE project researches and creates a novel VM repository and operational environment for federated Cloud infrastructures. There exists a large variety of industrial applications that can strongly benefit by the ENTICE environment. In this paper we present an interesting selection of complementary use cases that drive the definition of the essential requirements for the ENTICE environment, and more importantly, validate the introduced innovations.

Cloud is an eco-system in which virtual machine instances are starting and terminating asynchronously on user demand or automatically when the load is rapidly increased or decreased. Although this dynamic environment allows to rent computing or storage resources cheaper rather than buying them, still it does not guarantee the stable execution during a period of time as the traditional physical environment. This is emphasised even more for workflows execution, since they consist of many data and control dependencies, which cause the makespan to be instable when a workflow is being executed in different periods of time in Cloud. In this paper we analyse several parameters of workflow and the cloud environment that are expected to impact the workflow execution instability and investigate the correlation between them. The cloud parameters include the number of instances and their type, as well as the correlation with the efficient or inefficient execution of workflow parallel sections. We conduct a series of experiments, repeating each experiment by 30 test cases in order to evaluate instability for different cloud and workflow parameters. The results show a neglectful correlation between each pair of parameters, as well as the tasks and file transfers within the workflow. Oppose to the expectations, the distribution of the makespan per experiment does not always comply with the normal distribution, which is also not correlated to a particular cloud or workflow parameter.

Due to the difficulty of employing real data centres' infrastructure for assessing the effectiveness of energy-aware algorithms, many researchers resort on using Cloud simulators. These tools require precise and detailed models for virtualized data centres in order to deliver accurate results. In recent years, many models have been proposed, but most of them either do not consider energy consumption related to virtual machine(VM) migration or ignore some of the energy-impacting components (e.g. CPU, network, storage). In this paper, we propose a new model for data centre energy consumption that takes into account these omitted components. We implement this model in a framework that combines two Cloud simulators: GroudSim that provides the Cloud management side, and DISSECT-CF that provides the internal infrastructure side. We evaluated our model in a comprehensive set of scenarios and obtained an accuracy between 8% and 22% for instantaneous power consumption, and between 8% and 25% for energy consumption.

Following the wide adoption by industry of the cloud computing technologies, we can talk about a second generation of cloud services and products that are currently under design phase. However, it is not yet clear how the third generation of cloud products and services of the next decade will look like, especially at the delivery level of Infrastructure-as-a-Service. In order to answer at least partially to such a challenging question, we initiated a literature overview and two surveys involving the members of a cluster of European research and innovation actions. The results are interpreted in this paper and a set of topics of interest for the third generation are identified.

The speedup is usually limited by two main laws in high-performance computing, that is, the Amdahl's and Gustafson's laws.However, the speedup sometimes can reach far beyond the limited linear speedup, known as superlinear speedup, which means that the speedup is greater than the number of processors that are used.Although the superlinear speedup is not a new concept and many authors have already reported its existence, most of them reported it as a side effect, without explaining why and how it is happening.In this paper, we analyze several different superlinear speedup types and define a taxonomy for them.Additionally, we present several explanations and cases of superlinearity existence for different types of granular algorithms (tasks), which means that they can be divided into many sub-tasks and scattered to the processors for execution.Apart from frequent explanation that having more cache memory in parallel execution is the main reason, we summarize other different effects that cause the superlinearity, including the superlinear speedup in cloud virtual environment for both vertical and horizontal scaling.

This paper presents OpenSwarm, a lightweight easy-to-use open-source operating system. To our knowledge, it is the first operating system designed for and deployed on miniature robots. OpenSwarm operates directly on a robot's microcontroller. It has a memory footprint of 1 kB RAM and 12 kB ROM. OpenSwarm enables a robot to execute multiple processes simultaneously. It provides a hybrid kernel that natively supports preemptive and cooperative scheduling, making it suitable for both computationally intensive and swiftly responsive robotics tasks. OpenSwarm provides hardware abstractions to rapidly develop and test platform-independent code. We show how OpenSwarm can be used to solve a canonical problem in swarm robotics—clustering a collection of dispersed objects. We report experiments, conducted with five e-puck mobile robots, that show that an OpenSwarm implementation performs as good as a hardware-near implementation. The primary goal of OpenSwarm is to make robots with severely constrained hardware more accessible, which may help such systems to be deployed in real-world applications.

Summary Scientific workflow systems face new challenges when supporting Cloud computing, as the information on the state of the used infrastructures is much less detailed than before. Thus, organising virtual infrastructures in a way that not only supports the workflow execution but also optimises it for several service level objectives (e.g. maximum energy consumption limit, cost, reliability, availability) become reliant on good Cloud modelling and prediction information. While simulators were successfully aiding research on such workflow management systems, the currently available Cloud related simulation toolkits suffer from several issues (e.g. scalability and narrow scope) that hinder their applicability. To address these issues, this article introduces techniques for unifying two existing simulation toolkits by first analysing the problems with the current simulators, and then by illustrating the problems faced by workflow systems. We use for this purpose the example of the ASKALON environment, a scientific workflow composition and execution tool for cloud and grid environments. We illustrate the advantages of a workflow system with directly integrated simulation back‐end and how the unification of the selected simulators does not affect the overall workflow execution simulation performance. Copyright © 2015 John Wiley & Sons, Ltd.

Energy consumption has become a significant issue for data centres. Assessing their consumption requires precise and detailed models. In the latter years, many models have been proposed, but most of them either do not consider energy consumption related to virtual machine migration or do not consider the variation of the workload on (1) the virtual machines (VM) and (2) the physical machines hosting the VMs. In this paper, we show that omitting migration and workload variation from the models could lead to misleading consumption estimates. Then, we propose a new model for data centre energy consumption that takes into account the previously omitted model parameters and provides accurate energy consumption predictions for paravirtualised virtual machines running on homogeneous hosts. The new model's accuracy is evaluated with a comprehensive set of operational scenarios. With the use of these scenarios we present a comparative analysis of our model with similar state-of-the-art models for energy consumption of VM Migration, showing an improvement up to 24% in accuracy of prediction.

We address the problem of scheduling a class of large-scale applications inspired from real-world on hybrid Clouds, characterized by a large number of homogeneous and concurrent tasks that are the main sources of bottlenecks but open great potential for optimization. We formulate the scheduling problem as a new sequential cooperative game and propose a communication- and storage-aware multi-objective algorithm that optimizes two user objectives (execution time and economic cost) while fulfilling two constraints (network bandwidth and storage requirements). We present comprehensive experiments using both simulation and real-world applications that demonstrate the efficiency and effectiveness of our approach in terms of algorithm complexity, make span, cost, system-level efficiency, fairness, and other aspects compared with other related algorithms.

Scheduling multiple large-scale parallel workflow applications on heterogeneous computing systems like hybrid clouds is a fundamental NP-complete problem that is critical to meeting various types of QoS (Quality of Service) requirements. This paper addresses the scheduling problem of large-scale applications inspired from real-world, characterized by a huge number of homogeneous and concurrent bags-of-tasks that are the main sources of bottlenecks but open great potential for optimization. The scheduling problem is formulated as a new sequential cooperative game and propose a communication and storage-aware multi-objective algorithm that optimizes two user objectives (execution time and economic cost) while fulfilling two constraints (network bandwidth and storage requirements). We present comprehensive experiments using both simulation and real-world applications that demonstrate the efficiency and effectiveness of our approach in terms of algorithm complexity, makespan, cost, system-level efficiency, fairness, and other aspects compared with other related algorithms.

While in the past scheduling algorithms were almost exclusively targeted at optimizing applications' make span, today they must simultaneously optimise several goals. Among these goals, energy efficiency is receiving increasing attention for environmental and financial reasons. In contrast to related work that optimises energy consumption as a single objective function, we reformulate in this paper the problem as a bi-objective optimisation by considering both make span and energy as goals. We study the potential benefits of using a Pareto-based workflow scheduling algorithm called MOHEFT using realistic energy consumption and performance models for task executions. We analyse the tradeoff solutions computed by MOHET for different workflows (different in shapes and sizes) in different execution scenarios (different resources in terms of energy consumption). The obtained results show that our bi-objective approach found in some cases schedules that reduce the energy consumption up to 85% with only 3.3% of make span concessions.

Public commercial clouds emerged as new and attractive resource provisioning option for scientific computing. This new alternative raises new challenges for users of such clouds, since optimizing the completion time of scientific applications might substantially increase the monetary cost of leasing cloud resources. In this paper, we first propose a set of basic rescheduling operations covering a broad set of scenarios for reducing the costs of running scientific workflows in clouds. Based on them, we design two heuristic scheduling algorithms. The first algorithm aims at reducing the cost of resource provisioning while still attaining the optimal make span. The second algorithm further reduces the costs to meet a budget constraint with a small increase in the make span. The experiments conducted using real-world and synthetic workflow applications demonstrate important benefits compared to related state-of-the-art approaches.

Energy efficiency has become a key issue in computer science related research and development over the last years. While most approaches focus either on hardware or on software, we propose a solution incorporating both hardware and software enabling the measurement the energy consumption of code segments executed on physical machines. Our novel approach to energy measurement and instrumentation allows for both state-of-the-art offline analysis, and innovative online measurements associated with the code being executed. We present our modular architecture which shields the users from in-depth knowledge of their energy measurement hardware, and allows the development of code for measurement and instrumentation independent of each instruments' proprietary interface. Finally, we propose an efficient method for increasing the accuracy of measurements for low sampling rate measurement devices.

The ultimate goal of cloud providers by providing resources is increasing their revenues. This goal leads to a selfish behavior that negatively affects the users of a commercial multicloud environment. In this paper, we introduce a pricing model and a truthful mechanism for scheduling single tasks considering two objectives: monetary cost and completion time. With respect to the social cost of the mechanism, i.e., minimizing the completion time and monetary cost, we extend the mechanism for dynamic scheduling of scientific workflows. We theoretically analyze the truthfulness and the efficiency of the mechanism and present extensive experimental results showing significant impact of the selfish behavior of the cloud providers on the efficiency of the whole system. The experiments conducted using real-world and synthetic workflow applications demonstrate that our solutions dominate in most cases the Pareto-optimal solutions estimated by two classical multiobjective evolutionary algorithms.

Traditional scheduling research usually targets make span as the only optimization goal, while several isolated efforts addressed the problem by considering at most two objectives. In this paper we propose a general framework and heuristic algorithm for multi-objective static scheduling of scientific workflows in heterogeneous computing environments. The algorithm uses constraints specified by the user for each objective and approximates the optimal solution by applying a double strategy: maximizing the distance to the constraint vector for dominant solutions and minimizing it otherwise. We analyze and classify different objectives with respect to their impact on the optimization process and present a four-objective case study comprising make span, economic cost, energy consumption, and reliability. We implemented the algorithm as part of the ASKALON environment for Grid and Cloud computing. Results for two real-world applications demonstrate that the solutions generated by our algorithm are superior to user-defined constraints most of the time. Moreover, the algorithm outperforms a related bi-criteria heuristic and a bi-criteria genetic algorithm.

Nowadays, scientists and companies are confronted with multiple competing goals such as makespan in high-performance computing and economic cost in Clouds that have to be simultaneously optimized. Multi-objective scheduling of scientific workflows in distributed systems is therefore receiving increasing research attention. Most existing approaches typically aggregate all objectives in a single function, defined a-priori without any knowledge about the problem being solved, which negatively impacts the quality of the solutions. In contrast, Pareto-based approaches having as outcome a set of several (nearly-) optimal solutions that represent a tradeoff among the different objectives, have been scarcely studied. In this paper, we propose a new Pareto-based list scheduling heuristic that provides the user with a set of tradeoff optimal solutions from where the one that better suits the user requirements can be manually selected. We demonstrate the potential of MOHEFT for a bi-objective scheduling problem that optimizes makespan and economic cost in a Cloud-based computing scenario. We compare MOHEFT with two state-of-the-art approaches using different synthetic and real-world workflows: the classical HEFT algorithm used in single-objective scheduling and the SPEA2* genetic algorithm used for multi-objective optimisation problems.

Today, the use of GPUs as coprocessors to accelerate high-performance scientific applications is becoming an important practice. Still, some of the high-level programming languages such as Java require extensions or new interfaces for utilising the huge parallelism of these new devices. In this paper, we propose extensions to an existing Java-based programming and parallel computing environment called Java Symphony to enable Java applications use accelerating devices such as GPUs with little API programmability change. With Java Symphony, a parallel Java application can be uniformly programmed and executed on heterogeneous platforms consisting of conventional parallel computers enhanced with data-parallel coprocessors such as GPUs. We report results on using Java Symphony for programming and improving the performance of six real applications and benchmarks in a heterogeneous environment consisting of a combination of different multi-core CPU and GPU devices.

The Message Passing Interface (MPI) provides bindings for the three programming languages commonly used in High Performance Computing (HPC): C, C++ and Fortran. Unfortunately, MPI supports only the lowest common denominator of the three languages, providing a level of abstraction far lower than typical C++ libraries. Lately, after the decision of the MPI committee to deprecate and remove the C++ bindings from the MPI standard, programmers are forced to use either the C API or rely on third-party libraries. In this paper we present a lightweight, header-only C++ interface to MPI which uses object oriented and generic programming concepts to improve its integration into the C++ programming language. We compare our wrapper with a related approach called Boost. MPI showing how MPP facilitates the interaction with C++ objects. Performance wise, MPP outperforms Boost. MPI by reducing the interface overhead by a factor of eight. Additionally, MPP's handling of user-defined data types allows transferring of STL containers (e.g. std::list) up to 20 times faster than Boost. MPI for small linked lists by relying on software serialization.

We propose a new heuristic called Resubmission Impact to support fault tolerant execution of scientific workflows in heterogeneous parallel and distributed computing environments. In contrast to related approaches, our method can be effectively used on new or unfamiliar environments, even in the absence of historical executions or failure trace models. On top of this method, we propose a dynamic enactment and rescheduling heuristic able to execute workflows with a high degree of fault tolerance, while taking into account soft deadlines. Simulated experiments of three real-world workflows in the Austrian Grid demonstrate that our method significantly reduces the resource waste compared to conservative task replication and resubmission techniques, while having a comparable makespan and only a slight decrease in the success probability. On the other hand, the dynamic enactment method manages to successfully meet soft deadlines in faulty environments in the absence of historical failure trace information or models.

Commercial distributed systems such as Clouds are managed by selfish providers that strategically try to increase their revenues regardless of the utility of other providers and users. These selfish behaviors affect the efficiency of using such environments. In this paper, based on a general game theoretic truthful reverse auction mechanism, we investigate the scheduling problem of dependent tasks on distributed Cloud resources owned by selfish providers. The social cost of the game is to minimize the make span and monetary cost simultaneously. Extensive simulation experiments show that the schedules obtained are approximately Pareto optimal.

Over the past years there has been a steady change in CPU design towards both many-core processors and power-aware hardware architectures. These two trends are combined in the Intel Single-chip Cloud Computer (SCC), an experimental prototype with 48 Pentium cores created by Intel Labs. The SCC is a highly configurable many-core chip which provides unique opportunities to optimize run time, communication and memory access as well as power/energy consumption of parallel programs. The aim of this paper is to characterize the performance behavior of the chip with various power settings, mappings of processes/cores to memory controllers, etc through benchmarking. Analytical models are used to verify and interpret the results. Conclusions drawn from our benchmark outcomes are that data exchange based on message passing is faster than shared memory data exchange. Contrary to popular belief, lowest energy consumption is not achieved for the fastest execution time. Furthermore in order to improve the memory access behavior one should increase the clock frequency of both, mesh network and memory controllers. In general, the results of our investigations can be used to analyze the effect of power settings and architecture properties on the performance and energy consumption of parallel programs as well as assist in choosing appropriate settings for specific workloads.

Today, highly successful Massively Multiplayer Online Games (MMOGs) have millions of registered users and hundreds of thousands of active concurrent players. To sustain their highly variable load, game operators over-provision a large static infrastructure capable of sustaining the game peak load, even though a large portion of the resources is unused most of the time. This inefficient resource utilisation has negative economic impacts by preventing any but the largest hosting centres from joining the market and dramatically increases prices.In this paper, we propose a new business model of hosting and operating MMOGs based on Cloud computing principles involving four actors: resource provider, game operator, game provider, and client. Our model efficiently provisions on-demand virtualised resources to game sessions based on their dynamic client load, which dramatically decreases prices and gives small and medium enterprises the opportunity of joining the market through zero initial investment.We validate our new model and its underlying business relationships through trace-based simulations utilising six months worth of monitoring data from a real-life MMOG using emulated resources from 16 of the largest Cloud resource providers currently on the market. We demonstrate that our model can operate state-of-the-art MMOGs with an average monthly gross profit of nearly $6 million excluding game purchase prices, overheads and taxation, while being able to maintain and control the QoS offered to all clients. Finally, we show how our approach is capable of operating next generation very highly interactive MMOGs with a small increase of 5.8% in the subscription price.

Cloud computing is an emerging commercial infrastructure paradigm that promises to eliminate the need for maintaining expensive computing facilities by companies and institutes alike. Through the use of virtualization and resource time sharing, clouds serve with a single set of physical resources a large user base with different needs. Thus, clouds have the potential to provide to their owners the benefits of an economy of scale and, at the same time, become an alternative for scientists to clusters, grids, and parallel production environments. However, the current commercial clouds have been built to support web and small database workloads, which are very different from typical scientific computing workloads. Moreover, the use of virtualization and resource time sharing may introduce significant performance penalties for the demanding scientific computing workloads. In this work, we analyze the performance of cloud computing services for scientific computing workloads. We quantify the presence in real scientific computing workloads of Many-Task Computing (MTC) users, that is, of users who employ loosely coupled applications comprising many tasks to achieve their scientific goals. Then, we perform an empirical evaluation of the performance of four commercial cloud computing services including Amazon EC2, which is currently the largest commercial cloud. Last, we compare through trace-based simulation the performance characteristics and cost models of clouds and other scientific computing platforms, for general and MTC-based scientific computing workloads. Our results indicate that the current clouds need an order of magnitude in performance improvement to be useful to the scientific community, and show which improvements should be considered first to address this discrepancy between offer and demand.

Today's Massively Multiplayer Online Games (MMOGs) can include millions of concurrent players spread across the world and interacting with each other within a single session. Faced with high resource demand variability and with misfit resource renting policies, the current industry practice is to overprovision for each game tens of self-owned data centers, making the market entry affordable only for big companies. Focusing on the reduction of entry and operational costs, we investigate a new dynamic resource provisioning method for MMOG operation using external data centers as low-cost resource providers. First, we identify in the various types of player interaction a source of short-term load variability, which complements the long-term load variability due to the size of the player population. Then, we introduce a combined MMOG processor, network, and memory load model that takes into account both the player interaction type and the population size. Our model is best used for estimating the MMOG resource demand dynamically, and thus, for dynamic resource provisioning based on the game world entity distribution. We evaluate several classes of online predictors for MMOG entity distribution and propose and tune a neural network-based predictor to deliver good accuracy consistently under real-time performance constraints. We assess using trace-based simulation the impact of the data center policies on the quality of resource provisioning. We find that the dynamic resource provisioning can be much more efficient than its static alternative even when the external data centers are busy, and that data centers with policies unsuitable for MMOGs are penalized by our dynamic resource provisioning method. Finally, we present experimental results showing the real-time parallelization and load balancing of a real game prototype using data center resources provisioned using our method and show its advantage against a rudimentary client threshold approach.

The drift towards new challenges in Grid computing including scientific workflow management implies the need for new, robust, multicriteria scheduling algorithms that can be applied by the user in an intuitive way. Currently existing bi-criteria scheduling approaches for scientific workflows are usually restricted to certain criterion pairs and require the user to define his preferences either as weights assigned each criterion or as fixed constraints defined for one criterion. The first approach has the drawback that combining multiple criteria into a single objective function is not always intuitive to the end-user, while the second requiresa prioriknowledge about the result of the first criteria scheduling result. We propose a new bi-criteria scheduling specification method defining for the secondary criterion a sliding constraint as a function of the primary criterion. We model the problem as an extension of the multiple-choice knapsack problem and propose a general bi-criteria scheduling heuristic called dynamic constraint algorithm (DCA) based on dynamic programming. We show through simulation that in most of the experimental cases DCA outperforms two existing algorithms designed for the same problem at the expense of an increased execution time, which is still relatively low for workflows of medium size. Finally, we confirm our simulation results for a real-world hydrological application executed in the Austrian Grid environment.

Grid schedulers require individual activity performance predictions to map workflow activities on different Grid sites. The effectiveness of the scheduling systems is hampered by inaccurate predictions due to the inability of existing predictors to effectively model the dynamic and heterogeneous nature of Grid resources, or the wide range of problem sizes and runtime arguments. To address this deficiency, we propose a hybrid Bayesian-neural network approach to dynamically model and predict the execution time of activities in real workflow applications. Bayesian network is used for a high-level representation of activities performance probability distribution against different factors affecting the performance. The important attributes are dynamically selected by the Bayesian network and fed into a radial basis function neural network to make further predictions. Our approach is generic to any type of scientific applications, and flexible to import expert knowledge to further improve accuracies. Experimental results for activities from three realworld workflow applications are presented to show effectiveness of our approach.

Even though highly distributed environments such as Clouds and Grids are increasingly used for e-science high performance applications, they still cannot deliver the robustness and reliability needed for widespread acceptance as ubiquitous scientific tools. To overcome this problem, existing systems resort to fault tolerance mechanisms such as task replication and task resubmission. In this paper we propose a new heuristic called resubmission impact to enhance the fault tolerance support for scientific workflows in highly distributed systems. In contrast to related approaches, our method can be used effectively on systems even in the absence of historic failure trace data. Simulated experiments of three real scientific workflows in the Austrian Grid environment show that our algorithm drastically reduces the resource waste compared to conservative task replication and resubmission techniques, while having a comparable execution performance and only a slight decrease in the success probability.

The workflow paradigm is one of the most important programming models for the Grid. The composition of Grid workflows has been widely studied in the Grid community. However, there is still a lack of a general and efficient approach for automatic composition of Grid workflows. In this paper, we present a STRIPS (Stanford Research Institute Problem Solver) based formal definition of the Grid workflow composition problem, followed by a novel graph based algorithm for automatic composition of high quality (portable, fault tolerant and optimized) Grid workflows. Our algorithm searches for semantic descriptions of workflow activities, i.e., Activity Functions (AFs), defined by ontologies and composes them into Grid workflows using AF Data Dependence (ADD) graphs. The composition process consists of three phases: ADD graph creation, workflow extraction, and workflow optimization. The worst case complexity of our algorithm is quadratic in the number of AFs. An extension of our algorithm to compose Grid workflows with branches and loops is also presented. Experimental results illustrate the effectiveness and efficiency of our approach: (i) the measured worst case execution time of our algorithm further proofs the analyzed time complexity; (ii) the composition of the real world meteorology Grid workflow application MeteoAG with our algorithm takes approximate half a second; and (iii) the execution time of the MeteoAG workflow when running on the Austrian Grid is reduced by up to 25% and the speedup is increased by up to 2.24 by applying our workflow optimization techniques.

The functional heterogeneity of computational Grids has highly increased due to inclusion of resources other than dedicated to Grid, like from non-dedicated desktop Grids, on-demand systems and even from P2P systems and mobile Grids. At such a diversified scale, resources exhibit different availability properties mainly due to administrators' policies for resource availability in the Grid, and their failure/unavailability properties. These make resources' availability predictions for optimized resource selection, a challenging problem. Addressing this problem, we characterize resource availability properties against their availability policies to understand their availability behavior and quantify it through availability models. We further exploit the availability/failure properties to make predictions about their availability through pattern recognition and classification. We have achieved, on average, accuracy of more than 90% and 75% in our predictions for resource instance availability and lifetime respectively.

The drift towards new challenges in grid computing, including the utility grid paradigm and service level agreements based on quality-of-service guarantees, implies the need for new, robust, multi-criteria scheduling algorithms that can be applied by the user in an intuitive way. Multiple scheduling criteria addressed by the related grid research include execution time, the cost of running a task on a machine, reliability, and different data quality metrics. The existing bi-criteria scheduling approaches are usually dedicated for certain criterion pairs only that require the user to define one's preferences either as weights assigned to the criteria or as fixed constraints defined for one of the criteria. These requirements are often not feasible for the user and not suited to the specificity of the multi- criteria scheduling problem. We propose a novel requirement specification method based on a sliding constraint, and we model the problem as an extension of the multiple-choice knapsack problem. We propose a general bi-criteria scheduling heuristic called dynamic constraint algorithm (DCA) based on dynamic programming, dedicated to the problem model defined by us. In the experimental study, we show that in most of the problem variants, DCA outperforms two existing algorithms designed for the same problem. It also shows relatively low scheduling times for workflows of medium size.

Today's massively multiplayer online games (MMOGs) can include millions of concurrent players spread across the world. To keep these highly-interactive virtual environments online, a MMOG operator may need to provision tens of thousands of computing resources from various data centers. Faced with large resource demand variability, and with misfit resource renting policies, the current industry practice is to maintain for each game tens of self-owned data centers. In this work we investigate the dynamic resource provisioning from external data centers for MMOG operation. We introduce a novel MMOG workload model that represents the dynamics of both the player population and the player interactions. We evaluate several algorithms, including a novel neural network predictor, for predicting the resource demand. Using trace-based simulation, we evaluate the impact of the data center policies on the resource provisioning efficiency; we show that dynamic provisioning can be much more efficient than its static alternative.

Grid economy models have long been considered as a promising alternative for the classical Grid resource management, due to their dynamic and decentralized nature, and because the financial valuation of resources and services is inherent in any such model. In particular, auction models are widely used in the existing Grid research, as they are easy to implement and are shown to successfully manage resource allocation on the Grid market. The focus on the current work is on workflow scheduling in the Grid resource allocation model based on Continuous Double Auctions (CDA). We analyze different scheduling strategies that can be applied by the user to execute workflows in such an environment, and try to identify the general behavioral patterns that can lead to a fast and cheap workflow execution. In the experimental study, we show that under certain circumstances some benefit can be gained by applying an ldquoaggressiverdquo scheduling strategy.

Scientific workflows are a topic of great interest in the grid community that sees in the workflow model an attractive paradigm for programming distributed wide-area grid infrastructures. Traditionally, the grid workflow execution is approached as a pure best effort scheduling problem that maps the activities onto the grid processors based on appropriate optimization or local matchmaking heuristics such that the overall execution time is minimized. Even though such heuristics often deliver effective results, the execution in dynamic and unpredictable grid environments is prone to severe performance losses that must be understood for minimizing the completion time or for the efficient use of high-performance resources. In this paper, we propose a new systematic approach to help the scientists and middleware developers understand the most severe sources of performance losses that occur when executing scientific workflows in dynamic grid environments. We introduce an ideal model for the lowest execution time that can be achieved by a workflow and explain the difference to the real measured grid execution time based on a hierarchy of performance overheads for grid computing. We describe how to systematically measure and compute the overheads from individual activities to larger workflow regions and adjust well-known parallel processing metrics to the scope of grid computing, including speedup and efficiency. We present a distributed online tool for computing and analyzing the performance overheads in real time based on event correlation techniques and introduce several performance contracts as quality-of-service parameters to be enforced during the workflow execution beyond traditional best effort practices. We illustrate our method through postmortem and online performance analysis of two real-world workflow applications executed in the Austrian grid environment.

The workflow model for composing Grid applications is based on an imperative model of computation prone to programming errors, which is an issue yet to consider in the Grid community. In this paper, we propose a new unconventional model for programming Grid applications based on two programming phases: (1) formal functional specification, written by the application scientist not interested in any Grid-related issues and representing "what" has to be computed; (2) imperative workflow-based coordination, written by the computer scientist which indicates "how" to efficiently execute the specification on the Grid. A correctness checker automatically connects both parts at compile- time and ensures the correct execution of the workflow coordination with respect to the formal specification. We validate our approach for three real-world applications and show experimental results that demonstrate the correctness and scalability of our coordination model.

We present a new distributed performance analysis service of the ASKALON integrated Grid environment for computing runtime overheads of dynamic workflows in realtime based on event correlation techniques. We illustrate a formal method to express precise overhead correlation rules, including several performance contracts as quality of service parameters based on fuzzy logic to be enforced in dynamic environments though rescheduling, various runtime optimisations, and steering techniques. We demonstrate experimental results for two real applications from material chemistry and graphics rendering domains.

Scheduling large-scale applications on the Grid is a fundamental challenge and is critical to application performance and cost. Large-scale applications typically contain a large number of homogeneous and concurrent activities which are main bottlenecks, but open great potentials for optimization. This paper presents a new formulation of the well-known NP-complete problems and two novel algorithms that addresses the problems. The optimization problems are formulated as sequential cooperative games among workflow managers. Experimental results indicate that we have successfully devised and implemented one group of effective, efficient, and feasible approaches. They can produce soultuins of significantly better performance and cost than traditional algorithms. Our algorithms have considerably low time complexity and can assign 1,000,000 activities to 10,000 processors within 0.4 second on one Opteron processor. Moreover, the solutions can be practically performed by workflow managers, and the violation of QoS can be easily detected, which are critical to fault tolerance.

Application execution time prediction is of key importance in making decisions about efficient usage of Grid resources. Grid services lack support of a generic application execution time prediction service due to environment specific solutions provided by the existing prediction techniques. To remedy this, we present a generic and comprehensive system to provide execution time predictions of applications on different Grid-sites. Our system is based on a two layered training phase to minimize the training effort, which is our first main contribution. The training phase is driven by a novel experimental design. We also introduce a mechanism of sharing performance measurements across the Grid, on the basis of soft benchmarks, which is our second contribution. Both of these phases support our prediction engine to serve robust predictions. Experiments from the prototype implementation are shown to demonstrate the effectiveness of our proposed system.

Static scheduling and execution of Grid workflows is prone to severe performance losses due to inaccurate predictions or the dynamic nature of the Grid environment. In this paper we present an online tool for analysing the performance overheads that appear during the real-time execution of workflow applications in Grid environments. We employ event correlation techniques and a distributed superpeer architecture in which each peer correlates local lowlevel activity and middleware events to infer performance overheads related to larger workflow regions at a higher level of abstraction. The rule-based correlation technique provides full extensibility to our approach that requires no source code modification. We demonstrate the functionality of our tool through online performance analysis of a real-world workflow application executed in a Grid environment. We present automatically generated online graphs of correlated events that promptly signal to the end-users the real reasons of run-time performance overheads in their executions.

In this paper we present an extension to devise and implement advance reservation as part of the scheduling and resource management services of the ASKALON Grid application development and runtime environment. The scheduling service has been enhanced to offer a list of resources that can execute a specific task and to negotiatewith the resource manager about resources capable of processing tasks in the shortest possible time. We introduce progressive reservation approach which tries to allocate resources based on a fair-share principle. Experiments are shown that demonstrate the effectiveness of our approach, and that reflect different QoS parameters including performance, predictability, resource usage and resource fairness.

This paper describes a novel approach to fault detection and prediction on the grid based on data mining techniques. Data mining techniques are here applied as a mean to effectively process the significant amount of captured data from grid sites, services, workflows and activities. The paper provides a first approach of proposed techniques in terms of its ability of utilizing relevant information and the fault tolerance requirements. Such approach is one intelligent, distributed framework of fault detection and prediction for anomaly and failed activity by using resource- and workflow-based information. We use fault predictions to improve the performance of the workflow execution by avoiding potential faults of activities

We propose a novel approach for bi-criteria scheduling of scientific workflows on the grid, using dynamic programming to balance the trade-off between the two contradicting criteria. We determine the primary and the secondary criterion, and establish a flexible limit for the primary criterion. We identify different classes of criteria and adjust the solution for different variants of the problem

Abstract Performance engineering of parallel and distributed applications is a complex task that iterates through various phases, ranging from modeling and prediction, to performance measurement, experiment management, data collection, and bottleneck analysis. There is no evidence so far that all of these phases should/can be integrated into a single monolithic tool. Moreover, the emergence of computational Grids as a common single wide‐area platform for high‐performance computing raises the idea to provide tools as interacting Grid services that share resources, support interoperability among different users and tools, and, most importantly, provide omnipresent services over the Grid. We have developed the ASKALON tool set to support performance‐oriented development of parallel and distributed (Grid) applications. ASKALON comprises four tools, coherently integrated into a service‐oriented architecture. SCALEA is a performance instrumentation, measurement, and analysis tool of parallel and distributed applications. ZENTURIO is a general purpose experiment management tool with advanced support for multi‐experiment performance analysis and parameter studies. AKSUM provides semi‐automatic high‐level performance bottleneck detection through a special‐purpose performance property specification language. The PerformanceProphet enables the user to model and predict the performance of parallel applications at the early stages of development. In this paper we describe the overall architecture of the ASKALON tool set and outline the basic functionality of the four constituent tools. The structure of each tool is based on the composition and sharing of remote Grid services, thus enabling tool interoperability. In addition, a data repository allows the tools to share the common application performance and output data that have been derived by the individual tools. A service repository is used to store common portable Grid service implementations. A general‐purpose Factory service is employed to create service instances on arbitrary remote Grid sites. Discovering and dynamically binding to existing remote services is achieved through registry services. The ASKALON visualization diagrams support both online and post‐mortem visualization of performance and output data. We demonstrate the usefulness and effectiveness of ASKALON by applying the tools to real‐world applications. Copyright © 2005 John Wiley & Sons, Ltd.

The need to conduct and manage large sets of experiments for scientific applications dramatically increased over the last decade. However, there is still very little tool support for this complex and tedious process. We introduce the ZENTURIO experiment management system for parameter studies, performance analysis, and software testing for cluster and Grid architectures. ZENTURIO uses the ZEN directive-based language to specify arbitrary complex program executions. ZENTURIO is designed as a collection of Grid services that comprise: (1) a registry service which supports registering and locating Grid services; (2) an experiment generator that parses files with ZEN directives and instruments applications for performance analysis and parameter studies; (3) an experiment executor that compiles and controls the execution of experiments on the target machine. A graphical user portal allows the user to control and monitor the experiments and to automatically visualise performance and output data across multiple experiments. ZENTURIO has been implemented based on Java/Jini distributed technology. It supports experiment management on cluster architectures via PBS and on Grid infrastructures through GRAM. We report results of using ZENTURIO for performance analysis of an ocean simulation application and a parameter study of a computational finance code.

This paper describes ZEN, a directive-based language for the specification of arbitrarily complex program executions by varying the problem, system, or machine parameters for parallel and distributed applications. ZEN introduces directives to substitute strings and to insert assignment statements inside arbitrary files, such as program, input, script, or make-files. The programmer thus can invoke experiments for arbitrary value ranges of any problem parameter, including program variables, file names, compiler options, target machines, machine sizes, scheduling strategies, data distributions, etc. The number of experiments can be controlled through ZEN constraint directives. Finally, the programmer may request a large set of performance metrics to be computed for any code region of interest. The scope of ZEN directives can be restricted to arbitrary file or code regions. We implemented a prototype tool for automatic experiment management that is based on ZEN. We report results for the performance analysis of an ocean simulation application and for the parameter study of a computational finance code.