As the scale of data continues to grow exponentially, managing resource allocation and energy consumption in big data systems becomes increasingly complex and critical. Moreover, with big data systems, energy efficiency is more important daily. In cloud environments, it can be the determining factor between reduced costs and lowered environmental damage. This paper presents a deep learning-based framework for accurately predicting instant energy consumption in real-time and detecting anomalies of different sizes in big data clusters. We use SmartMonit to gather task execution and real-time infrastructure data. A Feedforward Neural Network (FNN) predicts energy consumption from CPU utilisation, memory usage, and task profiling research. The system will track any deviation from predicted consumption with root cause analysis (RCA) if there are significant anomalies. We also integrate an Autoencoder to identify straggler tasks and inefficient resource utilisation. Userdefined functions are next applied to examine these anomalies and try to detect the underlying reasons, like distributed data processing, locality of computation exploitation, or resource waste. Given the scale and heterogeneity of big data workloads, the system's ability to dynamically adjust and optimise resource usage is essential for handling complex processing tasks. The experimental results prove that the proposed system effectively enhances resource allocation and decreases wasted energy.

Abstract As artificial intelligence (AI) continues to advance, Generative AI emerges as a transformative force, capable of generating novel content and revolutionizing anomaly detection methodologies. This paper presents CloudGEN, a pioneering approach to anomaly detection in cloud environments by leveraging the potential of Generative Adversarial Networks (GANs) and Convolutional Neural Network (CNN). Our research focuses on developing a state‐of‐the‐art Generative AI‐based anomaly detection system, integrating GANs, deep learning techniques, and adversarial training. We explore unsupervised generative modelling, multi‐modal architectures, and transfer learning to enhance expert systems' anomaly detection systems. We illustrate our approach by dissecting anomalies regarding job performance, network behaviour, and resource utilization in cloud computing environments. The experimental results underscore a notable surge in anomaly detection accuracy with significant development of approximately 11%.

Integrating advanced artificial intelligence (AI), the Internet of Things (IoT) and cutting-edge cloud computing epitomizes the transformative potential of Industry 5.0 technologies, enabling unprecedented automation and efficiency. However, this technological surge also brings serious environmental challenges, significantly increasing energy consumption and carbon emissions. This article introduces EcoCloud, a robust task scheduling mechanism based on ant colony optimization (ACO) principles that aims to improve energy and carbon efficiency in Industrial IoT-enabled cloud environments. EcoCloud dynamically schedules MapReduce jobs on Hadoop clusters in IoT-based cloud systems by leveraging real-time resource consumption metrics through a comprehensive energy model deployed via a multilayer perceptron (MLP) neural network. As a result, the model accurately predicts power consumption and distributes workloads to underutilized nodes to optimize energy usage and reduce carbon emissions. Extensive evaluations show that EcoCloud significantly outperforms traditional scheduling methods, improving energy consumption and overall system performance.

Drones are sophisticated machines that can hover over extreme locations, conduct aerial surveillance, collect surveillance data, and disseminate it to the distributed edge for processing and analysis. The distributed edge deploys advanced artificial intelligence (AI) models to detect any unwarranted activity or object based on surveillance data. However, these lightweight and low-power unmanned aerial vehicles (UAVs) may experience faults due to unprecedented workload when deployed in extreme surveillance domains. In this article, we have designed an AI framework to detect any safety concerns with drones deployed for aerial surveillance in extreme locations based on real-time drone critical parameters. We also propose a MapReduce-based object recognition and classification module to process large-scale images captured by drones efficiently. However, conventional AI systems behave like black box systems, leading to a lack of trust and transparency. Thus, we convert the traditional framework of AI into an explainable edge AI framework using Shapley additive explanations (SHAPs) that opens Pandora’s black box. The experimental results show the effectiveness of the proposed framework in detecting drone safety concerns through explainable health status tracking alongside ensuring an effective object detection mechanism.

In drone swarms, where multiple drones collaborate closely to achieve shared objectives within constrained spatial domains, the intricacies of these interrelated actions can lead to potential issues. Despite rigorous pre-deployment planning, the inherent probability of complications persists. These compli-cations stem from onboard computational resources, hardware failures, and network communication disruptions. While the malfunction of an individual drone may seem inconsequential, it can escalate into a substantial predicament when it disrupts the seamless coordination of the entire swarm. Therefore, the need to proactively monitor drones for predictive failure analysis and the subsequent examination of failed drones to mitigate future occurrences becomes imperative. This paper introduces a comprehensive framework for systematically collecting and processing data within drone swarms. The framework gathers critical information about onboard characteristics and commu-nication metrics. These data points are subjected to advanced analysis using Complex Bayesian Networks to probabilistically uncover complex and hidden relationships between random features. The results demonstrate exceptional accuracy, with influences ranging from 99 % to 79 %, that ensures the reliability and effectiveness of the predictive capabilities in enhancing drone safety and network performance.

With the increasing number of Internet of Things (IoT) devices being deployed and used in daily life, the load on computational devices has grown exponentially. This situation is more prevalent in smart cities where such devices are used for autonomous control and monitoring. Smart cities have different kinds of applications that are aided through IoT devices that collect data, send it to computational processing and storage devices, and get back decisions or actuate the actions based on the input data. There has been a stringent requirement to reduce the end-to-end delay in this process owing to the remote deployment of cloud data centres. This eventually led to the revolution of edge computing, wherein nano–micro-processing devices can be deployed closer to the premises of the smart application and process the data generated with a lower turnaround time. However, due to the limited computational power and storage, controlling the workload diverted to the edge devices has been challenging. The workload scheduling policies and task allocation schemes often fail to consider the run time health of the edge devices due to a lack of proper monitoring infrastructure. Thus, in this article, we proposed a health monitoring and diagnosis framework for geo-distributed edge clusters processing big data generated by smart city applications. This framework is built over the Map-Reduce approach for distributed processing of big data on edge clusters deployed across the smart city. Within this framework, SmartMonit (a monitoring agent) is deployed that collects the health statistics of edge devices and predicts the potential failures using an artificial neural network-based self-organising maps approach. The proposed framework is deployed over different clusters to test the efficacy concerning failure detection.

Abstract Twitter produces a massive amount of data due to its popularity that is one of the reasons underlying big data problems. One of those problems is the classification of tweets due to use of sophisticated and complex language, which makes the current tools insufficient. We present our framework HTwitt, built on top of the Hadoop ecosystem, which consists of a MapReduce algorithm and a set of machine learning techniques embedded within a big data analytics platform to efficiently address the following problems: (1) traditional data processing techniques are inadequate to handle big data; (2) data preprocessing needs substantial manual effort; (3) domain knowledge is required before the classification; (4) semantic explanation is ignored. In this work, these challenges are overcome by using different algorithms combined with a Naïve Bayes classifier to ensure reliability and highly precise recommendations in virtualization and cloud environments. These features make HTwitt different from others in terms of having an effective and practical design for text classification in big data analytics. The main contribution of the paper is to propose a framework for building landslide early warning systems by pinpointing useful tweets and visualizing them along with the processed information. We demonstrate the results of the experiments which quantify the levels of overfitting in the training stage of the model using different sizes of real-world datasets in machine learning phases. Our results demonstrate that the proposed system provides high-quality results with a score of nearly 95% and meets the requirement of a Hadoop-based classification system.

Big data processing systems, such as Hadoop and Spark, usually work in large-scale, highly-concurrent, and multi-tenant environments that can easily cause hardware and software malfunctions or failures, thereby leading to performance degradation. Several systems and methods exist to detect big data processing systems’ performance degradation, perform root-cause analysis, and even overcome the issues causing such degradation. However, these solutions focus on specific problems such as stragglers and inefficient resource utilization. There is a lack of a generic and extensible framework to support the real-time diagnosis of big data systems. In this article, we propose, develop and validate AutoDiagn. This generic and flexible framework provides holistic monitoring of a big data system while detecting performance degradation and enabling root-cause analysis. We present an implementation and evaluation of AutoDiagn that interacts with a Hadoop cluster deployed on a public cloud and tested with real-world benchmark applications. Experimental results show that AutoDiagn can offer a high accuracy root-cause analysis framework, at the same time as offering a small resource footprint, high throughput, and low latency.

Internet of Things (IoT)-based Healthcare services, which are becoming more widespread today, continuously generate huge amounts of data which is often called big data. Due to the magnitude and intricacy of the data, it is difficult to find valuable information that can be used for decision-making and prediction. Big data systems take on a significant infrastructure service to better serve the purpose of IoT systems and support critical decision making. On the other hand, privacy preservation, data integrity, and identity verification are essential requirements in healthcare big data service management. To overcome these problems, this article offers a scalable computing system that provides verifiable data access mechanism for IoT-enabled health data analytics in the big data ecosystem. There are two primary sub-architectures in the proposed architecture, namely a big data analytics tracking system and a derived blockchain-based data storage/access system. This approach leverages big data systems and blockchain architecture to analyze, and securely store data from IoT-enabled devices and allow verified access to the stored data. The zero-knowledge protocol is used to ensure that no information is accessible to unauthenticated users alongside avoiding data linkability. The results demonstrate the effectiveness of the our method to solve the problems of big data analytics and privacy issues in healthcare.

Modern big data processing systems are becoming very complex in terms of large-scale, high-concurrency and multiple talents. Thus, many failures and performance reductions only happen at run-time and are very difficult to capture. Moreover, some issues may only be triggered when some components are executed. To analyze the root cause of these types of issues, we have to capture the dependencies of each component in real-time. In this paper, we propose SmartMonit, a real-time big data monitoring system, which collects infrastructure information such as the process status of each task. At the same time, we develop a real-time stream processing framework to analyze the coordination among the tasks and the infrastructures. This coordination information is essential for troubleshooting the reasons for failures and performance reduction, especially the ones propagated from other causes.