The rise of Internet of Things (IoT) applications has led to massive data generation. However, dealing with such massive data is challenging. Nowadays, data pipelines are popular mechanisms used to properly deal with data operations at scale in the IoT continuum. Serverless Data Pipelines (SDP) is one such approach to performing event-driven data analysis on data streams. Data pipelines are composed of many components, and scaling the entire pipeline without leaving any bottlenecks is challenging. This study aims to assess the performance of scaling mechanisms in handling stochastic workloads efficiently and understanding critical resource utilization in fog environments. We applied workload-based techniques (Request per Second, Queue Length, Message Rate) and resource-based scaling (CPU) on SDP components of two IoT applications: Aeneas (long-running functions) and PuhatuMonitoring (short-running functions). Using Azure serverless workload patterns, we compared scaling approaches in real-time fog environments, evaluating QoS metrics like processing time and CPU utilization. Our analysis of suitability, using the weighted average scoring method on two QoS metrics, revealed that for compute-intensive tasks, the resource-based scaling approach works effectively for jump, steady, spike, and fluctuation workloads. For short execution time tasks, workload-based scaling suits all four workloads.

Intelligent task placement and management of tasks in large-scale fog platforms is challenging due to the highly volatile nature of modern workload applications and sensitive user requirements of low energy consumption and response time. Container orchestration platforms have emerged to alleviate this problem with prior art either using heuristics to quickly reach scheduling decisions or AI driven methods like reinforcement learning and evolutionary approaches to adapt to dynamic scenarios. The former often fail to quickly adapt in highly dynamic environments, whereas the latter have run-times that are slow enough to negatively impact response time. Therefore, there is a need for scheduling policies that are both reactive to work efficiently in volatile environments and have low scheduling overheads. To achieve this, we propose a Gradient Based Optimization Strategy using Back-propagation of gradients with respect to Input (GOBI). Further, we leverage the accuracy of predictive digital-twin models and simulation capabilities by developing a Coupled Simulation and Container Orchestration Framework (COSCO). Using this, we create a hybrid simulation driven decision approach, GOBI*, to optimize Quality of Service (QoS) parameters. Co-simulation and the back-propagation approaches allow these methods to adapt quickly in volatile environments. Experiments conducted using real-world data on fog applications using the GOBI and GOBI* methods, show a significant improvement in terms of energy consumption, response time, Service Level Objective and scheduling time by up to 15, 40, 4, and 82 percent respectively when compared to the state-of-the-art algorithms.