Stream grouping strategy plays an important role in stateful stream computing environments. Many existing grouping strategies overlook various cost factors associated with grouping while balancing stream load. To overcome this limitation, we propose Pd-Stream, a popularity-aware discriminative grouping strategy that identifies the hot keys in dynamic real-time streams and assigns them to instances with high balance and low cost. Our solution includes: (1) A stream application model is constructed, along with a skewed data stream model and a data stream grouping model. Data stream grouping optimization problems are formalized. (2) A hot key probability estimation algorithm is designed, which estimates real-time probabilities of hot keys based on their popularity within the sampling window. (3) An instance assignment algorithm is designed using dynamic routing. This algorithm determines the minimal number of candidate instances based on the probabilities of hot keys, and selects the target instance with the lowest load through a dynamic routing table. Experimental results show that Pd-Stream provides near-optimal load balancing with low memory, achieving load imbalance as low as$10^{-5}$and replication factor as low as 1.74. It outperforms state-of-the-art works, reducing latency by 27%–46% and improving throughput by 23%–52%.

Multi-objective collaborative optimization is essential for improving performance in stream computing systems. However, existing approaches often neglect the interdependencies among communication overhead, load balancing, and energy consumption, and lack predictive capabilities, resulting in delayed scheduling decisions that degrade system latency and throughput. To overcome these limitations, we propose a prediction-driven collaborative framework, named Pc-Stream, which proactively identifies overloaded compute nodes and triggers task migrations in advance. This paper presents this strategy through two key components: (1) A temperature-driven neighborhood adjustment method for task topology partitioning. This method dynamically adjusts the number of migrated tasks based on a predefined temperature. Tasks with high communication volume are batchmigrated to nodes with lower utilization rates during the hightemperature phase, and migrated individually during the lowtemperature phase. (2) A sliding window mechanism that generates multiple sub-sequences for training multiple predictive models. These models enable the system to monitor load trends and proactively migrate tasks from overloaded nodes to those with sufficient resources, thereby reducing communication costs and improving load balance. Experimental results demonstrate that, under dynamic and fluctuating data stream conditions, Pc-Stream significantly enhances overall system performance: reducing average system latency by 49.9 %, and increasing average throughput by 16.9 %.

ABSTRACT The importance of fault tolerance strategies for distributed streaming computing systems becomes more evident due to the increased diversity of failures. Checkpointing is considered a general and efficient method for ensuring fault tolerance. However, determining the checkpoint interval poses a challenge: shorter checkpoint intervals lead to higher overhead, while longer intervals result in extended fault recovery time. Therefore, optimizing the checkpoint interval becomes crucial for the efficient operation of streaming applications. There has been relatively limited exploration and analysis of optimal checkpoint interval settings in the context of stream computing. Many existing works considered adjusting this interval based on a single factor. This article proposes a checkpoint adaptive strategy with high availability, named Ca‐Stream. It considers multiple factors when adjusting checkpoint intervals. Specifically, it addresses the following aspects: (1) Using linear regression to predict the system's fault rate and dynamically adjusting the checkpoint interval based on these predictions. (2) Monitoring CPU time and memory consumption per task to dynamically trigger checkpoints, achieving high reliability, especially in resource‐constrained scenarios. (3) Detecting task execution times on nodes and volume of input data for tasks to identify slow tasks within the cluster. Experiments conducted on a Flink system demonstrate Ca‐Stream's benefits. It reduces checkpoint consumption time by over 38%, system recovery latency by 33%, CPU occupancy by up to 47%, and memory occupancy by 37% compared to Flink's approaches.

Load imbalance can lead to the emergence of stragglers, i.e., join instances that significantly lag behind others in processing data streams. Currently, state-of-the-art solutions are capable of balancing the load between join instances to mitigate stragglers by managing hot keys and random partitioning. However, these solutions rely on either complicated routing strategies or resource-inefficient processing structures, making them susceptible to frequent changes in load between instances. Therefore, we present Ls-Stream, a data stream scheduler that aims to support dynamic workload assignment for join instances to lighten stragglers. This paper outlines our solution from the following aspects: (1) The models for partitioning, communication, matrix, and resource are developed, formalizing problems like imbalanced load between join instances and state migration costs. (2) Ls-Stream employs a two-level routing strategy for workload allocation by combining hash-based and key-based data partitioning, specifying the destination join instances for data tuples. (3) Ls-Stream also constructs a fine-grained model for minimizing the state migration cost. This allows us to make tradeoffs between data transfer overhead and migration benefits. (4) Experimental results demonstrate significant improvements made by Ls-Stream: reducing maximum system latency by 49.3% and increasing maximum throughput by more than 2x compared to existing state-of-the-art works.

Effective task scheduling in stream computing systems can reduce the latency by minimizing inter-node communication. However, this approach often requires restarting tasks to change their deployment locations, resulting in significant system overhead and making it inadequate especially in dynamically changing data stream environments. To address this issue, we propose Ns-Stream, a hierarchical data scheduler that dynamically adjusts data distribution weights between near-source and off-source tasks. Our solution includes: (1) We observe that communication overhead from off-source data processing significantly impacts system latency when tasks' resources are ample. However, as the resources become limited, the computational power required by tasks becomes the key constraint on system performance. (2) During initialization scheduling, we deploy tasks with potential communication to the same node using the graph convolutional network, thus avoiding the need for runtime task scheduling. (3) We dynamically adjust data distribution weights between near-source and off-source tasks based on their computing capabilities, prioritizing local processing of data tuples (within the same worker and node) to optimize resource utilization and reduce data transmission overhead. (4) Experimental results demonstrate significant improvements made by Ns-Stream: reducing maximum system latency by 40% and increasing maximum throughput by 55% compared to existing state-of-the-art works.

In cross-domain stream computing, assigning highly dependent tasks to different domains causes poor performance. Existing methods ignore cross-domain and focus on load balancing and resource allocation. To address this scheduling challenge, this paper proposes a task dependency-aware scheduling strategy named Td-Stream. This strategy is discussed in the following aspects: (1) Impact analysis: Analyzing the adverse impact of communication dependencies between tasks on system performance under traditional scheduling methods in cross-domain environments. (2) Model construction: Constructing models for stream topology, task dependency, resource cost.(3) Cross-domain task allocation: Introducing a cross-domain dependent task allocation method that incorporates a resource elasticity mechanism. Experimental results demonstrate significant improvements made by Td-Stream compared to existing state-of-theart works.

Summary An effective scheduling strategy is critical for achieving better performance in real‐time stream processing systems. How to quickly and efficiently process real‐time data stream is always challenging, especially when clusters are collaborating in a Geo‐Distributed computing environment. To address these challenges, we propose an elastic scheduling strategy with Latency Constraints in Geo‐Distributed stream computing environments called Lc‐Stream. This article discusses our work from the following aspects: (1) An optimized data stream redirection method that is proposed based on queuing network algorithm, along with a computing resource model, a latency constrained scheduling model and a communication energy consumption model. (2) An updated node selection method based on the inter‐layer task correlation, to reduce the communication latency between groups at the executor granularity. (3) A network cluster distribution for Geo‐Distributed computing environment to ensure energy saving under low transmission latency. Experimental results show that compared to R‐Storm, Lc‐Stream reduces total latency by over 19% and increases throughput by over 37% in typical cross‐domain multi‐task topologies. Compared to Ts‐Stream, Lc‐Stream also reduces total latency by over 15% and increases throughput by over 21%. At the same time, it helps to balance the load among the systems and avoid overuse of compute nodes.

Elastic scaling of parallel operators has emerged as a powerful approach to reduce response time in stream applications with fluctuating inputs. Many state-of-the-art works focus on stateless operators and change the operator parallelism from one aspect. They often lack efficient management of operator states and overlook the costs associated with resource over-provisioning. To overcome these limitations, we introduce Es-Stream for elastic scaling of stateful operators over fluctuating data streams, which includes: 1) We observe that under-provisioning of operator parallelism leads to data pile-up, resulting in longer system latency, while over-provisioning of operator parallelism causes idle instances and additional resource consumption. 2) The Es-Stream system scales in two dimensions: the parallelism of operators and the number of resources. It dynamically adjusts operators to an optimal parallelism while scaling the resources used by the stream application. 3) When the parallelism of stateful operators changes, upstream operators backup downstream operators’ state and cache the emitted data tuples at dynamic time intervals, ensuring the operator parallelism is adjusted in a low-overhead way. 4) Experimental results demonstrate that Es-Stream provides promising performance improvements, reducing the maximum system latency by 3x and saving the maximum state recovery time by 2x, compared to existing state-of-the-art works.

Current optimization for stateful flow processing computation tends to focus on load balancing without considering the utilization of downstream instance resources. To address this issue, we propose a data stream grouping method called Fa-Stream, specifically designed for stateful operators and incorporating field values frequency-awareness. Fa-Stream is implemented in three main aspects: (1) A data stream grouping model is built using Count-Min Sketch and Gated Recurrent Unit (GRU) to predict and analyze the frequency of field values. It selectively chooses high-frequency field values, and the communication distance model and instance resource constraint model are designed to adjust the weights of downstream instances for high-frequency field values. (2) A cyclic access routing table is generated, and weights are dynamically adjusted by a rebalancing scheme to avoid load skewness. Consistent hash grouping is implemented for low-frequency field values, and dual mapping is used to prevent large-scale migration caused by scaling. To validate the effectiveness of Fa-Stream, comparative experiments between Partial Key Grouping (PKG) and Fa-Stream are conducted using the Storm platform. Results demonstrate that Fa-Stream improves tuple throughput by 12.3%, reduces system delay by 14.2%, and increases load balancing degree by 42.8%. Furthermore, fa-Stream exhibits efficiency and stability across different data skews and tuple input rates.

Efficient utilization of computing resources in a stream computing environment is crucial for system performance. Existing scheduling strategies can hardly guarantee latency under performance constraints, let alone accounting for the communication cost incurred by scheduling itself. To address these issues, we propose Lg-Stream, a latency guaranteed scheduling strategy under performance constraints. This paper discusses the Lg-Stream strategy from the following aspects: (1) We model the topology as a queuing network to evaluate the system latency; (2) For scenarios with limited resources and latency constraints, we ensure that each executor-to-component allocation optimizes the system's processing latency to the maximum, consequently altering the components’ parallelism; (3) We place executors that communicate with each other on the same node as much as possible. Experimental results demonstrated that in comparison to existing state-of-the-art scheduling strategies, it reduces the average system latency up to 30%.

Abstract Rapid increase in energy consumption is a serious problem in cloud storage systems. Data accessed in large‐scale storage systems usually exhibit temporal and spatial characteristics, which make it possible to reduce energy consumption by clustering data with similar access characteristics for storage in the same zone of cloud storage systems. Existing works usually only focus on the frequency of data access. However, widely existing phenomena show data access with seasonal and tidal characteristics in cloud storage systems. The seasonal and tidal characteristics of data access are extracted thoroughly in this paper. According to the extracted data access characteristics, energy‐aware data clustering through a machine learning algorithm (K‐ear) is proposed. K‐ear classifies data into five seasonal categories according to their seasonal access characteristics and then classifies every seasonal category into three tidal categories according to its tidal access characteristics. The 15 classified categories are stored in different storage zones with different energy and performance modes. Simulation experiments using CloudSimDisk with the constructed mathematic models demonstrate that the proposed K‐ear algorithm is more energy‐efficient than the default data clustering algorithms in Hadoop and the classical data clustering storage strategy according to the data access frequency (Striping‐Based Energy‐Aware Strategy).

With the increasing popularity of location technologies and location-based service applications, a large number of user's trajectory data have been collected. Publishing the real-time statistics data of trajectory streams can be useful in many fields such as intelligent transportation system, urban road planning and road congestion detection. As the trajectory data itself contains a wealth of user's privacy information, the privacy leakage problem has aggravated the risk of data publishing. In order to realize the personalized and uniform privacy preserving of user's trajectory data, the differential privacy model based on data perturbation is introduced, and a privacy preserving algorithm is proposed. The algorithm contains three modules of dynamic privacy budget allocation, privacy approximation and privacy publishing. In the experiment, the performances of the proposed method are verified by using real-life datasets.

This article describes how with the continuous expansion on the volume of data produced by sensors in Cyber Physical Systems, the scale of the cloud storage system has become larger. This will lead to the problems of a high energy consumption rate and a low utilization becoming a serious issue. In order to enhance the effective energy consumption, reduce the invalid energy consumption, and supply more flexible QoS for users in CPS, this article proposes an automatic energy gear-shifting mechanism with flexible QoS constraints (QGLG). The QGLG predicts system load of the follow-up period through a support vector machine model. According to the current system load, the predicted load, and the flexible QoS, QGLG automatically up-shifts and down-shifts among nodes. Substantive results from the simulation experiments done on GridSim show that the QGLG can achieve energy consumption reduction while satisfying the user's flexible QoS requirements. Compared with a similar energy-reducing mechanism, QGLG has its obvious advantage when considering the requirements of user with energy saved notwithstanding.

One of the issues for tour planning applications is to adaptively provide personalized advices for different types of tourists and tour activities. This paper proposes a high level Petri Nets based approach to providing some level of adaptation by implementing adaptive navigation in a tour node space. The new model supports dynamic reordering or removal of tour nodes along a tour path; it supports multiple travel modes and incorporates multimodality within its tour planning logic to derive adaptive tour. Examples are given to demonstrate how to realize adaptive interfaces and personalization. Future directions are also discussed at the end of this paper.