As increasingly complex functions are implemented in applications, directed acyclic graphs (DAGs) are widely used to model the inter-dependencies between individual functions. Cloud-based data processing platforms need to consider the complex topology of DAGs and arbitrary deadlines given by users for job scheduling, leading to an NP-hard decision-making problem. Leveraging spot instances in data processing platforms can achieve significant cost savings, but the unpredictable interruption of spot instances makes the problem of VM scaling and job scheduling more difficult. In this paper, a Reinforcement Learning (RL) based approach called SpotDAG is proposed to solve the auto-scaling problem for jobs modeled as DAGs on a data processing platform where spot instances are introduced. SpotDAG makes cluster scaling and job scheduling decisions at the same time by mapping its output to several meta-policies. This paper introduces the self-attention mechanism for feature extraction to help the intelligent agent learn faster. A mask layer after the output of the proposed RL-based algorithm circumvents illegal actions to ensure that a job is completed by its deadline. Extensive experimental results show that the proposed approach can significantly reduce the cost of instances for data processing platforms while ensuring that jobs are completed in time.

Nowadays, the improvement of the data acquisition capability of the Industrial Internet of Things (IIoT) systems has brought about higher data throughput. Users can deploy industrial big data analysis applications on cloud platforms in a pay-as-you-go way to deal with the unstable data generation and data analysis workloads in the IIoT scenario. To procure stable and flexible computing resources, as a traditional pay-as-you-go cloud instance procurement option, on-demand instances are widely used, but their expensive prices also significantly increase users’ cost burden. To reduce the data analysis cost, in this article, we establish a per-job cost-effective framework adopting spot instances, on-demand instances, and cloud storage for industrial big data analysis applications. In our framework, we take advantage of spot instances, which are computing instances provided at low prices under a pay-as-you-go model, to achieve cost savings. However, using spot instances carries the risk of being interrupted. Therefore, we propose to use a checkpointing mechanism to back up intermediate results to cloud storage to reduce the potential loss caused by spot instance interruptions. Considering the time sensitivity of industrial big data analysis applications, we use on-demand instances as alternative computing resources after spot instance interruptions to ensure that users’ jobs can be completed without high time latencies. Evaluation results show that our framework can achieve cost savings as well as minimize time latencies for users’ jobs.