Abstract Many computing workloads in big data and machine learning applications are structured as directed acyclic graphs (DAG) and deployed on PC clusters for parallel execution using multiple physical or virtual machines. The scheduling of such workloads is critical to the application performance such as execution time and a plethora of techniques have been developed, taking into account various aspects such as data locality, network bandwidth, and server capability. We formulate DAG-structured workload scheduling as a nonlinear integer programming (NIP) problem and prove it to be NP-complete. Our empirical study reveals a positive correlation between scheduling plan distance (SPD) and finish time gap (FTG), and based on this finding, we propose a running time gap strategy (RTGS) to tackle this scheduling problem in multiprocessor environments. RTGS follows the main optimization strategy in the family of whale optimization algorithm (WOA). We derive a new function and use a greedy algorithm to generate an effective scheduling plan in RTGS. Extensive experiments with real production traces from Alibaba on simulation environments and realistic Hadoop environments show that our approach significantly improves the stability of WOA when applied to the scheduling problem of DAG-structured workloads, and also reduces the workload completion time by up to $$93\%$$ 93 % in comparison with seven state-of-the-art baseline algorithms.

The rapidly increasing demand for big data processing has necessitated the development of advanced scheduling policies that can effectively accommodate urgent job requirements. This paper presents the Urgent Job Priority Scheduler (UJPS) for Hadoop YARN, aimed at handling urgent jobs efficiently in big data processing. UJPS uses an Aging model to cut waiting times and prevent job starvation, a Dynamic Priority model for urgency-based prioritization, and a Container Load model to boost data locality and efficiency. Tested on Hadoop with benchmark tasks, UJPS outperforms five advanced schedulers, lowering waiting times by up to 81.42% and reducing job runtime by 32.90%. It prioritizes urgent tasks while ensuring overall efficiency, offering benefits to organizations using Hadoop YARN for timely job execution.

In Hadoop’s big data processing systems, YARN is responsible for resource management and job scheduling. The built-in job scheduling algorithms in YARN are simple to execute, but have some limitations such as job starvation, excessive server load, and load imbalance. In this paper, we propose a new Hybrid Dynamic Priority job Scheduling algorithm (HDPS) to address these limitations. HDPS dynamically adjusts the priority of a job as its waiting time increases to prevent job starvation. It also features a task assignment strategy designed specifically to address data locality by considering the available resources of servers and the distribution of data blocks stored on servers to reduce data transfer time and improve job execution efficiency. We implement and integrate HDPS into YARN and conduct experiments in a real Hadoop system using built-in benchmark test cases of Hadoop. Experimental results show that HDPS exhibits comprehensive superior performance over existing algorithms in terms of execution efficiency and load balance.

In this paper, a new sparse array structure, which further improves the degrees of freedom (DOFs) and enhanced the DOA estimation performance, for the fourth-order cumulant based direction of arrival (DOA) estimation is proposed. The new-formed array is hole-free and can achieve a large consecutive range in its fourth-order difference coarray. By analyzing its second-order sum coarray and fourth-order difference coarray, the closed form expression for the physical sensor locations and the corresponding virtual sensor configurations are derived. Compared with the existing fourth-order based sparse array structures, such as FLNA and SAFOE-NA, when the number of sensors is less than 23, the proposed sparse array can obtain longer consecutive virtual array, leading to more detected sources with a higher accuracy. Numerical simulations are performed to verify the superiorities of the proposed sparse array for fourth-order cumulant based DOA estimation.

Clustering offers a general methodology and comes with a remarkably rich conceptual and algorithmic framework for data analysis and data interpretation. As one of the most representative algorithms of fuzzy clustering, fuzzy C-means (FCM) is a widely used objective function-based clustering method exploited in various applications. In this study, a virtual-based fuzzy clustering algorithm is proposed to improve the classification performance coming as a result of using fuzzy clustering. This improvement is achieved by forming a virtual space based on the original data space. First, we construct a piecewise linear transformation function to modify the similarity matrix of the original data and build the so-called virtual similarity matrix (VSM). Considering the VSM, the effect of closeness becomes amplified; in other words, high similarity values (say, larger than α which is a cutoff value of the large and small similarity in this paper) present in the original similarity matrix are made higher, whereas lower similarity levels (say, smaller than α) are further reduced. In addition, data with high similarity (say, larger than a certain threshold value) observed in the original space will overlap (the attributes of the samples are exactly the same) significantly in the virtual space; the overlapping samples can be treated as one sample. This modification makes possible easier to identify clusters. Second, we build a relationship matrix between the original dataset and the determined similarity values and present two closed-form solutions to the problem of building the relationship matrix. Subsequently, a virtual space of the original data space is derived through the modified similarity matrix and the introduced relationship matrix. We offer a thorough analysis behind the developed clustering algorithm. The experimental results are in agreement with the underlying conceptual basis. Furthermore, the resulting classification performance is significantly improved compared with the results produced by the FCM and the kernel-based fuzzy C-means.

Active antenna system (AAS) is a promising technology to boost the capacity of next generation wireless communication systems. As a key feature of AAS, vertical sectorization can help form new sub-sectors vertically in a conventional macro cell, facilitates reusing the frequency resources for multiple users, and thus has the potential to improve the system peroformance. In this paper, we investigate the performance of vertical sectorization by optimizing the antenna downtilt and transmit power in LTE-A downlink systems. We first derive the achievable data rate of a downlink wireless communication system considering vertical sectorization and then formulate the problem based on the derived data rate. Finally, antenna downtilt and transmit power are optimized to improve the performance of vertical sectorization. The simulation results demonstrate the effectiveness of the proposed algorithm.

This poster introduces JRD, a novel device-free localization system which can achieve high accuracy with low cost and little human effort, and is even robust to different scenarios. Unlike the previous Radio Signal Strength (RSS)-based systems which depend on the dense deployment to provide high accuracy, JRD extracts the fine-grained RSS distributions of a single link and presents a voting algorithm based on multi-link to identify the object location accurately while maintaining a low-cost deployment. Furthermore, JRD is flexible to different scenarios by using the transferring technique with less time-consuming and human effort. Experimental results show that JRD can improve the localization accuracy by up to 50% with less cost as compared with the existing RSS approaches.

This poster introduces MoveLoc, a system that locates a moving object without carrying any devices from detecting Doppler shifts reflected off the moving object. It works even if the object walking or running in different directions without any training. MoveLoc does not require the user to carry any communication devices, yet its accuracy exceeds current moving object localization systems using Radio Signal Strength(RSS). We implement the system and evaluate the performance on moving person by experiments. Experimental result shows that MoveLoc achieves an average location accuracy of 0.69 meters and reduces the equipment deployment density compared with other known wireless moving object localization approaches using RSS.