Modern data processing workloads often have highly unpredictable end-to-end latency characteristics that are caused by heterogeneity, time-variation, and parallelized processing. The increase in unpredictability is in part attributable to job "straggling", and is symptomatic of a new class of stochastic scheduling challenges that will degrade the performance of current and future applications at scale. While the job scheduling literature for data processing frameworks is rich with ideas; there is little coordination between research groups on methodology and presentation, stunting the ability for designers to survey a collection of results and draw generalizable conclusions about good design patterns. We introduce an abstract system model for data processing on warehouse scale machines that aids in eliminating ambiguity in the job scheduling research by categorizing schedulers based on where they act on the system in the job processing data pathway. Furthermore, we demonstrate that although the scheduling problem is NP-Hard in the general case, it is still possible to derive scheduler design principles using bounds and asymptotics.

Public conversations on Twitter comprise many pertinent topics including disasters, protests, politics, propaganda, sports, climate change, epidemics/pandemic outbreaks, etc., that can have both regional and global aspects. Spatial discourse analysis rely on geographical data. However, today less than 1% of tweets are geotagged; in both cases—point location or bounding place information. A major issue with tweets is that Twitter users can be at location A and exchange conversations specific to location B, which we call the Location A/B problem. The problem is considered solved if location entities can be classified as either origin locations (Location As) or non-origin locations (Location Bs). In this work, we propose a simple yet effective framework—the True Origin Model—to address the problem that uses machine-level natural language understanding to identify tweets that conceivably contain their origin location information. The model achieves promising accuracy at country (80%), state (67%), city (58%), county (56%) and district (64%) levels with support from a Location Extraction Model as basic as the CoNLL-2003-based RoBERTa. We employ a tweet contexualizer (locBERT) which is one of the core components of the proposed model, to investigate multiple tweets’ distributions for understanding Twitter users’ tweeting behavior in terms of mentioning origin and non-origin locations. We also highlight a major concern with the currently regarded gold standard test set (ground truth) methodology, introduce a new data set, and identify further research avenues for advancing the area.

High performance distributed systems such as distributed stream processing systems and message-passing parallel programs are often deployed on platforms that make use of vanilla TCP/IP communication, which in turn uses the conventional Nagle’s algorithm for congestion control. Recent research in Reinforcement Learning (RL) techniques to either replace or control the conventional TCP approach shows promise in achieving a greater degree of performance, especially when the demand for network resources in a multi-tenant platform is highly dynamic and infeasible to model. Existing results are, however, focused on RL for general Internet communication, with the learning objective being some combination of throughput, loss, and latency, and predominately use a continuous action space to adjust the packet rate at the sender. In this work, we propose a coefficient-free RL objective that perfectly matches the data transmission rate to the underlying communication system’s bottleneck, which naturally deters packet loss and thereby converges to the ideal throughput even in lock-free and latency-constrained Big Data applications where packets are dropped due to load shedding or exceeding latency thresholds. Our results compare favorably to other state-of-the-art objective functions using an RL framework, e.g., providing up to 48% reduction in packet loss while obtaining up to a 4% increase in overall throughput when packet latency is highly constrained.