LLM serving is increasingly dominated by decode attention, which is a memory-bound operation due to massive KV cache loading from global memory. Meanwhile, real-world workloads exhibit substantial, hierarchical shared prefixes across requests (e.g., system prompts, tools/templates, RAG). Existing attention implementations fail to fully exploit prefix sharing: one-query-per-CTA execution repeatedly loads shared prefix KV cache, while one-size-fits-all tiling leaves on-chip resources idle and exacerbates bubbles for uneven KV lengths. These choices amplify memory bandwidth pressure and stall memory-bound decode attention.

Modern deep neural network (DNN) and large language model (LLM) applications integrate multiple models into inference pipelines with stringent latency requirements for customized tasks. To mitigate extensive request timeouts caused by accumulation, systems for inference pipelines commonly drop a subset of requests so the remaining ones can satisfy latency constraints. Since it is commonly believed that request dropping adversely affects goodput, existing systems only drop requests when they have to, which we call reactive dropping. However, this reactive policy can not maintain high goodput, as it neither makes timely dropping decisions nor identifies the proper set of requests to drop, leading to issues of dropping requests too late or dropping the wrong set of requests.

The hardware scheduler on NVIDIA GPUs is highly inefficient in utilizing micro-architectural hardware resources. It places blocks from the same kernel within the same GPU Streaming Multiprocessor (SM) core, resulting in a stacking colocating problem, where identical blocks are placed within the same SM core, saturating only a subset of intra-SM hardware resources while leaving others underutilized. The primary challenge in addressing this issue is that the NVIDIA hardware is closed-source, preventing us from directly modifying the hardware scheduler. To bridge the semantic gap between the resource demands of kernels and the scheduler, we introduce µ Share, which enables intra-SM scattered colocating of kernels through a non-intrusive half-plus blocksize shaping method. It shapes the blocksize of kernels to a halfplus blocksize (i.e., slightly more than half of the SM's thread capacity), scattering identical blocks of the same kernel across different SMs. It further adopts a time-shifted launching method to reduce intra-SM resource contention. Compared to state-of-the-art systems, µ Share does not require intrusive modifications to hardware or kernel code, yet it can still improve inference throughput by 26.90%-54.09% and increases low-level hardware utilization by 38.53%-61.15%.

Function-as-a-Service (FaaS) is becoming increasingly prevalent, and the number of daily service requests in our system ByteDance has even reached 49.6 billion. These requests exhibit “3H” characteristics:high concurrency,high burstiness, andhigh fluctuation, causing the instance scheduling system to face severe bottlenecks even earlier than the data plane.In this paper, inspired by the idea of on-demand scaling of FaaS applications, we proposeGalaxy, a highly scalable instance scheduling system for FaaS platforms.Galaxydecouples the instance scheduling system into functions and introduces a top controller to dynamically adjust the number of instances for each function based on the request load, effectively eliminating bottlenecks under high concurrent requests. We evaluateGalaxywith real-world workloads and the experimental results show thatGalaxycan complete the scaling of over 89,935 instances within 1 second, excluding cold start overhead.Galaxyincreases the scheduling throughput by 8.6 ~ 12.7× compared to the existing control plane.

Function caching is one of the fundamental techniques in FaaS platforms to alleviate coldstart overhead. However, as cache instances consume significant cloud resources (e.g., memory), it is challenging to balance function performance and cache cost. Current systems use simple and rudderless cache polices with a “local cache control” design, which ignores function characteristics such as workload skewness from hot functions and results in either cache contentions or cache resource waste.In this paper, inspired by software-defined networks, we proposeFlame, an efficient cache system to manage cached functions with hotspot-aware instance scheduling and cache allocation. It consists of a two-layer design. Firstly, by decoupling the cache control plane from worker nodes and introducing a centralized cache controller,Flamecan schedule functions from a global view of the cluster’s status, thereby reducing inter-node workload skew. Second,Flamedivides the prior monolithic cache pool within each node into multiple partitions and dynamically assigns them to different hot functions, thereby further mitigating intra-node cache contention. Experimental results from realworld workloads show thatFlamecan reduce cache resource usage by 36% on average while improving function performance by nearly 7× compared to the state-of-the-art method.

Website Fingerprinting (WF) attack can be mitigated throughrandom camouflageorpair camouflage.Random camouflageinserts random dummy packets into the traces according to pre-defined rules. It can be compromised easily by machine learning-based WF attacks.Pair camouflageobfuscates the distinguishing features of paired websites by inserting elaborated perturbations into raw traces, thereby misleading the attacker. It is costly in maintaining a perturbation generator for each pair of websites. Based on these insights, we proposeStinger, a novel data poisoning based WF defense, which enables effective defense against WF attacks with low bandwidth overhead and only maintains one generator for all websites.Stingerexploits the idea of poisoning by contaminating the model directly in such a way that the WF attacks only classify based on the inserted poison sequences, thus being low overhead and website independent. We experimentally evaluateStingerusing the DF and AWF datasets. The results show that Stinger improves the successful defending rate by an average of 20.37% and 22.83% while reducing overhead by 85.88% and 81.35%, respectively.

Video large language models (VideoLLMs) have significantly advanced video question answering (VideoQA) applications, which demand both low latency and high accuracy. To meet the requirements, VideoLLMs are typically deployed on GPUs for parallel acceleration. However, the massive computational load from long video contexts often makes such acceleration insufficient. Existing techniques like token pruning and speculative decoding attempt to address this challenge by altering the computational load, but often fail to balance both speed and accuracy. Sparse self-speculation mitigates these limitations via selecting a subset of tokens on a specific token budget to draft the output and then verify it using all tokens. However, existing sparse self-speculation is designed for text-based scenarios and cannot be directly applied to VideoQA tasks, as it fails to account for discrepancies in critical multimodal tokens and the dynamic nature of optimal token budget in VideoQA, leading to suboptimal scale and inappropriate composition of parallel computational load. We argue that achieving both high accuracy and low latency in VideoQA tasks requires managing the computational load with awareness of these discrepancies and dynamics. To achieve this, we introduce ViDA, a selective sparse self-speculation inference system. It progressively searches token budgets by iteratively refining lower and upper bounds of search space derived from long contexts, aiming to find and allocate varying optimal budgets in real-time adaptively. Additionally, it leverages insights from discrepancies in critical multimodal tokens to perform a discrepancy-aware token selection approach for identifying critical tokens. Evaluations across various VideoQA workloads show that compared to state-of-the-art methods, ViDA preserves exact model outputs while reducing average time-per-outputtoken (TPOT) by 15% to 46% and average end-to-end latency by up to 28%, while decreasing the divergence from optimal token budget distribution by up to 93

Detecting euphemisms is essential for content security on various social media platforms, but existing methods designed for detecting euphemisms are ineffective in impromptu euphemisms. In this work, we make a first attempt to an exploration of impromptu euphemism detection and introduce the Impromptu Cybercrime Euphemisms Detection (ICED) dataset. Moreover, we propose a detection framework tailored to this problem, which employs context augmentation modeling and multi-round iterative training. Our detection framework mainly consists of a coarse-grained and a fine-grained classification model. The coarse-grained classification model removes most of the harmless content in the corpus to be detected. The fine-grained model, impromptu euphemisms detector, integrates context augmentation and multi-round iterations training to better predicts the actual meaning of a masked token. In addition, we leverage ChatGPT to evaluate the mode’s capability. Experimental results demonstrate that our approach achieves a remarkable 76-fold improvement compared to the previous state-of-the-art euphemism detector.

Serverless workflow applications, composed of multiple serverless functions, are increasingly popular in production. However, inter-function communication and cold start latency remain key performance bottlenecks. This paper introduces AlloyStack, a library operating system (LibOS) tailored for serverless workflows. AlloyStack addresses two major challenges: (1) reducing cold start latency through on-demand OS component loading and (2) minimizing data transfer overhead by enabling functions within the same workflow to share a single address space, eliminating unnecessary data copying. To ensure secure isolation, AlloyStack uses Memory Protection Keys (MPK) to separate user functions from the LibOS while maintaining efficient data sharing. Our evaluation shows that AlloyStack reduces cold start times by 98.5% to just 1.3ms. Compared to SOTA systems, AlloyStack achieves a 7.3× to 38.7× speedup in Rust end-to-end latency and a 4.8× to 78.3× speedup in other languages for intermediate data-intensive workflows.

Training large models in parallel by GPU clusters significantly accelerates the computation in each iteration. However, the frequent collective communication for synchronizing the huge number of gradients poses a scalability challenge, whose performance gradually becomes the bottleneck as the number of workers increases. Ring-reduce is a favorable architecture since it can balance the communication and computation load among workers. In this paper, we discover that the communication resources are underutilized when using the ring-reduce synchronization method in clusters. Accordingly, Shuffle-Exchange Synchronization (SES), a novel method is proposed to improve the communication efficiency for distributed large model training. SES organizes all the worker nodes into several groups, within which they perform small-scale ring-reduce synchronizations during each iteration. To achieve better convergence performance, a gradient correction operation is integrated into SES. Experiments in 16 workers on a real-world industrial computing platform, show that SES can accelerate the large model training to 1.97× without losing model performance.

The increasing parameter size of Deep Neural Networks (DNNs) has significantly enhanced model performance. As large-scale DNN models typically require partitioning into multiple blocks for distributed training, existing research has predominantly focused on offline scheduling for individual or batched training tasks. However, the stochastic arrival of such tasks in edge networks poses a critical challenge for efficiently scheduling them in resource-constrained edge clusters. In this paper, aiming to minimize the average training completion time across all tasks, we first extract DNN operator graphs and partition them into coarse-grained subgraphs using a max-flow mincut algorithm. Then, we formulate the online scheduling problem for multiple distributed DNN training tasks as a Markov Decision Process (MDP) and propose a reinforcement learning-based (RL-based) solution. Extensive experiments comparing our method with three conventional baselines (FIFO, SJF, and Greedy) under diverse configurations show that our approach reduces the average training completion time by 12.95%, demonstrating its effectiveness in resource-constrained edge environments with dynamic workloads.

Programmable packet scheduling enables users to customize scheduling algorithms flexibly without designing new ASICs. Existing schemes prefre to approximate optimal Push-In First-Out (PIFO) using First-In First-Out (FIFO) queues in commodity programmable switches. Despite its availability, these schemes suffer performance degradation due to the unawareness of available switch buffer. To be specific, when the port buffer is drained, existing schemes discard all incoming packets, even though these packets have higher priorities than the enqueued packets. In this paper, we reveal that the problem's key culprit is the lack of coordination between buffer management and packet scheduling in the switch. To fill this gap, we present Lark, a buffer-aware building block for programmable scheduling schemes designed to solve the above problem. Its key idea is to proactively drop the low-priority packets when the allocated buffer is to be drained, thereby admitting the later-arriving high-priority packets. Lark contains two modules, a lightweight gradient-based online prediction module and a simple priority-based decision module. Lark relies the former module to identify whether the allocated buffer is to be drained and uses the later to determine whether to drop the incoming packet. We have integrated Lark into two representative schemes, SP-PIFO and AIFO. Our large-scale evaluations over three realistic workloads show that Lark can significantly optimize their key metrics without sacrificing throughnut.

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, largely due to their substantial model size. However, this also results in significant GPU memory demands during inference. To address these challenges on hardware with limited GPU memory, existing approaches employ offloading techniques that offload unused tensors to CPU memory, thereby reducing GPU memory usage. Since offloading involves data transfer between GPU and CPU, it introduces transfer overhead. To mitigate this, prior works typically overlap data transfer with GPU computation using a fixed pipelining strategy applied uniformly across all inference iterations, referred to asstaticoffloading. However, static offloading policies fail to maximize inference throughput because they cannot adapt to the dynamically changing transfer overhead during the inference process, leading to increasing GPU idleness and reduced inference throughput.We propose that offloading policies should beadaptiveto the varying transfer overhead across inference iterations to maximize inference throughput. To this end, we design and implement an adaptive offloading-based inference system called TightLLM with two key innovations. First, its key-value (KV) distributor employs atrade-compute-for-transferstrategy to address growing transfer overhead by dynamically recomputing portions of the KV cache, effectively overlapping data transfer with computation and minimizing GPU idleness. Second, TightLLM’s weight loader slices model weights and distributes the loading processacross multiple batches, amortizing the excessive weight loading overhead and significantly improving throughput. Evaluation across various combinations of GPU hardware and LLM models shows that TightLLM achieves 1.3 to 23 times higher throughput during the decoding phase and 1.2 to 22 times higher throughput in the prefill phase compared to state-of-the-art offloading systems. Due to the higher throughput in prefill and decoding phases, TightLLM can reduce the completion time for large-scale tasks, which involve processing and generating a substantial number of tokens, by 59.6% to 94.9%.

The rise of Machine Learning as a Service (MLaaS) has driven the demand for complex and customized real-time inference tasks, often requiring cascading multiple deep neural network (DNN) models into inference pipelines. However, these pipelines pose significant challenges due to scheduling complexity, particularly in maintaining strict latency service level objectives (SLOs). Existing systems serve pipelines with model-independent scheduling policies, which ignore the unique workload characteristics introduced by model cascading in the inference pipeline, leading to SLO violations and resource inefficiencies. In this paper, we propose that the serving system should exploit the model-cascading nature and inter-model workload dependency of the inference pipeline to ensure strict latency SLO cost-effectively. Based on this, we design and implementSLOpt, a serving system optimized for real-time inference pipelines with a three-stage co-design of workload estimation, resource provisioning, and request execution.SLOptproposes cascade workload estimation and ahead-of-time tuning, which together address the challenge of cascade blocking and head-of-line blocking in workload estimation and resource provisioning.SLOptfurther implements an adaptive batch drop policy to mitigate latency amplification issues within the pipeline. These innovations enableSLOptto reduce the 99th percentile latency (P99 latency) by 1.4 to 2.5 times compared to the state of the arts while lowering serving costs by up to 29%. Moreover, to achieve comparable P99 latency,SLOptrequires up to 70% less cost than existing systems. Extensive evaluations on a 64-GPU cluster demonstrateSLOpt’s effectiveness in meeting strict P99 latency SLOs under diverse real-world workloads.

Serverless computing typically relies on the third-party forwarding method to transmit data between functions. This method couples control flow and data flow together, resulting in significantly slow data transmission speeds. This challenge makes it difficult for the serverless computing paradigm to meet the low-latency requirements of web services.

Logging-based fault-tolerant methods have recently been widely used in serverless computing, which can provide exactly-once execution semantics for stateless functions. However, this would also introduce considerable system overhead in serverless platforms, especially for the function workflow scenarios. How to build an efficient fault-tolerant logging system for latency-sensitive stateful functions is still challenging. We find the prior synchronous logging design causing the workflow block on waiting for every log request, hinders the performance of workflow execution. In this paper, we present DoubleTrack, a serverless runtime system for fault-tolerant workflows. The key of DoubleTrack is the asynchronous shared log, which decouples fault-tolerant logging and workflow execution, thus mitigating the overhead of synchronous logging. Our evaluation with real-world workloads shows that compared to the synchronous logging approach, DoubleTrack achieves up to 45.8% end-to-end latency and 46.5% logging overhead reduction.

Serverless computing enhances application development and deployment through elastic resource provisioning. However, the cold start problem, caused by the stateless nature of serverless functions, remains a significant challenge, particularly in high-traffic scenarios. Existing solutions focus on optimizing function instance caches and reducing sandbox initialization times, yet resource inefficiencies in multi-level cache architectures persist. This paper proposes Photia, a cross-layer collaborative cache management system that integrates decision-making between function instance and sandbox caches. Experimental results demonstrate that Photia reduces function startup latency by up to 2x and improves sandbox cache resource utilization by 20%.

Existing works explore various consistency levels to address transactional consistency gaps in serverless platforms but have not explored achieving strong session serializability (3SER) for serverless. For applications with 3SER requirements, stronger transactional consistency is unnecessary and can compromise performance. However, the stateless nature of serverless architectures and their fault-tolerance mechanisms pose challenges in designing a 3SER transaction protocol. We propose TFaaS, the first transactional serverless platform that provides a 3SER guarantee for serverless workflows. TFaaS employs caching to manage session states, transaction execution states, and workflow logs. It selectively persists the necessary states based on time fence techniques to ensure consistency and fault-tolerance requirements are met. Our evaluation demonstrates that TFaaS can effectively reduce latency by up to 6.5× and improve throughput by up to 8×.

Serverless computing enables a new way of building and scaling cloud applications by allowing developers to write fine-grained functions. The execution duration of a cloud function is typically short, usually ranging from a few milliseconds to a few seconds. FaaS providers charge users based on the execution duration of cloud functions with a granularity of 1 millisecond. Existing mixed scheduling methods collocate functions with varying execution times, which may prolong their execution duration and lead to unfair charges for FaaS users. To address this problem, we propose a partition scheduling approach, placing functions with varying execution times on different partitions. We introduce Synergy, a solution in serverless computing that leverages a collaboration of central and local scheduling to partition functions and employs suitable scheduling algorithms for these partitions respectively. Synergy also supports the dynamic switching of scheduling algorithms for partitioned nodes to adapt to highly fluctuating loads. We evaluate Synergy using real-world, representative benchmarks. Experimental results demonstrate that, compared to state-of-the-art and conventional approaches, Synergy can reduce the average function execution duration by $63 \%$.

Serverless platforms use sandbox technology to provide isolated environments for workloads, preventing security attacks from untrusted workloads in multi-tenant environments. Among existing sandbox mechanisms, those based on memory partitioning have the advantage of lightweight isolation, enabling efficient instance scheduling and switch operations, and good scalability. The CAP-VM system, which combines CHERI memory partitioning primitives and library operating system mechanisms, provides good runtime performance and security isolation levels. However, this sandbox system still has an issue of high startup latency, which cannot meet the service quality requirements in Serverless scenarios. To solve the problems existing in the CAP-VM sandbox system, we propose a new snapshot mechanism and sandbox fork caching mechanism. Through this snapshot mechanism, the sandbox system can lower the cold start latency of the sandbox by loading the pre-initialized instance state at startup, and reduce the memory overhead of the sandbox by sharing the instance state. The fork caching mechanism optimizes the sandbox creation process by introducing CAP-VM sandbox fork primitives, reducing the sandbox cold start latency. Compared with the existing CAP-VM sandbox system, the sandbox snapshot system and fork caching mechanism can reduce the sandbox cold start latency by more than ${6 0 \%}$ and ${9 3 . 3 \%}$ respectively.

Traditional network functions are typically implemented using specialized hardware appliances, which are expensive and difficult to upgrade. Network Function Virtualization (NFV) offers an effective approach to address these challenges by implementing comparable functionalities on commercial servers through software-based virtualization. In NFV, a sequence of Virtual Network Functions (VNFs) is orchestrated to form a Service Function Chain (SFC) that provides flexible network services. However, scheduling SFCs with multiple resource constraints to achieve high reliability poses a critical challenge. Existing approaches often assume offline scheduling and overlook the dynamic nature of heterogeneous resource loads across regions. Moreover, they primarily focus on individual VNFs rather than considering the cross-region scheduling of the entire SFC, which can result in increased transmission delay. In this paper, we investigate the problem of service function chain scheduling across multiple regions (SFCS-MR) with deadline constraints, aiming to maximize the success rate of requests. We formulate this problem as an Integer Linear Programming (ILP) model and prove its NP-hardness. To address this problem effectively, we propose a two-stage algorithm that determines whether an SFC requires cross-region scheduling and selects the suitable regions for its execution. Through extensive experimental evaluations, we demonstrate that our cross-region SFC scheduling solution can achieve a maximum improvement of 32.42% in the overall request success rate compared to benchmarks.

Cloud service providers improve resource utilization by co-locating latency-critical (LC) workloads with best-effort batch (BE) jobs in datacenters. However, they usually treat multi-component LCs as monolithic applications and treat BEs as “second-class citizens” when allocating resources to them. Neglecting the inconsistent interference tolerance abilities of LC components and the inconsistent preemption loss of BE workloads can result in missed co-location opportunities for higher throughput. We present Rhythm , a co-location controller that deploys workloads and reclaims resources rhythmically for maximizing the system throughput while guaranteeing LC service’s tail latency requirement. The key idea is to differentiate the BE throughput launched with each LC component, that is, components with higher interference tolerance can be deployed together with more BE jobs. It also assigns different reclamation priority values to BEs by evaluating their preemption losses into a multi-level reclamation queue. We implement and evaluate Rhythm using workloads in the form of containerized processes and microservices. Experimental results show that it can improve the system throughput by 47.3%, CPU utilization by 38.6%, and memory bandwidth utilization by 45.4% while guaranteeing the tail latency requirement.

Caching function is a promising way to mitigate coldstart overhead in serverless computing. However, as caching also increases the resource cost significantly, how to make caching decisions is still challenging. We find that the prior "local cache control" designs are insufficient to achieve high cache efficiency due to the workload skewness across servers.

Reinforcement Learning (RL) algorithms require large amounts of computational resources and time to train due to the simultaneous model training and real-time interactions with simulation environments. This results in an RL algorithm's training time potentially exceeding days to months. We present HRL, a comprehensive optimization system designed to improve the system throughput of RL algorithms. HRL addresses the challenges of discovering and identifying bottlenecks in the sampling, training, or communication stages and promptly improving their performance. We propose a group-parallel pipeline method to improve the sampling efficiency and apply data quantization and multi-learner training to resolve the network and learning bottlenecks. The HRL system has been fully implemented and integrated with XingTian. The results of the experiments show that the HRL system can improve the throughput by 18.6%-90.6%.

Serverless computing commonly adopts strong isolation mechanisms for deploying functions, which may bring significant performance overhead because each function needs to run in a completely new environment (i.e., the "one-to-one" model). To accelerate the function computation, prior work has proposed using sandbox sharing to reduce the overhead, i.e., the "many-to-one" model. Nonetheless, either process-based true parallelism or thread-based pseudo-parallelism still causes high latency, preventing its adaptation for latency-sensitive web services.

Domain generation algorithms (DGAs) can be categorized into three types:zero-knowledge,partial-knowledge, andfull-knowledge. While prior research merely focused onzero-knowledgeandfull-knowledgetypes, we characterize their anti-detection ability and practicality and find thatzero-knowledgeDGAs present low anti-detection ability againstdetectors, andfull-knowledgeDGAs suffer from low practicality due to the strong assumption that they are fullydetector-aware. Given these observations, we proposePKDGA, a partial knowledge-based domain generation algorithm with high anti-detection ability and high practicality.PKDGAemploys the reinforcement learning architecture, which makes it evolve automatically based only on the easily-observable feedback from detectors. We evaluatePKDGAusing a comprehensive set of real-world datasets, and the results demonstrate that it reduces the detection performance of existingdetectorsfrom 91.7% to 52.5%. We further applyPKDGAto theMiraimalware, and the evaluations show that the proposed method is quite lightweight and time-efficient.

Facing the ever-increasing demands of microprocessors for computing power nowadays, trigonometric functions, as the basis of various algorithms, have problems with latency, resource overhead and power consumption in traditional software or coprocessor schemes. In this paper, a lightweight RISC-V standard microprocessor is designed. In addition, the optimization of the CORDIC algorithm realizes a low-latency and high-precision trigonometrical kernel-level accelerator， and the instruction set is extended. The experiment shows that the instruction expansion method used in this paper can reduce the iteration cycle by more than 92% on average compared with the pure software solution and it can reduce the error and the iteration cycle by 33% and 46% on average compared with the solution of the traditional CORDIC algorithm. Under the SMIC180nm technology, the extension instruction consumes 3.9k equivalent gates and only occupies 13.9% of the processor area. The RISC-V processor after extending trigonometric function still has several advantages in area and main frequency relative to the ARM processor in the same period.

Modern websites increasingly rely on machine learning (ML) to improve their business efficiency. Developing and maintaining ML services incurs high costs for developers. Although serverless systems are a promising solution to reduce costs, we find that the current general purpose serverless systems cannot meet the low latency, high throughput demands of ML services.

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

Computing systems always face a “resource allocation dilemma” that shows the great difficulties in trading off resource efficiency for tail latency, due to the internal uncertainty of cluster status and execution behavior. Inspired by the imaginary “Maxwell’s demon” in thermodynamics who can reduce the uncertainty through a per-gas molecule-level control policy, we consider the “one-to-one mapping” feature of serverless computing and build a novel resource allocator, named Maxwell, that can achieve low tail latency and high resource efficiency in serverless simultaneously. Like the “Maxwell’s demon Maxwell is able to optimize the resource allocation for every request. It observes the state of each request and makes decisions about the minimum resource allocation through a reinforcement learning predictor. As the per-request-grained control incurs significant overhead, we further design a pipeline for avoiding the accumulated effect on a workflow. Experimental results show that Maxwell not only saves up to 31% CPU resources but also reduces the standard deviation of latency by 1.9×. Its time overhead is negligible and the resource overhead is also limited when the query per second $\leq$500.

Modern resource management frameworks guarantee low tail latency for long-running services using the resource over-provisioning method, resulting in serious waste of resources and increasing the service costs greatly. To reduce the over-provisioning cost, we present HRSE, a hybrid resource scaling engine that enables much more efficient resource provisioning for both periodic and non-periodic workloads of long-running services while guaranteeing the tail latency Service Level Objective (SLO). HRSE employs a convolution-based time series analysis to identify periodic patterns in workloads. If periodic patterns are discovered, HRSE estimates the just-right amount of resources based on the periodic features through atop-$K$based collaborative filtering approach. Otherwise, it leverages wavelet-clustering to capture the short-term patterns in non-periodic workloads and predict the resource demands for the near future. To further enforce the tail latency SLO, HRSE uses an online reprovisioning mechanism that dynamically adjusts the resources to mitigate the performance uncertainty due to workload burstinesses. We fully implement HRSE on top of Docker and conduct extensive experiments using traces from production systems. Testbed experiments show that HRSE is able to increase the average resource utilization to 43 and 45 percent for periodic and non-periodic workloads respectively while guaranteeing the same tail latency objective.

In the era of cloud computing, network services are deployed on geographically distributed cloud platforms, which results in a large amount of inter-datacenter traffic. Multi-tier pricing schemes are widely adopted by cloud service providers (CSPs) to charge cloud users for inter-datacenter transmission services. To avoid a severe penalty associated with missing a deadline, cloud users are prone to selecting a sufficiently high service level. However, they are usually unaware of the total traffic volume before accessing the network; hence, a high transmission cost is introduced. In this paper, we propose an online cost-efficient transmission scheme for cloud users with information-agnostic traffic. The basic idea is to split a long-term transmission request into a series of short-term ones. In this scheme, we take into account the CSP’s countermeasures, and model the interactions between the cloud users and the CSP as a Stackelberg game. We show that the optimal number of short-term requests and the associated transmission service levels can be determined with an online algorithm based on Lyapunov optimization. The experimental results reveal that the CSP and the cloud users can achieve a win-win outcome, whereby the transmission cost of cloud users can be reduced by 59 percent.

In software defined networking, theflatdesign of distributed control plane enables the management of multi-domain networks that are incapable of deploying a root controller. However, it is very difficult to avoid policy conflicts between independent local controllers due to the lack of centralized arbitration. Moreover, domains without trust may not be always cooperative and could even cheat to maximize their own interests. In this article, we first consider the cooperative scenario and address the problem of traffic engineering in a flat distributed control plane. We propose a fully distributed algorithm, calledDisTE, which can provide max-min fair bandwidth allocation for flows and maximize resource utilization.DisTEalso preserves the local topology of each domain and achieves policy consistency by multiple rounds of synchronization. We then consider the non-cooperative scenario, where selfish domains may discriminate bandwidth requests from other domains or overstate theirs owns to squeeze more bandwidths.

The cloud resource provision policy of a content provider in the presence of competitors on globally distributed cloud platforms plays a significant role in maximizing its profit. However, developing an optimal resource provision policy is quite challenging, due to the difficulty to capture the competition relationship between two competitive CPs and to obtain the budget of the competitors which is usually kept private. To solve this problem, in this article, we propose a learning-driven cloud resource provision policy for a CP with competitors. We formulate the competition between the CPs as alottery Colonel Blottogame in which the payoff of each region is positively related to the resource advantage achieved by the CP, formulate the budget allocation problem as a Markov decision process, and obtain the sub-optimal resource provision policy by reinforcement learning and deep reinforcement learning-based algorithms. We also prove the convergence of the sub-optimal solution. Finally, we validate our proposed method using real-world CPs statistics. The results show that the budget information is critical for a CP to make policy decisions, and it is better for CPs with smaller budget to focus their budget resources in regions with higher values.

As the scale and complexity of deep learning models continues to grow, model training is becoming an expensive job and only a small number of well-financed organizations can afford. Are the resources in commodity clusters well utilized for training? or how much potential space are still there for further improving the training efficiency in commodity clusters? is an urgent question to answer.

Datacenters' excessive energy consumption has become an increasingly significant pain point to cloud service providers. However, existing research usually tends to focus on the server power cap to increase application throughput and do not address the rapid growth in the energy consumption of datacenters. Although the serverless architecture makes it more difficult to improve datacenter energy efficiency due to its high density and high dynamic nature, we observe that the emerging serverless workloads bring new opportunities for energy reduction. First, the energy consumption of serverless workloads during their execution time can be divided into three stages with clear boundaries: startup, runtime and idle. As the three stages have quite different energy usage patterns, we are able to reduce overall energy consumption through orchestrating the energy consumption of the stages. Second, we observe the energy fungibility phenomenon, i.e., the different combinations of multidimensional resource allocations may lead to the same latency but different energy consumption. Exploiting the balanced combination of energy consumption and performance, we can reduce the overall energy consumption without violating the latency service level agreement (SLA). Based on this, we propose RAEF, a function-level resource allocator that proactively adjusts the resources of the functions for minimizing the energy consumption with SLA guarantees. Evaluation results demonstrate that RAEF reduces energy consumption by up to 21.2 % compared to the state-of-the-art technique while guaranteeing the SLA.

Interference among distributed cloud applications can be classified into three types: full, partial and zero. While prior research merely focused on full interference, the partial interference that occurs at parts of applications is far more common yet still lacks in-depth study. Serverless computing that structures applications into small-sized, short-lived functions further exacerbate partial interference. We characterize the features of partial interference in serverless as exhibiting high volatility, spatial-temporal variation, and propagation. Given these observations, we propose an incremental learning predictor, named Gsight, which can achieve high precision by harnessing the spatial-temporal overlap codes and profiles of functions via an end-to-end call path. Experimental results show that Gsight can achieve an average error of 1.71%. Its convergence speed is at least 3X faster than that in a serverful system. A scheduling case study shows that the proposed method can improve function density by ≥ 18.79% while guaranteeing the quality of service (QoS).

In mobile social networks (MSNs), mobile users communicate with each other via mobile devices, such as smartphones and tablets, transmitting data through intermittent connections. Mobile users have high mobility, which creates higher requirements for efficient data forwarding in MSNs. Therefore, forwarding data efficiently and quickly becomes a key problem. To tackle this problem, this article proposes a routing method based on a dynamic transient social community (DTSC) to optimize the routing and forwarding performance in MSNs. In this process, combined with the duration of intensive contact between nodes and the social relations of mobile users, the similarity of each pair of contact nodes is calculated, and community detection is carried out. Then, by analyzing the emergence mode of the DTSC, the measurement value and corresponding routing algorithm of the community’s ability to deliver messages are designed. Our algorithm fully considers the duration of the node’s direct encounter and the social connection of the indirect contact to ensure that the node can deliver successfully in a short time. The experimental results show that the DTSC has an excellent performance in data forwarding.

Detecting abnormal service performance is significant for Internet-based service management and operation. Recent advances in anomaly detection methods prefer unsupervised learning algorithms since they can work without manually labelled data. However, existing unsupervised methods converge into suboptimal solutions due to their heuristic-based objectives. Moreover, they frequently rely on the strong assumption that noise follows a Gaussian distribution, and their detection accuracy is also highly sensitive to threshold settings. To detect anomalies precisely and robustly, we presentAdran, an unsupervised anomaly detection model that introduces adversarial learning into a reconstructive model, generating a reconstructive adversarial network with an anomaly detection-based training objective. It tolerates non-Gaussian noise by activating the discriminator with a non-smooth function. Our experimental results demonstrate thatAdranachieves an improvement of$\geq 32\%$over the state-of-the-art methods in terms ofF-score. Moreover, the robustness analysis demonstrates that it is reasonably easy and straightforward to set an appropriate threshold usingAdran.

Cloud service providers improve resource utilization by co-locating latency-critical (LC) workloads with best-effort batch (BE) jobs in datacenters. However, they usually treat an LC workload as a whole when allocating resources to BE jobs and neglect the different features of components of an LC workload. This kind of coarse-grained co-location method leaves a significant room for improvement in resource utilization.

Detecting anomaly in images is challenging due to the high dimension nature of image data. While the previous learning-based anomaly detection approaches can detect a particular type of anomaly precisely, they often fail in detecting multiple types of abnormal samples simultaneously.We identify the two specific types of anomalies that can be precisely detected by either compress-based or reconstruction-based anomaly detection approaches, named global anomaly and local anomaly. We then propose Glad, an anomaly detector that can precisely detect both of them at the same time. Glad adopts a joint approach combining the density estimation and auto-encoder. Firstly, it designs a multimodal density estimation model to derive the latent representation probability for identifying the global anomaly. Then, it uses structural similarity to measure the reconstruction loss for characterizing local anomaly. Finally, both anomalies can be diagnosed according to the joint density of latent representation and reconstruction loss. Experimental results on public benchmark datasets demonstrate that Glad outperforms the state-of-the-art methods significantly.

Accurate and quick failure localization is critical for automatic network troubleshooting. While it is particularly difficult to solve the problem in the traditional network due to the uncertain routing, Software Defined Networking (SDN) enables the deterministic routing for packet transmission through the traffic engineering algorithm in the centralized controller.

Recent trends show that cloud computing is growing to span more and more globally distributed datacenters. For geo-distributed datacenters, there is an increasingly need for scheduling algorithms to place tasks across datacenters, by jointly considering WAN traffic and computation. This scheduling must deal with situations such as wide-area distributed data, data sharing, WAN bandwidth costs and datacenter capacity limits, while also minimizing makespan. However, this scheduling problem is NP-hard. We propose a new resource allocation algorithm called HPS+, an extension to Hypergraph Partition-based Scheduling. HPS+ models the combined task-data dependencies and data-datacenter dependencies as an augmented hypergraph, and adopts an improved hypergraph partition technique to minimize WAN traffic. It further uses a coordination mechanism to allocate network resources closely following the guidelines of task requirements, for minimizing the makespan. Evaluation across the real China-Astronomy-Cloud model and Google datacenter model show that HPS+ saves the amount of data transfers by upto 53 percent and reduces the makespan by 39 percent compared to existing algorithms.

The increasing scale of software defined networks (SDN) raises the requirement of distributed control plane, for providing scalable, reliable and high performance network management capabilities. In particular, the flat design of distributed control plane enables the management of networks with multiple independent domains that are incapable of deploying a root controller. However, it is very difficult to avoid policy conflicts between multiple controllers in flat plane due to the lack of arbitration. In this paper, we address the problem of traffic engineering in a flat control plane, and design a distributed traffic engineering algorithm, called DisTE, which can provide max-min fair bandwidth allocation for flows and maximize the resource utilization, using a fully distributed arbitration mechanism. DisTE also preserves the local topology of each domain using the topology aggregation method, and supports consistency by multiple rounds of synchronizations. We examine four strategies for determining the synchronization timings, and find that linearly decreasing interval method provides a better trade-off between network utilization and time costs. Experiments on a 717-switches 5-domain network topology demonstrate that DisTE could drive the link utilization ratio to more than 93%, and reduce up to 95% convergence time at cost of 3% relative error on fairness, compared to the centralized approach.

A new broad of video services that support live streaming has become tremendously popular in recent years. Compared with traditional video-on-demand (VOD) services, live video streaming has much higher requirements on Quality-of-Experience (QoE), including low rebuffering, high definition, low latency and low bitrate oscillations. While previous adaptive bitrate algorithms (ABR) solely optimize bitrate for ensuring QoE of VOD, live video streaming has a larger decision space, making the optimization problem more difficult to solve. We propose Deeplive, which maximizes QoE through deep reinforcement learning (DRL), so it does not rely on fixed rules. To accelerate the training process of Deeplive, we further propose optimization including window completion with historical data and quick-start with rate-based algorithm. We compare Deeplive with other advanced ABR algorithms in a frame-level dynamic adaptive video streaming simulator using different network traces, QoE definitions, and video categories. In all experiments, we find that Deeplive not only has significant improvement in training time, but also shows an average of 15-55% improvement on QoE than the state-of-the-art ABR algorithms.

In-memory data processing frameworks (e.g., Spark) make big data analysis greatly simpler and efficient. However, stragglers that take much longer to finish than other tasks significantly degrade performance. There exist multiple factors that cause stragglers, either from the hardware resource layer or application layer, e.g. hardware heterogeneity, interference, data locality and data skew. While state-of-the-art straggler mitigation techniques have presented partial solutions on data skew and data locality, we find that the other factors can also result in serious problems. We present Clio, a cross-layer interference-aware optimization system that can effectively mitigate stragglers for data processing frameworks. Clio supports the scheduling of both map and reduce tasks. It heuristically dispatches intermediate data in proportion to the actual computing ability of each worker node, which is estimated considering various straggler factors, to balance the completion times of tasks in a much finer way. We implement Clio in Apache Spark, and evaluate its performance using both synthetic and real datasets. Experiment results show that, Clio can speed up the execution of applications by up to 67%, compared with the existing algorithms.

Modern resource management frameworks guarantee low tail latency for long-running services using the resource over-provisioning method, resulting in serious waste of resource and increasing the service costs greatly. To reduce the over-provisioning cost, we present EFRA, an elastic and fine-grained resource allocator that enables much more efficient resource provisioning while guaranteeing the tail latency Service Level Objective (SLO). EFRA achieves this through the cooperation of three key components running on a containerized platform: The period detector identifies the period features of the workload through a convolution-based time series analysis. The resource reservation component estimates the just-right amount of resources based on the period analysis through a top-K based collaborative filtering approach. The online reprovisioning component dynamically adjusts the resources for further enforcing the tail latency SLO. Testbed experiments show that EFRA is able to increase the average resource utilization to 43%, and save up to 66% resources while guaranteeing the same tail latency objective.

Parallel and distributed computing has been under many years of development, and paved the way that what information and communication technology looks like nowadays. With the advance of new techniques, such as 5G, cloud computing, and big data, the theory, design, analysis, evaluation, and application of parallel and distributed computing have encountered great challenges to meet the increasing requirements on high performance, energy efficiency, as well as reliability and security. To achieve these goals, interdisciplinary knowledge and some specialized technical skills are required. This special issue is a collection of many examples of how researchers, scholars, vendors, and practitioners are collaborating to address these challenges.

Cost savings have become a significant challenge in the management of data centers. In this paper, we show that, besides energy consumption, service level agreement(SLA) violations also severely degrade the cost-efficiency of data centers. We present online VM placement algorithms for increasing cloud provider's revenue. First, First-Fit and Harmonic algorithm are devised for VM placement without considering migrations. Both algorithms get the same performance in the worst-case analysis, and equal to the lower bound of the competitive ratio. However, Harmonic algorithm could create more revenue than First-Fit by more than 10 percent when job arriving rate is greater than 1.0. Second, we formulate an optimization problem of maximizing revenue from VM migration, and prove it as NP-Hard by a reduction from 3-Partition problem. Therefore, we propose two heuristics: Least-Reliable-First (LRF) and Decreased-Density-Greedy (DDG). Experiments demonstrate that DDG yields more revenue than LRF when migration cost is low, yet leads to losses when SLA penalty is low or job arriving rate is high, due to the large number of migrations. Finally, we compare the four algorithms above with algorithms adopted in Openstack using a real trace, and find that the results are consistent with the ones using synthetic data.

As cloud services grow to span more and more globally distributed datacenters, there is an increasingly need for scheduling algorithms to automatically place tasks across these datacenters. In geo-distributed cloud, the limited WAN bandwidth has become the major bottleneck in fast big data analytics. The scheduling algorithm needs to minimize the global completion time, by jointly optimizing task scheduling and WAN data transfer. In this paper, we model the task scheduling as a community detection problem, with respect to the dependency relations between task, data, and datacenters, and propose a Community Detection-based Scheduling (CDS) algorithm, which is able to minimize the WAN data transfer volume. We utilize the real China-Astronomy-Cloud network to evaluate the proposed algorithms. Experimental results show that we can reduce the total data transfer volume by up to 40.7%, and the global completion time by up to 35.8%, compared with the Hypergraph Partition-based scheduling algorithm and the greedy scheduling algorithm.

Recent trends show that cloud computing is growing to span more and more globally distributed data centers. For geo-distributed data centers, there is an increasing need for scheduling algorithms to place tasks across data centers, by jointly considering data and computation. This scheduling must deal with situations such as wide-area distributed data, data sharing, WAN bandwidth costs and data center capacity limits, while also minimizing completion time. However, this kind of scheduling problems is known to be NP-Hard. In this paper, inspired by real applications in astronomy field, we propose a two-phase scheduling algorithm that addresses these challenges. The mapping phase groups tasks considering the data-sharing relations, and dispatches groups to data centers by way of one-to-one correspondence. The reassigning phase balances the completion time across data centers according to relations between tasks and groups. We utilize the real China-Astronomy-Cloud model and typical applications to evaluate our proposal. Simulations show that our algorithm obtains up to 22% better completion time and effectively reduces the amount of data transfers compared with other similar scheduling algorithms.

Decoupling of control and data plane in Software Define Networks (SDN) creates significant flexibility in network management. As networks are evolving into a complex multi-domain multi-layer architecture, traffic engineering across multiple domains and layers entails challenges for the control plane, especially when each separate administrative domain does not disclose their network topology and resource information. In this paper, we present a hierarchical controller design over multidomain and multi-layer networks, by adopting a root controller at the top layer. We allow to aggregate network topology and QoS information into a hierarchical Network Information Base (NIB) for the confidentiality concern. Then, we devise a communication protocol, which enables controllers at different layers and domains to work collaboratively on bandwidth allocation by reading to the hierarchical NIB. We also present an improved traffic engineering algorithm by considering bandwidth and delay simultaneously, to maximize the network utilization while respecting max-min fairness. Experiments on a 717-switches 5-domain network topology demonstrate that our proposal could drive the link utilization ratio to more than 85%.

We introduce a new Space-Filling Multidimensional Data Visualization (SFMDVis) that can be used to facilitate the viewing, interaction and analysis of the multidimensional data with a fully utilized display space. The existing multidimensional visualizations typically create visual clutter and over-plotting that make it difficult for interaction with data items directly. Our new space filling technique uses horizontal lines to represent multidimensional data items. Each line is logically divided into segments based on color mapping in order to denote the data item with its value. The proposed visualization is space efficient and also avoids the visual clutter and over-plotting problems as we have often observed in other visualizations. In addition, we allow user to interact directly with data on the display which is more intuitive and efficient than other means.

The distributed system made the large-scale scientific computing possible in a cost effective way. And the hardware resources in such systems are also getting much cheaper than years before. However, the problem of executing the job using minimum resources is still reasonable and important, especially for the cloud environment, who has to save energy and control cost. Unfortunately, only a few existing scheduling algorithms have taken into account the resource usage issue. In this study, with considering the realistic network topology and communication model, we firstly propose the Deadline, Reliability, Resources-aware (DRR) scheduling algorithm. The theory analysis fully demonstrate that, the output schedule of our algorithm can satisfy the user's requirement on reliability and deadline. Through the experiments, with setting the deadline less than the make span of the MaxRe algorithm's output schedule, we find that our algorithm can complete the job under this deadline. Besides, our algorithm can save almost 50% computation resources and 70% communication resources than FTSA (bl) and FTSA (tl+bl) algorithms.

In the existing studies on fault-tolerant scheduling, the active replication schema makes use of ε + 1 replicas for each task to tolerate E failures. However, in this paper, we show that it does not always lead to a higher reliability with more replicas. Besides, the more replicas implies more resource consumption and higher economic cost. To address this problem, with the target to satisfy the user's reliability requirement with minimum resources, this paper proposes a new fault tolerant scheduling algorithm: MaxRe. In the algorithm, we incorporate the reliability analysis into the active replication schema, and exploit a dynamic number of replicas for different tasks. Both the theoretical analysis and experiments prove that the MaxRe algorithm's schedule can certainly satisfy user's reliability requirements. And the MaxRe scheduling algorithm can achieve the corresponding reliability with at most 70% fewer resources than the FTSA algorithm.