In the realm of AI, large language models (LLMs) like GPT-5, central to the operation of AI agents, predominantly operate in the cloud, incurring high operational costs. With local-based small language models (SLMs) becoming more accurate, the necessity of cloud-exclusive processing is being reconsidered. An AI agent's response to a user's request comprises a series of subtasks or iterations. Existing approaches choose either an LLM or SLM for the entire request to ensure similar outputs, but this is ineffective for AI agents as SLMs may generate differing subtasks, compromising final accuracy. In this paper, we first conduct experimental analysis to understand the features of AI agent operations. Leveraging our findings, we propose the Adaptive Iteration-level Model Selector (AIMS), a lightweight scheduler to automatically partition an AI agent's subtasks between local-based SLM and cloud-based LLM. AIMS considers the varying subtask features and strategically decides the location for each subtask in order to use SLM as much as possible while maintaining the accuracy level. Our experimental results demonstrate that AIMS achieves up to a 27.5% relative improvement in accuracy and up to 31.4% relative increase in SLM usage compared to HybridLLM. It offloads 83.4% of subtasks to a local SLM while attaining similar accuracy on average compared with the cloud-only LLM approach.

The inference workloads of Large Language Models (LLMs) pose significant latency and cost challenges due to increasing model sizes and demand for real-time responses. Existing cluster schedulers for multi-instance LLM serving primarily focus on load balancing to optimize memory usage, which is insufficient for workloads with diverse request characteristics. In such cases, the compute layout — the arrangement of tokens across iterations within each instance—plays a crucial role in determining latency. We propose AdaGen, a workload-adaptive cluster scheduler that minimizes latency and thus maximizes SLO attainment by optimizing compute layouts across instances. AdaGen employs a multi-step scheduling strategy: it first classifies requests based on prefill and decode lengths, then balances load, and finally performs selective distributed execution across instances. Each step incrementally refines the scheduling based on the compute layouts derived from the decision of the previous step. To avoid the overhead of actual execution to generate the layouts, AdaGen introduces a novel simulation-based estimator. Extensive experiments using production workloads show that AdaGen achieves up to 3.6× higher SLO attainment and 2× better cost-efficiency compared to the existing systems, while ensuring scalability.

Large language models increasingly rely on pipeline parallelism for distributed inference, but existing systems face critical challenges in serverless environments: heterogeneous request distributions across pipeline stages and unpredictable workload patterns requiring rapid elasticity. Traditional static resource allocation fails to address pipeline-specific bottlenecks and cold start delays inherent in serverless architectures. We propose QUART, a workload-adapted resource allocation system for LLM distributed serving in serverless clusters. QUART introduces pipeline-aware resource management through: (1) latency-aware critical stage identification using coefficient of variation (CV)-based burst propagation analysis, (2) dynamic replica allocation with proportional-integral-derivative (PID) control for congested stages, and (3) hierarchical parameter caching with copy-on-write mechanisms enabling sub-second serverless scaling. The system addresses serverless-specific challenges through cache-aware scheduling that maintains model parameters in memory, eliminating disk I/O overhead during rapid scaling events. Evaluation with real-world workloads shows QUART reduces average response latency by up to 87.1% compared to existing serverless inference systems while achieving 2.37x improvement in goodput.

Cold start delays are a main pain point for today's FaaS (Function-as-a-Service) platforms. A widely used mitigation strategy is keeping recently invoked function containers alive in memory to enable warm starts with minimal overhead. This paper identifies new challenges that state-of-the-art FaaS keep-alive policies neglect. These challenges are caused by concurrent function invocations, a common FaaS workload behavior. First, concurrent requests present a tradeoff between reusing busy containers (delayed warm starts) versus cold-starting containers. Second, concurrent requests cause imbalanced evictions of containers that will be reused shortly thereafter. To tackle the challenges, we propose a novel serverless function container orchestration algorithm called CIDRE. CIDRE makes informed decisions to speculatively choose between a delayed warm start and a cold start under concurrency-driven function scaling. CIDRE uses both fine-grained container-level and coarse-grained concurrency information to make balanced eviction decisions. We evaluate CIDRE extensively using two production FaaS workloads. Results show that CIDRE reduces the cold start ratio and the average invocation overhead by up to 75.1% and 39.3% compared to state-of-the-art function keep-alive policies.

In this work, from our trace-driven experimental measurements, we observed that despite employing homogeneous GPUs for distributed deep learning (DDL) training, stragglers persistently emerge, significantly prolonging time-to-accuracy (TTA) and squandering GPU resources. Previous approaches typically react to stragglers as they occur or are imminent during execution after scheduling, resulting in training delays before they are addressed and introducing additional overhead. To reduce the number of stragglers, this paper introduces a novel DDL training job scheduler for proactive straggler reduction (STRN), the first effort in mitigating stragglers during scheduling. STRN is devised based on our findings that various DDL jobs exhibit distinct sensitivities to straggling and a particular resource's overload, and the optimal synchronization strategy for a job hinges on its specific characteristics and operational environment. Thus, STRN assesses each job's sensitivity to each resource type and straggling. It aims to minimize the likelihood of resource overload for jobs with higher sensitivity to that resource and to ensure that jobs with higher sensitivity to straggling encounter less straggling. STRN initially runs a heuristic method and then transitions to a Reinforcement Learning (RL)-based method once trained, facilitating expedited and optimal job scheduling. STRN also optimizes synchronization strategies for each DDL job to reduce overall TTA. Trace-driven real experiments demonstrate that STRN reduces average TTA by up to 59% and enhances average accuracy by up to 91% compared to state-of-the-art methods. We have made the source code available for distribution.

This paper presents a comprehensive analysis of diffusion model serving challenges in production cloud environments. We examine the unique computational patterns and resource requirements that distinguish diffusion model serving from traditional ML workloads, revealing fundamental systemlevel challenges from their multi-stage pipeline architectures. Our analysis is based on a dataset collected from a commercial image generation service processing 3.5 million requests across 300+ GPUs of production operation.

Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environments poses significant challenges due to their complex model architectures and heterogeneous characteristics across their multi-stage inference pipelines and modalities.

Small deep learning models deployed at edge servers suffer from decreasing accuracy due to data drift and hence require continual learning, which leads to resource competition with inference execution, decreasing accuracy and latency service-level-objective (SLO) fulfillment. Previous methods fail to maximize both accuracy and SLO fulfillment. To address this problem, we propose a Checkpointed Inference-based system for accurate and SLO-guaranteed data drift-resilient model Serving (CIS). CIS incorporates checkpointed inference - maximizing accuracy by continuously switching to intermediately retrained models during inference. However, model switching introduces significant inference latency overhead. To mitigate this, first, CIS proposes a lightweight allocation and placement scheduler to minimize switching time. Second, during inference execution, CIS reorders retraining samples to reduce switching frequency. Third, it temporarily reallocates GPU space from retraining tasks to inference tasks to address request queuing issue caused by the switching, with minimal impact on accuracy. Trace-driven experiments show that CIS achieves up to 25.1% higher accuracy without affecting SLO fulfillment, and requires 4× lower GPU cost compared to the existing method in achieving similar or higher accuracy.

Significant b reakthroughs in machine learning (ML) and the advantages of on-device processing have led to edge devices increasingly incorporating accelerators like GPUs, NPUs, and DSPs. However, these accelerators consume energy, prompting users to limit their floating-point precision. Many edge device users are in regions where including high-fidelity accelerators is too costly, leading to low-cost devices with low precision, sacrificing accuracy. Previous work predetermined layer assignments between the CPU and accelerator offline for high accuracy and low latency without considering the input, but we observe that input affects optimal layer assignment. To address this, we present Flex, a system for Fast, Accurate DNN Inference on Low-Cost Edges using Heterogeneous Accelerator eXecution. Leveraging common observations from models on various edge devices, Flex uses a lightweight heuristic and reinforcement learning (RL) to dynamically assign layers across the CPU and accelerator. Experiments show Flex improves average inference time by up to 39%, accuracy by up to 22%, and energy consumption by up to 61% compared to state-of-the-art methods, and is only 4.2% less optimal than the best achievable results.

The straggler problem has been extensively studied in CPU-based distributed deep learning (DL) training but has not received significant attention in homogeneous GPU-based distributed training, possibly because GPUs do not typically become bottlenecks in this scenario. In this paper, we conduct experiment measurements and find that the straggler problems persist in this scenario, primarily stemming from communication hurdles, compounded by computation delays, and stragglers substantially inflate resource consumption and training time by ∼50%. Existing straggler mitigation methods do not directly address the communication stragglers in this scenario, and they suffer from drawbacks such as prolonged latency in straggler removal, high resource consumption, or compromised training accuracy. To tackle these limitations, based on the insights derived from thorough measurements, we propose a Straggler-aware Time and Resource Efficient distributed DL Training system (STRET). STRET is tailored for both homogeneous and heterogeneous GPU-based distributed training, encompassing both the parameter server (PS) and all-reduce architectures. It creates a hybrid architecture that connects a straggler to a non-straggler possessing high communication bandwidth with it to reduce communication delay. If this method fails to eliminate stragglers, it runs two complementary methods in sequence to remove the stragglers. First, it further reduces communication overhead by withholding reporting gradients when the accuracy increment is marginal. Second, it conducts one-time batch size tuning to reduce iteration time. Real experimental results on TensorFlow show that STRET can reduce up to 56% and 41% training time and save up to 94% and 96% resources in the heterogeneous and homogeneous scenarios, respectively, compared to state-of-the-art approaches while preserving accuracy.

Disaggregated Large Language Model (LLM) inference decouples the compute-intensive prefill stage from the memory-intensive decode stage, allowing low-end, compute-focused GPUs for prefill and high-end, memory-rich GPUs for decode, which reduces cost while maintaining high throughput. However, transmitting Key-Value (KV) data between the two stages can be a bottleneck, especially for long prompts. Additionally, the computational overhead in the two stages is key for optimizing Job Completion Time (JCT), and KV data size can become prohibitive for long prompts and sequences. Existing KV quantization methods can alleviate transmission and memory bottlenecks, but they introduce significant dequantization overhead, exacerbating the computation time.

Effective dispatching of rescue teams under flooding disasters is crucial. However, previous methods are either incapable of handling flooding disaster situations, or cannot accurately estimate the distribution of rescue requests and accordingly adjust the search of the rescue teams. We proposeMobiRescue, a humanMobility basedRescueteam dispatching system, which aims to maximize the total number of rescued people, minimize the rescue delay and the number of serving rescue teams. We studied a city-scale human mobility dataset collected under the Hurricane Florence, and observed that several natural and demographic factors are closely related to impact severity, and road segment passability must be considered. Accordingly, we first propose a Support Vector Machine based method to predict the distribution of rescue requests considering the disaster-related factors. Then, we design an Euler path based method to determine the search paths for rescue team dispatching. Subsequently, we develop a Reinforcement Learning based method to guide the search of the rescue teams. Finally, we design a multi-objective optimization problem based method to adapt to the changed road segment passability. Our experiments demonstrate that compared with the other methods,MobiRescueincreases the total number of timely served rescue requests by 43.4% in average.

Connected and autonomous vehicles (CAVs) will be equipped with onboard deep neural network (DNN) models for processing the data from different sensors and communication units in CAVs and on the roads. In the CAV scenario, each vehicle receives time-series driving signals (e.g., speed, brake status) from nearby vehicles, uses its onboard DNN model for driving maneuver prediction for the nearby vehicles, and then takes corresponding actions for safe and efficient driving. Several black-box adversarial attacks for DNN maneuver classifiers in the CAV scenario have been proposed, in which an attacker deliberately sends false driving signals (i.e., add minimum perturbation to the predicted input) to its nearby vehicle to fool its onboard DNN model to output a wrong class label and cause unwanted traffic incidents or congestion. Though previous research proposed adversarial attack detection methods, the methods are for general DNN models rather than specifically for the time-series driving maneuver classification. To detect such adversarial attacks on the DNN maneuver classifiers in the CAV scenario, in this paper, we analyze the adversarial attacks and propose four different approaches comprised of both statistical and machine learning (ML) based methods. Our trace-driven and real experiments show the combination of all four approaches performs 14% better in accuracy rate and 12% better in precision rate compared to the state-of-the-art statistical and ML-based methods. These improvements are critical for driving safety in CAV scenarios.

Pipeline parallelism is a key mechanism to ensure the performance of large model serving systems. These systems need to deal with unpredictable online workloads with low latency and high good put. However, due to the specific characteristics of large models and resource constraints in pipeline parallelism, existing systems struggle to balance resource allocation across pipeline stages. The primary challenge resides in the differential distribution of requests across various stages of the pipeline. We propose QUART, a large model serving system that focuses on optimizing the performance of key stages in pipeline parallelism. QUART dynamically identifies the key stages of the pipeline and introduces an innovative two-level model parameter caching system based on forks to achieve rapid scaling of key stages within seconds. In evaluations with real-world request workloads, QUART reduces average response latency by up to 87.1%) and increases good put by 2.37x compared to the baseline. The experiments demonstrate that QUART effectively reduces tail latency and the average queue length of the pipeline.

Federated Learning (FL) has been increasingly used in various edge device applications. Cluster-wise separate Federated Learning (CFL) is an effective means of addressing the low accuracy issue resulting from data heterogeneity in FL. In CFL, nearby edge devices form a cluster and each cluster trains a separate machine learning (ML) model. As many edge devices (e.g., cellular phones) are mobile, a device may leave its cluster during training, which degrades the training accuracy. However, the existing works on CFL do not consider the mobility of the devices. We propose enabling such devices to keep participating in the trainings for its previous clusters and also the current cluster. However, due to constrained resources and high workload of multiple CFLs, such devices may become stragglers, affecting training time and accuracy. Also, cluster environment changes and less-visited clusters will generate low model accuracy. In this paper, we first conducted experimental analysis to verify the motivation and illustrate the problems. Then, to address the problems, we propose Clustered Federated Learning for Mobile edge devices (CFLM). CFLM decreases training time by sharing training data and computation results between multiple trainings. Additionally, CFLM increases accuracy by handling dynamic environment and by increasing the accuracy of lessvisited clusters. Our extensive evaluations on both CPU and GPU devices show that CFLM decreases training time by up to 68% and increases accuracy by up to 18% compared to the existing works.

Delay Tolerant Networks (DTNs) are wireless mobile networks resilient to disruptions, significant latency, and incomplete paths between source and destination (S-D). Vehicular Delay Tolerant Network (VDTNs) is a subclass of DTNs. Most existing routing algorithms for VDTNs rely on predetermined rules that are inflexible to changes in node features (such as location, contact history, and node relationships). Moreover, given the extremely dynamic network architecture of the VDTN, the routing protocols must be flexible to the varying node features. Thus, the purpose of this work is to develop an intelligent routing system capable of determining the optimal routing method given a pair of S-D nodes. In this study, a novel self-adaptive routing method for VDTNs based on machine learning (ML) is proposed. Using real-time features of S-D nodes, the proposed method selects the optimal routing method. Using the Opportunistic Network Environment (ONE) simulator based on a real-world taxi trace, we evaluated the performance of our ML-based self-adaptive routing method employing various ML models. Evaluation results show that our method achieves up to 34.34% greater success rate and up to 23.75% lower average message delivery delay than existing methods.

With the emergence of edge computing along with its local computation advantage over the cloud, methods for distributed deep learning (DL) training on edge nodes have been proposed. The increasing scale of DL models and large training dataset poses a challenge to run such jobs in one edge node due to resource constraints. However, the proposed methods either run the entire model in one edge node, collect all training data into one edge node, or still involve the remote cloud. To handle the challenge, we propose a fully distributed training system that realizes both Data and Model Parallelism over a network of edge devices (called DMP). It clusters the edge nodes to build a training structure by taking advantage of the feature that distributed edge nodes sense data for training. For each cluster, we propose a heuristic and a Reinforcement Learning (RL) based algorithm to handle the problem of how to partition a DL model and assign the partitions to edge nodes for model parallelism to minimize the overall training time. Taking advantage of the feature that geographically close edge nodes sense similar data, we further propose two schemes to avoid transferring duplicated data to the first-layer edge node as training data without compromising accuracy. Our container-based emulation and real edge node experiments show that our systems reduce up to 44% training time while maintaining the accuracy comparing with the state-of-the-art approaches. We also open sourced our source code.

To reduce the Deep Learning (DL) model training time and hence resource consumption, it is critical to avoid stragglers. However, the dynamics and uncertainty features of resource availability pose a challenge to avoiding stragglers caused. To handle this challenge, we propose a Straggler-Avoiding job Scheduling approach (SAS), which smartly ensures that the tasks of a job receive resources with similar dynamics and uncertainty so that the tasks can complete at approximately the same time. Specifically, SAS uses an ML method to predict available resource amounts with probability in future times, groups nodes with similar available resource amounts and probabilities, and then assigns each job to one node group with the objective of minimizing job completion time (JCT). To reduce the decision making time, we also propose a reinforcement learning (RL) based scheduling approach (SAS-RL) that assigns each job to a node group. In addition, we propose a distributed parameter server (PS) load reassignment method to handle PS stragglers. Our trace-driven real experiments show that SAS reduce up to 45% JCT and 63% stragglers compared with existing job schedulers, and our PS load reassignment reduces up to 48% JCT compared with the previous PS load distribution scheme.

Compared to previous epidemics, COVID-19 spreads much faster in people gatherings. Thus, we need not only more accurate epidemic spread prediction considering the people gatherings but also more time-efficient prediction for taking actions (e.g., allocating medical equipments) in time. Motivated by this, we analyzed a time-varying people mobility graph of the United States (US) for one year and the effectiveness of previous methods in handling time-varying graphs. We identified several factors that influence COVID-19 spread and observed that some graph changes are transient, which degrades the effectiveness of the previous graph repartitioning and replication methods in distributed graph processing since they generate more time overhead than saved time. Based on the analysis, we propose an accurate and time-efficient Distributed Epidemic Spread Prediction system (DESP). First, DESP incorporates the factors into a previous prediction model to increase the prediction accuracy. Second, DESP conducts repartitioning and replication only when a graph change is stable for a certain time period (predicted using machine learning) to ensure the operation improves time-efficiency. We conducted extensive experiments on Amazon AWS based on real people movement datasets. Experimental results show DESP reduces communication time by up to 52%, while enhancing accuracy by up to 24% compared to existing methods.

Various audio and video applications rely on multiple deep neural network (DNN) models deployed on edge servers to conduct inference with ms-level latency service-level-objectives (SLOs). To avoid accuracy decreases caused by data drift, continual retraining is necessary. However, this poses a challenge for GPU resource allocation to satisfy the tight SLOs while maintaining high accuracy in this scenario. There has been no research devoted to tackling this issue. In this paper, we conducted trace-based experimental analysis in this particular scenario, which shows that different models have varying degrees of impact from data drift, incremental retraining (proposed by us that retrains certain samples before inference) and early-exit model structures can help increase accuracy, and the interdependencies among tasks may lead to significant CPU-GPU memory communications. Leveraging these unique observations, we propose a data drift Adaptive scheduler for accurate and SLO-guaranteed Inference serving at edge servers (AdaInf). AdaInf uses incremental retraining and allocates GPU amount among applications based on their SLOs. For each application, it splits GPU time between retraining and inference to satisfy its SLO, and then allocates GPU time among retraining tasks based on their impact degrees. In addition, AdaInf proposes strategies that leverage the job features in this scenario to reduce the impact of CPU-GPU memory communications on latency. Our real trace-driven experimental evaluation shows that AdaInf can increase accuracy by up to 21% and reduce SLO violations by up to 54% compared to existing methods. Achieving similar accuracy as AdaInf requires 4× more GPU resources on the edge server for the existing method.

An autonomous vehicle (AV) is a combination of subsystems, measuring its driving environments with different sensors (e.g., camera, RADAR, and LiDAR) in real time. AVs follow their control policies and make real-time control decisions based on sensor measurements to ensure driving safety. Control policies in an AV are usually implemented with codes and not open to the public and drivers, which results in people’s strong concerns about driving safety. In this article, we propose a data-driven control policy-based driving safety analysis system (PoSa) to analyze the driving safety of a target AV. In PoSa, we first build a data-driven control policy extraction method to extract control policies of a target AV based on its historical driving data. Then, we develop a hazard driving scenario identification method to identify possible hazard driving scenarios of a target AV by executing the extracted control policies under different driving scenarios. Finally, we use vehicle driving data from one industry-standard AV platform (Baidu Apollo) to evaluate PoSa’s hazard driving scenario identification performance. We compared its identification results with Baidu AV accident reports from California DMV and the hazard driving scenario identification results cover as many as 89% hazard driving scenarios in the Baidu AV accident report, which demonstrates that PoSa has good performance on identifying hazard driving scenarios and its identification results can be used to optimize control policies for driving safety improvement.

Pure electric vehicles (EVs) become more and more popular in current automotive markets. With the wide application of EVs in the market, more EVs will be charged in residential homes. An advanced metering infrastructure measures real-time electricity consumption of individual residential homes. However, the advanced metering infrastructure is exposed to attacks given the fact that its components (e.g., smart meters) are located at public places. Under this situation, electricity consumption information of an individual residential home with an EV can be easily stolen. In this article, we propose an EV trip information inference system (TIIS) to infer EV trip information (origin-destination of each driving trip in a day) of a residential home based on its real-time electricity consumption measurements in a day. To the best of our knowledge, this is the first driving privacy relevant work caused only by residential electricity consumption data. For a residential home with an EV, TIIS derives the relationship between trip numbers in a day and charging-starting time, and the relationship between total driving distance and total charging energy. To derive the EV trip information of a residential home, TIIS first extracts an EV charging profile from its electricity consumption data and identifies EV model type and trip numbers as well as total driving distance based on EV charging profiles. Then, TIIS infers EV trip information based on the identified information. We used electricity consumption data and EV driving data to evaluate inference performance of TIIS. The experimental results demonstrate TIIS has as high as 81% trip inference accuracy and works on residential homes with different EV model types.

In this article, we demonstrate that by using a temporal sequence of applied brake signals collected from a vehicle, attackers can still possibly infer the vehicle’s route over the period, even though brake-signal data does not reveal any specific location information. Our route inference is basically composed of three steps. At first, we categorize brake-signal subsequences into four different driving maneuvers (i.e., stopping from a certain speed, reducing speed to adjust with the traffic flow, and taking left and right turns). Second, we estimate the number of intersections traversed by the vehicle using the applied brake signals and their corresponding maneuvers. Finally, we design a graph-based route-selection algorithm to find a list of (paths) routes from the regional map using the predicted driving maneuvers and the speed profile. We evaluate our approach using over 450 km of transportation data, which has been collected from 25 individuals. The experimental results demonstrate that, by resorting to our solution, 92.04% of the original drivers’ trajectory can be successfully recovered from their brake data regardless of driver and vehicle models.

In data-intensive parallel computing clusters, it is important to provide deadline-guaranteed service to jobs while minimizing resource usage (e.g., network bandwidth and energy). Under the current computing framework (that first allocates data and then schedules jobs), in a busy cluster with many jobs, it is difficult to achieve high data locality (hence low bandwidth consumption), deadline guarantee, and high energy savings simultaneously. We model the problem to simultaneously achieve these three objectives using integer programming. Due to the NP-hardness of the problem, we propose a heuristic Cooperative job Scheduling and data Allocation method (CSA).CSAnovelly reverses the order of data allocation and job scheduling in the current computing framework. Job-scheduling-first enablesCSAto proactively consolidate tasks with more common requested data to the same server when conducting deadline-aware scheduling, and also consolidate the tasks to as few servers as possible to maximize energy savings. This facilitates the subsequent data allocation step to allocate a data block to the server that hosts most of this data's requester tasks, thus maximally enhancing data locality. To achieve the tradeoff between data locality and energy savings with specified weights,CSAhas a cooperative recursive refinement process that recursively adjusts the job schedule and data allocation schedule. We further propose two enhancement algorithms (i.e., minimum k-cut data reallocation algorithm and bipartite based task reassignment algorithm) to further improve the performance ofCSAthrough additional data reallocation and task reassignment, respectively. Trace-driven experiments in the simulation and the real cluster show thatCSAoutperforms other schedulers in supplying deadline-guarantee and resource-efficient services and the effectiveness of each enhancement. Also, the enhancement algorithms are effective in improvingCSA.

With the advancement of dynamic wireless charging for Electric Vehicles (EVs), Mobile Energy Disseminator (MED), which can charge an EV in motion, becomes available. However, existing wireless charging scheduling methods for wireless sensors, which are the most related works to MED deployment, are not directly applicable for city-scale EV-to-EV dynamic wireless charging. We presentMobiCharger:aMobile wirelessChargerguidance system that determines the number of serving MEDs, and their optimal routes. We studied a metropolitan-scale vehicle mobility dataset, and found: most vehicles have routines, and the number of driving EVs changes over time, which means MED deployment should adaptively change as well. We combine EVs' current trajectories and routines to estimate EV density and the cruising graph for MED coverage. Then, we develop an offline MED deployment method that utilizes multi-objective optimization to determine the number of serving MEDs and the driving route of each MED, and an online method that utilizes Reinforcement Learning to adjust the MED deployment when the real-time vehicle traffic changes. Our trace-driven experiments show that compared with previous methods,MobiChargerincreases the medium State-of-Charge of all EVs by 50% during all time slots, and the number of charges of EVs by almost 100%.

With the rapid proliferation of Machine Learning (ML) and Deep learning (DL) applications running on modern platforms, it is crucial to satisfy application performance requirements such as meeting deadline and ensuring accuracy. To this end, researchers have proposed several job schedulers for ML clusters. However, none of the previously proposed schedulers consider ML model parallelism, though it has been proposed as an approach to increase the efficiency of running large-scale ML and DL jobs. Thus, in this paper, we propose an ML job Feature based job Scheduling system (MLFS) for ML clusters running both data parallelism and model parallelism ML jobs. MLFS first uses a heuristic scheduling method that considers an ML job’s spatial and temporal features to determine task priority for job queue ordering in order to improve job completion time (JCT) and accuracy performance. It uses the data from the heuristic scheduling method for training a deep reinforcement learning (RL) model. After the RL model is well trained, it then switches to the RL method to automatically make decisions on job scheduling. In addition, MLFS has a system load control method that selects tasks from overloaded servers to move to underloaded servers based on task priority, and also intelligently removes the tasks that generate little or no improvement on the desired accuracy performance when the system is overloaded to improve JCT and accuracy by job deadline. Furthermore, we propose Optimal ML iteration stopping method that determines the proper time to stop training ML model when this model reaches the minimum loss value. Our real experiments and large-scale simulation based on real trace show that MLFS reduces JCT by up to 53% and makespan by up to 52%, and improves accuracy by up to 64% when compared with existing ML job schedulers. We also open sourced our code.

Renewable energy supply is a promising solution for datacenters' increasing electricity monetary cost, energy consumption and harmful gas emissions. However, due to the instability of renewable energy, insufficient renewable energy supply may lead to the use of stored energy or brown energy. To handle this problem, in this paper, we propose an instability-resilient renewable energy allocation system. We define a job's service-level-objective (SLO) as the successful running probability by only using supplied renewable energy. The system allocates jobs with the same SLO level to the same physical machine (PM) group, and powers each PM group with renewable energy generators that have probability no less than its SLO to produce the amount no less than its energy demand. We use a deep learning technique to predict the probability of producing the amount no less than each value of each renewable energy source, and predict the energy demands of each PM area. We formulate an optimization problem to match renewable energy resources with different instabilities to different PM groups for supply, and use reinforcement learning method and linear programming method to solve it. We further propose an energy-driven computing resource assignment method, which adjusts the amount of computing resource of each job based on job deadline and failure probability in each PM group, and a failure prediction based energy saving method. Real trace driven experiments show that our methods achieve much lower SLO violations, total energy monetary cost and total carbon emission compared to other methods and the effectiveness of individual methods.

Evacuation during natural disasters is an important preventive measure. Previous works on evacuation traffic analysis and evacuation path planning problems are mainly based on small size road networks and human mobility datasets. In this paper, we use a large-scale human mobility dataset of the North Carolina (NC) state during hurricane Florence in 2018 in designing an emergency and safe evacuation path planning system (ESEP). ESEP consists of three main parts: mobility dataset analysis, latent variable neural network model based evacuation prediction model, and evacuation path planning system to evacuate people within the shortest possible time while ensuring maximum connectivity to safe cities along the routing path at minimum possible risk of being affected by the hurricane and also avoiding congestion. We additionally propose a map partitioning based method for dividing the problem into several smaller equivalent sub-problems to reduce the computation complexity. Our trace-driven evaluations show that the ESEP-suggested evacuation paths achieve better performance in terms of all design factors. The experimental result also shows that the map partitioning based method reduces the computation time of finding evacuation paths by 5 0% while maintaining the most objective values within 10% of the global optimal solution.

With the increased usage of edge devices having local computation capabilities, deep neural network (DNN) training in a network of edge devices becomes promising. Several recent works have proposed fully edge-based distributed training systems for situations when the communication to cloud is unstable or intermittent. However, such distributed systems become vulnerable when there are untrusted devices that launch data and model poisoning attacks during training, deteriorating the accuracy of the DNN model. To handle this challenge, we propose a Trustworthy distributed system for Machine learning training in an edge device network (TrustMe). TrustMe realizes both data and model parallelisms. It detects the untrusted devices producing illegitimate outputs. Next, it reassigns the training tasks of the untrusted devices to other trusted devices in such a way that the reassignment and the training that is restarted after the reassignment require minimal time. Our container-based emulation and real device experiments demonstrate that TrustMe achieves up to 12% higher accuracy and 45% less training time compared to existing methods in the presence of untrusted devices.

With the emergence of edge devices along with their local computation advantage over the cloud, distributed deep learning (DL) training on edge nodes becomes promising. In such a method, the cluster head of a cluster of edge nodes schedules all the DL training jobs from the cluster nodes. Using such a centralized scheduling method, the cluster head knows all the loads of the cluster nodes, which can avoid overloading the cluster nodes, but the head itself may become overloaded. To handle this problem, we first propose a multi-agent RL (MARL) system that enables each edge node to schedule its own jobs using RL. However, without the coordination between the nodes, action collision may occur, in which multiple nodes may schedule tasks to the same node and make it overloaded. To avoid these problems, we propose a system called Shielded ReinfOrcement learning (RL) based DL training on Edges (SROLE). In SROLE, each edge node schedules its own jobs using multi-agent RL. The shield deployed in a node checks action collisions and provides alternative actions to avoid the collisions. As the central shield node for the entire cluster may become a bottleneck, we further propose a decentralized shielding method, in which different shields are responsible for different regions in the cluster and they coordinate to avoid action collisions on the region boundaries. Our container-based emulation experiments show that SROLE reduces training time by up to 59% with 29% lower median resource utilization and reduces the number of action collisions by up to 48% compared to multi-agent RL and the centralized RL. Our real device experiments show that SROLE still reduces the training time by up to 53% with 28% lower median resource utilization than multi-agent RL and the centralized RL.

The challenges of low latency, high throughput datacenter networks create new traffic management problems that require new congestion control mechanisms. Generally, the proposals to solve this problem have focused either on refining existing window-based congestion control like in TCP or on introducing a central controller to make congestion control decisions. In this paper, we propose a third approach, where nodes share network information with their neighbors and apply this information to make local decisions that limit global congestion. In our implementation, the rate limiting decisions on one node are driven by the local agent that uses reinforcement learning to optimize a combination of overall latency, throughput and the shared information. To make this approach efficient, the local agents choose overall rate limits for each node, and then a separate process assigns the traffic of individual flows within these limits. We show that, in trace-driven real implementation, our method achieves better congestion avoidance than several end-to-end and centralized mechanisms in prior work.

Recent research has demonstrated that machine learning algorithms are vulnerable to adversarial attacks, in which small but carefully crafted input perturbations can lead to algorithm failure. It has been demonstrated that certain adversarial attack algorithms are capable of producing these types of perturbations. These attack methods are inapplicable when the attack must be generated in near real time. The use of a hardware accelerator, such as a Process-in-Memory (PIM) archi-tecture, is a potential method for addressing this issue. The PIM architecture is regarded as a superior option for data-intensive applications such as solving optimization problems and Deep Neural Networks (DNN) due to its capacity for ultra-low-latency parallel processing. However, implementing an adversarial attack algorithm directly on the PIM platform is inefficient due to the PIM architecture's complexity and overhead costs. To address this issue, we utilize a novel adversarial attack scheme based on the PIM that leverages Look-up-Table (LUT)-based processing. The proposed LUT-based PIM architecture is capable of being dynamically programmed to execute the operations necessary for an adversarial attack algorithm. Our simulations reveal that the proposed method is capable of achieving an ultra-low operating delay and energy-efficiency performance.

Previous methods for passenger demand inference are unable to capture the effect of all possible random factors (e.g., accident and weather), hence resulting in insufficient accuracy. Moreover, due to the lack of charging optimization, existing taxicab dispatching methods cannot be applied to electric taxicabs directly. We propose CD-Guide, which provides Charging and Dispatching Guide for electric taxicabs based on customized selection and training of historical passenger demand data, multiobjective optimization, and reinforcement learning (RL). By analyzing a large-scale electric taxicab data set, we found that: 1) the histogram of passengers’ origin buildings is effective in illustrating the suitability of historical data for learning; 2) passenger demands in different regions vary a lot due to various random factors; and 3) charging time must be considered in dispatching electric taxicabs. We first develop a passenger demand inference model based on customized selection and training of suitable historical passenger demand data. Then, we develop two taxicab guidance methods that utilize multiobjective optimization and RL, respectively, to maximize the taxicab’s likelihood of finding passengers, maximally prevent the taxicab from missing passengers due to charging, and, meanwhile, maintain the continuous service of the taxicab. Extensive experiments on real-world data sets demonstrate that compared with the state of the art, CD-Guide increases the total number of served passengers by 100%, and the minimum State-of-Charge of all taxicabs by 75% during all time slots.

In metropolitan areas with heavy transit demands, electric vehicles (EVs) are expected to be continuously driving without recharging downtime. Wireless power transfer (WPT) provides a promising solution for in-motion EV charging. Nevertheless, previous works are not directly applicable for the deployment of in-motion wireless chargers due to their different charging characteristics. The challenge of deploying in-motion wireless chargers to support the continuous driving of EVs in a metropolitan road network with the minimum cost remains unsolved. We proposeCatChargerto tackle this challenge. By analyzing a metropolitan-scale data set, we found that traffic attributes like vehicle passing speed, daily visit frequency at intersections (i.e., landmarks), and their variances are diverse, and these attributes are critical to in-motion wireless charging performance. Driven by these observations, we first group landmarks with similar attribute values using the entropy minimization clustering method, and select candidate landmarks from the groups with suitable attribute values. Then, we use the kernel density estimator (KDE) to deduce the expected vehicle residual energy at each candidate landmark and consider EV drivers’ routing choice behavior in charger deployment. Finally, we determine the deployment locations by formulating and solving a multiobjective optimization problem, which maximizes vehicle traffic flow at charger deployment positions while guaranteeing the continuous driving of EVs at each landmark. Trace-driven experiments demonstrate thatCatChargerincreases the ratio of driving EVs at the end of a day by 12.5% under the same deployment cost.

Reducing tail latency becomes increasingly important to improve the user-perceived service experience. User-facing latency-sensitive cloud applications typically contain multiple interactive tiers (e.g., Web, App, Database) running in different virtual machines (VMs) with complex interaction patterns. However, such interactions between VMs in different tiers are often neglected in previous VM consolidation methods, resulting in poor application performance. In this article, we study the consolidation of multi-tier interactive workloads from a new perspective of user-perceived tail latency. We propose a novel profiling-based consolidation methodology to satisfy tail latency requirements while reducing the number of used physical machines. To achieve such a goal, we first perform large-scale profiling experiments under various consolidation settings in a KVM virtualized private cluster to establish the empirical performance values. We consider two key factors that affect the tail latency of multi-tier workloads:interferencewith co-located VMs andinteractionbetween tiers. We model the consolidation of multi-tier workloads as an optimization problem with different objectives and constraints, and derive the consolidation schedule. We implement and evaluate the proposed models, as well as comparing with other methods (i.e.,withoutprofiling orwithoutconsidering interaction influence). Extensive experimental results show that the proposed method is able to reduce up to5Xtail latency, compared with the methodwithoutprofiling and up to1.3Xtail latency, compared with the methodwithoutconsidering the interaction influence between different tiers.

For a modern vehicle, if the sensor in a vehicle anti-lock braking system (ABS) or controller area network (CAN) bus is attacked during a brake process, the vehicle will lose driving direction control and the driver’s life will be highly threatened. However, current methods for detecting attacks are not sufficiently accurate, and no method can provide attack mitigation. To ensure vehicle ABS security, we propose an attack detection method to accurately detect both sensor attack (SA) and CAN bus attack in a vehicle ABS, and an attack mitigation strategy to mitigate their negative effects on the vehicle ABS. In our attack detection method, we build a vehicle state space equation that considers the real-time road friction coefficient to predict vehicle states (i.e., wheel speed and longitudinal brake force) with their previous values. Based on sets of historical measured vehicle states, we develop a search algorithm to find out attack changes (vehicle state changes because of attack) by minimizing errors between the predicted vehicle states and the measured vehicle states. In our attack mitigation strategy, attack changes are subtracted from the measured vehicle states to generate correct vehicle states for a vehicle ABS. We conducted the first real SA experiments to show how a magnet affects sensor readings. Our simulation results demonstrate that our attack detection method can detect SA and CAN bus attack more accurately compared with existing methods, and also that our attack mitigation strategy almost eliminates the attack’s effects on a vehicle ABS.

To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the highreconfigurationdelay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation ofCo-scheduler, a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computesguidelineson how many racks to place the job’s input data and the job’s tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

Even though memory denial-of-service attacks can cause severe performance degradations onco-locatedvirtual machines, a previous detection scheme against such attacks cannot accurately detect the attacks and also generates high detection delay and high performance overhead since it assumes that cache-related statistics of an application follow the same probability distribution at all times, which may not be true for all types of applications. In this paper, we present the experimental results showing the impacts of memory DoS attacks on different types of cloud-based applications. Based on these results, we propose two lightweight and responsive Statistical based Detection Schemes (SDS/B and SDS/P) that can detect such attacks accurately. SDS/B constructs a profile of normal range of cache-related statistics for all applications and use statistical methods to infer an attack when the real-time collected statistics exceed this normal range, while SDS/P exploits the increased periods of access patterns for periodic applications to infer an attack. Upon SDS, we further leverage deep neural network (DNN) techniques to design a DNN-based detection scheme that is general to various types of applications and more robust to adaptive attack scenarios. Our evaluation results show that SDS/B, SDS/P and DNN outperform the state-of-the-art detection scheme, e.g., with 65% higher specificity, 40% shorter detection delay, and 7% less performance overhead. We also discuss how to use SDS and DNN-based detection schemes under different situations.

In Named Data Networking, cache pollution, cache poisoning and interest flooding are three popular types of attacks that can drastically degrade the network performance. However, previous methods for mitigating these attacks are not sufficiently effective or efficient. Also, they cannot simultaneously handle the three attacks, or the case that core routers or edge routers are compromised. To handle these problems, we propose an effective and lightweight countermeasure scheme. It consists of token-based router monitoring policy (TRM), hierarchical consensus-based trust management (HCT), and popularity-based probabilistic caching and caching replacement policy (PPC). In TRM, each edge router monitors and evaluates each data requester’s probability of launching the cache pollution attack and each data provider’s probability of launching the cache poisoning attack, and accordingly assigns, rewards and penalizes tokens to them to control their data request and data provision activities. Thus, the interest flooding attack can also be mitigated by limiting the consumption of tokens. In HCT, each core router manages its directly connected edge routers using TRM, and the core routers trust each other through adopting the concept of consensus in Blockchain. Thus, the edge and core routers executing monitoring and evaluation are trustable. PPC uses probabilistic caching and caching replacement based on the popularity of received content to further mitigate the attacks and reduce caching and data verification overhead. Results from simulation experiments demonstrate that our proposed scheme has better performance, in terms of interest satisfaction ratio and average end-to-end delay than current mechanisms.

Live virtual machine migration technique allows migrating an entire OS with running applications from one physical host to another, while keeping all services available without interruption. It provides a flexible and powerful way to balance system load, save power, and tolerate faults in data centers. Meanwhile, with the stringent requirements of latency, scalability, and availability, an increasing number of applications are deployed across distributed data-centers. However, existing live migration approaches still suffer from long downtime and serious performance degradation in cross data-center scenes due to the mass of dirty retransmission, which limits the ability of cross data-center scheduling. In this paper, we propose a system named Memory/disk operation aware Lightweight VM Live Migration across data-centers with low performance impact (MLLM). It significantly improves the cross data-center migration performance by reducing the amount of dirty data in the migration process. In MLLM, we predict disk read workingset (i.e., more frequently read contents) and memory write workingset (i.e., more frequently write contents) based on the access sequence traces. And then we adjust the migration models and data transfer sequence by the workingset information. We further proposed an improved algorithm for workingset estimation. Moreover, we discussed the potential use of machine learning (ML) to enhance the performance of the VM migration and also propose a two-level hierarchical network model to make the ML-based prediction more efficient. We implement MLLM and its improved versions on the QEMU/KVM platform and conduct several experiments. The experimental results show that 1) MLLM averagely reduces 62.9% of total migration time and 36.0% service downtime over existing methods; 2) The improved workingset estimation algorithm reduces 9.32% memory pre-copy time on average over the original algorithm.

In cloud systems, demand-prediction based resource provisioning schemes help assure the SLOs (service level objectives) of cloud tenants. We notice that if a provisioning scheme does not exclude bursts from historical resource demands in normal demand prediction or always uses a large padding to correct under-prediction, it will lead to resource over-provisioning and low resource utilization. To improve the previous schemes, in this paper, we present a Resource-efficient Predictive Resource Provisioning system in cloud systems (RPRP) that excludes bursts in demand prediction and has algorithms to specifically handle bursts to avoid resource over-provisioning. Rather than setting padding to a possibly high value, RPRP has a load-dependent padding algorithm that adaptively determines padding based on predicted demands. To handle bursts, RPRP has a burst-resilient shared padding algorithm that reserves resource shared by multiple co-located VMs rather than for individual VMs. It also embodies a responsive padding algorithm that adaptively adjusts padding to recover from both under-provisioning and over-provisioning. We implemented RPRP on top of Xen and conducted both trace-driven simulation and real-world testbed experiments. The experimental results show that RPRP achieves higher resource utilization, more accurate demand predictions, and fewer SLO violations than previous schemes.

A large-scale cloud data center needs to provide high service reliability and availability with low failure occurrence probability. However, current large-scale cloud data centers still face high failure rates due to many reasons such as hardware and software failures, which often result in task and job failures. Such failures can severely reduce the reliability of cloud services and also occupy huge amount of resources to recover the service from failures. Therefore, it is important to predict task or job failures before occurrence with high accuracy to avoid unexpected wastage. Many machine learning and deep learning based methods have been proposed for the task or job failure prediction by analyzing past system message logs and identifying the relationship between the data and the failures. In order to further improve the failure prediction accuracy of the previous machine learning and deep learning based methods, in this article, we propose a failure prediction algorithm based on multi-layer Bidirectional Long Short Term Memory (Bi-LSTM) to identify task and job failures in the cloud. The goal of Bi-LSTM failure prediction algorithm is to predict whether the tasks and jobs are failed or completed. The trace-driven experiments show that our algorithm outperforms other state-of-art prediction methods with 93 percent accuracy and 87 percent for task failure and job failures respectively.

Pandemic like COVID-19 poses severe challenges to public health and causes great damages to human lives and economy. Computational epidemiology allows the authorities (e.g., federal, state, city governments) to predict the future states of pandemic and take preventive measures based on that. However, due to widespread human mobility, epidemic prediction on a small-scale (e.g., city) may not produce effective results. Therefore, to obtain a clearer picture of pandemics, authorities often rely on large-scale human mobility data. Processing such a large graph dataset is highly computation intensive, and thus generates long latency. Distributed computing, in which the graph is partitioned and processed by multiple servers in parallel, is a solution to this problem. However, as human mobility changes over time, the graph varies over time, which may make the previous graph partition not effective anymore in limiting the communication overhead. In this paper, we study the existing works in the literature for processing large-scale human mobility graph data to predict pandemic spread. We conducted comprehensive experiments based on real-world large-scale human mobility data that covers the entire state of Virginia, USA. Based on the experimental evaluations, we present our findings and discuss the possible future research directions for the aforementioned challenge.

Epidemic simulation traditionally serves as one of the important methods to forecast how an epidemic may spread among a population. However, there are two key limitations that restrict the scope of such methods. The first limitation is that the existing tools rely on different sets of static parameters (e.g., infection probability, recovering probability) for simulating an epidemic spread that may fail to capture the dynamic nature of population interactions that acts as a dominant factor in an epidemic spread scenario such as COVID-19 pandemic. To handle this challenge, we propose a machine learning based model that combines a Graph Convolutional Neural Network (GCN) and a Recurrent Neural Network (RNN). It integrates the ability of the GCN to capture spatial dependency in human interaction and the ability of the RNN to incorporate temporal effects of the virus spread. The second limitation is that these methods do not address the computation overhead problem when dealing with time-dynamic graphs. Training a GCN on a very large graph suffers from the communication overhead from different graph partitions and the computation overheads stemming from partitioning dynamic graphs. This limitation impacts the scalability of the existing systems. To solve this challenge, we partition the graph in a computationally less expensive manner by partitioning the graph using the min-cut principle. We conducted comprehensive large scale real-world human mobility data driven experiments. Our experimental result shows that the proposed machine learning based forecasting model achieves overall 84% classification accuracy with greater than 72% precision and 62% recall. Also, the proposed graph partitioning approach reduces computation time and commutation overhead by a significant margin.

The COVID-19 pandemic is an ongoing pandemic of coronavirus disease since 2019. Millions of cases and deaths attributed to it have been confirmed in the world. So far the detection of COVID-19 heavily relies on the specialized tests (e.g., based on saliva or respiratory swabs). Some approaches use smart devices (e.g., Whoop) for coronavirus infection detection using respiratory rate. Machine learning (ML) techniques have become a promising approach for the coronavirus infection detection. Therefore, in this paper, we introduce a machine learning based COVID infection predictor. We measure the prediction accuracy of five ML models. We use Chi-square test and knowledge-based manual feature selection to select important features for prediction to reduce prediction time overhead without compromising prediction accuracy. We also study the accuracy with different input features (those that can be measured by medical devices and by smart devices) and find that removing some features has no or slight influence on the prediction accuracy. Since insufficient or unbalanced training data decreases the prediction accuracy, we further propose a Generative Adversarial Network (GAN) ML based predictor that produces synthetic data (close to real data) for ML training. Our extensive experiments show the effectiveness of our methods in improving the detection accuracy. Our study results can provide guidance on developing the coronavirus infection predictors based on different data sources and devices. We open sourced our code in GitHub.

In the cloud datacenter, the resource utilization of different virtual machine (VM) and physical machine (PM) varies with time and it may lead to SLO violation and then degrade the application performance. In order to minimize the probability of SLO violation, load balancing is used to dynamically migrate VMs from overloaded PMs to underloaded PMs. Previous load balancing methods fail to achieve long term load balance. To address this problem, in this paper, we propose different load balancing methods and evaluate their performance on several metrics. We use the Fast Fourier Transform (FFT) method, an improved FFT method considering more frequencies in FFT and the long short term memory (LSTM) machine learning model to predict the resource utilization of VM and PM in the future. LSTM can always achieve the best prediction performance in the prediction. Taking advantage of the ML technique, we then propose a heuristic based method and a reinforcement learning (RL) based method relying on ML workload prediction to generate the VM migration plan in the datacenter. We conduct experiments in both trace-driven simulation (based on Google cluster trace, PlanetLab trace, Worldcup trace) and real implementation in terms of the SLO violation rate, the number of migrations and overhead. The experimental results show that the workload prediction helps reduce the SLO violation rate and/or the number of migrations, which improves the load balance performance in a datacenter. Also, the RL based VM migration method outperforms the heuristic based method in a heavily loaded system but does not show obvious advantages in a lightly loaded system.

The current autonomous vehicles are equipped with onboard deep neural network (DNN) models to process the data from different sensor and communication units. In the connected autonomous vehicle (CAV) scenario, each vehicle receives time-series driving signals (e.g., speed, brake status) from nearby vehicles through the wireless communication technologies. In the CAV scenario, several black-box adversarial attacks have been proposed, in which an attacker deliberately sends false driving signals to its nearby vehicle to fool its onboard DNN model and cause unwanted traffic incidents. However, the previously proposed black-box adversarial attack can be easily detected. To handle this problem, in this paper, we propose a Suspicion-free Boundary Black-box Adversarial (SBBA) attack, where the attacker utilizes the DNN model's output to design the adversarial perturbation. First, we formulate the attack design problem as a goal satisfying optimization problem with constraints so that the proposed attack will not be easily detectable by detection methods. Second, we solve the proposed optimization problem using the Bayesian optimization method. In our Bayesian optimization framework, we use the Gaussian process to model the posterior distribution of the DNN model, and we use the knowledge gradient function to choose the next sample point. We devise a gradient estimation technique for the knowledge gradient method to reduce the solution searching time. Finally, we conduct extensive experimental evaluations using two real driving datasets. The experimental results show that SBBA outperforms the previous adversarial attacks by 56% higher success rate under detection methods, 238% less time to launch the attacks, and 76% less perturbation (to avoid being detected), and 257% fewer queries (to the DNN model to verify the attack success).

More and more web applications are deployed on the cloud storage services that store data objects of the web applications in the geo-distributed datacenters belonging to Cloud Service Providers (CSPs). In order to provide low request latency to the web application users, in the previous work, the web application developers need to store more data object replicas in a large number of datacenters or send redundant requests to multiple datacenters (e.g., closest datacenters), both of which increase monetary cost. In this paper, we conducted request latency measurement from a GENI server (as a client) to AWS S3 datacenters for one month, and our observations lay the foundation for our proposed system called GeoCol, a geo-distributed cloud storage system with low cost and latency using reinforcement learning (RL). To achieve the optimal tradeoff between the monetary cost and the request latency, GeoCol encompasses a request split method and a storage planning method. The request split method uses the SARIMA machine learning (ML) technique to predict the request latency as an input to an RL model to determine the number of sub-requests and the datacenter for each sub-request for a request in order to enable the parallel transmissions for a data object. In the storage planning method, each datacenter uses RL to determine whether each data object should be stored and the storage type of each stored data object. Our trace-driven experiment on AWS S3 and GENI platform shows that GeoCol outperforms other comparison methods in monetary cost with 32 % reduction and data object request latency with 51 % reduction.

Applications running in edge computing system seamlessly collect data, process data and take actions accordingly. In many cases, the applications need to assist people in real time, and even have life-or-death consequences such as heart attack detection in healthcare and object detection in driving, which requires low job latency. Moreover, power and bandwidth are constrained resources in edge computing systems. Therefore, a challenge is how to handle data efficiently to reduce job latency, and meanwhile reduce power and bandwidth consumption. Previous works mainly focus on where to store collected source data to reduce the communication latency for source data sharing. Noticing that intermediate and final processing results may be shared by many applications, we propose to store intermediate and final results for sharing to avoid the duplicated computation. We also propose data collection that reduces data collection frequency based on context-related factors to achieve an optimal tradeoff between the overhead and decision making accuracy. We further propose data redundancy elimination to reduce the redundant data transmitted between edge and fog nodes. Our combined data operation strategies show significant improvement over the state-of-the-art methods in terms job latency, power and bandwidth consumption for experiments on both simulated and real edge environment.

The rapid growth of cloud computing in cloud datacenters in recent decades greatly increases the brown energy consumption in datacenters, and hence significant increase of carbon emission that negatively impacts on the environment as well as the monetary cost. More and more cloud service providers are adopting renewable energy as the energy supply to offset the consumption of brown energy. Meanwhile, an increasing number of renewable energy generators have been built to meet the needs. However, the instability of the renewable energy cannot guarantee the support to the datacenter and the energy competition of different datacenters may lead to datacenter energy outage. In this paper, we focus on the problem of how to match different renewable energy generators to the datacenters from different cloud providers to minimize the carbon emission, monetary cost, and service level objective (SLO) violation due to renewable energy shortage. The challenges here are that the datacenters may compete in energy requesting, the renewable energy generation is not stable and the decision should be made quickly. There have been no previous efforts devoting to this problem. To solve the problem, we first test several machine learning techniques on long-term prediction accuracy on renewable energy generation and energy demand using real traces and identify SARIMA for the prediction. We then propose a multi-agent reinforcement learning based method (MARL) for each datacenter to determine how much renewable energy to request from each generator based on the predicted results. We also propose a deadline guaranteed job postponement method (DGJP) to postpone executing unurgent jobs upon insufficient renewable energy supply. The trace-driven experiments show that MARL outperforms other methods by increasing up to 35% SLO satisfaction ratio, and reducing up to 19% (0.33 billion dollars in 90 days) total monetary cost and 33% total carbon emission, and DGJP further improves the performance.

Autonomous vehicles (AVs) follow their control policies and make real-time decisions based on measured signals from sensors to ensure driving safety. With the development of sensor measurement technologies, an AV usually adopts more sensors to measure its driving environments and develops more control policies to satisfy people’s expectations on driving safety. However, these control policies are usually implemented with codes that are not open to the public and usually difficult to understand for people, which causes people’s strong concerns about AVs’ driving safety. In this paper, we propose a control policy based driving safety system (Polsa) to help improve $\overline{\mathrm{d}\mathrm{r}\mathrm{i}}$ving safety of a given AV. For a given AV, Polsa extracts its control policies and determines the safest control behavior among multiple control behaviors for each given trigger condition, which can be used by AV companies that produce the AV to improve the AV’s driving safety. Accordingly, first, Polsa has a control policy extraction method that uses dynamic time warping and k-means clustering technologies to cluster historical driving data with the same control behavior type together and then analyzes positions and driving speeds in each cluster to extract control policies of a target AV. Second, Polsa has an optimal control policy determination method to determine the safest control behavior for each given trigger condition. Unlike previous works that consider that the state of the target AV’s nearby vehicle is constant during a time period, Polsa considers time-varying driving state of its nearby vehicle, thus deriving more safer control behavior. We use an industry-standard AV platform (Baidu Apollo) to evaluate optimal control policy success rate of Polsa in comparison with two state-of-the-art methods. The comparison results show that Polsa can extract control policies with 83% accuracy, and improve optimal control policy success rate by 28% compared with existing methods, which demonstrates high performance of Polsa in extracting control policies and determining optimal control behavior.

The battery pack in an electric vehicle (EV) usually contains thousands of single cells and these cells are connected together to generate the required voltage and capacity. Cells usually become aged after a certain number of charging and discharging processes and State-of-Health (SOH) of each cell needs to be monitored periodically to ensure normal work of EVs. However, SOHs of cells cannot be directly measured in practice. Besides, aging propagation phenomena between cells (cells in the same battery pack have different SOHs and the aged cell accelerates aging processes of other nearby cells) exists in a battery pack. In this paper, we propose a data-driven Battery Health Estimation System (BaHeS) to estimate SOH of each cell in a battery pack based on cell information (e.g., voltage and temperature). BaHeS firstly proposes a voltage entropy based detection method to detect abnormal cells by calculating voltage entropies of all cells in a time period. And then, BaHeS builds a cell aging propagation model to model the effect of the detected abnormal cell on its nearby cell by estimating SOH change. Lastly, BaHeS applies a Long Short Term Memory (LSTM) based neural network with the estimated SOH change and cell information as inputs to estimate SOH of each cell. We used battery pack usage datasets from total 50 EVs to evaluate SOH estimation accuracy of BaHeS. The experimental results demonstrate that BaHeS has good SOH estimation performance with as high as 93% accuracy and improves estimation accuracy by 18% compared with existing methods.

Electronic control units and actuators in a modern vehicle transmit messages through a controller area network (CAN) bus to ensure driving safety. However, adversaries can easily inject abnormal messages into the CAN bus through an external interface to affect vehicle driving safety. Existing abnormal message detection methods only detect abnormal messages, which are included in their training data and do not consider individual vehicle’s message transmission behaviors, which makes existing methods difficult to detect all abnormal messages accurately and results in their limited application in different vehicles. In this paper, we propose a neural network based abnormal message detection system (NaDS), to detect abnormal messages in the CAN bus. In NaDS, we firstly build a Long Short-Term Memory (LSTM) network to detect both known abnormal messages (included in the training data) and unknown abnormal messages (not included in the training data) in the CAN bus. In the LSTM network, we select message IDs and data field values as network inputs and use a generative adversarial network (GAN) based abnormal message generator to train the LSTM network. We then develop a transfer learning method to transfer a pre-trained LSTM network from one vehicle into a new LSTM network based on a small amount of training data to detect abnormal messages in another vehicle. We use real-vehicle CAN bus message datasets including three different vehicle types to evaluate abnormal message detection accuracy of NaDS. The experimental results demonstrate that NaDS can improve abnormal message detection accuracy by 29% compared with existing methods and keep high abnormal message detection accuracy during network transfer processes.

Connected and autonomous vehicles have been introduced to increase roadway safety and traffic flow efficiency, where an autonomous vehicle shares its current and near-future driving maneuver information using different CAN-bus signals (e.g., speed, brake pedal pressure, and so on) with its nearby vehicles using wireless communication technologies. In order to classify these driving maneuver data, deep neural network (DNN) models are widely utilized as DNN models result in high prediction accuracy. However, the existing DNN-based classification models are not suitable for the driving maneuver time-series datasets, which are often imbalanced (i.e., different number of instances for different types of maneuvers). The raw driving signals also carry (univanted) traffic context information (e.g., congestions, signs, curvatures, etc.), which can significantly reduce the driving maneuver classification accuracy. This paper presents a Deep Driving Maneuver Classification (DeepDMC) framework to classify different driving maneuvers more accurately. There are three modules in the proposed framework: maneuver classification, context discrimination, and maneuver generation. First, the maneuver classification module takes both labeled and unlabeled data to predict different maneuvers. Second, the context discriminator module detects different traffic context information from the data, and then reduces traffic contexts’ impacts to classify different driving maneuvers. Third, the maneuver generation module generates synthetic maneuver data to handle the imbalanced data problem. To better understand the proposed framework’s robustness, we evaluate the proposed framework with other existing DNN-based classification models using two real driving datasets. From the experimental evaluations, we find that the proposed DeepDMC framework outperforms other existing classification models by up to 42.82% and 37.53% in terms of precision and recall values, respectively.

Deep Neural Network (DNN) models are expected to be widely used in self-driven autonomous vehicles to understand surrounding environments and enhance driving safety. In this paper, we propose a Fast Black-box Adversarial (FBA) attack for time-series DNN models in connected autonomous vehicle (CAV) scenarios. In this attack, an attacker sends false driving signals to a vehicle to misclassify its DNN model (e.g., maintaining speed is misclassified to stopping). Though different black-box adversarial attacks have been proposed previously, they are mainly for image classification, which cannot be directly adopted in the CAV scenarios due to two challenges. First, the attack needs to be generated in near real time. Second, it should not be noticeable based on the driving time-series signals. To handle these two challenges, FBA consists of two steps for the adversarial signal generation: offline and online. First, based on our real data analysis observation that each driving maneuver has maneuver-specific similar patterns (in the time-series) regardless of drivers or vehicles, FBA finds the influential input portion for each maneuver as the offline adversarial signal portion. Second, given a benign driving signal input, FBA replaces its influential input portion with the offline adversarial signal portion and smooths the signals, and uses this input as the initial solution to find the optimal perturbation (that leads to successful attack while minimizing the perturbation values) online using the zeroth-order gradient descent method. It significantly reduces the time to find the optimal perturbation since the initial solution is closer to the optimal solution. Our experiments based on real-driving datasets show the effectiveness of FBA in dealing with the two challenges compared with the existing black-box adversarial attacks.

Many outages of cloud computing services are caused by the natural disasters or common causes such as software failure, hardware failure, cyber attacks, and power outage. As a result, it is critical to develop resilient and robust cyber-physical system (CPS) to support continuity of service for smart and connected communities (S&CC). In spite of considerate research efforts on virtual machine (VM) migration for enhancing failure-resilience of datacenters, one issue still needs to be effectively addressed: how to determine the best destination for VM migration to avoid the influence from a given failure and enable edge nodes to connect to the VMs continuously. In this paper, we aim to handle these important issues to build a resilient and robust edge-cloud (or fog) network system supporting CPS for S&CC. We propose a machine learning based method for VM migration destination determination for the VMs in the predicted failure domains. We also propose a continuous edge-cloud connection method in wireless network component that enables edge nodes to continuously connect with the VMs through intermediate edge nodes when they cannot directly connect to their original VMs due to VM migration. Our experimental results show our proposed system reduces the number of job failures by 1.8 times and reduces the total job completion time by 48% for completed jobs compared to the case without our system.

With the fast development of autonomous vehicle (AV) technology and possible popularity of AVs in the near future, a mixed-vehicle type driving environment where both AVs and their surrounding human-driving vehicles drive on the same road will exist and last for a long time. An AV measures its driving environments in real time and make control decisions to ensure driving safety. However, surrounding human-driving vehicles may conduct aggressive driving behaviors (e.g., sudden deceleration, sudden acceleration, sudden left or right lane change) in practice, which requires an AV to make correct control decisions to eliminate the effect of aggressive driving behaviors on its driving safety. In this paper, we propose a reinforcement learning based decision-making system (ReDS) which considers aggressive driving behaviors of surrounding human-driving vehicles during the decision making process. In ReDS, we firstly build a mixture density network based aggressive driving behavior detection method to detect possible aggressive driving behaviors among surrounding vehicles of an AV. We then build a reward function based on aggressive driving behavior detection results and incorporate the reward function into a reinforcement learning model to make optimal control decisions considering aggressive driving behaviors. We use a real-world traffic dataset from the United States Department of Transportation Federal Highway Administration to evaluate optimal control decision determination performance of ReDS in comparison with the state-of-the-art methods. The comparison results show that ReDS can improve optimal control decision success rate by 43% compared with existing methods, which demonstrates that ReDS has good optimal control decision determination performance.

In a connected autonomous vehicle (CAV) scenario, each vehicle utilizes an onboard deep neural network (DNN) model to understand its received time-series driving signals (e.g., speed, brake status) from its nearby vehicles, and then takes necessary actions to increase traffic safety and roadway efficiency. In the scenario, it is plausible that an attacker may launch an adversarial attack, in which the attacker adds unnoticeable perturbation to the actual driving signals to fool the DNN model inside a victim vehicle to output a misclassified class to cause traffic congestion and/or accidents. Such an attack must be generated in near real-time and the adversarial maneuver must be consistent with the current traffic context. However, previously proposed adversarial attacks fail to meet these requirements. To handle these challenges, in this paper, we propose a Context- aware Black-box Adversarial Attack (CBAA) for time-series DNN models in CAV scenarios. By analyzing real driving datasets, we observe that specific driving signals at certain time points have a higher impact on the DNN output. These influential spatio-temporal factors differ in different traffic contexts (a combination of different traffic factors (e.g., congestion, slope, and curvature)). Thus, CBAA first generates the perturbation only on the influential spatio-temporal signals for each context offline. In generating an attack online, CBAA uses the offline perturbation for the current context to start searching the minimum perturbation using the zeroth-order gradient descent method that will lead to the misclassification. Limiting the spatio-temporal searching scope with the constraint of context greatly expedites finding the final perturbation. Our extensive experimental studies using two different real driving datasets show that CBAA requires 43% fewer queries (to the DNN model to verify the attack success) and 53% less time than existing adversarial attacks.

Many applications such as healthcare and transportation on edge devices will use deep neural network (DNN) prediction based on time-series data collected by the devices. However, the existence of noises in the on-device sensors negatively impacts the sensing output of the DNN models. The state-of-the-art time-series based DNN approaches can deal with Gaussian noise but cannot effectively handle other types of noises in spite of the existence of different types of noises such as shot, burst, transient noises, and their combination. In this paper, we propose an ensemble-based DNN model, namely E-Sense, which consists of different expert models for different noises and shows higher prediction accuracy. Since an edge device may have limited resources to run a large DNN model, we further propose a novel searching-based model compression method called E-Comp that uses knowledge distillation to compress E-Sense to a smaller DNN model while maintaining the accuracy. Our real experiments on live sensor data and trace-driven experiments on three real traces show that E-Sense outperforms other methods in accuracy, and E-Comp reduces 27% inference time without sacrificing accuracy compared with other DNN compression methods. We also distributed our source code.

The key to optimizing the performance of an anycast-based system (e.g., the root DNS or a CDN) is choosing the right set of sites to announce the anycast prefix. One challenge here is predicting catchments. A naïve approach is to advertise the prefix from all subsets of available sites and choose the best-performing subset, but this does not scale well. We demonstrate that by conducting pairwise experiments between sites peering with tier-1 networks, we can predict the catchments that would result if we announce to any subset of the sites. We prove that our method is effective in a simplified model of BGP, consistent with common BGP routing policies, and evaluate it in a real-world testbed. We then present AnyOpt, a system that predicts anycast catchments. Using AnyOpt, a network operator can find a subset of anycast sites that minimizes client latency without using the naïve approach. In an experiment using 15 sites, each peering with one of six transit providers, AnyOpt predicted site catchments of 15,300 clients with 94.7% accuracy and client RTTs with a mean error of 4.6%. AnyOpt identified a subset of 12 sites, announcing to which lowers the mean RTT to clients by 33ms compared to a greedy approach that enables the same number of sites with the lowest average unicast latency.

To maximize its own profit, cloud service brokerage (CSB) aims to distribute tenant demands to reserved servers such that the total reservation cost is minimized with the tenants’ service level agreement (SLA) being satisfied. Thedemand allocationproblem for CSB is non-trivial to solve due to uncertainty of tenants’ behavior. To avoid possible violations among demands, existing schemes allocate additional padding resources on the predicted demands, which leads to under-utilization of reserved resources. Accordingly, we propose aProbabilistic Demand Allocation(PDA) system to address the demand allocation problem for CSB. In PDA, we not only predict tenants’ demands based on their historical records, but also estimate the probability distribution of prediction errors. As over- and under-estimation are equally likely to happen with our prediction method, when allocating demands to a single server, their errors are possibly offset. Hence, it is unnecessary to allocate additional resource to each demand for violation prevention. Given the predication results, we formulate the demand allocation problem by probabilistic optimization, of which the objective is to minimize the overall cost from reserved servers while satisfying tenants’ SLA with high probability. Both simulation and real-world experimental results demonstrate the superiority of PDA in reducing servers’ reservation cost.

Charger lanes, which are road segments equipped with in-motion wireless chargers, are expected to keep Electric Vehicles (EVs) continuously driving without recharging downtime. To maximize the service efficiency of the in-motion wireless chargers, we must properly coordinate the traffic of the EVs to avoid the generation of congestion at the charger lanes and on the road segments to them. In this article, we propose WPT-Opt , a game-theoretic approach for optimizing in-motion wireless charging efficiency, minimizing EVs’ driving time to the charger, and avoiding traffic congestion at the charger lanes to fulfill this task. We studied a metropolitan-scale dataset of public transportation EVs and observed the EVs’ spatial and temporal preference in selecting chargers, competition for chargers during busy charging times, the relationship between vehicle density and driving velocity on a road segment, the normal distribution of travel time of road segments, and the fact that vehicles have similar frequently driven trajectories. Based on the observations, a central controller estimates the vehicle density of the road segments by measuring the vehicles’ trajectory travel time, the friendship among the vehicles, and the vehicles’ routing choice given the presence of charger lanes. Then, we formulate a non-cooperative Stackelberg game between all the EVs and the central controller , in which each EV aims at minimizing its charging time cost to its selected target charger, while the central controller tries to maximally avoid the generation of congestion on the way through the in-motion wireless chargers. Our trace-driven experiments on SUMO demonstrate that WPT-Opt can maximally reduce the average charging time cost of the EVs by approximately 200% during different hours of a day.

So far, there is no file replication method that tries to reduce data loss in correlated node failures, which however are common in Disruption Tolerant Networks (DTNs). In this paper, we propose a distributed file replication method (called MobileCopy) in DTNs, which aims to achieve low probability of totally losing a file at the expense of having a high number of impacted files in an individual large-scale correlated node failure. MobileCopy is designed for community-based file sharing systems. It has two main components: i) data loss resistant and popularity aware file replication, and ii) distributed hash table (DHT)-based file replica indexing. MobileCopy considers file popularity to determine the number of replicas of a file in each community. Through limiting the possible combination of candidate replica holders, MobileCopy greatly reduces the probability of node failures that will lead to data loss, i.e., losing all replicas of a file. Moreover, MobileCopy enables nodes to efficiently store and fetch the placement information of file replicas through competition based file replication and considering node mobility throughput among communities. Extensive trace-driven experiments demonstrate the effectiveness of MobileCopy against correlated node failures compared with previous methods.

Data availability is one of the most important performance factors in cloud storage systems. To enhance data availability, replication is a common approach to handle the machine failures. However, previously proposed replication schemes cannot effectively handle both correlated and non-correlated machine failures, especially while increasing the data availability with limited resources. The schemes for correlated machine failures must create a constant number of replicas for each data object, which often neglects diverse data popularities and does not utilize the resource to maximize the expected data availability. Also, the previous schemes neglect the consistency maintenance cost and the storage cost caused by replication. It is critical for cloud providers to maximize data availability (hence minimize SLA violations) while minimizing costs caused by replication in order to maximize the revenue. In this paper, we build a nonlinear integer programming model to maximize data availability in both types of failures, and therefore minimize the cost caused by replication. Based on the model's solution for the replication degree of each data object, we propose a low-cost multi-failure (correlated and non-correlated machine failures) resilient replication scheme (MRR). MRR can effectively handle both correlated and non-correlated machine failures, considers data popularities to enhance data availability, and also tries to minimize consistency maintenance and storage cost. Extensive numerical results from trace parameters and experiments from real-world Amazon S3 demonstrate that MRR achieves high data availability, low data loss probability and low consistency maintenance and storage costs when compared to previous replication schemes.

More and more organizations move their data and workload to commercial cloud storage systems. However, the multiplexing and sharing of the resources in a cloud storage system present unpredictable data access latency to tenants, which may make online data-intensive applications unable to satisfy their deadline requirements. Thus, it is important for cloud storage systems to provide deadline guaranteed services. In this paper, to meet a current form of service level objective (SLO) that constrains the percentage of each tenant's data access requests failing to meet its required deadline below a given threshold, we build a mathematical model to derive the upper bound of acceptable request arrival rate on each server. We then propose a Deadline Guaranteed storage service (called DGCloud) that incorporates three basic algorithms. Its deadline-aware load balancing scheme redirects requests and creates replicas to release the excess load of each server beyond the derived upper bound. Its workload consolidation algorithm tries to maximally reduce servers while still satisfying the SLO to maximize the resource utilization. Its data placement optimization algorithm re-schedules the data placement to minimize the transmission cost of data replication. We further propose three enhancement methods to further improve the performance of DGCloud. A dynamic load balancing method allows an overloaded server to quickly offload its excess workload. A data request queue improvement method sets different priorities to the data responses in a server's queue so that more requests can satisfy the SLO requirement. A wakeup server selection method selects a sleeping server that stores more popular data to wake up, which allows it to handle more data requests. Our trace-driven experiments in simulation and Amazon EC2 show the superior performance of DGCloud compared with previous methods in terms of deadline guarantees and system resource utilization, and the effectiveness of its individual algorithms.

As video streaming applications are deployed on the cloud, cloud providers are charged by ISPs for inter-datacenter transfers under the dominant percentile-based charging models. In order to minimize the payment costs, existing works aim to keep the traffic on each link under the charging volume. However, these methods cannot fully utilize each link's available bandwidth capacity. As a solution, we propose an economical and deadline-driven video flow scheduling system, called EcoFlow. Considering that different video flows have different transmission deadlines, EcoFlow transmits videos in the order of their deadline tightness and postpones the deliveries of later-deadline videos to later time slots. The flows that are expected to miss their deadlines are divided into subflows to be rerouted to other under-utilized links. We also propose setting each link's initial charging volume to reduce the scheduling latency at the beginning of the charging period and discuss how to deal with issues such as the prediction errors of link available bandwidth and the lack of charging volume's prior knowledge. Furthermore, we designed implementation strategies for using EcoFlow in both centralized and distributed situations. Experimental results demonstrate that EcoFlow achieves lower bandwidth costs and higher video flow transmission rates when compared to existing methods.

With the rapid proliferation of Machine Learning (ML) and Deep learning (DL) applications running on modern platforms, it is crucial to satisfy application performance requirements such as meeting deadline and ensuring accuracy. To this end, researchers have proposed several job schedulers for ML clusters. However, none of the previously proposed schedulers consider ML model parallelism, though it has been proposed as an approach to increase the efficiency of running large-scale ML and DL jobs. Thus, in this paper, we propose an ML job Feature based job Scheduling system (MLFS) for ML clusters running both data parallelism and model parallelism ML jobs. MLFS first uses a heuristic scheduling method that considers an ML job's spatial and temporal features to determine task priority for job queue ordering in order to improve job completion time (JCT) and accuracy performance. It uses the data from the heuristic scheduling method for training a deep reinforcement learning (RL) model. After the RL model is well trained, it then switches to the RL method to automatically make decisions on job scheduling. Furthermore, MLFS has a system load control method that selects tasks from overloaded servers to move to underloaded servers based on task priority, and also intelligently removes the tasks that generate little or no improvement on the desired accuracy performance when the system is overloaded to improve JCT and accuracy by job deadline. Real experiments and large-scale simulation based on real trace show that MLFS reduces JCT by up to 53% and makespan by up to 52%, and improves accuracy by up to 64% when compared with existing ML job schedulers. We also open sourced our code.

For the situation where plenty of trust models are built to improve node cooperation in mobile peer-to-peer environments, node cooperation is analyzed in this paper. An Android APP is first developed to investigate users' satisfaction on cooperation degree based on their internal states, and then according to the investigation results, nodes' utility on cooperation degree based on internal states is designed. A node cooperation optimization model (COM) is built to maximize the utilities of all involved nodes and an intelligence algorithm is applied to solve this optimization problem. Because the results show that the cooperation degrees before and after optimization are very similar, each node can decide its cooperation degree only based on own information. Finally, the model is verified and applied to design a new routing protocol. The experiment results demonstrate that the proposed routing protocol can achieve higher routing success ratio and average node utility than other popular routing protocols.

As a widely used IT service, more and more companies shift their services to cloud datacenters. It is important for cloud service providers (CSPs) to provide cloud service resources with high elasticity and cost-effectiveness and then achieve good quality of service (QoS) for their clients. However, meeting QoS with cost-effective resource is a challenging problem for CSPs because the workloads of Virtual Machines (VMs) experience variation over time. It is highly necessary to provide an accurate VMs workload prediction method for resource provisioning to efficiently manage cloud resources. In this paper, we first compare the performance of representative state-of-the-art workload prediction methods. We suggest a method to conduct the prediction a certain time before the predicted time point in order to allow sufficient time for task scheduling based on predicted workload. To further improve the prediction accuracy, we introduce a clustering based workload prediction method, which first clusters all the tasks into several categories and then trains a prediction model for each category respectively. The trace-driven experiments based on Google cluster trace demonstrates that our clustering based workload prediction methods outperform other comparison methods and improve the prediction accuracy to around 90% both in CPU and memory.

For a modern vehicle, if a sensor in the vehicle Anti-lock Braking System (ABS) or the controller area network (CAN) bus is attacked during a brake process, the vehicle will lose driving direction control and driver's life will be highly threatened. However, current methods for detecting attacks are not sufficiently accurate and no methods can provide attack mitigation. To ensure vehicle ABS security, we propose an attack detection method to accurately detect both sensor attack and CAN bus attack in a vehicle ABS, and an attack mitigation strategy to mitigate their negative effects on the vehicle ABS. In our attack detection method, we build a vehicle state space equation which considers real-time road friction coefficient to predict vehicle states (i.e., wheel speed and longitudinal brake force) with their previous values. Based on sets of historical measured vehicle states, we develop a search algorithm to find out attack changes (vehicle state changes because of the attack) by minimizing errors between predicted vehicle states and measured vehicle states. In our attack mitigation strategy, attack changes are subtracted from measured vehicle states to generate correct vehicle states for the vehicle ABS. We conducted first real sensor attack experiments to show how a magnet affects sensor readings. Our experimental results demonstrate that our attack detection method can detect sensor attack and CAN bus attack more accurately compared with existing methods, and also our attack mitigation strategy almost eliminates attack's effects on the vehicle ABS.

In Named Data Networks, cache pollution, cache poisoning and interest flooding are three popular types of attacks that can drastically degrade the network performance. However, previous methods for mitigating these attacks are not sufficiently effective or efficient. Also, they cannot simultaneously handle the three attacks. To handle these problems, we propose an effective and lightweight countermeasure scheme. It consists of token-based router monitoring policy (TRM), hierarchical consensus-based trust management (HCT), and popularity-based probabilistic caching policy (PPC). In TRM, each edge router monitors and evaluates each data requester's probability of launching the cache pollution attack and each data provider's probability of launching the cache poisoning attack, and accordingly assigns, rewards and penalizes tokens to them to control their data request and data provision activities. In HCT, each core router manages its directly connected edge routers using TRM, and the core routers trust each other through adopting the concept of consensus in Blockchain. PPC uses probabilistic caching based on the popularity of received content to further mitigate the attacks and reduce caching and data verification overhead. Results from simulation experiments demonstrate that our proposed scheme has better performance, in terms of interest satisfaction ratio and average end-to-end delay than current mechanisms.

The effectiveness of dispatching rescue teams under a flooding disaster is crucial. However, previous emergency vehicle dispatching methods cannot handle flooding disaster situations, and previous rescue team dispatching methods cannot accurately estimate the positions of potential rescue requests or dispatch the rescue teams according to the real-time distribution of rescue requests. In this paper, we propose MobiRescue, a human Mobility based Rescue team dispatching system, that aims to maximize the total number of fulfilled rescue requests, minimize the rescue teams' driving delay to the rescue requests' positions and also the number of dispatched rescue teams. We studied a city-scale human mobility dataset for the Hurricane Florence, and found that the disaster impact severities are quite different in different regions, and people's movement was significantly affected by the disaster, which means that the rescue teams' driving routes should be adaptively adjusted. Then, we propose a Support Vector Machine (SVM) based method to predict the distribution of potential rescue requests on each road segment. Based on the predicted distribution, we develop a Reinforcement Learning (RL) based rescue team dispatching method to achieve the aforementioned goals. Our trace-driven experiments demonstrate the superior performance of MobiRescue over other comparison methods.

With ever increasing concerns on environmental issues caused by gasoline fuel based vehicles, electric vehicles (EVs) have attracted more and more attention from governments, industries, and customers [1] . The recent advancements in EVs have great potential to create a more environmentally friendly smart city. However, due to limited battery capacity, most current mainstream EVs still have quite limited driving range (e.g., 100 miles) [2] . How to ensure the continuous running of EVs on a large-scale road network (e.g., metropolitan city, interstate) becomes a major concern.

Electronic control units and actuators in a modern vehicle communicate with each other through controller area network (CAN) bus to ensure driving safety. Current methods assume that time interval or frequency of a normal message always keeps constant and detect abnormal messages by checking whether a message's time interval or frequency changes. However, through real CAN bus dataset analysis, we find that time interval and frequency of a normal message may change greatly in different vehicle driving conditions (such as Acceleration, Deceleration, and Driving mode shift), which can cause low detection accuracy of current methods. To handle this problem, in this paper, we propose a message transmission behavior based abnormal message detection system (MetraDS), which detects abnormal messages in CAN bus based on message transmission behaviors including frequency and message sequence. Through real dataset analysis, we find that a message's sequence usually does not change in different vehicle driving conditions, and also an abnormal message affects the frequency of itself and its subsequent messages. Accordingly, MetraDS checks message sequence dissimilarity to determine whether abnormal messages exist in a message sequence and then checks each message's influence to itself and its subsequent messages to locate the abnormal messages in the sequence. We used the real dataset to test the abnormal message detection performance of MetraDS in comparison with state-of-the-art methods. The experimental results show that MetraDS improves the abnormal message detection accuracy of other methods by as much as 33.8%, and its memory and computational cost requirements are acceptable.

In this poster, we present measurement studies of the impact of memory DoS attacks on different types of cloud-based applications. Based on the observations, we propose a lightweight, responsive Statistical based Detection Scheme (SDS) that can detect such attacks accurately. Our initial evaluation results show that SDS outperforms the state-of-the-art detection scheme up to 2% higher recall, up to 65% higher specificity, and up to 40% shorter detection delay.

Even though memory-based denial-of-service attacks can cause severe performance degradations on co-located virtual machines, a previous detection scheme against such attacks cannot accurately detect the attacks and also generates high detection delay and high performance overhead since it assumes that cache-related statistics of an application follow the same probability distribution at all times, which may not be true for all types of applications. In this paper, we present the experimental results showing the impacts of memory DoS attacks on different types of cloud-based applications. Based on these results, we propose two lightweight, responsive Statistical based Detection Schemes (SDS/B and SDS/P) that can detect such attacks accurately. SDS/B constructs a profile of normal range of cache-related statistics for all applications and use statistical methods to infer an attack when the real-time collected statistics exceed this normal range, while SDS/P exploits the increased periods of access patterns for periodic applications to infer an attack. Our evaluation results show that SDS/B and SDS/P outperform the state-of-the-art detection scheme, e.g., with 65% higher specificity, 40% shorter detection delay, and 7% less performance overhead.

Currently, many web applications are deployed on cloud storage service provided by cloud service providers (CSPs). A CSP offers different types of storage including hot, cold and archive storage and sets unit prices for these different types, which vary substantially. By properly assigning the data files of a web application to different types of storage based on their usage profiles and the CSP’s pricing policy, a cloud customer potentially can achieve substantial cost savings and minimize the payment to the CSP. However, no previous research handles this problem. Towards this goal, we present a Markov Decision Process formulation for the cost minimization problem, and then develop a reinforcement learning based approach to effectively solve the problem, which changes the type of storage of each data file periodically to minimize money cost in long term. We then propose a method to aggregate concurrently requested data files to further reduce the cloud storage service payment for a web application. Our experiments with Wikipedia traces show the effectiveness of the proposed methods for minimizing cloud customer cost in comparison with other methods.

The size and energy consumption of datacenters have been increasing significantly over the past years. As a result, datacenters' increasing electricity monetary cost, energy consumption and energy harmful gas emissions have become a severe problem. Renewable energy supply is widely seen as a promising solution. However, the instability of renewable energy brings about a new challenge since insufficient energy supply may lead to job running interruptions or failures. Though previous works attempt to more accurately predict the amount of produced renewable energy, due to the instability of its influencing factors (e.g., wind, temperature), sufficient renewable energy supply cannot be always guaranteed. To handle this problem, in this paper, we propose allocating jobs with the same service-level-objective (SLO) level to the same physical machine (PM) group, and power each PM group with renewable energy generators that have probability no less than its SLO to produce the amount no less than its energy demand. It ensures that insufficient renewable energy supply will not lead to SLO violations. We use a deep learning technique to predict the probability of producing amount no less than each value of each renewable energy source and predict the energy demands of each PM area. We formulate an optimization problem: how to match renewable energy resources with different instabilities to different PM groups as energy supply in order to minimize the number of SLO violations (due to interruption from insufficient renewable energy supply), total energy monetary cost and total carbon emission. We then use reinforcement learning method and linear programming method to solve the optimization problem. The real trace driven experiments show that our method can achieve much lower SLO violations, total energy monetary cost and total carbon emission compared to other methods.

MapReduce is a popular distributed framework for big data analysis. However, the current MapReduce framework is insufficiently efficient in handling intermediate data, which may cause bottlenecks in I/O operations, computation, and network bandwidth. Previous work addresses the I/O problem by aggregating map task outputs (i.e. intermediate data) for each single reduce task on one machine. Unfortunately, when there are a large number of reduce tasks, their concurrent requests for intermediate data generate a large amount of I/O operations. In this paper, we present APA (Aggregation, Partition, and Allocation), a new intermediate data management system for the MapReduce framework. APA aggregates the intermediate data from the map tasks in each rack to one file, and the file host pushes the needed intermediate data to each reduce task. Thus, it reduces the number of disk seeks involved in handling intermediate data within one job. Rather than evenly distributing the intermediate data among reduce tasks based on the keys as in current MapReduce, APA partitions the intermediate data to balance the execution latency of different reduce tasks. APA further decides where to allocate each reduce task to minimize the intermediate data transmission time between map tasks and reduce tasks. Through experiments on a real MapReduce Hadoop cluster using the HiBench benchmark suite, we show that APA improves the performance of the current Hadoop by 40%-50%.

Pure electric vehicles (EVs) are more and more popular in current automotive markets. Many services for EVs such as intelligent battery charging systems and mobile apps are developed to monitor battery energy information (energy consumption time series) for users. However, it is neglected that such EV energy consumption time series may lead to driver privacy related problems such as the disclosure of driving path. In this paper, we propose the first battery energy based path inference attacks (Bepath), which can estimate EV driving paths only based on EV energy consumption time series. Bepath firstly estimates appliance states of an EV including vehicle speed state, air condition state and intersection turn state based on EV energy consumption time series. And then, Bepath figures out the driving path based on the estimated appliance states and map information. We used real-life daily driving data with different participants to test the performance of Bepath and compared inferred path results between Bepath and an existing method to evaluate the path inference accuracy of Bepath. The experimental results show that Bepath can estimate appliance states accurately and infers 50% trips with distance errors within 0.5 km, while the previous work that infers paths based on vehicle speed from insurance companies can only infer 26% trips with distance errors within 0.5 km.

Many usage-based automotive insurance programs do not directly use any GPS-based location tracking devices. Instead, CAN-bus data, such as brake signal data, can be collected by these programs to evaluate drivers’ driving habits and vehicle usage policies. In this paper, we demonstrate that by using a temporal sequence of applied brake signals collected from a vehicle, attackers can still possibly infer the vehicle’s route over the period, even though brake signal data does not reveal any specific location information. Our route inference is basically composed of three steps: At first, we categorize brake signal subsequences into four different driving maneuvers (i.e., stopping from a certain speed, reducing speed to adjust with the traffic flow, and taking left and right turns). Second, we estimate the number of intersections traversed by the vehicle using the applied brake signals and their corresponding maneuvers. Particularly, we also estimate the overall speed profile based on the magnitude and interval of different brake signals. From the estimated speed profile, we infer the distances, traveling time, and traffic signs corresponding to the candidate edges. Finally, we design a graph-based route-selection algorithm to find a list of (paths) routes from the regional map using the predicted driving maneuvers and the speed profile. We use a score function based on three factors (i.e., distance, traveling time, and traffic signs) to identify a candidate edge. We evaluate our approach using over 450km of transportation data, which has been collected from 24 individuals. The experimental results demonstrate that, by resorting to our solution, 89% of the original drivers’ trajectory can be successfully recovered from their brake data regardless of driver and vehicle models.

Previous passenger demand inference methods have insufficient accuracy because they fail to catch the influence of all random factors (e.g., weather, holiday). Also, existing taxicab dispatching methods are not directly applicable for electric taxicabs because they cannot optimize their charging. We present CD-Guide: an electric taxicab dispatching and charging approach based on customized training and Reinforcement Learning (RL). We studied a metropolitan-scale taxicab dataset, and found: histogram of passengers' origin buildings (i.e., where they come from) is useful for selecting suitable training data for inference model, passenger demand in different regions may be influenced by various unpredictable random factors, and taxicabs' charging time must be considered to avoid missing potential passengers. By saying suitable historical data, we mean the data that are under the influence of random factors similar as current time. Then, we develop a RL based method to guide a taxicab to maximize its probability of picking up a passenger, minimize the number of its missed passengers due to charging, and meanwhile avoid the taxicab from battery exhaustion. Our trace-driven experiments show that compared with previous methods, CD-Guide increases the total number of served passengers by 100%.

Previous Electric Vehicle (EV) charging scheduling methods and EV route planning methods require EVs to spend extra waiting time and driving burden for a recharge. With the advancement of dynamic wireless charging for EVs, Mobile Energy Disseminator (MED), which can charge an EV in motion, becomes available. However, existing wireless charging scheduling methods for wireless sensors, which are the most related works to the deployment of MEDs, are not directly applicable for the scheduling of MEDs on city-scale road networks. We present MobiCharger: a Mobile wireless Charger guidance system that determines the number of serving MEDs, and the optimal routes of the MEDs periodically (e.g., every 30 minutes). Through analyzing a metropolitan-scale vehicle mobility dataset, we found that most vehicles have routines, and the temporal change of the number of driving vehicles changes during different time slots, which means the number of MEDs should adaptively change as well. Then, we propose a Reinforcement Learning based method to determine the number and the driving route of serving MEDs. Our experiments driven by the dataset demonstrate that MobiCharger increases the medium state-of-charge and the number of charges of all EVs by 50% and 100%, respectively.

Driving route recommendation systems have been becoming popular due to high demands on such systems and their high socio-economic impacts. Existing route recommendation systems cannot provide a well-balanced route by considering the user preference on multiple criteria or make route recommendation in a short time. This paper presents a multi-objective route recommendation system considering three different attributes (i.e., fuel consumption, travel time, and air quality). The proposed route recommendation system uses the Q-learning based reinforcement learning algorithm to leverage the available datasets to make route recommendations in a timely manner. First, we build a road network graph using a publicly available map service (i.e., OpenStreetMap) and other real-world datasets on traffic, weather, and air substances. Second, we utilize the existing predictors for air quality, travel time, and fuel consumption estimations to update the road network graph periodically. Third, we design the route recommendation system using the Q-learning reinforcement learning approach considering the given user's preference for travel time, fuel consumption, and air quality. To evaluate the proposed approach's performance, we conduct experimental evaluations based on the real-world datasets with publicly available map service.

Connected and autonomous vehicles (CAV) have been introduced to increase roadway safety and traffic flow efficiency. In CAV scenarios, an autonomous vehicle shares its current and near-future driving maneuvers in terms of different driving signals (e.g., speed, brake pedal pressure) with its nearby vehicles using wireless communication technologies. Deep neural network (DNN) models are usually used to process the driving maneuver time-series data over other machine learning algorithms due to the high prediction accuracy of DNN models. In this scenario, an attacker can send false driving maneuver signals to fool the DNN model to misclassify an input. The existing black-box adversarial attacks (which are for image datasets) require many queries to the DNN model to check if a generated attack will be successful (hence long time) or high amount of perturbation (low imperceptibility), and thus cannot be applied to the time-sensitive CAV scenarios featured by multi-dimensional time series driving data. In this paper, we present an Advanced black-box Adversarial Attack (A3) for the deep driving maneuver classification models. We first formulate an optimization problem for the attack generation with continuous search space to reduce the search time. To solve the optimization problem, A3innovatively combines the binary search and optimization algorithm to improve the time-efficiency of searching the optimal solution. It first uses a binary partition technique to reduce the perturbation search space in solving the problem to improve time-efficiency. It then uses the zeroth-order stochastic gradient descent approach, which is featured by searching a solution faster for high-dimensional datasets, thus further improving time-efficiency. We evaluate the proposed A3attack in terms of different metrics using two real driving datasets. The experimental results show that the A3attack requires up to 84.12% fewer queries and 57.67% less perturbation with 94.87% higher success rates than the existing black-box adversarial attacks.

Nowadays, smart building infrastructures are equipped with hundreds of sensors to monitor building environments and provide smart solutions for occupant comfortability and energy efficiency. Ideally, an automated system can predict and adjust the physical features (e.g., lighting, air quality, temperature, and so on) in a person’s office based on his/her personalized preferences and activities. However, since the data is from one person, there may not be sufficient data for machine learning model training, and the data’s quality may be low (e.g., with noises). Then, it is a challenge to conduct accurate predictions to provide personalized environment adjustment. To handle this problem, in this paper, we propose a smart building assistance system consisting of different sensor data analysis approaches and a deep neural network (DNN)-based prediction model to make a more accurate prediction despite low-quality sensor data. First, we collected a year-long smart building dataset from four different data sources (i.e., sensors, calendar, weather, and survey). Second, we perform different feature engineering approaches (i.e., concretization, one-hot encoding, and multiple feature combination) on the data as inputs for the prediction models. Third, we identify a support vector regression-based prediction model and propose a hybrid DNN model consisting of several recurrent neural network blocks and a feed-forward DNN block to predict different preferred physical features considering different activities of a person (e.g., meeting, lunch, research activities). Finally, we conduct experimental studies to evaluate the performance of the proposed prediction models compared to other existing machine learning models in terms of accuracy. Our predicted preferred physical features match the occupant’s preferred ranges of different physical features during a specific activity. We also open-sourced our code on GitHub.

Edge computing usage in many applications, such as transportation and healthcare, has been becoming popular nowadays. These applications often use deep learning (DL) prediction, which are highly dependent on time-series data collected by the sensors in the edge devices. However, the presence of noise in the on-device sensors negatively affects the sensing output of the DL models. Recently proposed time-series based DL approaches (e.g., SADeepSense) address this issue with the assumption that in the presence of noise, the correlation of sensor inputs in an edge device changes. In this paper, through real experiments, we notice that this assumption may not hold true in the presence of shot noise. To handle this problem, in order to further improve the prediction accuracy, we propose a DL model, namely NoiseSenseDNN, which more accurately extracts the correlation between different sensor inputs over time in the presence of both shot and white noise due to its unique architecture. We further propose a compressed version of NoiseSenseDNN that minimizes the inference time and consumed energy of the edge device while meeting the accuracy requirement. Our experiments on a workstation and a real edge device and three real traces show that NoiseSenseDNN outperforms SADeepSense in accuracy, and the compressed NoiseSenseDNN significantly reduces inference time and energy consumption while meeting the required accuracy.

The Internet of Things (IoT) paradigm promises to make “things” such as physical objects with sensing capabilities and/or attached with tags, mobile objects such as smartphones and vehicles, consumer electronic devices, and home appliances such as fridge, television, health care devices, as part of the Internet environment. In cloud-centric IoT applications, the sensor data from these “things” is extracted, accumulated, and processed at the public/private clouds, leading to significant latencies. To satisfy the ever increasing demand for cloud computing resources from emerging applications such as IoT, academics and industry experts are now advocating for going from large-centralized cloud computing infrastructures to micro data centers located at the edge of the network. These micro data centers are often closer to a user (geographically and in access latency) compared to the centralized cloud data center. The aim of utilizing such edge resources is to off load computation that would have “traditionally” been carried out at the cloud data center to a resource that is closer to a user or edge devices. This vision also acknowledges the variation in network latency from an end-user to cloud data center. While the network around a data center is often high capacity and speed, that near the user device may have variable properties (in terms of resilience, bandwidth, latency, etc.). Referred to as “fog/edge computing,” this paradigm is expected to improve the agility of cloud service deployments in addition to bringing computing resources closer to end-users. The emergence of computing paradigms such as edge and fog computing supports the data analysis near the data sources for a wide range of applications. Edge computing is the middle layer between users and cloud data centers, and it plays an important role in the IoT use cases where applications required near real-time actions. The intermediate edge layer provides limited computing and storage resources, which consists of network gateways and micro data centers. Edge computing application orchestration is capable of big data processing and can be installed in heterogeneous hardware configurations. Due to the large-scale deployment and device heterogeneity, edge computing infrastructure designing and implementation are challenging, including model analysis, system integration, protocol designing, energy, and security modeling. In addition, since edge data centers are installed in network gateways with open network configuration, they are prone to several network threats, less trustworthy, and easy to compromise. On the one hand, the development of fog and edge clouds includes dedicated facilities, operating system, network, and middleware techniques to build and operate such micro data centers that host virtualized computing resources. On the other hand, the use of fog and edge clouds requires extension to current programming models and proposes new abstractions that will allow developers to design new applications that take benefit from such massively distributed systems. The use of this approach also opens up other challenges in security and privacy (as a user now needs to “trust” every micro data center they interact with), support for resource management for mobile users who transfer session from one micro data center to another, and support for “embedding” such micro data centers into devices (eg, cars, buildings, etc). The objective of this special issue is to disseminate original contributions and research findings concerning the challenges and changes (both evolutionary and disruptive) in edge and fog computing. It provides cutting-edge research from both academia and industry, with emphasis on current developments and future directions in security and privacy issues of emerging fog computing. The call for special issues received a number of submissions. Each paper was reviewed by at least three reviewers and went through at least two rounds of reviews. After a two-phase peer review process, we have accepted 14 high-quality papers related to the aforementioned areas of interest. The accepted papers focus on recent solutions by developing novel research ideas around edge and fog computing for several applications, such as health care, smart city, urban pollution monitoring, etc. The brief contributions of these papers are discussed in the following section. The first paper titled “Abnormal visual event detection based on multi-instance learning and autoregressive integrated moving average model in edge-based Smart City surveillance” by Xu et al proposes an abnormal event detection approach based on multi-instance learning and autoregressive integrated moving average model for video surveillance of crowded scenes in urban public places. It utilizes an unsupervised method for abnormal event detection by combining multi-instance visual feature selection and the autoregressive integrated moving average model. This approach has thoroughly experimented, and the experimental results demonstrate the efficiency of the proposed approach by achieving better abnormal event detection performance for a crowded scene of urban public places with an edge environment. The second paper titled “User allocation-aware edge cloud placement in mobile edge computing” by Guo et al studies the edge cloud placement problem, which is to place the edge clouds at the candidate locations and allocate the mobile users to the edge clouds. Further, it formulates as a multi-objective optimization problem with the objective to balance the workload between edge clouds and minimize the service communication delay of mobile users. The experiment results show the performance of the proposed approach in terms of workload balance and communication delay for validation. The third paper titled “A secure fog-based platform for SCADA-based IoT critical infrastructure” by Baker et al contributes a novel security “toolbox” to reinforce the integrity, security, and privacy of SCADA-based IoT critical infrastructure at the fog layer. The toolbox incorporates a key feature, that is, a cryptographic-based access approach to the cloud services using identity-based cryptography and signature schemes at the fog layer. This paper also presents the implementation details of a prototype for our proposed secure fog-based platform and provides performance evaluation results to demonstrate the appropriateness of the proposed platform in a real-world scenario. The results from the experiments demonstrate a superior performance of the secure fog-based platform, which is around 2.8 seconds when adding five virtual machines (VMs), 3.2 seconds when adding 10 VMs, and 112 seconds when adding 1000 VMs, compared to the multilevel user access control platform. The fourth paper titled “Developing applications in large scale, dynamic fog computing: A case study” by Giang et al presents a case study in building fog computing applications using an open-source platform distributed node-RED. It shows how applications can be decomposed and deployed to a geographically distributed infrastructure using distributed node-RED, and how existing software components can be adapted and reused to participate in fog applications. This case study is implemented in a lab-based fog infrastructure and simulated for large-scale evaluation. The fifth paper titled “An osmotic computing infrastructure for urban pollution monitoring” by Longo et al focuses on the design and development of a middleware that integrates data coming from mobile and IoT devices specifically deployed in urban contexts using the osmotic computing paradigm. Moreover, a component of the osmotic membrane has been developed in this paper for security management. The sixth paper titled “Characterizing application scheduling on edge, fog, and cloud computing resources” by Varshney and Simmhan offers a taxonomy of concepts essential for specifying and solving the problem of scheduling applications on edge, fog, and cloud computing resources. The proposed model is divided into multiple steps, initially characterized by the resource capabilities and limitations of these infrastructures and offers a taxonomy of application models, quality-of-service constraints and goals, and scheduling techniques based on a literature review, followed by tabulated key research prototypes and papers using this taxonomy. It also highlights gaps in the literature and open problems remain. The seventh paper titled “Cloud-aided online electroencephalography (EEG) classification system for brain healthcare: A case study of depression evaluation with a lightweight CNN” by Ke et al presents the design of an online EEG classification system aided by cloud centering on a lightweight convolutional neural network (CNN). The system incrementally trains the CNN on cloud and enables hot deployment of the trained classifier without the need to restart the gateway to adapt to the users' needs. The classifier maintains a high convolutional layer to gain the ability of processing high-dimensional EEG segments. Finally, the model is experimented to validate the contribution. The eighth paper titled “SEWMS: An Edge-based Smart Wearable Maintenance System in Communication Network” by Rui et al proposes a dynamic context-aware information push algorithm (DCAIP) by focusing on the current low level in information, complicated scenes, and various information on on-site maintenance. It also presents a smart wearable maintenance system (SEWMS), an edge computing-assisted IoT platform for the real-time guidance of technical experts and systems for on-site maintenance personnel, aiming to improve the efficiency and quality of on-site maintenance. The ninth paper titled “A crosswalk pedestrian recognition system by using deep learning and zebra-crossing recognition techniques” by Dow et al investigates a real-time pedestrian recognition system that ensures high accuracy by using a deep learning classifier and zebra-crossing recognition techniques. The proposed system was designed to improve pedestrian safety and reduce accidents at intersections. Environmental feature vectors were first used to detect zebra crossings and to determine crossing areas. An adaptive mapping technique was then used to map the pedestrian waiting area based on the crossing area. A dual-camera mechanism was used to maintain detection accuracy and improve system fault tolerance. Finally, the you-only-look-once model was used to recognize pedestrians at intersections. The 10th paper titled “Intelligent sentiment analysis approach using edge computing-based deep learning technique” by Sankar et al studies machine learning algorithms to extract the best features from the training review dataset. Then, the selected features are fed into the CNN and other fully connected layers for further processing. This work has also employed a pretrained sentiment analysis model over an Android application framework to classify reviews on a smartphone without the need for any cloud or server-side application programming interface. The 11th paper titled “Pipeline provenance for cloud-based big data analytics” by Wang et al proposes a solution, named LogProv toward realizing the functionalities for big data provenance, which needs to renovate data pipelines or some of big data software infrastructure to generate structured logs for pipeline events, and then stores data and logs separately in cloud space. The LogProv is implemented and deployed in Nectar Cloud, associated with Apache Pig, Hadoop ecosystem, and adopted Elasticsearch to provide query service. The 12th paper titled “Socially aware microcloud service overlay optimization in community networks” by Apolónia et al presents a model, named Select in Community Networks (SELECTinCN), which enhances the overlay creation for pub/sub systems over peer-to-peer (P2P) networks. Moreover, SELECTinCN includes social information based on cooperation within CNs by exploiting the social aspects of the community of practice. The model organizes the peers in a ring topology and provides an adaptive P2P connection establishment algorithm, where each peer identifies the number of connections needed based on the social structure and user availability. The 13th paper titled “DewSim: A trace-driven toolkit for simulating mobile device clusters in Dew computing environments” by Hirsch et al models and develops a trace-based toolkit built on modular software artifacts to speed up research in resource management techniques in Dew environments. A trace-driven methodology is adopted to assure the practical value of simulated scenarios. The toolkit comprises a device profiler application for Android to capture generic battery and central processing unit traces from real devices, a profile mixer to create user interaction baseline traces through generic ones, and an extensible engine to simulate the execution of workloads configurable via text files. The 14th paper titled “How to Place Your Apps in the Fog-State of the Art and Open Challenges” Borgi et al review the existing methodologies to solve the application placement problem in the fog, while pursuing three main objectives. First, it offers a comprehensive overview of the currently employed algorithms, on the availability of open-source prototypes, and on the size of test use cases. Second, it classifies the literature based on the application and fog infrastructure characteristics that are captured by available models, with a focus on the considered constraints and the optimized metrics. Finally, it identifies some open challenges in application placement in the fog. The 15th paper titled ‘SELFNET 5G mobile edge computing infrastructure: Design and prototyping’ by Chirivella-Perez et al. presented the design and prototype implementation of the fifth-generation (5G) mobile edge infrastructure based on a mobile edge computing paradigm. This mobile edge infrastructure SELFNET is an amlganmation of cloud computing, software-defined networking, and network function virtualization to end up with a portable 5G infrastructure testbed which enabled the realistic execution and testing. Finally, in the last paper titled ‘SDN/NFV security framework for fog-to-things computing infrastructure’ by Krishnan et al. proposed DTARS which is a System for Distributed Threat Analytics and Response for an Edge/Fog and SDN integrated architecture. In this system, the detection scheme runs at the data plane wherein a coarse-grained behavioral, anti-spoofing, flow monitoring and fine-grained traffic multi-feature entropy-based algorithms are deployed. The proposed framework has been developed for defense applications on malware testbed. We hope that the research contributions and findings in this special issue would benefit the readers in terms of enhancing their knowledge and encouraging them to work on various aspects of edge and fog computing. We express our sincere thanks to the editor-in-chief for allowing us to organize this special issue. The editorial office staffs are excellent and thanks for their support. We are also thankful to all the authors who made this special issue possible, and to the reviewers for their thoughtful contributions.

To provide robust infrastructure as a service (IaaS), clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. The unique features of clouds pose formidable challenges to achieving effective and efficient load balancing. First, VMs in clouds use different resources (e.g., CPU, bandwidth, memory) to serve a variety of services (e.g., high performance computing, web services, file services), resulting in different over utilized resources in different PMs. Also, the over utilized resources in a PM may vary over time due to the time-varying heterogeneous service requests. Second, there is intensive network communication between VMs. However, previous load balancing methods statically assign equal or predefined weights to different resources, which lead to degraded performance in terms of speed and cost to achieve load balance. Also, they do not strive to minimize the VM communications between PMs. We propose a Resource Intensity Aware Load balancing method (RIAL). For each PM, RIAL dynamically assigns different weights to different resources according to their usage intensity in the PM, which significantly reduces the time and cost to achieve load balance and avoids future load imbalance. It also tries to keep frequently communicating VMs in the same PM to reduce bandwidth cost, and migrates VMs to PMs with minimum VM performance degradation. We also propose an extended version of RIAL with three additional algorithms. First, it optimally determines the weights for considering communication cost and performance degradation due to VM migrations. Second, it has a more strict migration triggering algorithm to avoid unnecessary migrations while still satisfying Service Level Objects (SLOs). Third, it conducts destination PM selection in a decentralized manner to improve scalability. Our extensive trace-driven simulation results and real-world experimental results show the superior performance of RIAL compared to other load balancing methods.

Cloud computing offers a cost-effective and elastic computing paradigm that facilitates large scale data storage and analytics. By deploying virtualization technologies in the datacenter, cloud enables efficient resource management and isolation for various big data applications. Since the hotspots (i.e., overloaded machines) can degrade the performance of these applications, virtual machine migration has been utilized to perform load balancing in the datacenters to eliminate hotspots and guarantee Service Level Agreements (SLAs). However, the previous load balancing schemes make migration decisions based on deterministic resource demand estimation and workload characterization, without considering their stochastic properties. By studying real world traces, we show that the resource demand and workload of virtual machines are highly dynamic and bursty, which can cause these schemes to make inefficient migrations for load balancing. To address this problem, in this paper we propose a stochastic load balancing scheme which aims to provide probabilistic guarantee against the resource overloading with virtual machine migration, while minimizing the total migration overhead. Our scheme effectively addresses the prediction of the distribution of resource demand and the multidimensional resource requirements with stochastic characterization. Moreover, as opposed to the previous works that measure the migration cost without considering the network topology, our scheme explicitly takes into account the distance between the source physical machine and the destination physical machine for a virtual machine migration. The trace-driven experiments show that our scheme outperforms the previous schemes in terms of SLA violation and the migration cost.

Traffic congestion control is pivotal for intelligent transportation systems. Previous works optimize vehicle speed for different objectives such as minimizing fuel consumption and minimizing travel time. However, they overlook the possible congestion generation in the future (e.g., in 5 minutes), which may degrade the performance of achieving the objectives. In this article, we propose a vehicle Trajectory–based driving speed OPtimization strategy ( TOP ) to minimize vehicle travel time and meanwhile avoid generating congestion. Its basic idea is to adjust vehicles’ mobility to alleviate road congestion globally. TOP has a framework for collecting vehicles’ information to a central server, which calculates the parameters depicting the future road condition (e.g., driving time, vehicle density, and probability of accident). Based on the collected information, the central server also measures the friendship among the vehicles and considers the delay caused by red traffic signals to help estimating the vehicle density of the road segments. The server then formulates a non-cooperative Stackelberg game considering these parameters, in which when each vehicle aims to minimize its travel time, the road congestion is also proactively avoided. After the Stackelberg equilibrium is reached, the optimal driving speed for each vehicle and the expected vehicle density that maximizes the utilization of the road network are determined. Our real trace analysis confirms some characteristics of vehicle mobility to support the design of TOP . Extensive trace-driven experiments show the effectiveness and superior performance of TOP in comparison with other driving speed optimization methods.

As Electric Vehicles (EVs) become increasingly popular, their battery-related problems (e.g., short driving range and heavy battery weight) must be resolved as soon as possible. Velocity optimization of EVs to minimize energy consumption in driving is an effective alternative to handle these problems. However, previous velocity optimization methods assume that vehicles will pass through traffic lights immediately at green traffic signals. Actually, a vehicle may still experience a delay to pass a green traffic light due to a vehicle waiting queue in front of the traffic light. Also, as velocity optimization is for individual vehicles, previous methods cannot avoid rear-end collisions. That is, a vehicle following its optimal velocity profile may experience rear-end collisions with its frontal vehicle on the road. In this article, for the first time, we propose a velocity optimization system that enables EVs to immediately pass green traffic lights without delay and to avoid rear-end collisions to ensure driving safety when EVs follow optimal velocity profiles on the road. We collected real driving data on road sections of US-25 highway (with two driving lanes in each direction and relatively low traffic volume) to conduct extensive trace-driven simulation studies. Results show that our velocity optimization system reduces energy consumption by up to 17.5% compared with real driving patterns without increasing trip time. Also, it helps EVs to avoid possible collisions compared with existing collision avoidance methods.

Information-aware connected and automated vehicles (CAVs) have drawn great attention in recent years due to their potentially significant positive impacts on roadway safety and operational efficiency. In this paper, we conduct an in-depth review of three basic and key interrelated aspects of a CAV: sensing and communication technologies; human factors; and information-aware controller design. First, the different vehicular sensing and communication technologies and their protocol stacks, to provide reliable information to the information-aware CAV controller, are thoroughly discussed. Diverse human factors, such as user comfort, preferences, and reliability, to design the CAV systems for mass adaptation are also discussed. Then, the different layers of a CAV controller (route planning, driving mode execution, and driving model selection) considering human factors and information through connectivity are reviewed. In addition, the critical challenges for the sensing and communication technologies, human factors, and information-aware controller are identified to support the design of a safe and efficient CAV system while considering user acceptance and comfort. Finally, the promising future research directions of these three aspects are discussed to overcome existing challenges to realize a safe and operationally efficient CAV.

Always keeping a certain distance between vehicles in a platoon system is important for collision avoidance. Centralized platoon systems let the leader vehicle determine and notify the velocities of all the vehicles in the platoon. Unfortunately, such a centralized method generates high packet drop rate and communication delay due to the leader vehicle's limited communication capability. Therefore, we propose a decentralized platoon network, in which each vehicle determines its velocity by only communicating with the vehicles in a short range. However, the multiple simultaneous transmissions between different pairs of vehicles may interfere with each other. By leveraging a typical feature of a platoon, we devise a channel allocation algorithm, called the Fast and Lightweight Autonomous channel selection algorithm (FLA), in which each vehicle determines its channel simply based on its distance to the leader vehicle. We also devise a strategy, in which a succeeding vehicle uses its stored common velocity profile when it is disconnected from its preceding vehicle and then adjusts its velocity once the connection is built. We conduct experiments on NS-3 and Matlab to evaluate the performance of our proposed methods and implement a real-world prototype by equipping vehicles with Android mobile devices. The experimental results demonstrate the superior performance of our decentralized platoon network over the previous centralized platoon networks.

Resource under-utilization is prevalent in data centers as a result of substantial stochastic workload variations and resource provision guarantee for the requirements to provide high-level service availability. To avoid resource under-utilization, recently, the Economy cloud computing service class (Economy class in short) is proposed that sells allocated but unused (i.e., reclaimed) physical resources with long-term service level objectives (SLOs). Though the Economy class would lead to revenue benefit, how much of the reclaimed resources should be used as the Economy class (i.e., capacity planning) and how to set its price and SLAs to maximize revenue benefit have not been studied. To address this problem, in this paper, we propose a multi-level SLA for Economy class, which has multiple levels of SLOs and penalty. We then present a rigorous theoretical analysis on the capacity planning and pricing of the new Economy class with an objective to maximize the revenue of the cloud computing service provider. Theoretical results are validated using resource utilization data derived from a publicly available Google cluster workload trace dataset.

A large-scale cloud data center needs to provide high service reliability and availability with low failure occurrence probability. However, current large-scale cloud data centers still face high failure rates due to many reasons such as hardware and software failures, which often result in task and job failures. Such failures can severely reduce the reliability of cloud services and also occupy huge amount of resources to recover the service from failures. Therefore, it is important to predict task or job failures before occurrence with high accuracy to avoid unexpected wastage. Many machine learning and deep learning based methods have been proposed for the task or job failure prediction by analyzing past system message logs and identifying the relationship between the data and the failures. In order to further improve the failure prediction accuracy of the previous machine learning and deep learning based methods, in this paper, we propose a failure prediction algorithm based on multi-layer Bidirectional Long Short Term Memory (Bi-LSTM) to identify task and job failures in the cloud. The goal of Bi-LSTM prediction algorithm is to predict whether the tasks and jobs are failed or completed. The trace-driven experiments show that our algorithm outperforms other state-of-art prediction methods with 93% accuracy and 87% for task failure and job failures respectively.

With recent experience in multiple large-scale disasters, it has been widely confirmed that the severity of a disaster is greatly dependent on the effectiveness of ambulance dispatching during disaster phase. However, previous base station (i.e., temporary or permanent hospital) based ambulance redeployment methods and dynamic ambulance scheduling methods cannot handle the ambulance dispatching problem in catastrophic situations. In this paper, we present MobiAmbulance: a human Mobility based Ambulance dispatching system that aims to maximize the total number of fulfilled patient pick-up requests, and minimize the driving delay of the fulfilled requests. We studied a state-scale human mobility dataset and found that the change of vehicle flow rate can be utilized to determine the connection status between road segments, and the distribution of people in catastrophic situations is drastically different from that in normal situations. Then, we develop a method to determine the road network connection status and the set of road segments that can still be driven through by ambulances after disaster. Based on the updated road network graph, we develop an ambulance dispatching method based on weighted driving route to maximize the total number of fulfilled patient pick-up requests, and minimize the driving delays of the fulfilled requests. Our trace-driven experiments demonstrate the superior performance of MobiAmbulance over other comparison methods.

Current routing algorithms for Vehicle Delay Tolerant Networks (VDTNs) use contact information, centrality information or location information to deliver packets. Location based routing algorithms can achieve better performance than contact and centrality based routing algorithms since VDTNs have high vehicle mobility, sparsity of vehicles, short contact durations, disruption and intermittent connectivity, which increase the difficulties of routing by contact or centrality information. However, location based routing algorithms always build on the assumption that the trajectories of vehicles are always known beforehand. However, in reality, only a small percentage of vehicles would like to disclose their trajectories and destinations. To solve this problem, in this paper, we propose a Distributed Trajectory Tracking System (DTTS) to track the trajectory of each vehicle distributively by taking advantage of road side units (RSUs) and the special feature of road map. Furthermore, we implement a previous routing algorithm on top of DTTS. Finally, we design an efficient forward tracing based routing algorithm based on DTTS. Base on the trajectory information stored in RSUs, the packet is forwarded along the shortest path from its current location to the target vehicle. Extensive trace-driven simulation demonstrates that the DTTS based routing has better routing success rate, shorter average delays and fewer routing hops in comparison with state-of-the-art routing algorithms in VDTNs.

Production data analytic workloads typically consist of a majority of jobs with small input data sizes and a small number of jobs with large input data sizes. Recent works advocate scale-up/scale-out heterogeneous clusters (in short Hybrid clusters) to handle these heterogeneous workloads, since scaleup machines (i.e., adding more resources to a single machine) can process small jobs faster than simply scaling out the cluster with cheap machines. However, there are several challenges for job placement and data placement to implement such a Hybrid cluster. In this paper, we propose a job placement strategy and a data placement strategy to solve the challenges. The job placement strategy places a job to either scale-up or scale-out machines based on the job's characteristics, and migrates jobs from scale-up machines to under-utilized scale-out machines to achieve load balance. The data placement strategy allocates data replicas in the two types of machines accordingly to increase the data locality in Hybrid cluster. We implemented a Hybrid cluster on Apache YARN, and evaluated its performance using a Facebook production workload. With our proposed strategies, a Hybrid cluster can reduce the makespan of the workload up to 37% and the median job completion time up to 60%, compared to traditional scale-out clusters with state-of-the-art schedulers.

In recent years, driving route recommendation has attracted growing interest from researchers and industries. However, previously proposed route recommendation systems cannot jointly consider different factors (e.g., fuel consumption, travel time, air quality) in parallel with different weights entered by users. In addition, as users set the weights based on their own evaluation (e.g., much higher weight on air quality than fuel consumption), which may lead to a very unbalanced route (e.g., worst travel time and worst fuel consumption) that actually is not what the users desire. To handle these issues, in this paper, we propose a routing recommendation system, called Vehicle Routing Trifecta (VRT), which can jointly blend different considered factors with different weights entered by users while still producing well-balanced routes that conform user normal desire. VRT consists of two innovative components. First, we establish three different predictors for air quality, travel time, and fuel consumption estimations of each road segment in the road network. Second, we design an optimal route selector, which consists of the solution of a multi-criteria optimization problem based on the given user preference on three different aspects (e.g., air quality, travel time, and fuel consumption). We conduct extensively simulation studies based on the real-world, geo-tagged datasets to evaluate VRT. The comparative studies with other existing routing recommendation systems show the superior performance of VRT in terms of recommending routes that meet user entered preference.

Accurate estimations on vehicle fuel consumption and pollution emission on roads are important for vehicle velocity optimization and driving route planning. Existing methods for such estimations only consider vehicle driving speed and acceleration but neglect the influence of road gradient. This is mainly because the road gradients for most road networks are not available and none of existing methods for road gradient estimation can be conducted inexpensively in practice and keep high road gradient estimation accuracy simultaneously. Thus, how to estimate the road gradient conveniently and accurately is an important but challenging problem. To handle this challenge, we propose a new road gradient estimation system which estimates the road gradient only using a smartphone. When a vehicle is driving, a smartphone in the vehicle continuously measures vehicle states (velocity, acceleration, steering rate, position), which are used to estimate the road gradient. To eliminate measuring noise and drift noise, the deviation between the measured value and estimated value is used to adjust the estimated value. Since measured vehicle states when a vehicle changes lane adversely influence the accuracy of road gradient estimation, we design lane change detection to eliminate such influences. Finally, given a group of road gradient estimates for a given route, we use the track fusion algorithm to further eliminate measuring noise and drift noise and improve road gradient estimation accuracy. We conducted driving experiments in a city area to evaluate our system. The experimental results show that our system's estimation error is reduced by 22% compared with existing methods. The results also demonstrate the accuracy of our lane change detection. Finally, we integrated the road gradient values into vehicle fuel consumption and air pollution emission model to estimate fuel consumption and air pollution emission and found that the estimation values increase by 33.4% compared with the values without considering road gradient.

The optical circuit switch (OCS) in recently proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) can only be used to transfer large flows (i.e., flows with a large size of data). Current job schedulers for data-parallel frameworks are not suitable for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we propose Co-scheduler, a job scheduler for dataparallel frameworks that aims to improve job performance by attempting to place the tasks of a job to aggregate enough data traffic to take advantage of OCS and minimize the CCT in Hybrid-DCN. Specifically, for each job, Co-scheduler computes a guideline on the number of racks to place the job's input data and to run the job's map tasks, so that the job can potentially take full advantage of OCS to transfer its data. When the map tasks of a job complete, Co-scheduler computes all the possible schedules of the reduce tasks of the job. Each possible schedule includes the number of racks to schedule the reduce tasks that enables the job to use OCS to transfer its data, and the number of reduce tasks to place on each of the racks that minimizes CCT of the job. Next, Co-scheduler selects a best schedule among all the possible schedules so that the job completion time is minimized. Finally, Co-scheduler schedules the map tasks and reduce tasks of the job based on the computed guideline and best schedule. The evaluation demonstrates that compared to the state-of-theart schedulers, Co-scheduler achieves performance improvements on makespan, average job completion time, and average CCT by up to 51.2%, 54.6% and 73.6%, respectively.

The proliferation in the use of the Internet of Things (IoT) and Machine Learning (ML) techniques in edge computing systems have paved the way of using Intelligent Cognitive Assistants (ICA) for assisting people in working, learning, transportation, healthcare, and other activities. A challenge here is how to schedule application tasks between the three tiers in the edge computing system (i.e., remote cloud, fog and edge devices) according to several considered factors such as latency, energy, and bandwidth consumption. However, the state-of-the-art approaches for this challenge fall short in providing a schedule in real time for critical ICA tasks due to complex calculation phase. In this paper, we propose a novel ReInforcement Learning based Task Assignment approach, RILTA, that ensures the timeliness guaranteed execution of ICA tasks with high energy efficiency. We first formulate the task-scheduling problem in the edge computing systems considering timeliness and energy consumption in ICA applications. We then propose a heuristic for solving the problem and design the reinforcement model based on the output of the proposed heuristic. Our simulation results show that RILTA can reduce the task processing time and energy consumption with higher timeliness guarantee in comparison to other existing methods by 13 - 22% and 1 - 10% respectively.

In spite of many advantages of hybrid electrical/optical datacenter networks (Hybrid-DCN), current job schedulers for data-parallel frameworks are not suitable for Hybrid-DCN, since the schedulers do not aggregate data traffic to facilitate using optical circuit switch (OCS). In this paper, we propose JobPacker, a job scheduler for data-parallel frameworks in Hybrid-DCN that aims to take full advantage of OCS to improve job performance. JobPacker aggregates the data transfers of a job in order to use OCS to improve data transfer efficiency. It first explores the tradeoff between parallelism and traffic aggregation for each shuffle-heavy recurring job, and then generates an offline schedule including which racks to run each job and the sequence to run the recurring jobs in each rack that yields the best performance. It has a new sorting method to prioritize recurring jobs in offline-scheduling to prevent high resource contention while fully utilizing cluster resources. In real-time scheduler, JobPacker uses the offline schedule to guide the data placement and schedule recurring jobs, and schedules non-recurring jobs to the idle resources not assigned to recurring jobs. Trace-driven simulation and GENI-based emulation show that JobPacker reduces the makespan up to 49% and the median completion time up to 43%, compared to the state-of-the-art schedulers in Hybrid-DCN.

In data-intensive parallel computing clusters, it is important to provide deadline-guaranteed service to jobs while minimizing resource usage (e.g., network bandwidth and energy). Under the current computing framework (that first allocates data and then schedules jobs), in a busy cluster with many jobs, it is difficult to achieve these objectives simultaneously. We model the problem to simultaneously achieve the objectives using integer programming, and propose a heuristic Cooperative job Scheduling and data Allocation method (CSA). CSA novelly reverses the order of data allocation and job scheduling in the current computing framework, i.e., changing data-first-job-second to job-first-data-second. It enables CSA to proactively consolidate tasks with more common requested data to the same server when conducting deadline-aware scheduling, and also consolidate the tasks to as few servers as possible to maximize energy savings. This facilitates the subsequent data allocation step to allocate a data block to the server that hosts most of this data's requester tasks, thus maximally enhancing data locality and reduce bandwidth consumption. CSA also has a recursive schedule refinement process to adjust the job and data allocation schedules to improve system performance regarding the three objectives and achieve the tradeoff between data locality and energy savings with specified weights. We implemented CSA and a number of previous job schedulers on Apache Hadoop on a real supercomputing cluster. Trace-driven experiments in the simulation and the real cluster show that CSA outperforms other schedulers in supplying deadline-guarantee and resource-efficient services.

Live virtual machine migration technique allows migrating an entire OS with running applications from one physical host to another, while keeping all services available without interruption. It provides a flexible and powerful way to balance system load, save power and tolerant faults in data centers. Meanwhile, with the stringent requirements of latency, scalability, and availability, an increasing number of applications are deployed across distributed cloud data-centers. However, existing live migration approaches still suffer from long downtime and serious performance degradation in cross data-center scenes due to the mass of dirty retransmission, which limits the ability of cross data-center scheduling. In this paper, we propose a system named Memory/disk operation aware Lightweight VM Live Migration across data-centers with low performance impact (MLLM). It significantly improves the cross data-center migration performance by reducing the amount of dirty data in the migration process. In MLLM, we predict disk read workingset (i.e., more frequently read contents) and memory write workingset (i.e., more frequently write contents) based on the access sequence trace. And then we adjust the migration models and data transfer sequence based on the workingset information. We also present two optimizing methods to filter unused blocks and to de-duplicate data content by a hot data cache, thereby greatly decreasing the amount of data to be transferred. We implement MLLM on the QEMU/KVM platform and conduct several real-world experiments. The experimental results show that our method averagely reduces 67.0% of total migration time and 41.6% service downtime over existing methods.

With the application of Wireless Power Transfer (WPT) techniques for Electric Vehicles (EVs), public transportation EVs are expected to be continuously operable without recharging downtime. A road segment equipped with an inmotion wireless charger is called a charger lane. To maximize the service efficiency of deployed in-motion wireless chargers without suffering from traffic congestion, we must properly manage the traffic of the EVs and coordinate their arrivals at the charger lanes to avoid the generation of traffic congestion at the charger lanes and on the road segments to them. In this paper, we propose WPT-Opt, a game theoretic approach for Optimizing in-motion wireless charging service efficiency, minimizing EVs' time spent on the way to the charger, and avoiding traffic congestion at the charger lanes, to fulfill this task. We studied a metropolitan-scale dataset of public transportation EVs, and observed the EVs' spatial and temporal preference in selecting chargers, competition for chargers during busy charging times, and the relationship between vehicle density and driving velocity on a road segment. Then, we formulate a non-cooperative Stackelberg game between all the EVs and a central controller, in which each EV aims at minimizing its charging time cost to its selected target charger, while the central controller tries to maximally avoid the generation of congestion on the in-motion wireless chargers and the road segments to them in the near future. Our tracedriven experiments on SUMO demonstrate that WPT-Opt can maximally reduce the average charging time cost of the EVs by approximately 200% during different hours of a day.

Functioning as an intermediary between tenants and cloud providers, cloud service brokerages (CSBs) can bring about great benefits to the cloud market. As energy costs of cloud computing have been increasing rapidly, there is a need for cloud providers to optimize energy efficiency while maintain high service level performance to tenants, not only for their own benefit but also for social welfares. Thus, for green cloud companies, two questions have arisen: 1) under what pricing policies from the cloud providers to the CSB, a profit-driven CSB is willing to minimize the total energy cost while satisfy tenant demands and 2) how should a CSB distribute tenants demands to achieve this objective? To address question 1), we find a pricing policy for cloud providers such that maximizing CSBs profit is equivalent to minimizing cloud providers energy cost. To address question 2), we first devise a greedy solution, and then propose an approximation algorithm and a decomposition-based solution with a constant approximation ratio. Both simulation and real-world Amazon EC2 experimental results demonstrate the effectiveness of our pricing policy to incentivize CSBs to save energy and the superior performance of our algorithms in energy efficiency and resource utilization over the previous algorithms.

As the typical delay tolerant networks, the satellite networks possess the intermittent connections, the large-scale time delays and the time-varying topologies. Obviously, such features seriously affect the delivery of mission data with certain requirements on the traffic or latency. To decrease the delivery time, existing relay-based contact graph routing (CGR) method selects the earliest reachable path to forward mission data. However, this method cannot guarantee the transmission of a large amount data and wastes the rare contact opportunities. Therefore, in this paper, we focus on the transmission QoS problem of multiple missions over the satellite networks, and design a QoS support routing strategy to achieve multiple flow-maximizing paths with acceptable delivery delays. Especially, with the storage time aggregated graph (STAG), we construct an on-demand mission model to depict both the network dynamic characteristics and the different mission requirements, and then reduce the QoS support problem as a graph-based maximum flow problem. To solve this problem, one STAG-based multiple flow-maximizing routing scheme is proposed to ensure the mission QoS and match the rare network resources, which has low computation complexity. Finally, compared with CGR, simulation results demonstrate the proposed scheme can achieve a higher mission completion rate and resource utilization rate.

Data collection is a fundamental and critical function of wireless sensor networks (WSNs) for the cyber-physical systems (CPS) to estimate the state of the physical world. However, unstable network conditions impose significant challenges in guaranteeing the data accuracy that is essential for the reliable estimation of physical states. Without efficiently resolving congestion during data transmission in WSNs, packet loss due to congestion can significantly degrade the data quality. Various congestion control schemes have been proposed to address this issue. Most of them rely on reducing transmitted data samples to eliminate the congestion, which, however, could lead to abysmally high estimation error. In this paper, we analyze the impact of congestion control on the data accuracy and propose a Congestion-Adaptive Data Collection scheme (CADC) to efficiently resolve the congestion under the guarantee of data accuracy. CADC mitigates congestion by adaptive lossy compression while ensuring a given overall data estimation error bound in a distributed manner. Considering that for a CPS application different data items may have different priorities, we also propose a weighted CADC scheme such that the data with higher priority has less distortion. We further adapt CADC to guarantee the accuracy of specific aggregate computations. Extensive simulations demonstrate the effectiveness and efficiency of CADC.

Large-scale data stores are an increasingly important component of cloud datacenter services. However, cloud storage system usually experiences data loss, hindering data durability. Three-way random replication is commonly used to lead better data durability in cloud storage systems. However, three-way random replication cannot effectively handle correlated machine failures to prevent data loss. Although Copyset Replication and Tiered Replication can reduce data loss in correlated and independent failures, and enhance data durability, they fail to leverage different data popularities to substantially reduce the storage cost and bandwidth cost caused by replication. To address these issues, we present a popularity-aware multi-failure resilient and cost-effective replication (PMCR) scheme for high data durability in cloud storage. PMCR splits the cloud storage system into primary tier and backup tier, and classifies data into hot data, warm data and cold data based on data popularities. To handle both correlated and independent failures, PMCR stores the three replicas of the same data into one Copyset formed by two servers in the primary tier and one server in the backup tier. For the third replicas of warm data and cold data in the backup tier, PMCR uses the compression methods to reduce storage cost and bandwidth cost. Extensive numerical results based on trace parameters and experimental results from real-world Amazon S3 show that PMCR achieves high data durability, low probability of data loss, and low storage cost and bandwidth cost compared to previous replication schemes.

Hybrid peer-to-peer assisted cloud-based video-on-demand (VoD) systems augment cloud-based VoD systems with P2P networks to improve scalability and save bandwidth costs in the cloud. In these systems, the VoD service provider (e.g., NetFlix) relies on the cloud to deliver videos to users and pays for the cloud bandwidth consumption. The users can download videos from both the cloud and peers in the P2P network. It is important for the VoD service provider to i) minimize the cloud bandwidth consumption, and ii) guarantee users' satisfaction (i.e., quality-of-experience). Though previous adaptive bitrate streaming (ABR) methods improve video playback smoothness, they cannot achieve these two goals simultaneously. To tackle this challenge, we propose AutoTune, a game-based adaptive bitrate streaming method. In AutoTune, we formulate the bitrate adaptation problem in ABR as a noncooperative Stackelberg game, where VoD service provider and the users are players. The VoD service provider acts as a leader and it decides the VoD service price for users with the objective of minimizing cloud bandwidth consumption while ensuring users' participation. In response to the VoD service price, the users select video bitrates that lead to maximum utility (defined as a function of its satisfaction minus associated VoD service fee). Finally, the Stackelberg equilibrium is reached in which the cloud bandwidth consumption is minimized while users are satisfied with selected video bitrates. To enhance the performance of AutoTune, we further propose the reputation-based incentive scheme and the popularity-based cache management scheme. Experimental results from the PeerSim simulator and the PlanetLab real-world testbed show that compared to existing methods, AutoTune can provide high user satisfaction and save cloud bandwidth consumption. Also, the proposed enhancement schemes are effective in improving the performance of AutoTune.

In this paper, price-controlled energy management is investigated in a bi-level optimization framework, that is, energy scheduling problem of smart homes (SHs) and generation scheduling and unit commitment (UC) problems of a generation company (GENCO). SHs as the responsive customers (respect to the energy management) include a variety of sources such as photovoltaic (PV) panels, diesel generator, and battery as an energy storage. In addition, SHs are able to transact electricity with the GENCO through the power system. In this paper, the goal of GENCO is to design an optimal energy management scheme (optimal price of electricity) to maximize its daily profit. Herein, each SH reacts to the energy management scheme and reschedules its energy resources to minimize its daily operation cost. In this paper, a scenario-based stochastic optimization approach is applied in the energy scheduling problem of an SH to address the variability and uncertainty issues of the PV panels. Also, a combination of genetic algorithm (GA) and linear programming is applied as the optimization tool for the energy scheduling problem of an SH. Moreover, lambda-iteration economic dispatch and GA techniques are applied to solve the generation scheduling and UC problems of the GENCO, respectively. The numerical study demonstrates that in order to reach the maximum profit of GENCO, the energy management must be optimally designed and implemented; otherwise, the energy management scheme may result in detriment. Moreover, it is shown that each SH is able to get benefit from the energy management scheme and minimize its daily operation cost.

The planning and operation problems of parking lots of plug-in electric vehicles (PEVs) are studied in this paper. Herein, each distribution company (DISCO) allocates the parking lots to the electrical feeders to minimize the power loss and expected energy not supplied of the system, and consequently minimize the total cost of the planning problem over the given time horizon. In addition, the generation company (GENCO) manages the charging time of PEVs parked in the parking lots to defer the more expensive and pollutant generation units, and as a result maximize its daily profit. In both planning and operation problems, the behavioral model of PEVs' drivers are modeled with respect to the value of incentive and their distance from the parking lots. To achieve the realistic results in the planning problem of each DISCO, several economic and technical factors including yearly inflation and interest rates, hourly and daily variations of the load demand, yearly load growth of the system, and yearly growth rate of PEVs' application are considered. The optimization problems of each DISCO and GENCO are solved applying quantum-inspired simulated annealing algorithm and genetic algorithm, respectively. It is demonstrated that the behavioral model of drivers, their driving patterns, and even the type of PEVs can remarkably affect the outcomes of planning and operation problems. It is shown that the optimal allocation of parking lots can minimize every DISCO's planning cost and optimal charging management of PEVs can increase the GENCO's daily profit.

In spite of many shuffle-heavy jobs in current commercial data-parallel clusters, few previous studies have considered the network traffic in the shuffle phase, which contains a large amount of data transfers and may adversely affect the cluster performance. In this paper, we propose a network-aware scheduler (NAS) that handles two main challenges associated with the shuffle phase for high performance: i) balancing cross-node network load, and ii) avoiding and reducing cross-rack network congestion. NAS consists of three main mechanisms: i) map task scheduling (MTS), ii) congestion-avoidance reduce task scheduling (CA-RTS) and iii) congestion-reduction reduce task scheduling (CR-RTS). MTS constrains the shuffle data on each node when scheduling the map tasks to balance the cross-node network load. CA-RTS distributes the reduce tasks for each job based on the distribution of its shuffle data among the racks in order to minimize cross-rack traffic. When the network is congested, CR-RTS schedules reduce tasks that generate negligible shuffle traffic to reduce the congestion. We implemented NAS in Hadoop on a cluster. Our trace-driven simulation and real cluster experiment demonstrate the superior performance of NAS on improving the throughput (up to 62%), reducing the average job execution time (up to 44%) and reducing the cross-rack traffic (up to 40%) compared with state-of-the-art schedulers.

Task scheduling and preemption are two important functions in data-parallel clusters. Though directed acyclic graph task dependencies are common in data-parallel clusters, previous task scheduling and preemption methods do not fully utilize such task dependency to increase throughput since they simply schedule precedent tasks prior to their dependent tasks or neglect the dependency. We notice that in both scheduling and preemption, choosing a task with more dependent tasks to run allows more tasks to be runnable next, which facilitates to select a task that can more increase throughput. Accordingly, in this paper, we propose a Dependency-aware Scheduling and Preemption system (DSP) to achieve high throughput. First, we build an integer linear programming model to minimize the makespan (i.e., the time when all jobs finish execution) with the consideration of task dependency and deadline, and derive the target server and start time for each task, which can minimize the makespan. Second, we utilize task dependency to determine tasks' priorities for preemption. Finally, we propose a method to reduce the number of unnecessary preemptions that cause more overhead than the throughput gain. Extensive experimental results based on a real cluster and Amazon EC2 cloud service show that DSP achieves much higher throughput compared to existing strategies.

There is a dramatic increase in the variety of virtual machines (VMs) and complexity of VM placement problems in clouds. Previous VM placement approaches attempt to accommodate more VMs efficiently on fewer PMs by balancing the resource usages across multiple dimensions. However, these approaches are not sufficiently accurate in measuring the quality of the PMs in terms of fully utilizing PM resource and having the potential to accommodate more VMs. Therefore, it is critical to design a new method that can more accurately measure the probability of a PM of fully utilizing its resources after accommodating a given VM with the consideration of different types of VMs. In addition, anti-collocation constraints must be handled efficiently. We propose a PageRank based VM placement algorithm with anti-collocation constraints (PageRankVM). PageRankVM defines the best PM resource usage profile, which means that the PM has full resource utilization for every resource dimension, and then ranks PM profiles according to their convergence of transferring (by accommodating VMs) to the best profile. PageRankVM then places a given VM to the PM based on the ranks of the resulted PM profiles after accommodating the VM with the consideration of anti-collocation constraints. Compared to previous approaches, PageRankVM effectively measures the ability of different PM profiles to reach the best profiles by accommodating a given VM, and hence differentiates the effectiveness of different VM placement decisions. We conducted extensive trace-driven simulation and GENI testbed experiments and demonstrated that PageRankVM has superior performance compared with other methods in terms of reducing the number of PMs, the energy consumption, the number of VM migrations, and the service level objective (SLO) violations.

In a modern cloud datacenter, a cascading failure will cause many Service Level Objective (SLO) violations. In a cascading failure, when a set of physical machines (PMs) in a failure domain are failed, their workloads are transferred to the PMs in another failure domain to continue. However, the new domain receiving additional workloads may become overloaded due to the resource oversubscription feature in the cloud, which easily leads to domain failures and subsequent workload transfer to other domains. This process repeats and a cascading failure is created finally. However, few previous methods can effectively handle the cascading failures. To handle this problem, we propose a Cascading Failure Resilience System (CFRS), which incorporates three methods: Overload-Avoidance VM Reassignment (OAVR), VM backup set placement (VMset) and Dynamic Oversubscription Ratio Adjustment (DOA). The experiments in trace-driven simulation show that CFRS outperforms other comparison methods in terms of the number of domain failures, the number of failed PMs and the number of SLO violations.

With the rapid development of web applications in datacenters, network latency becomes more important to user experience. The network latency will be greatly increased by incast congestion, in which a huge number of requests arrive at the front-end server simultaneously. Previous incast problem solutions usually handle the data transmission between the data servers and the front-end server directly, and they are not sufficiently effective in proactively avoiding incast congestion. To further improve the effectiveness, in this paper, we propose a Proactive Incast Congestion Control system (PICC). Since each connection has bandwidth limit, PICC novelly limits the number of data servers concurrently connected to the front-end server to avoid the incast congestion through data placement. Specifically, the front-end server gathers popular data objects (i.e., frequently requested data objects) into as few data servers as possible, but without overloading them. It also re-allocates the data objects that are likely to be concurrently or sequentially requested into the same server. As a result, PICC reduces the number of data servers concurrently connected to the front-end server (which avoids the incast congestion), and also the number of connection establishments (which reduces the network latency). Since the selected data servers tend to have long queues to send out data, to reduce the queuing latency, PICC incorporates a queuing delay reduction algorithm that assigns higher transmission priorities to data objects with smaller sizes and longer queuing times. The experimental results on simulation and a real cluster based on a benchmark show the superior performance of PICC over previous incast congestion problem solutions.

The exceptional increase in the usage of smartphones has contributed to a massive increase in data traffic from application servers to the smartphones, which not only strains their computation capacities and batteries but also bogs down the last hop in data transmission. For this problem, traffic redundancy elimination (TRE) is an effective solution, in which a chunk to be transmitted could be directly fetched from the receiver's cache. However, existing TRE solutions either cannot be directly applied to or are not suitable for smartphones due to high computing and energy overhead imposed on smartphones. To address this problem, in this paper, we propose a novel TRE system, called TailoredRE, which consists of three components. First, each smartphone has a clone in the cloud that is responsible for computation intensive tasks including parsing traffic and detecting redundancy. Second, considering that each mobile user has certain applications (e.g., YouTube) to use in daily life, each smartphone's clone selectively chooses the applications that are most frequently used by the user and also have high redundancy ratios to cache data. Third, considering that some users always have common favorite applications, TailoredRE clusters their clones together to cooperatively conduct the redundancy detection task in order to reduce the cache resource consumption in the cloud. We collected traces from eleven applications including Web Browser, YouTube, CNN, Quora, Instagram and Facebook, and used the traces in simulation. We also implemented and open-sourced TailoredRE and conducted prototype-based experiments. Experiment results show that TailoredRE can achieve much higher cache hit rate, end-to-end throughput, bandwidth saving and energy efficiency compared with previous TRE methods.

In recent years, many efforts have been devoted to reducing vehicles' energy consumption through optimizing their velocities. However, all previous methods neglect avoiding vehicle collision when calculating vehicles' optimal velocity profiles. In this paper, we formulate a new problem, called the Collision-Aware vehicle Energy consumption Minimization (CAEM) problem that calculates the optimal velocity profiles which avoid vehicle collision. CAEM is more difficult to solve than the traditional velocity optimization problem that is for a single vehicle, since CAEM needs to take into account the mobility of all the vehicles together to avoid collision. This problem is a convex problem that cannot be directly solved by existing methods. Further, it is impractical to get the mobility information of all vehicles at the beginning. Even if it is feasible, the computation efficiency is very low. We propose a novel method that can tackle these challenges. In order to keep each vehicle velocity as stable as possible to reduce the energy consumption, it builds a light schematic map to help identify the green light time interval of each traffic light in the source-destination route of a vehicle, during which the vehicle must drive through the traffic light. Rather than considering the mobility of all vehicles at a time, it calculates each vehicle's velocity profile in sequence based on the starting time to prevent the vehicle from colliding with all the previously scheduled vehicles. Finally, CAEM is transformed to non-convex problem and can be solved by existing optimization methods. Simulation and real-world testbed experimental results demonstrate the superior performance of our method over the previous methods.

DISboards is a discussion forum that provides a platform for knowledge sharing on planning and resources for Disney-related travel (Disney World, Disney Cruise Line, etc.). Since no previous work has been devoted to studying the online social networks (SNs) in the forums, we examine the SN and knowledge sharing activities in DISboards as a case study of discussion forums. Based on a large amount of data collected, we provide an in-depth study of DISboards. In particular, we analyzed SN structure, effect of SN in the forum, category characteristics and so on. We found that users with more friends are generally more active in the forum; teens are more active and constitute a significant part of the SN. We clustered the selected categories (e.g., resorts, dining, and hotels) into three groups: report, fact, discussion, and characterized their properties. Most users focus narrowly on only a few categories, while very few users participate in many categories. The development of SN should be able to attract more users to involve in the forum. We believe that the results presented in this paper are crucial in understanding SN and knowledge sharing in the forums. The paper also gives an instruction for the enhancement of SNs to incentivize users' activeness in the forums.

Question and Answer (Q&A) systems aggregate the collected intelligence of all users to provide satisfying answers for questions. A well-developed Q&A system should incorporate features such as high question response rate, high answer quality, a spam-free environment for users and bridging disjoint social clusters. Previous works use reputation systems to achieve the goals. However, these reputation systems evaluate a user with an overall rating for all questions the user has answered regardless of the question categories, thus the reputation score does not accurately reflect the user's ability to answer a question in a specific category. We propose SmartQ: a reputation based Q&A System. SmartQ employs a category and theme based reputation management system to evaluate users' willingness and capability to answer various kinds of questions. The reputation system facilitates the forwarding of a question to favorable experts, which improves the question response rate and answer quality. SmartQ bridges disjoint social clusters by calculating reputation scores for each cluster on each question theme; SmartQ incorporates a lightweight spammer detection method to identify potential spammers. In order to reduce the loads of experts, we propose a strategy to recommend suggested answers from similar questions to each new question. Our trace-driven simulation on PeerSim demonstrates the effectiveness of SmartQ in providing good user experience. We then develop a real application of SmartQ and deploy it for use in a student group in Clemson University. The user feedback shows that SmartQ can provide high-quality answers for users in a community.

Mobile crowdsensing serves as a critical building block for emerging Internet of Things (IoT) applications. However, the sensing devices continuously generate a large amount of data, which consumes much resources (e.g., bandwidth, energy, and storage) and may sacrifice the Quality-of-Service (QoS) of applications. Prior work has demonstrated that there is significant redundancy in the content of the sensed data. By judiciously reducing redundant data, data size and load can be significantly reduced, thereby reducing resource cost and facilitating the timely delivery of unique, probably critical information and enhancing QoS. This article presents a survey of existing works on mobile crowdsensing strategies with an emphasis on reducing resource cost and achieving high QoS. We start by introducing the motivation for this survey and present the necessary background of crowdsensing and IoT. We then present various mobile crowdsensing strategies and discuss their strengths and limitations. Finally, we discuss future research directions for mobile crowdsensing for IoT. The survey addresses a broad range of techniques, methods, models, systems, and applications related to mobile crowdsensing and IoT. Our goal is not only to analyze and compare the strategies proposed in prior works, but also to discuss their applicability toward the IoT and provide guidance on future research directions for mobile crowdsensing.

As a popular routing protocol in wireless sensor networks (WSNs), greedy routing has received great attention. The previous works characterize its data deliverability in WSNs by the probability of all nodes successfully sending their data to the base station. Their analysis, however, neither provides the information of the quantitative relation between successful data delivery ratio and transmission power of sensor nodes nor considers the impact of the network congestion or link collision on the data deliverability. To address these problems, in this paper, we characterize the data deliverability of greedy routing by the ratio of successful data transmissions from sensors to the base station. We introduce n-guaranteed delivery which means that the ratio of successful data deliveries is not less than n, and study the relationship between the transmission power of sensors and the probability of achieving n-guaranteed delivery. Furthermore, with considering the effect of network congestion, link collision, and holes (e.g., those caused by physical obstacles such as a lake), we provide a more precise and full characterization for the deliverability of greedy routing. Extensive simulation and real-world experimental results show the correctness and tightness of the upper bound of the smallest transmission power for achieving n-guaranteed delivery.

Present approaches to achieve k-coverage for Wireless Sensor Networks still rely on centralized techniques. In this paper, we devise a distributed method for this problem, namely Distributed VOronoi based Cooperation scheme (DVOC), where nodes cooperate in hole detection and recovery. In previous Voronoi based schemes, each node only monitors its own critical points. Such methods are inefficient for k-coverage because the critical points are far away from their generating nodes in k-order Voronoi diagram, causing high cost for transmission and computing. As a solution, DVOC enables nodes to monitor others' critical points around themselves by building local Voronoi diagrams (LVDs). Further, DVOC constrains the movement of every node to avoid generating new holes. If a node cannot reach its destination due to the constraint, its hole healing responsibility will fall to other cooperating nodes. The experimental results from the real world testbed demonstrate that DVOC outperforms the previous schemes.

The past few years have seen an explosion in the popularity of online short-video sharing in YouTube. As the number of users continue to grow, the bandwidth required to maintain acceptable quality of service (QoS) has greatly increased. Peer-to-peer (P2P) architectures have shown promise in reducing the bandwidth costs; however, the previous works build one P2P overlay for each video, which provides limited availability of video providers and produces high overlay maintenance overhead. To handle these problems, in this work, we novelly leverage the existing social network in YouTube, where a user subscribes to another user’s channel to track all his/her uploaded videos. The subscribers of a channel tend to watch the channel’s videos and common-interest nodes tend to watch the same videos. Also, the popularity of videos in one channel varies greatly. We study real trace data to confirm these properties. Based on these properties, we propose SocialTube, which builds the subscribers of one channel into a P2P overlay and also clusters common-interest nodes in a higher level. It also incorporates a prefetching algorithm that prefetches higher-popularity videos. To enhance the system performance, we further propose the demand/supply-based cache management scheme and reputation-based neighbor management scheme. Extensive trace-driven simulation results and PlanetLab real-world experimental results verify the effectiveness of SocialTube at reducing server load and overlay maintenance overhead and at improving QoS for users.

In the cloud, multiple tenants share the resource of datacenters and their applications compete with each other for scarce network bandwidth. Current studies have shown that the lack of bandwidth guarantee causes unpredictable network performance, leading to poor application performance. To address this issue, several virtual network abstractions have been proposed which allow the tenants to reserve virtual clusters with specified bandwidth between the Virtual Machines (VMs) in the datacenters. However, all these existing proposals require the tenants to deterministically characterize the bandwidth demands in the abstractions, which can be difficult and result in inefficient bandwidth reservation due to the demand uncertainty. In this paper, we explore a virtual cluster abstraction with stochastic bandwidth characterization to address the bandwidth demand uncertainty. We propose Stochastic Virtual Cluster (SVC), which models the bandwidth demand between VMs in a probabilistic way. Based on SVC, we develop a stochastic framework for virtual cluster allocation, in which the admitted virtual cluster's bandwidth demands are satisfied with a high probability. Efficient VM allocation algorithms are proposed to implement the framework while reducing the possibility of link congestion through minimizing the maximum bandwidth occupancy of a virtual cluster on physical links. Using simulations, we show that SVC achieves the trade-off between the job concurrency and the average job running time, and demonstrate its effectiveness for accommodating cloud application workloads with highly volatile bandwidth demands and its improvement to work-conserving bandwidth enforcement.

The multi-commodity flow problem plays an important role in network optimization, routing, and service scheduling. With the network partitioning and the intermittent connectivity, the commodity flows in delay tolerant networks (DTNs) are time-dependent, which is very different from that over the static networks. As an NP-hard problem, existing works can only obtain sub-optimal results on maximizing the multi-commodity flow of dynamic networks. To overcome these bottlenecks, in this paper, we propose a graph-based algorithm to solve the maximum two-commodity flow problem over the DTNs. Through analyzing the relationship between the two commodities, we propose a maximum two-commodity flow theorem to simplify the coupling two-commodity flow problem as the two single-commodity flow ones. Then, with the help of the storage time aggregated graph (STAG) (a DTN model with less memory), we construct a pair of flow graphs to describe the reduced two single-commodity flows (addition flow and subtraction flow), and design the corresponding flow calculation methods. Moreover, we design a STAG-based dynamic combined flow algorithm to maximize the two-commodity flow. Finally, the computational complexity of the proposed algorithm is analyzed, and its efficacy has also been demonstrated through an illustrative example and numerical simulations.

Image storage systems are designed to support images sharing and retrieving applications, especially for smartphone and other mobile devices. Existing image storage systems store images in random data servers without considering their similarities. When a front-end server sends out an image query, it receives similar images from a large number of data servers, causing possible network incast congestion and long query latency. To solve this problem, we propose CStorage, an efficient classification-based image storage system. In CStorage, similar images are stored in the same data server. Thus, a front-end server receives query results from a single data server, which reduces the occurrence of incast congestion as well as the image retrieval latency. CStorage also leverages the deep learning technique to cluster images and provides a high precision and recall rates. Experimental results show the effectiveness of CStorage in reducing image retrieval latency and improving precision and recall rates of searching results.

Nowadays, a large amount of services are deployed on the edge of the network from the cloud since processing data at the edge can reduce response time and lower bandwidth cost for applications such as healthcare in smart homes. Resource management is very important in the edge computing since it is able to increase the system efficiency and improve the quality of service. A common approach for resource management in edge computing is to assign tasks to the remote cloud or edge devices just according to several factors such as energy, bandwidth consumption, and latency. However, the approach is insufficiently efficient and falls short in meeting the requirements of handling health emergency when being applied in smart homes for healthcare. In this paper, we propose a task scheduling approach called HealthEdge that sets different processing priorities for different tasks based on the collected data on human health status and determines whether a task should run in a local device or a remote cloud in order to reduce its total processing time as much as possible. Based on a real trace from five patients, we conduct a trace-driven experiment to evaluate the performance of HealthEdge in comparison with other methods. The results show that HealthEdge can optimally assign tasks between the network edge and cloud, which can reduce the task processing time, reduce bandwidth consumption and increase local edge workstation utilization.

In spite of many advantages of hybrid electrical/optical datacenter networks (Hybrid-DCN), current job schedulers for data-parallel frameworks are not suitable for Hybrid-DCN, since the schedulers do not aggregate data traffic to facilitate using optical circuit switch (OCS). We propose SchedOCS, a job scheduler for data-parallel frameworks in Hybrid-DCN that aims to take full advantage of the OCS to improve the job performance.

In this study, distributed energy resources scheduling problem of the set of smart homes (SHs) is investigated considering their cooperation with their neighbors and applying a stochastic model predictive control (MPC). Herein, every SH has a variety of sources and each SH is able to transact power with the local distribution company (DISCO) through the grid and with other connected SHs. The challenges of problem include modeling the technical and economic constraints of sources and dealing with the variability and uncertainties concerned with the power of photovoltaic (PV) panels that make the problem a mixed-integer nonlinear programming (MINLP), dynamic, and stochastic optimization problem. In order to handle the variability and uncertainties of problem, a stochastic MPC is applied. The numerical study demonstrates that cooperation of SHs in the energy scheduling problem has a high potential for minimizing operation cost of SHs.

As Electric Vehicles (EVs) become increasingly popular, their battery-related problems (e.g., short driving range and heavy battery weight) must be resolved as soon as possible. Velocity optimization of EVs to minimize energy consumption in driving is an effective alternative to handle these problems. However, previous velocity optimization methods assume that vehicles will pass through traffic lights immediately at green traffic signals. Actually, a vehicle may still experience a delay to pass a green traffic light due to a vehicle waiting queue in front of the traffic light. In this paper, for the first time, we propose a velocity optimization system which enables EVs to immediately pass green traffic lights without delay. We collected real driving data on a 4.0 km long road section of US-25 highway to conduct extensive trace-driven simulation studies. The experimental results from Matlab and Simulation for Urban MObility (SUMO) traffic simulator show that our velocity optimization system reduces energy consumption by up to 17.5% compared with real driving patterns without increasing trip time.

Electric vehicles (EVs) have great potential to reduce dependency on fossil fuels. The recent surge in the development of online EV (OLEV) will help to address the drawbacks associated with current generation EVs, such as the heavy and expensive batteries. OLEVs are integrated with the smart grid of power infrastructure through a wireless power transfer system (WPT) to increase the driving range of the OLEV. However, the integration of OLEVs with the grid creates a tremendous load for the smart grid. The demand of a power grid changes over time and the price of power is not fixed throughout the day. There should be some congestion avoidance and load balancing policy implications to ensure quality of services for OLEVs. In this paper, first, we conduct an analysis to show the existence of unpredictable power load and congestion because of OLEVs. We use the Simulation for Urban MObility tool and hourly traffic counts of a road section of the New York City to analyze the amount of energy OLEVs can receive at different times of the day. Then, we present a game theory based on a distributed power schedule framework to find the optimal schedule between OLEVs and smart grid. In the proposed framework, OLEVs receive the amount of power charging from the smart grid based on a power payment function which is updated using best response strategy. We prove that the updated power requests converge to the optimal power schedule. In this way, the smart grid maximizes the social welfare of OLEVs, which is defined as mixed consideration of total satisfaction and its power charging cost. Finally, we verify the performance of our proposed pricing policy under different scenarios in a simulation study.

In this paper, we study the link scheduling problem considering the fluctuating fading effect in transmissions. We extend the previous deterministic physical interference model to the Rayleigh-fading model that uses the stochastic propagation to address fading effects. Based on this model, we formulate a problem called Fading-Resistant Link Scheduling (Fading-R-LS) problem, which aims to maximize the throughput of all links in a single time slot. We prove that this problem is NP-hard. Based on the geometric structure of Fading-R-LS, we then propose two centralized schemes with O(g(L)) and O(1) performance guarantee, respectively, where g(L) is the number of magnitudes of transmission link lengths. Our experimental results show that the superior performance of our proposed schemes compared to previous schemes.

Previous resource provisioning strategies in cloud datacenters allocate physical resources to virtual machines (VMs) based on the predicted resource utilization pattern of VMs. The pattern for VMs of a job is usually derived from historical utilizations of multiple VMs of the job. We observed that these utilization curves are usually misaligned in time, which would lead to resource over-prediction and hence over-provisioning. Since this resource utilization misalignment problem has not been revealed and studied before, in this paper, we study the VM resource utilization from public datacenter traces to verify the existence of the utilization misalignments. Then, to reduce resource over-provisioning, we propose three VM resource utilization pattern refinement algorithms to improve the original generated pattern by lowering the cap of the pattern, reducing cap provision duration and varying the minimum value of the pattern. These algorithms can be used in any resource provisioning strategy that considers predicted resource utilizations of VMs of a job. We then adopt these refinement algorithms in an initial VM allocation mechanism and test them in trace-driven experiments and real-world cluster experiments. The experimental results show that each improved mechanism can increase resource efficiency up to 74%, and reduce the number of PMs needed to satisfy tenant requests up to 47% while conforming the SLO requirement.

The future generation of transportation system will be featured by electrified public transportation. To fulfill metropolitan transit demands, electric vehicles (EVs) must be continuously operable without recharging downtime. Wireless Power Transfer (WPT) techniques for in-motion EV charging is a solution. It however brings up a challenge: how to deploy charging lanes in a metropolitan road network to minimize the deployment cost while enabling EVs' continuous operability. In this paper, we propose CatCharger, which is the first work that handles this challenge. From a metropolitan-scale dataset collected from multiple sources of vehicles, we observe the diversity of vehicle passing speed and daily visit frequency (called traffic attributes) at intersections (i.e., landmarks), which are important factors for charging lane deployment. To select landmarks for deployment, we first group landmarks with similar traffic attribute values using the entropy minimization clustering method, and choose better candidate landmarks from each group suitable for deployment. To determine the deployment locations from the candidate landmarks, we infer the expected vehicle residual energy at each landmark using a Kernel Density Estimator fed by the vehicles' mobility, and formulate and solve an optimization problem to minimize the total deployment cost while ensuring a certain level of expected residual energy of EVs at each landmark. Our trace-driven experiments demonstrate the superior performance of CatCharger over other methods.

Nowadays, a large amount of services are deployed on the edge of the network from the cloud since processing data at the edge can reduce response time and lower bandwidth cost for applications such as healthcare in smart homes. Resource management is very important in the edge computing since it is able to increase the system efficiency and improve the quality of service. A common approach for resource management in edge computing is to assign tasks to the remote cloud or edge devices just according to several factors such as energy, bandwidth consumption, and latency. However, the approach is insufficiently efficient and falls short in meeting the requirements of handling health emergency when being applied in smart homes for healthcare. Possible health emergency needs immediate attention and different health tasks have different priorities to be processed. In this paper, we propose a task scheduling approach called HealthEdge that sets different processing priorities for different tasks based on the collected data on human health status and determines whether a task should run in a local device or a remote cloud in order to reduce its total processing time as much as possible. Based on a real trace from five patients, we conduct a trace-driven experiment to evaluate the performance of HealthEdge in comparison with other methods. The results show that HealthEdge can optimally assign tasks between the network edge and cloud, which can reduce the task processing time, reduce bandwidth consumption and increase local edge workstation utilization.

This paper aims to improve wireless network throughput by suppressing duplicate data transmissions from network links. It has been demonstrated that IP-layer Redundancy Elimination (RE) with content overhearing can significantly improve the goodput and utilization of wireless channels in wireless environment. However, the integration of IP-layer RE and wireless overhearing introduces a challenge. That is, probabilistic wireless overhearing and the possibility of a receiver overhearing from multiple transmitters cause the caches of a sender and a receiver far from synchronization, which can disrupt IP-layer RE's correctness and degrade its performance. The previous work deals with this challenge by the overhearing probability estimation, which however is not efficient or scalable. In this paper, we propose a Prediction-based Redundancy Elimination with Content Overhearing method (PRECO) to address this challenge. By exploiting prediction-based RE, PRECO does not require cache synchronization and overhearing probability estimation, which enables its efficient and scalable deployment. Based on PRECO, we exploit the benefits of deploying sub-packet level RE as a primitive IP-layer service on all nodes in wireless mesh networks by proposing a redundancy-aware routing protocol. Trace-driven performance evaluation shows the effectiveness and efficiency of PRECO compared with other RE methods.

In state-of-the-art Delay Tolerant Network (DTN) routing algorithms, two encountering nodes rely on routing utilities (e.g., probabilities of meeting other nodes) to decide the better carrier (defined forwarder) for their packets. As the utilities and forwarder information reflect user privacy, nodes may be reluctant to disclose such information, which however is indispensable in routing. To handle this challenge, we propose two distributed strategies to protect the aforementioned private information in utility-based DTN routing algorithms while still guarantying the correctness of packet forwarding, namely meeting Relationship Anonymity (ReHider) and Forwarder Anonymity (FwHider). ReHider anonymizes routing utilities between two encountered nodes, while FwHider additionally hides the forwarder information among the group of encountered nodes on top of the routing utility anonymity. We also present enhanced versions of the two strategies that can better prevent certain malicious behaviors such as probing attack and brute-force attack. The proposed strategies are distributed without the need of a central authority for authentication or key management. They can be applied to any utility-based DTN routing algorithms. Extensive analysis, trace-driven simulation, and smartphone based test demonstrate the effectiveness and energy efficiency of the proposed strategies.

Electric vehicles (EVs) will become a component of the future generation intelligent transportation system. Because of EVs' limited battery power, the wireless power transfer (WPT) system has drawn much attention in recent years. The WPT system charges EVs in motion when they pass the charging lanes installed in roads without requiring physical contact between utility power supply and vehicle battery. A charging lane has limited power that can be transferred to EVs on the charging lane. A challenge here is how to allocate the limited power to the EVs so that they have sufficient power to arrive at the next charging lane or their destinations (when there are no charging lanes ahead). In this paper, we study this power distribution scheduling problem.We provide solutions to handle this challenge and also achieve each of the following goals as much as possible: i) balancing the state of charge (SOC) of the EVs, ii) balancing the amount of stored power of the EVs, and iii) minimizing the total power charged. This paper is the first work that handles such a power distribution scheduling problem in WPT systems. Our extensive experiments on MatLab and Simulation for Urban MObility (SUMO) show the effectiveness of our scheduling solutions in achieving the different goals compared with other scheduling methods including first-come- first-serve and equal share.

In Web applications served by datacenter nowadays, the incast congestion at the front-end server seriously degrades the data request latency performance due to the vast data transmissions from a large number data servers for a data request in a short time. Previous incast congestion control methods usually consider the direct data transmissions from data servers to the front-end server, which makes it difficult to control the sending speed or adjust workloads due to the transient transmission of only a few data objects from each data server. In this paper, we propose a Swarm-based Incast Congestion Control (SICC) system. SICC forms all target data servers of one request in the same rack into a swarm. In each swarm, a data server (called hub) is selected to forward all data objects to the front-end server, so that the number of data servers concurrently connected to the front-end server is reduced, which avoids the incast congestion. Also, the continuous data transmission from hubs to the front-end server facilitates the development of other strategies to further control the incast congestion. To fully utilize the bandwidth, SICC uses a two-level data transmission speed control method to adjust the data transmission speeds of hubs. A query redirection method further reduces the request latency by balancing the transmission remaining times between hubs. Our experiments in simulation and on a real cluster demonstrate that SICC outperforms other incast control methods in improving throughput and reducing the data request latency.

Question and Answer (Q&A) systems play a vital role in our daily life for information and knowledge sharing. Users post questions and pick questions to answer in the system. Due to the rapidly growing user population and the number of questions, it is unlikely for a user to stumble upon a question by chance that (s)he can answer. Also, altruism does not encourage all users to provide answers, not to mention high quality answers with a short answer wait time. The primary objective of this paper is to improve the performance of Q&A systems by actively forwarding questions to users who are capable and willing to answer the questions. To this end, we have designed and implemented SocialQ&A, an online social network based Q&A system. SocialQ&A leverages the social network properties of common-interest and mutual-trust friend relationship to identify an asker through friendship who are most likely to answer the question, and enhance the user security. We also improve SocialQ&A with security and efficiency enhancements by protecting user privacy and identifies, and retrieving answers automatically for recurrent questions. We describe the architecture and algorithms, and conducted comprehensive large-scale simulation to evaluate SocialQ&A in comparison with other methods. Our results suggest that social networks can be leveraged to improve the answer quality and asker's waiting time. We also implemented a real prototype of SocialQ&A, and analyze the Q&A behavior of real users and questions from a small-scale real-world SocialQ&A system.

Discovering the densest subgraph is important in graph analysis, which has wide-ranging applications from social network community mining to the discovery of biological network modules. However, the previous algorithms neglect the connectivity of the dense subgraph since it is a challenge to give consideration to both subgraph structure and time efficiency. As a result, it may lead to isolated subgraphs in the output though they aim to find one connected dense subgraph. Also, there are lack of efficient algorithms for big natural graphs, especially considering datasets become increasingly larger in this era of Big Data. Furthermore, previous algorithms fail to take advantage of various features of natural graphs (e.g., power-law degree distribution, homophyly of vertices, and power-law community size) which can be applied to improve the efficiency and precision of the densest subgraph discovery. To handle these problems, in this paper, we study the densest subgraph problem by designing two different algorithms based on different features that natural graphs have. First, by analyzing the features of natural graphs, we design a heuristic algorithm for discovering the connected densest subgraph for massive undirected graphs in a MapReduce framework by taking advantage of the features of natural graphs. Second, we propose an exact algorithm for big data for the problem of discovering the densest subgraph. Experimental results show that our algorithms are more time-efficient and precise than other algorithms.

In mobile opportunistic social networks (MOSNs), mobile devices carried by people communicate with each other directly when they meet for proximity-based MOSN services (e.g., file sharing) without the support of infrastructures. In current methods, when nodes meet, they simply communicate with their real IDs, which leads to privacy and security concerns. Anonymizing real IDs among neighbor nodes solves such concerns. However, this prevents nodes from collecting real ID-based encountering information, which is needed to support MOSN services. Therefore, in this paper, we propose FaceChange that can support both anonymizing real IDs among neighbor nodes and collecting real ID-based encountering information. For node anonymity, two encountering nodes communicate anonymously. Only when the two nodes disconnect with each other, each node forwards an encrypted encountering evidence to the encountered node to enable encountering information collection. A set of novel schemes are designed to ensure the confidentiality and uniqueness of encountering evidences. FaceChange also supports fine-grained control over what information is shared with the encountered node based on attribute similarity (i.e., trust), which is calculated without disclosing attributes. Advanced extensions for sharing real IDs between mutually trusted nodes and more efficient encountering evidence collection are also proposed. Extensive analysis and experiments show the effectiveness of FaceChange on protecting node privacy and meanwhile supporting the encountering information collection in MOSNs. Implementation on smartphones also demonstrates its energy efficiency.

Many cloud service providers (CSPs) provide data storage services with datacenters distributed worldwide. These datacenters provide different get/put latencies and unit prices for resource utilization and reservation. Thus, when selecting different CSPs' datacenters, cloud customers of globally distributed applications (e.g., online social networks) face two challenges: 1) how to allocate data to worldwide datacenters to satisfy application service level objective (SLO) requirements, including both data retrieval latency and availability and2) how to allocate data and reserve resources in datacenters belonging to different CSPs to minimize the payment cost. To handle these challenges, we first model the cost minimization problem under SLO constraints using the integer programming. Due to its NP-hardness, we then introduce our heuristic solution, including a dominant-cost-based data allocation algorithm and an optimal resource reservation algorithm. We further propose three enhancement methods to reduce the payment cost and service latency: 1) coefficient-based data reallocation; 2) multicast-based data transferring; and 3) request redirection-based congestion control. We finally introduce an infrastructure to enable the conduction of the algorithms. Our trace-driven experiments on a supercomputing cluster and on real clouds (i.e., Amazon S3, Windows Azure Storage, and Google Cloud Storage) show the effectiveness of our algorithms for SLO guaranteed services and customer cost minimization.

To provide robust infrastructure as a service, clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. Previous reactive load balancing algorithms migrate VMs upon the occurrence of load imbalance, while previous proactive load balancing algorithms predict PM overload to conduct VM migration. However, both methods cannot maintain long-term load balance and produce high overhead and delay due to migration VM selection and destination PM selection. To overcome these problems, in this paper, we propose a proactive Markov Decision Process (MDP)-based load balancing algorithm. We handle the challenges of allying MDP in virtual resource management in cloud datacenters, which allows a PM to proactively find an optimal action to transit to a lightly loaded state that will maintain for a longer period of time. We also apply the MDP to determine destination PMs to achieve long-term PM load balance state. Our algorithm reduces the numbers of service level agreement (SLA) violations by long-term load balance maintenance, and also reduces the load balancing overhead (e.g., CPU time and energy) and delay by quickly identifying VMs and destination PMs to migrate. We further propose enhancement methods for higher performance. First, we propose a cloud profit oriented reward system in the MDP model so that when the MDP tries to maximize the rewards for load balance, it concurrently improves the actual profit of the datacenter. Second, we propose a new MDP model, which considers the actions for both migrating a VM out of a PM and migrating a VM into a PM, in order to reduce the overhead and improve the effectiveness of load balancing. Our trace-driven experiments show that our algorithm outperforms both previous reactive and proactive load balancing algorithms in terms of SLA violation, load balancing efficiency, and long-term load balance maintenance. Our experimental results also show the effectiveness of our proposed enhancement methods.

The pay-as-you-go service model impels cloud customers to reduce the usage cost of bandwidth. Traffic Redundancy Elimination (TRE) has been shown to be an effective solution for reducing bandwidth costs, and thus has recently captured significant attention in the cloud environment. By studying the TRE techniques in a trace driven approach, we found that both short-term (time span of seconds) and long-term (time span of hours or days) data redundancy can concurrently appear in the traffic, and solely using either sender-based TRE or receiver-based TRE cannot simultaneously capture both types of traffic redundancy. Also, the efficiency of existing receiver-based TRE solution is susceptible to the data changes compared to the historical data in the cache. In this paper, we propose a Cooperative end-to-end TRE solution (CoRE) that can detect and remove both short-term and long-term redundancy through a two-layer TRE design with cooperative operations between layers. An adaptive prediction algorithm is further proposed to improve TRE efficiency through dynamically adjusting the prediction window size based on the hit ratio of historical predictions. Besides, we enhance CoRE to adapt to different traffic redundancy characteristics of cloud applications to improve its operation cost. Extensive evaluation with several real traces show that CoRE is capable of effectively identifying both short-term and long-term redundancy with low additional cost while ensuring TRE efficiency from data changes.

MapReduce is a popular computing model for parallel data processing on large-scale datasets, which can vary from gigabytes to terabytes and petabytes. Though Hadoop MapReduce normally uses Hadoop Distributed File System (HDFS) local file system, it can be configured to use a remote file system. Then, an interesting question is raised: for a given application, which is the best running platform among the different combinations of scale-up and scale-out Hadoop with remote and local file systems. However, there has been no previous research on how different types of applications (e.g., CPU-intensive, data-intensive) with different characteristics (e.g., input data size) can benefit from the different platforms. Thus, in this paper, we conduct a comprehensive performance measurement of different applications on scale-up and scale-out clusters configured with HDFS and a remote file system (i.e., OFS), respectively. We identify and study how different job characteristics (e.g., input data size, the number of file reads/writes, and the amount of computations) affect the performance of different applications on the different platforms. Based on the measurement results, we also propose a performance prediction model to help users select the best platforms that lead to the minimum latency. Our evaluation using a Facebook workload trace demonstrates the effectiveness of our prediction model. This study is expected to provide a guidance for users to choose the best platform to run different applications with different characteristics in the environment that provides both remote and local storage, such as HPC cluster and cloud environment.

Scale-up machines perform better for jobs with small and median (KB, MB) data sizes, while scale-out machines perform better for jobs with large (GB, TB) data size. Since a workload usually consists of jobs with different data size levels, we propose building a hybrid Hadoop architecture that includes both scale-up and scale-out machines, which however is not trivial. The first challenge is workload data storage. Thousands of small data size jobs in a workload may overload the limited local disks of scale-up machines. Jobs from scale-up and scale-out machines may both request the same set of data, which leads to data transmission between the machines. The second challenge is to automatically schedule jobs to either scale-up or scale-out cluster to achieve the best performance. We conduct a thorough performance measurement of different applications on scale-up and scale-out clusters, configured with Hadoop Distributed File System (HDFS) and a remote file system (i.e., OFS), respectively. We find that using OFS rather than HDFS can solve the data storage challenge. Also, we identify the factors that determine the performance differences on the scale-up and scale-out clusters and their cross points to make the choice. Accordingly, we design and implement the hybrid scale-up/out Hadoop architecture. Our trace-driven experimental results show that our hybrid architecture outperforms both the traditional Hadoop architecture with HDFS and with OFS in terms of job completion time, throughput and job failure rate.

With the increasing popularity of Massively Multiplayer Online Game (MMOG) and fast growth of mobile gaming, cloud gaming exhibits great promises over the conventional MMOG gaming model as it frees players from the requirement of hardware and game installation on their local computers. However, as the graphics rendering is offloaded to the cloud, the data transmission between the end-users and the cloud significantly increases the response latency and limits the user coverage, thus preventing cloud gaming to achieve high user Quality of Service (QoS). To solve this problem, previous research suggested deploying more datacenters, but it comes at a prohibitive cost. We propose a lightweight system called CloudFog, which incorporates “fog” consisting of supernodes that are responsible for rendering game videos and streaming them to their nearby players. Fog enables the cloud to be only responsible for the intensive game state computation and sending update information to supernodes, which significantly reduce the traffic hence the latency and bandwidth consumption. To further enhance QoS, we propose the reputation based supernode selection strategy to assign each player with a suitable supernode that can provide satisfactory game video streaming service, the receiver-driven encoding rate adaptation strategy to increase the playback continuity, the social network based server assignment strategy to avoid the communication interaction between servers in a datacenter to reduce latency, and the dynamic supernode provisioning strategy to deal with user churns. Experimental results from PeerSim and PlanetLab show the effectiveness and efficiency of CloudFog and our individual strategies in increasing user coverage, reducing response latency and bandwidth consumption.

In data intensive clusters, a large amount of files are stored, processed and transferred simultaneously. To increase the data availability, some file systems create and store three replicas for each file in randomly selected servers across different racks. However, they neglect the file heterogeneity and server heterogeneity, which can be leveraged to further enhance data availability and file system efficiency. As files have heterogeneous popularities, a rigid number of three replicas may not provide immediate response to an excessive number of read requests to hot files, and waste resources (including energy) for replicas of cold files that have few read requests. Also, servers are heterogeneous in network bandwidth, hardware configuration and capacity (i.e., the maximal number of service requests that can be supported simultaneously), it is crucial to select replica servers to ensure low replication delay and request response delay. In this paper, we propose an Energy-Efficient Adaptive File Replication System (EAFR), which incorporates three components. It is adaptive to time-varying file popularities to achieve a good tradeoff between data availability and efficiency. Higher popularity of a file leads to more replicas and vice versa. Also, to achieve energy efficiency, servers are classified into hot servers and cold servers with different energy consumption, and cold files are stored in cold servers. EAFR then selects a server with sufficient capacity (including network bandwidth and capacity) to hold a replica. To further improve the performance of EAFR, we propose a dynamic transmission rate adjustment strategy to prevent potential incast congestion when replicating a file to a server, a networkaware data node selection strategy to reduce file read latency, and a load-aware replica maintenance strategy to quickly create file replicas under replica node failures. Experimental results on a real-world cluster show the effectiveness of EAFR and proposed strategies in reducing file read latency, replication time, and power consumption in large clusters.

Traditional web-based Question and Answer (Q&A) websites cannot easily solve non-factual questions to match askers' preference. Recent research efforts begin to study social-based Q&A systems that rely on an asker's social friends to provide answers. However, this method cannot find answerers for a question not belonging to the asker's interests. To solve this problem, we propose a distributed Q&A system incorporating both social community intelligence and global collective intelligence, named as iASK. iASK improves the response latency and answer quality in both the social domain and global domain. It uses a neural network based friend ranking method to identify answerer candidates by considering social closeness and Q&A activities. To efficiently identify answerers in the global user base, iASK builds a virtual server tree that embeds the hierarchical structure of interests, and also maps users to the tree based on user interests. To accurately locate the cooperative experts, iASK has a fine-grained reputation system to evaluate user reputation based on their cooperativeness and expertise, and uses a reputation based reward strategy to encourage users to be cooperative. To further improve the performance of iASK, we propose a weak tie assisted social based potential answerer location algorithm and an interest coefficient based uncategorized question forwarding algorithm. To further improve the response quality and cooperativeness, we propose a reputation based reward strategy that motivates users to answer questions from unknown users. Experimental results from large-scale trace-driven simulation and real-world daily usages of the iASK prototype show the superior performance of iASK. It achieves high answer quality with 24 percent higher accuracy, short response latency with 53 percent less delay and effective cooperative incentives with 16 percent more answers compared to other social-based Q&A systems. The results also show the effectiveness of the enhancement algorithms in improving the performance of iASK.

In large-scale computing clusters, when the server storing a task's requested data does not have sufficient computing capacity for the task, current job schedulers either schedule the task to the closest server and transmit to it the requested data, or let the task wait until the server has sufficient computing capacity. The former solution generates network load while the latter solution increases task delay. To handle this problem, load balancing methods are needed to reduce the number of overloaded servers due to computing workloads. However, current load balancing methods do not aim to balance the computing load for the long term. Through trace analysis, we demonstrate the diversity of computing workloads of different tasks and the necessity of balancing the computing workloads among servers. Then, we propose a cost-efficient Computing load Aware and Long-View load balancing approach (CALV). CALVis novel in that it achieves long-term computing load balance by migrating out an overloaded server data blocks contributing more computing workloads when the server is more overloaded and contribute less computing workloads when the server is more underloaded at different epochs during a time period. Based upon the task schedules, we further propose a task reassignment algorithm that reassigns tasks from an overloaded server to other data servers of the tasks to make it non-overloaded before CALVis conducted. The above methods are for the tasks whose submission times and execution latencies can be predicted. To handle unexpected tasks or insufficiently accurate predictions, we propose a dynamic load balancing method, in which an overloaded server dynamically redirects tasks to other data servers of the tasks, or replicates the tasks' requested data to other servers and redirects the tasks to those servers in order to become non-overloaded. Finally, we propose a proximity-aware tree based distributed load balancing method to reduce the reallocation cost and improve the scalability of CALV. Trace-driven experiments in simulation and a real computing cluster show that CALV outperforms other methods in terms of balancing the computing workloads and cost efficiency.

It is important for cloud service brokers to provide a multi-cloud storage service to minimize their payment cost to cloud service providers (CSPs) while providing service level objective (SLO) guarantee to their customers. Many multi-cloud storage services have been proposed or payment cost minimization or SLO guarantee. However, no previous works fully leverage the current cloud pricing policies (such as resource reservation pricing) to reduce the payment cost. Also, few works achieve both cost minimization and SLO guarantee. In this paper, we propose a multi-cloud Economical and SLO-guaranteed Storage Service (ES3), which determines data allocation and resource reservation schedules with payment cost minimization and SLO guarantee. ES3 incorporates (1) a coordinated data allocation and resource reservation method, which allocates each data item to a datacenter and determines the resource reservation amount on datacenters by leveraging all the pricing policies; (2) a genetic algorithm based data allocation adjustment method, which reduce data Get/Put rate variance in each datacenter to maximize the reservation benefit. We also propose several algorithms to enhance the cost efficient and SLO guarantee performance of ES3including i) dynamic request redirection, ii) grouped Gets for cost reduction, iii) lazy update for cost-efficient Puts, and iv) concurrent requests for rigid Get SLO guarantee. Our trace-driven experiments on a supercomputing cluster and on real clouds (i.e., Amazon S3, Windows Azure Storage and Google Cloud Storage) show the superior performance of ES3in payment cost minimization and SLO guarantee in comparison with previous methods.

Community question answering services (CQAS) (e.g., Yahoo! Answers) provides a platform where people post questions and answer questions posed by others. Previous works analyzed the answer quality (AQ) based on answer-related features, but neglect the question-related features on AQ. Previous work analyzed how asker- and question-related features affect the question quality (QQ) regarding the amount of attention from users, the number of answers and the question solving latency, but neglect the correlation between QQ and AQ (measured by the rating of the best answer), which is critical to quality of service (QoS). We handle this problem from two aspects. First, we additionally use QQ in measuring AQ, and analyze the correlation between a comprehensive list of features (including answer-related features) and QQ. Second, we propose the first method that estimates the probability for a given question to obtain high AQ. Our analysis on the Yahoo! Answers trace confirmed that the list of our identified features exert influence on AQ, which determines QQ. For the correlation analysis, the previous classification algorithms cannot consider the mutual interactions between multiple (>2) classes of features. We then propose a novel Coupled Semi-Supervised Mutual Reinforcement-based Label Propagation (CSMRLP) algorithm for this purpose. Our extensive experiments show that CSMRLP outperforms the Mutual Reinforcement-based Label Propagation (MRLP) and five other traditional classification algorithms in the accuracy of AQ classification, and the effectiveness of our proposed method in AQ prediction. Finally, we provide suggestions on how to create a question that will receive high AQ, which can be exploited to improve the QoS of CQAS.

Yahoo! Answers is an online platform where users can post questions and answer other users' questions. Our previous work studied the online social network (OSN) of Yahoo! Answers by analyzing information from the profiles (including fans, contacts, and interests) of top contributors and their related users. Rather than using the static profile information from the top-contributor-centered dataset, in this paper, we particularly analyze the actual questioning and answering (Q/A) behaviors of normal users. We build a Q/A network that unidirectionally connects each asker to his/her answerers. We analyze the structural characteristics of the Q/A network, user Q/A activities, and knowledge base of all users. In addition to the observations similar to our previous study, which indicates that the OSN of Yahoo! Answers can reflect user Q/A activities to a certain extent, we additionally observe that: 1) a large portion of users only ask questions without answering others' questions; 2) users are active in more knowledge categories than those indicated in their profiles; and 3) the knowledge categories of the top-contributor-related users cannot represent those of normal users. Finally, we analyze the characteristics of questions and answers in different knowledge categories. This paper not only provides an understanding of actual Q/A activities of users but also showcases the aspects of Q/A activities that the OSN of Yahoo! Answers can and cannot accurately reflect. Based on the insights gained from this paper, we propose a few methods to help improve the quality of service of Yahoo! Answers.

The cloud networks use switches to transfer inbound and outbound traffic through the data centers. Access of multiple tenants to the limited bandwidth capacity over the network switches increases the data traffic congestion in the network. The highly congested switches are vulnerable to get overloaded, and consequently slow down the flow of data traffic in the network. This paper proposes a nonlinear pricing policy for on-demand bandwidth allocation that jointly maximizes the total satisfaction of tenants and minimizes the congestion in the core switches. The optimal schedule is found through the best response strategy, in which each tenant updates its bandwidth allocation at each step based on the updated load-dependent predetermined nonlinear bandwidth pricing functions. The updated bandwidth allocations converge to the optimal bandwidth schedule that balances the load over the core switches. The performance of proposed pricing policy is evaluated under different scenarios.

MapReduce is a popular computing model for parallel data processing on large-scale datasets, which can vary from gigabytes to terabytes and petabytes. Though Hadoop MapReduce normally uses Hadoop Distributed File System (HDFS) local file system, it can be configured to use a remote file system. Then, an interesting question is raised: for a given application, which is the best running platform among the different combinations of scale-up and scale-out Hadoop with remote and local file systems. However, there has been no previous research on how different types of applications (e.g., CPU-intensive, data-intensive) with different characteristics (e.g., input data size) can benefit from the different platforms. Thus, in this paper, we conduct a comprehensive performance measurement of different applications on scale-up and scaleout clusters configured with HDFS and a remote file system (i.e., OFS), respectively. We identify and study how different job characteristics (e.g., input data size, the number of file reads/writes, and the amount of computations) affect the performance of different applications on the different platforms. This study is expected to provide a guidance for users to choose the best platform to run different applications with different characteristics in the environment that provides both remote and local storage, such as HPC cluster.

More and more organizations move their data and workload to commercial cloud storage systems. However, the multiplexing and sharing of the resources in a cloud storage system present unpredictable data access latency to tenants, which may make online data-intensive applications unable to satisfy their deadline requirements. Thus, it is important for cloud storage systems to provide deadline guaranteed services. In this paper, to meet a current form of service level objective (SLO) that constrains the percentage of each tenant's data access requests failing to meet its required deadline below a given threshold, we build a mathematical model to derive the upper bound of acceptable request arrival rate on each server. We then propose a Deadline Guaranteed storage service (called DGCloud) that incorporates three algorithms. Its deadline-aware load balancing scheme redirects requests and creates replicas to release the excess load of each server beyond the derived upper bound. Its workload consolidation algorithm tries to maximally reduce servers while still satisfying the SLO to maximize the resource utilization. Its data placement optimization algorithm re-schedules the data placement to minimize the transmission cost of data replication. Our trace-driven experiments in simulation and Amazon EC2 show the higher performance of DGCloud compared with previous methods in terms of deadline guarantees and system resource utilization, and the effectiveness of its individual algorithms.

Demand-prediction based resource provisioning schemes help assure service level objectives (SLO) in cloud systems. We notice that if a provisioning scheme does not exclude bursts from historical resource demands in normal demand prediction or always uses a large padding to correct under-prediction, it will lead to resource over-provisioning and low resource utilization. To improve the previous schemes, in this paper, we present a Resource-efficient Predictive Resource Provisioning system in clouds (RPRP) that excludes bursts in demand prediction and has algorithms to specifically handle bursts to avoid resource over-provisioning. Rather than setting padding to a possibly high value, RPRP has a load-dependent padding algorithm that adaptively determines padding based on predicted demands. To handle bursts, RPRP embodies a responsive padding algorithm that adaptively adjusts padding to recover from both under-provisioning and over-provisioning. We implemented RPRP on top of Xen and conducted both trace-driven simulation and real-world testbed experiments. The experimental results show that RPRP achieves higher resource utilization, more accurate demand predictions, and fewer SLO violations than previous schemes.

Many high-performance computing (HPC) sites extend their clusters to support Hadoop MapReduce for a variety of applications. However, HPC cluster differs from Hadoop cluster on the configurations of storage resources. In the Hadoop Distributed File System (HDFS), data resides on the compute nodes, while in the HPC cluster, data is stored on separate nodes dedicated to storage. Dedicated storage offloads I/O load from the compute nodes and provides more powerful storage. Local storage provides better locality and avoids contention for shared storage resources. To gain an insight of the two platforms, in this paper, we investigate the performance and resource utilization of different types (i.e., I/O-intensive, data-intensive and CPU-intensive) of applications on the HPC-based Hadoop platforms with local storage and dedicated storage. We find that the I/O-intensive and data-intensive applications with large input file size can benefit more from the dedicated storage, while these applications with small input file size can benefit more from the local storage. CPU-intensive applications with a large number of small-size input files benefit more from the local storage, while these applications with large-size input files benefit approximately equally from the two platforms. We verify our findings by trace-driven experiments on different types of jobs from the Facebook synthesized trace. This work provides guidance on choosing the best platform to optimize the performance of different types of applications and reduce system overhead.

Cloud storage system usually experiences data loss, hindering data durability. Three-way random replication is commonly used to prevent data loss in cloud storage systems. However, it cannot effectively handle correlated machine failures. Although Copyset Replication and Tiered Replication can reduce data loss in correlated and independent failures and enhance data durability, they fail to leverage different data popularities to substantially reduce the storage cost and bandwidth cost caused by replication. To address these issues, we present a popularity-aware multi-failure resilient and cost-effective replication (PM-CR) scheme for high data durability in cloud storage. PMCR splits the cloud storage system into primary tier and backup tier, and classifies data into hot data, warm data and cold data based on data popularities. To handle both correlated and independent failures, PMCR stores the three replicas of the same data into one Copyset formed by two servers in the primary tier and one server in the backup tier. For the third replicas of warm data and cold data in the backup tier, PMCR uses the Similar Compression method for read-intensive data and uses the Delta Compression method for write-intensive data to reduce storage cost and bandwidth cost. As a result, these costs are reduced and data durability and availability are enhanced without compromising data request delay greatly. Extensive experiment results based on trace parameters show that PMCR achieves high data durability, low probability of data loss, and low storage cost and bandwidth cost compared to previous replication schemes.

In cloud systems, efficient resource provisioning is needed to maximize the resource utilization while reducing the Service Level Objective (SLO) violation rate, which is important to cloud providers for high profit. Several methods have been proposed to provide efficient provisioning. However, the previous methods do not consider leveraging the complementary of jobs' requirements on different resource types and job size concurrently to increase the resource utilization. Also, by simply packing complementary jobs without considering job size in the job packing, it can decrease the resource utilization. Therefore, in this paper, we consider both jobs' demands on different resource types (in the spatial space) and jobs' execution time (in the temporal space); we pack the complementary jobs (whose demands on multiple resource types are complementary to each other) belonging to the same type and assign them to a Virtual Machine (VM) to increase the resource utilization. Moreover, the previous methods do not provide efficient resource allocation for heterogeneous jobs in current cloud systems and do not offer different SLO degrees for different job types to achieve higher resource utilization and lower SLO violation rate. Therefore, we propose a Customized Cooperative Resource Provisioning (CCRP) scheme for the heterogeneous jobs in clouds. CCRP uses the hybrid resource allocation and provides SLO availability customization for different job types. To test the performance of CCRP, we compared CCRP with existing methods under various scenarios. Extensive experimental results based on a real cluster and Amazon EC2 show that CCRP achieves 50% higher or more resource utilization and 50% lower or less SLO violation rate compared to the previous resource provisioning strategies.

In the cloud, the network links are shared among tenants, which makes them easy to get fully congested (overloaded). Overloaded links degrade the performance of tenants' applications, and impose additional costs to the cloud provider. In this paper, we propose a nonlinear bandwidth pricing policy for congestion control in the cloud network. In order to maximize social welfare (i.e., maximize the total satisfaction of the tenants while minimizing the congestion over the link), the cloud provider uses the nonlinear pricing policy that increases the unit price with increment of bandwidth usage. Each tenant competes for bandwidth allocation to maximize its utility (i.e., both maximize its own individual satisfaction and minimize its bandwidth payment cost). We design a game between tenants and the cloud provider, and show that there exists a unique optimal bandwidth schedule (Nash equilibrium) that jointly maximizes the social welfare and the utility of each tenant at the same time. In order to find the optimal schedule, we use an asynchronous-based best response strategy, in which each tenant updates its optimal bandwidth allocation based on the updated bandwidth payment function from the cloud provider. We prove that the updated bandwidth allocations converge to the optimal bandwidth schedule. In our simulation study and real implementation, we verify the performance of our proposed pricing mechanism under different scenarios.

Data analytics frameworks shift towards larger degrees of parallelism. Efficient scheduling of data-parallel jobs (tasks) is critical for improving job performance such as response time, and resource utilization. It is an important challenge for large scale data analytics frameworks in which jobs are more complex and have diverse characteristics (e.g., diverse resource requirements). Prior work on scheduling cannot achieve low response time and high resource utilization simultaneously because they cannot accurately estimate the durations of tasks in the queue of a worker machine by using sampling-based approach (including sampling with late binding) for task placement, and thus they fail to place tasks at the best possible worker machine. Also, they do not sufficiently consider the diverse resource requirements of jobs (tasks) for placing tasks on worker machines. To address this challenge, we propose a Dependency-aware and Resource-efficient Scheduling (DRS) to achieve low response time and high resource utilization. DRS takes into account task dependency and assigns tasks that are independent of each other to different worker machines. Also, DRS considers tasks' resource requirements and packs complementary tasks whose resource demands on multiple resources are complementary to each other to increase the resource utilization. In addition, DRS uses the mutual reinforcement learning to estimate the task's waiting time (the duration of tasks in the queue of a worker), and assigns tasks to workers with the consideration of tasks' waiting time to reduce the response time. Extensive experimental results based on a real cluster and experiments using real-world Amazon EC2 cloud service show that DRS achieves low response time and high resource utilization compared to previous strategies.

To minimize the fuel consumption for driving, several methods have been proposed to calculate vehicles' optimal velocity profiles on a remote cloud. Considering the traffic dynamism, each vehicle needs to keep updating the velocity profile, which requires low latency for information uploading and profile calculation. However, these proposed methods cannot satisfy this requirement due to (1) high queuing delay for information uploading caused by a large number of vehicles, and (2) the neglect of the traffic light and high computation delay for velocity profile. For (1), considering the driving features of close vehicles on a road, e.g., similar velocity and interdistances, we propose to group vehicles within a certain range and let the leader vehicle in each group to upload the group information to the cloud, which then derives the velocity of each vehicle in the group. For (2), we propose spatial-temporal DP (ST-DP) that additionally considers the traffic lights. We innovatively find that the DP process makes it well suited to run on Spark (a fast parallel cluster computing framework) and then present how to run ST-DP on Spark. Finally, we demonstrate the superiority of our method using both trace-driven simulation (NS-2.33 simulator and MATLAB) and real-world experiments.

The shared nature of cloud network infrastructures causes unpredictable network performance, which may degrade the performance of these applications. Recently, several works propose to explicitly reserve the network bandwidth in the cloud with virtual network abstraction models, which pre-specify the network bandwidth between virtual machines (VMs) for a tenant job. However, the pre-specification fails to exploit the elastic feature of the bandwidth resource (i.e., more reserved bandwidth within no-elongation threshold bandwidth leads to shorter job execution time and vice versa) in job scheduling. It is difficult for ordinary tenants (without specialized network knowledge) to estimate the exact needed bandwidth. In this paper, we propose a new cloud job scheduler, in which each tenant only needs to specify job deadline and each job's reserved bandwidth is elastically determined by leveraging the elastic feature to maximize the total job rewards, which represent the worth of successful completion by deadlines. Finally, the scheduler tries to reduce the execution time of each job. It also jointly considers the computational capacity of VMs and reserved VM bandwidth in job scheduling. Using trace-driven and real cluster experiments, we show the efficiency and effectiveness of our job scheduler in comparison with other scheduling strategies.

In cloud systems, achieving high resource utilization and low Service Level Objective (SLO) violation rate are important to the cloud provider for high profit. For this purpose, recently, some methods have been proposed to predict allocated but unused resources and reallocate them to long-running service jobs. However, the accuracy of their prediction method relies on the existence of patterns in jobs' resource utilization. Therefore, these methods cannot be used for short-lived jobs, which usually do not have certain patterns but exhibit frequent fluctuations in resource requirements. Also, these methods may result in resource fragmentation and lead to low resource utilization because they neglect job resource intensity in multi-resource allocation and may allocate much more resources to jobs. To handle this problem, we propose a Cooperative Opportunistic Resource Provisioning scheme (CORP) for short-lived jobs. CORP uses the deep learning method to predict the amount of temporarily-unused resource of each short-lived job. It also predicts the fluctuations of the amount of unused resource using Hidden Markov Model, and adjusts the predicted amount for the peak and valley of unused resource, and dynamically allocates the corrected amount of resource to jobs. Further, CORP uses a job packing strategy by leveraging complementary jobs' requirements on different resource types and allocates such jobs to the same VM to fully utilize unused resources, which increases resource utilization. Extensive experimental results based on a real cluster and Amazon EC2 show that CORP achieves high resource utilization and low SLO violation rate compared to previous resource provisioning schemes.

Maximizing data locality in task scheduling is critical for the performance of MapReduce job execution. Manyexisting works on MapReduce scheduling decide the placementof map and reduce tasks on a coarse granularity of locationsmeasured by located machines and racks. They do not explicitlyconsider the network topology and data transmission cost, whichmay cause task straggling and degrade the job performance. Inorder to improve MapReduce job performance, in this paper, we consider the task placement with the goal of minimizing theoverall data transmission cost for a job execution while balancingthe transmission cost reduction and resource utilization. Wepropose a probabilistic network-aware scheduling algorithm thatselects a task (map task or reduce task) to be scheduled on a givenavailable task slot that leads to the minimum transmission costamong the task candidates, and then schedule the selected taskon the slot with a probability determined by its transmission cost, a lower expected transmission cost leads to a higher probabilityand vice versa. We also propose a method to more accuratelyestimate the intermediate data size based on the progress ofmap tasks, which is needed to calculate the transmission cost ofreduce tasks but is unknown at the time of reduce task scheduling. We implement our probabilistic network-aware schedulingalgorithm on Apache Hadoop and conduct experiments on ahigh-performance computing platform. The experimental resultsshow that our scheduling algorithm outperforms the previousapproaches in terms of job completion time and cluster resource utilization.

Replication is a common approach to enhance data availability in cloud storage systems. Previously proposed replication schemes cannot effectively handle both correlated and non-correlated machine failures while increasing the data availability with the limited resource. The schemes for correlated machine failures must create a constant number of replicas for each data object, which neglects diverse data popularities and cannot utilize the resource to maximize the expected data availability. Also, the previous schemes neglect the consistency maintenance cost and the storage cost caused by replication. It is critical for cloud providers to maximize data availability (hence minimize SLA violations) while minimizing cost caused by replication in order to maximize the revenue. In this paper, we build a nonlinear integer programming model to maximize data availability in both types of failures and minimize the cost caused by replication. Based on the model's solution for the replication degree of each data object, we propose a low-cost multi-failure resilient replication scheme (MRR). MRR can effectively handle both correlated and non-correlated machine failures, considers data popularities to enhance data availability, and also tries to minimize consistency maintenance cost and storage cost. Extensive numerical results from trace parameters and experiments from real-world Amazon S3 show that MRR achieves high data availability, low data loss probability and low consistency maintenance cost and storage cost compared to previous replication schemes.

In this paper, we introduce RoadAware, an infrastructure-less system that leverages a smartphone to collect road information on people's daily routes to and from work. Unlike previous research that utilizes moving vehicles equipped with sensors to collect road information, RoadAware does not need the support from infrastructure, road-side facilities or other users and focuses on serving individuals during his/her daily commute to and back from work. RoadAware can provide road information including 1) travel distance and time between traffic lights; 2) duration of red and green signals of each traffic light; and 3) traffic volume on the road between traffic lights. RoadAware builds a model that can deduce the road information based on the wait time and the length of the waiting queue in front of the car when it stops at a traffic light. The road information not only benefits individuals in their daily commutes (e.g., predict waiting at red lights and remind whether there is a delay on daily route) but can also be collected for traffic optimization that otherwise requires costly sensor deployment. We developed RoadAware on Windows smartphones, and our extensive real-world test shows that RoadAware can provide road information with acceptable accuracy.

We consider learning the underlying graph structure of a network in which infection spreads based on the observations of node infection times. We give an algorithm based on minimal hitting set to learn the exact underlying graph structure and provide sufficient condition on number of cascades required (i.e. sample complexity) for reliable recovery, which is shown to be O(logn), where n is the number of nodes in the graph. We then analytically evaluate performance of minimal hitting set approach in learning the degree distribution and detecting leaf nodes of a graph and provide a sufficient condition for its sample complexity which is shown to be lower than that of learning the whole graph. We also generalize the exact graph estimation problem to the problem of estimating the graph within a certain distortion, measured by edit distance. We show that this edit distance based graph estimator has a lower sample complexity. Our experimental results based on both synthetic network topologies and a real-world network trace show that our algorithm achieves superior performance than a previously proposed algorithm based on maximum likelihood.

Mobile crowdsensing serves as a critical building block for the emerging Internet of Things (IoT) applications. However, the sensing devices continuously generate a large amount of data, which consumes much resources (e.g., bandwidth, energy and storage), and may sacrifice the quality-of-service (QoS) of applications. Prior work has demonstrated that there is significant redundancy in the content of the sensed data. By judiciously reducing the redundant data, the data size and the load can be significantly reduced, thereby reducing resource cost, facilitating the timely delivery of unique, probably critical information and enhancing QoS. This paper presents a survey of existing works for the mobile crowdsensing strategies with emphasis on reducing the resource cost and achieving high QoS. We start by introducing the motivation for this survey, and present the necessary background of crowdsensing and IoT. We then present various mobile crowdsensing strategies and discuss their strengths and limitations. Finally, we discuss the future research directions for mobile crowdsensing. The survey addresses a broad range of techniques, methods, models, systems and applications related to mobile crowdsensing and IoT. Our goal is not only to analyze and compare the strategies proposed in the prior works but also to discuss their applicability towards the IoT, and provide the guidance on the future research direction of mobile crowdsensing.

Though intensive research efforts have been devoted to the study of the link scheduling problem in wireless networks, no previous work has discussed this problem for cooperative communication networks, in which receivers are allowed to combine messages from different senders to combat transmission errors. In this paper, we study the link scheduling problem in wireless cooperative communication networks, in which receivers are allowed to combine copies of a message to combat fading. We formulate two problems named cooperative link scheduling problem (CLS) and one-shot cooperative link scheduling problem (OCLS). The first problem aims to find a schedule of links that uses the minimum number of time slots to inform all the receivers. The second problem aims to find a set of links that can inform the maximum number of receivers in one time slot. As a solution, we propose an algorithm for both CLS and OCLS with g(K) approximation ratio, where g(K) is so called diversity of key links. In addition, we propose a greedy algorithm with O(1) approximation ratio for OCLS when the number of links for each receiver is upper bounded by a constant. Simulation results indicate that our cooperative link scheduling approaches outperform non-cooperative ones.

A smart home (SH) can have a variety of sources such as diesel generator (DG), photovoltaic (PV) panels installed on the roof of the SH, and plug-in electric vehicle (PEV) as an energy storage. In addition, a SH is able to transact power with the local distribution company (DISCO) through the grid. This study intends to investigate the energy scheduling problem of a SH so that the daily electricity consumption cost of the SH is minimized. Herein, the challenges include modeling the economic and technical constraints of the DG and the battery of the PEV, and also dealing with the variability and uncertainties concerned with the power of the PV panels that make the problem a nonlinear, dynamic (time-varying), and stochastic optimization problem. In order to handle the variability and uncertainties of the power of the PV panels, a stochastic model predictive control (MPC) is applied. Herein, a combination of genetic algorithm (GA) and linear programming (GA-LP) approach is applied as the optimization tool. The numerical study demonstrates the competence of the proposed approach for decreasing the operation cost.

Many Cloud Service Providers (CSPs) provide data storage services with datacenters distributed worldwide. These datacenters provide different Get/Put latencies and unit prices for resource utilization and reservation. Thus, when selecting different CSPs' datacenters, cloud customers of globally distributed applications (e.g., online social networks) face two challenges: (i) how to allocate data to worldwide datacenters to satisfy application SLO (service level objective) requirements including both data retrieval latency and availability, and (ii) how to allocate data and reserve resources in datacenters belonging to different CSPs to minimize the payment cost. To handle these challenges, we first model the cost minimization problem under SLO constraints using integer programming. Due to its NP-hardness, we then introduce our heuristic solution, including a dominant-cost based data allocation algorithm and an optimal resource reservation algorithm. We finally introduce an infrastructure to enable the conduction of the algorithms. Our trace-driven experiments on a supercomputing cluster and on real clouds (i.e., Amazon S3, Windows Azure Storage and Google Cloud Storage) show the effectiveness of our algorithms for SLO guaranteed services and customer cost minimization.

Nodes in Delay Tolerant Networks (DTN) rely on routing utilities (e.g., probabilities of meeting nodes) to decide the packet forwarder. As the utilities reflect user privacy, nodes may be reluctant to disclose such information directly. Therefore, we propose a distributed strategy to protect the aforementioned private information in utility-based DTN routing algorithms while still guarantying the correctness of packet forwarding, namely meeting Relationship Anonymity (ReHider). We also present an enhanced version that can better prevent certain malicious behaviors (probing attack and brute-force attack). Initial analysis show the effectiveness of the proposed strategy.

In cloud datacenters, multiple Virtual Machines (VMs) are co-located in a Physical Machine (PM) to serve different applications. Prior VM consolidation methods for cloud datacenters schedule VMs mainly based on resource (CPU and memory) constraints in PMs but neglect serious shared Last Level cache contention between VMs. This may cause severe VM performance degradation due to cache thrashing and starvation for VMs. Current cache contention aware VM consolidation strategies either estimate cache contention by coarse VM classification for each individual VM without considering co-location and (or) require the aid of hardware to monitor the online miss rate of each VM. Therefore, these strategies are insufficiently accurate and (or) difficult to adopt for VM consolidation in clouds. In this paper, we formalize the problem of cache contention aware VM placement and migration in cloud datacenters using integer linear programming. We then propose a cache contention aware VM placement and migration algorithm (CacheVM). It estimates the total cache contention degree of co-locating a given VM with a group of VMs in a PM based on the cache stack distance profiles of the VMs. Then, it places the VM to the PM with the minimum cache contention degree and chooses the VM from a PM that generates the maximum cache contention degree to migrate out. We implemented CacheVM and its comparison methods on a supercomputing cluster. Trace-driven simulation and real-testbed experiments show that CacheVM outperforms other methods in terms of the number of cache misses, execution time and throughput.

The future generation transportation system will be featured by electrified public transportation. To fulfill metropolitan transit demands, electric vehicles (EVs) must be continuously operable without recharging downtime. Wireless Power Transfer (WPT) techniques for in-motion EV charging is a solution [1], [2]. It however brings up a challenge: how to deploy charging lanes in a metropolitan road network to minimize the deployment cost while enabling EVs' continuous operability.

The explosive growth of personal mobile devices (e.g., smartphones and pads) has brought about significant potential distributed sensing resources. However, such resources have not been fully utilized due to two problems: i) mobile device mobility usually is not dedicated to data sensing, and ii) users may not be willing to participate in the data sensing proactively, i.e., move to or wait in a specific area. To address these problems, we propose a sensing system, namely MobiSensing, with a low intervention to device owners. It uses the semi-Markov process to model node mobility for future mobility prediction. While moving around, mobile devices connect to the central task assignment server opportunistically through their owners' daily usage. In each connection, the server predicts the connected device's next connection and its mobility between current and the next connection. Then, the server assigns sensing tasks in this period of time that the node is likely to complete to the node. As a result, no proactive operations or movements are required for device owners, and sensing tasks can be completed passively and efficiently. Trace-driven experiments demonstrate the high successful rate of MobiSensing.

As an alternate form in the road transportationsystem, electric vehicle (EV) can help reduce the fossil-fuelconsumption. However, the usage of EVs is constrained by thelimited capacity of battery. Wireless Power Transfer (WPT) can increase the driving range of EVs by charging EVs inmotionwhen they drive through a wireless charging laneembedded in a road. The amount of power that can be suppliedby a charging lane at a time is limited. A problem here iswhen a large number of EVs pass a charging lane, how toefficiently distribute the power among different penetrationslevels of EVs? However, there has been no previous researchdevoted to tackling this challenge. To handle this challenge, wepropose a system to balance the State of Charge (called BSoC) among the EVs. It consists of three components: i) fog-basedpower distribution architecture, ii) power scheduling model, and iii) efficient vehicle-to-fog communication protocol. The fogcomputing center collects information from EVs and schedulesthe power distribution. We use fog closer to vehicles ratherthan cloud in order to reduce the communication latency. Thepower scheduling model schedules the power allocated to eachEV. In order to avoid network congestion between EVs and thefog, we let vehicles choose their own communication channelto communicate with local controllers. Finally, we evaluateour system using extensive simulation studies in NetworkSimulator-3, MatLab, and Simulation for Urban MObilitytools, and the experimental results confirm the efficiency ofour system.

It is important for cloud service brokers to provide a multi-cloud storage service to minimize their payment cost to cloud service providers (CSPs) while providing service level objective (SLO) guarantee to their customers. Many multi-cloud storage services have been proposed or payment cost minimization or SLO guarantee. However, no previous works fully leverage the current cloud pricing policies (such as resource reservation pricing) to reduce the payment cost. Also, few works achieve both cost minimization and SLO guarantee. In this paper, we propose a multi-cloud Economical and SLO-guaranteed Storage Service (ES3), which determines data allocation and resource reservation schedules with payment cost minimization and SLO guarantee. ES3 incorporates (1) a coordinated data allocation and resource reservation method, which allocates each data item to a datacenter and determines the resource reservation amount on datacenters by leveraging all the pricing policies; (2) a genetic algorithm based data allocation adjustment method, which reduce data Get/Put rate variance in each datacenter to maximize the reservation benefit; and (3) a dynamic request redirection method, which dynamically redirects a data request from a reservation-overutilized datacenter to a reservation-underutilized datacenter to further reduce the payment. Our trace-driven experiments on a supercomputing cluster and on real clouds (i.e., Amazon S3, Windows Azure Storage and Google Cloud Storage) show the superior performance of ES3 in payment cost minimization and SLO guarantee in comparison with previous methods.

Functioning as an intermediary between cloud tenants and providers, cloud service brokerages (CSBs) bring about great benefits to the cloud market. To maximize its own profit, a CSB is faced with a challenge: how to reserve servers and distribute tenant demands to the reserved servers such that the total reservation cost is minimized while the reserved servers can satisfy the tenant service level agreement (SLA)? Demand prediction and demand allocation are two steps to solve this problem. However, previous demand prediction methods cannot accurately predict tenant demands since they cannot accurately estimate prediction errors and also assume the existence of seasonal periods of demands. Previous demand allocation methods only aim to minimize the number of reserved servers rather than the server reservation cost, which is more challenging. To solve this challenge, we propose a Probabilistic Demand Allocation system (PDA). It predicts demands and more accurate prediction errors without the assumption of the existence of seasonal periods. It then formulates a nonlinear programming problem and has a decentralized method to find the problem solution. In addition to overcoming the shortcomings in previous methods, PDA is novel in that rather than separately conducting the prediction and demand allocation, it considers prediction errors in demand allocation in order to allocate demands with offsetting prediction errors (e.g., -1 and +1) to the same server, which helps find the problem solution. Both simulation and real-world experimental results demonstrate the superior performance of our system in reducing servers' reservation cost.

Always keeping a certain distance between vehicles in a platoon is important for collision avoidance. Centralized platoon systems let the leader vehicle determine and notify the velocities of all the vehicles in the platoon. Unfortunately, such a centralized method generates high packet drop rate and communication delay due to the leader vehicle's limited communication capability. Therefore, we propose a decentralized platoon network, in which each vehicle determines its own velocity by only communicating with the vehicles in a short range. However, the multiple simultaneous transmissions between different pairs of vehicles may interfere with each other. Directly applying current channel allocation methods for interference avoidance leads to high communication cost and delay in vehicle joins and departures (i.e., vehicle dynamics). As a result, a challenge is how to reduce the communication delay and cost for channel allocation in decentralized platoon networks? To handle this challenge, by leveraging a typical feature of a platoon, we devise a channel allocation algorithm, called the Fast and Lightweight Autonomous channel selection algorithm (FLA), in which each vehicle determines its own channel simply based on its distance to the leader vehicle. We conduct experiments on NS-3 and Matlab to evaluate the performance of our proposed methods. The experimental results demonstrate the superior performance of our decentralized platoon network over the previous centralized platoon networks and of FLA over previous channel allocation methods in platoons.

Traffic congestion control is pivotal for intelligent transportation systems. Previous works optimize vehicle speed for different objectives such as minimizing fuel consumption and minimizing travel time. However, they overlook the possible congestion generation in the future (e.g., in 5mins), which may degrade the performance of achieving the objectives. In this paper, we propose a vehicle Trajectory based driving speed OPtimization strategy (TOP) to minimize vehicle travel time and meanwhile avoid generating congestion. Its basic idea is to adjust vehicles'mobility to alleviate road congestion globally. TOP has a framework for collecting vehicles' information to a central server, which calculates the parameters depicting the future road condition (e.g., driving time, vehicle density, and probability of accident). The server then formulates a non-cooperative Stackelberg game considering these parameters, in which when each vehicle aims to minimize its travel time, the road congestion is also proactively avoided. After the Stackelberg equilibrium is reached, the optimal driving speed for each vehicle and the expected vehicle density that maximizes the utilization of the road network are determined. Our real trace analysis confirms some characteristics of vehicle mobility to support the design of TOP. Extensive trace-driven experiments show the effectiveness and superior performance of TOP in comparison with other driving speed optimization methods.

Though a cooperative broadcast scheme has been proposed for fading environments, it has two defects: First, it only handles a packet flow from a single source node in the network, but does not consider the scenario of multiple packet flows simultaneously broadcasted from different source nodes. Second, it only allows a single relay node to forward a packet in each time slot, though multiple relay nodes forwarding in a time slot can significantly reduce broadcast latency. In this paper, we aim achieve low-latency multi-flow broadcast in wireless multi-hop networks with fading channels. To describe the interference among the transmission in different flows, we incorporate the Rayleigh fading model to the signal to noise ratio (SNR) model. Then, we introduce a cooperative diversity scheme which allows multiple relays forwarding in a time slot to reduce broadcast latency. We then formulate an interesting problem: In a fading environment, what is the optimal relay allocation schedule to minimize the broadcast latency? We propose a warm up heuristic algorithm for single-flow cooperative broadcast, based on which, we further propose a heuristic algorithm for multi-flow cooperative broadcast. Simulation results demonstrate that the two algorithms achieve lower broadcast latency than a previous method.

Cooperative adaptive cruise control (CACC) systems have the potential to increase traffic throughput by allowing smaller headway between vehicles and moving vehicles safely in a platoon at a harmonized speed. CACC systems have been attracting significant attention from both academia and industry since connectivity between vehicles will become mandatory for new vehicles in the USA in the near future. In this paper, we review three basic and important aspects of CACC systems: communications, driver characteristics, and controls to identify the most challenging issues for their real-world deployment. Different routing protocols that support the data communication requirements between vehicles in the CACC platoon are reviewed. Promising and suitable protocols are identified. Driver characteristics related issues, such as how to keep drivers engaged in driving tasks during CACC operations, are discussed. To achieve mass acceptance, the control design needs to depict real-world traffic variability such as communication effects, driver behavior, and traffic composition. Thus, this paper also discusses the issues that existing CACC control modules face when considering close to ideal driving conditions.

In delay tolerant networks (DTNs), enabling a mobile node to search and find another interested mobile node is an important function in many applications. However, the movement of nodes in DTNs makes the problem formidable. Current node searching methods in disconnected networks mainly rely on fixed stations in the network and infrastructure-based communication to collect node position information, which is difficult to implement in DTNs. In this paper, we present DSearching, a distributed mobile node searching scheme for DTNs that requires no infrastructure. In DSearching, the entire DTN area is split into sub-areas, and each node summarizes its mobility information as both transient sub-area visiting record and long-term movement pattern. Upon arriving at a sub-area, a node generates a new visiting record for the sub-area and distributes it to nodes that are likely to stay in the previous sub-area, so that visiting records form a chain for the locators to trace the node. Each node also stores different parts of its long-term mobility pattern to long-staying nodes in different sub-areas for others to trace it when visiting records are absent. Considering that nodes in DTNs usually have limited resources, DSearching constrains the communication and storage cost in the information distribution while enabling efficient node searching. Advanced extensions that can further improve the searching efficiency is also proposed in this paper. Extensive trace-driven experiments with real traces demonstrate the high efficiency and high effectiveness of DSearching.

Wireless sensor and actuator networks (WSANs) are composed of sensors and actuators to perform distributed sensing and actuating tasks. Most WSAN applications (e.g., fire detection) demand that actuators rapidly respond to observed events. Therefore, real-time (i.e., fast) and fault-tolerant transmission is a critical requirement in WSANs to enable sensed data to reach actuators reliably and quickly. Due to limited power resources, energy-efficiency is another crucial requirement. Such requirements become formidably challenging in large-scale WSANs. However, existing WSANs fall short in meeting these requirements. To this end, we first theoretically study the Kautz graph for its applicability in WSANs to meet these requirements. We then propose a Kautz-based REal-time, Fault-tolerant and EneRgy-efficient WSAN (REFER). REFER embeds Kautz graphs into the physical topology of a WSAN for real-time communication and connects the Kautz graphs using distributed hash table (DHT) for high scalability. We also theoretically study routing paths in the Kautz graph, based on which we develop an efficient fault-tolerant routing protocol. It enables a relay node to quickly and efficiently identify the next shortest path from itself to the destination based only on node IDs upon routing failure, rather than relying on retransmission from the source. REFER is advantageous over previous Kautz graph based works in that it does not need an energy-consuming protocol to find the next shortest path and it preserves the consistency between the overlay and physical topology. We further improve routing in REFER by multi-path based routing and energy-efficient multicasting within and between Kautz graph cells, respectively. Extensive experimental results demonstrate the superior performance of REFER in comparison with existing WSAN systems in terms of real-time communication, energy-efficiency, fault-tolerance and scalability.

Node searching in delay tolerant networks is of great importance for different applications, in which a locator node finds a target node in person. In the previous distributed node searching method, a locator traces the target along its movement path from its most frequently visited location. For this purpose, nodes leave traces during their movements and also store their long-term movement patterns in their frequently visited locations (i.e., preferred locations). However, such tracing leads to a long delay and high overhead on the locator by long-distance moving. Our trace data study confirms these problems and provides the foundation of our design of a new node searching method, called target-oriented method (TSearch). By leveraging social network properties, TSearch aims to enable a locator to directly move toward the target. Nodes create encounter records (ERs) indicating the locations and times of their encounters and make the ERs easily accessible by locators through message exchanges or a hierarchical structure. In node searching, a locator follows the target's latest ER, the latest ERs of its friends (i.e., frequently meeting nodes), its preferred locations, and the target's possible locations deduced from additional information for node searching. Extensive trace-driven and real-world experiments show that TSearch achieves significantly higher success rate and lower delay in node searching compared with previous methods.

Distributed file searching in delay tolerant networks formed by mobile devices can potentially support various useful applications. In such networks, nodes often present certain social network properties of their holders in terms of contents (i.e., interests) and contacts. However, current methods in DTNs only consider either content or contact for file searching or dissemination, which limits the file sharing efficiency. In this paper, we first analyze real traces to confirm the importance and necessity of considering both content and contact in file search. We then propose Cont2, a social-aware file search method that exploits both node contents and contact patterns. First, considering people with common interests tend to share files and gather together, Cont2virtually groups common-interest nodes into a community to direct file search. Second, considering human mobility follows a certain pattern, Cont2exploits nodes' contact frequencies with a community to expedite file searching. To further improve the searching efficiency, Cont2also integrates sub-communities and parallel forwarding as optional components for file searching. Trace-driven experiments on the GENI testbed and NS-2 simulator show that Cont2can effectively improve the search efficiency compared to current methods.

Though the new OSN model, which deploys datacenters globally, helps reduce service latency, it causes higher inter-datacenter communication load. In Facebook, each datacenter has a full copy of all data, and the master datacenter updates all other datacenters, generating tremendous load in this new model. Distributed data storage, which only stores a user's data to his/her geographically closest datacenters mitigates the problem. However, frequent interactions between distant users lead to frequent inter-datacenter communication and hence long service latencies. In this paper, we aim to reduce inter-datacenter communications while still achieving low service latency. We first verify the benefits of the new model and present OSN typical properties that underlie the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters ($SD^3$). Since replicas need inter-datacenter data updates, datacenters in$SD^3$jointly consider update rates and visit rates to select user data for replication; furthermore,$SD^3$atomizes users’ different types of data (e.g., status update, friend post, music) for replication, ensuring that a replica always reduces inter-datacenter communication.$SD^3$also incorporates three strategies to further enhance its performance: locality-aware multicast update tree, replica deactivation, and datacenter congestion control. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency and effectiveness of$SD^3$in comparison to other replication methods and the effectiveness of its three schemes.

Content Delivery Networks (CDNs) play a central role in today's Internet infrastructure and have seen a sharp increase in scale. More and more internet sites are armed with live contents, such as live sports game statistics, e-commerce, and online auctions, and they rely on CDNs to deliver such contents freshly at scale. However, the problem of maintaining consistency for live (dynamic) contents while achieving high scalability is non-trivial in CDNs. The large number of widely scattered replicas guarantees the QoS of end-users while substantially increasing the complexity of consistency maintenance under frequent updates. Current consistency maintenance infrastructures and methods cannot simultaneously satisfy both scalability and consistency. In this paper, we first analyze our crawled trace data of cached sports game content on thousands of content servers of a major CDN. We analyze the content consistency from different perspectives, from which we break down the reasons for inconsistency among content servers. We verify that the CDN uses unicast instead of multicast trees as the update infrastructure, which may not scale effectively. Then, we further evaluate the performance in consistency, scalability and overhead for different infrastructures with different update methods. We itemize the advantages and disadvantages of different methods and infrastructures in different scenarios through the evaluation. Based on this evaluation, we propose our hybrid and self-adaptive update method to reduce network load and improve scalability under the conditions recorded in the trace and prove its effectiveness through trace-driven experiments. We aim to give guidance for appropriate selections of consistency maintenance infrastructures and methods for a CDN, and for choosing a CDN service with different considerations.

It is imperative for cloud storage systems to be able to provide deadline guaranteed services according to service level agreements (SLAs) for online services. In spite of many previous works on deadline aware solutions, most of them focus on scheduling work flows or resource reservation in datacenter networks but neglect the server overload problem in cloud storage systems that prevents providing the deadline guaranteed services. In this paper, we introduce a new form of SLAs, which enables each tenant to specify a percentage of its requests it wishes to serve within a specified deadline. We first identify the multiple objectives (i.e., traffic and latency minimization, resource utilization maximization) in developing schemes to satisfy the SLAs. To satisfy the SLAs while achieving the multi-objectives, we propose a Parallel Deadline Guaranteed (PDG) scheme, which schedules data reallocation (through load re-assignment and data replication) using a tree-based bottom-up parallel process. The observation from our model also motivates our deadline strictness clustered data allocation algorithm that maps tenants with the similar SLA strictness into the same server to enhance SLA guarantees. We further enhance PDG in supplying SLA guaranteed services through two algorithms: i) a prioritized data reallocation algorithm that deals with request arrival rate variation, and ii) an adaptive request retransmission algorithm that deals with SLA requirement variation. Our trace-driven experiments on a simulator and Amazon EC2 show the effectiveness of our schemes for guaranteeing the SLAs while achieving the multi-objectives.

For predictable application performance or fairness in network sharing in clouds, many bandwidth allocation policies have been proposed. However, with these policies, tenants are not incentivized to use idle bandwidth or prevent link congestion, and may even take advantage of the policies to gain unfair bandwidth allocation. Increasing network utilization while avoiding congestion not only benefits cloud provider but also the tenants by improving application performance. In this paper, we propose a new pricing model that sets different unit prices for reserved bandwidth, the bandwidth on congested links and on uncongested links, and makes the unit price for congested links proportional to their congestion degrees. We use game theory model to analyze tenants' behaviors in our model and the current pricing models, which shows the effectiveness of our model in providing the incentives. With the pricing model, we propose a network sharing policy to achieve both min-guarantee and proportionality, while prevent tenants from earning unfair bandwidth. We further propose methods for each virtual machine to arrange its traffic to reduce its unsatisfied demand and maximize its utility, while increase network utilization. As a result, our solution creates a win-win situation, where tenants strive to increase their benefits in bandwidth sharing, which also concurrently increases the utilities of cloud provider and other tenants. Our simulation and trace-driven experimental results show the effectiveness of our solutions in creating the win-win situation.

In many countries, the Internet is under stringent censorship for political or religious reasons which severely undermines the free flow of information. A censorship-resistant web browsing system must be scalable, blocking resistant, and tracing resistant. However, current censorship-resistant web browsing systems, which use a group of dedicated proxies to bypass censorship, fail to meet these requirements. To tackle these challenges, we propose Freeweb, which relies on widely-distributed peer-to-peer (P2P) nodes in a decentralized manner rather than specified proxies in a centralized manner. We also proposed enhancement methods to reduce file access delay and avoid node overloads in Freeweb. Freeweb is built on top of a Distributed Hash Table (DHT)-based P2P network, where nodes not under censorship help nodes under censorship to access blocked webpages. Freeweb has a web browser front-end whose user interface resembles existing web browsers. The underlying complex process of retrieving blocked webpages is therefore hidden from users. We implemented an open-sourced Freeweb and conducted extensive real-world experiments on PlanetLab. The experimental results show that Freeweb has a high success rate and reasonable browsing latency, and its enhancement reduces much network load and file access latency, and avoids node overloads.

Real-world applications, such as peer-to-peer (P2P) networks, e-commerce and social networks, usually employ reputation systems to provide guidance in selecting trustworthy node for high system reliability and security. A reputation system computes and publishes reputation score for each node based on a collection of opinions from others about the node. However, collusion behaviors impair the effectiveness of reputation systems in trustworthy node selection. Though many reputation calculation methods have been proposed to mitigate collusion's influence, little effort has been devoted to specifically tackling collusion. Based on the important collusion behavior characteristics in reputation evaluation and influence on reputation values, we propose a basic collusion detection method to specifically detect suspicious collusion behaviors in pairs. We further optimize the method by reducing the computing overhead. We also propose two pre-processing methods to firstly identify partial reputation raters of a node that are more likely to be colluders before applying the collusion detection method on them, thus reducing the collusion detection overhead. Extensive experimental results show that our proposed methods can significantly enhance the capability of existing reputation systems to detect collusion with low overhead. Also, the pre-processing methods are effective in reducing the collusion detection overhead without affecting the collusion detection accuracy.

In large-scale computing clusters, when the server storing a task's input data does not have sufficient computing capacity, current job schedulers either schedule the task and transmit the input data to the closest server or let the task wait until the server has sufficient computing capacity, which generates network load or task delay. To handle this problem, load balancing methods are needed to reduce the number of overloaded servers due to computing workloads. However, current load balancing methods either do not consider the computing workload or assume that it is proportional to the number of data blocks in a server. Through trace analysis, we demonstrate the diversity of computing workloads of different tasks and the necessity of balancing the computing workloads among servers. Then, we propose a cost-efficient Computing load Aware and Long-View load balancing approach (CALV). In addition to the computing load awareness, CALV is also novel in that it achieves long-term load balance by migrating out data blocks from an overloaded server that contribute more computing workloads when the server is more overloaded and contribute less computing workloads when the server is more underloaded at different epochs during a time period. CALV also has a lazy data block transmission method to improve the load balanced state and avoid network load peak. Trace-driven experiments in simulation and a real computing cluster show that CALV outperforms other methods in terms of balancing the computing workloads and cost efficiency.

Functioning as an intermediary between tenants and cloud providers, cloud service brokerages (CSBs) can bring about great benefits to the cloud market. CSBs buy the cloud resources, i.e., servers, with lower prices from cloud providers and sell the resources to the tenants with higher prices. To maximize its own profit, a CSB may distribute tenants' requests to the clouds that waste energy resources. However, as energy costs of cloud computing have been increasing rapidly, there is a need for cloud providers to optimize energy efficiency while maintain high service level performance to tenants, not only for their own benefit but also for social welfares (e.g., protecting environment). Thus, for green cloud companies, two questions have arisen: 1) under what pricing policies from the cloud providers to the CSB, a profit-driven CSB is willing to minimize the total cloud energy cost while satisfy tenant demands and 2) how should a CSB distribute tenants' demands to achieve this objective? To address question 1), we find a pricing policy for cloud providers such that maximizing CSB's profit is equivalent to minimizing cloud providers' energy cost. To address question 2), we first devise a greedy solution, and then propose an approximation algorithm with a constant approximation ratio. Both simulation and real-world Amazon EC2 experimental results demonstrate the effectiveness of our pricing policy to incentivize CSBs to save energy for cloud providers and the superior performance of our algorithms in energy efficiency and resource utilizations in comparison with the previous algorithms.

In data intensive clusters, a large amount of files are stored, processed and transferred simultaneously. To increase the data availability, some file systems create and store three replicas for each file in randomly selected servers across different racks. However, they neglect the file heterogeneity and server heterogeneity, which can be leveraged to further enhance data availability and file system efficiency (in terms of replication delay and request response delay). As files have heterogeneous popularities, a rigid number of three replicas may not provide immediate response to an excessive number of read requests to hot files, and waste resources (including energy) for replicas of cold files that have few read requests. Also, servers are heterogeneous in network bandwidth, hardware configuration and capacity (i.e., the maximal number of service requests that can be supported simultaneously), it is crucial to select replica servers to ensure low replication delay and request response delay. In this paper, we propose an Energy-Efficient Adaptive File Replication System (EAFR), which incorporates three components. It is adaptive to time-varying file popularities to achieve a good tradeoff between data availability and efficiency. Higher popularity of a file leads to more replicas and vice versa. Also, to achieve energy efficiency, servers are classified into hot servers and cold servers with different energy consumption, and cold files are stored in cold servers. Further, EAFR selects a server with sufficient capacity (including network bandwidth and capacity) to hold a replica. Experimental results on a real-world cluster show the effectiveness of EAFR in reducing file read latency, replication time, and power consumption in large clusters.

Question and Answer (Q&A) systems play a vital role in our daily life for information and knowledge sharing. Users post questions and pick questions to answer in the system. Due to the rapidly growing user population and the number of questions, it is unlikely for a user to stumble upon a question by chance that (s)he can answer. Also, altruism does not encourage all users to provide answers, not to mention high quality answers with a short answer wait time. The primary objective of this paper is to improve the performance of Q&A systems by actively forwarding questions to users who are capable and willing to answer the questions. To this end, we have designed and implemented SocialQ&A, an online social network based Q&A system. SocialQ&A leverages the social network properties of common-interest and mutual-trust friend relationship to identify an asker's friends who are most likely to answer the question. We describe the architecture, algorithms and user interface of SocialQ&A, and analyze the Q&A behavior of real users and questions from a small-scale real-world SocialQ&A system. We also conducted comprehensive large-scale simulation to evaluate SocialQ&A in comparison with other methods. Our results suggest that social networks can be leveraged to improve the answer quality and asker's waiting time.

Discovering the densest subgraph is important in graph analysis, which has wide-ranging applications from social network community mining to the discovery of biological network modules. However, the previous algorithms neglect the connectivity of the dense subgraph since it is a challenge to give consideration to both subgraph structure and time efficiency. As a result, it may lead to isolated subgraphs in the output though they aim to find one connected dense subgraph. Also, there are lacking of efficient algorithms for big natural graphs, especially considering datasets become increasingly larger in this era of Big Data. Furthermore, previous algorithms fail to take advantage of various features of natural graphs (e.g., power-law degree distribution, homophyly of vertices, and power-law community size) which can be applied to improve the efficiency and precision of the densest subgraph discovery. To handle these problems, we design a heuristic algorithm for discovering the connected densest subgraph for massive undirected graphs in a MapReduce framework by taking advantage of the features of natural graphs. Experimental results show that our algorithm is capable of discovering the connected densest subgraph. Also, it can reduce the running times by 62\% for average and discover denser subgraphs in 50\% of the real datasets and 88\% of the simulated datasets comparing with previous algorithm in a MapReduce framework.

In Vehicle Delay Tolerant Networks (VDTNs), current routing algorithms select relay vehicles based on either vehicle encounter history or predicted future locations. The former method may fail to find relays that can encounter the target vehicle in a large-scale VDTN while the latter method may not provide accurate location prediction due to traffic variance. Therefore, these methods cannot achieve high performance in terms of routing success rate and delay. In this paper, we aim to improve the routing performance in VDTNs. We first analyze vehicle network traces and observe that i) each vehicle has only a few active sub-areas that it frequently visits, and ii) two frequently encountered vehicles usually encounter each other in their active sub-areas. We then propose Active Area based Routing method (AAR) which consists of two steps based on the two observations correspondingly. AAR first distributes a packet copy to each active sub-area of the target vehicle using a traffic-considered shortest path spreading algorithm, and then in each sub-area, each packet carrier tries to forward the packet to a vehicle that has high encounter frequency with the target vehicle. Extensive trace-driven simulation demonstrates that AAR produces higher success rates and shorter delay in comparison with the state-of-the-art routing algorithms in VDTNs.

With the increasing popularity of Massively Multiplayer Online Game (MMOG) and fast growth of mobile gaming, cloud gaming exhibits great promises over the conventional MMOG gaming model as it frees players from the requirement of hardware and game installation on their local computers. However, as the graphics rendering is offloaded to the cloud, the data transmission between the end-users and the cloud significantly increases the response latency and limits the user coverage, thus preventing cloud gaming to achieve high user Quality of Experience (QoE). To solve this problem, previous research suggested deploying more data centers, but it comes at a prohibitive cost. We propose a lightweight system called Cloud Fog, which incorporates "fog" consisting of super nodes that are responsible for rendering game videos and streaming them to their nearby players. Fog enables the cloud to be only responsible for the intensive game state computation and sending update information to super nodes, which significantly reduce the traffic hence the latency and bandwidth consumption. Experimental results from PeerSim and Planet Lab show the effectiveness and efficiency of Cloud Fog in increasing user coverage, reducing response latency and bandwidth consumption.

As video streaming applications are deployed on the cloud, cloud providers are charged by ISPs for inter-data enter transfers under the dominant percentile-based charging models. In order to minimize the payment costs, existing works aim to keep the traffic on each link under the charging volume (i.e., 95th percentile traffic volume from the beginning of a charging period up to current time). However, these methods cannot fully utilize each link's available bandwidth capacity, and may increase the charging volumes. To further reduce the bandwidth payment cost by fully utilizing link bandwidth, we propose an economical and deadline-driven video flow scheduling system, called EcoFlow. Considering different video flows have different transmission deadlines, EcoFlow transmits videos in the order of their deadline tightness and postpones the deliveries of later-deadline videos to later time slots so that the charging volume at current time interval will not increase. The flows that are expected to miss their deadlines are divided into sub flows to be rerouted to other underutilized links in order to meet their deadlines without increasing charging volumes. Experimental results on Planet Lab and EC2 show that compared to existing methods, EcoFlow achieves the least bandwidth costs for cloud providers.

It is critical for a cloud service broker to minimize its payment cost to cloud service providers for serving the broker's customers while providing SLA-guaranteed services. In this paper, we propose an Economical and SLA-guaranteed cloud Storage Service (ES3), which finds a data allocation and resource reservation schedule with cost minimization. ES3 incorporates (1) a data allocation and reservation algorithm, which allocates each data item to a datacenter and determines the reservation amount on data centers by leveraging all the pricing policies, (2) a genetic algorithm based data allocation adjustment approach, which makes data Get/Put rates stable in each datacenter to maximize the reservation benefit. Our trace-driven experiments show the superior performance of ES3.

In mobile opportunistic social networks (MOSNs), mobile devices carried by people communicate with each other directly when they meet for proximity-based MOSN services (e.g., file sharing) without the support of infrastructures. In current methods, when nodes meet, they simply communicate with their real IDs, which leads to privacy and security concerns. Anonymizing real IDs among encountering neighbor nodes solves such concerns. However, this prevents nodes from collecting real ID based encountering information, which is needed to support MOSN services. Therefore, in this paper, we propose FaceChange that can support both anonymizing real IDs among neighbor nodes and collecting real ID based encountering information. To realize neighbor node anonymity, two encountering nodes communicate anonymously. Then, when the two nodes disconnect, each node forwards an encrypted encountering evidence to the encountered node to enable encountering information collection. A set of novel schemes are designed to protect the confidentiality and uniqueness of encountering evidences. FaceChange also supports fine-grained control over what encountering information should be forwarded based on attribute similarity (i.e., trust) without disclosing attributes. Extensive analysis and experiments show the effectiveness of FaceChange on protecting node privacy and meanwhile supporting the encountering information collection in MOSNs. Real implementation on smartphones also demonstrates its energy efficiency.

Virtualization is a key technology for cloud data centers to implement infrastructure as a service (IaaS) and to provide flexible and cost-effective resource sharing. It introduces an additional layer of abstraction that produces resource utilization overhead. Disregarding this overhead may cause serious reduction of the monitoring accuracy of the cloud providers and may cause degradation of the VM performance. However, there is no previous work that comprehensively investigates the virtualization overhead. In this paper, we comprehensively measure and study the relationship between the resource utilizations of virtual machines (VMs) and the resource utilizations of the device driver domain, hypervisor and the physical machine (PM) with diverse workloads and scenarios in the Xen virtualization environment. We examine data from the real-world virtualized deployment to characterize VM workloads and assess their impact on the resource utilizations in the system. We show that the impact of virtualization overhead depends on the workloads, and that virtualization overhead is an important factor to consider in cloud resource provisioning. Based on the measurements, we build a regression model to estimate the resource utilization overhead of the PM resulting from providing virtualized resource to the VMs and from managing multiple VMs. Finally, our trace-driven real-world experimental results show the high accuracy of our model in predicting PM resource consumptions in the cloud datacenter, and the importance of considering the virtualization overhead in cloud resource provisioning.

Since scale-up machines perform better for jobs with small and median (KB, MB) data sizes while scale-out machines perform better for jobs with large (GB, TB) data size, and a workload usually consists of jobs with different data size levels, we propose building a hybrid Hadoop architecture that includes both scale-up and scale-out machines, which however is not trivial. The first challenge is workload data storage. Thousands of small data size jobs in a workload may overload the limited local disks of scale-up machines. Jobs from scale-up and scale-out machines may both request the same set of data, which leads to data transmission between the machines. The second challenge is to automatically schedule jobs to either scale-up or scale-out cluster to achieve the best performance. We conduct a thorough performance measurement of different applications on scale-up and scale-out clusters, configured with Hadoop Distributed File System (HDFS) and a remote file system (i.e., OFS), respectively. We find that using OFS rather than HDFS can solve the data storage challenge. Also, we identify the factors that determine the performance differences on the scale-up and scale-out clusters and their cross points to make the choice. Accordingly, we design and implement the hybrid scale-up/out Hadoop architecture. Our trace-driven experimental results show that our hybrid architecture outperforms both the traditional Hadoop architecture with HDFS and with OFS in terms of job completion time.

With the increasing popularity of Massively Multiplayer Online Game (MMOG) and fast growth of mobile gaming, cloud gaming exhibits great promises over the conventional MMOG gaming model as it frees players from the requirement of hardware and game installation on their local computers. However, as the graphics rendering is offloaded to the cloud, the data transmission between the end-users and the cloud significantly increases the response latency and limits the user coverage, thus preventing cloud gaming to achieve high user Quality of Experience (QoE). To solve this problem, previous research suggested deploying more data centers, but it comes at a prohibitive cost. We propose a lightweight system called Cloud Fog, which incorporates "fog" consisting of super nodes that are responsible for rendering game videos and streaming them to their nearby players. Fog enables the cloud to be only responsible for the intensive game state computation and sending update information to super nodes, which significantly reduce the traffic hence the latency and bandwidth consumption. To further enhance QoE, we propose the receiver-driven encoding rate adaptation strategy to increase the playback continuity and the deadline-driven sender buffer scheduling strategy to ensure that the segments arrive at the players within their response latency. Experimental results from Peer Sim and Planet Lab show the effectiveness and efficiency of Cloud Fog and our individual strategies in increasing user coverage, reducing response latency and bandwidth consumption.

In this paper, we study the minimum energy broadcast problem in time-varying graphs (TVGs), which are a very useful high level abstraction for studying highly dynamic wireless networks. To this end, we first incorporate a channel model, called energy-demand functions, to the current TVGs, namely time-varying energy-demand graphs (TVEGs). Based on this model, we formulate the problem: given a TVEG, what is the optimal schedule (i.e., Which nodes should forward a packet in what times and at what power levels) to minimize the energy consumption of the broadcast? We prove the problem to be NP-hard and o(log N) in approximable. It is a challenge to find a solution for this problem on continuous time. Fortunately, we prove that the problem on continuous time is equivalent to the problem on certain discrete time points, called discrete time set (DTS). Based on this property, we propose polynomial time solutions for this problem with different channel models, and evaluate the performance of these methods from real-life contact traces.

Node searching in delay tolerant networks (DTNs) is of great importance for different applications, in which a locator node finds a target node in person. In the previous distributed node searching method, a locator traces the target along its movement path from its most frequently visited location. For this purpose, nodes leave traces during their movements and also store their long-term movement patterns in their frequently visited locations (i.e., preferred locations). However, such tracing leads to a long delay and high overhead on the locator by longdistance moving. Our trace data study confirms these problems and provides foundation of our design of a new node searching method, called target-oriented method (TSearch). By leveraging social network properties, TSearch aims to enable a locator to directly move towards the target. Nodes create encounter records (ERs) indicating the locations and times of their encounters and make the ERs easily accessible by locators through message exchanges or a hierarchical structure. In node searching, a locator follows the target's latest ER, the latest ERs of its friends (i.e., frequently meeting nodes), and its preferred locations in order. Extensive trace-driven and real-world experiments show that TSearch achieves significantly higher success rate and lower delay in node searching compared with previous methods.

Delay Tolerant Networks (DTNs) have attracted significant interests due to the adaptability in areas without infrastructures. In such scenarios, moving data from one place (landmark) to another place (landmark) is essential for data communication between different areas. However, current DTN routing algorithms either fail to fully utilize node mobility or have additional requirements that cannot be satisfied easily (i.e., Require base stations or the global traffic distribution). Therefore, in this paper, we propose a distributed greedy routing algorithm, namely Greedy Flow, for efficient packet routing between landmarks. Greedy Flow builds a local traffic map and a global landmark map on each node. The local traffic map indicates the node's knowledge about the amount of traffic (node transition) between landmarks in the area where it primarily visits. It is constructed by collecting encountered nodes' transit frequencies between these landmarks. The global landmark map shows the distribution of landmarks in the system and is built offline. In packet routing, the global landmark map shows the general packet forwarding direction, while the local traffic map helps determine the next-hop landmark on the fastest path in the forwarding direction. As a result, packets are greedily forwarded toward their destination landmarks. Extensive real trace driven experiments demonstrate the high efficiency of Greedy Flow.

In this paper, we study the link scheduling problem in wireless cooperative communication networks, in which receivers are allowed to combine copies of a message from different senders to combat fading. We formulate a problem called cooperative link scheduling problem (CLS), which aims to find a schedule of links that uses the minimum number of time slots to inform all the receivers. As a solution, we propose an algorithm for CLS with g(K) approximation ratio, where g(K) is so called diversity of key links. Simulation results indicate that our cooperative link scheduling approaches outperform noncooperative ones.

Present approaches to achieve k-coverage for Wireless Sensor Networks still rely on centralized techniques. In this paper, we devise a distributed method for this problem, namely Distributed VOronoi based Cooperation scheme (DVOC), where nodes cooperate in hole detection and recovery. In previous Voronoi based schemes, each node only monitors its own critical points. Such methods are inefficient for k-coverage because the critical points are far away from their generating nodes in k-order Voronoi diagram, causing high cost for transmission and computing. As a solution, DVOC enables nodes to monitor others' critical points around themselves by building local Voronoi diagrams (LVDs). Further, DVOC constrains the movement of every node to avoid generating new holes. If a node cannot reach its destination due to the constraint, its hole healing responsibility will fall to other cooperating nodes. The experimental results from the real world testbed demonstrate that DVOC outperforms the previous schemes.

Cloud providers are charged by ISPs for inter-datacenter transfers under the dominant percentile-based charging models. In order to minimize the payment costs, existing works aim to keep the traffic on each link under the charging volume. However, these methods cannot fully utilize each link's available bandwidth capacity, and may increase the charging volumes. To further reduce the bandwidth payment cost by fully utilizing link bandwidth, we propose an economical and deadline-driven video flow scheduling system, called EcoFlow. Considering different video flows have different transmission deadlines, EcoFlow transmits videos in the order of their deadline tightness and postpones the deliveries of later-deadline videos to later time slots so that the charging volume at current time interval will not increase. The flows that are expected to miss their deadlines are divided into subflows to be rerouted to other under-utilized links in order to meet their deadlines without increasing charging volumes. We also design the implementation of EcoFlow in a distributed manner. Experimental results on EC2 show that compared to existing methods, EcoFlow achieves the least bandwidth costs for cloud providers.

In this paper, we propose a time-efficient and exact algorithm for the problem of discovering the densest subgraph in big data. Current algorithms for solving this problem have three problems: i) they cannot handle the dilemma between the efficiency of handing big data and the precision of the discovered densest subgraph; ii) they cannot take advantage of both the parallel computing on MapReduce and in-memory computing on one computer; iii) their applicability to different kinds of graphs has not been discussed. Our proposed algorithm combines the MapReduce parallel computing with in-memory computing on one computer together to improve the efficiency and precision of discovering the densest subgraphs. The algorithm consists of two computational phases: i) the graph reduction in the MapReduce framework; ii) the densest subgraph discovery in memory. Further, we theoretically analyze the correctness of this algorithm and its applicability in different natural graphs. We conduct extensive experimental evaluations in a MapReduce framework on both massive real-world graphs and simulated graphs to test our algorithm in comparison with other algorithms. Experimental results show that our algorithm is more time-efficient and precise than other algorithms.

Current reputation systems for peer-to-peer (P2P) file sharing systems either fail to utilize existing trust within social networks or suffer from certain attacks (e.g., free-riding and collusion). To handle these problems, we introduce a trust management system, called SocialLink, that utilizes social network and historical transaction links. SocialLink manages file transactions through both the social network and a novel weighted transaction network, which is built based on previous file transaction history. First, SocialLink exploits the trust among friends in social networks by enabling two friends to share files directly. Second, the weighted transaction network is utilized to 1) deduce the trust of the client on a server in reliably providing the requested file and 2) check the fairness of the transaction. In this way, SocialLink prevents potential misbehaving transactions (i.e., providing faulty files), encourages nodes to contribute file resources to non-friends, and avoids free-riding. Furthermore, the weighted transaction network helps SocialLink resist whitewashing, collusion and Sybil attacks. Extensive simulation demonstrates that SocialLink can efficiently ensure trustable and fair P2P file sharing and resist the aforementioned attacks.

Hybrid peer-to-peer assisted cloud-based video-on-demand (VoD) systems augment cloud-based VoD systems with P2P networks to improve scalability and save bandwidth costs in the cloud. In these systems, the VoD service provider (e.g., NetFlix) relies on the cloud to deliver videos to users and pays for the cloud bandwidth consumption. The users can download videos from both the cloud and peers in the P2P network. It is important for VoD service provider to (i) minimize the cloud bandwidth consumption, and (ii) guarantee users' satisfaction (i.e., quality-of-experience). Though previous adaptive bitrate streaming (ABR) methods improve video playback smoothness, they cannot achieve these two goals simultaneously. To tackle this challenge, we propose AutoTune, a game-based adaptive bitrate streaming method. In AutoTune, we formulate the bitrate adaptation problem in ABR as a noncooperative Stackelberg game, where VoD service provider and the users are players. The VoD service provider acts as a leader and it decides the VoD service price for users with the objective of minimizing cloud bandwidth consumption while ensuring users' participation. In response to the VoD service price, the users select video bitrates that lead to maximum utility (defined as a function of its satisfaction minus associated VoD service fee). Finally, the Stackelberg equilibrium is reached in which the cloud bandwidth consumption is minimized while users are satisfied with selected video bitrates. Experimental results from the PeerSim simulator and the PlanetLab real-world testbed show that compared to existing methods, AutoTune can provide high user satisfaction and save cloud bandwidth consumption.

It is imperative for cloud storage systems to be able to provide deadline guaranteed services according to service level agreements (SLAs) for online services. However, there are no previous works that provide the deadline guarantee specifically for cloud storage systems. In this paper, we introduce a new form of SLAs, which enables each tenant to specify a percentage of its requests it wishes to serve within a specified deadline. We first identify the multiple objectives (i.e., traffic and latency minimization, resource utilization maximization) in developing schemes to satisfy the SLAs. To satisfy the SLAs while achieving the multi-objectives, we propose a Parallel Deadline Guaranteed (PDG) scheme, which schedules data reallocation (through load re-assignment and data replication) using a tree-based bottom-up parallel process. We also enhance PDG through a prioritized data reallocation algorithm, which enables highly overloaded servers to autonomously probe nearby servers and enables the load balancer to instantly handle them. Our trace-driven experiments on a simulator show the effectiveness of our schemes for guaranteeing the SLAs while achieving the multi-objectives.

Traditional Web-based Question and Answer (Q&A) Web sites cannot easily solve non-factual questions to match askers' preference. Recent research efforts begin to study social-based Q&A systems that rely on an asker's social friends to provide answers. However, this method cannot find answerers for a question not belonging to the asker's interests. To solve this problem, we propose a distributed Q&A system incorporating both social community intelligence and global collective intelligence, named as iASK. iASK improves the response latency and answer quality in both the social domain and global domain. It uses a neural network based friend ranking method to identify answerer candidates by considering social closeness and Q&A activities. To efficiently identify answerers in the global user base, iASK builds a virtual server tree that embeds the hierarchical structure of interests, and also maps users to the tree based on user interests. To accurately locate the cooperative experts, iASK has a fine-grained reputation system to evaluate user reputation based on their cooperativeness and expertise. Experimental results from large-scale trace-driven simulation and realworld daily usages of the iASK prototype show the superior performance of iASK. It achieves high answer quality with 24% higher accuracy, short response latency with 53% less delay and effective cooperative incentives with 16% more answers compared to other social-based Q&A systems.

Data collection by wireless sensor networks is a fundamental and critical function for cyber-physical systems (CPS) to estimate the state of the physical world. However, unstable network conditions impose great challenges in guaranteeing data accuracy, which is essential for reliable state estimation of the physical phenomena. For underlying sensor networks, without efficiently resolving congestion in data transmission, packet loss at congested nodes can considerably increase the estimation error. However, previous congestion control schemes relying on reducing transmitted data samples also increase the estimation error. Thus, we propose a Congestion-Adaptive Data Collection scheme (CADC) to efficiently resolve the network congestion while guaranteeing the overall data estimation accuracy. CADC mitigates congestion by adaptive lossy compression with guarantee that a given overall data estimation error bound is satisfied. Besides, since a CPS application may have different priorities for different data items, we further propose a weighted CADC scheme such that the data with higher priority has less distortion. Extensive experimental results demonstrate the effectiveness and efficiency of our CADC schemes.

As a popular routing protocol in wireless sensor networks (WSNs), greedy routing has received great attention. The previous works characterize its data deliverability in WSNs by the probability of all nodes successfully sending their data to the base station. Their analysis, however, neither provides the information of the quantitative relation between successful data delivery ratio and transmission power of sensor nodes nor considers the impact of the network congestion or link collision on the data deliverability. To address these problems, in this paper, we characterize the data deliverability of greedy routing by the ratio of successful data transmissions from sensors to the base station. We introduce η-guaranteed delivery which means that the ratio of successful data deliveries is not less than η, and study the relationship between the transmission power of sensors and the probability of achieving η-guaranteed delivery. Furthermore, with considering the effect of network congestion and link collision, we provide a more precise and full characterization for the deliverability of greedy routing. Extensive simulation and real-world experimental results show the correctness and tightness of the upper bound of the smallest transmission power for achieving η-guaranteed delivery.

Motivated by the growing popularity of mobile devices and their increasing capacities, file sharing in disruption tolerant networks (DTNs) has attracted significant attention recently. Since nodes are sparsely distributed in separated areas and are intermittently disconnected in DTNs, it is difficult to achieve high data availability in file sharing. Many previous methods enhance file availability in DTNs through file replication. However, there has been no file replication method that tries to reduce data loss in correlated node failures, which however are common in wireless networks. In this paper, we propose a distributed file replication method (called MobileCopy) in DTNs, which aims to achieve low probability of totally losing a file at the expense of having a high number of impacted files in an individual large-scale correlated node failure. MobileCopy is designed for community-based file sharing systems. It has two main components: i) data loss resistant and popularity aware file replication, and ii) distributed hash table (DHT)-based file replica indexing. MobileCopy considers file popularity to determine the number of replicas of a file in each community. Through limiting the possible combination of candidate replica holders, MobileCopy greatly reduces the probability of node failures that will lead to data loss, i.e., losing all replicas of a file. Moreover, MobileCopy enables nodes to efficiently store and fetch the placement information of file replicas for efficient file searching. Extensive trace-driven experiments show that MobileCopy is robust against correlated node failures and efficient in file sharing in comparison with previous methods.

File sharing applications in mobile ad hoc networks (MANETs) have attracted more and more attention in recent years. The efficiency of file querying suffers from the distinctive properties of such networks including node mobility and limited communication range and resource. An intuitive method to alleviate this problem is to create file replicas in the network. However, despite the efforts on file replication, no research has focused on the global optimal replica creation with minimum average querying delay. Specifically, current file replication protocols in mobile ad hoc networks have two shortcomings. First, they lack a rule to allocate limited resources to different files in order to minimize the average querying delay. Second, they simply consider storage as available resources for replicas, but neglect the fact that the file holders’ frequency of meeting other nodes also plays an important role in determining file availability. Actually, a node that has a higher meeting frequency with others provides higher availability to its files. This becomes even more evident in sparsely distributed MANETs, in which nodes meet disruptively. In this paper, we introduce a new concept of resource for file replication, which considers both node storage and meeting frequency. We theoretically study the influence of resource allocation on the average querying delay and derive a resource allocation rule to minimize the average querying delay. We further propose a distributed file replication protocol to realize the proposed rule. Extensive trace-driven experiments with synthesized traces and real traces show that our protocol can achieve shorter average querying delay at a lower cost than current replication protocols.

Efficient and trustworthy file querying is important to the overall performance of peer-to-peer (P2P) file sharing systems. Emerging methods are beginning to address this challenge by exploiting online social networks (OSNs). However, current OSN-based methods simply cluster common-interest nodes for high efficiency or limit the interaction between social friends for high trustworthiness, which provides limited enhancement or contradicts the open and free service goal of P2P systems. Little research has been undertaken to fully and cooperatively leverage OSNs with integrated consideration of proximity and interest. In this work, we analyze a BitTorrent file sharing trace, which proves the necessity of proximity- and interest-aware clustering. Based on the trace study and OSN properties, we propose a social network integrated P2P file sharing system with enhanced efficiency and trustworthiness (SOCNET) to fully and cooperatively leverage the common-interest, proximity-close, and trust properties of OSN friends. SOCNET uses a hierarchical distributed hash table (DHT) to cluster common-interest nodes, then further cluster proximity-close nodes into a subcluster, and connects the nodes in a subcluster with social links. Thus, when queries travel along trustable social links, they also gain higher probability of being successfully resolved by proximity-close nodes, and it simultaneously enhances efficiency and trustworthiness. We further propose different strategies to guide nodes to forward a file query to friends that are more trustworthy and more likely to resolve the queries or forward the query to file holders. We also propose follower- and cluster-based file replication algorithms to enhance file search efficiency. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency, trustworthiness, and dynamism-resilience of SOCNET compared with other systems. Experimental results also confirm the effectiveness of the proposed strategies to improve SOCNET’s performance.

In peer-to-peer file sharing systems, file replication helps to avoid overloading file owners and improve file query efficiency. There exists a tradeoff between minimizing the number of replicas (i.e., replication overhead) and maximizing the replica hit rate (which reduces file querying latency). More replicas lead to increased replication overhead and higher replica hit rates and vice versa. An ideal replication method should generate a low overhead burden to the system while providing low query latency to the users. However, previous replication methods either achieve high hit rates at the cost of many replicas or produce low hit rates. To reduce replicas while guaranteeing high hit rate, this paper presents SWARM, a file replication mechanism based on swarm intelligence. Recognizing the power of collective behaviors, SWARM identifies node swarms with common node interests and close proximity. Unlike most earlier methods, SWARM determines the placement of a file replica based on the accumulated query rates of nodes in a swarm rather than a single node. Replicas are shared by the nodes in a swarm, leading to fewer replicas and high querying efficiency. In addition, SWARM has a novel consistency maintenance algorithm that propagates an update message between proximity-close nodes in a tree fashion from the top to the bottom. Experimental results from the real-world PlanetLab testbed and the PeerSim simulator demonstrate the effectiveness of the SWARM mechanism in comparison with other file replication and consistency maintenance methods. SWARM can reduce querying latency by 40-58 percent, reduce the number of replicas by 39-76 percent, and achieves more than 84 percent higher hit rates compared to previous methods. It also can reduce the consistency maintenance overhead by 49-99 percent compared to previous consistency maintenance methods.

Advances in ubiquitous sensing, computing and wireless communication technologies are leading to the development of cyber-physical systems (CPS), which promise to revolutionize the way we interact with the physical world. CPS applications, such as healthcare monitoring, may involve many users and objects scattered over a wide area. One critical function of CPS is object search in the physical world through the cyber sphere that enables interaction between the cyber and physical spheres. Some of the previously proposed physical object search engines use RFID tracking, and others collect the information of object locations into a hierarchical centralized server. The difficulty of widely deploying RFID devices, the centralized search, and the need for periodical location information collection prevent CPS from achieving higher scalability and efficiency. To deal with this problem, we propose a Social-aware distributed Cyber-Physical human-centric Search engine (SCPS) that leverages the social network formed by wireless device users for object search. Without requiring periodical location information collection, SCPS locates objects held by users based on the routine user movement pattern. Moreover, using a social-aware Bayesian network, it can accurately predict the users’ locations even at the occurrence of exceptional (i.e., non-routine) events (e.g., raining) that break user movement pattern. Thus, SCPS is more advantageous than all previous social network based works which assume that user behaviors always follow a certain pattern. Further, SCPS conducts the search in a fully distributed manner by relying on a distributed hash table (DHT) structure. As a result, SCPS achieves high scalability, efficiency and location accuracy. Extensive real-trace driven simulation results show the superior performance of SCPS compared to other representative search methods including a hierarchical centralized method, a decentralized method, and two social network based methods. The results also show the effectiveness of different components of SCPS.

A large-scale resource sharing system (e.g., collaborative cloud computing and grid computing) creates a virtual supercomputer by providing an infrastructure for sharing tremendous amounts of resources (e.g., computing, storage, and data) distributed over the Internet. A resource information service, which collects resource data and provides resource search functionality for locating desired resources, is a crucial component of the resource sharing system. In addition to resource discovery speed and cost (i.e., efficiency), the ability to accurately locate all satisfying resources (i.e., fidelity) is also an important metric for evaluating service quality. Previously, a number of resource information service systems have been proposed based on Distributed Hash Tables (DHTs) that offer scalable key-based lookup functions. However, these systems either achieve high fidelity at low efficiency, or high efficiency at low fidelity. Moreover, some systems have limited flexibility by only providing exact-matching services or by describing a resource using a pre-defined list of attributes. This paper presents a resource information service that offers high efficiency and fidelity without restricting resource expressiveness, while also providing a similar-matching service. Extensive simulation and PlanetLab experimental results show that the proposed service outperforms other services in terms of efficiency, fidelity, and flexibility; it dramatically reduces overhead and yields significant enhancements in efficiency and fidelity.

Question and Answer (Q&A) websites such as Yahoo! Answers provide a platform where users can post questions and receive answers. These systems take advantage of the collective intelligence of users to find information. In this paper, we analyze the online social network (OSN) in Yahoo! Answers. Based on a large amount of our collected data, we studied the OSN's structural properties, which reveals strikingly distinct properties such as low link symmetry and weak correlation between indegree and outdegree. After studying the knowledge base and behaviors of the users, we find that a small number of top contributors answer most of the questions in the system. Also, each top contributor focuses only on a few knowledge categories. In addition, the knowledge categories of the users are highly clustered. We also study the knowledge base in a user's social network, which reveals that the members in a user's social network share only a few knowledge categories. Based on the findings, we provide guidance in the design of spammer detection algorithms and distributed Q&A systems. We also propose a friendship-knowledge oriented Q&A framework that synergistically combines current OSN-based Q&A and web Q&A. We believe that the results presented in this paper are crucial in understanding the collective intelligence in the web Q&A OSNs and lay a cornerstone for the evolution of next-generation Q&A systems.

Hybrid wireless networks combining the advantages of both mobile ad-hoc networks and infrastructure wireless networks have been receiving increased attention due to their ultra-high performance. An efficient data routing protocol is important in such networks for high network capacity and scalability. However, most routing protocols for these networks simply combine the ad-hoc transmission mode with the cellular transmission mode, which inherits the drawbacks of ad-hoc transmission. This paper presents a Distributed Three-hop Routing protocol (DTR) for hybrid wireless networks. To take full advantage of the widespread base stations, DTR divides a message data stream into segments and transmits the segments in a distributed manner. It makes full spatial reuse of a system via its high speed ad-hoc interface and alleviates mobile gateway congestion via its cellular interface. Furthermore, sending segments to a number of base stations simultaneously increases throughput and makes full use of widespread base stations. In addition, DTR significantly reduces overhead due to short path lengths and the elimination of route discovery and maintenance. DTR also has a congestion control algorithm to avoid overloading base stations. Theoretical analysis and simulation results show the superiority of DTR in comparison with other routing protocols in terms of throughput capacity, scalability, and mobility resilience. The results also show the effectiveness of the congestion control algorithm in balancing the load between base stations.

In a hybrid wireless network that combines a mobile ad-hoc network and an infrastructure network, efficient and reliable data routing is important for high throughput. Existing routing schemes that simply combine ad-hoc and infrastructure routings inherit the drawbacks of ad-hoc routing including congestion and high overhead for route discovery and maintenance. Although current reputation systems help increase routing reliability, they rely on local information exchanges between nodes to evaluate node reputations, so they are not sufficiently effective and efficient. A challenge here is if we can coordinately develop an efficient routing algorithm and effective cooperation incentives for reliable routing. To handle this challenge, this paper presents a peer-to-peer (P2P)-based Market-guided Distributed Routing mechanism (MDR). MDR takes advantage of widespread base stations to coordinately realize highly efficient data routing, and effective reputation management and trading market management for reliable data routing. The packets from a source node are distributively transmitted to base stations directly or indirectly, and then they are transmitted to the destination. The base stations form a P2P structure for reputation collection and querying to avoid local information exchanges, and for managing the service transactions between nodes in the trading market. By leveraging the single-relay transmission feature, base stations can monitor the actual transmitted packets of relay nodes to more accurately and efficiently evaluate their reputations and execute trading market management, as well as detect falsely reported reputation information. We further propose market-based policies to strengthen cooperation incentives. Simulation results show that MDR outperforms the traditional hybrid routing schemes and reputation systems in achieving high throughput.

In this paper, we focus on the efficient routing of data among different areas in delay tolerant networks (DTNs). In current algorithms, packets are forwarded gradually through nodes with higher probability of visiting the destination node or area. However, the number of such nodes usually is limited, leading to insufficient throughput performance. To solve this problem, we propose an inter-landmark data routing algorithm, namely DTN-FLOW. It selects popular places that nodes visit frequently as landmarks and divides the entire DTN area into subareas represented by landmarks. Nodes transiting between landmarks relay packets among landmarks, even though they rarely visit the destinations of these packets. Specifically, the number of node transits between two landmarks is measured to represent the forwarding capacity between them, based on which routing tables are built on each landmark to guide packet routing. Each node predicts its transits based on its previous landmark visiting records using the order- k Markov predictor. When routing a packet, the landmark determines the next-hop landmark based on its routing table and forwards the packet to the node with the highest probability of transiting to the selected landmark. Thus, DTN-FLOW fully utilizes all node movements to route packets along landmark-based paths to their destinations. We analyzed two real DTN traces to support the design of DTN-FLOW. We deployed a small DTN-FLOW system on our campus for performance evaluation. We also proposed advanced extensions to improve its efficiency and stability. The real deployment and trace-driven simulation demonstrate the high efficiency of DTN-FLOW in comparison to state-of-the-art DTN routing algorithms.

Underlying link-layer protocols of well-established wireless networks that use the conventional “store-and-forward” design paradigm cannot provide highly sustainable reliability and stability in wireless communication, which introduce significant barriers and setbacks in scalability and deployments of wireless networks. In this paper, we propose a Code Embedded Distributed Adaptive and Reliable (CEDAR) link-layer framework that targets low latency and balancing en/decoding load among nodes. CEDAR is the first comprehensive theoretical framework for analyzing and designing distributed and adaptive error recovery for wireless networks. It employs a theoretically sound framework for embedding channel codes in each packet and performs the error correcting process in selected intermediate nodes in a packet's route. To identify the intermediate nodes for the decoding, we mathematically calculate the average packet delay and formalize the problem as a nonlinear integer programming problem. By minimizing the delays, we derive three propositions that: 1) can identify the intermediate nodes that minimize the propagation and transmission delay of a packet; and 2) and 3) can identify the intermediate nodes that simultaneously minimize the queuing delay and maximize the fairness of en/decoding load of all the nodes. Guided by the propositions, we then propose a scalable and distributed scheme in CEDAR to choose the intermediate en/decoding nodes in a route to achieve its objective. The results from real-world testbed “NESTbed” and simulation with MATLAB prove that CEDAR is superior to schemes using hop-by-hop decoding and destination decoding not only in packet delay and throughput but also in energy-consumption and load distribution balance.

In current peer-to-peer (P2P) live streaming systems, nodes in a channel form a P2P overlay for video sharing. To watch a new channel, a node depends on the centralized server to join in the overlay of the channel. In today's live streaming applications, the increase in the number of channels triggers users' desire of watching multiple channels successively or simultaneously. However, the support of such watching modes in current applications is no better than joining in different channel overlays successively or simultaneously through the centralized server, which if widely used, poses a heavy burden on the server. In order to achieve higher efficiency and scalability, we propose a Social-network-Aided efficient liVe strEaming system (SAVE). SAVE regards users' channel switching or multichannel watching as interactions between channels. By collecting the information of channel interactions, nodes' interests, and watching times, SAVE forms nodes in multiple channels with frequent interactions into an overlay, constructs bridges between overlays of channels with less frequent interactions, and enables nodes to identify friends sharing similar interests and watching times. Thus, a node can connect to a new channel while staying in its current overlay, using bridges or relying on its friends, reducing the need to contact the centralized server. We further propose the channel-closeness-based chunk-pushing strategy and capacity-based chunk provider selection strategy to enhance the system performance. Extensive experimental results from the PeerSim simulator and PlanetLab verify that SAVE outperforms other systems in system efficiency and server load reduction, as well as the effectiveness of the two proposed strategies.

Encouraging cooperation and deterring selfish behaviors are important for proper operations of mobile ad hoc networks (MANETs). For this purpose, most previous efforts rely on either reputation systems or price systems. However, these systems are neither sufficiently effective in providing cooperation incentives nor sufficiently efficient in resource consumption. Nodes in both systems can be uncooperative while still being considered trustworthy. Also, information exchange between mobile nodes in reputation systems and credit circulation in price systems consumes significant resources. This paper presents a hierarchical Account-aided Reputation Management system (ARM) to efficiently and effectively provide cooperation incentives. ARM builds a hierarchical locality-aware distributed hash table (DHT) infrastructure for efficient and integrated operation of both reputation and price systems. The infrastructure helps to globally collect all node reputation information in the system, which can be used to calculate more accurate reputation and detect abnormal reputation information. Also, ARM integrates reputation and price systems by enabling higher-reputed nodes to pay less for their received services. Theoretical analysis demonstrates the properties of ARM. Simulation results show that ARM outperforms the individual reputation system and price system in terms of effectiveness and efficiency of providing cooperation incentives and deterring selfish behaviors.

Current reputation systems for peer-to-peer (P2P) file sharing networks suffer from high overhead on reputation querying. Also, purely relying on a threshold to detect malicious nodes may make a high-reputed node be reluctant to further increase its reputation in these reputation systems. On the other side, the social network concept of “friendship foster cooperation” can be utilized to alleviate the high overhead in reputation systems. However, the limited number of friends limits the availability of file resources in these approaches. To overcome the drawbacks, we propose a social network based reputation system, namely SocialTrust, that synergistically leverages the social network connections and traditional credit based reputation system to provide efficient reputation management for P2P file sharing. In SocialTrust, each node favors friends for service transactions, which are resulted from both real life acquaintances and online partnerships established between high-reputed and frequently-interacted nodes. When no friends are available for a request, a node chooses the server with the highest reputation. The benefits of friendship and partnership on file sharing and cost saving encourage nodes to be continuously cooperative. Further, SocialTrust considers the number of friends/partners and the reputation of a node in reputation rewarding/punishment in order to realize accurate reputation evaluation. SocialTrust can also prevent certain attacks such as deny of service and collusion. Extensive trace-driven simulation demonstrates the effectiveness of SocialTrust.

In Delay Tolerant Networks (DTNs), nodes meet opportunistically and exchange packets only when they meet with each other. Therefore, routing is usually conducted in a store-carry-forward manner to exploit the scarce communication opportunities. As a result, different packet routing strategies, i.e., which packet to be forwarded or stored with priority, can lead to different routing performance objectives, such as minimal average delay and maximal hit rate. On the other hand, incentive systems are necessary for DTNs since nodes may be selfish and may not be cooperative on packet forwarding/storage. However, current incentive systems for DTNs mainly focus on encouraging nodes to participate in packet forwarding/storage but fail to further encourage nodes to follow a certain packet routing strategy to realize a routing performance objective. We name the former as the first aspect of cooperation and the latter as the second aspect of cooperation in DTN routing. Therefore, in this paper, we first discuss the routing strategy that can realize different performance objectives when nodes are fully cooperative, i.e., are willing to follow both aspects of cooperation. We then propose Multicent, a game theoretical incentive scheme that can encourage nodes to follow the two aspects of cooperation even when they are selfish. Basically, Multicent assigns credits for packet forwarding/storage in proportional to the priorities specified in the routing strategy. Multicent also supports adjustable Quality of Service (QoS) for packet routing between specific sources and destinations. Extensive trace-driven experimental results verify the effectiveness of Multicent.

A peer-to-peer (P2P) file sharing system provides a platform that enables a tremendous number of nodes to share their files. Retrieving desired files efficiently and trustworthily is critical in such a large and jumbled system. However, the issues of efficient searching and trustworthy searching have only been studied separately. Simply combining the methods to achieve the two goals doubles system overhead. In this paper, we first study trace data from Facebook and BitTorrent. Guided by the observations, we propose a system that integrates a social network into a P2P network, named Social-P2P, for simultaneous efficient and trustworthy file sharing. It incorporates three mechanisms: (1) interest/trust-based structure, (2) interest/trust-based file searching, and (3) trust relationship adjustment. By exploiting the social interests and relationships in the social network, the interest/trust-based structure groups common-multi-interest nodes into a cluster and further connects socially close nodes within a cluster. The comparably stable nodes in each cluster form a Distributed Hash Table (DHT) for inter-cluster file searching. In the interest/trust-based file searching mechanism, a file query is forwarded to the cluster of the file by the DHT routing first. Then, it is forwarded along constructed connections within a cluster, which achieves high hit rate and reliable routing. Moreover, sharing files among socially close friends discourages nodes from providing faulty files because people are unlikely to risk their reputation in the real-world. In the trust relationship adjustment mechanism, each node in a routing path adaptively decreases its trust on the node that has forwarded a faulty file in order to avoid routing queries towards misbehaving nodes later on. We conducted extensive trace-driven simulations and implemented a prototype on PlanetLab. Experimental results show that Social-P2P achieves highly efficient and trustworthy file sharing compared to current file sharing systems and trust management systems.

Efficient file query is important to the overall performance of peer-to-peer (P2P) file sharing systems. Clustering peers by their common interests can significantly enhance the efficiency of file query. Clustering peers by their physical proximity can also improve file query performance. However, few current works are able to cluster peers based on both peer interest and physical proximity. Although structured P2Ps provide higher file query efficiency than unstructured P2Ps, it is difficult to realize it due to their strictly defined topologies. In this work, we introduce a Proximity-Aware and Interest-clustered P2P file sharing System (PAIS) based on a structured P2P, which forms physically-close nodes into a cluster and further groups physically-close and common-interest nodes into a sub-cluster based on a hierarchical topology. PAIS uses an intelligent file replication algorithm to further enhance file query efficiency. It creates replicas of files that are frequently requested by a group of physically close nodes in their location. Moreover, PAIS enhances the intra-sub-cluster file searching through several approaches. First, it further classifies the interest of a sub-cluster to a number of sub-interests, and clusters common-sub-interest nodes into a group for file sharing. Second, PAIS builds an overlay for each group that connects lower capacity nodes to higher capacity nodes for distributed file querying while avoiding node overload. Third, to reduce file searching delay, PAIS uses proactive file information collection so that a file requester can know if its requested file is in its nearby nodes. Fourth, to reduce the overhead of the file information collection, PAIS uses bloom filter based file information collection and corresponding distributed file searching. Fifth, to improve the file sharing efficiency, PAIS ranks the bloom filter results in order. Sixth, considering that a recently visited file tends to be visited again, the bloom filter based approach is enhanced by only checking the newly added bloom filter information to reduce file searching delay. Trace-driven experimental results from the real-world PlanetLab testbed demonstrate that PAIS dramatically reduces overhead and enhances the efficiency of file sharing with and without churn. Further, the experimental results show the high effectiveness of the intra-sub-cluster file searching approaches in improving file searching efficiency.

To provide robust infrastructure as a service (IaaS), clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. Previous reactive load balancing algorithms migrate VMs upon the occurrence of load imbalance, while previous proactive load balancing algorithms predict PM overload to conduct VM migration. However, both methods cannot maintain long-term load balance and produce high overhead and delay due to migration VM selection and destination PM selection. To overcome these problems, in this paper, we propose a proactive Markov Decision Process (MDP)-based load balancing algorithm. We handle the challenges of allying MDP in virtual resource management in cloud datacenters, which allows a PM to proactively find an optimal action to transit to a lightly loaded state that will maintain for a longer period of time. We also apply the MDP to determine destination PMs to achieve long-term PM load balance state. Our algorithm reduces the numbers of Service Level Agreement (SLA) violations by long-term load balance maintenance, and also reduces the load balancing overhead (e.g., CPU time, energy) and delay by quickly identifying VMs and destination PMs to migrate. Our trace-driven experiments show that our algorithm outperforms both previous reactive and proactive load balancing algorithms in terms of SLA violation, load balancing efficiency and long-term load balance maintenance.

Content Delivery Networks (CDNs) play a central role of today's Internet infrastructure, and have seen a sharp increment in scale. More and more internet sites are armed with dynamic (or live) content (such as live sports game statistics, e-commerce and online auction), and there is a need to deliver dynamic content freshly in scale. To achieve high scalability, the consistency maintenance problem for dynamic content (contents with frequent updates) served by CDNs is non-trivial. The large number of widely scattered replicas guarantee the service QoS of end-users, meanwhile largely increase the complexity of consistency maintenance. Current consistency maintenance infrastructures and methods cannot simultaneously satisfy the two requirements: scalability and consistency. In this paper, we first analyze our crawled trace data of a cached sports game content on thousands of content servers of a major CDN. We analyze the content consistency from different perspectives, from which we try to break down the reasons for inconsistency among content servers. Finally, we further evaluate the performance in consistency, scalability and overhead for different infrastructures with different update methods. We itemize the advantages and disadvantages of different methods and infrastructures in different scenarios through the evaluation. We aim to give guidance for appropriate selections of consistency maintenance infrastructures and methods for a CDN, and for choosing a CDN service with different considerations.

The past few years have seen an explosion in the popularity of online short-video sharing in You Tube. As the number of users continued to grow, the bandwidth required to maintain acceptable quality of service (QoS) has greatly increased. Peer-to-peer (P2P) architectures have shown promise in reducing the bandwidth costs, however, the previous works build one P2P overlay for each video, which provides limited availability of video providers and produces high overlay maintenance overhead. To handle these problems, in this work, we novelly leverage the existing social network in You Tube, where a user subscribes to another user's channel to track all his uploaded videos. The subscribers of a channel tend to watch the channel's videos and common-interest nodes tend to watch the same videos. Also, the popularity of videos in one channel varies greatly. We study real trace data to confirm these properties. Based on these properties, we propose Social Tube that builds the subscribers of one channel into a P2P overlay and also clusters common-interest nodes in a higher level. It also incorporates a prefetching algorithm that prefetches higher-popularity videos. Extensive trace-driven simulation results and Planet Lab real world experimental results verify the effectiveness of Social Tube at reducing server load and overlay maintenance overhead and at improving QoS for users.

The shared multi-tenant nature of cloud network infrastructures has caused poor application performance in the clouds due to unpredictable network performance. To provide bandwidth guarantee, several virtual network abstractions have been proposed which allow the tenants to specify and reserve virtual clusters with required network bandwidth between the VMs. However, all of these existing proposals require the tenants to deterministically characterize the exact bandwidth demands in the abstractions, which can be difficult and result in inefficient bandwidth reservation due to the demand uncertainty. In this paper, we propose a virtual cluster abstraction with stochastic bandwidth requirements between VMs, called Stochastic Virtual Cluster (SVC), which probabilistically models the bandwidth demand uncertainty. Based on SVC, we propose a network sharing framework and efficient VM allocation algorithms to ensure that the bandwidth demands of tenants on any link are satisfied with a high probability, while minimizing the bandwidth occupancy cost on links. Using simulations, we demonstrate the effectiveness of SVC for accommodating cloud application workloads with highly volatile bandwidth demands, in the way of achieving the trade-off between the job concurrency and average job running time.

Question and Answer (Q&A) systems aggregate the collected intelligence of all users to provide satisfying answers for questions. A well-developed Q&A system should incorporate features such as high question response rate, high answer quality, a spam-free environment for users. Previous works use reputation systems to achieve the goals. However, these reputation systems evaluate a user with an overall rating for all questions the user has answered regardless of the question categories, thus the reputation score does not accurately reflect the user's ability to answer a question in a specific category. We propose SmartQ: a reputation based Q&A System. SmartQ employs a category and theme based reputation management system to evaluate users' willingness and capability to answer various kinds of questions. The reputation system facilitates the forwarding of a question to favorable experts, which improves the question response rate and answer quality. Also, SmartQ incorporates a lightweight spammer detection method to identify potential spammers. Our trace-driven simulation on PeerSim demonstrates the effectiveness of SmartQ in providing a good user experience. We then develop a real application of SmartQ and deploy it for use in a student group in Clemson University. The user feedback shows that SmartQ can provide high-quality answers for users in a community.

In cloud datacenters, effective resource provisioning is needed to maximize energy efficiency and utilization of cloud resources while guaranteeing the Service Level Agreement (SLA) for tenants. Previous resource provisioning strategies either allocate physical resources to virtual machines (VMs) based on static VM resource demands or dynamically handle the variations in VM resource requirements through live VM migrations. However, the former fail to maximize energy efficiency and resource utilization while the latter produce high migration overhead. To handle these problems, we propose an initial VM allocation mechanism that consolidates complementary VMs with spatial/temporal-awareness. Complementary VMs are the VMs whose total demand of each resource dimension (in the spatial space) nearly reaches their host's capacity during VM lifetime period (in the temporal space). Based on our observation of the existence of VM resource utilization patterns, the mechanism predicts the lifetime resource utilization patterns of short-term VMs or periodical resource utilization patterns of long-term VMs. Based on the predicted patterns, it coordinates the requirements of different resources and consolidates complementary VMs in the same physical machine (PM). This mechanism reduces the number of PMs needed to provide VM service hence increases energy efficiency and resource utilization and also reduces the number of VM migrations and SLA violations. Simulation based on two real traces and real-world testbed experiments show that our initial VM allocation mechanism significantly reduces the number of PMs used, SLA violations and VM migrations of the previous resource provisioning strategies.

In delay tolerant networks (DTNs), enabling a node to search and find an interested mobile node is an important function in many applications. However, the movement of nodes in DTNs makes the problem formidable. Current methods in disconnected networks mainly rely on fixed stations and infrastructure-based communication to collect node position information, which is difficult to implement in DTNs. In this paper, we present DSearching, a distributed mobile node searching scheme for DTNs that requires no infrastructure except the GPS on mobile nodes. In DSearching, the entire DTN area is split into sub-areas, and each node summarizes its mobility information as both transient sub-area visiting record and long-term movement pattern among sub-areas. Each node distributes its transient visiting record for a newly entered sub-area to nodes that are likely to stay in the sub-area that it just moves out, so that the information flows in the network for the locator to trace it along its actual movement path. Each node also stores different parts of its long-term mobility pattern to long-staying nodes in different sub-areas for the locator to trace it when visiting records are absent. Considering that nodes in DTNs usually have limited resources, DSearching constrains the communication and storage cost in the information distribution process. Extensive trace-driven experiments with real traces demonstrate the high efficiency and high effectiveness of DSearching.

To provide robust infrastructure as a service (IaaS), clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. The unique features of clouds pose formidable challenges to achieving effective and efficient load balancing. First, VMs in clouds use different resources (e.g., CPU, bandwidth, memory) to serve a variety of services (e.g., high performance computing, web services, file services), resulting in different overutilized resources in different PMs. Also, the overutilized resources in a PM may vary over time due to the time-varying heterogenous service requests. Second, there is intensive network communication between VMs. However, previous load balancing methods statically assign equal or predefined weights to different resources, which leads to degraded performance in terms of speed and cost to achieve load balance. Also, they do not strive to minimize the VM communications between PMs. We propose a Resource Intensity Aware Load balancing method (RIAL). For each PM, RIAL dynamically assigns different weights to different resources according to their usage intensity in the PM, which significantly reduces the time and cost to achieve load balance and avoids future load imbalance. It also tries to keep frequently communicating VMs in the same PM to reduce bandwidth cost, and migrate VMs to PMs with minimum VM performance degradation. Our extensive trace-driven simulation results and real-world experimental results show the superior performance of RIAL compared to other load balancing methods.

Cooperative broadcast, in which receivers are allowed to combine received packet from different senders to combat transmission errors, has gained increasing attention. Previous studies showed that broadcast optimization solutions are sufficient in non-fading environments but may suffer a low delivery ratio under wireless channel fading. Though previous work analyzed the tradeoff between energy and delay in cooperative broadcast, no works investigated the tradeoff in a fading environment. Thus, in this paper, we study this tradeoff with the consideration of fading. We formulate this problem as a Fading-resistant Delay-constrained Minimum Energy Cooperative Broadcast (FDMECB) problem, and prove that it is NP-complete. We then propose an approximation algorithm for theoretical interests. We further propose a heuristic algorithm that makes approximately optimal local decision to achieve global optimization. Our experimental results show that our algorithms outperform a previous non-fading resistant algorithm.

For predictable application performance or fairness in network sharing in clouds, many bandwidth allocation policies have been proposed. However, with these policies, tenants are not incentivized to use idle bandwidth or prevent link congestion, and may even take advantage of the policies to gain unfair bandwidth allocation. Increasing network utilization while avoiding congestion not only benefits cloud provider but also the tenants by improving application performance. In this paper, we propose a new pricing model that sets different unit prices for reserved bandwidth, the bandwidth on congested links and on uncongested links, and makes the unit price for congested links proportional to their congestion degrees. We use game theory model to analyze tenants' behaviors in our model and the current pricing models, which shows the effectiveness of our model in providing the incentives. With the pricing model, we propose a network sharing policy to achieve both min-guarantee and proportionality, while prevent tenants from earning unfair bandwidth. We further propose methods for each virtual machine to arrange its traffic to maximize its utility. As a result, our solution creates a win-win situation, where tenants strive to increase their benefits in bandwidth sharing, which also concurrently increases the utilities of cloud provider and other tenants. Our simulation and trace-driven experimental results show the effectiveness of our solution in creating the win-win situation.

The explosive growth of unsolicited e-mails has prompted the development of numerous spam filter techniques. Bayesian spam filters are superior to static keyword-based spam filters in that they can continuously evolve to tackle new spam by learning keywords in new spam emails. However, Bayesian spam filters are easily poisoned by clever spammers who avoid spam keywords and add many innocuous words in their emails. Also, Bayesian spam filters need a significant amount of time to adapt to a new spam based on user feedback. Moreover, few current spam filters exploit social networks to assist in spam detection. In order to develop an accurate and user-friendly spam filter, we propose a SOcial network Aided Personalized and effective spam filter (SOAP) in this paper. In SOAP, each node connects to its social friends; i.e., nodes form a distributed overlay by directly using social network links as overlay links. Each node uses SOAP to collect information and check spam autonomously in a distributed manner. Unlike previous spam filters that focus on parsing keywords (e.g., Bayesian filters) or building blacklists, SOAP exploits the social relationships among email correspondents and their (dis)interests to detect spam adaptively and automatically. In each node, SOAP integrates four components into the basic Bayesian filter: social closeness-based spam filtering, social interest-based spam filtering, adaptive trust management, and friend notification. We have evaluated the performance of SOAP using simulation based on trace data from Facebook. We also have implemented a SOAP prototype for real-world experiments. Experimental results show that SOAP can greatly improve the performance of Bayesian spam filters in terms of accuracy, attack-resilience, and efficiency of spam detection. The performance of the Bayesian spam filter is SOAP’s lower bound.

Current peer-to-peer (P2P) file sharing methods in mobile ad hoc networks (MANETs) can be classified into three groups: flooding-based, advertisement-based, and social contact-based. The first two groups of methods can easily have high overhead and low scalability. They are mainly developed for connected MANETs, in which end-to-end connectivity among nodes is ensured. The third group of methods adapts to the opportunistic nature of disconnected MANETs but fails to consider the social interests (i.e., contents) of mobile nodes, which can be exploited to improve the file searching efficiency. In this paper, we propose a P2P content-based file sharing system, namely SPOON, for disconnected MANETs. The system uses an interest extraction algorithm to derive a node's interests from its files for content-based file searching. For efficient file searching, SPOON groups common-interest nodes that frequently meet with each other as communities. It takes advantage of node mobility by designating stable nodes, which have the most frequent contact with community members, as community coordinators for intracommunity searching, and highly mobile nodes that visit other communities frequently as community ambassadors for intercommunity searching. An interest-oriented file searching scheme is proposed for high file searching efficiency. Additional strategies for file prefetching, querying-completion, and loop-prevention, and node churn consideration are discussed to further enhance the file searching efficiency. We first tested our system on the GENI Orbit testbed with a real trace and then conducted event-driven experiment with two real traces and NS2 simulation with simulated disconnected and connected MANET scenarios. The test results show that our system significantly lowers transmission cost and improves file searching success rate compared to current methods.

As wireless communication gains popularity, significant research has been devoted to supporting real-time transmission with stringent Quality of Service (QoS) requirements for wireless applications. At the same time, a wireless hybrid network that integrates a mobile wireless ad hoc network (MANET) and a wireless infrastructure network has been proven to be a better alternative for the next generation wireless networks. By directly adopting resource reservation-based QoS routing for MANETs, hybrids networks inherit invalid reservation and race condition problems in MANETs. How to guarantee the QoS in hybrid networks remains an open problem. In this paper, we propose a QoS-Oriented Distributed routing protocol (QOD) to enhance the QoS support capability of hybrid networks. Taking advantage of fewer transmission hops and anycast transmission features of the hybrid networks, QOD transforms the packet routing problem to a resource scheduling problem. QOD incorporates five algorithms: 1) a QoS-guaranteed neighbor selection algorithm to meet the transmission delay requirement, 2) a distributed packet scheduling algorithm to further reduce transmission delay, 3) a mobility-based segment resizing algorithm that adaptively adjusts segment size according to node mobility in order to reduce transmission time, 4) a traffic redundant elimination algorithm to increase the transmission throughput, and 5) a data redundancy elimination-based transmission algorithm to eliminate the redundant data to further improve the transmission QoS. Analytical and simulation results based on the random way-point model and the real human mobility model show that QOD can provide high QoS performance in terms of overhead, transmission delay, mobility-resilience, and scalability.

Previous delay-tolerant network (DTN) routing algorithms exploit either past encounter records or social network properties to derive a node's probability of delivering packets to their destinations. However, they only have a local view of the network, which limits the routing efficiency. Also, when two nodes meet, they have to exchange the delivery abilities to the destinations of all packets in the two nodes, which incurs high resource consumption. In this paper, we propose SMART, which utilizes a distributed social map for lightweight routing in delay-tolerant networks. In SMART, each node builds its own social map consisting of nodes it has met and their frequently encountered nodes in a distributed manner. Based on both encountering frequency and social closeness of the two linked nodes in the social map, we decide the weight of each link to reflect the packet delivery ability between the two nodes. The social map enables more accurate forwarder selection through a broader view. Moreover, nodes exchange much less information for social map update, which reduces resource consumption. Trace-driven experiments and tests on the GENI ORBIT testbed demonstrate the high efficiency of SMART in comparison to previous algorithms.

Recently, many schemes have been proposed for detecting and healing coverage holes to achieve full coverage in wireless sensor networks (WSNs). However, none of these schemes aim to find the shortest node movement paths to heal the coverage holes, which could significantly reduce energy usage for node movement. Also, current hole healing schemes require accurate knowledge of sensor locations; obtaining this knowledge consumes high energy. In this paper, we propose a Delaunay-based coordinate-free mechanism (DECM) for full coverage. Based on rigorous mathematical analysis, DECM can detect coverage holes and find the locally shortest paths for healing holes in a distributed manner without requiring accurate node location information. Also, DECM incorporates a cooperative movement mechanism that can prevent generating new holes during node movements in healing holes. Simulation results and experimental results from the real-world GENI Orbit testbed show that DECM achieves superior performance in terms of the energy-efficiency, effectiveness of hole healing, energy consumption balance and lifetime compared to previous schemes.

Advancements in cloud computing are leading to a promising future for collaborative cloud computing (CCC), where globally-scattered distributed cloud resources belonging to different organizations or individuals (i.e., entities) are collectively used in a cooperative manner to provide services. Due to the autonomous features of entities in CCC, the issues of resource management and reputation management must be jointly addressed in order to ensure the successful deployment of CCC. However, these two issues have typically been addressed separately in previous research efforts, and simply combining the two systems generates double overhead. Also, previous resource and reputation management methods are not sufficiently efficient or effective. By providing a single reputation value for each node, the methods cannot reflect the reputation of a node in providing individual types of resources. By always selecting the highest-reputed nodes, the methods fail to exploit node reputation in resource selection to fully and fairly utilize resources in the system and to meet users' diverse QoS demands. We propose a CCC platform, called Harmony, which integrates resource management and reputation management in a harmonious manner. Harmony incorporates three key innovations: integrated multi-faceted resource/reputation management, multi-QoS-oriented resource selection, and price-assisted resource/reputation control. The trace data we collected from an online trading platform implies the importance of multi-faceted reputation and the drawbacks of highest-reputed node selection. Simulations and trace-driven experiments on the real-world PlanetLab testbed show that Harmony outperforms existing resource management and reputation management systems in terms of QoS, efficiency and effectiveness.

This paper addresses the data search problem in large-scale highly mobile and dense wireless networks. Current wireless network data search systems are not suitable for large-scale highly mobile and dense wireless networks. This paper presents a scalable and mobility-resilient LOcality-based distRibuted Data search system (LORD) for large-scale wireless networks with high mobility and density. Taking advantage of the high density, rather than mapping data to a location point, LORD maps file metadata to a geographical region and stores it in multiple nodes in the region, thus enhancing mobility-resilience. LORD has a novel region-based geographic data routing protocol that does not rely on flooding or GPSs for data publishing and querying, and a coloring-based partial replication algorithm to reduce data replicas in a region while maintaining the querying efficiency. LORD also works for unbalanced wireless networks with sparse regions. Simulation results show the superior performance of LORD compared to representative data search systems in terms of scalability, overhead, and mobility resilience in a highly dense and mobile network. The results also show the high scalability and mobility-resilience of LORD in an unbalanced wireless network with sparse regions, and the effectiveness of its coloring-based partial replication algorithm.

Tremendous advances in pervasive networking have enabled wide-area distributed systems to connect distributed resources or users such as corporate data centers and high-performance computing centers. These distributed pervasive systems take advantage of resources and enhance collaborations worldwide. However, due to lack of central management, they are severely threatened by a variety of malicious users in today's Internet. Current reputation- and anonymity-based technologies for node communication enhance system trustworthiness. However, most of these technologies gain trustworthiness at the cost of efficiency degradation. This paper presents a P2P-based infrastructure for trustworthy and efficient node communication in wide-area distributed systems. It jointly addresses trustworthiness and efficiency in its operation in order to meet the high-performance requirements of a diversified wealth of distributed pervasive applications. The infrastructure includes two policies: trust/efficiency-oriented request routing and trust-based adaptive anonymous response forwarding. This infrastructure not only offers a trustworthy environment with anonymous communication but also enhances overall system efficiency through harmonious trustworthiness and efficiency trade-offs. Experimental results from simulations and the real-world PlanetLab testbed show the superior performance of the P2P-based infrastructure in achieving both high trustworthiness and high efficiency in comparison to other related approaches.

Recently, emerging research efforts have been focused on question and answer (Q&A) systems based on social networks. The social-based Q&A systems can answer non-factual questions, which cannot be easily resolved by web search engines. These systems either rely on a centralized server for identifying friends based on social information or broadcast a user's questions to all of its friends. Mobile Q&A systems, where mobile nodes access the Q&A systems through Internet, are very promising considering the rapid increase of mobile users and the convenience of practical use. However, such systems cannot directly use the previous centralized methods or broadcasting methods, which generate high cost of mobile Internet access, node overload, and high server bandwidth cost with the tremendous number of mobile users. We propose a distributed Social-based mObile Q&A System (SOS) with low overhead and system cost as well as quick response to question askers. SOS enables mobile users to forward questions to potential answerers in their friend lists in a decentralized manner for a number of hops before resorting to the server. It leverages lightweight knowledge engineering techniques to accurately identify friends who are able to and willing to answer questions, thus reducing the search and computation costs of mobile nodes. The trace-driven simulation results show that SOS can achieve a high query precision and recall rate, a short response latency and low overhead. We have also deployed a pilot version of SOS for use in a small group in Clemson University. The feedback from the users shows that SOS can provide high-quality answers.

Video sharing has been an increasingly popular application in online social networks (OSNs). However, its sustainable development is severely hindered by the intrinsic limit of the client/server architecture deployed in current OSN video systems, which is not only costly in terms of server bandwidth and storage but also not scalable with the soaring amount of users and video content. The peer-assisted Video-on-Demand (VoD) technique, in which participating peers assist the server in delivering video content, has been proposed recently. Unfortunately, videos can only be disseminated through friends in OSNs. Therefore, current VoD works that explore clustering nodes with similar interests or close location for high performance are suboptimal, if not entirely inapplicable, in OSNs. Based on our long-term real-world measurement of over 1,000,000 users and 2,500 videos on Facebook, we propose SocialTube, a novel peer-assisted video sharing system that explores social relationship, interest similarity, and physical location between peers in OSNs. Specifically, SocialTube incorporates four algorithms: a social network (SN)-based P2P overlay construction algorithm, an SN-based chunk prefetching algorithm, chunk delivery, and scheduling algorithm, and a buffer management algorithm. Experimental results from a prototype on PlanetLab and an event-driven simulator show that SocialTube can improve the quality of user experience and system scalability over current P2P VoD techniques.

Radio frequency identification (RFID) and wireless sensor networks (WSNs) have been popular in the industrial field, and both have undergone dramatic development. RFID and WSNs are well known for their abilities in identity identification and data transmission, respectively, and hence widely used in applications for environmental and health monitoring. Though the integration of a sensor and an RFID tag was proposed to gather both RFID tag and sensed information, few previous research efforts explore the integration of data transmission modes in the RFID and WSN systems to enhance the performance of the applications. In this paper, we propose a hybrid RFID and WSN system (HRW) that synergistically integrates the traditional RFID system and WSN system for efficient data collection. HRW has hybrid smart nodes that combine the function of RFID tags, the reduced function of RFID readers, and wireless sensors. Therefore, nodes can read each other's sensed data in tags, and all data can be quickly transmitted to an RFID reader through the node that first reaches it. The RFID readers transmit the collected data to the back-end servers for data processing and management. We also propose methods to improve data transmission efficiency and to protect data privacy and avoid malicious data selective forwarding in data transmission. Comprehensive simulation and trace-driven experimental results show the high performance of HRW in terms of the cost of deployment, transmission delay and capability, and tag capacity requirement.

Continuous aggregation is usually required in many sensor applications to obtain the temporal variation information of aggregates. However, in a hostile environment, the adversary could fabricate false temporal variation patterns of the aggregates by manipulating a series of aggregation results through compromised nodes. Existing secure aggregation schemes conduct one individual verification for each aggregation result, which could incur great accumulative communication cost and negative impact on transmission scheduling for continuous aggregation. In this paper, we identify distinct design issues for protecting continuous in-network aggregation and propose a novel scheme to detect false temporal variation patterns. Compared with the existing schemes, our scheme greatly reduces the verification cost by checking only a small part of aggregation results to verify the correctness of the temporal variation patterns in a time window. A sampling-based approach is used to check the aggregation results, which enables our scheme independent of any particular in-network aggregation protocols as opposed to existing schemes. We also propose a series of security mechanisms to protect the sampling process. Both theoretical analysis and simulations show the effectiveness and efficiency of our scheme.

In current reputation systems for peer-to-peer (P2P) file sharing networks, reputation querying generates high overhead in file service. Also, a high-reputed node may be reluctant to further increase its reputation in these reputation systems. Recently, social network-based P2P systems that favor file sharing among friends have been proposed for reliable file sharing. However, this approach contradicts the objective of widely file sharing in P2P systems. To overcome these drawbacks, we propose a social network integrated reputation system, namely SocialTrust, that synergistically leverages the concept of "friendship fosters cooperation''. In SocialTrust, each node maintains trusted relationships for the purpose of file sharing, which are resulted from both real life acquaintance and online partnership established between high-reputed and frequently-interacted nodes. Given a number of server options, a client first chooses a friend or a partner, if available, without querying their reputations. Otherwise, it chooses the server with the highest reputation. The benefits of friendship and partnership on file sharing and cost saving encourage nodes to be continuously cooperative. Further, in order to realize accurate reputation evaluation and strong cooperation incentives, SocialTrust considers the number of friends/partners and reputation of a node in the process of reputation rewarding and punishment. Extensive trace-driven simulation demonstrates the effectiveness of SocialTrust.

Though the new OSN model with many worldwide distributed small datacenters helps reduce service latency, it brings a problem of higher inter-datacenter communication load. In Facebook, each datacenter has a full copy of all data and the master datacenter updates all other datacenters, which obviously generates tremendous load in this new model. Distributed data storage that only stores a user's data to his/her geographically-closest datacenters mitigates the problem. However, frequent interactions between far-away users lead to frequent inter-datacenter communication and hence long service latency. In this paper, we aim to reduce inter-datacenter communications while still achieve low service latency. We first verify the benefits of the new model and present OSN typical properties that lay the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters (SD3). In SD3, a datacenter jointly considers update rate and visit rate to select user data for replication, and further atomizes a user's different types of data (e.g., status update, friend post) for replication, making sure that a replica always reduces inter-datacenter communication. The results of trace-driven experiments on the real-world PlanetLab testbed demonstrate the higher efficiency and effectiveness of SD3in comparison to other replication methods.

In many countries, the Internet is under stringent censorship for political or religious reasons which severely undermines the free flow of information. A censorship-resistant web browsing system must be scalable, blocking resistant, and tracing resistant. However, current censorship-resistant web browsing systems, which use a group of dedicated proxies to bypass censorship, fail to meet these requirements. To tackle these challenges, we propose Free web, which relies on widely-distributed peer-to-peer (P2P) nodes in a decentralized manner rather than specified proxies in a centralized manner. Free web is built on top of a Distributed Hash Table (DHT)-based P2P network, where nodes not under censorship help nodes under censorship to access blocked web pages. Free web has a web browser front end whose user interface resembles existing web browsers. The underlying complex process of retrieving blocked web pages is therefore hidden from users. We implemented and open-sourced Free web and conducted extensive real-world experiments on Planet Lab. The experimental results show that Free web has a high success rate and reasonable browsing latency.

Underlying link-layer protocols of wireless networks use the conventional “store and forward” design paradigm cannot provide highly sustainable reliability and stability in wireless communication, which introduce significant barriers and setbacks in scalability and deployments of wireless networks. In this paper, we propose a Code Embedded Distributed Adaptive and Reliable (CEDAR) link-layer framework that targets low latency and high throughput. CEDAR is the first comprehensive theoretical framework for analyzing and designing distributed and adaptive error recovery for wireless networks. It employs a theoretically-sound framework for embedding channel codes in each packet and performs the error correcting process in selected intermediate nodes in packet's route. To identify the intermediate nodes for the en/decoding for minimizing average packet latency, we mathematically analyze the average packet delay, using Finite State Markovian Channel model and priority queuing model, and then formalize the problem as a non-linear integer programming problem. Also, we propose a scalable and distributed scheme to solve this problem. The results from real-world testbed “NESTbed” and simulation with Matlab prove that CEDAR is superior to the schemes using hop-by-hop decoding and destination-decoding not only in packet delay but also in throughput. In addition, the simulation results show that CEDAR can achieve the optimal performance in most cases.

Cooperative broadcast, in which a packet receiver cooperatively combines received weak signal power from different senders to decode the original packet, has gained increasing attention. However, existing approaches are developed based on the assumption that there is a single flow in the network; thus, they are not suitable for multi-flow broadcasting in which broadcasts are initiated by different nodes and consist of more than one packet at any point in time. In this paper, we aim to achieve low-latency multi-flow broadcast in wireless multihop networks with fading channels. We formulate this problem as a Minimum Slotted Delay Cooperative Broadcast (MSDCB) problem, and prove that it is NP-complete and o(logN) inapproximable. We then propose two heuristic algorithms named PCBHS and PCBH-M to solve MSDCB. Our experimental results show that our algorithms outperform previous methods.

In this paper, we focus on the efficient routing of data among different areas in Delay Tolerant Networks (DTNs). In current algorithms, packets are forwarded gradually through nodes with higher probability of visiting the destination node or area. However, the number of such nodes usually is limited, leading to insufficient throughput performance. To solve this problem, we propose an inter-landmark data routing algorithm, namely DTN-FLOW. It selects popular places that nodes visit frequently as landmarks and divides the entire DTN area into sub-areas represented by landmarks. Nodes transiting between landmarks relay packets among landmarks, even though they rarely visit the destinations of these packets. Specifically, the number of node transits between two landmarks is measured to represent the forwarding capacity between them, based on which routing tables are built on each landmark to guide packet routing. Each node predicts its transits based on its previous landmark visiting records using the order-k Markov predictor. In a packet routing, a landmark determines the next hop landmark based on its routing table, and forwards the packet to the node with the highest probability of transiting to the selected landmark. Thus, DTN-FLOW fully utilizes all node movements to route packets along landmark paths to their destinations. We analyzed two real DTN traces to support the design of DTN-FLOW. We also deployed a small DTN-FLOW system in our campus for performance evaluation. This deployment and trace-driven simulation demonstrate the high efficiency of DTN-FLOW in comparison with state-of-the-art DTN routing algorithms.

Due to intermittent connection and limited communication opportunity, routing in Delay Tolerant Networks (DTNs) is usually conducted in a store-carry-forward manner. Consequently, different packet forwarding or storage strategies can lead to different performance objectives, such as minimal average delay, maximal hit rate, and minimal maximal delay, which are desired by applications with different purposes. However, nodes may not be willing to follow these strategies. Further, selfish nodes may even refuse to carry or forward packets for others if they cannot obtain benefits in return. Though many incentive systems have been proposed to encourage packet forwarding, none of them aim to encourage nodes to realize the aforementioned performance objectives. In this paper, we first discuss the strategies that can realize different performance objectives and then propose Multicent, a game theoretical incentive scheme that not only provides cooperative incentives but also encourages nodes to follow defined rules to realize the desired performance objective. Multicent also makes the Quality of Service (QoS) of packet routing adjustable for specific sources, destinations, or source-destination pairs. Extensive trace-driven experimental results verify the effectiveness of Multicent.

This work focuses on Delay Tolerant Networks (DTNs) in a social network environment. DTNs do not have a complete path from a source to a destination most of the time. Previous data routing approaches in DTNs are primarily based on either flooding or single-copy routing. However, these methods incur either high overhead due to excessive transmissions or long delays due to suboptimal choices for relay nodes. Probabilistic forwarding that forwards a message to a node with a higher delivery utility enhances single-copy routing. However, current probabilistic forwarding methods only consider node contact frequency in calculating the utility while neglecting the influence of contact duration on the throughput, though both contact frequency and contact duration reflect the node movement pattern in a social network. In this paper, we theoretically prove that considering both factors leads to higher throughput than considering only contact frequency. To fully exploit a social network for high throughput and low routing delay, we propose a Social network oriented and duration utility-based distributed multicopy routing protocol (SEDUM) for DTNs. SEDUM is distinguished by three features. First, it considers both contact frequency and duration in node movement patterns of social networks. Second, it uses multicopy routing and can discover the minimum number of copies of a message to achieve a desired routing delay. Third, it has an effective buffer management mechanism to increase throughput and decrease routing delay. Theoretical analysis and simulation results show that SEDUM provides high throughput and low routing delay compared to existing routing approaches. The results conform to our expectation that considering both contact frequency and duration for delivery utility in routing can achieve higher throughput than considering only contact frequency, especially in a highly dynamic environment with large routing messages.

In peer-to-peer networks (P2Ps), many autonomous peers without preexisting trust relationships share resources with each other. Due to their open environment, the P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high reliability and security; however, node collusion impairs the effectiveness of reputation systems in trustworthy node selection. Although some reputation systems have certain mechanisms to counter collusion, the effectiveness of the mechanisms is not sufficiently high. In this paper, we leverage social networks to enhance the capability of reputation systems in combating collusion. We first analyzed real trace of the reputation system in the Overstock online auction platform, which incorporates a social network. The analysis reveals the impact of the social network on user purchasing and reputation rating patterns. We, thus, identified suspicious collusion behavior patterns and propose a social network-based mechanism, SocialTrust, to counter collusion. SocialTrust adaptively adjusts the weight of ratings based on the social distance and interest relationship between peers. Experiment results show that SocialTrust can significantly strengthen the capability of current reputation systems in combating collusion.

File replication is widely used in structured P2P systems to avoid hot spots in servers and enhance file availability. The number of replicas and replication distance affect the file replication cost. These two elements and the replica update frequency determined in the file replication stage also affect the cost of subsequent consistency maintenance. However, most existing file replication protocols focus on improving file lookup efficiency without considering its cost and its subsequent influence on consistency maintenance. This paper studies the problem about how a server chooses files to replicate and where to replicate files to achieve low cost in both file replication and consistency maintenance stages without compromising the effectiveness of file replication. This paper presents a lightweight and Cooperative multifactOr considered file Replication Protocol (CORP) to achieve this goal. CORP simultaneously takes into account multiple factors including file popularity, update rate, node available capacity, file load, and node locality, aiming to minimize the number of replicas, update frequency, and replication distance. CORP also dynamically adjusts the number of replicas based on ever-changing file popularity and visit pattern. Extensive experimental results from simulation and PlanetLab real-world testbed demonstrate the efficiency and effectiveness of CORP in comparison with other file replication protocols. It dramatically reduces the overhead of both file replication and consistency maintenance. In addition, it exhibits high adaptiveness to skewed lookups and yields significant improvement in reducing overloaded nodes. Specifically, compared to the other replication protocols, CORP can reduce more than 71 percent of file replicas, 84 percent of overloaded nodes, 94 percent of consistency maintenance cost, and 72 percent of file replication and consistency maintenance latency.

Mobile Ad Hoc Networks (MANETs) use anonymous routing protocols that hide node identities and/or routes from outside observers in order to provide anonymity protection. However, existing anonymous routing protocols relying on either hop-by-hop encryption or redundant traffic, either generate high cost or cannot provide full anonymity protection to data sources, destinations, and routes. The high cost exacerbates the inherent resource constraint problem in MANETs especially in multimedia wireless applications. To offer high anonymity protection at a low cost, we propose an Anonymous Location-based Efficient Routing proTocol (ALERT). ALERT dynamically partitions the network field into zones and randomly chooses nodes in zones as intermediate relay nodes, which form a nontraceable anonymous route. In addition, it hides the data initiator/receiver among many initiators/receivers to strengthen source and destination anonymity protection. Thus, ALERT offers anonymity protection to sources, destinations, and routes. It also has strategies to effectively counter intersection and timing attacks. We theoretically analyze ALERT in terms of anonymity and efficiency. Experimental results exhibit consistency with the theoretical analysis, and show that ALERT achieves better route anonymity protection and lower cost compared to other anonymous routing protocols. Also, ALERT achieves comparable routing efficiency to the GPSR geographical routing protocol.

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

File consistency maintenance in P2P systems is a technique for maintaining consistency between files and their replicas. Most previous consistency maintenance methods depend on either message spreading or structure-based pushing. Message spreading generates high overhead due to a large amount of messages; structure-based pushing methods reduce this overhead. However, both approaches cannot guarantee that every replica node receives an update in churn, because replica nodes passively wait for updates. As opposed to push-based methods that are not effective in high-churn and low-resource P2P systems, polling is churn resilient and generates low overhead. However, it is faced with a number of challenges: 1) ensuring a limited inconsistency; 2) realizing polling in a distributed manner; 3) considering physical proximity in polling; and 4) leveraging polling to further reduce polling overhead. To handle these challenges, this paper introduces a poll-based distributed file consistency maintenance method called geographically aware wave (GeWave). GeWave further reduces update overhead, enhances the fidelity of file consistency, and takes proximity into account. Using adaptive polling in a dynamic structure, GeWave avoids redundant file updates and ensures that every node receives an update in a limited time period even in churn. Furthermore, it propagates updates between geographically close nodes in a distributed manner. Extensive experimental results from the PlanetLab real-world testbed demonstrate the efficiency and effectiveness of GeWave in comparison with other representative consistency maintenance schemes. It dramatically reduces the overhead and yields significant improvements on effectiveness, scalability, and churn resilience of previous file consistency maintenance methods.

Internet-based video streaming applications are becoming more and more popular, attracting millions of online viewers every day. The incredible growth of viewers, dynamics of participants, and high video quality of service (QoS) requirement pose scalability, availability, and low-latency challenges to peer-to-peer (P2P) live video streaming systems. Tree-based systems have low-delay but are vulnerable to churn, while mesh-based systems are churn-resilient but suffer from high delay and overhead. Also, both structures cannot make full utilization of the bandwidth in the system. To tackle the challenges, we propose a DHT-aided chunk-driven overlay (DCO). It introduces a scalable DHT ring structure into a mesh-based overlay to efficiently manage video stream sharing. DCO includes a two-layer hierarchical DHT-based infrastructure, a chunk sharing algorithm, and a video provider selection algorithm. It selects stable nodes to form a scalable DHT-based infrastructure. The nodes in the DHT serve as distributed matchmakers between video providers and requesters. To motivate stable nodes to serve as the DHT nodes, we introduce an incentive mechanism based on the game theory. Aided by DHT, DCO guarantees stream chunk availability and assigns to a chunk requester a provider among all available providers in the system so that stream chunks are transmitted along a dynamic tree with top-down decreasing node bandwidth. In this way, DCO takes full advantage of available bandwidth in the system and, at the same time, provides high scalability and low latency. Experimental results show the superior performance of DCO compared with mesh-based and tree-based systems, and the effectiveness of the incentive mechanism and provider selection algorithm.

Peer-to-peer networks (P2Ps) use reputation systems to provide incentives for nodes to offer high quality of service (QoS) and thwart the intentions of dishonest or selfish nodes. Existing reputation systems have two problems. First, they directly regard node reputation as trust. Rather, reputation represents the opinions formed by others about a node's QoS behavior, while trust represents a node's honesty and willingness to cooperate. In addition to trust, factors such as node capacity and lifetime also influence reputation. Due to these factors' heterogeneity and variance over time, reputation cannot directly reflect a node's trust or current QoS. Second, existing reputation systems guide a node to select the server with the highest reputation, which may not actually select the highest QoS server and would overload the highest reputed nodes. This work aims to accurately reflect node trust and provide guidance for high-QoS server selection. Through experimental study, we find that node trust, available capacity, and lifetime positively affect node reputation. Based on this observation, we first propose a manual trust model and an automatic trust model that remove the influence of additional factors on reputation to truly reflect node trust. We then propose a high-QoS server selection algorithm that separately considers node trust, current available capacity, and lifetime. Extensive simulation results demonstrate the effectiveness of the trust models in accurate node trust reflection compared with an existing reputation system. Moreover, the server selection algorithm dramatically increases the success rate of service requests and avoids overloading nodes.

Question and Answer (Q&A) websites such as Yahoo!Answers provide a platform where users can post questions and receive answers. These systems take advantage of the collective intelligence of users to find information. In this paper, we analyze the online social network (OSN) in Yahoo!Answers. Based on a large amount of our collected data, we studied the OSN's structural properties, which reveals strikingly distinct properties such as low link symmetry and weak correlation between indegree and outdegree. After studying the knowledge base and behaviors of the users, we find that a small number of top contributors answer most of the questions in the system. Also, each top contributor focuses on only a few knowledge categories. In addition, the knowledge categories of the users are highly clustered. We also study the knowledge base in a user's social network, which reveals that the members in a user's social network share only a few knowledge categories. Based on the findings, we provide guidance in the design of spammer detection algorithms and distributed Q&A systems. We also propose a friendship-knowledge oriented Q&A framework that synergically combines current OSN-based Q&A and web Q&A. We believe that the results presented in this paper are crucial in understanding the collective intelligence in the web Q&A OSNs and lay a cornerstone for the evolution of next-generation Q&A systems.

Wireless Sensor and Actuator Networks (WSANs) are composed of sensors and actuators to perform distributed sensing and actuating tasks. Most WSAN applications (e.g., fire detection) demand that actuators rapidly respond to events under observation. Therefore, real-time and fault-tolerant transmission is a critical requirement in WSANs to enable sensed data to reach actuators reliably and quickly. Due to limited power resources, energy-efficiency is another crucial requirement. Such requirements become formidably challenging in large-scale WSANs. However, existing WSANs fall short in meeting these requirements. To this end, we first theoretically study the Kautz graph for its applicability in WSANs to meet these requirements. We then propose a Kautz-based Real-time, Fault-tolerant and Energy-efficient WSAN (REFER). REFER has a protocol that embeds Kautz graphs into the physical topology of a WSAN for real-time communication and connects the graphs using Distributed Hash Table (DHT) for high scalability. We also theoretically study routing paths in the Kautz graph, based on which we develop an efficient fault-tolerant routing protocol. It enables a relay node to quickly and efficiently identify the next shortest path from itself to the destination only based on node IDs upon routing failure. REFER is advantageous over previous Kautz graph based works in that it does not need an energy-consuming protocol to find the next shortest path and it can maintain the consistency between the overlay and physical topology. Experimental results demonstrate the superior performance of REFER in comparison with existing systems in terms of real-time communication, energy-efficiency, fault-tolerance and scalability.

Previous Delay Tolerant Network (DTN) routing algorithms exploit either past encounter records (probabilistic routing) or social network properties (social network based routing) to derive a node's probability of delivering packets to their destinations. However, they only have a local view of the network, which limits the routing efficiency. Also, when two nodes meet, they have to exchange the delivery probabilities to the destinations of all packets in the two nodes, which incurs high resource consumption. In a social network, the people a person frequently meets are usually stable, which makes them play a more important role in forwarding message for the person. Based on this, we propose a lightweight distributed Social MAp based Routing algorithm in delay Tolerant networks (SMART). In SMART, each node builds its own social map consisting of nodes it has met and their frequently encountered nodes in a distributed manner. Based on both encountering frequency and social closeness of the two linked nodes in the social map, we decide the weight of each link to reflect the packet delivery probability between the two nodes. The social map enables more accurate forwarder selection through a broader view. Moreover, nodes exchange much less information for social map update and need fewer updates due to social map stability, which reduces resource consumption. Trace-driven experiments and tests on the GENI ORBIT testbed demonstrate the high efficiency of SMART in comparison with previous algorithms.

A peer-to-peer (P2P) file sharing system provides a platform that enables users to share their files. Retrieving files efficiently and trustworthily in such a large and jumbled system is critically important. However, the issues of efficient searching and trustworthy searching have only been studied separately. Simply combining two separate strategies dealing with each issue doubles system overhead. In this paper, we first study trace data from Facebook and BitTorrent. Guided by the study observations, we propose a P2P system based on social networks for simultaneous efficient and trustworthy file sharing, namely Social-P2P. Social-P2P groups common-multi-interest nodes into a cluster and further connects socially close nodes within a cluster. The comparably stable nodes in each cluster form a DHT for inter-cluster file searching. A file query is forwarded to the cluster of the file by the DHT routing and then is forwarded along constructed connections within a cluster, which achieves high hit rate and reliable routing. Sharing files among socially close friends discourages nodes from providing faulty files since people are unlikely to risk their reputation in the real world. Experimental results show that by leveraging a social network, Social-P2P achieves highly efficient and trustworthy file sharing.

The pay-as-you-go service model impels cloud customers to reduce the usage cost of bandwidth. Traffic Redundancy Elimination (TRE) has been shown to be an effective solution for reducing bandwidth costs, and has recently captured significant attention in the cloud environment. By studying the TRE techniques with a trace driven approach, we found that solely using either sender-based TRE or receiver-based TRE cannot simultaneously capture traffic redundancy in both short-term (time span of seconds) and long-term (time span of hours or days) data redundancy, which concurrently appear in the traffic. Additionally, the TRE efficiency of existing receiver-based TRE solution is susceptible to data changes compared to historical data in the cache. In this paper, we propose a sender and receiver Cooperative end-to-end TRE solution (CoRE) for efficiently identifying and removing both short-term and long-term redundancy. Through a two-layer redundancy detection design and one single pass algorithm for chunking and fingerprinting, CoRE efficiently carries out cooperative operations between the sender and the receiver. By extensive evaluation with several real traces, we show that CoRE is able to identify both short-term and longterm redundancy with low additional cost, while ensuring TRE efficiency from data changes.

In peer-to-peer networks (P2Ps), many autonomous nodes without preexisting trust relationships share resources (e.g., files) between each other. Due to their open environment, P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high system reliability and security. A reputation system computes and publishes reputation score for each node based on a collection of opinions from others about the node. However, collusion behaviors impair the effectiveness of reputation systems in trustworthy node selection. Though many reputation calculation methods have been proposed to mitigate collusion's influence, little effort has been devoted to specifically tackling collusion. In this paper, we analyze transaction ratings in the Amazon and Overstock online transaction platforms during one year. The analysis of real trace confirms the existence of collusion as well as its important behavior characteristics and influence on reputation values in real reputation systems. Accordingly, we propose a collusion detection method to specifically thwart collusion behaviors. We further optimize the method by reducing the computing cost. Experimental results show that the proposed method can significantly enhance the capability of existing reputation systems to deter collusion with low cost.

Recently, many schemes have been proposed for detecting and healing coverage holes to achieve full coverage in wireless sensor networks (WSNs). However, none of these schemes aim to find the shortest node movement paths to heal the coverage holes, which could significantly reduce energy usage for node movement. Also, current hole healing schemes require accurate knowledge of sensor locations, obtaining this knowledge consumes high energy. In this paper, we propose a DElaunay-based Coordinate-free Mechanism (DECM) for full coverage. Based on rigorous mathematical analysis, DECM can detect coverage holes and find the locally shortest paths for healing holes in a distributed manner without requiring accurate node location information. Simulation results and experimental results from the real-world GENI Orbit test bed show that DECM achieves superior performance in terms of the energy-efficiency and effectiveness of hole healing compared to previous schemes.

Video sharing has been an increasingly popular application in online social networks (OSNs). However, its sustainable development is severely hindered by the intrinsic limit of the client/server architecture deployed in current OSN video systems, which is not only costly in terms of server bandwidth and storage but also not scalable. The peer-assisted Video-on-Demand (VOD) technique, in which participating peers assist the server in delivering video content has been proposed recently. Unfortunately, videos can only be disseminated through friends in OSNs. Therefore, current VOD works that explore clustering nodes with similar interests or close location for high performance are suboptimal, if not entirely inapplicable, in OSNs. Based on our long-term real-world measurement of over 1,000,000 users and 2,500 videos in Facebook, we propose SocialTube, a novel peer-assisted video sharing system that explores social relationship, interest similarity, and physical location between peers in OSNs. Specifically, SocialTube incorporates three algorithms: a social network (SN)-based P2P overlay construction algorithm, a SN-based chunk prefetch algorithm, and a buffer management algorithm. The trace driven based simulation results show that SocialTube can improve the quality of user experience and system scalability over current P2P VOD techniques.

When monitoring environments with wireless sensor networks, optimal sensor deployment is a fundamental issue and an effective means to achieve desired performance. Selecting best sensor deployment has a dependence on the deployment environments. Existing works address sensor deployment within three types of environments including one dimensional line, 2-D field and 3-D space. However, in many applications the deployment environments usually have network structures, which cannot be simply classified as the three types. The deployed locations and communications of sensor nodes are limited onto the network edges, which make the deployment problem distinct from that in other types of environments. In this paper, we study sensor deployment in network-structured environments and aim to achieve k-coverage while minimizing the number of sensor nodes. Furthermore, we jointly consider the optimization of sink deployment and routing strategies with the goal to minimize the network communication cost of data collection. To the best of our knowledge, this paper is the first one to tackle sensor/sink deployment under the deployment constraints imposed by the network structure. The hardness of the problems is shown. Polynomial-time algorithms are proposed to determine optimal sensor/sink deployment and routing strategies in tree-topology network structure. Efficient approximation algorithms are proposed for the general graph network structure and their performances are analyzed. Theoretical results and extensive simulations show the efficiency of the proposed algorithms.

Wireless sensor networks(WSNs) are being widely deployed for many monitoring applications, of which a popular one is anti-theft. However, current WSN technologies for anti-theft are either very susceptible to the environment interference or easily compromised by the thieves. Hence, they fail to achieve the desired effectiveness in many anti-theft scenarios. To address these limitations, this paper proposes a novel Hidden Anti-theft System (HAS) to monitor theft intrusion, which is based on the influences of theft intrusion on signal strength according to the shadowing effect in wireless communication. The theft intrusion is detected by finding abnormal RSSI samples of a wireless link compared with the stable range of signal strength in the link's normal state. Through the proposed monitoring approach, HAS can effectively detect intrusion while keeping invisible. In HAS system, to achieve load balance for detection task, we propose an efficient algorithm to determine the set of links that each node monitors. To reduce the response time, a dual-layer scanning solution with dual-radio nodes is proposed to scan the monitoring area more intensively. The experiment results show that the efficiency of HAS. HAS achieves very low false positive and false negative rates, and the response time of dual-layer scanning is 54.2% less than single layer scanning.

In current peer-to-peer (P2P) live streaming systems, nodes in a channel form a P2P overlay for video sharing. To watch a new channel, a node depends on the centralized server to join in the overlay of the channel. The increase in the number of channels in today's live streaming applications triggers users' desire of watching multiple channels successively or simultaneously. However, the support of such watching modes in current applications is no better than joining in different channel overlays successively or simultaneously, which if widely used, poses heavy burden on the centralized server. In order to achieve higher efficiency and scalability, we propose a Social network-Aided efficient liVe strEaming system (SAVE). SAVE regards users' channel switching or multi-channel watching as interactions between channels. By collecting the information of channel interactions and nodes' interests and watching times, SAVE forms nodes in multiple channels with frequent interactions into an overlay, constructs bridges between overlays of channels with less frequent interactions, and enables nodes to identify friends sharing similar interests and watching times. Thus, a node can connect to a new channel while staying in its current overlay, using bridges or relying on its friends, reducing the need to contact the centralized server. Extensive experimental results from the PeerSim simulator and PlanetLab verify that SAVE outperforms other popular protocols in system efficiency and server load reduction.

Efficient and trustworthy file querying is important to the overall performance of peer-to-peer (P2P) file sharing systems. Emerging methods are beginning to address this challenge by exploiting online social networks (OSNs). However, current OSN-based methods simply cluster common-interest nodes for high efficiency or limit the interaction between social friends for high trustworthiness, which provides limited enhancement or contradicts the open and free service goal of P2P systems. Little research has been undertaken to fully and cooperatively leverage OSNs with integrated consideration of proximity and interest. In this work, we analyze a BitTorrent file sharing trace, which proves the necessity of proximityand interest-aware clustering. Based on the trace study and OSN properties, we propose a SOcial Network integrated P2P file sharing system with enhanced Efficiency and Trustworthiness (SoNet) to fully and cooperatively leverage the common-interest, proximity-close and trust properties of OSN friends. SoNet uses a hierarchical distributed hash table (DHT) to cluster common-interest nodes, then further cluster proximity-close nodes into subcluster, and connects the nodes in a subcluster with social links. Thus, when queries travel along trustable social links, they also gain higher probability of being successfully resolved by proximityclose nodes, simultaneously enhancing efficiency and trustworthiness. The results of trace-driven experiments on the realworld PlanetLab testbed demonstrate the higher efficiency and trustworthiness of SoNet compared with other systems.

A hybrid wireless network combines a mobile ad-hoc network and an infrastructure network. Efficient and reliable data routing is important for high throughput in such networks. Existing routing schemes that simply combine ad-hoc and infrastructure routings inherit the drawbacks of ad-hoc routing and fail to take advantage of the infrastructure for high efficiency. Current reputation systems relying on local information exchange are not sufficiently effective and efficient in guiding reliable routing. This paper presents a peer-to-peer (P2P)-based Market-guided Distributed Routing mechanism (MDR) to increase the throughput of hybrid networks by achieving a high efficiency and reliability. Taking advantage of the high density of base stations, the packets from a source node are distributively transmitted to base stations directly or indirectly. The packet transmission in MDR is modeled as a market trading behaviors, in which source nodes pay credits to relay nodes. The service price is determined by the supply and demand equilibrium of the nodes in the system. MDR organizes base stations into a P2P structure to facilitate high efficient data operation for service price determination. An erasure coding-based distributed routing algorithm is also proposed to facilitate an efficient and reliable market trading. Theoretical analysis demonstrates the distinguishing features of MDR and simulation results show that MDR outperforms the traditional hybrid routing schemes and reputation systems.

In large-scale computational Grids, discovery of heterogeneous resources as a working group is crucial to achieving scalable performance. This paper presents a resource management scheme including a hierarchical cycloid overlay architecture, resource clustering and discovery algorithms for wide-area distributed Grid systems. We establish program/data locality by clustering resources based on their physical proximity and functional matching with user applications. We further develop dynamism-resilient resource management algorithm, cluster-token forwarding algorithm, and deadline-driven resource management algorithms. The advantage of the proposed scheme lies in low overhead, fast and dynamism-resilient multiresource discovery. The paper presents the scheme, new performance metrics, and experimental simulation results. This scheme compares favorably with other resource discovery methods in static and dynamic Grid applications. In particular, it supports efficient resource clustering, reduces communications cost, and enhances resource discovery success rate in promoting large-scale distributed supercomputing applications.

Resource discovery is critical to the usability and accessibility of grid computing systems. Distributed Hash Table (DHT) has been applied to grid systems as a distributed mechanism for providing scalable range-query and multi-attribute resource discovery. Multi-DHT-based approaches depend on multiple DHT networks with each network responsible for a single attribute. Single-DHT-based approaches keep the resource information of all attributes in a single node. Both classes of approaches lead to high overhead. In this paper, we propose a Low-Overhead Range-query Multi-attribute (LORM) DHT-based resource discovery approach. Unlike other DHT-based approaches, LORM relies on a single compound graph-based DHT network and distributes resource information among nodes in balance by taking advantage of the compound graph structure. Moreover, it has high capability to handle the large-scale and dynamic characteristics of resources in grids. Experimental results demonstrate the efficiency of LORM in comparison with other resource discovery approaches. LORM dramatically reduces maintenance and resource discovery overhead. In addition, it yields significant improvements in resource location efficiency. We also analyze the performance of the LORM approach rigorously by comparing it with other multi-DHT-based and single-DHT-based approaches with respect to their overhead and efficiency. The analytical results are consistent with experimental results, and prove the superiority of the LORM approach in theory.

In mobile ad hoc networks (MANETs), tasks are conducted based on the cooperation of nodes in the networks. However, since the nodes are usually constrained by limited computation resources, selfish nodes may refuse to be cooperative. Reputation systems and price-based systems are two main solutions to the node noncooperation problem. A reputation system evaluates node behaviors by reputation values and uses a reputation threshold to distinguish trustworthy nodes and untrustworthy nodes. A price-based system uses virtual cash to control the transactions of a packet forwarding service. Although these two kinds of systems have been widely used, very little research has been devoted to investigating the effectiveness of the node cooperation incentives provided by the systems. In this paper, we use game theory to analyze the cooperation incentives provided by these two systems and by a system with no cooperation incentive strategy. We find that the strategies of using a threshold to determine the trustworthiness of a node in the reputation system and of rewarding cooperative nodes in the price-based system may be manipulated by clever or wealthy but selfish nodes. Illumined by the investigation results, we propose and study an integrated system. Theoretical and simulation results show the superiority of the integrated system over an individual reputation system and a price-based system in terms of the effectiveness of cooperation incentives and selfish node detection.

Online forums have long since been the most popular platform for people to communicate and share ideas. Nowadays, with the boom of multimedia sharing, users tend to share more and more with their online peers within online communities such as forums. The server-client model of forums has been used since its creation in the mid-1990s. However, this model has begun to fall short in meeting the increasing need of bandwidth and storage resources as an increasing number of people share more and more multimedia content. In this work, we first investigate the unique properties of forums based on the data collected from the Disney discussion boards. According to these properties, we design a scheme to support P2P-based multimedia sharing in forums called Multimedia Board (MBoard). Extensive trace-driven simulation results utilizing real trace data show that MBoard can significantly reduce the load on the server while maintaining a high quality of service for the users.

One critical function of cyber-physical systems (CPS) is object search in the physical world through the cyber sphere that enables interaction between the cyber and physical spheres. Some of the previously proposed physical object search engines use RFID tracking, and others collect the information of object locations into a hierarchical centralized server. The difficulty of widely deploying RFID devices, the centralized search, and the need for periodical location information collection prevent CPS from achieving higher scalability and efficiency. To deal with this problem, we propose a Social-aware distributed Cyber-Physical human-centric Search engine (SCPS) that leverages the social network formed by wireless device users for object search. Without requiring periodical location information collection, SCPS locates objects held by users based on the routine user movement pattern. Moreover, using a social-aware Bayesian network, it can accurately predict the users' locations when exceptional events (e.g., inclement weather) occur, which breaks user movement pattern. Thus, SCPS is more advantageous than all previous social network based works which assume that user behaviors always follow a certain pattern. Further, SCPS conducts the search in a fully distributed manner by relying on a DHT structure. As a result, SCPS achieves high scalability, efficiency and location accuracy. Extensive real-trace driven simulation results show the superior performance of SCPS compared to other representative search methods.

Advancements in technology over the past decade are leading to a promising future for large-scale distributed systems, where globally-scattered distributed resources are collectively pooled and used in a cooperative manner to achieve unprecedented petascale supercomputing capabilities. The issues of resource management (resMgt) and reputation management (repMgt) need to be addressed in order to ensure the successful deployment of large-scale distributed systems. However, these two issues have typically been addressed separately, despite the significant interdependencies between them: resMgt needs repMgt to provide a cooperative environment for resource sharing, and in turn facilitates repMgt to evaluate multi-faceted node reputations for providing different resources. Current repMgt methods provide a single reputation value for each node in providing all types of resources. However, a node willing to provide one resource may not be willing to provide another resource. In addition, current repMgt methods often guide node selection policy to select the highest-reputed nodes, which may overload these nodes. Also, few works exploited node reputation in resource selection in order to fully and fairly utilize resources in the system and to meet users' diverse QoS demands. We propose a system called Harmony that integrates resMgt and repMgt in a harmonious manner. Harmony incorporates two key innovations: integrated multi-faceted resource/reputation management and multi-QoS-oriented resource selection. The trace data we collected from an online trading platform confirms the importance of multi-faceted reputation and potential problems with highest-reputed node selection. Trace-driven experiments performed on PlanetLab show that Harmony outperforms existing resMgt and repMgt in terms of the success rate, service delay, and efficiency.

Peer-to-peer networks (P2Ps) use reputation systems to provide incentives for nodes to offer high quality of service (QoS) and thwart the intentions of dishonest or selfish nodes. Existing reputation systems have two problems. First, they directly regard node reputation as trust. Reputation represents the opinion formed by others about a node's QoS behavior, while trust represents a node's honesty and willingness to cooperate. In addition to trust, factors such as node capacity and lifetime also influence reputation. Due to the heterogeneous and time-varying attributes of these factors, reputation actually cannot directly reflect a node's trust or current QoS. Second, existing reputation systems guide a node to select the server with the highest reputation, which may not actually select the highest QoS server and would overload the highest-reputed nodes. This work aims to accurately reflect node trust and provide accurate guidance for high QoS server selection. Through experimental study, we find that node trust, available capacity and lifetime positively affect node reputation. Based on this observation, we first propose a manual and an automatic trust model that removes the influence of additional factors on reputation to truly reflect node trust. We then propose a high-QoS server selection algorithm that separately considers node trust, current available capacity, and lifetime. Simulation results demonstrate the effectiveness of the trust models in accurate node trust reflection. Moreover, the server selection algorithm dramatically increases the success rate of service requests and avoids overloading nodes.

File sharing applications in mobile ad hoc networks (MANETs) have attracted more and more attention in recent years. The efficiency of file querying suffers from the distinctive properties of MANETs including node mobility and limited communication range and resource. An intuitive method to alleviate this problem is to create file replicas in the network. However, despite the efforts on file replication, no research has focused on the global optimal replica sharing with minimum average querying delay. Specifically, current file replication protocols in MANETs have two shortcomings. First, they lack a rule to allocate limited resource to different files in order to minimize the average querying delay. Second, they simply consider storage as resource for replicas, but neglect the fact that the file holders' frequency of meeting other nodes also plays an important role in determining file availability. A node having a higher meeting frequency with others provides higher availability to its files. In this paper, we introduce a new concept of resource for file replication, which considers both node storage and meeting frequency. We theoretically study the influence of resource allocation on the average querying delay and derive a resource allocation rule to minimize the average querying delay. We further propose a distributed file replication protocol that follows the rule. The trace-driven experiments on both the real-world GENI testbed and NS-2 show that our protocol can achieve shorter average querying delay at lower cost than current replication protocols, which justifies the correctness of our theoretical analysis and the effectiveness of the proposed protocol.

Mobile Ad Hoc Networks (MANETs) use anonymous routing protocols that hide node identities and/or routes from outside observers in order to provide anonymity protection. However, existing anonymous routing protocols, relying on either hop-by-hop encryption or redundant traffic, either generate high cost or cannot provide full anonymity protection to data sources, destinations, and routes. To offer high anonymity protection at a low cost, we propose an Anonymous Location-based Efficient Routing proTocol (ALERT). ALERT dynamically partitions the network field into zones and randomly chooses nodes in zones as intermediate relay nodes, which form a non-traceable anonymous route. In addition, it hides the data initiator/receiver among many initiators/receivers. Thus, ALERT offers anonymity protection to sources, destinations, and routes. It also has strategies to effectively counter intersection and timing attacks. Experimental results show that ALERT achieves better route anonymity protection and lower cost compared to other anonymous routing protocols. Also, ALERT achieves comparable routing efficiency to the GPSR geographical routing protocol.

Encouraging cooperative and deterring selfish behaviors are important for proper operations of MANETs. For this purpose, most previous efforts either rely on reputation systems or price systems. However, both systems are neither sufficiently effective in providing cooperation incentives nor efficient in resource consumption. Nodes in both systems can be uncooperative while still being considered trustworthy. Also, information exchange between mobile nodes in reputation systems and credit circulation in price systems consume significant resources. This paper presents a hierarchical Account-aided Reputation Management system (ARM) to efficiently and effectively provide cooperation incentives. ARM builds a hierarchical locality-aware DHT infrastructure for efficient and integrated operations of both reputation and price systems. The infrastructure helps to globally collect all reputation information in the system, which helps to calculate more accurate reputation and detect abnormal reputation information. Also, ARM coordinately integrates resource and price systems by enabling higher-reputed nodes to pay less for their received services. Theoretical analysis demonstrates the properties of ARM. Simulation results show that ARM outperforms both a reputation system and price system in terms of effectiveness and efficiency.

The explosive growth of unsolicited emails has prompted the development of numerous spam filtering techniques. A Bayesian spam filter is superior to a static keywordbased spam filter because it can continuously evolve to tackle new spam by learning keywords in new spam emails. However, Bayesian spam filters can be easily poisoned by avoiding spam keywords and adding many innocuous keywords in the emails. In addition, they need a significant amount of time to adapt to a new spam based on user feedback. Moreover, few current spam filters exploit social networks to assist spam detection. In order to develop an accurate and user-friendly spam filter, in this paper, we propose a SOcial network Aided Personalized and effective spam filter (SOAP). Unlike previous filters that focus on parsing keywords (e.g, Bayesian filter) or building blacklists, SOAP exploits the social relationship among email correspondents to detect the spam adaptively and automatically. SOAP integrates three components into the basic Bayesian filter: social closeness-based spam filtering, social interest-based spam filtering, and adaptive trust management. We evaluate performance of SOAP based on the trace data from Facebook. Experimental results show that SOAP can greatly improve the performance of Bayesian spam filters in terms of the accuracy, attack-resilience and efficiency of spam detection. We also find that the performance of Bayesian spam filters is the lower bound of SOAP.

Online forums have long since been the most popular platform for people to communicate and share ideas. Nowadays, with the boom of multimedia sharing, users tend to share more and more with their online peers within online communities such as forums. The server-client model of forums has been used since its creation in the mid-nineties. However, this model has begun to fall short in meeting the increasing need of bandwidth and storage as an increasing number of people share more and more multimedia content. In this work, we first investigate the unique properties of forums based on the data collected from the Disney discussion boards. According to these properties, we design a scheme to support P2P-based multimedia sharing in forums called Multimedia Board (MBoard). Extensive simulation results utilizing real trace data show that MBoard can significantly reduce the load on the server while maintaining a high quality of service for the users.

In peer-to-peer networks (P2Ps), many autonomous peers without preexisting trust relationships share resources with each other. Due to their open environment, the P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high reliability and security. However, node collusion impairs the effectiveness of reputation systems in trustworthy node selection. Although some reputation systems have certain mechanisms to counter collusion, the effectiveness of the mechanisms is not sufficiently high. In this paper, we leverage social networks to enhance the capability of reputation systems in combating collusion. We first analyzed real trace of the reputation system in the Overstock online auction platform which incorporates a social network. The analysis reveals the important impact of the social network on user purchasing and reputation rating patterns. We thus identified suspicious collusion behavior patterns and propose a social network based mechanism, namely Social Trust, to counter collusion. Social Trust adaptively adjusts the weight of ratings based on the social distance and interest relationship between peers. Experimental results show that Social Trust can significantly strengthen the capability of current reputation systems in combating collusion.

Current P2P file sharing methods in mobile ad hoc networks (MANETs) can be classified into three groups: flooding-based, advertisement-based and social contact-based. The first two groups of methods can easily generate high overhead and low scalability, and the third group fails to consider the social interests (content) of mobile nodes, which otherwise can improve file searching efficiency. In this paper, we propose a P2P content-based file sharing system for MANETs. The system uses an interest extraction algorithm to derive a node's interests from its files for complex queries. For efficient file searching, it groups common-interest nodes that frequently meet with each other as communities. Further, it takes advantage of node mobility by designating stable nodes, which has frequent contact with community members, as community coordinators for intra-community searching, and highly-mobile nodes as community ambassadors for inter-community searching. An interest-oriented file searching scheme further enhances the file searching success rate. We first deployed our system on the real-world GENI Orbit testbed with a real trace and then conducted experiment on the ns2 simulator with both real trace and simulated disconnected and connected MANET scenario. The test results show that our system significantly lowers transmission cost and improves file searching success rate compared to current methods.

Since centralized data storage and search schemes often lead to high overhead and latency, distributed data-centric storage becomes a preferable approach in large-scale wireless sensor networks (WSNs). However, most of existing distributed methods lack optimization for spatial-temporal search to query events occurred in a certain geographical area and a certain time period. Furthermore, for data search routing, most methods rely on locating systems (e.g., GPS), which consume high energy. This paper proposes a distributed spatial-temporal Similarity Data Storage (SDS) scheme. SDS provides efficient spatial-temporal and similarity data searching service, and is applicable for both static and dynamic WSNs. It disseminates event data in such a way that the distance between WSN neighborhoods represents the similarity of data stored in them. In addition, SDS carpooling routing algorithm efficiently routes messages without the aid of GPS. Theoretical and experimental results show that SDS yields significant improvements on the efficiency of data querying compared with existing approaches, and obtains stable performance in dynamic environments.

Internet-based video streaming applications is becoming more and more popular and attract millions of online viewers every day. The incredible growth of viewers, dynamics of participants, and high video quality-of-service (QoS) requirement pose scalability, availability and low-latency challenges to peer-to-peer (P2P) live video streaming systems. Tree-based systems have low-delay but are vulnerable to churn, while mesh-based systems are churn-resilient but suffers high delay and overhead. Both systems cannot make full utilization of the bandwidth in the system. To tackle the challenges, we propose a DHT-aided Chunk-driven Overlay (DCO). It introduces a scalable DHT ring structure into a mesh-based overlay to efficiently manage video stream sharing. DCO includes a hierarchical DHT-based infrastructure and a chunk sharing algorithm. Aided by DHT, DCO guarantees stream chunk availability. In this way, DCO flexibly takes full advantage of available bandwidth in the system and at the same time provides high scalability and low-latency. Simulation results show the superiority of DCO compared with mesh-based and tree-based systems.

As wireless communication gains popularity, significant research has been devoted to supporting realtime transmission with stringent Quality of Service (QoS) requirements for wireless applications. At the same time, a wireless hybrid network that integrates a mobile wireless ad hoc network (MANET) and a wireless infrastructure network has been proven to be a better alternative for the next generation wireless networks. By directly adopting resource reservation-based QoS routing for MANETs, hybrids networks inherit invalid reservation and race condition problems in MANETs. How to guarantee the QoS in hybrid networks remains as an open problem. In this paper, we propose a QoS-oriented Distributed routing protocol (QOD) to enhance the QoS support capability of hybrid networks. Taking advantage of fewer transmission hops and anycast transmission features of the networks, QOD transforms the packet routing problem to a resource scheduling problem. QOD includes a QoS-guaranteed neighbor selection algorithm to meet the transmission delay requirement, a distributed packet scheduling algorithm to further reduce transmission delay, and mobility-based segment resizing algorithm that adaptively adjusts segment size according to node mobility in order to reduce transmission time. Analytical results show the QOD's properties of lower transmission delay and dynamism-resilience. Simulation results show that QOD can provide high QoS performance in terms of overhead, transmission delay, dynamism-resilience and scalability compared to a resource reservation-based mechanism.

In peer-to-peer file sharing systems, file replication and consistency maintenance are widely used techniques for high system performance. Despite significant interdependencies between them, these two issues are typically addressed separately. Most file replication methods rigidly specify replica nodes, leading to low replica utilization, unnecessary replicas and hence extra consistency maintenance overhead. Most consistency maintenance methods propagate update messages based on message spreading or a structure without considering file replication dynamism, leading to inefficient file update and hence high possibility of outdated file response. This paper presents an Integrated file Replication and consistency Maintenance mechanism (IRM) that integrates the two techniques in a systematic and harmonized manner. It achieves high efficiency in file replication and consistency maintenance at a significantly low cost. Instead of passively accepting replicas and updates, each node determines file replication and update polling by dynamically adapting to time-varying file query and update rates, which avoids unnecessary file replications and updates. Simulation results demonstrate the effectiveness of IRM in comparison with other approaches. It dramatically reduces overhead and yields significant improvements on the efficiency of both file replication and consistency maintenance approaches.

In peer-to-peer file sharing systems, file replication technology is widely used to reduce hot spots and improve file query efficiency. Most current file replication methods replicate files in all nodes or two end points on a client-server query path. However, these methods either have low effectiveness or come at a cost of high overhead. File replication in server side enhances replica hit rate, hence, lookup efficiency but produces overloaded nodes and cannot significantly reduce query path length. File replication in client side could greatly reduce query path length, but cannot guarantee high replica hit rate to fully utilize replicas. Though replication along query path solves these problems, it comes at a high cost of overhead due to more replicas and produces underutilized replicas. This paper presents an Efficient and Adaptive Decentralized (EAD) file replication algorithm that achieves high query efficiency and high replica utilization at a significantly low cost. EAD enhances the utilization of file replicas by selecting query traffic hubs and frequent requesters as replica nodes, and dynamically adapting to nonuniform and time-varying file popularity and node interest. Unlike current methods, EAD creates and deletes replicas in a decentralized self-adaptive manner while guarantees high replica utilization. Theoretical analysis shows the high performance of EAD. Simulation results demonstrate the efficiency and effectiveness of EAD in comparison with other approaches in both static and dynamic environments. It dramatically reduces the overhead of file replication, and yields significant improvements on the efficiency and effectiveness of file replication in terms of query efficiency, replica hit rate, and overloaded nodes reduction.

Consistent hashing-based DHT networks have an inherent load balancing problem. The problem becomes more severe in heterogeneous networks with nonuniform and time-varying popular files. Existing DHT load balancing algorithms are mainly focused on the issues caused by node heterogeneity. To deal with skewed lookups, this paper presents an elastic routing table (ERT) mechanism for query load balancing, based on the observation that high-degree nodes tend to receive more traffic load. The mechanism allows each node to have a routing table of variable size corresponding to node capacities. The indegree and outdegree of the routing table can also be adjusted dynamically in response to the change of file popularity and network churn. Theoretical analysis proves that the routing table degree is bounded. The ERT mechanism facilitates locality-aware randomized query forwarding to further improve lookup efficiency. By relating query forwarding to a supermarket customer service model, we prove that a two-way randomized query forwarding policy should lead to an exponential improvement in query processing time over random walking. Simulation results demonstrate the effectiveness of the ERT mechanism and its related query forwarding policy for congestion and query load balancing. In comparison with existing "virtual-server¿-based load balancing algorithms and other routing table control approaches, the ERT-based congestion control protocol yields significant improvement in query lookup efficiency.

Efficient file query is important to the overall performance of peer-to-peer (P2P) file sharing systems. Clustering peers by their common interests can significantly enhance the efficiency of file query. On the other hand, clustering peers by their physical proximity can also improve file query performance. Few current works are able to cluster peers based on both peer interest and physical proximity. It is even harder to realize it in structured P2Ps due to their strictly defined topologies, although they provide higher file query efficiency than unstructured P2Ps. In this paper, we introduce a proximity-aware and interest-clustered P2P file sharing system (PAIS) based on a structured P2P. It groups peers based on both interest and proximity. PAIS supports sophisticated routing and clustering strategies based on a hierarchical topology. Theoretical analysis and simulation results demonstrate that PAIS dramatically reduces the overhead and enhances efficiency in file sharing.

Efficient file query is important to the overall performance of Peer-to-Peer (P2P) file sharing systems. In this paper, we introduce an interest-based P2P file sharing system based on a structured P2P. It groups peers based on both interest and proximity. The proposed system is able to support sophisticated routing and clustering strategies based on the file attribute and topology used. Simulation results demonstrate the efficiency of the proposed system in comparison with another P2P file sharing system. It dramatically reduces the overhead and yields significant improvements in file query efficiency.

Wireless sensor networks can monitor physical and environmental conditions inexpensively, and the networks have been widely used in military and civilian applications. The sensor nodes in the networks not only have the ability to sense, but also can supply data storage, computation and communicate with each other. However, energy efficiency is the major concern in wireless sensor networks, which usually rely on battery for energy source. Centralized data storage and query processing schemes lead to high energy consumption. This paper proposes a distributed data storage scheme (DDS) that disseminates related event data in a neighborhood in wireless sensor networks. Besides offering data query according to event names, DDS also provides distributed querying. Experimental results show that the DDS scheme yields higher efficiency and effectiveness than the centralized scheme.

File replication and file consistency maintenance are two important protocols for achieving overall high performance of peer-to-peer file sharing systems. In spite of the significant dependency of consistency maintenance on file replication, there has been little file replication research devoted to tackling the dependency issue. Most traditional file replication protocols focus on improving file lookup efficiency without considering the subsequent consistency maintenance. This paper presents a COoperative file Replication Protocol (CORP). CORP observes a set of factors including file update rate, file popularity, node available capacity, file load and node locality that affect the efficiency of consistency maintenance. It considers these factors to achieve high efficiency not only in file replication but also in consistency maintenance. CORP generates less replicas and optimally chooses infrequently-updated or popular files to replicate. In addition, it makes file replicas in physically close nodes. Further, CORP can dynamically adjust the number of replicas based on ever-changing file popularity and visit pattern. Simulation results demonstrate the efficiency and effectiveness of CORP in comparison with other file replication protocols. It dramatically reduces the overhead of both file replication and consistency maintenance. Moreover, it yields significant improvement in reducing overloaded nodes.

One primary goal in P2P networks is to provide high search performance for users to retrieve interested documents distributed over nodes. Document indexing is the key to search performance. However, it is challenging to guarantee high search performance with small document index. In this paper, we present iSearch which aims to build small document index to deliver high search performance on Gnutella-like P2P networks. The number of index terms per document is typically 4, which dramatically reduces associated cost in index storage and dissemination. iSearch explores two options to build index: top term-based indexing (TTI) and query-driven indexing (QDI). TTI bases selection of document index terms on term statistics, while QDI progressively refines document index by past queries. Our simulations show that that TTI and QDI improve search performance over random walk significantly. By dynamically adapting index based on past queries, QDI outperforms TTI greatly, by up to 2× recall improvement.

In cooperative systems such as wireless ad hoc networks, tasks are conducted based on the cooperation of nodes in the system. However, selfish nodes may refuse to be cooperative. Reputation system and price-based system are two main solutions to this problem. In this paper, we use game theory to investigate the underlying cooperation incentives of both systems. We found that the strategies of using a threshold to determine the trustworthiness of a node in the reputation system and of rewarding cooperative nodes in the price-based system may be manipulated by clever selfish nodes to gain benefits while still being selfish. Illumined by the investigation results, an integrated system is proposed and studied. Theoretical and simulation results show the superiority of the integrated system over individual reputation system and price-based system in terms of the effectiveness of cooperation incentives.

Since centralized data storage and search schemes often lead to high overhead and latency, distributed data centric storage becomes a preferable approach in large-scale WSNs. However, most of existing methods lack optimization for spatial- temporal search and similarity search for multi-attribute data. Some methods are optimized under circumstances where nodes are equipped with locating systems (e.g., GPS) which consumes high energy. This paper proposes a distributed spatial-temporal similarity data storage scheme (SDS). It disseminates event data in such a way that the distance between WSN neighborhoods represents the similarity of data stored in them. In addition, SDS carpooling routing algorithm efficiently routes messages without the aid of a locating system. SDS provides efficient spatial- temporal and similarity data searching service. Experimental results show that SDS yields significant improvements on the efficiency of data querying compared with existing approaches.

Peer-to-peer file sharing applications are increasingly popular and they enable users to share various files with high scalability. However, in intermittently connected mobile network environment, existing file sharing systems are hindered by the lack of node stable connectivity and constant network topology change. We propose a social network based file sharing system in mobile P2P networks. It takes advantage of node mobility and builds communities based on social network. Instead of being hindered by the mobility of the nodes, the proposed system utilizes the node mobility to facilitate communications and provide efficient message routing and file retrieval services. Experiment results show that the proposed system significantly lowers the transmission cost and achieves satisfactory file retrieval success rate.

With the wide use of mobile devices in mobile ad hoc networks, maintaining anonymity is becoming an increasingly important issue. Existing routing algorithms either rely on hop-by-hop encryption or local broadcasting for anonymous routing, which lead to high overhead. We propose a low-cost anonymous routing algorithm, which can provide both source/destination and routing anonymity. It dynamically divides the network into hierarchical zones to obscure routing path and randomly chooses nodes as intermediate relay nodes. Therefore it splits routing path to multiple steps that contains no specific routing pattern. Furthermore, we present analysis of the ability of our algorithm to withstand certain attacks. Experiment results show that our algorithm trades some routing efficiency for anonymity, but gains better delivery rate than GPSR under short transmission range.

The blind flooding algorithm under a time-to-live (TTL) constraint used in unstructured peer-to-peer (P2P) networks, such as Gnutella, may cause a large amount of network traffic. The algorithm cannot guarantee for acquiring the required data objects, especially for rare ones. To mitigate these problems, this paper proposes PercolationNET, a multi-tree sub-overlay, which is built on top of an existing P2P overlay (named original overlay). PercolationNET organizes peers in a tree-based structure which facilitates reliable and efficient message dissemination for search. The search process is divided into two stages. A query message is first propagated on the original overlay, and then broadcast along the sub-overlay PercolationNET. PercolationNET combines the advantages of fast coverage speed in flooding-based scheme and low traffic cost in tree-based scheme. The experimental results of PercolationNET compared with FloodNet confirm the superiority of PercolationNET in achieving faster coverage speed and lower message cost.

In large-scale computational or P2P grids, discovery of heterogeneous resources as a working group is crucial to achieving scalable performance. This paper presents a hierarchical cycloid overlay (HCO) architecture with resource clustering and discovery algorithms for efficient and robust resource discovery in wide-area distributed grid systems. We establish program/data locality by clustering resources based on their physical proximity and functional matching with user applications. We further develop randomized probing and cluster-token forwarding algorithms. The novelty of the HCO scheme lies in low overhead, fast speed and dynamism resilience in multi-resource discovery. The paper presents the HCO framework, new performance metrics, and simulation experimental results. This HCO scheme compares favorably with other resource management methods in static and dynamic grid applications. In particular, it supports efficient resource clustering, reduces communications cost, and enhances resource discovery success rate in promoting large-scale distributed supercomputing applications.

Hybrid wireless networks combining the advantages of both ad-hoc networks and infrastructure wireless networks have been receiving increasingly attentions because of their ultra-high performance. An efficient data routing protocol is an important component in such networks for high capacity and scalability. However, most routing protocols for the networks simply combine an ad-hoc transmission mode and a cellular transmission mode, which fail to take advantage of the dual-feature architecture. This paper presents a distributed Three-hop Routing (DTR) protocol for hybrid wireless networks. DTR divides a message data stream into segments and transmits the segments in a distributed manner. It makes full spatial reuse of system via high speed ad-hoc interface and alleviate mobile gateway congestion via cellular interface. Furthermore, sending segments to a number of base stations simultaneously increases the throughput, and makes full use of wide-spread base stations. In addition, DTR significantly reduces overhead due to short path length and eliminates route discovery and maintenance overhead. Theoretical analysis and simulation results show the superiority of DTR in comparison with other routing protocols in terms of throughput capacity, scalability and mobility resilience.

Resource discovery is critical to the usability and accessibility of grid computing systems. Distributed hash table (DHT) has been applied to grid systems as a distributed mechanism for providing scalable range-query and multiattribute resource discovery. Multi-DHT-based approaches depend on multiple DHT networks with each network responsible for a single attribute. Single-DHT-based approaches keep the resource information of all attributes in a single node. Both classes of approaches lead to high overhead. Recently, we proposed a heuristic Low-Overhead Range-query Multiattribute DHT-based resource discovery approach (LORM). It relies on a single hierarchical DHT network and distributes resource information among nodes in balance by taking advantage of the hierarchical structure. We demonstrated its effectiveness and efficiency via simulation. In this paper, we analyze the performance of the LORM approach rigorously by comparing it with other multi-DHT-based and single-DHT-based approaches with respect to their overhead and efficiency. The analytical results are consistent with simulation results. The results prove the superiority of the LORM approach in theory.

Data sharing is one of the most popular applications that dominate 70% of data traffic on the Internet. This application has been penetrating wireless mobile networks with dramatic speed, which allows the sharing of data whenever and wherever. However, traditional client-server data sharing model suffers from single point failure and low-scalable transmission in a highly dense and dynamic scenario. MANET is a promising alternative structure for flexible and distributed data sharing. However, the topology-based flooding employed in MANETs for data routing and querying prevents the data sharing system from achieving high scalability and robustness. Building a DHT on a MANET reduces data query overhead, but the inconsistency between the overlay and the underlying topology degrade system's performance. Geographic routing based data sharing in MANETs reduces the data routing overhead. However, it has more requirements (e.g. GPS) on the nodes. This paper presents a locality-based distributed data sharing system (LORD) for mobility and congestion resilient data management. LORD consists of a DHT-based data index and retrieval protocol and a locality-aware region-based data routing protocol. It provides highly efficient, scalable and dynamism-resilient data sharing with low overhead. Moreover, it offers similarity data searching function. Theoretical analysis and simulation results show the superiority of LORD compared with other data sharing systems in terms of scalability, overhead and dynamism-resilience in a highly dense and dynamic MANET.

In peer-to-peer file sharing systems, file replication helps to avoid overloading file owners and improve file query efficiency. Aiming to achieve high replica utilization and efficient file query with low overhead, this paper presents a file replication mechanism based on swarm intelligence, namely SWARM. Recognizing the power of collective behaviors, SWARM identifies node swarms with common node interests and close proximity. It determines the location of a file replica based on the accumulated query rates of nodes in a swarm. Replicas are shared by the nodes in a swarm, leading to less replicas and high query efficiency. Simulation results demonstrate the efficiency and effectiveness of the SWARM mechanism in comparison with other file replication methods.

In delay tolerant network, a complete routing path from a source to a destination can not be guaranteed at most of the time. Therefore, traditional routing method for ad hoc network is not applicable in these situations. Current approaches for such networks are primarily based on redundant transmissions and single copy direct routing. However, they incur either high overhead due to excessive transmissions or long delays due to the incorrect path choices during forwarding. In this paper, we propose a direction based geographic routing scheme (DIG) for the intermittently connected network. Relying on the geographic location information, the packets are routed in a approximate ideal path to the destination, which significantly reduces the resource required in flooding-based algorithm and lead to decreased delay compared to the direct routing. Theoretical analyzes and simulations show that compared to the epidemic routing and direct routing, DIG provides nearly optimal delay with very low overhead.

File replication is a widely used technique for high performance in peer-to-peer file sharing systems. A file replication technique should be efficient and meanwhile facilitates efficient file consistency maintenance. However, most traditional methods don't consider node available capacity and physical location in file replication, leading to high overhead for both file replication and consistency maintenance. This paper presents a proactive low-overhead file replication scheme, namely Plover. By making file replication among physically close nodes based on node available capacities, Plover not only achieves high efficiency in file replication but also supports low-cost and timely consistency maintenance. It also includes an efficient file query redirection algorithm for load balancing between replica nodes. Simulation results demonstrate the effectiveness of Plover in comparison with other file replication schemes. It dramatically reduces the overhead of both file replication and consistency maintenance compared to other schemes. In addition, it yields significant improvements in reduction of overloaded nodes.

In peer-to-peer file sharing systems, file replication and consistency maintenance are widely used techniques for high system performance. Despite significant interdependencies between them, these two issues are typically addressed separately. Most file replication methods rigidly specify replica nodes, leading to low replica utilization, unnecessary replicas and hence extra consistency maintenance overhead. Most consistency maintenance methods propagate update messages based on message spreading or a structure without considering file replication dynamism, leading to inefficient file update and hence high possibility of outdated file response. This paper presents an Integrated file replication and consistency maintenance mechanism (IRM), that integrates the two techniques in a systematic and harmonized manner. It achieves high efficiency in file replication and consistency maintenance at a significantly lower cost. Instead of passively accepting replica and update, each node determines file replication and update polling by dynamically adapting to time-varying file query and update rates, which avoids unnecessary file replications and updates. Simulation results demonstrate the effectiveness of IRM in comparison with other approaches. It dramatically reduces overhead and yields significant improvements on the efficiency of both file replication and consistency maintenance approaches.

Internet-based distributed systems enable globally scattered resources to be collectively pooled and used in a cooperative manner to achieve unprecedented petascale super computing capabilities. Numerous resource discovery approaches have been proposed to help achieve this goal. To report or discover a multi-attribute resource, most approaches use multiple messages with each message for an attribute, leading to high overhead. Anther approach can reduce multi-attribute to one index, but it is not practically effective in an environment with a large number of different resource attributes. Furthermore, few approaches are able to locate resources geographically close to the requesters, which is critical to system performance. This paper presents a P2P-based intelligent resource discovery (PIRD) mechanism that weaves all attributes into a set of indices using locality sensitive hashing, and then maps the indices to a structured P2P. It further incorporates Lempel-Ziv-Welch algorithm to compress attribute information for higher efficiency. In addition, it helps to search resources geographically close to requesters by relying on a hierarchical P2P structure. PIRD significantly reduces overhead and improves the efficiency and effectiveness of resource discovery. Theoretical analysis and simulation results demonstrate the efficiency of PIRD in comparison with other approaches. It dramatically reduces overhead and yields significant improvements on the efficiency of resource discovery.

Radio frequency identification (RFID) and wireless sensor networks (WSN) have been popular in industrial and academic field due to their ability in identity identification and data transmission respectively. Both RFID and wireless sensor networks have undergone huge development separately. In this paper, it has been discussed how helpful and effective it will be when RFID and WSN have been combined together. It also discusses about some applications of this integrated technology in the field of precious animal and patient health monitoring where real time information is of utmost importance. A system structure of the integrated RFID and WSN has been introduced and the simulation results show that the new system outperforms traditional RFID monitory system in terms of the cost of deployment, updating delay and tag capacity requirement.

Intermittently connected mobile networks don't have a complete path from a source to a destination at most of the time. Such an environment can be found in very sparse mobile networks where nodes meet only occasionally or in wireless sensor networks where nodes always sleep to conserve energy. Current transmission approaches in such networks are primarily based on: multi-copy flooding scheme and single-copy forwarding scheme. However, they incur either high overheads due to excessive transmissions or long delay due to possible incorrect choices during forwarding. In this paper, we propose a A utility-based distributed routing algorithm with multi-copies called UDM, in which a packet is initially replicated to a certain number of its neighbor nodes, which sequentially forward those packets to the destination node based on a probabilistic routing scheme. Some buffer management methods are also proposed to further improve its performance. Theoretical analyze and simulations show that compared to epidemic routing, spray and wait routing, UDM routing scheme provides a nearly optimal delay performance with a stable packet arrive rate in the community mobility model.

File consistency maintenance in P2P systems is a technique for maintaining consistency between files and their replicas. Most traditional consistency maintenance methods depend on either message spreading or structure for update propagation by pushing. Message spreading generates high overhead due to redundant messages, and cannot guarantee that every replica node receives an update. Structure-based pushing methods reduce the overhead but cannot guarantee timely consistency in churn. Moreover, most methods are unable to consider physical proximity to improve efficiency. To further reduce update overhead, enhance guarantee of consistency, and take proximity into account, this paper presents a geographically-aware Wave method (GeWave). Depending on adaptive polling in a dynamic structure, GeWave avoids redundant file updates by dynamically adapting to time-varying file update and query rates, and ensures the consistency of query results even in churn. Furthermore, it conducts update propagation between geographically close nodes in a distributed manner. Simulation results demonstrate the efficiency of GeWave in comparison with other representative consistency maintenance schemes. It dramatically reduces the overhead and yields significant improvements on efficiency and scalability of file consistency maintenance schemes.

Peer-to-peer (P2P) networks based on consistent hashing functions have an inherent load uneven distribution problem. Things are even worse in unstructured P2P systems. The objective of load balancing in P2P networks is to balance the workload of the network nodes in proportion to their capacity so as to eliminate traffic bottleneck. It is challenging because of the dynamic nature of overlay networks and time-varying load characteristics. Random choices schemes can balance load effectively while incurring only a small overhead, making such schemes appealing for practical systems. Existing theoretical work analyzing properties of random choices algorithms can not be applied in the highly dynamic and heterogeneous P2P systems. In this paper, we characterize the behaviors of randomized search schemes in the general P2P environment. We extend the supermarket model by investigating the impact of node heterogeneity and churn to the load distribution in P2P networks. We prove that by using d-way random choices schemes, the length of the longest queue in P2P systems with heterogeneous nodal capacity and node churn for d ≥ 2 is clog logn/logd + O(1) with high probability, where c is a constant.

In mobile ad hoc network, all the packets are forwarded in a multi-hop fashion relying on the contribution of each participants. In order to encourage the cooperation between the nodes in the system, many incentive mechanisms have been proposed. Although these incentive schemes can improve the cooperation to some certain extend, they are still suffering from some drawbacks. In this paper, the efficiency of these incentive mechanisms has been analyzed based on game theory modules. Their performances have also been compared with a proposed hybrid cooperation enforcement mechanism ARM. The simulation and theoretical results show the superiority of the ARM over traditional reputation based scheme and price-based scheme.

In peer-to-peer file sharing systems, file replication technology is widely used to reduce hot spots and improve file query efficiency. Most current file replication methods replicate files in all nodes or two endpoints on a client-server query path. However, these methods either have low effectiveness or come at a cost of high overhead. This paper presents an Efficient and adaptive decentralized file replication algorithm (EAD) that achieves high query efficiency and high replica utilization at a significantly low cost. EAD enhances the utilization of file replicas by selecting query traffic hubs and frequent requesters as replica nodes, and dynamically adapting to non-uniform and time-varying file popularity and node interest. Unlike current methods, EAD creates and deletes replicas in a decentralized self-adaptive manner while guarantees high replica utilization. Simulation results demonstrate the efficiency and effectiveness of EAD in comparison with other approaches in both static and dynamic environments. It dramatically reduces the overhead of file replication, and yields significant improvements on the efficiency and effectiveness of file replication in terms of query efficiency, replica hit rate and overloaded nodes reduction.

Resource management is critical to the usability and accessiblity of grid computing systems. Conventional approaches to grid resource discovery are either centralized or hierarchical, and these prove to be inefficient as the size of the grid system increases. The Peer-to-Peer (P2P) paradigm has been applied to grid systems as a mechanism for providing scalable range-query and multi-attribute resource management. However, most current P2P-based resource management approaches support multi-attribute range queries at a high cost. They either depend on multiple P2P networks with each P2P network responsible for a single attribute, or they keep the resource information of all attributes in a single node. This paper presents a low-overhead range-query multi-attribute P2P-based resource management approach, LORM. Unlike other P2P-based approaches, it relies on a single P2P network and allocates resource information to different nodes based on resource attributes and values. Moreover, it has high capability to handle the large-scale and dynamic characteristics of resources in grids. Simulation results demonstrate the efficiency of LORM in comparison with other resource management approaches in terms of resource management overhead and resource discovery efficiency.

Peer-to-peer (P2P) networks are becoming increasingly popular and are an exciting new class of innovative, internet-based resource sharing systems. In a P2P network, a reputation system is essential to evaluate the trustworthiness of participating peers and to combat the selfish, dishonest, and malicious peer behaviors. The system collects locally-generated peer feedbacks and aggregates them to yield the global reputation values to represent peer trustworthiness. Reputation system guides peers in choosing a server for services/resources. Most of the approaches for peers to exploit reputation metrics are to select the one with the highest reputation value as a providing server. However, it may lead to unexpectedly low efficiency for high-reputed peers, and prevents P2P networks from taking full advantage of all resources for high performance. Other approaches restrict a peer to select a server in the same or lower reputation values. However, these approaches prevent P2P networks from achieving their goal of widely resource sharing and service exchanges. In this paper, we introduce a trust-based fairness-oriented server selection policy, FairTrust, for a peer to choose another peer to interact. FairTrust takes into account both reputation and capacity factors in server selection. It helps to create a secure P2P communication environment, and meanwhile to take full advantage of system resources for high performance by fair load distribution. Simulation results show the superiority of the FairTrust policy in achieving both high security and high performance in P2P networks in comparison with other related policies.

Structured peer-to-peer overlay networks, like distributed hash tables (DHTs), map data items to the network based on a consistent hashing function. Such mapping for data distribution has an inherent load balance problem. Data redistribution algorithms based on randomized matching of heavily loaded nodes with light ones can deal with the dynamics of DHTs. However, they are unable to consider the proximity of the nodes simultaneously. There are other methods that rely on auxiliary networks to facilitate locality-aware load redistribution. Due to the cost of network construction and maintenance, the locality-aware algorithms can hardly work for DHTs with churn. This paper presents a locality-aware randomized load-balancing algorithm to deal with both the proximity and network churn at the same time. We introduce a factor of randomness in the probing of lightly loaded nodes in a range of proximity. We further improve the efficiency by allowing the probing of multiple candidates (d-way) at a time. Simulation results show the superiority of the locality-aware two-way randomized algorithm in comparison with other random or locality-aware algorithms. In DHTs with churn, it performs no worse than the best chum-resilient algorithm. It takes advantage of node capacity heterogeneity and achieves good load balance effectively even in a skewed distribution of items

Distributed hash table (DHT) networks based on consistent hashing functions have an inherent load balancing problem. The problem becomes more severe due to the heterogeneity of network nodes and the non-uniform and timevarying file popularity. Existing DHT load balancing algorithms are mainly focused on the issues caused by node heterogeneity. To deal with skewed lookups, this paper presents an elastic routing table (ERT) mechanism for query load balancing, based on the observation that high degree nodes tend to experience more traffic load. The mechanism allows each node to have a routing table of variable size corresponding to its capacity. The indegree and outdegree of the routing table can also be adjusted dynamically in response to the change of file popularity and network churn. Theoretical analysis proves the routing table degree is bounded. The ERT mechanism facilitates locality-aware randomized query forwarding to further improve lookup efficiency. By relating query forwarding to a supermarket customer service model, we prove a 2-way randomized query forwarding policy leads to an exponential improvement in query processing time over random walking. Simulation results demonstrate the effectiveness of the ERT mechanism and its related query forwarding policy for congestion and query load balancing. In comparison with the existing "virtual-server"-based load balancing algorithm and other routing table control approaches, the ERT-based congestion control protocol yields significant improvements in query lookup efficiency.

DHT networks based on consistent hashing functions have an inherent load uneven distribution problem. The objective of DHT load balancing is to balance the workload of the network nodes in proportion to their capacity so as to eliminate traffic bottleneck. It is challenging because of the dynamism nature of DHT networks and time-varying load characteristics. In this paper, we present a hash-based proximity clustering approach for load balancing in heterogeneity DHTs. In the approach, DHT nodes are classified as regular nodes and supernodes according to their computing and networking capacities. Regular nodes are grouped and associated with supernodes via consistent hashing of their physical proximity information on the Internet. The supernodes form a self-organized and churn resilient auxiliary network for load balancing. The hierarchical structure facilitates the design and implementation of a locality-aware randomized load balancing algorithm. The algorithm introduces a factor of randomness in the load balancing processes in a range of neighborhood so as to deal with both the proximity and dynamism. Simulation results show the superiority of the approach, in comparison with a number of other DHT load balancing algorithms. The approach performs no worse than existing proximity-aware algorithms and exhibits strong resilience to the effect of churn. It also greatly reduces the overhead of resilient randomized load balancing algorithms due to the use of proximity information.

Structured P2P overlay networks based on a consistent hashing function have an aftermath load balance problem that needs to be dealt with. A load balancing method should take into account both proximity and dynamic features of DHTs, Randomized matching between heavily loaded nodes with lightly loaded nodes can deal with the dynamic feature. But current randomized methods are unable to consider physical proximity of the node simultaneously. There are locality-aware methods that rely on an additional logical network to capture the physical locality in load balancing. Due to the cost for network construction and maintenance, these locality-aware algorithms can hardly deal with DHTs with churn. This paper presents a locality-aware randomized load balancing algorithm to deal with both of the proximity and dynamic features of DHTs. We introduce a factor of randomness in the probing process in a range of proximity to deal with the DHT churn. We further improve the randomized load balancing efficiency by d-way probing. Simulation results show the superiority of a locality-aware 2-way randomized load balancing in DHTs, in comparison with other pure random policies and locality-aware sequential algorithms. In DHTs with churn, it performs no worse than the best churn resilient algorithm.

Summary form only given. There are many structured P2P systems that use DHT technologies to map data items onto the nodes in various ways for scalable routing and location. Most of the systems require O(logn) hops per lookup request with O(logn) neighbors per node, where n is the network size. We present a constant-degree P2P architecture, namely Cycloid, which emulates a cube-connected-cycles (CCC) graph in the routing of lookup requests. It achieves a time complexity of O(d) per lookup request by using O(1) neighbors per node, where n = d/spl middot/ 2/sup d/. We compare Cycloid with other two constant-degree systems, Viceroy and Koorde in various architectural aspects via simulation. Simulation results show that Cycloid has more advantages for large scale and dynamic systems that have frequent node arrivals and departures. In particular, Cycloid delivers a higher location efficiency in the average case and exhibits a more balanced distribution of keys and query loads between the nodes.