This demo presents QARTA; an open-source full-fledged system for highly accurate and scalable map services. QARTA employs machine learning techniques to: (a) construct its own highly accurate map in terms of both map topology and edge weights, and (b) calibrate its query answers based on contextual information, including transportation modality, underlying algorithm, and time of day/week. The demo is based on actual deployment of QARTA in all Taxis in the State of Qatar and in the third-largest food delivery company in the country, and receiving hundreds of thousands of daily API calls with a real-time response time. Audience will be able to interact with the demo through various scenarios that show QARTA map and query accuracy as well as internals of QARTA.

Maps services are ubiquitous in widely used applications including navigation systems, ride sharing, and items/food delivery. Though there are plenty of efforts to support such services through designing more efficient algorithms, we believe that efficiency is no longer a bottleneck to these services. Instead, it is the accuracy of the underlying road network and query result. This paper presents QARTA; an open-source full-fledged system for highly accurate and scalable map services. QARTA employs machine learning techniques to construct its own highly accurate map, not only in terms of map topology but more importantly, in terms of edge weights. QARTA also employs machine learning techniques to calibrate its query answers based on contextual information, including transportation modality, location, and time of day/week. QARTA is currently deployed in all Taxis and the third largest food delivery company in the State of Qatar, replacing the commercial map service that was in use, and responding in real-time to hundreds of thousands of daily API calls. Experimental evaluation of QARTA shows its comparable or higher accuracy than commercial services.

Travel time estimation is an important component in modern transportation applications. The state of the art techniques for travel time estimation use GPS traces to learn the weights of a road network, often modeled as a directed graph, then apply Dijkstra-like algorithms to find shortest paths. Travel time is then computed as the sum of edge weights on the returned path. In order to enable time-dependency, existing systems compute multiple weighted graphs corresponding to different time windows. These graphs are often optimized offline before they are deployed into production routing engines, causing a serious engineering overhead. In this paper, we present STAD, a system that adjusts - on the fly - travel time estimates for any trip request expressed in the form of origin, destination, and departure time. STAD uses machine learning and sparse trips data to learn the imperfections of any basic routing engine, before it turns it into a full-fledged time-dependent system capable of adjusting travel times to real traffic conditions in a city. STAD leverages the spatio-temporal properties of traffic by combining spatial features such as departing and destination geographic zones with temporal features such as departing time and day to significantly improve the travel time estimates of the basic routing engine. Experiments on real trip datasets from Doha, New York City, and Porto show a reduction in median absolute errors of 14% in the first two cities and 29% in the latter. We also show that STAD performs better than different commercial and research baselines in all three cities.

Commercial maps often offer traffic awareness which is critical for many location based services. On the other hand free and open map services (such as government maps or OSM) are traffic oblivious and hence are of limited value for such services. In this paper we show that coarse information available from a commercial map routing API, can be dissected into fine-grained per-road-segment traffic information which can be reused in any application requiring traffic-awareness. Our system MapReuse queries a commercial map for a (relatively small) number of routes, and uses the returned routes and expected travel times, to infer travel time on each individual edge of the road network. Such fine-grained travel time information can be used not only to infer travel time on any given route but also to compute complex spatial queries (such as traffic-aware isochrone map) for free. We test our system on four representative metropolitan areas: Bogota, Doha, NYC and Rome, and report very encouraging results. Namely, we observe the median and mean percentage errors of MapReuse, measured against the travel times reported by the commercial map, to be in the range of 4% to 8%, implying that MapReuse is capable to accurately reconstruct the traffic conditions in all four studied cities.

Understanding link travel times (LTT) has received significant attention in transportation and spatial computing literature but they often remain behind closed doors, primarily because the data used for capturing them is considered confidential. Consequently, free and open maps such as OpenStreetMap (OSM) or TIGER, while being remarkably accurate in capturing geometry and topology of the road network are oblivious to actual travel times. Without LTTs computing the optimal routes or estimated time of arrival is challenging and prone to substantial errors. In this work we set to enrich the underlying map information with LTT by using a most basic data about urban trajectories, which also becomes increasingly available for public use: set of origin/destination location/timestamp pairs. Our system, W-edge utilizes such basic trip information to calculate LTT to each individual road segment, effectively assigning a weight to individual edges of the underlying road network. We demonstrate that using appropriately trained edge weights, the errors in estimating travel times are up to 60% lower than the errors observed in OSRM or GraphHopper, two prominent OSM-based, traffic-oblivious, routing engines.

In this paper we address the challenge of inferring the road network of a city from crowd-sourced GPS traces. While the problem has been addressed before, our solution has the following unique characteristics: (i) we formulate the road network inference problem as a network alignment optimization problem where both the nodes and edges of the network have to be inferred, (ii) we propose both an offline (Kharita) and an online (Kharita) algorithm which are intuitive and capture the key aspects of the optimization formulation but are scalable and accurate. The Kharita in particular is, to the best of our knowledge, the first known online algorithm for map inference, (iii) we test our approach on two real data sets and both our code and data sets have been made available for research reproducibility.

For an auto insurer, understanding the risk of individual drivers is a critical factor in building a healthy and profitable portfolio. For decades, assessing the risk of drivers has relied on demographic information which allows the insurer to segment the market in several risk groups priced with an appropriate premium. In the recent years, however, some insurers started experimenting with so called Usage-Based Insurance (UBI) in which the insurer monitors a number of additional variables (mostly related to the location) and uses them to better assess the risk of the drivers. While several studies have reported results on the UBI trials these studies keep the studied data confidential (for obvious privacy and business concerns) which inevitably limits their reproducibility and interest by the data-mining community. In this paper we discuss a methodology for studying driver risk assessment using a public dataset of 173M taxi rides in NYC with over 40K drivers. Our approach for risk assessment utilizes not only the location data (which is significantly sparser than what is normally exploited in UBI) but also the revenue, tips and overall activity of the drivers (as proxies of their behavioral traits) and obtain risk scoring accuracy on par with the reported results on non-professional driver cohorts in spite of sparser location data and no demographic information about the drivers.

Through their normal operation, cellular networks are a repository of continuous location information from their subscribed devices. Such information, however, comes at a coarse granularity both in terms of space, as well as time. For otherwise inactive devices, location information can be obtained at the granularity of the associated cellular sector, and at infrequent points in time, that are sensitive to the structure of the network itself, and the level of mobility of the device. In this paper, we are asking the question of whether such sparse information can help to identify the paths followed by mobile connected devices throughout the day. If such a task is possible, then we would not only enable continuous mobility path estimation for smartphones, but also for the millions of future connected "things".

Mobile broadband is predominantly priced following tiered plans that involve a certain prepaid emph{commit volume} and additional metered volume priced at a higher emph{penalty rate}. An individual's demand, however, may vary wildly from month to month and thus users inevitably purchase packages that are either too small or too large for their needs. By collaborating in predefined closed (eg, family) or open groups (eg, through tethering) users can reduce both the amount of paid-but-left-unused capacity or the high penalty rates. In this paper we present a quantitative study of collaborative consumption using data from 40K mobile subscribers and tariffs from ten operators around the world. We show that small 2-person family plans offer modest expected savings in the range of 3% to 14%, whereas getting more substantial savings requires rather impractically large groups of approximately 10 people or more. Going over to open groups, where users can freely trade their data capacity, we characterize the impact of the secondary market price on the user costs and the operator revenues, and show that Telcos might be better off to embrace secondary markets (eg, let them integrate with billing), and thus have some control on, as opposed to letting them operate unsupervised through tethering. The latter may severely harm the revenues of a Telco, in an uncontrolled manner, especially in densely populated areas.

We present the first study of broadband services in their broader context, evaluating the impact of service characteristics (such as capacity, latency and loss), their broadband pricing and user demand. We explore these relationships, beyond correlation, with the application of natural experiments. Most efforts on broadband service characterization have so far focused on performance and availability, yet we lack a clear understanding of how such services are being utilized and how their use is impacted by the particulars of the market. By analyzing over 23-months of data collected from 53,000 end hosts and residential gateways in 160 countries, along with a global survey of retail broadband plans, we empirically study the relationship between broadband service characteristics, pricing and demand. We show a strong correlation between capacity and demand, even though subscribers rarely fully utilize their links, but note a law of diminishing returns with relatively smaller increases in demand at higher capacities. Despite the fourfold increase in global IP traffic, we find that user demand on the network over a three year period remained constant for a given bandwidth capacity. We exploit natural experiments to examine the causality between these factors. The reported findings represent an important step towards understanding how user behavior, and the market features that shape it, affect broadband networks and the Internet at large.

A majority of Internet service providers (ISPs) support connectivity to the entire Internet by transiting their traffic via other providers. Although the transit prices per megabit per second (Mbps) decline steadily, the overall transit costs of these ISPs remain high or even increase due to the traffic growth. The discontent of the ISPs with the high transit costs has yielded notable innovations such as peering, content distribution networks, multicast, and peer-to-peer localization. While the above solutions tackle the problem by reducing the transit traffic, this paper explores a novel approach that reduces the transit costs without altering the traffic. In the proposed Cooperative IP Transit (CIPT), multiple ISPs cooperate to jointly purchase Internet Protocol (IP) transit in bulk. The aggregate transit costs decrease due to the economies-of-scale effect of typical subadditive pricing as well as burstable billing: Not all ISPs transit their peak traffic during the same period. To distribute the aggregate savings among the CIPT partners, we propose Shapley-value sharing of the CIPT transit costs. Using public data about IP traffic and transit prices, we quantitatively evaluate CIPT and show that significant savings can be achieved, both in relative and absolute terms. We also discuss the organizational embodiment, relationship with transit providers, traffic confidentiality, and other aspects of CIPT.

The co-existence of cellular and wired networks has been exploited almost exclusively in the direction of OffLoading traffic from the former onto the latter. In this paper we claim that there exist cases that call for the exact opposite, i.e, use the cellular network to assist a fixed wired network. In particular, we show that by "OnLoading'' traffic from the wired broadband network onto the cellular network we can usefully speedup wired connections, on the downlink or the uplink. We consider the technological challenges pertaining to this idea and implement a prototype 3G OnLoading service that we call 3GOL, that can be deployed by an operator providing both the wired and cellular network services. By strategically OnLoading a fraction of the data transfers to the 3G network, one can significantly enhance the performance of particular applications. In particular we demonstrate non-trivial performance benefits of 3GOL to two widely used applications: video-on-demand and multimedia upload. We also consider the case when the operator that provides wired and cellular services is different, adding the analysis on economic constraints and volume cap on cellular data plans that need to be respected. Simulating 3GOL}over a DSLAM trace we show that 3GOL can reduce video pre-buffering time by at least 20% for 50% of the users while respecting data caps and we design a simple estimator to compute the daily allowance that can be used towards 3GOL while respecting caps. Our prototype is currently being piloted in 30 households in a large European city by a large network provider.

Voice and data traffic growth over the last several years has become a major challenge for cellular operators with a direct impact on revenues, infrastructure investments, and end-user performance. The economics of these operators depend on various incentives used to attract users in the form of unlimited, buffet-like voice/sms/data packages. However, our understanding of the effects of user behavior under these offerings on operator revenues/costs remains poor. Using two years of detailed usage information of ~1 million users across three services, voice, sms and data, combined with payment and cost information, we study how user behavior affects the economics of cellular operators. We discover that around 20% of the users consume more resources than what they pay for and hence are non-profitable. In addition to the individual user behavior, we study how the user interactions in the call graph affect the operator's revenues and cost, drawing on tools from social network analysis. We develop a framework that incorporates both the individual and social user behavior for studying how volume caps influence the revenues and the traffic costs. Using this framework we empirically show that volume caps can increase the difference between the revenues and the traffic costs of the studied operator by a factor of 2, while affecting only 16% of the existing user base.

Many emerging scientific and industrial applications require transferring multiple terabytes of data on a daily basis. Examples include pushing scientific data from particle accelerators/colliders to laboratories around the world, synchronizing datacenters across continents, and replicating collections of high-definition videos from events taking place at different time-zones. A key property of all above applications is their ability to tolerate delivery delays ranging from a few hours to a few days. Such delay-tolerant bulk (DTB) data are currently being serviced mostly by the postal system using hard drives and DVDs, or by expensive dedicated networks. In this paper, we propose transmitting such data through commercial ISPs by taking advantage of already-paid-for off-peak bandwidth resulting from diurnal traffic patterns and percentile pricing. We show that between sender-receiver pairs with small time-zone difference, simple source scheduling policies are able to take advantage of most of the existing off-peak capacity. When the time-zone difference increases, taking advantage of the full capacity requires performing store-and-forward through intermediate storage nodes. We present an extensive evaluation of the two options based on traffic data from 200+ links of a large transit provider with points of presence (PoPs) at three continents. Our results indicate that there exists huge potential for performing multiterabyte transfers on a daily basis at little or no additional cost.

The assumption of rationality is fundamental to large part of network economics literature. In this paper, we use a simple definition of rationality based on economic self-interest and test for such behavior using real data on how users purchase and consume mobile network services. If users acted in their best (optimal) interest, then they would opt for the tariff that best suits their demands. However, that need not be the case, as users can fall prey to biases that can lead them to make seemingly sub-optimal choices. Such biases are hard to characterize and in this paper we empirically study how end-users purchase and use network services.

Access link can often be the bottleneck for application performance. In this paper, we propose to augment wired connections using cellular ones, that we term "3G onloading (3GOL)". 3GOL utilizes available mobile devices and already-paid-for data volumes to augment and improve performance of applications on wired network. We motivate 3GOL by understanding bottlenecks present in the wired and the cellular networks. In order to understand the potential benefits of 3GOL, we conduct active experiments using mobile devices. We show that capacity gains can scale linearly with the number of devices on the downlink while also seeing improvements on the uplink. Using real traces we show how video on demand can benefit with 3GOL, even when volume caps are in place. We design 3GOL as an over the top service, and highlight research challenges.

We study the problem of how to share the cost of a backbone network among its customers. A variety of empirical cost-sharing policies are used in practice by backbone network operators but very little ever reaches the research literature about their properties. Motivated by this, we present a systematic study of such policies focusing on the discrepancies between their cost allocations. We aim at quantifying how the selection of a particular policy biases an operator's understanding of cost generation. We identify F-discrepancies due to the specific function used to map traffic into cost (e.g., volume vs. peak rate vs. 95-percentile) and M-discrepancies, which have to do with where traffic is metered (per device vs. ingress metering). We also identify L-discrepancies relating to the liability of individual customers for triggered upgrades and consequent costs (full vs. proportional), and finally, TCO-discrepancies emanating from the fact that the cost of carrying a bit is not uniform across the network (old vs. new equipment, high vs. low energy or real estate costs, etc.). Using extensive traffic, routing, and cost data from a tier-1 network we show that F-discrepancies are large when looking at individual links but cancel out when considering network-wide cost-sharing. Metering at ingress points is convenient but leads to large M-discrepancies, while TCO-discrepancies are huge. Finally, L-discrepancies are intriguing and esoteric but understanding them is central to determining the cost a customer inflicts on the network.

A majority of ISPs (Internet Service Providers) support connectivity to the entire Internet by transiting their traffic via other providers. Although the transit prices per Mbps decline steadily, the overall transit costs of these ISPs remain high or even increase, due to the traffic growth. The discontent of the ISPs with the high transit costs has yielded notable innovations such as peering, content distribution networks, multicast, and peer-to-peer localization. While the above solutions tackle the problem by reducing the transit traffic, this paper explores a novel approach that reduces the transit costs without altering the traffic. In the proposed CIPT (Cooperative IP Transit), multiple ISPs cooperate to jointly purchase IP (Internet Protocol) transit in bulk. The aggregate transit costs decrease due to the economies-of-scale effect of typical subadditive pricing as well as burstable billing: not all ISPs transit their peak traffic during the same period. To distribute the aggregate savings among the CIPT partners, we propose Shapley-value sharing of the CIPT transit costs. Using public data about IP traffic of 264 ISPs and transit prices, we quantitatively evaluate CIPT and show that significant savings can be achieved, both in relative and absolute terms. We also discuss the organizational embodiment, relationship with transit providers, traffic confidentiality, and other aspects of CIPT.

Access networks include modems, home gateways, and DSL Access Multiplexers (DSLAMs), and are responsible for 70-80% of total network-based energy consumption. In this paper, we take an in-depth look at the problem of greening access networks, identify root problems, and propose practical solutions for their user- and ISP-parts. On the user side, the combination of continuous light traffic and lack of alternative paths condemns gateways to being powered most of the time despite having Sleep-on-Idle (SoI) capabilities. To address this, we introduce Broadband Hitch-Hiking (BH2), that takes advantage of the overlap of wireless networks to aggregate user traffic in as few gateways as possible. In current urban settings BH2 can power off 65-90% of gateways. Powering off gateways permits the remaining ones to synchronize at higher speeds due to reduced crosstalk from having fewer active lines. Our tests reveal speedup up to 25%. On the ISP side, we propose introducing simple inexpensive switches at the distribution frame for batching active lines to a subset of cards letting the remaining ones sleep. Overall, our results show an 80% energy savings margin in access networks. The combination of B2 and switching gets close to this margin, saving 66% on average.

This paper presents and develops a novel delay-based additive increase, multiplicative decrease (AIMD) congestion control algorithm. The main features of the proposed solution include: 1) low standing queues and delay in homogeneous environments (with delay-based flows only); 2) fair coexistence of delay- and loss-based flows in heterogeneous environments; 3) delay-based flows behave as loss-based flows when loss-based flows are present in the network; otherwise they revert to delay-based operation. It is also shown that these properties can be achieved without any appreciable increase in network loss rate over that which would be present in a comparable network of standard TCP flows (loss-based AIMD). To demonstrate the potential of the presented algorithm, both analytical and simulation results are provided in a range of different network scenarios. These include stability and convergence results in general multiple-bottleneck networks and a number of simulation scenarios to demonstrate the utility of the proposed scheme. In particular, we show that networks employing our algorithm have the features of networks in which RED AQM's are deployed. Furthermore, in a wide range of situations (including high-speed scenarios), we show that low delay is achieved irrespective of the queueing algorithm employed in the network, with only sender-side modification to the basic AIMD algorithm.

Cost control for the Internet access providers (AP) influences not only the nominal speeds offered to the customers, but also other, more controversial, policies related to traffic shaping and discrimination. Given that the cost for the AP is determined by the peak-hour traffic (e.g. through the 95th-percentile), the individual user contribution towards the aggregate cost is not a linear function of its byte usage. In this paper we propose a metric for evaluating the contribution each individual user has on the peak demand, that is based on Shapley value, a well known game-theoretic concept. Given the computational complexity of calculating the Shapley value, we use a Monte Carlo method for approximating it with reasonable accuracy. We employ our methodology to study a dataset that logs per-subscriber temporal usage patterns over one month period for 10K broadband subscribers of a European AP and report observed results.

In recent years we have witnessed a great interest in large distributed computing platforms, also known as clouds. While these systems offer enormous computing power, they are major energy consumers. In existing data centers CPUs are responsible for approximately half of the energy consumed by the servers. A promising technique for saving CPU energy consumption is dynamic speed scaling, in which the speed at which the processor is run is adjusted based on demand and performance constraints. In this paper we look at the problem of allocating the demand in the network of processors (each being capable to perform dynamic speed scaling) to minimize the global energy consumption/cost subject to a performance constraint. The nonlinear dependence between the energy consumption and the performance as well as the high variability in the energy prices result in a nontrivial resource allocation. The problem can be abstracted as a fully distributed convex optimization with a linear constraint. On the theoretical side, we propose two low-overhead fully decentralized algorithms for solving the problem of interest and provide closed-form conditions that ensure stability of the algorithms. Then we evaluate the efficacy of the optimal solution using simulations driven by the real-world energy prices. Our findings indicate a possible cost reduction of 10-40% compared to power-oblivious 1/N load balancing, for a wide range of load factors.

With the expansion of cloud-based services, the question as to how to control usage of such large distributed systems has become increasingly important. Load balancing (LB), and recently proposed distributed rate limiting (DRL) have been used independently to reduce costs and to fairly allocate distributed resources. In this paper we propose a new mechanism for cloud control that unifies the use of LB and DRL: LB is used to minimize the associated costs and DRL makes sure that the resource allocation is fair. From an analytical standpoint, modelling the dynamics of DRL in dynamic workloads (resulting from LB cost-minimization scheme) is a challenging problem. Our theoretical analysis yields a condition that ensures convergence to the desired working regime. Analytical results are then validated empirically through several illustrative simulations. The closed- form nature of our result also allows simple design rules which, together with extremely low computational and communication overhead, makes the presented algorithm practical and easy to deploy.

Delay-based TCP variants continue to attract a large amount of attention in the networking community. Potentially, they offer the possibility to efficiently use network resources while at the same time achieving low queueing delay and virtually zero packet loss. One major impediment to the deployment of delay-based TCP variants is their inability to coexist fairly with standard loss-based TCP. In this paper we propose a simple strategy to make the fair coexistence possible and to ensure that delay-based flows will revert back to the delay-based operation when loss-based flows are no longer present. Analytical and ns-2 simulation results are presented to validate the proposed algorithm.

The distributed rate limiting (DRL) paradigm is a recently proposed mechanism for decentralized control of cloud-based services. DRL is a simple and efficient approach to resolve the issues of pricing and resource control/engineering of cloud based services. The existing DRL schemes focus on very specific performance metrics (such as loss rate and fair-share) and their design heavily depends on the assumption that the traffic is generated by elastic TCP sources. In this paper we tackle the DRL problem for general workloads and performance metrics and propose an analytic framework for the design of stable DRL algorithms. The closed-form nature of our results allows simple design rules which, together with extremely low communication overhead, makes the presented algorithms practical and easy to deploy with guaranteed convergence properties under a wide range of possible scenarios.

Control of large distributed cloud-based services is a challenging problem. The Distributed Rate Limiting (DRL) paradigm was recently proposed as a mechanism for tackling this problem. The heuristic nature of existing DRL solutions makes their behavior unpredictable and analytically untractable. In this paper we treat the DRL problem in a mathematical framework and propose two novel DRL algorithms that exhibit good and predictable performance. The first algorithm Cloud Control with Constant Probabilities (C3P) solves the DRL problem in best effort environments, emulating the behavior of a single best-effort queue in a fully distributed manner. The second problem we approach is the DRL in processor sharing environments. Our algorithm, Distributed Deficit Round Robin (D2R2), parameterized by parameter α, converges to a state that is, at most, O(1/α) away from the exact emulation of centralized processor sharing queue. The convergence and stability properties are fully analyzed for both C3P and D2R2. Analytical results are validated empirically through a number of representative packet level simulations. The closed-form nature of our results allows simple design rules which, together with extremely low communication overhead, makes the presented algorithms practical and easy to deploy.

The need for efficient counter architecture has arisen for the following two reasons. Firstly, a number of data streaming algorithms and network management applications require a large number of counters in order to identify important traffic characteristics. And secondly, at high speeds, current memory devices have significant limitations in terms of speed (DRAM) and size (SRAM). For some applications no information on counters is needed on a per-packet basis and several methods have been proposed to handle this problem with low SRAM memory requirements. However, for a number of applications it is essential to have the counter information on every packet arrival. In this paper we propose two, computationally and memory efficient, randomized algorithms for approximating the counter values. We prove that proposed estimators are unbiased and give variance bounds. A case study on multistage filters (MSF) over the real Internet traces shows a significant improvement by using the active counters architecture.

Small Flow Completion Time (FCT) of short-lived flows, and fair bandwidth allocation of long-lived flows have been two major, usually concurrent, goals in the design of resource allocation algorithms. In this paper we present a framework that naturally unifies these two objectives under a single umbrella; namely by proposing resource allocation algorithm Markov Active Yield (MAY). Based on a probabilistic strategy: "dropproportionaltotheamountofpastdrops", MAY achieves very small FCT among short-lived flows as well as max-min fair bandwidth allocation among long-lived flows, using only the information of short history of already dropped packets. It turns out that extremely small amount of on-chip SRAM (roughly 1 bit per flow in Pareto-like flow size distributions) is enough for storing this drop history. Analytical models are presented and analyzed and accuracy of results is verified experimentally using packet level ns2 simulations.