Telecommunications operators (telcos) traditional sources of income, voice and SMS, are shrinking due to customers using over-the-top (OTT) applications such as WhatsApp or Viber. In this challenging environment it is critical for telcos to maintain or grow their market share, by providing users with as good an experience as possible on their network. But the task of extracting customer insights from the vast amounts of data collected by telcos is growing in complexity and scale everey day. How can we measure and predict the quality of a user's experience on a telco network in real-time? That is the problem that we address in this paper. We present an approach to capture, in (near) real-time, the mobile customer experience in order to assess which conditions lead the user to place a call to a telco's customer care center. To this end, we follow a supervised learning approach for prediction and train our Restricted Random Forest model using, as a proxy for bad experience, the observed customer transactions in the telco data feed before the user places a call to a customer care center. We evaluate our approach using a rich dataset provided by a major African telecommunication's company and a novel big data architecture for both the training and scoring of predictive models. Our empirical study shows our solution to be effective at predicting user experience by inferring if a customer will place a call based on his current context. These promising results open new possibilities for improved customer service, which will help telcos to reduce churn rates and improve customer experience, both factors that directly impact their revenue growth.

Together with the availability of new mobility data, the development of new intelligent transport systems (ITS) have increased, in order to provide new key performance indicators toward the improvement of the management of traffic awareness in cities. ITS rely on accurate transit infrastructure data that often contain erroneous information (e.g., inconsistencies or is out of date). In this paper, we propose a new methodology that makes use of GPS traces to automatically detect or correct bus stop locations, reconstruct bus route shapes, and estimate time schedules. The methodology performs different steps: 1) data cleaning and detection of trips; 2) bus stop extraction through data mining techniques; 3) route shape reconstruction; and 4) time schedule estimation. A case study using real GPS data from the City of Dublin, Ireland, is performed.

Large-scale critical infrastructures such as transportation, energy, or water distribution networks are increasingly equipped with smart sensor technologies. Low-latency analytics on the resulting times series would open the door to many exciting opportunities to improve our grasp on complex urban systems. However, sensor-generated time series often turn out to be noisy, non-uniformly sampled, and misaligned in practice, making them ill-suited for traditional data processing. In this paper, we introduce TRISTAN (massive TRIckletS Time series ANalysis), a new data management system for efficient storage and real-time processing of fine-grained time series data. TRISTAN relies on a dedicated, compressed sparse representation of the time series using a dictionary. In contrast to previous approaches, TRISTAN is able to execute most analytics queries on the compressed data directly, and supports efficient and approximate query answering based on the most significant atoms of the dictionary only. We present the overall architecture of our system and discuss its performance on several smarter city datasets, showing that TRISTAN can achieve up to 20:1 compression ratios and 250x speedup compared to a state-of-the-art system.

The seamless confluence of data in motion and data at rest has the potential to redefine the Big Data analytics landscape in a diverse range of domains. To make this happen, existing data intensive computing frameworks need to be repurposed and integrated at control, data, and management levels. Towards this end, we present the system level integration of IBM InfoSphere Streams with Apache YARN. Our design leverages the key differentiating features of the two frameworks to blend high throughput batch-processing with near line-rate, low latency stream-processing. In addition, both frameworks are able to share resources and offer the same interfaces that their users are accustomed to. Using two real-world examples, we illustrate how such a system can be used in production.

This work is motivated by a real-life application that exploits sensor data available from traffic light control systems currently deployed in many cities around the world. Each sensor consists of an induction loop that generates a stream of events triggered whenever a metallic object e.g. car, bus, or a bicycle, is detected above the sensor. Because of the red phase of traffic lights objects are usually divided into groups that move together. Detecting these groups of objects as long as they pass through the sensor is useful for estimating the status of the toad networks such as car queue length or detecting traffic anomalies. In this work, given a data stream that contains observations of an event, e.g. detection of a moving object, together with the timestamps indicating when the events happen, we study the problem that clusters the events together in real-time based on the proximity of the event's occurrence time. We propose an efficient real-time algorithm that scales up to the large data streams extracted from thousands of sensors in the city of London. Moreover, our algorithm is better than the baseline algorithms in terms of clustering accuracy. We demonstrate motivations of the work by showing a real-life use-case in which clustering results are used for estimating the car queue lengths on the road and detecting traffic anomalies.

Smart sensors are increasingly being used to manage and monitor critical urban infrastructures, e.g., for telecommunication, transport, water, or energy networks, as well as for healthcare or smart buildings. Sensor-based monitoring systems offer ways of continuously monitoring low frequency activities, and open the door to new analytic and predictive applications in Smarter Cities. Such sensors generate “tricklets”, i.e., noisy and continuous time series. Tricklets are typically misaligned, non-uniformly sampled, and comprise low frequency activities and recurring patterns. Storing and making sense of such data in a typical database management system is difficult, due to the impedance mismatch between classical (e.g., relational) data and tricklets. In this paper, we investigate the management of large amounts of tricklets from an architectural perspective, and propose MiSTRAL (MaSsive TRicklets anALysis), an architecture designed for executing low-latency analytics on time series warehouses. MiSTRAL uses a dictionary based representation for tricklets that allows queries to be run natively on compressed representations and thus to achieve the low-latency goal. The architecture of MiSTRAL is presented in detail in the following, along with early experimental results on several Smarter Cities datasets.

General-purpose streaming systems support diverse application domains with powerful and user-defined stream operators. Most general-purpose streaming systems have their own, non-XML, internal data representation. However, streaming input is often either a sequence of small XML documents, or a scan of a huge document. Prior work on XML streaming focuses on filtering, not transforming, XML, and does not describe how to integrate with a general-purpose streaming system. This paper describes how to integrate an XML transformer with a streaming system by designing a specification syntax that is both consistent with the existing system and familiar to XML users. After type-checking the specification, we compile it to an efficient automaton driven by SAX events. Our approach extends the underlying streaming system with XML support without changing its core architecture, and the same technique could be used for other extensions beyond XML.

In this paper we discuss our experience with the design of a public transport awareness application developed for the city of Dublin. The application is capable to ingest, analyze and visualize in real-time high volumes of traffic data coming from a variety of sources. We address the challenges encountered during the design of the application and propose novel solutions to tackle those challenges.

Vehicle GPS data is an essential "raw" material for a broad range of applications such as traffic management and control, routing, and navigation. To become useful, the data has to be related to the underlying road network by means of map matching algorithms, which are often computationally expensive. In addition, GPS data is not accurate and often needs to be cleaned to remove erroneous observations. In this paper, we describe how map matching can be run on IBM's System S, which provides a platform to run stream processing applications in a scalable manner. We show how various features of System S, including a component based programming model, data pipelining and parallelization of computation, help us to scale the map-matching and data cleaning processes, both as the rate of incoming GPS data increases and as the size of the underlying road network increases. We provide results of performance evaluations, where we show our system can match GPS data arriving at a rate of 1 million points per second onto a map with 1 billion links.

With the widespread adoption of location tracking technologies like GPS, the domain of intelligent transportation services has seen growing interest in the last few years. Services in this domain make use of real-time location-based data from a variety of sources, combine this data with static location-based data such as maps and points of interest databases, and provide useful information to end-users. Some of the major challenges in this domain include i) scalability, in terms of processing large volumes of real-time and static data; ii) extensibility, in terms of being able to add new kinds of analyses on the data rapidly, and iii) user interaction, in terms of being able to support different kinds of one-time and continuous queries from the end-user. In this paper, we demonstrate the use of IBM InfoSphere Streams, a scalable stream processing platform, for tackling these challenges. We describe a prototype system that generates dynamic, multi-faceted views of transportation information for the city of Stockholm, using real vehicle GPS and road-network data. The system also continuously derives current traffic statistics, and provides useful value-added information such as shortest-time routes from real-time observed and inferred traffic conditions. Our performance experiments illustrate the scalability of the system. For instance, our system can process over 120000 incoming GPS points per second, combine it with a map containing over 600,000 links, continuously generate different kinds of traffic statistics and answer user queries.

The Web services research community has proposed a number of approaches for service composition, ranging from manual to semi-automatic to completely automatic. However, it is often difficult to take independently developed services and compose them, since they may not work together correctly. For service composition to occur, the services in question must be designed and developed in a manner that facilitates their composition. In this paper, we propose a novel approach for service design and composition that combines top-down and bottom-up elements. Our approach is driven by faceted, tag-based functional requirements provided by end-users. These requirements describe, at a high-level, the families of compositions that end-users desire. The requirements kick off a top-down service development lifecycle, where enterprise architects and service developers design, develop and test workflows and services, possibly reusing existing flows and services in the process. At runtime, end-users can specify goals, which are satisfied through a bottom-up composition of flows from the available services. The composed flows include those explicitly designed by the architects as well as new ones that are assembled in a serendipitous manner from the available services. With examples from a case study in the financial services domain, we demonstrate our approach for designing and developing services that can be composed into myriad workflows based on end-user goals.

In the realm of component-based software systems, pursuers of the holy grail of automated application composition face many significant challenges. In this paper we argue that, while the general problem of automated composition in response to high-level goal statements is indeed very difficult to solve, we can realize composition in a restricted context, supporting varying degrees of manual to automated assembly for specific types of applications. We propose a novel paradigm for composition in flow-based information processing systems, where application design and component development are facilitated by the pervasive use of faceted, tag-based descriptions of processing goals, of component capabilities, and of structural patterns of families of application. The facets and tags represent different dimensions of both data and processing, where each facet is modeled as a finite set of tags that are defined in a controlled folksonomy. All data flowing through the system, as well as the functional capabilities of components are described using tags. A customized AI planner is used to automatically build an application, in the form of a flow of components, given a high-level goal specification in the form of a set of tags. End-users use an automatically populated faceted search and navigation mechanism to construct these high-level goals. We also propose a novel software engineering methodology to design and develop a set of reusable, well-described components that can be assembled into a variety of applications. With examples from a case study in the Financial Services domain, we demonstrate that composition using a faceted, tag-based application design is not only possible, but also extremely useful in helping end-users create situational applications from a wide variety of available components.

With the emergence of Yahoo Pipes and several similar services, data mashup tools have started to gain interest of business users. Making these tools simple and accessible ton users with no or little programming experience has become a pressing issue. In this paper we introduce MARIO (Mashup Automation with Runtime Orchestration and Invocation), a new tool that radically simplifies data mashup composition. We have developed an intelligent automatic composition engine in MARIO together with a simple user interface using an intuitive "wishful search" abstraction. It thus allows users to explore the space of potentially composable data mashups and preview composition results as they iteratively refine their "wishes", i.e. mashup composition goals. It also lets users discover and make use of system capabilities without having to understand the capabilities of individual components, and instantly reflects changes made to the components by presenting an aggregate view of changed capabilities of the entire system. We describe our experience with using MARIO to compose flows of Yahoo Pipes components.

Intelligence Transportation Systems are critical to improve the efficiency of modern transportation. A system that is flexible and powerful enough to handle diverse demands from a large user base, is still elusive. Studies have shown that developing and integrating the various components constitute a significant portion of the capital cost and complexity of such systems. In this paper, we present a stream processing infrastructure we call System S. System S enables the deployment of large scale applications. It supports a mechanism for sharing data sources, software components, and even intermediate results allowing a reduction in the cost of software integration, and ownership. We experiment the stream processing infrastructure with a Fleet Management Center, and demonstrate how the infrastructure can be used to address unique issues in traffic management.

Technologies that use the Internet network to deliver voice communications have the potential to reduce costs and improve access to communications services around the world. However, these new technologies pose several challenges in terms of confidentiality of the conversations and anonymity of the conversing parties. Call authentication and encryption techniques provide a way to protect confidentiality, while anonymity is typically preserved by an anonymizing service (anonymous call). This work studies the feasibility of revealing pairs of anonymous and encrypted conversing parties (caller/callee pair of streams) by exploiting the vulnerabilities inherent to VoIP systems. In particular, by exploiting the aperiodic inter-departure time of VoIP packets, we can trivialize each VoIP stream into a binary time-series. We first define a simple yet intuitive metric to gauge the correlation between two VoIP binary streams. Then we propose an effective technique that progressively pairs conversing parties with high accuracy and in a limited amount of time. Our metric and method are justified analytically and validated by experiments on a very large standard corpus of conversational speech. We obtain impressively high pairing accuracy that reaches 97% after 5 minutes of voice conversations.

Intelligent mesh optical networks deployed today offer unparalleled capacity, flexibility, availability, and, inevitably, new challenges to master all these qualities in the most efficient and practical manner. More specifically, demands are routed according to the state of the network available at the moment. As the network and the traffic evolve, the lightpaths of the existing demands becomes sub-optimal. In this paper we study two algorithms to re-optimize lightpaths in resilient mesh optical networks. One is a complete re-optimization algorithm that re-routes both primary and backup paths, and the second is a partial re-optimization algorithm that re-routes the backup paths only. We show that on average, these algorithms allow bandwidth savings of 3% to 5% of the total capacity in scenarios where the backup path only is re-routed, and substantially larger bandwidth savings when both the working and backup paths are re-routed. We also prove that trying all possible demand permutations with an online algorithm does not guarantee optimality, and in certain cases does not achieve it, while for the same scenario optimality is achieved through re-optimization. This observation motivates the needs for a re-optimization approach that does not just simply look at different sequences, and we propose and experiment with such an approach. Re-optimization has actually been performed in a nationwide live optical mesh network and the resulting savings are reported in this paper, validating reality and the usefulness of re-optimization in real networks.

This paper presents a collection of approximation formulas that allow a network planner to quickly estimate the size of a mesh optical network with limited inputs.In particular, it provides a set of equations that relate number of sites, average fiber connectivity, demand load and capacity for various mesh protection architectures.These results can be used to easily and quickly estimate the amount of traffic that can be carried over a given network, or, conversely, given the traffic to be supported, to assess the characteristics of the topology required (in terms of number of nodes, connectivity).Finally, this analysis can be used to estimate the restoration performance that can be expected without requiring any extensive simulation studies.

Distributed, IP-based control architecture has been proposed for switched optical mesh networks, as a means to automate operations, enhance interoperability and facilitate the deployment of new applications. While distributed control in general enhances scalability and flexibility, it also offers challenges to path computation, especially for shared mesh restored paths, because of the summarization of link state information disseminated by the routing protocol. This paper discusses the tradeoffs between the path computation efficiency and the abstraction level of link state information. Several heuristic algorithms for computing shared mesh restored paths are described, which require different levels of abstraction and summarization of link resource sharing information. The performance of these algorithms is compared in term of the efficiency of network capacity utilization, the computation complexity, and the amount of required network information. We show that with appropriately aggregated link state information the proposed path computation algorithms are able to utilize the network resource very efficiently.

Efficient methods are needed to aggregate and disseminate the routing information, including the optical link resource availability and sharing, so that the amount advertised by the routing protocol is minimized and the information necessary for path computation is not lost. In this paper, the author proposed to extend the generalized multi-protocol label switching (GMPLS) OSPF-TE routing protocol to carry the necessary sharing information of the reserved resource on a TE link in support of computing the shared mesh restored paths. It is based on the OSPF routing extensions required to support traffic engineering (TE) and GMPLS. New optional sub-TLVs (type/length/value) are added to the link TLV of the TE link state advertisements (LSA) so that the sharing information of the restoration resource is disseminated.

Optical mesh network infrastructure has emerged as the technology of choice for next-generation transport networks. At the same time, distributed, IP-based, control architecture has been proposed for intelligent optical networks, as a means to automate operations, enhance interoperability, and facilitate the deployment of new applications. While distributed control in general enhances scalability and flexibility, it has also been observed that the requisite network topology and link-state information summarization may result in suboptimal path computation, especially for shared mesh restored paths. This paper presents a distributed control plane for optical mesh networks, focusing on distributed path computation and provisioning mechanisms. It discusses the tradeoffs between the path computation efficiency for shared mesh restored paths and the amount of network topology and link-state information that is disseminated via routing protocols. We show that with appropriately aggregated link resource availability and sharing information, the proposed distributed path computation algorithms are able to determine the shareability of restoration links with remarkable accuracy. A local channel assignment scheme, which is used in conjunction with the distributed path computation algorithms to assign shared channels when provisioning restoration paths, is also proposed. The additional information that signaling messages are required to carry in order to perform the local channel assignment at each node along the restoration path is discussed. Furthermore, we specify the extensions to the open shortest-path first (OSPF) routing protocol in support of shared mesh restoration. We analyze the performance of the proposed distributed path computation algorithms and the local channel assignment scheme, as well as the overhead of OSPF extensions. In particular, we study the tradeoffs between network capacity utilization, restoration path computation complexity, OSPF extension overhead, and memory requirements for storing the modified link-state database.

We assess the benefits of using statistical techniques to ascertain the shareability of protection channels when computing shared-mesh restored lightpaths in optical mesh networks. These optical networks support wavelength conversion everywhere as a byproduct of the electronic nature of the switching in the optical-electronic-optical optical cross connect used. Current deterministic approaches require a detailed level of information proportional to the number of active lightpaths. Although this is not an issue for good sized networks in the foreseeable future, these approaches are not practicable for distributed route computation involving larger networks. On the other hand, distributed approaches that do not make use of shareability information require a significant amount of additional capacity compared to a centralized approach with access to complete shareability information. With the proposed approach we show that even with less information, independent of the amount of traffic demand, it is possible to predict the shareability of protection channels with remarkable accuracy. In addition, we propose a local distributed channel assignment scheme that is used in conjunction with our distributed route computation proposal to assign shared channels when provisioning the backup path. This channel assignment scheme can also be used to further optimize capacity usage in individual links upon certain events or at regular intervals. Experiments are provided that demonstrate that our approach yields faster computation times with no significant penalty in terms of capacity usage than a centralized approach using complete information.

We assess the benefits of using statistical techniques to ascertain the shareability of protection channels when computing shared mesh restored lightpaths. Current deterministic approaches require a detailed level of information proportional to the number of active lightpaths, and do not scale well as traffic demands and network grow. With the proposed approach, we show that less information, independent of the amount of traffic demand, is sufficient to determine the shareability of protection channels with remarkable accuracy. Experiments also demonstrate that our approach yields faster computation times with no significant penalty in terms of capacity usage.

The paper proposes a structure for quality-of-service (QoS)-centered service level agreements (SLA), and a framework for their real-time management in multiservice packet networks. The SLA is structured to be fair to both parties, the service provider and their customer. The SLA considered here are for QoS assured delivery of aggregate bandwidth from ingress to egress nodes; however, the control and signaling is for the more granular flows or calls. A SLA monitoring scheme is presented in which revenue is generated by the admission of flows into the network, and penalty incurred when flows are lost in periods when the service provider is not SLA compliant. In the SLA management scheme proposed, the results of a prior off-line design are used, in conjunction with measurements taken locally at ingress nodes, to classify the loading status of routes. The routing and resource management are based on virtual partitioning and its supporting mechanism of bandwidth protection. The effectiveness of SLA management is measured by the robustness in performance in the presence of substantial diversity in actual traffic conditions. A simulation testbed called D'ARTAGNAN has been built from which we report numerical results for a case study. The results show that the SLA management scheme is robust, fair and efficient over a broad range of traffic conditions.

We quantify the improvement in blocking performance from the use of wavelength-interchange in WDM Rings and their interconnection. We show that using wavelength-interchange in WDM rings can significantly improve (by factors of 100 to 10000) the "fairness" in wavelength assignment between the longer and the shorter paths. More importantly, we found that for larger interconnected WDM rings this improvement can be achieved by including wavelength-interchange capability in just 10 to 20% of the network nodes. Finally, we develop two analytical models, an accurate but complex overflow model and a less accurate but simpler asymptotic model, for calculating the blocking performance of WDM rings with no wavelength-interchange that use the "maximum reuse" algorithm for wavelength assignment.