Data ingestion is an essential part of companies and organizations that collect and analyze large volumes of data. This paper describes Gobblin, a generic data ingestion framework for Hadoop and one of LinkedIn's latest open source products. At LinkedIn we need to ingest data from various sources such as relational stores, NoSQL stores, streaming systems, REST endpoints, filesystems, etc. into our Hadoop clusters. Maintaining independent pipelines for each source can lead to various operational problems. Gobblin aims to solve this issue by providing a centralized data ingestion framework that makes it easy to support ingesting data from a variety of sources. Gobblin distinguishes itself from similar frameworks by focusing on three core principles: generality, extensibility, and operability. Gobblin supports a mixture of data sources out-of-the-box and can be easily extended for more. This enables an organization to use a single framework to handle different data ingestion needs, making it easy and inexpensive to operate. Moreover, with an end-to-end metrics collection and reporting module, Gobblin makes it simple and efficient to identify issues in production.

Espresso is a document-oriented distributed data serving platform that has been built to address LinkedIn's requirements for a scalable, performant, source-of-truth primary store. It provides a hierarchical document model, transactional support for modifications to related documents, real-time secondary indexing, on-the-fly schema evolution and provides a timeline consistent change capture stream. This paper describes the motivation and design principles involved in building Espresso, the data model and capabilities exposed to clients, details of the replication and secondary indexing implementation and presents a set of experimental results that characterize the performance of the system along various dimensions.

In Internet architectures, data systems are typically categorized into source-of-truth systems that serve as primary stores for the user-generated writes, and derived data stores or indexes which serve reads and other complex queries. The data in these secondary stores is often derived from the primary data through custom transformations, sometimes involving complex processing driven by business logic. Similarly data in caching tiers is derived from reads against the primary data store, but needs to get invalidated or refreshed when the primary data gets mutated. A fundamental requirement emerging from these kinds of data architectures is the need to reliably capture, flow and process primary data changes.

Linked In is among the largest social networking sites in the world. As the company has grown, our core data sets and request processing requirements have grown as well. In this paper, we describe a few selected data infrastructure projects at Linked In that have helped us accommodate this increasing scale. Most of those projects build on existing open source projects and are themselves available as open source. The projects covered in this paper include: (1) Voldemort: a scalable and fault tolerant key-value store, (2) Data bus: a framework for delivering database changes to downstream applications, (3) Espresso: a distributed data store that supports flexible schemas and secondary indexing, (4) Kafka: a scalable and efficient messaging system for collecting various user activity events and log data.

We introduce and formalize a novel constrained path optimization problem that is the heart of the real-time ad serving task in the Yahoo! (formerly RightMedia) Display Advertising Exchange. In the Exchange, the ad server's task for each display opportunity is to compute, with low latency, an optimal valid path through a directed graph representing the business arrangements between the hundreds of thousands of business entities that are participating in the Exchange. These entities include not only publishers and advertisers, but also intermediate entities called "ad networks" which have delegated their ad serving responsibilities to the Exchange. Path optimality is determined by the payment to the publisher, and is affected by an advertiser's bid and also by the revenue-sharing agreements between the entities in the chosen path leading back to the publisher. Path validity is determined by constraints which focus on the following three issues: 1) suitability of the opportunity's web page and its publisher 2)suitability of the user who is currently viewing that web page, and 3) suitability of a candidate ad and its advertiser. Because the Exchange's constrained path optimization task is novel, there are no published algorithms for it. This paper describes two different algorithms that have both been successfully used in the actual Yahoo! ad server. The first algorithm has the advantage of being extremely simple, while the second is more robust thanks to its polynomial worst-case running time. In both cases, meeting latency caps has required that the basic algorithms be improved by optimizations; we will describe a candidate ordering scheme and a pre-computation scheme that have both been effective in reducing latency in the real ad serving system that serves over ten billion ad calls per day.

We introduce the problem of evaluating graph constraints in content-based publish/subscribe (pub/sub) systems. This problem formulation extends traditional content-based pub/sub systems in the following manner: publishers and subscribers are connected via a (logical) directed graph G with node and edge constraints, which limits the set of valid paths between them. Such graph constraints can be used to model a Web advertising exchange (where there may be restrictions on how advertising networks can connect advertisers and publishers) and content delivery problems in social networks (where there may be restrictions on how information can be shared via the social graph). In this context, we develop efficient algorithms for evaluating graph constraints over arbitrary directed graphs G. We also present experimental results that demonstrate the effectiveness and scalability of the proposed algorithms using a realistic dataset from Yahoo!'s Web advertising exchange.

Abstract — Real-time applications are going to play a major role in Vehicular Ad-hoc Networks (VANETs). In this context, nodes ’ IP addresses need to be automatically configured in a very small time and with a reduced need for re-configurations. Due to the very high mobility of vehicles, however, traditional mechanisms for address auto-configuration fail to perform well. Aimed at solving this problem, we propose a novel Leader-based scheme that exploits the topology of VANETs and a distributed DHCP service to guarantee fast and stable address configuration. Keywords: Ad-hoc networks, VANET, address configuration. I.

One of the most important sources of revenue for big Internet-based companies are advertisements. With vehicular networks poised to become part of the Internet, this new "edge" of the Internet represents the next frontier that advertising companies will be striving to reach. In this paper we investigate the design parameters for AdTorrent, an integrated system for search, ranking and content delivery in Digital Billboards, a scalable "push" model architecture for targeted dissemination of ad content in a car network. We present an analytical model to estimate the performance impact of key design parameters such as the scope of the query flooding on the query hit ratio in epidemic query dissemination.

The increasing ease of self-expression and web-publishing has resulted in an explosion in the amount of content being generated in the current Internet. Besides traditional sources such as news portals, regular users are documenting their lives and thoughts and other people are subscribing, downloading and viewing this content. A lot of content therefore is being generated at the edge and consumed by the edge. Traditional client-server architectures are known to be in-effective in handling large correlated bursts of user demands. However, with RSS becoming more popular, such flash crowd scenarios will be more and more commonplace due to automated polling and downloads. Peer to peer protocols such as BitTorrent provide an attractive solution for such scenarios. BitTorrent networks are scalable, and the expected download time is independent of the arrival rate of peers (content consumers). However, the base performance of a BitTorrent network may not be fast enough from a user or content publisher's perspective. Besides, BitTorrent gives poor results towards the end of a flash crowd when most of the large burst of arrivals have downloaded and left, and there are not too many peers online. We motivate the need for a content delivery network with well connected servers to participate in BitTorrent delivery streams. The servers are dynamically added and function as cushions to handle increase in demand as well as bolster a delivery stream when there is a paucity of users.

Grido is an architecture that targets a network operator intending to provide enhanced services to its customers. This is achieved by setting up a "backbone" overlay network. A backbone overlay is a set of Internet hosts dedicated to providing overlay services. A network operator can view Grido as a sandbox for rapid prototyping and market adoption assessment of novel services. In the past, overlay networks have been designed to mitigate deployment issues of functionalities such as multicast and QoS at the network layer. Grido provides a WS-agreement based negotiation interface complying with the current Global Grid Forum (GGF) standards. We propose to use a novel virtual coordinates-assisted overlay construction and maintenance protocol. We demonstrate using simulations, that Grido incurs a low latency overhead while maintaining sparse connectivity on the backbone overlay. Grido also incurs low overhead for virtual coordinates estimation and chooses the closest 5% overlay node to any IP address, 95% of the time

Today's emerging traffic is far removed from the traffic trends seen during the early days of Ethernet technology. As a result, the current IEEE 802.1 standards and its extensions to the spanning tree protocol fall short of providing satisfactory quality of service for traffic which has a significant amount of QoS-sensitive multimedia and VoIP traffic. In the current and near-future scenario of campus-wide networks with significantly large layer-2 clusters and numerous virtual LANs (VLANs), we show significant shortcomings of the basic spanning tree and the multiple spanning tree protocols with regard to QoS. We propose a novel, simple, and yet highly effective enhancement to the multiple spanning tree protocol to achieve a high degree of QoS by keeping in perspective the different characteristics of the various traffic types in the Diffserv framework. We discuss the problems of the current standards and present in detail our proposed extension to overcome them. Our simulation results show good improvements in throughput and significant benefits in delay for all classes of traffic to conclusively prove our claims.

Current traffic patterns are far removed from the traffic trends seen during the early days of Ethernet technology. Therefore, the current IEEE 802.1 standard and its extensions to the spanning tree protocol fall short of providing-satisfactory quality of service for traffic which has a significant amount of QoS-sensitive traffic. In addition, they do not provide a mechanism for stabilizing the traffic load among a whole bridged network where usually the root switch has to handle all the traffic from its child switches. In the current and near-future scenario of switching networks with significantly large layer-2 clusters and numerous VLANs, we show significant shortcomings of the basic spanning tree and the multiple spanning tree protocols with regard to QoS and load balancing. We propose to incorporate a bridge ID scheme of a regional root identifier which is used in multiple spanning tree instance (MSTI) to provide an efficient MSTI building method for QoS and load balancing. Through this scheme, the traffic source of a point-to-multipoint (P2MP) application can set itself as the root of MSTI satisfying the QoS constraint while roots of several regular MSTIs are built in a distributed way. We discuss the problem of building MSTI for P2MP applications and regular MSTIs and present in detail our proposed scheme. Our experiments for two cases show good improvement of QoS in terms of delay and load balancing of the whole bridge network.

Efforts to provide connection oriented service over the inherently best-effort Internet started almost right after its birth. Today, there exist a multitude of solutions that have been proposed but have never been implemented due to their impracticability. We propose a practical solution for fast, low cost, scalable, and yet accurate QoS routing. We propose to use hierarchical approaches to make the scheme practical and cost-effective. At the same time, we increase network utilization and decrease inaccuracy of stale information by the use of multiple paths. An extensive simulation of the various permutations of schemes over a large set of topologies and traffic conditions validate the proposed schemes and prove conclusively that hierarchical schemes with multiple path capabilities not only result in significantly lower overhead, but also give high levels of QoS performance. This paper presents the architecture of our schemes, the multiple path computation algorithm, and the simulation results validating our claims.