Accurate capacity measurement of Internet services is critical to ensure high-performing production computing environments. In this work, we present our solution of performing accurate capacity measurement. Referred to as "Redliner", it uses live traffic in production environments to drive the measurement, hence avoiding many pitfalls that prevent capacity measurement from obtaining accurate values in synthetic lab environment. Redliner works by intelligently redirecting a portion of production traffic to the SUT (Service Under Test) and realtime analyzing the performance. It has been adopted by hundreds of services inside LinkedIn and is executed for various types of capacity analysis on a daily basis.

Accurate capacity measurement of Internet services is critical to ensure high-performing production computing environments. In this work, we present our solution of performing accurate capacity measurement. Referred to as “LiveRedliner”, it uses live traffic in production environments to drive the measurement, hence avoiding many pitfalls that prevent capacity measurement from obtaining accurate values in synthetic lab environment.

For PaaS-deployed (Platform as a Service) customer-facing applications (e.g., online gaming and online chatting), ensuring low latencies is not just a preferred feature, but a must-have feature. Given the popularity and powerfulness of Java platforms, a significant portion of today's PaaS platforms run Java. JVM (Java Virtual Machine) manages a heap space to hold application objects. The heap space can be frequently GC-ed (Garbage Collected), and applications can be occasionally stopped for long time during some GC and JVM activities. In this work, we investigated the JVM pause problem. We found out that there are some (and large) JVM STW pauses cannot be explained by application-level activities and JVM activities during GC, instead, they are caused by OS mechanisms. We successfully reproduced such problems and root-cause-ed the reasons. The findings can be used to enhance JVM implementation. We also proposed a set of solutions to mitigate and eliminate these large STW pauses. We share the knowledge and experiences in this writing.

Today's applications are increasingly using memory mapped files for managing large volumes of data in hoping to enjoy the performance benefits of memory mapping compared with traditional file IO. Memory mapped files uses the OS page caching mechanism to save expensive system call and copying. However, as we find out, a naive usage of memory mapped files will cause severe performance problem due to the ineffective usage of physical memory. We propose a solution called SmartCache to address the performance issue. SmartCache maintains an application-layer caching space to more effectively use the physical memory. SmartCache can be implemented inside an application or as an independent library for applications to use.

SSD (Solid State Drive) is being increasingly adopted to alleviate the IO performance bottlenecks of applications. Numerous measurement results have been published to showcase the performance improvement brought by SSD as compared to HDD (Hard Disk Drive). However, in most deployment scenarios, SSD is simply treated as a “faster HDD”. Hence its potential is not fully utilized. Though applications gain better performance when using SSD as the storage, the gains are mainly attributed to the higher IOPS and bandwidth of SSD. The naive adoption of SSD does not utilize SSD to its full potential. In other words, application performance improvements brought by SSD could be more significant if applications are designed to be SSD-friendly. In this work, we propose a set of 9 SSD-friendly design changes at application layer. Applying the design changes by applications can result in three types of benefits: (1) improved application performance; (2) increased SSD IO efficiency; and (3) longer SSD life.

Modern cloud computing platforms (e.g. Linux on Intel CPUs) feature ACPI-based (Advanced Configuration and Power Interface) mechanism, which dynamically scales CPU frequencies/voltages to adjust the CPU frequencies based on the workload intensity. With this feature, CPU frequency is reduced when the workload is relatively light in order to save energy, while increased when the workload intensity is relatively high. In business cloud computing environments, software products/services often need to "scale out"' to multiple machines to form a cluster to achieve a pre-defined aggregated performance goal (e.g., SLA-devised throughput). To reduce business operation cost, minimizing the provisioned cluster size is critical. However, as we show in this work, the working of ACPI in today's modern OS may result in more machines being provisioned, hence higher business operation cost, To deal with this problem, we propose a SLA-aware CPU scaling algorithm based on business SLA (Service Level Agreement aware). The proposed design rational and algorithm are a fundamental rethinking of how ACPI mechanisms should be implemented in business cloud computing environments. Contrary to the current forms of ACPI which simply adapt CPU power levels only based on workload intensity, the proposed SLA-aware algorithm is primarily based on current application performance relative to the pre-defined SLA. Specifically, the algorithm targets at achieving the pre-defined SLA as the top-level goal, while saving energy as the second-level goal.

Data quality is essential in big data paradigm as poor data can have serious consequences when dealing with large volumes of data. While it is trivial to spot poor data for small-scale and offline use cases, it is challenging to detect and fix data inconsistency in large-scale and online (real-time or near-real time) big data context. An example of such scenario is spotting and fixing poor data using Apache Samza, a stream processing framework that has been increasingly adopted to process near-real-time data at LinkedIn. To optimize the deployment of Samza processing and reduce business cost, in this work we propose a memory capacity model for Apache Samza to allow denser deployments of high performing data-filtering applications built on Samza. The model can be used to provision just-enough memory resource to applications by tightening the bounds on the memory allocations. We apply our memory capacity model on Linkedln's real use cases in production, which significantly increases the deployment density and saves business costs. We will share key learning in this paper.

Internet companies like LinkedIn handle a large amount of incoming web traffic. Events generated in response to user input or actions are stored in a source database. These database events feature the typical characteristics of Big Data: high volume, high velocity and high variability. Database events are replicated to isolate source database and form a consistent view across data centers. Ensuring a low replication latency of database events is critical to business values. Given the inherent characteristics of Big Data, minimizing the replication latency is a challenging task.

Cloud Computing promises a cost-effective and administration-effective solution to the traditional needs of computing resources. While bringing efficiency to the users thanks to the shared hardware and software, the multi-tenency characteristics also bring unique challenges to the backend cloud platforms. In particular, the JVM mechanisms used by Java applications, coupled with OS-level features, give rise to a set of problems that are not present in other deployment scenarios. In this work, we consider the problem of ensuring high-performance of mission-critical Java applications in multi-tenant cloud environments. Based on our experiences with Linkedin's platforms, we identify and solve a set of problems caused by multi-tenancy. We share the lessons and knowledge we learned during the course.

Many business-oriented services will be gradually offered in the Cloud. Java Message Service (JMS) is a critical messaging technology in Java-based business applications, particularly to those that are based on the Java Enterprise Edition (Java EE) open standard. Maintaining high performance in the horizontally scaled, and elastic, cloud environment is critical to the success of the business applications. In this paper, we present practical considerations in optimizing JMS performance for the cloud deployment, where some of the findings may also serve to improve the design of JMS container so it adapts well to cloud computing. Our work also includes performance evaluation on the proposed strategies.

Video streaming traffic of both VoD (Video on Demand) and Live is exploding. Various types of businesses and many people are relying on video streaming to attract customers/users and for other purposes. Given the vast number of video stream formats (e.g., MP4, FLV) and transmission protocols (e.g., HTTP, RTMP, RTSP) for supporting varying types of playback terminals (particularly mobile devices such as iphone/ipad and Android phones), video content providers often need to transcode videos to multiple formats in order to stream to different types of users. Being time-sensitive and requiring high bandwidth, video streaming exerts high pressure on underlying delivery networks. Content Delivery Network (CDN) providers can help their customers quickly and reliably distribute stream contents to end users. In addition to distributing video streams, CDN providers typically allow their customers to perform video transcoding on CDN platforms. With the high volume of video streams and the bursty transcoding workload, CDN providers are eager to deploy elastic and optimized cloud-based transcoding platforms. In this paper, we design and build such a transcoding platform.

Content Delivery Networks (CDNs) are increasingly being used to deliver live streaming on today's Internet. The new application type of live streaming exposes unique characteristics and challenges that require more advanced design of CDN infrastructure. Unlike traditional web-object delivering, which allows CDN content servers to cache contents and thus typically only involves certain CDN servers (e.g. edge content servers) for delivering contents, live streaming requires a real-time full CDN streaming path that spans across the streaming source, the CDN ingest server, the CDN content server, and end-viewers. Though the ingest server is typically fixed for a particular live stream, appropriate content servers need to be selected for delivering the stream to end viewers. Though today's CDNs typically employ layered infrastructure for delivering live streaming, in this work, we propose a flat-layered design which is referred to as Constrained Server Chaining (CSC) for selecting optimal content servers to deliver live streams. Rather than employing a strictly layered infrastructure, CSC allows CDN streaming servers to dynamically choose upstream servers, thus saves transit cost for CDN providers.

Studies have shown the presence of considerable amounts of redundancy in Internet traffic content. Recent works are exploring possibilities for exploiting network traffic redundancy, but these works invariably focus on fixed wireline networks. Unlike wireline networks, wireless and mobile environments exhibit unique challenges and opportunities in the context of redundancy elimination. In this work, we explore leveraging network traffic redundancy, but exclusively focus on wireless and mobile environments. We first analyze real Wi-Fi traces, and based on insights obtained from the analysis, we propose Wireless Memory (WM), a two-ended AP-client solution to effectively exploit traffic redundancy for such environments. Trace-driven evaluation results show that WM can help deliver significant throughput improvement.

Location-based applications have become increasingly popular on smartphones over the past years. The active use of these applications can however cause device battery drain owing to their power-intensive location-sensing operations. This paper presents an adaptive location sensing framework that significantly improves the energy efficiency of smartphones running location-based applications. The underlying design principles of the proposed framework involve substitution, suppression, piggybacking, and adaptation of applications' location-sensing requests to conserve energy. We implement these design principles on Android-based smartphones as a middleware. Our evaluation results show that the design principles reduce the usage of the power-intensive GPS (Global Positioning System) by up to 98% and improve battery life by up to 75%.

A tremendous amount of research has been done toward improving transport-layer performance over wireless data networks. The improved transport layer protocols are typically application-unaware. In this paper, we argue that the behavior of applications can and does dominate the actual performance experienced. More importantly, we show that for practical applications, application behavior all but completely negates any improvement achievable through better transport layer protocols. In this context, we motivate an application-aware, but application transparent, solution suite called A^3 (application-aware acceleration) that uses a set of design principles realized in an application-specific fashion to overcome the typical behavioral problems of applications. We demonstrate the performance of A^3 through both emulations using realistic application traffic traces and implementations using the NetFilter utility.

Peer-to-peer (P2P) data networks dominate Internet traffic. In this work, we study the problems that arise when mobile hosts participate in P2P networks. We primarily focus on the performance issues as experienced by the mobile host, but also study the impact on other fixed peers. Using Bit Torrent as a key example, we identify several unique problems that arise due to the design aspects of P2P networks being incompatible with typical characteristics of wireless and mobile environments. We then present a wireless P2P (wP2P) client application that is backward compatible with existing fixed peer client applications, but when used on mobile hosts can significant improve performance.

Current popular web-browsers simply fetch the entire web-page from the server in a greedy fashion. This simple web fetching mechanism employed by browsers is inappropriate for use in low-bandwidth networks, since they cause large response times for users unneccesarily. In this paper, we first analyze the reasons that cause large response times by considering several factors including the properties of typical web-pages and browsers, the interaction of the HTTP and TCP protocols, and the impact of server-side optimization techniques. We then propose three easy-to-deploy browser-side optimization mechanisms to reduce the user response time. Through simulations, we compare the performance of our solution with that of current browsers and show that the proposed scheme brings significant performance benefits in terms of user-perceived response times.

A tremendous amount of research has been done toward improving transport layer performance over wireless data networks. The improved transport layer protocols are typically application-unaware. In this paper, we argue that the behavior of applications can and do dominate the actual performance experienced. More importantly, we show that for practical applications, application behavior all but completely negates any improvements achievable through better transport layer protocols. In this context, we motivate an application-aware, but application transparent, solution suite called A 3 (application-aware acceleration) that uses a set of design principles realized in an application specific fashion to overcome the typical behavioral problems of applications. We demonstrate the performance of A3 through emulations using realistic application traffic traces.

A typical wireless sensor network (WSN) performs only one action: sensing the environment. The need for smart interaction with the environment has led to the emergence of wireless sensor and actor networks (WSANs). The evolution from WSNs, which can be thought of to perform only read operations, to WSANs, which can perform both read and write operations, introduces unique and new challenges that need to be addressed. In this context, we identify the problem of mutual exclusion, which is the requirement to act only to the desired level for any particular location and command. We define the different types of mutual exclusion and the associated challenges in the context of WSANs, and show the undesirable consequences of not providing mutual exclusion with example applications. To address this problem efficiently, we propose a greedy centralized approach, and a distributed and fully localized approach based on the centralized approach. Through simulations, we study the performance of the proposed solution with the centralized approach and a baseline strategy, and show that the proposed solution is efficient for a variety of network conditions

A typical wireless sensor network performs only one action: sensing the environment. The requirement for intelligent interaction with the environment has led to the emergence of wireless sensor and actor networks (WSANs). In WSANs, the sensors monitor the environment based on which the sink issues commands to the actors to act on the environment. In order to provide tight coupling between sensing and acting, an effective coordination mechanism is required among sensors and actors. In this context, we identify the problem of "hazards", which is the out-of-order execution of queries and commands due to a lack of coordination between sensors and actors. We identify three types of hazards and show with an example application, the undesirable consequences of these hazards. We also identify and enumerate the associated challenges in addressing hazards. In this context, we discuss the basic design needed to address this problem efficiently. We propose a distributed and fully localized hazard-free approach that addresses the problem and the associated challenges based on the design. Through simulations, we study the performance of the proposed solution and two basic strategies, and show that the proposed solution is efficient for a variety of network conditions.

Superpeer unstructured P2P systems have been found to be very effective by dividing the peers into two layers, superlayer and leaf-layer, in which message flooding is only conducted among superlayer and all leaf-peers are represented by corresponding superpeers. However, current superpeer systems do not employ any effective layer management schemes, so the transient and low-capacity peers are allowed to act as superpeers. Moreover, the lack of an appropriate size ratio maintenance mechanism on superlayer to leaf-layer makes the system's search performance far from being optimal. We present one workload model aimed at reducing the weighted overhead of a network. Using our proposed workload model, a network can determine an optimal layer size ratio between leaf-layer and superlayer. We then propose a dynamic layer management algorithm, DLM, which can maintain an optimal layer size ratio and adaptively elect and adjust peers between superlayer and leaf-layer. DLM is completely distributed in the sense that each peer decides to be a superpeer or a leaf-peer independently without global knowledge. DLM could effectively help a superpeer P2P system maintain the optimal layer size ratio and designate peers with relatively long lifetime and large capacities as superpeers, and the peers with short lifetime and low capacities as leaf-peers under highly dynamic network situations. We demonstrate that the quality of a superpeer system is significantly improved under the DLM scheme by comprehensive simulations.

In unstructured peer-to-peer (P2P) systems, the mechanism of a peer randomly joining and leaving a P2P network causes topology mismatching between the P2P logical overlay network and the physical underlying network, causing a large volume of redundant traffic in the Internet. In order to alleviate the mismatching problem, we propose adaptive connection establishment (ACE), an algorithm of building an overlay multicast tree among each source node and the peers within a certain diameter from the source peer, and further optimizing the neighbor connections that are not on the tree, while retaining the search scope. Our simulation study shows that this approach can effectively solve the mismatching problem and significantly reduce P2P traffic. We further study the tradeoffs between the topology optimization rate and the information exchange overhead by changing the diameter used to build the tree.

The emerging peer-to-peer (P2P) model has recently gained a significant attention due to its high potential of sharing various resources among networked users. Super-peer unstructured P2P systems have been found very effective by dividing the peers into two layers, super-layer and leaf-layer, in which message flooding is only conducted among super-layer. However, current super-peer systems do not employ any effective layer management schemes, which means the transient and low-capacity peers are allowed to act as super-peers. Moreover, the lack of an appropriate size ratio maintenance mechanism on super-layer to leaf-layer makes the system's search performance far from being optimal. We propose a dynamic layer management algorithm, DLM, which can maintain the optimal layer size ratio, and adoptively adjust peers between super-layer and leaf-layer. DLM is completely distributed in the sense that each peer decides to be a super-peer or a leaf peer independently without the global knowledge. DLM could effectively help a super-peer P2P system maintain the optimal layer size ratio, and designate peers with relatively long lifetime and large capacities as super-peers, and the peers with short lifetime and low capacities as leaf-peers under highly dynamic network situations. We demonstrate that the quality of a super-peer system is significantly improved under DLM scheme by comprehensive simulations.

Peer-to-peer (P2P) systems are self-organized and decentralized. However, the mechanism of a peer randomly joining and leaving a P2P network causes topology mismatching between the P2P logical overlay network and the physical underlying network. The topology mismatching problem brings great stress on the Internet infrastructure and seriously limits the performance gain from various search or routing techniques. We propose the adaptive overlay topology optimization (AOTO) technique, an algorithm for building an overlay multicast tree between each source node and its direct logical neighbors so as to alleviate the mismatching problem by choosing closer nodes as logical neighbors, while providing a larger query coverage range. AOTO is scalable and completely distributed in the sense that it does not require global knowledge of the whole overlay network when each node is optimizing the organization of its logical neighbors. The simulation shows that AOTO can effectively solve the mismatching problem and reduce more than 55% of the traffic generated by the P2P system itself.

Blind flooding is a popular search mechanism used in current commercial P2P systems because of its simplicity. However, blind flooding among peers or super-peers causes large volume of unnecessary traffic although the response time is short. Some improved statistics-based search mechanisms can reduce the traffic volume but also significantly shrink the query coverage range. In some search mechanisms, not all peers may be reachable creating the so-called partial coverage problem. Aiming at alleviating the partial coverage problem and reducing the unnecessary traffic, we propose an efficient and adaptive search mechanism, hybrid periodical flooding (HPF). HPF retains the advantages of statistics-based search mechanisms, alleviates the partial coverage problem, and provides the flexibility to adaptively adjust different parameters to meet different performance requirements. The effectiveness of HPF is demonstrated through simulation studies