We present LiRank, a large-scale ranking framework at LinkedIn that brings to production state-of-the-art modeling architectures and optimization methods. We unveil several modeling improvements, including Residual DCN, which adds attention and residual connections to the famous DCNv2 architecture. We share insights into combining and tuning SOTA architectures to create a unified model, including Dense Gating, Transformers and Residual DCN. We also propose novel techniques for calibration and describe how we productionalized deep learning based explore/exploit methods.

Analytics on Big Data is critical to derive business insights and drive innovation in today's Internet companies. Such analytics involve complex computations on large datasets, and are typically performed on MapReduce based frameworks such as Hive and Pig. However, in our experience, these systems are still quite limited in performing at scale. In particular, calculations that involve complex joins and aggregations, e.g. statistical calculations, scale poorly on these systems. In this paper we propose novel primitives for scaling such calculations. We propose a new data model for organizing datasets into calculation data units that are organized based on user-defined cost functions. We propose new operators that take advantage of these organized data units to significantly speed up joins and aggregations. Finally, we propose strategies for dividing the aggregation load uniformly across worker processes that are very effective in avoiding skews and reducing (or in some cases even removing) the associated overheads. We have implemented all our proposed primitives in a framework called Rubix, which has been in production at LinkedIn for nearly a year. Rubix powers several applications and processes TBs of data each day. We have seen remarkable improvements in speed and cost of complex calculations due to these primitives.

The broadcast nature of existing wireless LANs can produce excessive radio coverage which is often undesirable for applications like wireless IPTV or location-based service. To address this issue, we utilize the characteristics of signal attenuation common to all wireless communications and implement apoint-castingservice1(PCast) over existing wireless networks that can significantly restrict the coverage for transmissions. The principle of PCast is to split and encode the downlink traffic and deliver it to the client via a few surrounding access points (APs). Hence, an unintended client cannot decode a packet without all its fragments. Equivalently, the service area is bounded within the intersection of communication coverage of all APs. The result coverage area can be further reduced via rate and power control at the APs. In this paper, we present our design and implementation of the PCast service on top of an 802.11a wireless LAN. Experiments conducted in a lab environment show that PCast can bound the service region to be within 2 meters around a client, while incurring jitter of less than 1ms, and bandwidth capacity loss of less than 10%.

Environmental mobility refers to the ambient motion of people, vehicles and other objects in the vicinity of wireless communication. Whereas the effect of mobility of transmitting and receiving radios has been widely researched, the impact of environmental mobility has been relatively unexplored in wireless networks. In this paper, we present a systematic study to analyze the effect of environmental mobility on wireless link behavior and protocol performance using measurement, analysis, and simulations. Measurement data from channel traces was collected to isolate and quantify the effects of environmental mobility on the channel performance. Efficient models were developed to explain the observed behavior and integrated with network simulation to enable protocol level analysis under such dynamics. We found that the class of protocols that maintain state or have memory but are otherwise agnostic of the channel operations fare poorly by acquiring incorrect information about the channel and are not able to adequately exploit durations where channel conditions are good. Two case studies at different layers of protocol stack are used to quantify such behavior to show that protocol perform maybe as much as 20-50% below their expected performance. To remedy this situation we present a cross layer optimization scheme, called Recovery from Earlier Good State (REGS), that allows protocols to become aware of the underlying channel operations. We show that use of REGS together with existing optimizations can improve throughputs in the presence of EM by more than 100%.

Although there is growing interest in the use of physical testbeds to evaluate the performance of applications and protocols for sensor platforms, such studies also encounter significant challenges that include the lack of scalability and repeatability, as well as the inability to represent a diverse set of operational scenarios. On the other hand, simulators can typically address the preceding problems but of-ten lack the high degree of fidelity available to the analysts with physical testbeds. In this paper, we present the design and implementation of SenQ - an accurate and scalable evaluation framework for sensor networks that effectively addresses the preceding challenges. In particular, SenQ integrates sensor network operating systems with a very high-fidelity simulation of wireless networks such that sensor network applications and protocols can be executed, without modifications, in a repeatable manner under a diverse set of scalable environments. SenQ extends beyond the existing suite of simulators and emulators in four key aspects: first, it supports emulation of sensor network applications and protocols in an efficient and exible manner; second, it provides an efficient set of models of diverse sensing phenomena; third, it provides accurate models of both battery power and clock drift effect which have been shown to have a significant impact on sensor network studies; and finally it provides an efficient kernel that allows it to run experiments that provide substantial scalability in both the spatial and temporal contexts.

In this paper we address the effects of environmental mobility, that is, the ambient motion of entities like people and vehicles in the vicinity of wireless communication, on the channel characteristics and wireless network performance. We present a three step process of measurements, modeling and network simulations to quantify the significance of environmental mobility. Our field experiments show that presence of people not only cause deep fades but also distorts the fading distribution. We model the shadowing loss by three knife-edge diffraction model and propose a two-state Markov process channel fading behavior. The models are validated against measurement data and implemented in a network simulator. The models are scalable and incur execution overhead less than 15%. We also show the impact of environment mobility on protocol performance by means of two simulation case studies. We show that MAC layer data rate adaptation behavior is sensitive to environmental mobility and can result in 40% packets being delivered at lower rates. Second study on ad-hoc network performance show the throughput is decreased by 20%. We have identified that with environmental mobility the links are more sensitive to interference and the routes are less stable

No abstract available.

Recent trends in sensor network simulation can be divided between less flexible but accurate emulation based approach and more generic but less detailed network simulator models. We offer an approach with the flexibility of network simulators and provides the accuracy comparable to emulation based approaches. We describe the design and architecture of sensor network simulator which provides a rich suite of following models: sensing stack to model wave and diffusion based sensor channels, an accurate battery model, processor power consumption model, energy consumption model and sensor network based traffic model. We also present our study on the effects of detailed modeling on the performance of higher layer protocols. We describe the impact of using accurate models for battery, processor power consumption and traffic models on the network layer statistics as network lifetime and availability, throughput and routing overhead. Our results show there is high sensitivity of model accuracy on network and application level statistics. In extreme cases we have noticed an inversion of results with our accurate models as compared with previous generation of abstract models.

Mobile computing research has often focused on untethering an in-use computing device, rather than enabling the mobility of the computation task itself. This paper presents an architecture, implementation, and experimental evidence that together validate a new continuous computing concept, application session handoff. The iMASH architecture leverages previous work on proxies, content adaptation, and client awareness to provide a unique, middleware-enable capability for continuous computing. Implementation in both socket- and RPC-based environments shows that very fast, secure session handoff of non-trivial client/server applications across heterogeneous client devices and network is feasible: experiments on a number of applications yielded handoff latencies ranging from 0.5s to 2s.