Relevance ranking is a key component of many search engines, including the Tweet search engine at Twitter. Users often use Tweet search to discover live discussions and different voices on trending topics or recent events. Tweet search is thus unique due to its focus on real-time content, where both the retrieved content and queries change drastically on an hourly basis. Another important property of Tweet search is that its relevance ranking takes the social endorsements from other users into account, e.g., "likes" and "retweets", which is different from mainly relying on clicks as implicit feedback. The relevance ranking of Tweet search is also subject to strict latency constraints, because every second, a large amount of Tweets are posted and indexed, while tens of thousands of queries are issued to search posted Tweets. Considering the above properties and constraints, we present a relevance ranking system for Tweet search addressing all these challenges at Twitter. We first discuss the formation of the relevance ranking pipeline, which consists of a series of ranking models. We then present the methodology for training the models and the various groups of features we use, including real-time and personalized features. We also investigate approaches of achieving unbiased model training and building up automatic online tuning of system parameters. Experiments using online A/B testing demonstrate the effectiveness of the proposed approaches and we have deployed the proposed relevance ranking system in production for more than three years.

The Web abounds with dyadic data that keeps increasing by every single second. Previous work has repeatedly shown the usefulness of extracting the interaction structure inside dyadic data [21, 9, 8]. A commonly used tool in extracting the underlying structure is the matrix factorization, whose fame was further boosted in the Netflix challenge [26]. When we were trying to replicate the same success on real-world Web dyadic data, we were seriously challenged by the scalability of available tools. We therefore in this paper report our efforts on scaling up the nonnegative matrix factorization (NMF) technique. We show that by carefully partitioning the data and arranging the computations to maximize data locality and parallelism, factorizing a tens of millions by hundreds of millions matrix with billions of nonzero cells can be accomplished within tens of hours. This result effectively assures practitioners of the scalability of NMF on Web-scale dyadic data.

Value-added content resiliency services help to migrate the burden of resiliency provisioning and maintenance from service users, especially home users and small/medium organizations, who have difficulty in handling correlated failures that impact large areas. For service providers to achieve business success, however, the cost-effectiveness of their resiliency strategies is critical: while the content resiliency requirements specified by the end users have to be satisfied, excessive preventive operation costs caused by the over-reaction to potential risks should be avoided. In this paper, we study the problem of cost-effective configuration in content resiliency service networks under both independent and geographically correlated failures. We propose a new approach to modeling correlated failures in a representable, quantifiable and consistent way, which allows for both quantified availability guarantees and aggressive prevention cost optimization. We then formulate the costeffective configuration problem in content resiliency services and develop both real optimal and heuristic-based algorithms for solving the problem. Our experiments show that with the help of good models for correlated failures, the operation cost of the services can be significantly reduced without impairing the user-specified content resiliency.

Abstract — The routing infrastructure of the Internet has become resistant to fundamental changes and the use of overlay networks has been proposed to provide additional flexibility and control. One of the most prominent configurable components of an overlay network is its topology, which can be dynamically reconfigured to accommodate communication requirements that vary over time. In this paper, we study the problem of determining dynamic topology reconfiguration for service overlay networks with dynamic communication requirement, and the ideal goal is to find the optimal reconfiguration policies that can minimize the potential overall cost of using an overlay. We start by observing the properties of the optimal reconfiguration policies through studies on small systems and find structures in the optimal reconfiguration policies. Based on these observations, we propose heuristic methods for constructing different flavors of reconfiguration policies, i.e., never-change policy, always-change policy and cluster-based policies, to mimic and approximate the optimal ones. Our experiments show that our policy construction methods are applicable to large systems and generate policies with good performance. Our work does not only provide solutions to practical overlay topology design problems, but also provides theoretical evidence for the advantage of overlay network due to its configurability. I.

We consider heterogeneous group communication in which users have different preference for information related to various topics. The problem of disseminating such information to interested set of users in an overlay network is challenging when both low end-to-end latency and low incurred overlay traffic are desired. We investigate heuristic based overlay topology generation algorithm that exploits both network proximity in underlying network and similarity between users' preferences to reduce both the overlay latency and the amount of data forwarding done by nodes that do not prefer the data. Our results show that while these requirements conflict, an application specific tradeoff can be made to improve the overall system performance.

One problem in securing group communication is the scalability of group key management in dynamic multicast sessions. The main challenge arises when a member leaves the multicast session and a rekeying of the group is required to prevent the departing member from accessing the information being multicast after they leave. Recent research developed the logical key hierarchy (LKH) protocol which uses a tree structuring of receivers and requires O(log(n)) rekeying messages when a member leaves. It has also been demonstrated that /spl Omega/(log(n)) is the best one can achieve if strict confidentiality and non-collusion are required. While strict non-collusion is required for some highly sensitive data, we argue that some commercial content delivery applications will be extremely cost sensitive and willing to tolerate some small level of collusion. In this paper we consider the question of how one might trade off the message cost of rekeying with some increased vulnerability to collusion. We consider a range of protocols. In one extreme is LKH which is completely immune from collusion. On the other extreme is a protocol based on the linear ordering of receivers (LORE), which requires O(1) messages for rekeying but in which any two receivers can collude. We describe a scheme using a hybrid structuring of receivers (HySOR) which is tunable between the LKH and LORE extremes and by which one can trade off some vulnerability to collusion for a decrease in rekeying message cost. We provide analytical as wen as simulation results to investigate the performance of HySOR and its tunability along the collusion/scalability spectrum.

Real-world traffic traces are crucial for Internet research, but only a very small percentage of traces collected are made public. One major reason why traffic trace owners hesitate to make the traces publicly available is the concern that confidential and private information may be inferred from the trace. We focus on the problem of anonymizing IP addresses in a trace. More specifically, we are interested in prefix-preserving anonymization in which the prefix relationship among IP addresses is preserved in the anonymized trace, making such a trace usable in situations where prefix relationships are important. The goal of our work is two fold. First, we develop a cryptography-based, prefix-preserving anonymization technique that is provably as secure as the existing well-known TCPdpriv scheme, and unlike TCPdpriv, provides consistent prefix-preservation in large scale distributed setting. Second, we evaluate the security properties inherent in all prefix-preserving IP address anonymization schemes (including TCPdpriv). Through the analysis of Internet backbone traffic traces, we investigate the effect of some types of attacks on the security of any prefix-preserving anonymization algorithm. We also derive results for the optimum manner in which an attack should proceed, which provides a bound on the effectiveness of attacks in general.