The goal of Jobs Marketplace at LinkedIn is to match members to promoted job postings such that both job posters' ROI is optimized (amount of money spent per job clicks and applications) and the members are presented with relevant jobs that they are interested in and qualified for. This is achieved via a first-price auction mechanism where each job provides a bid for the member that comes to the job recommendations page. This bid depends on the match of the member to the job, as well as the daily budget that remains for the job, and its capability to spend it via clicks (e.g. some jobs might have more demand and have it easier to spend their budgets via clicks than others). In such a scheme, budget pacing, i.e. the capability of a job to spend its daily budget evenly, or according to a preset plan, is extremely important towards efficient utilization of its budget via reaching a higher number of candidates, and obey a variety of spending plans optimizing for different events such as clicks and applications.

Artificial Intelligence is increasingly playing an integral role in determining our day-to-day experiences. Moreover, with proliferation of AI based solutions in areas such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI are far-reaching. The dominant role played by AI models in these domains has led to a growing concern regarding potential bias in these models, and a demand for model transparency and interpretability. In addition, model explainability is a prerequisite for building trust and adoption of AI systems in high stakes domains requiring reliability and safety such as healthcare and automated transportation, and critical industrial applications with significant economic implications such as predictive maintenance, exploration of natural resources, and climate change modeling.

We present a framework for quantifying and mitigating algorithmic bias in mechanisms designed for ranking individuals, typically used as part of web-scale search and recommendation systems. We first propose complementary measures to quantify bias with respect to protected attributes such as gender and age. We then present algorithms for computing fairness-aware re-ranking of results. For a given search or recommendation task, our algorithms seek to achieve a desired distribution of top ranked results with respect to one or more protected attributes. We show that such a framework can be tailored to achieve fairness criteria such as equality of opportunity and demographic parity depending on the choice of the desired distribution. We evaluate the proposed algorithms via extensive simulations over different parameter choices, and study the effect of fairness-aware ranking on both bias and utility measures. We finally present the online A/B testing results from applying our framework towards representative ranking in LinkedIn Talent Search, and discuss the lessons learned in practice. Our approach resulted in tremendous improvement in the fairness metrics (nearly three fold increase in the number of search queries with representative results) without affecting the business metrics, which paved the way for deployment to 100% of LinkedIn Recruiter users worldwide. Ours is the first large-scale deployed framework for ensuring fairness in the hiring domain, with the potential positive impact for more than 630M LinkedIn members.

Talent Search systems aim to recommend potential candidates who are a good match to the hiring needs of a recruiter expressed in terms of the recruiter's search query or job posting. Past work in this domain has focused on linear and nonlinear models which lack preference personalization in the user-level due to being trained only with globally collected recruiter activity data. In this paper, we propose an entity-personalized Talent Search model which utilizes a combination of generalized linear mixed (GLMix) models and gradient boosted decision tree (GBDT) models, and provides personalized talent recommendations using nonlinear tree interaction features generated by the GBDT. We also present the offline and online system architecture for the productionization of this hybrid model approach in our Talent Search systems. Finally, we provide offline and online experiment results benchmarking our entity-personalized model with tree interaction features, which demonstrate significant improvements in our precision metrics compared to globally trained non-personalized models.

Previous efforts in recommendation of candidates for talent search followed the general pattern of receiving an initial search criteria and generating a set of candidates utilizing a pre-trained model. Traditionally, the generated recommendations are final, that is, the list of potential candidates is not modified unless the user explicitly changes his/her search criteria. In this paper, we are proposing a candidate recommendation model which takes into account the immediate feedback of the user, and updates the candidate recommendations at each step. This setting also allows for very uninformative initial search queries, since we pinpoint the user's intent due to the feedback during the search session. To achieve our goal, we employ an intent clustering method based on topic modeling which separates the candidate space into meaningful, possibly overlapping, subsets (which we call intent clusters) for each position. On top of the candidate segments, we apply a multi-armed bandit approach to choose which intent cluster is more appropriate for the current session. We also present an online learning scheme which updates the intent clusters within the session, due to user feedback, to achieve further personalization. Our offline experiments as well as the results from the online deployment of our solution demonstrate the benefits of our proposed methodology.

Talent search and recommendation systems at LinkedIn strive to match the potential candidates to the hiring needs of a recruiter or a hiring manager expressed in terms of a search query or a job posting. Recent work in this domain has mainly focused on linear models, which do not take complex relationships between features into account, as well as ensemble tree models, which introduce non-linearity but are still insufficient for exploring all the potential feature interactions, and strictly separate feature generation from modeling. In this paper, we present the results of our application of deep and representation learning models on LinkedIn Recruiter. Our key contributions include: (i) Learning semantic representations of sparse entities within the talent search domain, such as recruiter ids, candidate ids, and skill entity ids, for which we utilize neural network models that take advantage of LinkedIn Economic Graph, and (ii) Deep models for learning recruiter engagement and candidate response in talent search applications. We also explore learning to rank approaches applied to deep models, and show the benefits for the talent search use case. Finally, we present offline and online evaluation results for LinkedIn talent search and recommendation systems, and discuss potential challenges along the path to a fully deep model architecture. The challenges and approaches discussed generalize to any multi-faceted search engine.

In this talk, we present the overall system design and architecture, the challenges encountered in practice, and the lessons learned from the production deployment of the talent search and recommendation systems at LinkedIn. By presenting our experiences of applying techniques at the intersection of recommender systems, information retrieval, machine learning, and statistical modeling in a large-scale industrial setting and highlighting the open problems, we hope to stimulate further research and collaborations within the SIGIR community.

The field of online advertising, in essence, deals with the problem of presenting ads to online users in the most appropriate contexts to achieve a multitude of advertiser goals. A vast amount of work in online advertising has been focused on optimizing banner display advertising campaigns where the main goal lies in direct response metrics, often as clicks or conversions. In this paper, we explore the newly popularized space of online video advertising, where brand recognition is the key focus. We propose a framework based on a feedback mechanism where we optimize multiple video specific performance indicators while making sure the delivery constraints (budget and user reach) of advertisers are satisfied. While our main focus is on improving metrics such as engagement (amount of view time), and viewability (whether a campaign is within eyesight of a user), we also discuss the possibilities of expanding to other metrics. We demonstrate the benefit of our framework via empirical results in multiple real-world advertising campaigns. To the best of our knowledge, this is the first paper that deals with the unique challenges arising from the nature of online video advertising.

Real-Time Bidding allows an advertiser to purchase media inventory through an auction system that unfolds in the order of milliseconds. Media providers are increasingly being integrated into such programmatic buying platforms. It is typical for a contemporary Real-Time Bidding system to receive millions of bid requests per second at peak time, and have a large portion of these to be irrelevant to any advertiser. Meanwhile, given a valuable bid request, tens of thousands of advertisements might be qualified for scoring. We present our efforts in building selection models for both bid requests and advertisements to handle this scalability challenge. Our bid request model treats the system load as a hierarchical resource allocation problem and directs traffic based on the estimated quality of bid requests. Next, our exploration/exploitation advertisement model selects a limited number of qualified advertisements for thorough scoring based on the expected value of a bid request to the advertiser given its features. Our combined bid request and advertisement model is able to win more auctions and bring more value to clients by stabilizing the bidding pipeline. We empirically show that our deployed system is capable of handling 5x more bid requests.

In digital advertising, advertisers want to reach the right audience over media channels such as display, mobile, video, or social at the appropriate cost. The right audience for an advertiser consists of existing customers as well as valuable prospects, those that can potentially be turned into future customers. Identifying valuable prospects is called the audience extension problem because advertisers find new customers by extending the desirable criteria for their starting point, which is their existing audience or customers. The complexity of the audience extension problem stems from the difficulty of defining desirable criteria objectively, the number of desirable criteria (such as similarity, diversity, performance) to simultaneously satisfy, and the expected runtime (a few minutes) to find a solution over billions of cookie-based users. In this paper, we formally define the audience extension problem, propose an algorithm that extends a given audience set efficiently under multiple desirable criteria, and experimentally validate its performance. Instead of iterating over individual users, the algorithm takes in Boolean rules that define the seed audience and returns a new set of Boolean rules that corresponds to the extended audience that satisfy the multiple criteria.

Modeling of the mobility patterns arising in computer networks requires a compact and faithful representation of the mobility data collected from observations and measurements of the relevant network applications. This data can range from the information on the mobility of the agents that are being monitored by a wireless network to mobility information of nodes in mobile network applications. In this paper, we examine the use of probabilistic context-free grammars as the modeling framework for such data. We present a fast algorithm for deriving a concise probabilistic context-free grammar from the given training data. The algorithm uses an evaluation metric based on Bayesian formula for maximizing grammar a posteriori probability given the training data. We describe the application of this algorithm in two mobility modeling domains: 1) recognizing mobility patterns of monitored agents in different event data sets collected by sensor networks, and 2) modeling and generating node movements in mobile networks. We also discuss the model's performance in simulations utilizing both synthetic and real-world mobility traces.

Service modeling and service composition are software architecture paradigms that have been used extensively in web services where there is an abundance of resources. They mainly capture the idea that advanced functionality can be realized by combining a set of primitive services provided by the system. Many efforts in web services domain focused on detecting the initial composition, which is then followed for the rest of service operation. In sensor networks, however, communication among nodes is error-prone and unreliable, while sensor nodes have constrained resources. This dynamic environment requires a continuous adaptation of the composition of a complex service. In this paper, we first propose a graph-based formulation for modeling sensor services that maps to the operational model of sensor networks and is amenable to analysis. Based on this model, we formulate the process of sensor service composition as a cost-optimization problem and show that it is NP-complete. Two heuristic methods are proposed to solve the composition problem: the top-down and the bottom-up approaches. We discuss centralized and distributed implementations of these methods. Finally, using ns-2 simulations, we evaluate the performance and overhead of our proposed methods.

This paper introduces a novel method of generating mobility traces based on Probabilistic Context Free Grammars (PCFGs). A PCFG is a generalization of a context free grammar in which each production rule is augmented with a probability with which this production is applied during sentence generation. A concise PCFG can be inferred from the given real world trace collected from the actual mobile node behaviors. The resulting grammar can be used to generate sequences of arbitrary length mimicking the mobile node behavior. This is important when new protocol designs for mobile networks are tested by simulation. In the paper, we describe the methods developed to construct such grammars from training data (mobility history). We also discuss how to generate the synthetic data with an already constructed grammar. We present the experimental results on two real data sets, measuring similarity of the actual traces with the synthetic ones. We compare our grammar based method to a 2-level Markov Model based trace generation method. The results demonstrate that the grammar based approach works as an excellent compression method for the actual data. On many metrics, the synthetic data generated from the PCFG match the training data much better than the one generated by the Markov Model.

In a delay tolerant network (DTN), nodes are connected intermittently and the future node connections are mostly unknown. Since in these networks, a fully connected path from source to destination is unlikely to exist, message delivery relies on opportunistic routing. However, effective forwarding based on a limited knowledge of contact behavior of nodes is challenging. Most of the previous studies looked at only the pairwise node relations to decide routing. In contrast, in this paper, we analyze the correlation between the meetings of each node with other nodes and focus on the utilization of this correlation for efficient routing of messages. We introduce a new metric called conditional intermeeting time, which computes the average intermeeting time between two nodes relative to a meeting with a third node using only the local knowledge of the past contacts. Then, we show how we can utilize the proposed metric on the existing DTN routing protocols to improve their performance. For shortest-path based routing protocols in DTNs, we propose to route messages over conditional shortest paths in which the link cost between nodes are defined by conditional intermeeting times. Moreover, for metric-based forwarding protocols, we propose to use conditional intermeeting time as an additional delivery metric while making forwarding decisions of messages. Our trace-driven simulations on three different datasets show that the modified algorithms perform better than the original ones.

Delay tolerant networks are characterized by the sporadic connectivity between their nodes and therefore the lack of stable end-to-end paths from source to destination. Since the future node connections are mostly unknown in these networks, opportunistic forwarding is used to deliver messages. However, making effective forwarding decisions using only the network characteristics (i.e. average intermeeting time between nodes) extracted from contact history is a challenging problem. Based on the observations about human mobility traces and the findings of previous work, we introduce a new metric called conditional intermeeting time, which computes the average intermeeting time between two nodes relative to a meeting with a third node using only the local knowledge of the past contacts. We then look at the effects of the proposed metric on the shortest path based routing designed for delay tolerant networks. We propose Conditional Shortest Path Routing (CSPR) protocol that routes the messages over conditional shortest paths in which the cost of links between nodes is defined by conditional intermeeting times rather than the conventional intermeeting times. Through trace-driven simulations, we demonstrate that CSPR achieves higher delivery rate and lower end-to-end delay compared to the shortest path based routing protocols that use the conventional intermeeting time as the link metric.

Modern military and civilian surveillance applications should provide end users with the high level representation of events observed by sensors rather than with the raw data measurements. Hence, there is a need for a system that can infer higher level meaning from collected sensor data. We demonstrate that probabilistic context free grammars (PCFGs) can be used as a basis for such a system. To recognize events from raw sensor network measurements, we use a PCFG inference method based on Stolcke (1994) and Chen(1996). We present a fast algorithm for deriving a concise probabilistic context free grammar from the given observational data. The algorithm uses an evaluation metric based on Bayesian formula for maximizing grammar a posteriori probability given the training data. We also present a real-world scenario of monitoring a parking lot and the simulation based on this scenario. We described the use of PCFGs to recognize events in the results of such a simulation. We finally demonstrate the deployment details of such an event recognition system.