Yahoo Gemini native advertising marketplace serves billions of impressions daily, to hundreds millions of unique users, and reaches a yearly revenue of hundreds of millions USDs. Powering Gemini native models for predicting advertise (ad) event probabilities, such as conversions and clicks, is OFFSET - a feature enhanced collaborative-filtering (CF) based event prediction algorithm. The predicted probabilities are then used in Gemini native auctions to determine which ads to present for every serving event (impression). Dynamic creative optimization (DCO) is a recent Gemini native product that was launched two years ago and is increasingly gaining more attention from advertisers. The DCO product enables advertisers to issue several assets per each native ad attribute, creating multiple combinations for each DCO ad. Since different combinations may appeal to different crowds, it may be beneficial to present certain combinations more frequently than others to maximize revenue while keeping advertisers and users satisfied. The initial DCO offer was to optimize click-through rates (CTR), however as the marketplace shifts more towards conversion based campaigns, advertisers also ask for a conversion based solution. To accommodate this request, we present a post-auction solution, where DCO ads’ combinations are favored according to their predicted conversion rate (CVR). The predictions are provided by an auxiliary OFFSET based combination CVR prediction model, and used to generate the combination distributions for DCO ad rendering during serving time. An online evaluation of this explore-exploit like solution, via online bucket A/B testing, serving Gemini native DCO traffic, showed a 53.5% CVR lift, when compared to a control bucket serving all combinations uniformly at random. The impressive results demonstrate the ability of this practical yet effective solution to overcome many real-life issues such as data sparsity, reporting delays, trends, and various system constraints. The CVR prediction based DCO system is now fully deployed, serving all Gemini native traffic.

Ad click-though rate prediction (pCTR) is one of the core tasks of online advertising. Driving the pCTR models of Yahoo Gemini native advertising is OFFSET - a feature enhanced collaborative-filtering based event prediction algorithm. Due to data sparsity issues OFFSET models both users and items by mapping their features into a latent space, where the resulting user vector is a non-linear function of the user feature vectors (e.g., age, gender, hour, etc.) which allows pairwise dependencies. This pairwise dependencies concept is also used by other algorithms such as the Field-aware Factorization Machines (FFM). However, both in OFFSET and in FFM, the different pairwise interactions are modeled by latent vectors of constant and equal lengths. When prediction models are used online for serving real traffic, where the total serving model size is often limited, a non uniform representation of the pairwise interactions should be considered in order to maximize the accuracy of the model while consuming the same or even less space. In this work we present a Dynamic Length Factorization Machines (DLFM) algorithm that dynamically optimizes the length of the vectors for each feature interaction during training, while not exceeding a maximal overall latent vector size. After showing good online performance of 1.46% revenue lift and a 2.15% CTR lift, serving Gemini native traffic, the DLFM was pushed into production. Since integrated into production, the DLFM has not only improved the accuracy of the model by optimizing the length of each latent space, but has also reduced the total size of the model by 25%. Although the algorithm was applied to OFFSET, we show that DLFM can be applied to any FFM-like algorithm to optimize its pairwise feature vector lengths. We also present an Educated Model Initialization - a novel mechanism for initializing a new model based on an existing model that has some mutual user features. Using this mechanism, we managed to reduce the training time of our models by more than 90% when compared to an equivalent model that is trained from "scratch".

Real-world content recommendation marketplaces exhibit certain behaviors and are imposed by constraints that are not always apparent in common static offline data sets. One example that is common in ad marketplaces is swift ad turnover. New ads are introduced and old ads disappear at high rates every day. Another example is ad discontinuity, where existing ads may appear and disappear from the market for non negligible amounts of time due to a variety of reasons (e.g., depletion of budget, pausing by the advertiser, flagging by the system, and more). These behaviors sometimes cause the model's loss surface to change dramatically over short periods of time. To address these behaviors, fresh models are highly important, and to achieve this (and for several other reasons) incremental training on small chunks of past events is often employed. These behaviors and algorithmic optimizations occasionally cause model parameters to grow uncontrollably large, or diverge. In this work present a systematic method to prevent model parameters from diverging by imposing a carefully chosen set of constraints on the model's latent vectors. We then devise a method inspired by primal-dual optimization algorithms to fulfill these constraints in a manner which both aligns well with incremental model training, and does not require any major modifications to the underlying model training algorithm.We analyze, demonstrate, and motivate our method on OFFSET, a collaborative filtering algorithm which drives Verizon Media (VZM) native advertising, which is one of VZM's largest and faster growing businesses, reaching a run-rate of many hundreds of millions USD per year. Finally, we conduct an online experiment which shows a substantial reduction in the number of diverging instances, and a significant improvement to both user experience and revenue.

Verizon media (VZM) native advertising serves billions of impressions daily, reaching a yearly run-rate of many hundred of million USD. Driving VZM native models for predicting advertise (ad) event probabilities, such as clicks and conversions, is OFFSET - a feature enhanced collaborative-filtering ( CF) based e vent prediction algorithm. The predicted probabilities are then used in VZM native auctions to determine which ads to present for each serving event. Dynamic creative optimization (DCO) is a new VZM native product that was launched recently and is gaining increasingly more attention from advertisers. The DCO product allows advertisers to provide several assets per each native ad attribute, creating a plurality of combinations for each DCO ad. Since different combinations may appeal to different crowds, it may be beneficial to present certain combinations more frequently than others to maximize revenue. Inspired by the success of our Carousel asset optimization product, we present a post-auction successive elimination based approach for ranking DCO combinations according to their measured click through rates (CTR). This reinforcement learning multi-arm bandit like solution was evaluated during an online beta test phase done with selected advertisers, showing 21.8% CTR and 21.5% revenue lifts over a control bucket serving all combinations uniformly at random. The good performance of our DCO product attracts advertisers and it already shows a yearly run-rate of several million USD in revenue.

Verizon Media native advertising (also known as Yahoo Gemini native) serves billions of ad impressions daily, reaching several hundreds of millions USD in revenue yearly. Although we strive to provide the best experience for our users, there will always be some users that dislike our ads in certain cases. To address these situations Gemini native platform provides an ad close mechanism that enables users to close ads that they dislike and also to provide a reasoning for their action. Surprisingly, users do care about their ad experience and their engagement with the ad close mechanism is quite significant. While the ad close rate (ACR) is lower than the click through rate (CTR), they are of the same order of magnitude, especially on Yahoo mail properties. Since ad close events indicate bad user experience caused mostly by poor ad quality, we would like to exploit the ad close signals to improve user experience and reduce the number of ad close events while maintaining a predefined total revenue loss.

Yahoo's native advertising marketplace (also known as Gemini native) serves billions of ad impressions daily, reaching many hundreds of millions USD in yearly revenue. Driving Gemini native models that are used to predict ad click probability (pCTR) is OFFSET - a feature enhanced collaborative-filtering (CF) based event prediction algorithm. While some of the user features used by OFFSET have high coverage, other features, especially those based on click patterns, suffer from extremely low coverage. In this work, we present a framework that simplifies complex interactions between users and other entities in a bipartite graph. The one mode projection of this bipartite graph onto users represents a user similarity network, allowing us to quantify similarities between users. This network is combined with existing user features to create an enhanced feature set. In particular, we describe the implementation and performance of our framework using user Internet browsing data (e.g., visited pages URLs) to enhance the user category feature. Using our framework we effectively increase the feature coverage by roughly 15%. Moreover, online results evaluated on 1% of Gemini native traffic show that using the enhanced feature increases revenue by almost 1% when compared to the baseline operating with the original feature, which is a substantial increase at scale.

The item cold-start problem is inherent to collaborative filtering (CF) recommenders where items and users are represented by vectors in a latent space. It emerges since CF recommenders rely solely on historical user interactions to characterize their item inventory. As a result, an effective serving of new and trendy items to users may be delayed until enough user feedback is received, thus, reducing both users' and content suppliers' satisfaction. To mitigate this problem, many commercial recommenders apply random exploration and devote a small portion of their traffic to explore new items and gather interactions from random users. Alternatively, content or context information is combined into the CF recommender, resulting in a hybrid system. Another hybrid approach is to learn a mapping between the item attribute space and the CF latent feature space, and use it to characterize the new items providing initial estimates for their latent vectors.