Evaluating the quality of search systems traditionally requires a significant number of human relevance annotations. In recent times, several systems have explored the usage of Large Language Models (LLMs) as automated judges for this task while their inherent biases prevent direct use for metric estimation. We present a statistical framework extending Prediction-Powered Inference (PPI) that combines minimal human annotations with LLM judgments to produce reliable estimates of metrics which require sub-instance annotations. Our method requires as few as 100 human-annotated queries and 10,000 unlabeled examples, reducing annotation requirements significantly compared to traditional approaches. We formulate our proposed framework (PRECISE) for inference of relevance uplift for an LLM-based query reformulation application, extending PPI to sub-instance annotations at the query-document level. By reformulating the metric-integration space, we reduced the computational complexity from O(2^|C|) to O(2^K), where |C| represents corpus size (in order of millions). Detailed experiments across prominent retrieval datasets demonstrate that our method reduces the variance of estimates for the business-critical Precision@K metric, while effectively correcting for LLM bias in low-resource settings.

In-context learning (ICL) has emerged as a powerful paradigm for adapting Large Language Models (LLMs) to specific tasks without parameter updates. While various strategies exist for selecting relevant ICL exemplars from a labeled pool, the fundamental challenge of constructing this high-quality pool remains largely unexplored, especially for new tasks or domains with limited labeled data. We present IclForge, a novel active learning framework that efficiently selects informative examples from unlabeled datasets to be annotated and included in the ICL pool. Unlike traditional active learning methods that optimize for individual example informativeness, IclForge explicitly considers the interdependence of examples within the ICL context. Through extensive experiments across diverse datasets and LLM architectures, we show that IclForge outperforms standard active learning baselines by +180-450 basis points while requiring 50% fewer annotations. Our framework is complementary to existing ICL selection strategies and extends naturally to generative applications, which we demonstrate through experiments on Math Word Problem (MWP) tasks. These results highlight IclForge's effectiveness in constructing high-quality ICL exemplar pools in resource-constrained scenarios.

With increasing transistor density, modern heterogeneous embedded processors often exhibit high temperature gradients due to complex application scheduling scenarios which may have missed design considerations. In many use cases, off-chip ”active” cooling solutions are considered prohibitive in such reduced form factors. Core frequency throttling by existing dynamic thermal management techniques often compromises the Quality-of-Service (QoS) and violates real-time deadlines. This necessitates the adoption of intelligent resource management that simultaneously manages both thermal and latency performance. Coupled with the complexity of modern heterogeneous multi-cores, the periodic application updates that cater to ever-changing user requirements often render model-driven thermal-aware resource allocation approaches unsuitable for heterogeneous multi-core systems. For such application-architecture scenarios, we propose a novel self-learning based resource manager using Reinforcement Learning that intelligently manipulates core frequencies and task set mappings to fulfill thermal and latency objectives. Our framework employs a data-driven system modeling technique using Gaussian Process Regression to enable efficient offline training of this learning-based resource manager to avoid challenges associated with initial online training. We evaluate the approach on a heterogeneous embedded CPU-GPU platform with real workloads and observe a significant reduction in peak operating temperature when compared to the default onboard frequency governor as well as other learning-based state-of-the-art approaches.

Automatic extraction of attribute preferences from search queries is a critical problem in providing accurate product recommendations to customer. The task becomes even more challenging in cold-start settings where we do not have any supervised/labelled data available to train ML models. In this work, we implement a novel dataset generation pipeline (LLM-API) that leverages Large Language Models (LLMs), search logs and proprietary product information data from an ecommerce website to create a high quality dataset. Our proposed pipeline of LLM-API is robust as it can generalize to any product category with minimal changes in the LLM prompts. For the problem of converting product search queries to API calls we propose a multi-task schema generator model which we train on our generated dataset. Experiments on an internal test set reveals that our proposed model achieves an improvement of ≈9.6% and ≈5% in Exact Match and Micro-F1 respectively, over competitive baselines. Benchmarking our approach on public test set of search queries further reveals a gain of ≈8.6% and ≈10.5% in Exact Match and Micro-F1. We further demonstrate that our approach outperforms a state-of-the-art LLM (Claude) applied on our task using few-shot prompting and CoT reasoning, while at the same time, achieves improvement in inference latency.

For many recommender systems, the primary data source is a historical record of user clicks. The associated click matrix is often very sparse, as the number of users × products can be far larger than the number of clicks. Such sparsity is accentuated in cold-start settings, which makes the efficient use of metadata information of paramount importance. In this work, we propose a simple approach to address cold-start recommendations by leveraging content metadata, Metadata Alignment for cold-start Recommendation (MARec). We show that this approach can readily augment existing matrix factorization and autoencoder approaches, enabling a smooth transition to top performing algorithms in warmer set-ups. Our experimental results indicate three separate contributions: first, we show that our proposed framework largely beats SOTA results on 4 cold-start datasets with different sparsity and scale characteristics, with gains ranging from +8.4% to +53.8% on reported ranking metrics; second, we provide an ablation study on the utility of semantic features, and proves the additional gain obtained by leveraging such features ranges between +46.8% and +105.5%; and third, our approach is by construction highly competitive in warm set-ups, and we propose a closed-form solution outperformed by SOTA results by only 0.8% on average.

Click-through Rate (CTR) module is the foundation block of recommendation system and used for search, content selection, advertising, video streaming etc. CTR is modelled as a classification problem and extensive research is done to improve the CTR models. However, uncertainty method for these models are still an unexplored area. In this work we analyse popular uncertainty methods in the context of recommendation system. We found that popular uncertainty models fails to capture the predictive uncertainty of the CTR model that exist unique to the recommendation models and is not prevalent in the traditional classification models. We empirical show why a different uncertainty measure is required for the recommendation system CTR prediction models. We propose PRU (Predictive Relevance Uncertainty), a single forward pass uncertainty approach for a sample as a distance from the predictive relevance samples of the training data. We show the efficacy of the proposed predictive relevance uncertainty (PRU) on selective prediction. Further, we demonstrate the utility of the proposed framework on the downstream task of OOD detection and active learning while maintaining the latency of a single pass deterministic model.

Growth of e-commerce has enabled the creation of thousands of small-scale brands. However, these brands lack information on a) what new products to develop and b) how to refine existing products to improve on business metrics. We present a comprehensive Product Design Insights and Guidance service (named PRODIGY) that mines product attributes data available on e-commerce platforms and surface insights on a) new product development and b) product refinement. Our core contribution is a novel demand forecasting model for product designs based on a notable extension of the recently proposed FTTransformer architecture combined with a self-supervised pre-training task, akin to Masked Language Modeling (MLM) objective. For the product refinement use-case, we present a novel algorithm by embedding the design search in a data-density approximator, namely Conditional Variational Autoencoder. We run a thorough and comprehensive set of experiments and establish that PRODIGY achieves significant improvement in demand prediction as compared to state-of-the-art alternatives. Finally, we present our findings from an online experiment where PRODIGY helps to launch new products with +20% lift in sales and +1.3% lift in product ratings.

Rewards-based programs are popular within e-commerce online stores, with the goal of providing serendipitous incentives to delight customers. These rewards (or incentives) could be in the form of cashback, free-shipping or discount coupons on purchases within specific categories. The success of such programs relies on their ability to identify relevant rewards for customers, from a wide variety of incentives available on the online store. Estimating the likelihood of a customer redeeming an incentive is challenging due to 1) data sparsity: relatively rare occurrence of coupon redemptions as compared to issuances, and 2) delayed feedback: customers taking time to redeem, resulting in inaccurate model refresh, compounded by data drift due to new customers and coupons.

Learning individual level treatment effects from observational data is a problem of growing interest. For instance, inferring the effect of delivery promises on purchase of products on an e-commerce site or selecting the most effective treatment for a specific patient. Although the scenarios where we want to estimate the treatment effects in presence of multiple treatments is quite common in real life, most existing works related to individual treatment effect (ITE) are focused primarily on binary treatments and do not have a natural extension to the multi-treatment scenarios. In this paper we present MEMENTO ? a methodology and a framework to estimate individual treatment effect for multi-treatment scenarios, where the treatments are discrete and finite. Our approach is based on obtaining matching representations of the confounders for the various treatment types. This is achieved through minimization of an upper bound on the sum of factual and counterfactual losses. Experiments on real and semi-synthetic datasets show that MEMENTO is able to outperform known techniques for multi-treatment scenarios by close to 10% in certain use-cases. The proposed framework has been deployed for the problem of identifying minimum order quantity of a product in Amazon in an emerging marketplace and has re- sulted in a 4.7% reduction in shipping costs as proved from an A/B experiment.

Speed of delivery is critical for the success of e-commerce platforms. Faster delivery promise to the customer results in increased conversion and revenue. There are typically two mechanisms to control the delivery speed - a) replication of products across warehouses, and b) air-shipping the product. In this paper, we present a machine learning based framework to recommend air-shipping eligibility for products. Specifically, we develop a causal inference framework (referred to as Air Shipping Recommendation or ASPIRE) that balances the trade-off between revenue or conversion and delivery cost to decide whether a product should be shipped via air. We propose a doubly-robust estimation technique followed by an optimization algorithm to determine air eligibility of products and calculate the uplift in revenue and shipping cost.

The ability to accurately pinpoint the location of an event (e.g. loss, fault or bug) is of fundamental requirement in many systems. While we have state-of-the-art models to predict likelihood of an outcome, being able to pinpoint to the entity responsible for the outcome is also important. For example, in an e-commerce setup, a lost package detection system needs to infer the reason or location (delivery station, sort center, trucks) in case of a missing item, a network management system would like to diagnose nodes that are faulty based on end-end packet flow traces or a compiler needs to point out the exact location of a code that is erroneous. In this paper, we present an Attention based neural architecture for entity localization to accurately pinpoint the location of package loss in delivery network and bugs in erroneous programs. Our model performs well in scenarios where there is no annotation/ground truth for entities for localization. It can also adapt itself if annotations/ground truth is available for even a subset of entities by leveraging semi-supervision. The core of our model is a ladder-style architecture that helps us achieve state-of-the-art performance in both entity localization and detection. Further, to show the generality of our approach, we demonstrate its performance on a bug localization task for software programs. On a publicly available data-set, our solution outperforms the state-of-the-art technique by a significant margin.

Modern data intensive Cyber-physical Systems ubiquitously employ heterogeneous multiprocessor systems-on chips (MPSoCs) for real-time sensing, computation, and actuation. The low foot-print of such SoCs often leads to high operating temperatures beyond acceptable limits. In this context, conventional thermal management techniques such as Operating System (OS) governed frequency scaling result in drastic degradation of the quality of experience and violation of real-time requirements. In this work, we propose an analytical thermal model for heterogeneous CPU-GPU embedded platforms and demonstrate a Model Predictive Control (MPC) based scheduling strategy with a novel heuristics-based optimization technique that leverages information about future kernels to judiciously choose suitable task mapping options for minimization of the platform's peak (or maximum) temperature to prolong chip's life span while adhering to real-time performance requirements. To the best of our knowledge, this is the first work that considers future awareness along with a variety of online task mapping control actions such as partitioning, migration, and frequency tuning in the context of thermal management in heterogeneous CPU-GPU embedded platforms. We evaluate the proposed heterogeneous framework on an Odroid-XU4 board using OpenCL based workloads and demonstrate its effectiveness in reducing the platform peak temperature.

Online service platforms (OSPs), such as search engines, news-websites, ad-providers, etc., serve highly personalized content to the user, based on the profile extracted from her history with the OSP. In this paper, we capture OSP's personalization for an user in a new data structure called the personalization vector (?), which is a weighted vector over a set of topics, and present efficient algorithms to learn it.

In a team formation problem, one is required to find a group of users that can match the requirements of a collaborative task. Example of such collaborative tasks abound, ranging from software product development to various participatory sensing tasks in knowledge creation. Due to the nature of the task, team members are often required to work on a co-operative basis. Previous studies [1, 2] have indicated that co-operation becomes effective in presence of social connections. Therefore, effective team selection requires the team members to be socially close as well as a division of the task among team members so that no user is overloaded by the assignment. In this work, we investigate how such teams can be formed on a social network.

Viral Marketing, the idea of exploiting social interactions of users to propagate awareness for products, has gained considerable focus in recent years. One of the key issues in this area is to select the best seeds that maximize the influence propagated in the social network. In this paper, we define the seed selection problem (called t-Influence Maximization, or t-IM) for multiple products. Specifically, given the social network and t products along with their seed requirements, we want to select seeds for each product that maximize the overall influence. As the seeds are typically sent promotional messages, to avoid spamming users, we put a hard constraint on the number of products for which any single user can be selected as a seed. In this paper, we design two efficient techniques for the t-IM problem, called Greedy and FairGreedy. The Greedy algorithm uses simple greedy hill climbing, but still results in a 1/3-approximation to the optimum. Our second technique, FairGreedy, allocates seeds with not only high overall influence (close to Greedy in practice), but also ensures fairness across the influence of different products. We also design efficient heuristics for estimating the influence of the selected seeds, that are crucial for running the seed selection on large social network graphs. Finally, using extensive simulations on real-life social graphs, we show the effectiveness and scalability of our techniques compared to existing and naive strategies.

In this paper, we develop a framework for achieving scalable and communication-efficient dissemination of content in pub/sub systems. To maximize communication sharing across subscriptions, our routing framework groups subscriptions based on similarity, and transmits content matching one or more subscriptions in a group over a single dissemination tree for the group. We develop a cost model that uses published content samples in conjunction with the knowledge of consumer subscriptions to estimate the communication cost of a set of routing trees for subscription groups. The problem of computing a communication-optimal set of routing trees is then formulated as an optimization problem that seeks to find trees with the minimum cost. It turns out that the problem of computing a minimum-cost tree for a subscription group is a new generalization of the well-known Steiner tree problem, and an interesting problem in its own right. We develop an approximation algorithm that uses low-stretch spanning trees to compute a tree whose communication cost is within a polylogarithmic factor of the optimum. We use this to compute trees for various subscription- grouping configurations generated using a greedy clustering strategy, and select the one with the lowest cost. Our experimental study demonstrates the effectiveness of our content-aware routing approach compared to traditional routing based on content oblivious spanning trees.

Modern communication networks are vulnerable to attackers who send unsolicited messages to innocent users, wasting network resources and user time. Some examples of such attacks are spam emails, annoying tele-marketing phone calls, viral marketing in social networks, etc. Existing techniques to identify these attacks are tailored to certain specific domains (like email spam filtering), but are not applicable to a majority of other networks. We provide a generic abstraction of such attacks, called the Random Link Attack (RLA), that can be used to describe a large class of attacks in communication networks. In an RLA, the malicious user creates a set of false identities and uses them to communicate with a large, random set of innocent users. We mine the social networking graph extracted from user interactions in the communication network to find RLAs. To the best of our knowledge, this is the first attempt to conceptualize the attack definition, applicable to a variety of communication networks. In this paper, we formally define RLA and show that the problem of finding an RLA is NP-complete. We also provide two efficient heuristics to mine subgraphs satisfying the RLA property; the first (GREEDY) is based on greedy set-expansion, and the second (TRWALK) on randomized graph traversal. Our experiments with a real-life data set demonstrate the effectiveness of these algorithms.

Regular Expression (RE) matching has important applications in the areas of XML content distribution and network security. In this paper, we present the end-to-end design of a high performance RE matching system. Our system combines the processing efficiency of Deterministic Finite Automata (DFA) with the space efficiency of Non-deterministic Finite Automata (NFA) to scale to hundreds of REs. In experiments with real-life RE data on data streams, we found that a bulk of the DFA transitions are concentrated around a few DFA states. We exploit this fact to cache only the frequent core of each DFA in memory as opposed to the entire DFA (which may be exponential in size). Further, we cluster REs such that REs whose interactions cause an exponential increase in the number of states are assigned to separate groups -- this helps to improve cache hits by controlling the overall DFA size.

A k-set structure over data streams is a bounded-space data structure that supports stream insertion and deletion operations and returns the set of (item, frequency) pairs in the stream, provided, the number of distinct items in the stream does not exceed k; and returns nil otherwise. This is a fundamental problem with applications in data streaming [14], data reconciliation in distributed systems [12] and mobile computing [16], etc. In this paper, we present a deterministic algorithm for the k-set problem that matches the space lower bound to within a logarithmic factor.