In-context learning (ICL) has emerged as a powerful paradigm for adapting Large Language Models (LLMs) to specific tasks without parameter updates. While various strategies exist for selecting relevant ICL exemplars from a labeled pool, the fundamental challenge of constructing this high-quality pool remains largely unexplored, especially for new tasks or domains with limited labeled data. We present IclForge, a novel active learning framework that efficiently selects informative examples from unlabeled datasets to be annotated and included in the ICL pool. Unlike traditional active learning methods that optimize for individual example informativeness, IclForge explicitly considers the interdependence of examples within the ICL context. Through extensive experiments across diverse datasets and LLM architectures, we show that IclForge outperforms standard active learning baselines by +180-450 basis points while requiring 50% fewer annotations. Our framework is complementary to existing ICL selection strategies and extends naturally to generative applications, which we demonstrate through experiments on Math Word Problem (MWP) tasks. These results highlight IclForge's effectiveness in constructing high-quality ICL exemplar pools in resource-constrained scenarios.

Automatic extraction of attribute preferences from search queries is a critical problem in providing accurate product recommendations to customer. The task becomes even more challenging in cold-start settings where we do not have any supervised/labelled data available to train ML models. In this work, we implement a novel dataset generation pipeline (LLM-API) that leverages Large Language Models (LLMs), search logs and proprietary product information data from an ecommerce website to create a high quality dataset. Our proposed pipeline of LLM-API is robust as it can generalize to any product category with minimal changes in the LLM prompts. For the problem of converting product search queries to API calls we propose a multi-task schema generator model which we train on our generated dataset. Experiments on an internal test set reveals that our proposed model achieves an improvement of ≈9.6% and ≈5% in Exact Match and Micro-F1 respectively, over competitive baselines. Benchmarking our approach on public test set of search queries further reveals a gain of ≈8.6% and ≈10.5% in Exact Match and Micro-F1. We further demonstrate that our approach outperforms a state-of-the-art LLM (Claude) applied on our task using few-shot prompting and CoT reasoning, while at the same time, achieves improvement in inference latency.

Large language models (LLMs) have demonstrated remarkable performance in diverse tasks using zero-shot and few-shot prompting.Even though their capabilities of data synthesis have been studied well in recent years, the generated data suffers from a lack of diversity, less adherence to the prompt, and potential biases that creep into the data from the generator model.In this work, we tackle the challenge of generating datasets with high diversity, upon which a student model is trained for downstream tasks.Taking the route of decoding-time guidancebased approaches, we propose CORRSYNTH, which generates data that is more diverse and faithful to the input prompt using a correlated sampling strategy.Further, our method overcomes the complexity drawbacks of some other guidance-based techniques like classifier-based guidance.With extensive experiments, we show the effectiveness of our approach and substantiate our claims.In particular, we perform intrinsic evaluation to show the improvements in diversity.Our experiments show that CORRSYNTH improves both student metrics and intrinsic metrics upon competitive baselines across four datasets, showing the innate advantage of our method.

Vijit Malik, Sunipa Dev, Akihiro Nishi, Nanyun Peng, Kai-Wei Chang. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, Ashutosh Modi. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Deep learning models are susceptible to adversarial examples that have imperceptible perturbations in the original input, resulting in adversarial attacks against these models.Analysis of these attacks on the state of the art transformers in NLP can help improve the robustness of these models against such adversarial inputs.In this paper, we present Adv-OLM, a black-box attack method that adapts the idea of Occlusion and Language Models (OLM) to the current state of the art attack methods.OLM is used to rank words of a sentence, which are later substituted using word replacement strategies.We experimentally show that our approach outperforms other attack methods for several text classification tasks.