Generative AI has caused a paradigm shift in the area of Artificial Intelligence (AI) and as such has inspired much new research, especially on Large Language Models (LLMs). LLMs are transforming how people interact with computers in service-oriented fields in both the consumer (for example: retail, travel, education, healthcare) and enterprise (customer care, field service, sales, marketing, etc.) spaces. One barrier to widespread adoption is the current unpredictability of LLM behavior: users must trust that LLM-based services and systems are accurate, fair, and unbiased. Model responses that exhibit biases related to race, social status, and other sensitive topics can have serious consequences, ranging from lack of trust in the model to adverse social implications for consumers, all the way to damage to the reputations of the corporations that provide them. This study explores how to uncover biases related to social stigmas in LLM output, by using an adversarial prompt-based approach. Discovering model vulnerabilities of this type is a nontrivial task due to the large search space, making it resource-intensive. We present an evaluation framework for probing and analyzing the behaviors of multiple LLMs systematically. We use a curated set of adversarial prompts with a focus on uncovering biased responses to prompts associated with social attributes.

Data preparation is the first and a very important step towards any Large Language Model (LLM) development. This paper introduces an easy-to-use, extensible, and scale-flexible open-source data preparation toolkit called Data Prep Kit (DPK). DPK is architected and designed to enable users to scale their data preparation to their needs. With DPK they can prepare data on a local machine or effortlessly scale to run on a cluster with thousands of CPU Cores. DPK comes with a highly scalable, yet extensible set of modules that transform natural language and code data. If the user needs additional transforms, they can be easily developed using extensive DPK support for transform creation. These modules can be used independently or pipelined to perform a series of operations. In this paper, we describe DPK architecture and show its performance from a small scale to a very large number of CPUs. The modules from DPK have been used for the preparation of Granite Models [1] [2]. We believe DPK is a valuable contribution to the AI community to easily prepare data to enhance the performance of their LLM models or to fine-tune models with Retrieval-Augmented Generation (RAG).