Various approaches to LLM red-teaming leverage an attacker LLM to discover jailbreaks against a target.Several of them task attacker with formulating its own strategies to transform harmful queries into jailbreaks through trial and error, resulting in a semantically limited range of successful attacks.Another recent approach discovers diverse attacks by combining crowdsourced queries and tactics within the attacker's instructions, but does so at random, limiting effectiveness.This article introduces a novel framework, ADAPTIVE INSTRUCTION COM-POSITION, that combines crowdsourced texts according to an adaptive mechanism trained to jointly optimize attack effectiveness with diversity.We use reinforcement learning to balance exploration with exploitation in a combinatorial space of instructions to guide the attacker toward diverse generations tailored to target vulnerabilities.We show that our strategy substantially outperforms random combination on a set of effectiveness and diversity metrics, even under model transfer.Further, we show that it surpasses a host of recent adaptive approaches on the public benchmark Harmbench.We employ a lightweight neural contextual bandit that adapts to contrastively pretrained embeddings, and provide ablations to suggest that the contrastive property enables the network to generalize and scale to the massive space.Warning: this article discusses malicious content and methods for generating it using LLMs.