We present a study of using large language models (LLMs) to modify a document so as to have it highly ranked for a query by an undisclosed ranking function. We present different prompting methods inspired by work on using LLMs to induce ranking. Empirical evaluation attests to the merits of the best performing methods with respect to human modifications and a highly effective feature-based modification method.

The rise of generative AI and the emergence of Large Language Models (LLMs) have sparked a new line of research on the evolution of strategic information retrieval ecosystems. My research adopts an agent-based perspective to analyze sponsored search and competitive search ecosystems. The primary motivation of my work is to examine how agent strategies influence social welfare and the existence of equilibrium. To that end, I leverage frameworks from game theory and mechanism design.

In competitive search settings, document publishers (authors) respond to rankings induced for queries of interest: they modify the documents to improve their future ranking. Hence, for some queries there is an on-going ranking competition. Prior empirical studies of competitive search were based on controlled ranking competitions between humans. Large Language Models (LLMs), capable of generating high quality content, provide new opportunities for studying ranking competitions. Furthermore, there is a significant amount of content on the Web, which is a canonical example of a competitive search setting, generated by LLMs. In this paper, we introduce LEMSS: a multi-agent platform that leverages LLMs as publishers in competitive search settings. In addition to enabling the execution of large-scale and highly configurable ranking competitions, LEMSS includes tools to analyze and compare the competitions using a wide range of measures. We use these tools to analyze examples of datasets that result from ranking competitions executed using LEMSS. The analysis reveals, for example, that using LLMs as publishers reduced content diversity in the corpus to a larger extent than having human publishers.