Designing incentives for an adapting population is a ubiquitous problem in a wide array of economic applications and beyond. In this work, we study how to design additional rewards to steer multi-agent systems towards desired policies \emph{without} prior knowledge of the agents' underlying learning dynamics. Motivated by the limitation of existing works, we consider a new and general category of learning dynamics called \emph{Markovian agents}. We introduce a model-based non-episodic Reinforcement Learning (RL) formulation for our steering problem. Importantly, we focus on learning a \emph{history-dependent} steering strategy to handle the inherent model uncertainty about the agents' learning dynamics. We introduce a novel objective function to encode the desiderata of achieving a good steering outcome with reasonable cost. Theoretically, we identify conditions for the existence of steering strategies to guide agents to the desired policies. Complementing our theoretical contributions, we provide empirical algorithms to approximately solve our objective, which effectively tackles the challenge in learning history-dependent strategies. We demonstrate the efficacy of our algorithms through empirical evaluations.

Transaction costs are omnipresent in markets but are often omitted in economic models. We show that the presence of transaction costs can fundamentally alter incentive and welfare properties of Double Auctions, a canonical market organization. We further show that transaction costs can be categorized into two types. Double Auctions with homogeneous transaction costs---a category that includes fixed fees and price based fees---preserve the key advantages of Double Auctions without transaction costs: markets with homogeneous transaction costs are asymptotically strategyproof, and there is no efficiency-loss due to strategic behavior. In contrast, double auctions with heterogeneous transaction costs---such as spread fees---lead to complex strategic behavior (price guessing) and may result in severe market failures. Allowing for aggregate uncertainty, we extend these insights to market organizations other than Double Auctions.