The online advertising market, with its thousands of auctions run per second, presents a daunting challenge for advertisers who wish to optimize their spend under a budget constraint. Thus, advertising platforms typically provide automated agents to their customers, which act on their behalf to bid for impression opportunities in real time at scale. Because these proxy agents are owned by the platform but use advertiser funds to operate, there is a strong practical need to balance reliability and explainability of the agent with optimizing power. We propose a generalizable approach to optimizing bidding policies in production environments by learning from real data using offline reinforcement learning. This approach can be used to optimize any differentiable base policy (practically, a heuristic policy based on principles which the advertiser can easily understand), and only requires data generated by the base policy itself. We use a hybrid agent architecture that combines arbitrary base policies with deep neural networks, where only the optimized base policy parameters are eventually deployed, and the neural network part is discarded after training. We demonstrate that such an architecture achieves statistically significant performance gains in both simulated and at-scale production bidding environments compared to the default production bidding policy. Our approach does not incur additional infrastructure, safety, or explainability costs, as it directly optimizes the parameters of existing production routines without necessarily replacing them with black box-style models like neural networks.

Reinforcement learning (RL) is a versatile framework for optimizing long-term goals. Although many real-world problems can be formalized with RL, learning and deploying a performant RL policy requires a system designed to address several important challenges, including the exploration-exploitation dilemma, partial observability, dynamic action spaces, and safety concerns. While the importance of these challenges has been well recognized, existing open-source RL libraries do not explicitly address them. This paper introduces Pearl, a Production-Ready RL software package designed to embrace these challenges in a modular way. In addition to presenting benchmarking results, we also highlight examples of Pearl's ongoing industry adoption to demonstrate its advantages for production use cases. Pearl is open sourced on GitHub at github.com/facebookresearch/pearl and its official website is pearlagent.github.io.

Modern autonomous driving systems rely heavily on deep learning models to process point cloud sensory data; meanwhile, deep models have been shown to be susceptible to adversarial attacks with visually imperceptible perturbations. Despite the fact that this poses a security concern for the self-driving industry, there has been very little exploration in terms of 3D perception, as most adversarial attacks have only been applied to 2D flat images. In this paper, we address this issue and present a method to generate universal 3D adversarial objects to fool LiDAR detectors. In particular, we demonstrate that placing an adversarial object on the rooftop of any target vehicle to hide the vehicle entirely from LiDAR detectors with a success rate of 80%. We report attack results on a suite of detectors using various input representation of point clouds. We also conduct a pilot study on adversarial defense using data augmentation. This is one step closer towards safer self-driving under unseen conditions from limited training data.

Cap-and-trade schemes are designed to achieve target levels of regulated emissions in a socially efficient manner. These schemes work by issuing regulatory credits and allowing firms to buy and sell them according to their relative compliance costs. Analyzing the efficacy of such schemes in concentrated industries is complicated by the strategic interactions among firms producing heterogeneous products. We tackle this complexity via an agent-based microeconomic model of the US market for personal vehicles. We calculate Nash equilibria among credits-trading strategies in a variety of scenarios and regulatory models. We find that while cap-and-trade results improves efficiency overall, consumers bear a disproportionate share of regulation cost, as firms use credit trading to segment the vehicle market. Credits trading volume decreases when firms behave more strategically, which weakens the segmentation effect.

Due to complex interactions in financial markets, financial regulations can sometimes produce unexpected outcomes, and fail to achieve their macroeconomic goals. We replicate a previous agent-based simulation study which showed that the Basel banking regulations may increase financial instability, counter to their intended purpose. Our replication confirms that this is the case, following the original study's assumption that the financial firms' behaviors are fixed. We then extend the model to account for a possible strategic response, where financial firms adapt to the regulatory regime. Using empirical game-theoretic analysis, we derive equilibria with and without regulation. We find that in the new Basel-regulated equilibria, more funds stay out of default and banks lose less capital. The overall effect of regulation on financial stability becomes benign on most measures when accounting for the strategic adaptation of agents.

Credit networks provide a flexible model of distributed trust, which supports transactions between untrusted counterparties through paths of intermediaries. We extend this model by introducing interest rates (prices on lines of credit), both as a means to incentivize credit issuance and to provide a framework for modeling networks of financial relationships. Including interest rates poses a new constraint on transactions, as intermediaries will route payments only if the interest received covers any interest paid. We account for these constraints in an efficient algorithm for finding the maximum transaction flow between two agents in a financial network. There are generally many feasible payment paths serving a given transaction, and we show that the policy for selecting among such paths can have a substantial effect on liquidity, as measured by steady-state probability of transaction success. Finally, we consider the situation where the transaction source can choose among heuristic path selection mechanisms, in order to maximize their payoff. Through empirical game-theoretic analysis, we find that routing is inefficient due to the positive externality of choices promoting network liquidity. However, agent choices do reflect some consideration of overall network liquidity, in addition to their own interest payments.