This paper addresses a policy optimization task with the conditional value-at-risk (CVaR) objective. We introduce the *predictive CVaR policy gradient*, a novel approach that seamlessly integrates risk-neutral policy gradient algorithms with minimal modifications. Our method incorporates a reweighting strategy in gradient calculation -- individual cost terms are reweighted in proportion to their *predicted* contribution to the objective. These weights can be easily estimated through a separate learning procedure. We provide theoretical and empirical analyses, demonstrating the validity and effectiveness of our proposed method.

In nonstationary bandit learning problems, the decision-maker must continually gather information and adapt their action selection as the latent state of the environment evolves. In each time period, some latent optimal action maximizes expected reward under the environment state. We view the optimal action sequence as a stochastic process, and take an information-theoretic approach to analyze attainable performance. We bound per-period regret in terms of the entropy rate of the optimal action process. The bound applies to a wide array of problems studied in the literature and reflects the problem's information structure through its information-ratio.

We revisit the popular random matching market model introduced by Knuth (1976) and Pittel (1989), and shown by Ashlagi, Kanoria and Leshno (2013) to exhibit a “stark effect of competition”; in particular, with any difference in the number of agents on the two sides (“imbalance”), the short side agents obtain substantially better outcomes. We generalize the model to allow “partially connected” markets with each agent having an average degree d in a random (undirected) graph. Each agent has a (uniformly random) preference ranking over only their neighbors in the graph. We characterize stable matchings in large markets and find that the short side enjoys a significant advantage only for d exceeding log2 n where n is the number of agents on one side: For moderately connected markets with d = o(log2 n), we find that there is no advantage to being on the short side (for O(n1–∊) market imbalance), with agents on both sides getting a -ranked partner on average. Notably, this “mild competition” regime extends far beyond the connectivity threshold of d = Θ(log n). In contrast, for densely connected markets with d = ω(log2 n), we find a strong effect of competition, namely, short side agents get a log n-ranked partner on average, while the long side agents get a partner of (much larger) rank d/log n on average. Our results and analysis suggest that in general matching markets, being on the short side confers an advantage if and only if the number of short-side agents who remain unmatched is small relative to the market imbalance.

We consider a finite-horizon multi-armed bandit (MAB) problem in a Bayesian setting, for which we propose an information relaxation sampling framework. With this framework, we define an intuitive family of control policies that include Thompson sampling (TS) and the Bayesian optimal policy as endpoints. Analogous to TS, which, at each decision epoch pulls an arm that is best with respect to the randomly sampled parameters, our algorithms sample entire future reward realizations and take the corresponding best action. However, this is done in the presence of “penalties” that seek to compensate for the availability of future information. We develop several novel policies and performance bounds for MAB problems that vary in terms of improving performance and increasing computational complexity between the two endpoints. Our policies can be viewed as natural generalizations of TS that simultaneously incorporate knowledge of the time horizon and explicitly consider the exploration-exploitation trade-off. We prove associated structural results on performance bounds and suboptimality gaps. Numerical experiments suggest that this new class of policies perform well, in particular in settings where the finite time horizon introduces significant exploration-exploitation tension into the problem.