Visualizing optimization landscapes has resulted in many fundamental insights in numeric optimization, specifically regarding novel improvements to optimization techniques. However, visualizations of the objective that reinforcement learning optimizes (the "reward surface") have only ever been generated for a small number of narrow contexts. This work presents reward surfaces and related visualizations of 27 of the most widely used reinforcement learning environments in Gym for the first time. We also explore reward surfaces in the policy gradient direction and show for the first time that many popular reinforcement learning environments have frequent "cliffs" (sudden large drops in expected reward). We demonstrate that A2C often "dives off" these cliffs into low reward regions of the parameter space while PPO avoids them, confirming a popular intuition for PPO’s improved performance over previous methods. We additionally introduce a highly extensible library that allows researchers to easily generate these visualizations in the future. Our findings provide new intuition to explain the successes and failures of modern RL methods, and our visualizations concretely characterize several failure modes of reinforcement learning agents in novel ways.

This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle ("AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of the games commonly used with MARL, that they promote severe bugs that are hard to detect, and that the AEC games model addresses these problems.

Haptic devices allow a human user to physically interact with the virtual or simulated world by generating real forces to represent interactions in the virtual world. By definition, haptic devices exist in the set of robotic systems whose workspaces must overlap with the workspace of the human. For that reason, safety of the human becomes a central concern when developing both the physical hardware and the control systems for new and novel interfaces. Passive haptic devices, ones that display forces to the user by using passive actuators such as brakes, clutches of continuously variable transmissions, address the concerns of stability and safety by using actuators that physically guarantee a bounded output and the safety of the human user. However, a device using passive actuators can only produce a subset of the forces generated by a similar device built around active actuators. The work presented here will explore the use of a passive haptic device specifically with respect to teleoperation and will present a new algorithm for determining feedback that improves the transparency, a dynamic compensating controller. This controller will be implemented in LabVIEW on passive hardware and will be compared to two simpler control schemes in two typical teleoperation tasks, shape identification and point to point motion around an obstacle. The results will show that the passive device provides feedback that allows the user to recognize large surface features. Furthermore, results will show that the dynamic compensating controller proves especially useful in dynamic situations represented by the motion task.