To solve difficult tasks, humans ask questions to acquire knowledge from external sources. In contrast, classical reinforcement learning agents lack such an ability and often resort to exploratory behavior. This is exacerbated as few present-day environments support querying for knowledge. In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld. In addition to physical interactions, an agent can query an external knowledge source specialized for these environments to gather information. Second, we propose the ‘Asking for Knowledge’ (AFK) agent, which learns to generate language commands to query for meaningful knowledge that helps solve the tasks. AFK leverages a non-parametric memory, a pointer mechanism and an episodic exploration bonus to tackle (1) irrelevant information, (2) a large query language space, (3) delayed reward for making meaningful queries. Extensive experiments demonstrate that the AFK agent outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld environments.

While deep reinforcement learning (RL) promises freedom from hand-labeled data, great successes, especially for Embodied AI, require significant work to create supervision via carefully shaped rewards. Indeed, without shaped rewards, i.e., with only terminal rewards, present-day Embodied AI results degrade significantly across Embodied AI problems from single-agent Habitat-based PointGoal Navigation (SPL drops from 55 to 0) and two-agent AI2-THOR-based Furniture Moving (success drops from 58% to 1%) to three-agent Google Football-based 3 vs. 1 with Keeper (game score drops from 0.6 to 0.1). As training from shaped rewards doesn’t scale to more realistic tasks, the community needs to improve the success of training with terminal rewards. For this we propose GRIDTOPIX: 1) train agents with terminal rewards in gridworlds that generically mirror Embodied AI environments, i.e., they are independent of the task; 2) distill the learned policy into agents that reside in complex visual worlds. Despite learning from only terminal rewards with identical models and RL algorithms, GRIDTOPIX significantly improves results across tasks: from PointGoal Navigation (SPL improves from 0 to 64) and Furniture Moving (success improves from 1% to 25%) to football gameplay (game score improves from 0.1 to 0.6). GRIDTOPIX even helps to improve the results of shaped reward training.

Exploration is critical for good results in deep reinforcement learning and has attracted much attention. However, existing multi-agent deep reinforcement learning algorithms still use mostly noise-based techniques. Very recently, exploration methods that consider cooperation among multiple agents have been developed. However, existing methods suffer from a common challenge: agents struggle to identify states that are worth exploring, and hardly coordinate exploration efforts toward those states. To address this shortcoming, in this paper, we propose cooperative multi-agent exploration (CMAE): agents share a common goal while exploring. The goal is selected from multiple projected state spaces by a normalized entropy-based technique. Then, agents are trained to reach the goal in a coordinated manner. We demonstrate that CMAE consistently outperforms baselines on various tasks, including a sparse-reward version of multiple-particle environment (MPE) and the Starcraft multi-agent challenge (SMAC).

Solving complex real-world tasks, e.g., autonomous fleet control, often involves a coordinated team of multiple agents which learn strategies from visual inputs via reinforcement learning. Many existing multi-agent reinforcement learning (MARL) algorithms however don’t scale to environments where agents operate on visual inputs. To address this issue, algorithmically, recent works have focused on non-stationarity and exploration. In contrast, we study whether scalability can also be achieved via a disentangled representation. For this, we explicitly construct an object-centric intermediate representation to characterize the states of an environment, which we refer to as ‘semantic tracklets.’ We evaluate ‘semantic tracklets’ on the visual multi-agent particle environment (VMPE) and on the challenging visual multi-agent GFootball environment. ‘Semantic tracklets’ consistently outperform baselines on VMPE, and achieve a +2.4 higher score difference than baselines on GFootball. Notably, this method is the first to successfully learn a strategy for five players in the GFootball environment using only visual data. For more, please see our project page: https://ioujenliu.github.io/SemanticTracklets

In practice, imitation learning is preferred over pure reinforcement learning whenever it is possible to design a teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On a suite of challenging tasks set within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.

Various parallel actor-learner methods reduce long training times for deep reinforcement learning. Synchronous methods enjoy training stability while having lower data throughput. In contrast, asynchronous methods achieve high throughput but suffer from stability issues and lower sample efficiency due to ‘stale policies.’ To combine the advantages of both methods we propose High-Throughput Synchronous Deep Reinforcement Learning (HTS-RL). In HTS-RL, we perform learning and rollouts concurrently, devise a system design which avoids ‘stale policies’ and ensure that actors interact with environment replicas in an asynchronous manner while maintaining full determinism. We evaluate our approach on Atari games and the Google Research Football environment. Compared to synchronous baselines, HTS-RL is 2−6X faster. Compared to state-of-the-art asynchronous methods, HTS-RL has competitive throughput and consistently achieves higher average episode rewards.

Reinforcement learning (RL) has attracted much attention recently, as new and emerging AI-based applications are demanding the capabilities to intelligently react to environment changes. Unlike distributed deep neural network (DNN) training, the distributed RL training has its unique workload characteristics - it generates orders of magnitude more iterations with much smaller sized but more frequent gradient aggregations. More specifically, our study with typical RL algorithms shows that their distributed training is latency critical and that the network communication for gradient aggregation occupies up to 83.2% of the execution time of each training iteration.

Self-aligned double patterning (SADP) is one of the most promising techniques for sub-20 nm technology. Spacer-is-dielectric SADP using a cut process is getting popular because of its higher design flexibility; for example, it can decompose odd cycles without the need of inserting any stitch. This paper presents the first work that applies the cut process for decomposing odd cycles during routing. For SADP, further, overlay control is a critical issue for yield improvement; while published routers can handle only partial overlay scenarios, this paper identifies all the scenarios that induce overlays and proposes a novel constraint graph to model all overlays. With the developed techniques, our router can achieve high-quality routing results with significantly fewer overlays (and thus better yields). Compared with three state-of-the-art studies, our algorithm can achieve the best quality and efficiency, with zero cut conflicts, smallest overlay length, highest routability, and fastest running time.

Multiple e-beam lithography (MEBL) is one of the most promising next generation lithography technologies for high volume manufacturing, which improves the most critical issue of conventional single e-beam lithography, throughput, by simultaneously using thousands or millions of e-beams. For parallel writing in MEBL, a layout is split into stripes and patterns are cut by stripe boundaries, which are defined as stitching lines. Critical patterns cut by stitching lines could suffer from severe pattern distortion or even yield loss. Therefore, considering the positions of stitching lines and avoiding stitching line-induced bad patterns are required during layout design. In this paper, we propose the first work of stitch-aware routing framework for MEBL based on a two-pass bottom-up multilevel router. We first identify three types of stitching line-induced bad patterns which should not exist in an MEBL-friendly routing solution. Then, stitch-aware routing algorithms are, respectively, developed for global routing, layer/track assignment, and detailed routing. Experimental results show that our stitch-aware routing framework can effectively reduce stitching line-induced bad patterns and thus may not only improve the manufacturability but also facilitate the development of MEBL.

Self-aligned double patterning (SADP) is one of the most promising techniques for sub-20nm technology. Spacer-is-dielectric SADP using a cut process is getting popular because of its higher design flexibility; for example, it can decompose odd cycles without the need of inserting any stitch. This paper presents the first work that applies the cut process for decomposing odd cycles during routing. For SADP, further, overlay control is a critical issue for yield improvement; while published routers can handle only partial overlay scenarios, our work identifies all the scenarios that induce overlays and proposes a novel constraint graph to model all overlays. With the developed techniques, our router can achieve high-quality routing results with significantly fewer overlays (and thus better yields). Compared with three state-of-the-art studies, our algorithm can achieve the best quality and efficiency, with zero cut conflicts, smallest overlay length, highest routability, and fastest running time.

Multiple e-beam lithography (MEBL) is one of the most promising next generation lithography (NGL) technologies for high volume manufacturing, which improves the most critical issue of conventional single e-beam lithography, throughput, by simultaneously using thousands or millions of e-beams. For parallel writing in MEBL, a layout is split into stripes and patterns are cut by stripe boundaries, which are defined as stitching lines. Critical patterns cut by stitching lines could suffer from severe pattern distortion or even yield loss. Therefore, considering the positions of stitching lines and avoiding stitching line-induced bad patterns are required during layout design. In this paper, we propose the first work of stitch-aware routing framework for MEBL based on a two-pass bottom-up multilevel router. We first identify three types of stitching line-induced bad patterns which should not exist in an MEBL-friendly routing solution. Then, stitch-aware routing algorithms are respectively developed for global routing, layer/track assignment and detailed routing. Experimental results show that our stitch-aware routing framework can effectively reduce stitching line-induced bad patterns and thus may not only improve the manufacturability but also facilitate the development of MEBL.