Abstract Reinforcement learning (RL) agents naturally struggle with long-sequence sparse-reward tasks due to the lack of reward feedback during exploration and the problem of identifying the necessary action sequences required to reach the goal. Previous works have used abstract symbolic task knowledge models to speed up RL agents in these tasks by either splitting the task into easier to solve sub-tasks or by creating an artificial dense reward function. These approaches are often limited by their requirement of perfect symbolic knowledge models, which cannot be guaranteed when the abstract symbolic models are provided by humans and in real-world tasks. We introduce exponential plan-based reward shaping, which is able to leverage the ability to learn from experience of RL to compensate deficiencies in incomplete and incorrect abstract symbolic plans and use them to solve difficult tasks faster, while guaranteeing convergence to the optimal policy. Our approach is able to work with plans that miss important steps, include unnecessary extra steps, contain steps that refer ambiguously to both important and useless states, or encode an incorrect order of steps. We use action representations designed by human experts to automatically compute plans to capture the high-level task structure. The abstract symbolic subgoals defined by the plan are used to create dense reward feedback, which signals important states to the RL agent that should be achieved and explored to reach the goal. We show the theoretical advantages of our approach for plans with many steps and show its effectiveness empirically on multiple tasks with different kinds of incomplete or incorrect knowledge.

ProgressiveDB transforms any standard SQL database into a progressive database capable of continuous, approximate query processing. It introduces a few small extensions to the SQL query language that allow clients to express progressive analytical queries. These extensions are processed in the ProgressiveDB middleware that sits between a database application and the underlying database providing interactive query processing as well as query steering capabilities to the user. In our demo, we show how this system allows a database application with a graphical user interface to interact with different backends, while providing the user with immediate feedback during exploratory data exploration of an on-time flight database. ProgressiveDB also supports efficient query steering by providing a new technique, called progressive views, which allows the intermediate results of one progressive query to be shared and reused by multiple concurrent progressive queries with refined scope.