We propose a novel Reinforcement Learning model for discrete environments, which is inherently interpretable and supports the discovery of arbitrary deep subgoal hierarchies. In the model, an agent learns information about environment in the form of probabilistic rules, while policies for (sub)goals are learned as combinations thereof. No reward function is required for learning; an agent only needs to be given a primary goal to achieve. Subgoals of a goal G from the hierarchy are computed as descriptions of states, which if previously achieved increase the total efficiency of the available policies for G. These state descriptions are introduced as new sensor predicates into the rule language of the agent, which allows for sensing important intermediate states and for updating environment rules and policies accordingly.

In many tasks related to reasoning about consequences of a logical theory, it is desirable to decompose the theory into a number of weakly related or independent components. However, a theory may represent knowledge that is subject to change, as a result of executing actions that have effects on some of the initial properties mentioned in the theory. Having once computed a decomposition of a theory, it is advantageous to know whether a decomposition has to be computed again in the newly changed theory (obtained from taking into account changes resulting from execution of an action). In this article, we address this problem in the scope of the situation calculus, where a change of an initial theory is related to the notion of progression. Progression provides a form of forward reasoning; it relies on forgetting values of those properties, which are subject to change, and computing new values for them. We consider decomposability and inseparability, two component properties known from the literature, and contribute by studying the conditions (1) when these properties are preserved and (2) when they are lost wrt progression and the related operation of forgetting. To show the latter, we demonstrate the boundaries using a number of negative examples. To show the former, we identify cases when these properties are preserved under forgetting and progression of initial theories in local-effect basic action theories of the situation calculus. Our article contributes to bridging two different communities in knowledge representation, namely, research on modularity and research on reasoning about actions.

In many tasks related to reasoning about consequences of a logical theory, it is desirable to decompose the theory into a number of components with weakly-related or independent signatures. This facilitates reasoning when signature of a query formula belongs to only one of the components. However, an initial theory may be subject to change due to execution of actions affecting features mentioned in the theory. Having once computed a decomposition of a theory, one would like to know whether a decomposition has to be computed again for the theory obtained from taking into account the changes resulting from execution of an action. In the paper, we address this problem in the scope of the situation calculus, where change of an initial theory is related to the well-studied notion of progression. Progression provides a form of forward reasoning; it relies on forgetting values of those features which are subject to change and computing new values for them. We prove new results about properties of decomposition components under forgetting and show when a decomposition can be preserved in progression of an initial theory.