We present COIL (Counterfactual Object Interaction Learning), a novel way of learning skills of object interactions on entity-centric environments. The goal is to learn primitive behaviors that can induce interactions without external reward or any supervision. Existing skill discovery methods are limited to locomotion, simple navigation tasks, or single-object manipulation tasks, mostly not inducing interaction between objects. Unlike a monolithic representation usually used in prior skill learning methods, we propose to use a structured goal representation that can query and scope which objects to interact with, which can serve as a basis for solving more complex downstream tasks. We design a novel counterfactual intrinsic reward through the use of either a forward model or successor features that can learn an interaction skill between a pair of objects given as a goal. Through experiments on continuous control environments such as Magnetic Block and 2.5-D Stacking Box, we demonstrate that an agent can learn object interaction behaviors (e.g., attaching or stacking one block to another) without any external rewards or domain-specific knowledge.

We propose the k-Shortest-Path (k-SP) constraint: a novel constraint on the agent’s trajectory that improves the sample efficiency in sparse-reward MDPs. We show that any optimal policy necessarily satisfies the k-SP constraint. Notably, the k-SP constraint prevents the policy from exploring state-action pairs along the non-k-SP trajectories (e.g., going back and forth). However, in practice, excluding state-action pairs may hinder the convergence of RL algorithms. To overcome this, we propose a novel cost function that penalizes the policy violating SP constraint, instead of completely excluding it. Our numerical experiment in a tabular RL setting demonstrates that the SP-constraint can significantly reduce the trajectory space of policy. As a result, our constraint enables more sample efficient learning by suppressing redundant exploration and exploitation. Our experiments on MiniGrid, DeepMind Lab, Atari, and Fetch show that the proposed method significantly improves proximal policy optimization (PPO) and outperforms existing novelty-seeking exploration methods including count-based exploration even in continuous control tasks, indicating that it improves the sample efficiency by preventing the agent from taking redundant actions.

A synapse device array based on the gated Schottky diodes (GSDs) is fabricated. This GSD operates in reverse mode, so the synapse current is considerably low, helping to implement a low-power hardware-based neural networks (HNNs). The reverse Schottky diode current, which represents the synaptic weight, is modulated by applying program or erase pulses. In this GSD, the reverse diode current is used as the synapse current, but the forward diode current is cut off. This is an important feature to prevent the sneak path problem in the crossbar array. A synapse array consisting of fabricated 200 GSDs shows a variation (σ/μ) of 0.34, 0.22, and 0.14 for three different synaptic weight states. By using this GSD array, we perform the vector-by-matrix multiplication, and evaluate the inference accuracy of MNIST. As a baseline accuracy for MNIST classification, a convolutional neural network similar to Lenet-5 is designed and gives an accuracy of 99.53%. Normalization method is applied to the weights trained in the network to map the weights into the conductance range of synapse device. The adaptive weight quantization is then applied to the normalized weights. We verify that the HNN using GSDs works well in comparison to the baseline network even in the presence of non-ideal characteristics of synapse devices.

A gated Schottky diode is proposed for high-performance synapse devices and a means of designing a neural network using this device is described. The proposed gated Schottky diode operates in the saturation region with respect to the input voltage and is therefore immune to input noise and enables accurate vector-by-matrix multiplication. Moreover, by applying identical pulses to the bottom gate to store charges in a storage layer, the reverse saturation current increases almost linearly. Considering these special characteristics, we propose an architecture that uses a time-modulated input pulse and a learning rule based on a single conductance step. A three-layer perceptron network is trained using the conductance response of the synapse device and unidirectional weight-updating methods. In simulations using this network, the classification accuracy rate of MNIST training sets was found to be 94.50%. Compared to memristive devices, the improved linearity of the conductance response in our device is evidence of its higher accuracy.