It is difficult to solve complex tasks that involve large state spaces and long-term decision processes by reinforcement learning (RL) algorithms. A common and promising method to address this challenge is to compress a large RL problem into a small one. Towards this goal, the compression should be state-temporal and optimality-preserving (i.e., the optimal policy of the compressed problem should correspond to that of the uncompressed problem). In this paper, we propose a reward-restricted geodesic (RRG) metric, which can be learned by a neural network, to perform state-temporal compression in RL. We prove that compression based on the RRG metric is approximately optimality-preserving for the raw RL problem endowed with temporally abstract actions. With this compression, we design an RRG metric-based reinforcement learning (RRG-RL) algorithm to solve complex tasks. Experiments in both discrete (2D Minecraft) and continuous (Doom) environments demonstrated the superiority of our method over existing RL approaches.

Deep convolutional neural networks (CNNs) have demonstrated impressive performance on many visual tasks. Recently, they became useful models for the visual system in neuroscience. However, it is still not clear what is learned by CNNs in terms of neuronal circuits. When a deep CNN with many layers is used for the visual system, it is not easy to compare the structure components of CNNs with possible neuroscience underpinnings due to highly complex circuits from the retina to the higher visual cortex. Here, we address this issue by focusing on single retinal ganglion cells with biophysical models and recording data from animals. By training CNNs with white noise images to predict neuronal responses, we found that fine structures of the retinal receptive field can be revealed. Specifically, convolutional filters learned are resembling biological components of the retinal circuit. This suggests that a CNN learning from one single retinal cell reveals a minimal neural network carried out in this cell. Furthermore, when CNNs learned from different cells are transferred between cells, there is a diversity of transfer learning performance, which indicates that CNNs are cell specific. Moreover, when CNNs are transferred between different types of input images, here white noise versus natural images, transfer learning shows a good performance, which implies that CNNs indeed capture the full computational ability of a single retinal cell for different inputs. Taken together, these results suggest that CNNs could be used to reveal structure components of neuronal circuits, and provide a powerful model for neural system identification.

Hidden Markov models (HMMs) underpin the solution to many problems in computational neuroscience. However, it is still unclear how to implement inference of HMMs with a network of neurons in the brain. The existing methods suffer from the problem of being nonspiking and inaccurate. Here, we build a precise equivalence between the inference equation of HMMs with time-invariant hidden variables and the dynamics of spiking winner-take-all (WTA) neural networks. We show that the membrane potential of each spiking neuron in the WTA circuit encodes the logarithm of the posterior probability of the hidden variable in each state, and the firing rate of each neuron is proportional to the posterior probability of the HMMs. We prove that the time course of the neural firing rate can implement posterior inference of HMMs. Theoretical analysis and experimental results show that the proposed WTA circuit can get accurate inference results of HMMs.

Pro-active pushing is a promising emerging communication technology to improve the resource efficiency and quality of service provisioning in mobile networks. This paper characterizes users' requests with request delay information (RDI), and proposes to use the cache-hit ratio (CHR) as the metric of the system performance in pushing strategy design. Different from conventional instant on-demand services, the pro-active pushing system can merge different users' requests for the same file at different time instants. The base station can therefore push files more efficiently by employing multicasting technologies. However, it is revealed that if one of the users' channel condition is significantly poor, it is a better choice to ignore the user in the pushing system from the overall CHR performance perspective. In the two-user one-file scenario, we derive analytical expressions of the CHR to help the BS determine the optimal pushing rate, which is shown to be a two-value-selection. In particular, for two users with uniformly distributed RDIs, the decision space is specifically characterized through analysis and calculation.

Proactive pushing and caching has recently emerged as a promising technology to improve the quality of service in mobile networks. Given the fact that users' demand for content is largely driven by social networking, this paper combines the analysis of social networking with proactive pushing and caching by constructing a human-in-the-loop system model with a physical multicasting transmission. By taking social network structure, prediction and joint pushing and caching (JPC) into account, a closed-loop system is obtained, in which the input consists of arrivals of new content items, the control procedure is based on prediction and JPC, and the output is the cache-hit ratio (CHR). A prediction and JPC based adjustment algorithm is proposed to maximize the CHR of this system. It is shown that the prediction window, which refers to how far in the future predictions are made, and the prediction error have a significant impact on the performance of the system.