Significant progress has been made recently in deep reinforcement learning in the development of options. This idea consists in learning policies (or macro of actions) for sub-goals. An important bottleneck of this approach is that these options are often available as actions everywhere in the state space, hence, potentially enlarging the action space to search for the optimal policy. In this paper, we propose to use the fact that the state space is rarely fully connected, but instead has regions of highly connected states with fewer links between those regions. Our proposed model extends deep Q-Learning network (DQN) by splitting the top layers into multiple heads each specializing in learning the dynamics of a particular region of the state space as well as the optimal policy for that region. The state prediction quality of each head is used to determine which head is the local expert, rating its contribution to the current state's policy. We show that this approach is able to learn something similar to options and generalized value function, providing a promising alternative to the current approach.

The ability to make predictions is central to the artificial intelligence problem. While machine learning algorithms have difficulty in learning to predict events with hundreds of time-step dependencies, animals can learn event timing within tens of trials across a broad spectrum of time scales. This suggests strongly a need for new perspectives on the forecasting problem. This paper focuses on binary time series that can be predicted within some temporal precision. We demonstrate that the sum of squared errors (SSE) calculated at every time step is not appropriate for this problem. Next, we look at the advantages and shortcomings of using a dynamic time warping (DTW) cost function. Then, we propose the squared timing error (STE) that uses DTW on the event space and applies SSE on the timing error instead of at each time step. We evaluate all three cost functions on different types of timing errors, such as phase shift, warping, and missing events, on synthetic and real-world binary time series (heartbeats, finance, and music). The results show that STE provides more information about timing error, is differentiable, and can be computed online efficiently. Finally, we devise a gradient descent algorithm for STE on a simplified recurrent neural network. We then compare the performance of the STE-based algorithm to SSE- and logit-based gradient descent algorithms on the same network architecture. The results in real-world binary time series show that the STE algorithm generally outperforms all the other cost functions considered.

A resistive pulse sensing device is able to extract quantities such as concentration and size distribution of particles, e.g. cells or microspheres, as they flow through the device's sensor region, i.e. channel, in an electrolyte solution. The dynamic range of detectable particle sizes is limited by the channel dimensions. In addition, signal interference from multiple particles transiting the channel simultaneously, i.e. coincidence event, further hinder the dynamic range. Coincidence data is often considered unusable and is discarded, reducing the throughput and introducing possible biases and errors into the distributions. Here, we propose a two-step solution. We code the channel such that the system response results in a Manchester encoded Barker-Code sequence, allowing us to take advantage of the code's pulse compression properties. We pose the parameter estimation problem as a sparse inverse problem, which enables estimation of particle sizes and velocities while resolving coincidences, and solve it with a successive interference cancellation algorithm. We introduce modifications to the algorithm to account for device fabrication variations and natural stochastic variations in flow. We demonstrate the ability to resolve coincidences and possible increases in the device's dynamic range by screening particles of different size through a Barker encoded device.

In the field of surgical robotics, it is difficult to find objective assessment techniques for evaluating telepresence. In this study we try to merge the concepts from cognitive neuroscience more specifically from the field of tool incorporation (embodiment) into surgical robotics in order to assess quantitatively the affect of delayed feedback on the telepresence. Hence, this study investigates the relationship between tool-incorporation (in this case the haptic device of the surgical console) and the decrease of performance due to time delays. A cognitive neuroscience method called the crossmodal congruency effect (CCE) is used to measure the level of tool incorporation and thus indirectly the level of telepresence while performing a simulated surgical-like task. The result of this study shows that user telepresence level and performance measures decrease with increasing delayed visual and haptic feedback.

Successful application of reinforcement learning algorithms often involves considerable hand-crafting of the necessary non-linear features to reduce the complexity of the value functions and hence to promote convergence of the algorithm. In contrast, the human brain readily and autonomously finds the complex features when provided with sufficient training. Recent work in machine learning and neurophysiology has demonstrated the role of the basal ganglia and the frontal cortex in mammalian reinforcement learning. This paper develops and explores new learning algorithms that provides potential new approaches to the feature construction problem. The algorithms are compared and evaluated on the Acrobot task. inspired by neurological evidence

An algorithm for performing simultaneous growing and pruning of cascade-correlation (CC) neural networks is introduced and tested. The algorithm adds hidden units as in standard CC, and removes unimportant connections by using optimal brain damage (OBD) in both the input and output phases of CC. To this purpose, OBD was adapted to prune weights according to two separate objective functions that are used in CC to train the network, respectively. Application of the new algorithm to two databases of the PROBEN1 benchmarks reveals that this new dual-phase pruning technique is effective in significantly reducing the size of CC networks, while providing a speed-up in learning times and improvements in generalization over novel test sets.

Research with neural networks typically ignores the role of knowledge in learning by initializing the network with random connection weights. We examine a new extension of a well-known generative algorithm, cascade-correlation. Ordinary cascade-correlation constructs its own network topology by recruiting new hidden units as needed to reduce network error. The extended algorithm, knowledge-based cascade-correlation (KBCC), recruits previously learned sub-networks as well as single hidden units. This paper describes KBCC and assesses its performance on a series of small, but clear problems involving discrimination between two classes. The target class is distributed as a simple geometric figure. Relevant source knowledge consists ofvarious linear transformations ofthe target distribution. KBCC is observed to find, adapt and use its relevant knowledge to speed learning significantly.