Navigating and understanding the real world remains a key challenge in machine learning and inspires a great variety of research in areas such as language grounding, planning, navigation and computer vision. We propose an instruction-following task that requires all of the above, and which combines the practicality of simulated environments with the challenges of ambiguous, noisy real world data. StreetNav is built on top of Google Street View and provides visually accurate environments representing real places. Agents are given driving instructions which they must learn to interpret in order to successfully navigate in this environment. Since humans equipped with driving instructions can readily navigate in previously unseen cities, we set a high bar and test our trained agents for similar cognitive capabilities. Although deep reinforcement learning (RL) methods are frequently evaluated only on data that closely follow the training distribution, our dataset extends to multiple cities and has a clean train/test separation. This allows for thorough testing of generalisation ability. This paper presents the StreetNav environment and tasks, models that establish strong baselines, and extensive analysis of the task and the trained agents.

Navigating through unstructured environments is a basic capability of intelligent creatures, and thus is of fundamental interest in the study and development of artificial intelligence. Long-range navigation is a complex cognitive task that relies on developing an internal representation of space, grounded by recognisable landmarks and robust visual processing, that can simultaneously support continuous self-localisation ("I am here") and a representation of the goal ("I am going there"). Building upon recent research that applies deep reinforcement learning to maze navigation problems, we present an end-to-end deep reinforcement learning approach that can be applied on a city scale. Recognising that successful navigation relies on integration of general policies with locale-specific knowledge, we propose a dual pathway architecture that allows locale-specific features to be encapsulated, while still enabling transfer to multiple cities. A key contribution of this paper is an interactive navigation environment that uses Google Street View for its photographic content and worldwide coverage. Our baselines demonstrate that deep reinforcement learning agents can learn to navigate in multiple cities and to traverse to target destinations that may be kilometres away. A video summarizing our research and showing the trained agent in diverse city environments as well as on the transfer task is available at: https://sites.google.com/view/learn-navigate-cities-nips18

In this work, we investigate cellular simultaneous recurrent networks (CSRNs) to learn topological image mappings, particularly those of the affine transformations. While affine image transformation in conventional image processing is a relatively simple task, learning these transformations is an important part of having neural networks (NNs) function as generalized image processors. We introduce the CSRN and discuss its adaptation for image processing tasks. We report results for translation, rotation and scaling of both binary and grey-scale images. Our results suggest that the CSRN is capable of learning and performing these basic topological transformations.

At present, a major initiative in the research community is investigating new ways of processing data that capture the efficiency of the human brain in hardware and software. This has resulted in increased interest and development of bio-inspired computing approaches in software and hardware. One such bio-inspired approach is Cellular Simultaneous Recurrent Networks (CSRNs). CSRNs have been demonstrated to be very useful in solving state transition type problems, such as maze traversals. Although powerful in image processing capabilities, CSRNs have high computational demands with increasing input problem size. In this work, we revisit the maze traversal problem to gain an understanding of the general processing of CSRNs. We use a 2.67 GHz Intel Xeon X5550 processor coupled with an NVIDIA Tesla C2050 general purpose graphical processing unit (GPGPU) to create several novel accelerated CSRN implementations as a means of overcoming the high computational cost. Additionally, we explore the use of decoupled extended Kalman filters in the CSRN training phase and find a significant reduction in runtime with negligible change in accuracy. We find in our results that we can achieve average speedups of 21.73 and 3.55 times for the training and testing phases respectively when compared to optimized C implementations. The main bottleneck in training performance was a matrix inversion computation. Therefore, we utilize several methods to reduce the effects of the matrix inversion computation.

Cellular simultaneous recurrent networks (CSRN)s have been successfully exploited to solve the conventional maze traversing problem. In this work, for the first time, we investigate the use of CSRNs for image registration under affine transformations. In our simulations, we consider binary images with in-plane rotations between plusmn20deg. First, we experiment with a readily available CSRN with generalized multilayer perceptrons (GMLP)s as the basic core. We identify performance criteria for such CSRNs in affine correction. We then propose a modified MLP architecture with multi-layered feedback as the core for a CSRN to improve binary image registration performance. Simulation results show that while both the GMLP network and our modified network are able to achieve localized image registration, our modified architecture is more effective in moving pixels for registration. Finally, we use sub-image processing with our modified MLP architecture, to reduce training time and increase global registration accuracy. Overall, both CSRN architectures show promise for correctly registering a binary image.

In this paper, we investigate two novel techniques based on Cellular Simultaneous Recurrent Network (CSRN) that can address the problem of small in-plane image rotation and large out-of-plane pose invariant face recognition in image sequences. In our first technique, for the first time in literature, we investigated the CSRNs for static image registration under affine transformations. Both the readily available CSRN with generalized multilayer perceptrons (GMLPs) and a modified MLP architecture with multi-layered feedback are implemented. Simulation results show that while both the GMLP network and our modified network are able to achieve localized image registration, our modified architecture is more effective in registering image pixels. In our second technique, we investigate the recognition problem for face image sequences with large pose variation as an implicit temporal prediction task for CSRN. CSRN is trained by image sequences to capture the temporal information. The Euclidian distances between successive frames of test and output image sequences indicate either a match or mismatch between the two corresponding face classes. We extensively evaluate our CSRN-based face recognition technique with 5 persons using publicly available VidTIMIT Audio-Video face dataset. In order to verify the performance of CSRN, we also implement Elman neural network for comparison.

A fully automated, multistage system for real-time recognition of facial expression is presented. The system uses facial motion to characterize monochrome frontal views of facial expressions and is able to operate effectively in cluttered and dynamic scenes, recognizing the six emotions universally associated with unique facial expressions, namely happiness, sadness, disgust, surprise, fear, and anger. Faces are located using a spatial ratio template tracker algorithm. Optical flow of the face is subsequently determined using a real-time implementation of a robust gradient model. The expression recognition system then averages facial velocity information over identified regions of the face and cancels out rigid head motion by taking ratios of this averaged motion. The motion signatures produced are then classified using Support Vector Machines as either nonexpressive or as one of the six basic emotions. The completed system is demonstrated in two simple affective computing applications that respond in real-time to the facial expressions of the user, thereby providing the potential for improvements in the interaction between a computer user and technology.