Humans actively observe the visual surroundings by focusing on salient objects and ignoring trivial details. However, computer vision models based on convolutional neural networks (CNN) often analyze visual input all at once through a single feedforward pass. In this study, we designed a dual-stream vision model inspired by the human brain. This model features retina-like input layers and includes two streams: one determining the next point of focus (the fixation), while the other interprets the visuals surrounding the fixation. Trained on image recognition, this model examines an image through a sequence of fixations, each time focusing on different parts, thereby progressively building a representation of the image. We evaluated this model against various benchmarks in terms of object recognition, gaze behavior, and adversarial robustness. Our findings suggest that the model can attend and gaze in ways similar to humans without being explicitly trained to mimic human attention and that the model can enhance robustness against adversarial attacks due to its retinal sampling and recurrent processing. In particular, the model can correct its perceptual errors by taking more glances, setting itself apart from all feedforward-only models. In conclusion, the interactions of retinal sampling, eye movement, and recurrent dynamics are important to human-like visual exploration and inference.

Graph random walk sampling is becoming increasingly important with the widespread popularity of graph applications. It aims to capture the desirable graph properties by launching multiple walkers to collect feature paths. However, previous research suffers long sampling latency and severe memory access bottlenecks due to intrinsic data dependency and skewed vertex distribution. Thus, in this paper, we propose FastRW, a dedicated accelerator to boost graph random walk operation on FPGAs. Specifically, FastRW first integrates multiple parallel processing engines to achieve data-level parallelism, where each processing engine also leverages dataflow scheduling to resolve data dependency and hide long sampling latency. Secondly, FastRW leverages a combination of multiple storage resources to implement a hybrid memory architecture adapted to skewed vertex distribution. By integrating the above optimizations, FastRW develops a performance model to take advantage of the balance between computation parallelism and bandwidth demand. We evaluate FastRW with two classic sampling algorithms on a wide range of real-world graph datasets. The experimental results show that FastRW achieves a speedup of 37.52$\boldsymbol{\times}$on average over the system running on two 8-core Intel CPUs. FastRW also achieves an average of 28.04$\boldsymbol{\times}$speedup over the architecture implemented on V100 GPU.

The human visual system uses two parallel pathways for spatial processing and object recognition. In contrast, computer vision systems tend to use a single feedforward pathway, rendering them less robust, adaptive, or efficient than human vision. To bridge this gap, we developed a dual-stream vision model inspired by the human eyes and brain. At the input level, the model samples two complementary visual patterns to mimic how the human eyes use magnocellular and parvocellular retinal ganglion cells to separate retinal inputs to the brain. At the backend, the model processes the separate input patterns through two branches of convolutional neural networks (CNN) to mimic how the human brain uses the dorsal and ventral cortical pathways for parallel visual processing. The first branch (WhereCNN) samples a global view to learn spatial attention and control eye movements. The second branch (WhatCNN) samples a local view to represent the object around the fixation. Over time, the two branches interact recurrently to build a scene representation from moving fixations. We compared this model with the human brains processing the same movie and evaluated their functional alignment by linear transformation. The WhereCNN and WhatCNN branches were found to differentially match the dorsal and ventral pathways of the visual cortex, respectively, primarily due to their different learning objectives, rather than their distinctions in retinal sampling or sensitivity to attention-driven eye movements. These model-based results lead us to speculate that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity in retinal inputs. This dual-stream model takes a further step in brain-inspired computer vision, enabling parallel neural networks to actively explore and understand the visual surroundings.

Financial time series prediction and trading decision-making are priorities of computational intelligence for researchers in academia and the finance industry due to their broad application areas and substantial impact. However, these methods remain challenging because they retain various complex statistical properties, and the mechanism behind the processes is unknown to a large extent. A significant number of machine learning-based methods are proposed and demonstrate impressive results, especially deep learning-based models. Nevertheless, due to the high complexity of massive, nonlinear, and nonindependent data and the difficulties and time consumption of complicated training models of deep learning, the performance of online trading decisions is still inadequate for practical application. This paper proposes the Integrated Framework of Forecasting Based Online Trading Strategy (IFF-BOTS) to satisfy better prediction performance and dynamic decisions for real-world online trading systems. Our method adopts a novel isomorphic convolutional neural network (CNN)-based forecaster-classifier-executor architecture to exploit CNN-based price and trend integrated prediction and direct-reinforcement-learning-based trading decision-making. IFF-BOTS can also achieve better real-time performance for online trading. We empirically compare the proposed approach with state-of-the-art prediction and trading methods on real-world S&P and DJI datasets. The results show that the IFF-BOTS outperforms its competitors in predicting metrics, trading profits, and real-time performance.

In natural language processing, most models try to learn semantic representations merely from texts. The learned representations encode the “distributional semantics” but fail to connect to any knowledge about the physical world. In contrast, humans learn language by grounding concepts in perception and action and the brain encodes “grounded semantics” for cognition. Inspired by this notion and recent work in vision-language learning, we design a two-stream model for grounding language learning in vision. The model includes a VGG-based visual stream and a Bert-based language stream. The two streams merge into a joint representational space. Through cross-modal contrastive learning, the model first learns to align visual and language representations with the MS COCO dataset. The model further learns to retrieve visual objects with language queries through a cross-modal attention module and to infer the visual relations between the retrieved objects through a bilinear operator with the Visual Genome dataset. After training, the model’s language stream is a stand-alone language model capable of embedding concepts in a visually grounded semantic space. This semantic space manifests principal dimensions explainable with human intuition and neurobiological knowledge. Word embeddings in this semantic space are predictive of human-defined norms of semantic features and are segregated into perceptually distinctive clusters. Furthermore, the visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based on images, texts, or their combinations.

Based on the predictive coding theory in neuro- science, we designed a bi-directional and recur- rent neural net, namely deep predictive coding networks (PCN), that has feedforward, feedback, and recurrent connections. Feedback connections from a higher layer carry the prediction of its lower-layer representation; feedforward connec- tions carry the prediction errors to its higher-layer. Given image input, PCN runs recursive cycles of bottom-up and top-down computation to update its internal representations and reduce the differ- ence between bottom-up input and top-down pre- diction at every layer. After multiple cycles of recursive updating, the representation is used for image classification. With benchmark datasets (CIFAR-10/100, SVHN, and MNIST), PCN was found to always outperform its feedforward-only counterpart: a model without any mechanism for recurrent dynamics, and its performance tended to improve given more cycles of computation over time. In short, PCN reuses a single architecture to recursively run bottom-up and top-down pro- cesses to refine its representation towards more accurate and definitive object recognition.

Inspired by "predictive coding" - a theory in neuroscience, we develop a bi-directional and dynamic neural network with local recurrent processing, namely predictive coding network (PCN). Unlike feedforward-only convolutional neural networks, PCN includes both feedback connections, which carry top-down predictions, and feedforward connections, which carry bottom-up errors of prediction. Feedback and feedforward connections enable adjacent layers to interact locally and recurrently to refine representations towards minimization of layer-wise prediction errors. When unfolded over time, the recurrent processing gives rise to an increasingly deeper hierarchy of non-linear transformation, allowing a shallow network to dynamically extend itself into an arbitrarily deep network. We train and test PCN for image classification with SVHN, CIFAR and ImageNet datasets. Despite notably fewer layers and parameters, PCN achieves competitive performance compared to classical and state-of-the-art models. Further analysis shows that the internal representations in PCN converge over time and yield increasingly better accuracy in object recognition. Errors of top-down prediction also reveal visual saliency or bottom-up attention.

Three-dimensional (3-D) cardiac activation imaging (3-DCAI) is a recently developed technique that aims at imaging the activation sequence throughout the the ventricular myocardium. 3-DCAI entails the modeling and estimation of the cardiac equivalent current density (ECD) distribution from which the activation time at any myocardial site is determined as the time point with the peak amplitude of local ECD estimates. In this paper, we report, for the first time, an in vivo validation study assessing the feasibility of 3-DCAI in comparison with the 3-D intracardiac mapping, for a group of four healthy rabbits undergoing the ventricular pacing from various locations. During the experiments, the body surface potentials and the intramural bipolar electrical recordings were simultaneously measured in a closed-chest condition. The ventricular activation sequence noninvasively imaged from the body surface measurements by using 3-DCAI was generally in agreement with that obtained from the invasive intramural recordings. The quantitative comparison between them showed a root mean square (rms) error of 7.42 +/-0.61 ms, a relative error (RE) of 0.24 +/-0.03, and a localization error (LE) of 5.47 +/-1.57 mm. The experimental results were also consistent with our computer simulations conducted in well-controlled and realistic conditions. The present study suggest that 3-DCAI can noninvasively capture some important features of ventricular excitation (e.g., the activation origin and the activation sequence), and has the potential of becoming a useful imaging tool aiding cardiovascular research and clinical diagnosis of cardiac diseases.

We propose a new electrocardiographic (ECG) inverse approach for imaging the three-dimensional (3-D) ventricular activation sequence based on the modeling and estimation of the equivalent current density throughout the entire volume of the ventricular myocardium. The spatio-temporal coherence of the ventricular excitation process has been utilized to derive the activation time from the estimated time course of the equivalent current density. In the present study, we explored four different linear inverse algorithms (the minimum norm and weighted minimum norm estimates in combination with two regularization schemes: the instant-by-instant regularization and the isotropy method) to estimate the current density at each time instant during the ventricular depolarization. The activation time at any given location within the ventricular myocardium was determined as the time point with the occurrence of the maximum local current density estimate. Computer simulations were performed to evaluate this approach using single- and dual-site pacing protocols in a physiologically realistic cellular automaton heart model. The performance and stability of the proposed approach was evaluated with respect to the various levels of measurement noise (0, 5, 10, 20, 40, and 60 microV), the various numbers of ECG electrodes and the modeling errors on the torso geometry and heart position. The simulation results demonstrate that: 1) the single-site paced 3-D activation sequence can be well reconstructed from 200-channel body surface potential maps with additive Gaussian white noise of 20 microV (correlation coefficient = 0.90, relative error = 0.19, and localization error = 5.49 mm); 2) a higher imaging accuracy can be obtained when the activation is initiated from the left/right ventricle (LV/RV) compared to from the septum; 3) the isotropy method gives rise to a better performance than the conventional instant-by-instant regularization; 4) a decreased imaging accuracy results from a larger noise level, a fewer number of electrodes, or the volume conductor modeling errors; however, a reasonable imaging accuracy can still be obtained with a 60 microV noise level, 64 electrodes, or mild errors on both the torso geometry and heart position, respectively; 5) the dual-site paced 3-D activation sequence can be imaged when the two sites are paced either simultaneously or with a time delay of 20 ms; 6) two pacing sites can be resolved and localized in the imaged 3-D activation sequence when they are located at the contralateral sides of ventricles or at the ventricular lateral wall and the apex, respectively.