Combining Convolutional Neural Networks (CNNs) with Conditional Random Fields (CRFs) achieves great success among recent object segmentation methods. There are two advantages by such usage. First, CNNs can extract low-level features, which are very similar to the extracted features in primates' primary visual cortex (V1). Second, CRFs can set up the relationship between input features and output labels in a direct way. In this paper, we extend the first advantage by using CNNs for low-level feature extraction and a Structured Random Forest (SRF)-based border ownership detector for high-level feature extraction, which are similar to the outputs of primates secondary visual cortex (V2). Compared to the CRF model, an improved Conditional Boltzmann Machine (CBM), which has a multi-channel visible layer, is proposed to model the relationship between predicted labels, local and global contexts of objects with multi-scale and multilevel features. Besides, our proposed CBM model is extended for object parsing by using multivisible branches instead of a single visible layer of CBM, which cannot only segment the whole body but also the parts of the body under. These visible branches use each branch for the segmentation of the whole body or one of the body parts. All branches share the same hidden layers of CBM and train the branches under an iterative way. By exploiting object parsing, the whole body segmentation performance of object is improved. To refine the segmentation output, two kinds of optimization algorithms are proposed. The superpixel-based algorithm can re-label the overlapped regions of multiple kinds of objects. The other curve correction algorithm corrects the edges of segmented object parts by using smooth edges under a curve similarity criterion. Experiments demonstrate that our models yield competitive results for object segmentation on the PASCAL VOC 2012 dataset and for object parsing on the PennFudan Pedestrian Parsing dataset, Pedestrian Parsing Surveillance Scenes dataset, Horse-Cow parsing dataset, and PASCAL Quadrupeds dataset.

Existing visual tracking methods face many challenges: 1) the changed size and number of targets over time, occlusion in discrete frames, and mis-identification for crossing targets. Long short-term memory (LSTM) has the advantage of modeling long-term tasks and is suitable for tracking. We propose a novel online attentional recurrent neural network (ARNN) model for visual tracking, whose core component is a two-layer bidirectional LSTM along the x-and y-axes. Several bidirectional LSTMs can be cascaded or parallelly connected together to exploit multiscale target features and can give more precise tracked object locations. Each bidirectional LSTM utilizes the convolutional features of a convolutional neural network inside two bounding boxes from two frames to check whether the target in the current frame is the one in previous frames. An attention mechanism is also adopted to enhance the proposed model to better express the patch-level features of the tracking targets. Interattention and intra-attention models are proposed to imitate the temporal and spatial tracking mechanism of primate visual cortex. Interattention learns to overcome the occlusion problem, and intra-attention is able to mark important regions to better trace the target. The bidirectional LSTM and the attention mechanism are jointly trained. The combination of them further improves the accuracy of target tracking in videos. The outstanding performances in the experiments demonstrate the effectiveness of our proposed online method ARNN and yield competitive results compared with the state-of-the-art tracking methods.

We propose a novel online Attentional Recurrent Neural Network (ARNN) model for visual tracking, which exploits the feature maps of Convolutional Neural Network (CNN) inside a bounding box to identify whether this target is the one appeared in previous frames. Attention mechanism is adopted for both different parts of targets and different scales of object features. The former attention model is able to select important regions to better trace the target while the latter one learns to weight the multiple scale features for accurate object location. We jointly train the recurrent network with the region based and scale based attention mechanism. The outstanding performances in the experiments validate the effectiveness of our proposed ARNN and show that ARNN outperforms the state-of-the-art tracking methods.

Existing deep neural networks, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), typically treat volumetric video data as several single images and deal with one frame at one time, thus the relevance to frames can hardly be fully exploited. Besides, depth context plays the unique role in motion scenes for primates, but is seldom used in no depth label situations. In this paper, we use a more suitable architecture Multi-Scale Pyramidal Multi-Dimensional Long Short Term Memory (MSPMD-LSTM) to reveal the strong relevance within video frames. Furthermore, depth context is extracted and refined to enhance the performance of the model. Experiments demonstrate that our models yield competitive results on Youtube-Objects dataset and Segtrack v2 dataset.

Existing joint models of deep Convolutional Neural Networks (CNNs) and Conditional Random Fields (CRFs) face two problems for object segmentation: 1) CNNs can hardly extract high level features; 2) fully connected layers of CNNs are lack of capability of dealing with structured multi-level features. To address these problems, we utilize a Structured Random Forests based border ownership detection method to extract high level border features, which simulates the function of humans secondary visual cortex (V2). Moreover, an improved Conditional Boltzmann Machines (CBMs) are proposed to model predicted labels, local and global contexts of objects with multi-scale and multilevel features. Meanwhile, the proposed model inherits the merits of CNN, i.e., the good simulation of low level feature extraction ability in primary visual cortex (V1). Experiments demonstrate that our models yield competitive results on PASCAL VOC 2012 dataset.