This paper introduces architecture with high throughput, low on-chip memory, and efficient data access for Improved Dense Trajectories (iDT) as video representations for realtime action recognition. The iDT feature can capture longterm motion cues better than any existing deep feature, which makes it crucial in state-of-the-art action recognition systems. There are three major features in our architecture design, including a low bandwidth frame-wise feature extraction, low on-chip memory architecture for point tracking, and two-stage trajectory pruning architecture for low bandwidth. Using TSMC 40nm technology, our chip area is 3.1 mm2, and the size of on-chip memory is 40.8 kB. The chip can support videos in resolution of 320×240 with a throughput of 203 fps under 215 MHz, which is a 81.2 times speedup compared with CPU. Under the same operating frequency, it can also provide feature extraction for six windows of size 320 × 240 in higher resolution videos with a throughput of 34 fps.

In this paper, we propose a compact hardware architecture which can support HOG, HOF, MBHx and MBHy feature extraction in real-time. To achieve low-cost hardware, we first design a hardware-oriented algorithm. In addition to simplification of complicated numeric computation, we also modify the computation of the gradient at boundary to reduce on-chip buffers. The architecture is highly parallel in order to fully reuse input data and to achieve high throughput. Other features of our architecture include a four-bank memory composed of four SRAMs, a 2-cycle weighted binning, and a 2-port memory for histogram collection. The final specification can support HOG, HOF, MBHx, and MBHy feature extraction with a throughput of 65 fps in full-HD videos operating at 215 MHz, consuming only 57.6 kB on-chip memory and a gate count of 411k.

In this paper, we propose an accelerated local feature extraction in a reuse scheme for action recognition. Most local features of the previous frame could be reused due to the high correlation between successive frames. Feature extraction is only needed to be applied partially in the current frame. The full-frame features of the current frame are combined by features extracted at different times. Experimental results show that the proposed reuse scheme can achieve comparable performance to full-frame feature extraction while computational effort is reduced significantly.

In this paper, we focus on interaction prediction which infers to what interaction might happen in the near future. Each interaction is modeled by mixtures of deformable parts in order to provide higher tolerance to part configurations. In our weakly supervised learning setting, part detectors are learned from training data without bounding boxes around the true locations of the people in each frame. The discriminating features are obtained using a two-layer Linear Discriminant Analysis (LDA) classification to promise maximal separability for parts and interactions respectively. Experimental results demonstrate that the proposed system is effective in learning part-based models in less annotated information and achieves comparable performance to state-of-the-art fully supervised approaches.

In this paper, we propose a novel concept of region-of-unpredictable (ROU) to accelerate full-frame feature generation in video sequences. Due to the high correlation between successive frames, there are only few regions in which the features could not be estimated accurately from the previous frame called region-of-unpredictable (ROU). We develop a scheme combining partial feature extraction in ROU with feature prediction from the previous frame. The full-frame features of the current frame can then be obtained to minimize information loss. Experimental results show that the ROU determination algorithm supports 95.71% detection rate. The full-frame feature generation scheme using ROU determination saves 79.38% computational time compared with the full-frame feature extraction.

This paper presents a novel learning algorithm of fuzzy perceptron neural networks (FPNNs) for classifiers that utilize expert knowledge represented by fuzzy IF-THEN rules as well as numerical data as inputs. The conventional linear perceptron network is extended to a second-order one, which is much more flexible for defining a discriminant function. In order to handle fuzzy numbers in neural networks, level sets of fuzzy input vectors are incorporated into perceptron neural learning. At different levels of the input fuzzy numbers, updating the weight vector depends on the minimum of the output of the fuzzy perceptron neural network and the corresponding nonfuzzy target output that indicates the correct class of the fuzzy input vector. This minimum is computed efficiently by employing the modified vertex method. Moreover, the fuzzy pocket algorithm is introduced into our fuzzy perceptron learning scheme to solve the nonseparable problems. Simulation results demonstrate the effectiveness of the proposed FPNN model.

This paper proposes an automatic facial expression recognition system using neural network classifiers. First, we use the rough contour estimation routine, mathematical morphology, and point contour detection method to extract the precise contours of the eyebrows, eyes, and mouth of a face image. Then we define 30 facial characteristic points to describe the position and shape of these three organs. Facial expressions can be described by combining different action units that are used for describing the basic muscle movement of a human face. We choose six main action units, being composed of facial characteristic points movements, as the input vectors for two different neural network-based expression classifiers including radial basis function network and multilayer perceptron network. Using these two networks, we have obtained the same recognition rate as high as 92.1%. Simulation results by the computer demonstrate that computers are capable of extracting high-level or abstract information like human.

We propose a modified single-layer perceptron (MSLP) segmentation network for object extraction. We select a sigmoid gray level transfer function from the histogram of the input image and map the input gray levels into the interval [0,1]. Then we adopt the linear index of fuzziness of the output nodes as the error function of the image segmentation system to incorporate the learning capability of a neural network. Our scheme can successfully extract multiple objects with different gray levels. To further enhance the capability of the segmentation system, the proposed network is incorporated with fuzzy if-then rules to adaptively adjust the threshold of the activation function of the output neuron for best matching the local characteristics of the image. Fuzzy if-then rules involving the edge intensities and vertical positions of pixels are reasoned to determine the threshold adaptively. From the result of segmenting the forward looking infrared image, a good segmentation image was obtained by using the fuzzy MSLP segmentation technique.

In this paper, we describe an automatic unsupervised texture segmentation scheme using hidden Markov models (HMMs). First, the feature map of the image is formed using Laws' micromasks and directional macromasks. Each pixel in the feature map is represented by a sequence of 4-D feature vectors. The feature sequences belonging to the same texture are modeled as an HMM. Thus, if there are M different textures present in an image, there are M distinct HMMs to be found and trained. Consequently, the unsupervised texture segmentation problem becomes an HMM-based problem, where the appropriate number of HMMs, the associated model parameters, and the discrimination among the HMMs become the foci of our scheme. A two-stage segmentation procedure is used. First, coarse segmentation is used to obtain the approximate number of HMMs and their associated model parameters. Then, fine segmentation is used to accurately estimate the number of HMMs and the model parameters. In these two stages, the critical task of merging the similar HMMs is accomplished by comparing the discrimination information (DI) between the two HMMs against a threshold computed from the distribution of all DI's. A postprocessing stage of multiscale majority filtering is used to further enhance the segmented result. The proposed scheme is highly suitable for pipeline/parallel implementation. Detailed experimental results are reported. These results indicate that the present scheme compares favorably with respect to other successful schemes reported in the literature.

In this correspondence, we have presented a rotation and gray scale transform invariant texture recognition scheme using the combination of quadrature mirror filter (QMF) bank and hidden Markov model (HMM). In the first stage, the QMF bank is used as the wavelet transform to decompose the texture image into subbands. The gray scale transform invariant features derived from the statistics based on first-order distribution of gray levels are then extracted from each subband image. In the second stage, the sequence of subbands is modeled as a hidden Markov model (HMM), and one HMM is designed for each class of textures. The HMM is used to exploit the dependence among these subbands, and is able to capture the trend of changes caused by rotation. During recognition, the unknown texture is matched against all the models. The best matched model identifies the texture class. Up to 93.33% classification accuracy is reported.>

In the first stage of the proposed scheme the quadrature mirror filter (QMF) bank is used as the wavelet transform to decompose the texture image into subbands. Gray scale transform invariant features are then extracted from each subband image. In the second stage, the sequence of subbands is modeled as a hidden Markov model (HMM), and one HMM is designed for each class of textures. During recognition, the unknown texture is matched against all models. The best matched model identifies the texture class. The angles of rotation in the experiments are selected randomly in between -90 degrees and 90 degrees . Up to 95% classification accuracy is reported.>

An application of quadrature mirror filter (QMF) bank based subband decomposition to texture analysis is presented. Two-dimensional four bands QMF structure is used and the QMF features are introduced. The low-low band extracts the information of spatial dependence, and the low-high, high-low and high-high bands extract the structural information. This approach has the twin advantages of efficient information extraction and computation reduction. The classification abilities of QMF features are compared to those of Haralick features. The experiments demonstrate that the QMF features have better performance than the Haralick features.>