This paper proposes a method that utilizes two stages of deep learning networks for segmentation and recognition of spinal vertebrae. The first stage involves a y-shaped model that learns the basic shape of vertebrae to segment the spine. The second stage determines the order of each vertebra in the spine. This two-stage method provides more reliable spine segmentation results.

This paper proposes an object verification method by using sparse representation (SR) which has been applied for object representation and recognition. However, SR dictionary does not show sufficient compactness. Our method comprises three major modules. First, we train the sparse matrix by using boost K-Singular Value Decomposition (boost K-SVD) to obtain a sparse vector set. Second, we combine two training sparse vector sets of the same and different objects from two views to generate a positive/negative combined sparse vector set. Finally, a Support Vector Machine (SVM) classifier is applied for verification. Our contributions are (1) obtaining a sparser vector set using K-SVD, (2) demonstrating the SR matrix with better Restricted Isometry Property (RIP), and (3) applying the SR matrix to the object verification process with high accuracy. The experimental results prove that our method has higher accuracy than the other methods.

This paper presents a novice method for human and car identification in H.264/AVC compressed video domain. By analyzing the shape and motion vector homogeneity of the segmented objects, we can identify car and human. Our system consists of three main processes: (1) Moving object segmentation based on clustering MVs and Markov Random Field (MRF) iteration, (2) Feature Extraction based on motion analysis to obtain the difference of MVs direction (dMVD) and shape analysis to find the number of MBs (nMB) of an object, and (3) Object classification using Bayesian Classifier. In the experiments, we show that the recognition rate of car and human are 88% and 98% respectively.

This paper presents a vision-based continuous sign language recognition system to interpret the Taiwanese Sign Language (TSL). The continuous sign language, which consists of a sequence of hold and movement segments, can be decomposed into non-signs and signs. The signs can be either static signs or dynamic signs. The former can be found in the hold segment, whereas the latter can be identified in the combination of hold and movement segments. We use Support Vector Machine (SVM) to recognize the static sign and apply HMM model to identify the dynamic signs. Finally, we use the finite state machine to verify the correctness of the grammar of the recognized TSL sentence, and correct the miss-recognized signs.

This paper presents a new method to detect slip and fall events by analyzing the integrated spatiotemporal energy (ISTE) map. ISTE map includes motion and time of motion occurrence as our motion feature. The extracted human shape is represented by an ellipse that provides crucial information of human motion activities. We use this features to detect the events in the video with non-fixed frame rate. This work assumes that the person lies on the ground with very little motion after the fall accident. Experimental results show that our method is effective for fall and slip detection.

In this paper, we propose a gait analysis method which extracts the dynamic and static information from human walking for walking path and identity recognition. First, we utilize the periodicity of swing distances to estimate the gait period for each gait sequence. For each gait cycle, we extract the dynamic information by analyzing the statistic histogram of motion vectors and static information using Fourier descriptors. The extracted information is transformed into lower dimensional embedding space to represent the subject. Given a test feature vector, the nearest neighbor classifier is applied to compare with the feature vectors in the gait database for human object identification. The proposed algorithm is evaluated on the CASIA gait database, and the experimental results demonstrate this new system achieves a high recognition rate.

The attention analysis of multimedia data is challenging since different models have to be constructed according to different attention characteristics. This paper analyzes how people are excited about the watched video content and proposes a content-driven attention ranking strategy which enables client users to iteratively browse the video according to their preference. The proposed attention rank (AR) algorithm, which is extended from the Google PageRank algorithm that sorts the websites based on the importance, can effectively measure the user interest (UI) level for each video frame. The degree of attention is derived by integrating the object-based visual attention model (VAM) with the contextual attention model (CAM), which not only can more reliably take advantage of the human perceptual characteristics, but also can effectively identify which video content may attract users' attention. The information of users' feedback is utilized in re-ranking procedure to further improve the retrieving accuracy. The proposed algorithm is specifically evaluated on broadcasted baseball videos.

This paper proposes a novel multi-class hybrid-boost learning algorithm for multi-pose face detection and facial expression recognition. This system detects human face in different sizes, various poses, partial-occlusion, and different expressions. The contribution of this paper is the hybrid boosting algorithm combining the Haar-like (local) features and Gabor-like (global) features. The experimental results show that our system has better performance than the others.

This paper presents a real-time pointing arm tracking and identification system. It uses three cameras to track the pointing arm and identifies the pointed targets in 3-D space. The user make the arm pointing gesture at anyplace in the working space. Our system uses particle filtering and direct linear transformation (DLT) to track two 3-D points on the pointing arm and applies the epipolar geometry to refine the tracking results. In the experiments, we demonstrate that the hit rate is above 90%.

This paper proposes an automatic eye wink interpretation system for the severely handicapped people. First, we apply the support vector machine (SVM) and template matching algorithm to detect the eyes and then track the eye winks. Then, we convert the sequence of eye winks into code (binary digit) sequence. Finally, we use the dynamic programming to translate the eye wink sequence to a certain command for human-machine interface. In the experiments, our system demonstrates very good performance and high accuracy.

Gait recognition is a process of identifying individuals by the way they walk. Gait is often used as a unobstrusive biometric offering the possibility to identify people at a distance without any interaction or co-operation with the subject. This paper presents a novel method for both automatic viewpoint and person identification using only the silhouette sequence of gait. The gait silhouettes are nonlinearly transformed into low dimensional embedding and the dynamics in time-series images are modeled by HMM in the corresponding embedding space. The experimental results demonstrate that the proposed algorithm is an encouraging progress for the gait analysis research.

This paper proposes a semantics understanding system to interpret the superimposed caption box (SCB) in sports videos of which the template of SCB is presumably not a priori. The embedded captions in sports video programs represent digested key information of the video content. Most of the previous studies assume that the SCB template and the character bitmaps are known. Nevertheless, the representative character bitmaps are required for recognizing the captions. This paper has the following novelty: (1) SCB extraction and identification, (2) symbol extraction, and (3) semantic interpretation of the identified captions and symbols. Experimental results show that the algorithm performs the SCB contents understanding of several commercial sports video programs.

In this paper, we propose new video attention modeling and content-driven mining strategies which enable client users to browse the video according to their preference. By integrating the object-based visual attention model (V'AM) with the contextual attention model (CAM), the proposed scheme not only can more reliably take advantage of the human perceptual characteristics but also effectively discriminate which video contents may attract users' attention. In addition, extended from the Google PageRank algorithm which sorts the websites based on the importance, we introduce the so-call content-based attention rank (AR) to effectively measure the user interest (UI) level of each video frame. The information of users' feedback is treated as the enhanced query data to further improve the retrieving accuracy. The proposed algorithm is evaluated on commercial baseball game sequences and produces promising results.

We propose a subspace-based maximum-likelihood approach for acquiring the timing of wavelet-based multirate transmission systems. The S-curve analysis shows that our approach can correctly acquire the initial timing of a symbol from anywhere within a symbol-time interval at a cost of increasing jitter variance.

We propose a new framework for multiple scalable bitstream video communications over lossy channels. The major feature of the framework is that the encoder estimates the effects of postprocessing concealment and includes those effects in the rate-distortion analysis. Based on the framework, we develop a rate-distortion optimization algorithm to generate multiple scalable bitstreams. The algorithm maximizes the expected peak signal-to-noise ratio by optimally assigning forward error control codes and transmission schemes in a constrained bandwidth. The framework is a general approach motivated by previous methods that perform concealment in the decoder, as in our special case. Simulations show that the proposed approach can be implemented efficiently and that it outperforms previous methods by more than 2 dB.

We propose an error control scheme for video communications over lossy channels. The proposed algorithm uses receiver post-processing in analysis to coordinate with multiple encoded streams which is capable of handling error concealment and error protection to achieve robust transmissions. Unlike previous methods, our algorithm focuses on joint optimizing multiple substreams distortion with concealment over error prone channels. The algorithm minimizes the expected rate-distortion function to achieve the optimal FEC result. In the experiments, we demonstrate the effectiveness of our method by using a 3-D SPIHT algorithm. Simulation results show that the proposed protection strategy achieves about 2 dB higher peak signal-to-noise ratio compared to conventional method.

This paper presents a novel semantic-oriented video analysis system for the basketball game videos. Based on Bayesian belief network (BBN), it may bridge this gap between the low-level features describing image/video structure and the high-level knowledge. We apply the support vector machine (SVM) to identify and track the ball, the shooter, and the basket as the low-level features. Based on these features, our BBN framework can identify four categories of shot event such as short shot, medium shot, long shot, free throw, and the scoring event. In the experiments, we demonstrate that our system may interpret the video shots in terms of four different shot events and one scoring event effectively

This paper proposes a new algorithm for H.264 rate control. First, we estimate the frame target bit based on the proportional-integral-derivative (RAPID) buffer control to avoid the buffer overflow or underflow. With the frame target bit, we use the quadratic R-D model to estimate the frame quantization parameter (QP). Finally, we apply Kalman filter to further dynamically adjust macroblock QP to achieve accurate bit rate while maximizing the picture quality and simultaneously handling buffer fullness. In comparison with JM7.6, our scheme provides better video quality with reduced PSNR variations and less buffer overflow/underflow even at low bit rate. Most of all, our scheme also achieves an average PSNR gain of 0.127dB and generates coding bits close to target bits due to the accurate MB QP of Kalman filter estimator

In this paper, we propose an error control scheme for video communications over lossy channels. The proposed algorithm uses error concealment protection (ECP) approach to coordinate with multiple encoded streams which is capable of handling error concealment to achieve robust transmissions. Unlike previous methods, our algorithm focuses on joint optimizing multiple substreams distortion with concealment over error prone channels. The algorithm minimizes the expected rate-distortion function to achieve the optimal FEC result. In the experiments, we demonstrate the effectiveness of our method by using a 3-D SPIHT algorithm. Simulation results show that the proposed protection strategy achieves about 2 dB higher peak signal-to-noise ratio compared to conventional method.

This paper proposes a semantics understanding system to interpret the superimposed caption box (SCB) in sports videos of which the template of SCB is presumably not a priori. The semantics in SCB represent the digested key information of the sports video content. Most of the previous studies assume that the SCB template is known as a priori, and require the representative character bitmaps. The SCB understanding system consists of SCB extraction, character extraction, digit/text character recognition, digital object interpretation using labeling mechanism. In the experiments, we will illustrate that the system can interpret the SCB in basketball and baseball game videos successfully

Video semantic analysis is formulated based on the low-level image features and the high-level knowledge which is encoded in abstract, nongeometric representations. This paper introduces a semantic analysis system based on Bayesian network (BN) and dynamic Bayesian network (DBN). It is validated in the particular domain of soccer game videos. Based on BN/DBN, it can identify the special events in soccer games such as goal event, corner kick event, penalty kick event, and card event. The video analyzer extracts the low-level evidences, whereas the semantic analyzer uses BN/DBN to interpret the high-level semantics. Different from previous shot-based semantic analysis approaches, the proposed semantic analysis is frame-based for each input frame, it provides the current semantics of the event nodes as well as the hidden nodes. Another contribution is that the BN and DBN are automatically generated by the training process instead of determined by ad hoc. The last contribution is that we introduce a so-called temporal intervening network to improve the accuracy of the semantics output.

This paper proposes a novel semantics-based content analysis system for reliable media highlight extraction using dynamic Bayesian network (DBN). It extracts the low-level evidences and then converts the input video to high-level semantic meaning. Specific domains contain rich spatial and temporal transitional structures that help the transformation process. We introduce a robust audio-visual low-level evidence extraction scheme, and develop the so-called temporal intervening network to improve the performance of our system. In experiments, we show that our system can detect soccer events such as goal event, corner kick event, penalty kick event, and card event effectively.

We propose an efficient error-resilient video transmission algorithm over packet erasure channels using optimal source and channel bit-allocation. This algorithm uses FEC and rate-distortion optimization to find the optimal allocation of source and channel bits in each quality layer. The packet loss probability is periodically reported to the video server. This method can also be incorporated with any coding structure that generates a set of independent compressed bitstreams. The algorithm efficacy is demonstrated by simulations in which the video compression is an error-resilient 3D-SPIHT algorithm and the channel protection is provided by Reed-Solomon (RS) codes.

This paper presents a real time vision-based human motion capturing and recognition system using two calibrated CCD cameras. We propose a simple but effective method to estimate the motion parameters (BAPs) of the human object by analyzing the vertical projection profile and the horizontal projection profile in each view to identify different arm and leg postures. With the identified postures, we can apply the Kalman filtering to capture the motion parameters (joint angles). Our method is divided into macro motion analysis and micro motion analysis. The former identifies certain well-defined postures and the latter traces the variation of joint angle or BAPs. In the experiments, we test 22 different arm and leg postures and show the errors of the estimated BAPs.

This paper proposes a new error resilient video coding scheme using the matching pursuit (MP) residual coding and the multiple description coding. A conventional discrete cosine transform-based video coding scheme introduces undesirable blocking artifacts at low bit rates. MPs for residual coding has been developed to overcome the disadvantage. The parameters obtained after MP, called atoms, are encoded into two balanced descriptions using scalar quantizers. The two descriptions are respectively transmitted over two separate channels. If both descriptions are received, a high-quality reconstruction can be obtained, however, if either description is lost, a low-quality, but acceptable, reconstruction can be acquired.

We propose a clock acquisition algorithm for timing recovery for fractal modulation in additive white Gaussian noise (AWGN) and 1/f channels. Our acquisition algorithm uses exclusively the data redundancy inherently built in the fractal modulation to locate the beginning of timing of all the subbands simultaneously. Different diversity techniques can then be applied after the beginning of the data block is obtained. Our acquisition functions are derived by using the maximum-likelihood method. Using a serial search algorithm to maximize the acquisition function, we may find the timing point. Simulations have been conducted in evaluating the mean acquisition time of our algorithm.

The paper introduces a highlight detection method that consists of: (1) an inference scheme for semantic feature extraction; (2) a semantic highlight detection scheme using a multi-level semantic network (MSN). MSN is basically a Bayesian belief network (BBN) which consists of the low-level feature nodes connected to the mid-level semantic nodes and finally the high-level semantic nodes. A probabilistic structure can be combined by visual features and can be applied on highlight detection. Satisfactory results are shown.

The acquisition problem in wavelet-based modulation is very important. We discuss the problem of the ML-based method proposed by M. Luise et al. (see IEEE Trans. Commun., vol.48, p.1047-54, 2000) for timing acquisition and then we develop a novel acquisition algorithm. Our method uses the properties of the scaling function in the derivation of the acquisition function. We also show that the S-curve of our acquisition algorithm is smooth and has a unique zero during the signaling interval. Thus, we can acquire the correct symbol timing without ambiguity. The performance of our acquisition algorithm is evaluated by Monto-Carlo simulation using the Meyer wavelet.

The paper proposes a video communication scheme combing multiple description coding (MDC) using scalar quantizers with matching pursuit (MP) to transmit video over the Internet. The parameters obtained after MP, called atoms, are encoded into two balanced descriptions, which are transmitted over two separate channels. Simulation results show that our proposed method surpasses MP single description coding (SDC) and H.263 with FEC as well as MP MDC using shared atoms.

The exploitation of semantic information in videos is difficult because of the large difference in representations, levels of knowledge and abstract episodes. Traditional image/video understanding and indexing is formulated in terms of low-level features describing image/video structure and intensity, while high-level knowledge such as common sense and human perceptual knowledge are encoded. This paper attempts to bridge this gap through the integration of image/video analysis algorithms with multi-level semantic network to interpret the baseball video.

This paper introduces a real time model-based human motion analysis system using two orthogonal views. The goal of the system is to analyze the motion of the human object without using any markers on the object. The system will generate the corresponding body definition parameters (BDPs) and body animation parameters (BAPs) defined by MPEG-4. With the BDPs and a sequence of BAPs, a real-time virtual actor can be synthesized on various platforms.

Video streaming is the major subject of Amendment for MPEG-4 and it is developed in response to the growing needs on a video-coding standard for the video communication. The fine-granular scalability (FGS) combined with the temporal scalability addresses a variety of challenging problems in delivering video. The FGS video encoder makes the coding mode decision based on the video content and the current available bandwidth in order to achieve higher perceptual video quality. In this paper, we develop a mode selection method to find the most suitable scalable coding mode from six coding schemes: FGS, FGST, FGS-SE, and FGST with background composition based on the contents of the video sequences.

We propose a naive algorithm for the timing recovery for fractal modulation. We first investigate the bit error rate due to the clock error of a fractal modulation. Then, we propose clock synchronization techniques for fractal modulation. Our acquisition algorithm uses exclusively the self-similar property of fractal for signal modulation. The performance of our algorithm is also demonstrated.

This paper proposes a joint source and channel coding (JSCC) method for MPEG-2 video transmission. Based on the current channel status, this JSCC method finds the best source and channel code rate distribution. Due to the different bit error sensitivity of the source code. we apply unequal error protection (UEP) for the source bits by using so-called rate compatible punctured convolution (RCPC) channel coder for efficient protection during channel transmission. In the simulations. we illustrate that under the conditions of the same channel status and total bit rate. our method provides the minimal overall distortion of the MPEG-2 video compared to the equal error protection(EEP) coding.

This paper proposes a new method that combines the intensity and motion information to detect scene changes such as abrupt scene changes and gradual scene changes. Two major features are chosen as the basic dissimilarity measures, and self- and cross-validation mechanisms are employed via a static scene test. We also develop a novel intensity statistics model for detecting gradual scene changes. Experimental results show that the proposed algorithms are effective and outperform the previous approaches.

We propose a codebook adaptation algorithm for matching pursuit low bit rate video compression. The matching pursuit dictionary is a set of basis functions. In practice, a function is approximated as a vector. According to the incoming data, the basis functions are modified similar to the minimum-point-finding algorithm in stochastic regression. Although matching pursuit low bit rate compressions have been studied intensively, bases adaptation in the dictionary is a new approach. We demonstrate the performance of our adaptation method on MPEG-4 video sequences.

Describes an automatic new road sign interpretation method using matching pursuit (MP) filters. There are two processes. The detection process finds the relative position of the road sign in the original distant image and then extracts the internal content of the road sign from the closer view image. The recognition process consists of two stages: training and testing. In the first stage, it finds a set of best MP filter bases for each road sign. In the second stage, it projects the input unknown road sign to a different set of the MP filter bases to find the best match.

This paper introduces a multi-principal-distribution-model (PDM) method and hidden Markov model (HMM) for gesture image sequence interpretation. To track the hand-shape, it uses the PDM model which is built by learning patterns of variability from a training set of correctly annotated images. For gesture recognition, we need to deal with a large variety of hand-shapes. Therefore, we divide all the training hand shapes into a number of similar groups, with each group trained for an individual PDM shape model. Finally, we use the HMM to determine model transition among these PDM shape models. From the model transition sequence, it can identify the continuous gestures denoting one-digit or two-digit numbers.

We show that the techniques of gain-shape VQ can be used in optimizing the dictionary of the matching pursuit codec. Various performance of our method were evaluated.

A modified Hough transform based on a likelihood principle of connectivity and thickness is proposed for line detection. It makes short as well as thick line segments easier to detect in a noisy image. Certain desirable properties of the new method are justified by theory and simulations.

This paper introduces the complex subband transform (CST) that can be applied for both subband signal decomposition and motion estimation, which are the two major processes in subband-based image sequence coding. In the experiments, we compare the CST-based subband motion compensation with conventional block matching motion compensation, and find that the former has (i) higher peak signal-to-noise ratio (PSNR) of the reconstructed images and (ii) lower prediction error entropy of the motion vectors.

The duration of a positron emission tomography (PET) imaging scan can be reduced if the transmission scan of one patient which is used for emission correction can be synthesized by using the reference transmission scan of another patient. In this paper, we propose a new intersubjects PET emission scan registration method and PET transmission synthesis method by using the boundary information of the body or brain scan of the PET emission scans. The PET emission scans have poor image quality and different intensity statistics so that we preprocess the emission scans to have similar histogram and then apply the point distribution model (PDM) [15] to extract the contours of the emission scan. The extracted boundary contour of every slice is used to reconstruct the three-dimensional (3-D) surface of the reference set and the target set. Our registration is 3-D surface-based which uses the normal flow method [17] to find the correspondence vector field between two 3-D reconstructed surfaces. Since it is difficult to analyze internal organ using the PET emission scan imaging without correction, we assume that the deformation of internal organ is homogeneous. With the corresponding vector field between the two emission scans and the transmission scan of the reference set, we can synthesize the transmission scan of the target set.

This paper presents an object-based layer-structure very low bit rate video coding system. This system is different from the conventional object-based video coding algorithms in three aspects. First, it uses the elastic stretching technique to do the object-based predictive coding. Second, to reduce the coding cost of the contours, it applies contour matching and contour displacement to predict the contour in the next frame. Third, it uses the arbitrary shape transform (AST) coding to reduce the coding bit rate for the prediction error in the arbitrarily-shaped regions.

This paper introduces a ribbon-based motion analysis approach to describe human body movements. Here, we assume that there are no markers on human body. We develop a system to extract the moving ribbons (extremities) by processing the difference between current image frame and reference image frame. By analyzing the moving ribbons on the key frames, we may produce the motion parameter curves for each joint on the ribbon. These curves may not be continuous due to ribbon-torso occlusion, and both the interpolation and extrapolation processes can be used to predict the missing parts.

A new method called complex subband transform (CST) is introduced for subband signals decomposition, the motion estimation, and the subband image sequence coding. There are two subband-based methods proposed which combine the process of motion compensation and the process of subband decomposition namely out-band and in-band compensation. We modify the M-band perfect reconstruction modulation filter (PRMF) and propose a so-called complex subband transform (CST). We also derive a method using CST-based phase correlation functions to calculate motion vectors for overlapped windowed regions. In the experiments, we show that the CST-based motion estimation generates a more accurate and coherent motion field than the conventional block matching methods.

This paper presents a sign language recognition system which consists of three modules: model-based hand tracking, feature extraction, and gesture recognition using a 3-D Hopfield neural network. In the experiments, we illustrate that this system can recognize 15 different gestures accurately.

Presents a new motion estimation method for low bit-rate image sequence coding that uses a hierarchical decomposition of the current image frame to describe scene motion in terms of displacement and deformation of variable-sized rectangular regions. Because the conventional blockmatching motion compensation method (BMA) can only cope with the problem of translational movement of the scene, some researchers have proposed deformable-block-based motion compensation such as control grid interpolation (CGI) and the triangle-based method (TBM). CGI begins with a spatial displacement for a small number of points in an image, termed control points. TBM partitions the image frame into triangular patches with equal size that are deformable during the motion compensation. These methods do not consider using the different motion characteristics and the shape properties of the moving objects, but instead apply the same motion estimation algorithm to every part of the scene which may be stationary or moving. The present authors propose a new motion compensation method called hierarchical grid interpolation (HGI) which segments a frame into various quadrangles with different sizes (described by quadtree segmentation) according to the motion activity. In the experiments, the authors compare HGI with the conventional BMA and TBM. HGI requires less computation, produces less prediction error, and requires lower transmission bit-rate. It also achieves a subjective image quality that is better than the conventional motion compensation coder.>

This paper presents facial features extraction algorithms which can be used for automated visual interpretation and recognition of human faces. It is possible to capture the contours of eye and mouth by deformable template model because of their analytically describable shapes. However, the shapes of eyebrow, nostril and face are difficult to model using a deformable template. They are extracted by using an active contour model, the snake.>

This paper presents a face recognition system which can identify the unknown identity effectively using the front-view facial features. In front-view facial feature extractions, we can capture the contours of eyes and mouth by the deformable template model because of their analytically describable shapes. However, the shapes of eyebrows, nostrils and face are difficult to model using a deformable template. We extract them by using the active contour model (snake). After the contours of all facial features have been captured, we calculate effective feature values from these extracted contours and construct databases for unknown identities classification. In the database generation phase, 12 models are photographed, and feature vectors are calculated for each portrait. In the identification phase if any one of these 12 persons has his picture taken again, the system can recognize his identity.