Class incremental learning has yet to be explored in the field of lip-reading, which can circumvent data privacy issues and avoid the high training costs associated with joint training. In this paper, we introduce a benchmark for Class-Incremental Lip-Reading (CILR). To simultaneously improve the plasticity for new classes and stability for old classes in incremental learning, we propose a Slow-Fast Time Parameter Aggregation Network (TPAN) that decouples representation learning of new and old knowledge, taking into account the task characteristics of lip-reading. The TPAN comprises two dynamically evolving branches: one that uses fast gradient descent and the other employs slow momentum updates to retain old knowledge while adapting to new knowledge. Additionally, to achieve efficient knowledge transfer of the incremental model, we design a Hybrid Sequence-Distribution Distillation (HSDD) strategy to transfer knowledge in temporal feature view and classification probability view. We present a comprehensive comparison of the proposed method and previous state-of-the-art class incremental learning methods on the most commonly used lip-reading datasets LRW and LRW1000. The experimental result show that the proposed method can reduce the effect of catastrophic forgetting and improve the incremental accuracy.

Lip reading is the task of decoding text from speakers' mouth movements. Numerous deep learning-based methods have been proposed to address this task. However, these existing deep lip reading models suffer from poor generalization due to overfitting the training data. To resolve this issue, we present a novel learning paradigm that aims to improve the interpretability and generalization of lip reading models. In specific, a Variational Temporal Mask (VTM) module is customized to automatically analyze the importance of frame-level features. Furthermore, the prediction consistency constraints of global information and local temporal important features are introduced to strengthen the model generalization. We evaluate the novel learning paradigm with multiple lip reading baseline models on the LRW and LRW-1000 datasets. Experiments show that the proposed framework significantly improves the generalization performance and interpretability of lip reading models.

Identifying cell types is crucial for single-cell RNA sequencing (scRNA-seq) analysis and can be potentially utilized to understand high-level biological processes. Supervised models based on neural networks have recently been successfully applied in the scRNA-seq cell type classification problem and achieved promising results. While most existing works directly use the raw or transformed data, we argue that the original data are too sparse and high-dimensional, and extracting their effective low-dimensional features can better train downstream classifiers, thereby improving the cell type classification performance. In this paper, we propose a novel framework, named Deep Count Autoencoder-based Classifier (DCA-CLA), to leverage the discriminative low-dimensional features for classification. Specifically, DCA-CLA first denoises the original count matrix and extracts the data features from the hidden layer using a deep count autoencoder module, then it feeds these bottleneck features into the classifier network to train the learnable parameters and test the performance. Experimental results on eight separate datasets and four pairs of datasets demonstrate that the proposed DCA-CLA framework achieves competitive performance over the state-of-the-art frameworks.

Convolutional neural networks (CNNs) have attracted significant attention in visual recognition. Several recent studies have shown that, in addition to the fully connected layers, the features derived from the convolutional layers of CNNs can also achieve promising performance in image classification tasks. In this paper, we propose a new feature from the convolutional layers, called Bag of Surrogate Parts (BoSP), and its spatial variant, Spatial-BoSP (S-BoSP). The main idea is, we assume the feature maps in the convolutional layers as surrogate parts, and densely sample and assign image regions to these surrogate parts by observing the activation values. Together with BoSP/S-BoSP, we further propose another two schemes to enhance the performance: scale pooling and global-part prediction. Scale pooling aims to handle the objects with different scales and deformations, and global-part prediction combines the predictions of global and part features. By conducting extensive experiments on generic object, fine-grained object and scene datasets, we find the proposed scheme can not only achieve superior performance to the fully connected feature, but also produces competitive or, in some cases, remarkably better performance than the state of the art.

Scene recognition is an important and challenging task in computer vision. We propose an end-to-end pipeline by combing convolutional neural networks (CNNs) with explicit attention model to determine several meaningful regions of original images for scene recognition. In the proposed pipeline, the spatial transformer network is leveraged as the attention module, which can automatically learn the scales and movements of centers of attention windows. As for feature extraction, the basic CNN architecture is utilized. Furthermore, the stronger descriptors of scenes are constructed by feature fusion. The highlight of our proposed network is that it is capable to localize discriminative regions from an image in a data-driven manner without any additional supervision. We conduct experiments on a subset of the Places205 database to evaluate the performance of the proposed basic network and the involved parameters. Our model achieves state-of-the-art top-1 accuracy 82.10% on the evaluation dataset comparing with fine-tuned PlacesCNN (80.98%). We find that our model is able to learn informative attention regions for discriminating scene categories.

Local binary patterns (LBP) are considered among the most computationally efficient high-performance texture features. However, the LBP method is very sensitive to image noise and is unable to capture macrostructure information. To best address these disadvantages, in this paper, we introduce a novel descriptor for texture classification, the median robust extended LBP (MRELBP). Different from the traditional LBP and many LBP variants, MRELBP compares regional image medians rather than raw image intensities. A multiscale LBP type descriptor is computed by efficiently comparing image medians over a novel sampling scheme, which can capture both microstructure and macrostructure texture information. A comprehensive evaluation on benchmark data sets reveals MRELBP's high performance-robust to gray scale variations, rotation changes and noise-but at a low computational cost. MRELBP produces the best classification scores of 99.82%, 99.38%, and 99.77% on three popular Outex test suites. More importantly, MRELBP is shown to be highly robust to image noise, including Gaussian noise, Gaussian blur, salt-and-pepper noise, and random pixel corruption.

Local Binary Patterns (LBP) are among the most computationally efficient amongst high-performance texture features. However, LBP is very sensitive to image noise and is unable to capture macrostructure information. To best address these disadvantages, in this paper we introduce a novel descriptor for texture classification, the Median Robust Extended Local Binary Pattern (MRELBP). In contrast to traditional LBP and many LBP variants, MRELBP compares local image medians instead of raw image intensities. We develop a multiscale LBP-type descriptor by efficiently comparing image medians over a novel sampling scheme, which can capture both microstructure and macrostructure. A comprehensive evaluation on benchmark datasets reveals MRELBP's remarkable performance (robust to gray scale variations, rotation changes and noise) relative to state-of-the-art algorithms, but nevertheless at a low computational cost, producing the best classification scores of 99.82%, 99.38% and 99.77% on three popular Outex test suites. Furthermore, MRELBP is also shown to be highly robust to image noise including Gaussian noise, Gaussian blur, Salt-and-Pepper noise and random pixel corruption.

In this paper, we propose a simple, efficient, yet robust multiresolution approach to texture classification-binary rotation invariant and noise tolerant (BRINT). The proposed approach is very fast to build, very compact while remaining robust to illumination variations, rotation changes, and noise. We develop a novel and simple strategy to compute a local binary descriptor based on the conventional local binary pattern (LBP) approach, preserving the advantageous characteristics of uniform LBP. Points are sampled in a circular neighborhood, but keeping the number of bins in a single-scale LBP histogram constant and small, such that arbitrarily large circular neighborhoods can be sampled and compactly encoded over a number of scales. There is no necessity to learn a texton dictionary, as in methods based on clustering, and no tuning of parameters is required to deal with different data sets. Extensive experimental results on representative texture databases show that the proposed BRINT not only demonstrates superior performance to a number of recent state-of-the-art LBP variants under normal conditions, but also performs significantly and consistently better in presence of noise due to its high distinctiveness and robustness. This noise robustness characteristic of the proposed BRINT is evaluated quantitatively with different artificially generated types and levels of noise (including Gaussian, salt and pepper, and speckle noise) in natural texture images.

As the increasing affordability for capturing and storing video and the proliferation of Web 2.0 applications, video content is no longer necessarily created and supplied by a limited number of professional producers; any amateur can produce and publish his/her video quickly. Therefore, the amount of both professional-produced as well as amateur-produced video on the web is ever increasing. In this work, we propose a question; whether we can automatically classify an Internet video clip as being either professional-produced or amateur-produced? Hence, we investigate features and classification methods to answer this question. Based on the differences in the production processes of these two video categories, four features including camera motion, structure, audio feature and combined feature are adopted and studied along with with four popular classifiers KNN, SVM GMM and C4.5. Extensive experiments over representative datasets, evaluate these features and classifiers under different settings and compare to existing techniques. Experimental results demonstrate that SVMs with multimodal features from multi-sources are more effective at classifying video type. Finally, for answering the proposed question, results also show that automatically classifying a clip as professional-produced video or amateur-produced video can be achieved with good accuracy.

The white line marks on the soccer playfield can convey the positional information of the ongoing match. In this paper, we propose a new algorithm for automatic, robust line mark recognition for soccer video and its application in camera calibration is also investigated. The line mark recognition consists of four steps, playfield region detection, playfield area analysis, edge pixel detection and repairing, line mark location. The preliminary experiment shows that the line marks (including curved lines) can be located reliably (precision>90%), and its application in camera calibration is also effective especially when there are not enough straight lines.

Semantic concept detection is a very useful technique for developing powerful retrieval or filtering systems for multimedia data. To date, the methods for concept detection have been converging on generic classification schemes. However, there is often imbalanced dataset or rare class problems in classification algorithms, which deteriorate the performance of many classifiers. In this paper, we adopt three “under-sampling” strategies to handle this imbalanced dataset issue in a SVM classification framework and evaluate their performances on the TRECVid 2007 dataset and additional positive samples from TRECVid 2010 development set. Experimental results show that our well-designed “under-sampling” methods (method SAK) increase the performance of concept detection about 9:6% overall. In cases of extreme imbalance in the collection the proposed methods worsen the performance than a baseline sampling method (method SI), however in the majority of cases, our proposed methods increase the performance of concept detection substantially. We also conclude that method SAK is a promising solution to address the SVM classification with not extremely imbalanced datasets.

On one hand, goalmouth detection is a kind of high-level semantic concept detection methods in sports video, on the other hand, data mining for video processing that can benefit video analysis is a promising research frontier. In this paper, a robust method is proposed to detect the presence of goalmouth in field-ball game video based on fuzzy decision trees. Balance process is added in training procedure. The experiment result shows that our algorithm can improve the classification when comparing with the threshold based algorithm and the decision tree based algorithm. Fuzzy rules can also be easily deduced from the constructed tree to interpret the classification model.

The most common approach to automatic summarization and highlight detection in sports video is to train an automatic classifier to detect semantic highlights based on occurrences of low-level features such as action replays, excited commentators or changes in a scoreboard. We propose an alternative approach based on the detection of perception concepts (PCs) and the construction of Petri-Nets, which can be used for both semantic description and event detection within sports videos. Low-level algorithms to detect PCs using visual, aural and motion characteristics are proposed, and a series of Petri-Nets composed of PCs is formally defined to describe video content. We call this a perception concept network–Petri-Net (PCN–PN) model. Using PCN–PNs, personalized high-level semantic descriptions of video highlights can be facilitated and queries on high-level semantics can be achieved. A particular strength of this framework is that we can easily build semantic detectors based on PCN–PNs to search within sports videos and locate interesting events. Experimental results based on recorded sports video data across three types of sports games (soccer, basketball and rugby), and each from multiple broadcasters, are used to illustrate the potential of this framework.

In automatic video content analysis domain, the key challenges are how to recognize important objects and how to model the spatiotemporal relationships between them. In this paper we propose a semantic content analysis model based on Perception Concepts (PCs) and Finite State Machines (FSMs) to automatically describe and detect significant semantic content within sports video. PCs are defined to represent important semantic patterns for sports videos based on identifiable feature elements. PC-FSM models are designed to describe spatiotemporal relationships between PCs. And graph matching method is used to detect high-level semantic automatically. A particular strength of this approach is that users are able to design their own highlights and transfer the detection problem into a graph matching problem. Experimental results are used to illustrate the potential of this approach.

Anchorperson shot detection is an effective way of news video structuring. According to some features of anchorperson shots, such as visually similar with each other, long duration, and large span, etc., an anchorperson shot detection method based on clustering called AnchorClu is proposed. After fast clustering of shots’ key frames, some clusters including the anchorperson cluster can be got. Then by some proper rules, the anchorperson cluster is selected out, and the anchorperson shots can also be extracted to help the analysis of news video structuring. Experiments prove the effectiveness of the method, which is fast and gets the average recall of 93% and the average precision of 98%.

Feature analysis and extraction are the foundation of audio automatic classification. This paper divides audio streams into five classes: silence, noise, pure speech, speech over background sound and music. We present our work on audio feature analysis and extraction on the frame level and clip level. Four new features are proposed, including silence ratio, pitch frequency standard deviation, harmonicity ratio and smooth pitch ratio. We have presented an SVM based approach to classification. The effectiveness of the features is evaluated in experiments. Experiment results show that the features we selected and proposed are rational and effective.