This paper proposes a novel lightweight model training strategy called Knowledge Distillation Boosted Network Augmentation (KDBNA) for contact-free diagnosis and severity assessment of obstructive sleep apnea (OSA) through audio-based analysis of snoring events. Traditional OSA diagnosis relies on polysomnography (PSG), which is invasive, costly, and inconvenient for patients. To overcome these limitations, we developed a compact deep neural network by integrating knowledge distillation (KD) and network augmentation (NetAug), significantly reducing the parameter count of a ResNet-18 model to only 0.28M parameters—about 1/40 of its original size—without compromising performance. Experimental results show that our proposed method achieves higher sensitivity and F1-score compared to other lightweight approaches, while substantially reducing inference time on CPU-based platforms by approximately 87%. The proposed approach demonstrates strong potential for convenient, accurate, and cost-effective OSA screening, particularly suitable for implementation in edge computing devices and at-home healthcare scenarios.

Thermal image has many applications on image processing such as human detection, face recognition and physiological signal evaluation, etc. The respiratory rate is an important physiological signal, and it is highly related to emotion and some diseases. Therefore, we propose a non-contact method to estimate the respiratory rate from thermal image in this paper. Thermal image can provide the information about temperature distribution, so we can use this property to evaluate the breath signal on it. In this work, we use the visual image to detect the face and nose region to locate the ROI for breath signal extraction, then use the histogram equalization to enhance the contrast of thermal image. To make the breath signal more obvious, we proposed a method to find a mask from ROI with Otsu threshold method among some accumulated frames. In addition, Kanade-Lucas-Tomasi (KLT) tracking method is employed to tack the subject to prevent the ROI location error. Next, breath signal is decomposed by Hilbert-Huang Transform (HHT) into some intrinsic mode functions (IMFs). Finally, respiratory rate is estimated by counting the approximate zero-crossing points which are correlated to breathe cycle.

This paper proposes a novel vehicle color classification method which uses the concept of probabilistic latent semantic analysis (pLSA) to overcome the problem of sparse representation in data classification. Sparse representation is widely used and quite successful in many vision-based applications. However, it needs to calculate the sparse reconstruction cost (SRC) of each sample to find the best candidate. Because an optimization process is involved, it is very inefficient. In addition, it uses only the residual and does not consider the arrangement (or distribution) of combination coefficients of visual codes in classification. Thus, it often fails to classify categories if they are similar. In this paper, the pLSA concept is first introduced into the sparse representation to build a new classifier without using the SRC measure. The weakness of the pLSA scheme is the use of EM algorithm for updating the posteriori probability of latent class. Because it is very time-consuming, a novel weighting voting strategy is introduced to improve the pLSA scheme for recognizing objects in real time. The advantages of this classifier are: the accuracy is much higher than the SRC scheme and the efficiency is real-time in data classification. Vehicle color classification is demonstrated in this paper to prove the superiority of the new classifier.

This paper proposes a novel object classification method which uses the concept of probabilistic latent semantic analysis (pLSA) to overcome the problem of sparse representation in data classification. Sparse representation is widely used and quite successful in many vision-based applications. However, it needs to calculate the sparse reconstruction cost (SRC) of each sample to find the best candidate. Because an optimization process is involved, it is very inefficient. In addition, it uses only the residual and does not consider the arrangement (or distribution) of combination coefficients of visual codes in classification. Thus, it often fails to classify categories if they are similar. In this paper, the pLSA concept is first introduced into the sparse representation to build a new classifier without using the SRC measure. The weakness of the pLSA scheme is the use of EM algorithm for updating the posteriori probability of latent class. Because it is very time-consuming, a novel weighting voting strategy is introduced to improve the pLSA scheme for recognizing objects in real time. The advantages of this classifier are: the accuracy is much higher than the SRC scheme and the efficiency is real-time in data classification. Two applications are demonstrated in this paper to prove the superiority of the new classifier, i.e., vehicle make and model recognition, and action analysis.

Rain removal from a single color image is a challenging problem as no temporal information among successive images can be obtained. In this paper, we propose a single-color-image-based rain removal framework by properly formulating rain removal as an image decomposition problem based on sparse representation. In our framework, an input color image is first decomposed into a low-frequency part and a high-frequency part by using the guided image filter so that the rain streaks would be in the high-frequency part with nonrain textures/edges, and the high-frequency part is then decomposed into a rain component and a nonrain component by performing dictionary learning and sparse coding. To separate rain streaks from the high-frequency part, a hybrid feature set, including histogram of oriented gradients, depth of field, and Eigen color, is employed to further decompose the high-frequency part. With the hybrid feature set applied, most rain streaks can be removed; simultaneously nonrain component can be enhanced. To the best of our knowledge, compared with the state-of-the-art approaches, the proposed method is among the first to focus on the problem of single color image rain removal and achieves promising results with not only the rain component being removed more completely, but also the visual quality of restored images being improved.

Speeded-Up Robust Features (SURF) is a robust and useful feature detector for various vision-based applications but it is unable to detect symmetrical objects. This paper proposes a new symmetrical SURF descriptor to enrich the power of SURF to detect all possible symmetrical matching pairs through a mirroring transformation. A vehicle make and model recognition (MMR) application is then adopted to prove the practicability and feasibility of the method. To detect vehicles from the road, the proposed symmetrical descriptor is first applied to determine the region of interest of each vehicle from the road without using any motion features. This scheme provides two advantages: there is no need for background subtraction and it is extremely efficient for real-time applications. Two MMR challenges, namely multiplicity and ambiguity problems, are then addressed. The multiplicity problem stems from one vehicle model often having different model shapes on the road. The ambiguity problem results from vehicles from different companies often sharing similar shapes. To address these two problems, a grid division scheme is proposed to separate a vehicle into several grids; different weak classifiers that are trained on these grids are then integrated to build a strong ensemble classifier. The histogram of gradient and SURF descriptors are adopted to train the weak classifiers through a support vector machine learning algorithm. Because of the rich representation power of the grid-based method and the high accuracy of vehicle detection, the ensemble classifier can accurately recognize each vehicle.

SURF (Speeded Up Robust Features) is a robust and useful feature detector for various vision-based applications but lacks the ability to detect symmetrical objects. This paper proposes a new symmetrical SURF descriptor to enrich the power of SURF to detect all possible symmetrical matching pairs through a mirroring transformation. A vehicle make-and-model recognition (MMR) application is then adopted to prove the practicability and feasibility of the method. To detect vehicles from the road, the proposed symmetrical descriptor is first applied to determine the ROI of each vehicle from the road without using any motion features. This scheme provides two advantages; there is no need of background subtraction and it is extremely efficient for real-time applications. Two MMR challenges, i.e., multiplicity and ambiguity problems, are then addressed. The multiplicity problem stems from one vehicle model often having different model shapes on the road. The ambiguity problem results from vehicles from different companies often sharing similar shapes. To address these two problems, a grid division scheme is proposed to separate a vehicle into several grids; different weak classifiers that are trained on these grids are then integrated to build a strong ensemble classifier. Because of the rich representation power of the grid-based method and the high accuracy of vehicle detection, the ensemble classifier can accurately recognize each vehicle.

Given the rapid expansion of car ownership worldwide, vehicle safety is an increasingly critical issue in the automobile industry. The reduced cost of cameras and optical devices has made it economically feasible to deploy front‐mounted intelligent systems for visual‐based event detection. Prior to vehicle event detection, detecting vehicles robustly in real time is challenging, especially conducting detection process in images captured by a dynamic camera. Therefore in this study, a robust vehicle detector is developed. The proposed contribution is three‐fold. Road modelling is first proposed to confine detection area for maintaining low computation complexity and reducing false alarms as well. Haar‐like features and eigencolours are then employed for the vehicle detector. To tackle the occlusion problem, chamfer distance is used to estimate the probability of each individual vehicle. Consequently, to find the bounding box of vehicle candidate, the authors take the configuration of a normalised chamfer distance map that corresponds to the maximum score as the target using exponential entropy. Experiments on an extensive dataset show that the proposed system can effectively detect vehicles under different lighting and traffic conditions, and thus demonstrates its feasibility in real‐world environments.

State of the art video resizing methods usually produce perceivable visual discontinuities, especially in videos containing significant motion. To resolve the problem, contextual information about the focus of interest in consecutive video frames should be considered in order to preserve the visual continuity. In this paper, to detect the focus of interest with motion-tolerance, we propose a novel approach for modelling visual dynamics based on spatiotemporal slices (STS), which provide rich visual patterns along a large temporal scale. First, patch-based visual patterns are computed to generate a codebook of the automatically specified spatiotemporal extent determined by the contextual information in the STS. The codebook is then used to compute its associated response in each video frame, and eventually an importance map covering the focus of interest in a video clip can be obtained. To preserve the visual continuity of the content, particularly an important area, a multi-cue approach is used to guide a mesh-based non-homogeneous warping operation constrained by the trajectories in the STS. For the performance evaluation, we present a novel measure that utilizes patch-based Kullback-Leibler divergence (KL-divergence) to gauge the deformation of the focus of interest under the proposed video resizing approach. Experimental results show that the STS-based approach can generate retargeted videos effectively, while maintaining their isotropic manipulation and the continuous dynamics of visual perception.

This paper presents a novel color correction technique for classifying vehicles under different lighting conditions using their colors. To reduce the lighting effects, a reference image is first selected for building the mapping function between the current frame and the reference image. With this mapping function, the color distortions between frames can be reduced to minimum. In addition to lighting changes, the effect of sun light will make the vehicle window become white and lead to the errors of vehicle classification. To reduce this effect, a window-removing task is then applied for making vehicle pixels with the same color more concentrated on the foreground region. Then, vehicles can be more accurately classified to their categories even though strong sun light casts on them. To tackle the confusion problem that some vehicle colors are too similar, e.g., “deep-blue” and “deepgreen”, a novel tree-based classifier is then designed for classifying vehicles to more detailed labels. Experimental results have proved that the proposed method is a robust, accurate, and powerful tool for vehicle classification.

Detecting pedestrians with disability in surveillance videos is practical for the implementation of automated alert/assistance technology. This paper presents a novel approach for the dimensionality reduction which employs sparse representation to improve the generalization capability of a classifier. To characterize pedestrian with disability, we create directional maps by determining the dominant direction of motion in each local spatiotemporal region using 3D orientation filters, and then uses the maps in real-time surveillance settings to detect pre-defined types. Mathematically, the derived algorithm regards the input features as the dictionary in sparse representation, and selects the features that minimize the residual output error iteratively, thus the resulting features have a direct correspondence to the performance requirements of the given problem. Furthermore, the proposed algorithm can be regarded as a sparse classifier, which selects discriminative features and classifies the training data simultaneously. Experimental results obtained using the extensive dataset show the superior performance of our method and thus demonstrate its robustness with the novel sparse representation-based disabled pedestrian detector.

Given the rapid expansion of car ownership worldwide, vehicle safety is an increasingly critical issue in the automobile industry. The reduced cost of cameras and optical devices has made it economically feasible to deploy front-mounted intelligent systems for visual-based event detection for forward collision avoidance and mitigation. While driving at night, vehicles in front are generally visible by their taillights and brake lights. The brake lights are particularly important because they signal deceleration and potential collision. Therefore, in this paper, we propose a novel visual-based approach, based on the Nakagami-mdistribution, for detecting brake lights at night by analyzing the taillights. Rather than using the knowledge of the heuristic features, such as the symmetry, position, and size of the rear-facing vehicle, we focus on finding the invariant features to model brake light scattering by Nakagami imaging and therefore conduct the detection process in a part-based manner. Experiments on an extensive data set show that our proposed system can effectively detect vehicle braking under different lighting and traffic conditions and thus demonstrates its feasibility in real-world environments.

In this paper, in order to detect the focus of interest with motion-tolerance we propose a novel approach for modeling visual dynamics based on spatiotemporal slice (STS), which can provide rich visual patterns along a large temporal scale. Patch-based visual patterns are first computed to generate a codebook in the automatic specified spatiotemporal extent that is determined by contextual information in STS. The codebook is then used to compute its associated response in video frames and eventually an importance map for each frame covering the focus of interest can be obtained. To preserve the visual continuity an optimal video deformation constrained by the trajectories in STS is determined based on a multi-cue approach. Experimental results show that the proposed approach of video retargeting based on STS can effectively generate retargeted videos while maintain their isotropic manipulation and the continuous dynamics of visual perception.

Gender recognition is a challenging task in surveillance videos due to their relatively low-solution, uncontrolled environment and viewing angles of human subject. In this work, a surveillance system of real-time gender recognition is developed. The contribution of this work is four-fold. In order to make the system robust, a mechanism of decision making based on the combination of neighboring face detection, context-regions enhancement and confidence-based weighting assignment is designed. Considering the spatiotemporal consistency of the gender between consecutive faces in successive frames, the belief propagation is employed to model the temporal coherence. Experiment results obtained by using extensive dataset show that our system is effective and efficient in recognizing genders in real-time surveillance videos.

This paper proposes a non-orthogonal view iris recognition system comprising a new iris imaging module, an iris segmentation module, an iris feature extraction module and a classification module. A dual-charge-coupled device camera was developed to capture four-spectral (red, green, blue, and near-infrared) iris images which contain useful information for simplifying the iris segmentation task. An intelligent random sample consensus iris segmentation method is proposed to robustly detect iris boundaries in a four-spectral iris image. In order to match iris images acquired at different off-axis angles, we propose a circle rectification method to reduce the off-axis iris distortion. The rectification parameters are estimated using the detected elliptical pupillary boundary. Furthermore, we propose a novel iris descriptor which characterizes an iris pattern with multiscale step/ridge edge-type maps. The edge-type maps are extracted with the derivative of Gaussian and the Laplacian of Gaussian filters. The iris pattern classification is accomplished by edge-type matching which can be understood intuitively with the concept of classifier ensembles. Experimental results show that the equal error rate of our approach is only 0.04% when recognizing iris images acquired at different off-axis angles within ±30°.

This paper presents a novel people counting system for an environment in which a stationary camera can count the number of people watching a TV-wall advertisement or an electronic billboard without counting the repetitions in video streams in real time. The people actually watching an advertisement are identified via frontal face detection techniques. To count the number of people precisely, a complementary set of features is extracted from the torso of a human subject, as that part of the body contains relatively richer information than the face. In addition, for conducting robust people recognition, an online classifier trained by Fisher's Linear Discriminant (FLD) strategy is developed. Our experiment results demonstrate the efficacy of the proposed system for the people counting task.

Producing an appropriate extent of visually salient regions in video sequences is a challenging task. In this work, we propose a novel approach for modeling dynamic visual attention based on spatiotemporal analysis. Our model first detects salient points in three-dimensional video volumes, and then uses them as seeds to search the extent of salient regions in a motion attention map. To determine the extent of attended regions, the maximum entropy in the spatial domain is used to analyze the dynamics obtained from spatiotemporal analysis. The experiment results show that the proposed dynamic visual attention model can effectively detect visual saliency through successive video volumes.

This paper presents a video surveillance system that is capable of detecting and classifying moving targets in real-time. The system extracts moving targets from a video stream and classifies them into predefined categories according to their spatiotemporal properties. Classification of the moving targets is completed via a combination of a temporal boosted classifier and spatiotemporal “motion energy” analysis. We illustrate that a temporal boosted classifier can be designed that successfully recognizes five object categories: person(s), bicycle, motorcycle, vehicle, and person with umbrella. The proposed temporal boosted classifier has the unique ability to improve weak classifiers by allowing them to make use of previous information when evaluating the current frame. In addition, we demonstrate a method to further process targets in the “person(s)” category to determine if they are single moving individuals or crowds. It is shown that this challenging task of moving crowd recognition can be effectively performed using spatiotemporal motion energies.

This paper presents a video surveillance system in the environment of a stationary camera that can extract moving targets from a video stream in real time and classify them into predefined categories according to their spatiotemporal properties. Targets are detected by computing the pixel-wise difference between consecutive frames, and then classified with a temporally boosted classifier and ldquospatiotemporal-oriented energyrdquo analysis. We demonstrate that the proposed classifier can successfully recognize five types of objects: a person, a bicycle, a motorcycle, a vehicle, and a person with an umbrella. In addition, we process targets that do not match any of the AdaBoost-based classifier's categories by using a secondary classification module that categorizes such targets as crowds of individuals or non-crowds. We show that the above classification task can be performed effectively by analyzing a target's spatiotemporal-oriented energies, which provide a rich description of the target's spatial and dynamic features. Our experiment results demonstrate that the proposed system is extremely effective in recognizing all predefined object classes.

A framework for human action modeling and recognition in continuous action sequences is proposed. A star figure enclosed by a bounding convex polygon is used to effectively represent the extremities of the silhouette of a human body. Thus, human actions are recorded as a sequence of the star figure's parameters, which is then used for action modeling. To model human actions in a compact manner while characterizing their spatio-temporal patterns, star figure parameters are represented by a 2-D feature map, which is used and regarded as a spatio-temporal template. Experiments to evaluate the performance of the proposed framework show that it can recognize human actions in an efficient and effective manner.

In this paper, we propose the use of motion vectors embedded in MPEG bitstreams to generate so-called ldquomotion flowsrdquo, which are applied to perform video retrieval. By using the motion vectors directly, we do not need to consider the shape of a moving object and its corresponding trajectory. Instead, we simply ldquolinkrdquo the local motion vectors across consecutive video frames to form motion flows, which are then recorded and stored in a video database. In the video retrieval phase, we propose a new matching strategy to execute the video retrieval task. Motions that do not belong to the mainstream motion flows are filtered out by our proposed algorithm. The retrieval process can be triggered by query-by-sketch or query-by-example. The experiment results show that our method is indeed superb in the video retrieval process.

In recent years, real-time direct detection of events by surveillance systems has attracted a great deal of attention. In this paper, we propose a new video-based surveillance system that can perform real-time event detection. In the background modeling phase, we adopt a mixture of Gaussian approach to determine the background. Meanwhile, we use color blob-based tracking to track foreground objects. Due to the self-occlusion problem, the tracking module is designed as a multi-blob tracking process to obtain similar multiple trajectories. We devise an algorithm to merge these trajectories into a representative one. After applying the Douglas-Peucker algorithm to approximate a trajectory, we can compare two arbitrary trajectories. The above mechanism enables us to conduct real-time event detection if a number of wanted trajectories are pre-stored in a video surveillance system.

In this paper, a framework of automatic human action segmentation and recognition in continuous action sequences is proposed. A star-like figure is proposed to effectively represent the extremities in the silhouette of human body. The human action, thus, is recorded as a sequence of the star-like figure parameters, which is used for action modeling. To model human actions in a compact manner while characterizing their spatio-temporal distributions, star-like figure parameters are represented by Gaussian mixture models (GMM). In addition, to address the intrinsic nature of temporal variations in a continuous action sequence, we transform the time sequence of star-like figure parameters into frequency domain by discrete cosine transform (DCT) and use only the first few coefficients to represent different temporal patterns with significant discriminating power. The performance shows that the proposed framework can recognize continuous human actions in an efficient way

A novel human posture analysis framework that can perform automatic key posture selection and template matching for human behavior analysis is proposed. The entropy measurement, which is commonly adopted as an important feature to describe the degree of disorder in thermodynamics, is used as an underlying feature for identifying key postures. First, we use cumulative entropy change as an indicator to select an appropriate set of key postures from a human behavior video sequence and then conduct a cross entropy check to remove redundant key postures. With the key postures detected and stored as human posture templates, the degree of similarity between a query posture and a database template is evaluated using a modified Hausdorff distance measure. The experiment results show that the proposed system is highly efficient and powerful