Abstract Lightweight deep learning models for face recognition are becoming increasingly crucial for deployment on resource‐constrained devices such as embedded systems or mobile devices. This paper presents a highly efficient and compact deep learning (DL) model that achieves state‐of‐the‐art performance on various face recognition benchmarks. The developed DL model employs one‐ or few‐shot learning to obtain effective feature embeddings and draws inspiration from FaceNet with significant refinements to achieve a memory size of only 3.5 MB—about 30 times smaller than FaceNet—while maintaining high accuracy and real‐time performance. The study demonstrates the model's effectiveness through extensive experiments, which include testing on public datasets and the model's ability to recognize occluded faces in uncontrolled environments using grayscale input images. Compared to the state‐of‐the‐art lightweight models, the proposed model requires fewer FLOPs (0.06G), has a smaller number of parameters (1.2 M), and occupies a smaller model size (3.5 MB) while achieving a competitive level of recognition accuracy and real‐time performance. The results show that the model is well‐suited for deployment in embedded domains, including live entrance security checks, driver authorization, and in‐class attendance systems. The entire code of FN8 is available on GitHub .

Underwater image processing has been shown to exhibit significant potential for exploring underwater environments. It has been applied to a wide variety of fields, such as underwater terrain scanning and autonomous underwater vehicles (AUVs)-driven applications, such as image-based underwater object detection. However, underwater images often suffer from degeneration due to attenuation, color distortion, and noise from artificial lighting sources as well as the effects of possibly low-end optical imaging devices. Thus, object detection performance would be degraded accordingly. To tackle this problem, in this article, a lightweight deep underwater object detection network is proposed. The key is to present a deep model for jointly learning color conversion and object detection for underwater images. The image color conversion module aims at transforming color images to the corresponding grayscale images to solve the problem of underwater color absorption to enhance the object detection performance with lower computational complexity. The presented experimental results with our implementation on the Raspberry pi platform have justified the effectiveness of the proposed lightweight jointly learning model for underwater object detection compared with the state-of-the-art approaches.

Various weather conditions, such as rain, haze, or snow, can degrade visual quality in images/videos, which may significantly degrade the performance of related applications. In this paper, a novel framework based on sequential dual attention deep network is proposed for removing rain streaks (deraining) in a single image, called by SSDRNet (Sequential dual attentionbased Single image DeRaining deep Network). Since the inherent correlation among rain steaks within an image should be stronger than that between the rain streaks and the background (non-rain) pixels, a two-stage learning strategy is implemented to better capture the distribution of rain streaks within a rainy image. The two-stage deep neural network primarily involves three blocks: residual dense blocks (RDBs), sequential dual attention blocks (SDABs), and multi-scale feature aggregation modules (MAMs), which are all delicately and specifically designed for rain removal. The two-stage strategy successfully learns very fine details of the rain steaks of the image and then clearly removes them. Extensive experimental results have shown that the proposed deep framework achieves the best performance on qualitative and quantitative metrics compared with state-of-the-art methods. The corresponding code and the trained model of the proposed SSDRNet have been available online at https://github.com/fityanul/SDAN-for-Rain-Removal.

Images/videos captured from outdoor visual devices are usually degraded by turbid media, such as haze, smoke, fog, rain, and snow. Haze is the most common one in outdoor scenes due to the atmosphere conditions. In this paper, a novel deep learning-based architecture (denoted by MSRL-DehazeNet) for single image haze removal relying on multi-scale residual learning (MSRL) and image decomposition is proposed. Instead of learning an end-to-end mapping between each pair of hazy image and its corresponding haze-free one adopted by most existing learningbased approaches, we reformulate the problem as restoration of the image base component. Based on the decomposition of a hazy image into the base and the detail components, haze removal (or dehazing) can be achieved by both of our multi-scale deep residual learning and our simplified U-Net learning only for mapping between hazy and haze-free base components, while the detail component is further enhanced via the other learned convolutional neural network (CNN). Moreover, benefited by the basic building block of our deep residual CNN architecture and our simplified UNet structure, the feature maps (produced by extracting structural and statistical features), and each previous layer can be fully preserved and fed into the next layer. Therefore, possible color distortion in the recovered image would be avoided. As a result, the final haze-removed (or dehazed) image is obtained by integrating the haze-removed base and the enhanced detail image components. Experimental results have demonstrated good effectiveness of the proposed framework, compared with state-ofthe-art approaches.

With recent advances in embedded processing capability, vision-based real-time fire detection has been enabled in surveillance devices. This paper presents an image-based fire detection framework based on deep learning. The key is to learn a fire detector relying on tiny-YOLO (You Only Look Once) v3 deep model. With the advantage of lightweight architecture of tiny-YOLOv3 and training data augmentation by some parameter adjusting, our fire detection model can achieve better detection accuracy in real-time with lower complexity in the training stage. Experimental results have verified the effectiveness of the proposed framework.

Popularity prediction of social media becomes a more attractive issue in recent years. It consists of multi-type data sources such as image, meta-data, and text information. In order to effectively predict the popularity of a specified post in the social network, fusing multi-feature from heterogeneous data is required. In this paper, a popularity prediction framework for social media based on multi-modal feature mining is presented. First, we discover image semantic features by extracting their image descriptions generated by image captioning. Second, an effective text-based feature engineering is used to construct an effective word-to-vector model. The trained word-to-vector model is used to encode the text information and the semantic image features. Finally, an ensemble regression approach is proposed to aggregate these encoded features and learn the final regressor. Extensive experiments show that the proposed method significantly outperforms other state-of-the-art regression models. We also show that the multi-modal approach could effectively improve the performance in the social media prediction challenge.

To achieve intelligent manufacturing in Industry 4.0 for steel industry, this paper considers a smart steel manufacturing framework, where an automatic identification tracking method for steel products is proposed. Automatically online tracking and identifying steel products on a production line is essential for intelligent manufacturing management since those products would be frequently moved and processed at different locations on the product flow. Current approaches usually rely on embedding or marking a series of characters (or identification codes) on the steel surface. However, steel-making process is usually conducted under a high temperature environment, making it difficult to well embed the identification codes with high quality and further automatically well recognize them online. To tackle the problem, this paper presents a novel vision-based automatic identification tracking framework without needing to embed any identification codes into the steel product. The key is to employ the essential identity of a steel product without any extrinsic information embedded, achieved by extracting visual features from the product. The presented preliminary results have verified the efficiency of the proposed method, which will be further improved in the future.

Movie summarization aims at condensing a full-length movie to a significantly shortened version that still preserves the movie's major semantic content. In this paper, we propose a learning-based movie summarization framework via role-community social network analysis and feature fusion. In our framework, scene-based movie summarization is formulated as a 0-1 knapsack problem, where the scene attention value for each significant scene is calculated as its “value” and the length of this scene is used as its “cost.” To identify the significance of each scene, we propose a learning-based approach to fuse the information derived from visual saliency (based on low-level features and high-level cognitive process for an input movie), high-level semantic analysis (based on the global and local social networks constructed from the movie), and user preferences. Our evaluation results show that in most test cases, the proposed method subjectively outperforms attention-based and role-based summarization methods and our previous role-community-based method in terms of semantic content preservation.

This paper addresses the problem of hallucinating the missing high-resolution (HR) details of a low-resolution (LR) video while maintaining the temporal coherence of the reconstructed HR details using dynamic texture synthesis (DTS). Most existing multiframe-based video superresolution (SR) methods suffer from the problem of limited reconstructed visual quality due to inaccurate subpixel motion estimation between frames in an LR video. To achieve high-quality reconstruction of HR details for an LR video, we propose a texture-synthesis (TS)-based video SR method, in which a novel DTS scheme is proposed to render the reconstructed HR details in a temporally coherent way, which effectively addresses the temporal incoherence problem caused by traditional TS-based image SR methods. To further reduce the complexity of the proposed method, our method only performs the TS-based SR on a set of key frames, while the HR details of the remaining nonkey frames are simply predicted using the bidirectional overlapped block motion compensation. After all frames are upscaled, the proposed DTS-SR is applied to maintain the temporal coherence in the HR video. Experimental results demonstrate that the proposed method achieves significant subjective and objective visual quality improvement over state-of-the-art video SR methods.

A highly compressed image is usually not only of low resolution, but also suffers from compression artifacts (blocking artifact is treated as an example in this paper). Directly performing image super-resolution (SR) to a highly compressed image would also simultaneously magnify the blocking artifacts, resulting in an unpleasing visual experience. In this paper, we propose a novel learning-based framework to achieve joint single-image SR and deblocking for a highly-compressed image. We argue that individually performing deblocking and SR (i.e., deblocking followed by SR, or SR followed by deblocking) on a highly compressed image usually cannot achieve a satisfactory visual quality. In our method, we propose to learn image sparse representations for modeling the relationship between low- and high-resolution image patches in terms of the learned dictionaries for image patches with and without blocking artifacts, respectively . As a result, image SR and deblocking can be simultaneously achieved via sparse representation and morphological component analysis (MCA)-based image decomposition. Experimental results demonstrate the efficacy of the proposed algorithm.

This paper addresses the problem of hallucinating the missing high-resolution (HR) details of a low-resolution (LR) video while maintaining the temporal coherence of the hallucinated HR details by using dynamic texture synthesis (DTS). Most existing multi-frame-based video super-resolution (SR) methods suffer from the problem of limited reconstructed visual quality due to inaccurate sub-pixel motion estimation between frames in a LR video. To achieve high-quality reconstruction of HR details for a LR video, we propose a texture-synthesis-based video super-resolution method, in which a novel DTS scheme is proposed to render the reconstructed HR details in a time coherent way, so as to effectively address the temporal incoherence problem caused by traditional texture synthesis based image SR methods. To further reduce the complexity of the proposed method, our method only performs the DTS-based SR on a selected set of key-frames, while the HR details of the remaining non-key-frames are simply predicted using the bi-directional overlapped block motion compensation. Experimental results demonstrate that the proposed method achieves significant subjective and objective quality improvement over state-of-the-art video SR methods.

Rain removal from a single color image is a challenging problem as no temporal information among successive images can be obtained. In this paper, we propose a single-color-image-based rain removal framework by properly formulating rain removal as an image decomposition problem based on sparse representation. In our framework, an input color image is first decomposed into a low-frequency part and a high-frequency part by using the guided image filter so that the rain streaks would be in the high-frequency part with nonrain textures/edges, and the high-frequency part is then decomposed into a rain component and a nonrain component by performing dictionary learning and sparse coding. To separate rain streaks from the high-frequency part, a hybrid feature set, including histogram of oriented gradients, depth of field, and Eigen color, is employed to further decompose the high-frequency part. With the hybrid feature set applied, most rain streaks can be removed; simultaneously nonrain component can be enhanced. To the best of our knowledge, compared with the state-of-the-art approaches, the proposed method is among the first to focus on the problem of single color image rain removal and achieves promising results with not only the rain component being removed more completely, but also the visual quality of restored images being improved.

Decomposition of an image into multiple semantic components has been an effective research topic for various image processing applications such as image denoising, enhancement, and inpainting. In this paper, we present a novel self-learning based image decomposition framework. Based on the recent success of sparse representation, the proposed framework first learns an over-complete dictionary from the high spatial frequency parts of the input image for reconstruction purposes. We perform unsupervised clustering on the observed dictionary atoms (and their corresponding reconstructed image versions) via affinity propagation, which allows us to identify image-dependent components with similar context information. While applying the proposed method for the applications of image denoising, we are able to automatically determine the undesirable patterns (e.g., rain streaks or Gaussian noise) from the derived image components directly from the input image, so that the task of single-image denoising can be addressed. Different from prior image processing works with sparse representation, our method does not need to collect training image data in advance, nor do we assume image priors such as the relationship between input and output image dictionaries. We conduct experiments on two denoising problems: single-image denoising with Gaussian noise and rain removal. Our empirical results confirm the effectiveness and robustness of our approach, which is shown to outperform state-of-the-art image denoising algorithms.

Cross-view action recognition is a challenging problem, since one typically does not have sufficient training data at the target view of interest. With recent developments of domain adaptation, we propose a novel low-rank based domain adaptation model for mapping labeled data from the original source view to the target view, so that training and testing can be performed at that domain. Our model not only provides an effective way for associating image data across different domains, we further advocate the structural incoherence between transformed data of different categories. As a result, additional data discriminating ability is introduced to our domain adaptation model, and thus improved recognition can be expected. Experimental results on the IXMAS dataset verify the effectiveness of our proposed method, which is shown to outperform state-of-the-art domain adaptation approaches.

The recent deployment of very large-scale camera networks consisting of fixed/moving surveillance cameras and vehicle video recorders, has led to a novel field in object tracking problem. The major goal is to detect and track each vehicle within a large area, which can be applied to video forensics. For example, a suspected vehicle can be automatically identified for mining digital criminal evidences from a large amount of video data. In this paper, we propose an efficient cross-camera vehicle tracking technique via affine invariant object matching. More specifically, we formulate the problem as invariant image feature matching among different viewpoints of cameras. To achieve vehicle matching, we first extract invariant image feature based on ASIFT (affine and scale-invariant feature transform) for each detected vehicle in a camera network. Then, to improve the accuracy of ASIFT feature matching between images from different viewpoints, we propose to efficiently match feature points based on our observed spatially invariant property of ASIFT, as well as the min-hash technique. As a result, cross-camera vehicle tracking can be efficiently and accurately achieved. Experimental results demonstrate the efficacy of the proposed algorithm and the feasibility to video forensics applications.

In this paper, we propose a novel approach for visualizing and recognizing different emotion categories using facial expression images. Extended by the unsupervised nonlinear dimension reduction technique of locally linear embedding (LLE), we propose a supervised LLE (sLLE) algorithm utilizing emotion labels of face expression images. While existing works typically aim at training on such labeled data for emotion recognition, our approach allows one to derive subspaces for visualizing facial expression images within and across different emotion categories, and thus emotion recognition can be properly performed. In our work, we relate the resulting two-dimensional subspace to the valence-arousal emotion space, in which our method is observed to automatically identify and discriminate emotions in different degrees. Experimental results on two facial emotion datasets verify the effectiveness of our algorithm. With reduced numbers of feature dimensions (2D or beyond), our approach is shown to achieve promising emotion recognition performance.

Low-quality images are usually not only with low-resolution, but also suffer from compression artifacts (blocking artifact is treated as an example in this paper). Directly performing image super-resolution (SR) to a highly compressed (low-quality) image would also simultaneously magnify the blocking artifacts, resulting in unpleasing visual quality. In this paper, we propose a self-learning-based SR framework to simultaneously achieve single-image SR and compression artifact removal for a highly-compressed image. We argue that individually performing deblocking first, followed by SR to an image, would usually inevitably lose some image details induced by deblocking, which may be useful for SR, resulting in worse SR result. In our method, we propose to self-learn image sparse representation for modeling the relationship between low and high-resolution image patches in terms of the learned dictionaries, respectively, for image patches with and without blocking artifacts. As a result, image SR and deblocking can be simultaneously achieved via sparse representation and MCA (morphological component analysis)-based image decomposition. Experimental results demonstrate the efficacy of the proposed algorithm.

Video summarization techniques aim at condensing a full-length video to a significantly shortened version that still preserves the major semantic content of the original video. Movie summarization, being a special class of video summarization, is particularly challenging since a large variety of movie scenarios and film styles complicate the problem. In this paper, we propose a two-stage scene-based movie summarization method based on mining the relationship between role-communities since the role-communities in earlier scenes are usually used to develop the role relationship in later scenes. In the analysis stage, we construct a social network to characterize the interactions between role-communities. As a result, the social power of each role-community is evaluated by the community's centrality value and the role communities are clustered into relevant groups based on the centrality values. In the summarization stage, a set of feasible summary combinations of scenes is identified and an information-rich summary is selected from these candidates based on social power preservation. Our evaluation results show that in at most test cases the proposed method achieves better subjective performance than attention-based and role-based summarization methods in terms of semantic content preservation for a movie summary.

Rain removal from a single image is one of the challenging image denoising problems. In this paper, we present a learning-based framework for single image rain removal, which focuses on the learning of context information from an input image, and thus the rain patterns present in it can be automatically identified and removed. We approach the single image rain removal problem as the integration of image decomposition and self-learning processes. More precisely, our method first performs context-constrained image segmentation on the input image, and we learn dictionaries for the high-frequency components in different context categories via sparse coding for reconstruction purposes. For image regions with rain streaks, dictionaries of distinct context categories will share common atoms which correspond to the rain patterns. By utilizing PCA and SVM classifiers on the learned dictionaries, our framework aims at automatically identifying the common rain patterns present in them, and thus we can remove rain streaks as particular high-frequency components from the input image. Different from prior works on rain removal from images/videos which require image priors or training image data from multiple frames, our proposed self-learning approach only requires the input image itself, which would save much pre-training effort. Experimental results demonstrate the subjective and objective visual quality improvement with our proposed method.

Rain removal from an image/video is a challenging problem and has been recently investigated extensively. In our previous work, we have proposed the first single-image-based rain streak removal framework via properly formulating it as an image decomposition problem based on morphological component analysis (MCA) solved by performing dictionary learning and sparse coding. However, in this previous work, the dictionary learning process cannot be fully automatic, where the two dictionaries used for rain removal were selected heuristically or by human intervention. In this paper, we extend our previous work to propose an automatic self-learning-based rain streak removal framework for single image. We propose to automatically self-learn the two dictionaries used for rain removal without additional information or any assumption. We then extend our single-image-based method to video-based rain removal in a static scene by exploiting the temporal information of successive frames and reusing the dictionaries learned by the former frame(s) in a video while maintaining the temporal consistency of the video. As a result, the rain component can be successfully removed from the image/video while preserving most original details. Experimental results demonstrate the efficacy of the proposed algorithm.

Blocking artifact, characterized by visually noticeable changes in pixel values along block boundaries, is a common problem in block-based image/video compression, especially at low bitrate coding. Various post-processing techniques have been proposed to reduce blocking artifacts, but they usually introduce excessive blurring or ringing effects. This paper proposes a self-learning-based image/ video deblocking framework via properly formulating deblocking as an MCA (morphological component analysis)-based image decomposition problem via sparse representation. The proposed method first decomposes an image/video frame into the low-frequency and high-frequency parts by applying BM3D (block-matching and 3D filtering) algorithm. The high-frequency part is then decomposed into a “blocking component” and a “non-blocking component” by performing dictionary learning and sparse coding based on MCA. As a result, the blocking component can be removed from the image/video frame successfully while preserving most original image/video details. Experimental results demonstrate the efficacy of the proposed algorithm.

We present a novel learning-based method for single image super-resolution (SR). Given a single input low-resolution (LR) image (and its image pyramid), we propose to learn context-specific image sparse representation, which aims at modeling the relationship between low and high-resolution image patch pairs of different context categories in terms of the learned dictionaries. To predict the SR image, we derive the context-specific sparse representation of each image patch in the LR input with additional locality and group sparsity constraints. While the locality constraint searches for the most similar image patches and uses the corresponding highresolution outputs for SR, the group sparsity constraint allows us to utilize the information from most relevant context categories for predicting the final SR output. Experimental results show the proposed method is able to quantitatively and qualitatively achieve state-of-the-art performance.

Rain removal from a video is a challenging problem and has been recently investigated extensively. Nevertheless, the problem of rain removal from a single image was rarely studied in the literature, where no temporal information among successive images can be exploited, making the problem very challenging. In this paper, we propose a single-image-based rain removal framework via properly formulating rain removal as an image decomposition problem based on morphological component analysis. Instead of directly applying a conventional image decomposition technique, the proposed method first decomposes an image into the low- and high-frequency (HF) parts using a bilateral filter. The HF part is then decomposed into a "rain component" and a "nonrain component" by performing dictionary learning and sparse coding. As a result, the rain component can be successfully removed from the image while preserving most original image details. Experimental results demonstrate the efficacy of the proposed algorithm.

Rain removal from a video is a challenging problem and has been recently investigated extensively. Nevertheless, the problem of rain removal from a single image has been rarely studied in the literature, where no temporal information among successive images can be exploited, making it more challenging. In this paper, to the best of our knowledge, we are among the first to propose a single-frame-based rain removal framework via properly formulating rain removal as an image decomposition problem based on morphological component analysis (MCA). Instead of directly applying conventional image decomposition technique, we first decompose an image into the low-frequency and high frequency parts using a bilateral filter. The high-frequency part is then decomposed into "rain component" and "non rain component" via performing dictionary learning and sparse coding. As a result, the rain component can be successfully removed from the image while preserving most original image details. Experimental results demonstrate the efficacy of the proposed algorithm.

Compressive sensing (CS) has recently attracted much attention due to its unique feature of directly and simultaneously acquiring compressed and encrypted data based on their sparse or compressible properties. To securely transmit compressively sensed multimedia data over networks, it is required to support transcoder to securely convert compressed multimedia into several different types for diverse receivers. In this paper, a secure transcoding scheme for compressive multimedia sensing is proposed. We focus on securely converting compressively sensed multimedia data (not data compressed via standard codec) with a certain number of measurements into other different numbers of measurements without resorting to reconstruct the original data. We show that the security can be achieved via transforming multimedia re-sensing process into another secure domain at the transcoder. We also show that the computational security can be achieved while transmitting compressively sensed data between the sender (or each receiver) and the transcoder over networks.

Single-image super-resolution (SR) is to reconstruct a high-resolution image from a low-resolution input image. Nevertheless, most SR algorithms are performed in an iterative manner and are therefore time-consuming. In this paper, we propose an iteration-free single-image SR algorithm based on fast deconvolution with gradient prior. Based on the prior calculated from the initially upsampled image via current approach (e.g., bicubic interpolation or example/learning-based approaches), we make the deconvolution process well-posed, which can be efficiently solved in FFT domain. Moreover, the proposed algorithm can be directly applied to video SR, where the temporal coherence can be automatically maintained. Experimental results demonstrate that the proposed method can simultaneously obtain significant acceleration and quality improvement over several existing SR methods.

Assessment of image similarity is fundamentally important to numerous multimedia applications. The goal of similarity assessment is to automatically assess the similarities among images in a perceptually consistent manner. In this paper, we interpret the image similarity assessment problem as an information fidelity problem. More specifically, we propose a feature-based approach to quantify the information that is present in a reference image and how much of this information can be extracted from a test image to assess the similarity between the two images. Here, we extract the feature points and their descriptors from an image, followed by learning the dictionary/basis for the descriptors in order to interpret the information present in this image. Then, we formulate the problem of the image similarity assessment in terms of sparse representation. To evaluate the applicability of the proposed feature-based sparse representation for image similarity assessment (FSRISA) technique, we apply FSRISA to three popular applications, namely, image copy detection, retrieval, and recognition by properly formulating them to sparse representation problems. Promising results have been obtained through simulations conducted on several public datasets, including the Stirmark benchmark, Corel-1000, COIL-20, COIL-100, and Caltech-101 datasets.

In this paper, we formulate the problems of image copy detection and image recognition in terms of sparse representation. To achieve robustness, security, and efficient storage of image features, we propose to extract compact local feature descriptors via constructing the basis of the SIFT-based feature vectors extracted from the secure SIFT domain of an image. Image copy detection can be efficiently accomplished based on the sparse representations and reconstruction errors of the features extracted from an image possibly manipulated by signal processing or geometric attacks. For image recognition, we show that the features of a query image can be represented as sparse linear combinations of the features extracted from the training images belonging to the same cluster. Hence, image recognition can also be cast as a sparse representation problem. Then, we formulate our sparse representation problem as an l1-minimization problem. Promising results regarding image copy detection and recognition have been verified, respectively, through the simulations conducted on several content-preserving attacks defined in the Stirmark benchmark and Caltech-101 dataset.

We address an important issue of fully low-cost and low-complex video compression for use in resource-extremely limited sensors/devices. Conventional motion estimation-based video compression or distributed video coding (DVC) techniques all rely on the high-cost mechanism, namely, sensing/sampling and compression are disjointedly performed, resulting in unnecessary consumption of resources. That is, most acquired raw video data will be discarded in the (possibly) complex compression stage. In this paper, we propose a dictionary learning-based distributed compressive video sensing (DCVS) framework to “directly” acquire compressed video data. Embedded in the compressive sensing (CS)-based single-pixel camera architecture, DCVS can compressively sense each video frame in a distributed manner. At DCVS decoder, video reconstruction can be formulated as an l1-minimization problem via solving the sparse coefficients with respect to some basis functions. We investigate adaptive dictionary/basis learning for each frame based on the training samples extracted from previous reconstructed neighboring frames and argue that much better basis can be obtained to represent the frame, compared to fixed basis-based representation and recent popular “CS-based DVC” approaches without relying on dictionary learning.

We address an important issue of fully low-cost and low-complexity video encoding for use in resource limited sensors/devices. Conventional distributed video coding (DVC) does not actually meet this requirement because the acquisition of video sequences still relies on the high-cost mechanism (sampling + compression). Recently, we have proposed a distributed compressive video sensing (DCVS) framework to directly capture compressed video data called measurements, while exploiting correlations among successive frames for video reconstruction at the decoder. The core is to integrate the respective characteristics of DVC and compressive sensing (CS) to achieve CS-based single-pixel camera-compatible video encoder. At DCVS decoder, video reconstruction can be formulated as a convex unconstrained optimization problem via solving the sparse coefficients with respect to some basis functions. Nevertheless, the issue of measurement rate allocation has not been considered yet in the literature. Actually, different measurement rates should be adaptively assigned to different local regions by considering the sparsity of each region for improving reconstructed quality. This paper investigates dynamic measurement rate allocation in block-based DCVS, which can adaptively adjust measurement rates by estimating the sparsity of each block via feedback information. Simulation results have indicated the effectiveness of our scheme. It is worth noting that our goal is to develop a novel fully low-complexity video compression paradigm via the emerging compressive sensing and sparse representation technologies, and provide an alternative scheme adaptive to the environment, where raw video data is not available, instead of competing compression performances against the current compression standards (e.g., H.264/AVC) or DVC schemes which need raw data available for encoding.

Low-complexity video encoding has been applicable to several emerging applications. Recently, distributed video coding (DVC) has been proposed to reduce encoding complexity to the order of that for still image encoding. In addition, compressive sensing (CS) has been applicable to directly capture compressed image data efficiently. In this paper, by integrating the respective characteristics of DVC and CS, a distributed compressive video sensing (DCVS) framework is proposed to simultaneously capture and compress video data, where almost all computation burdens can be shifted to the decoder, resulting in a very low-complexity encoder. At the decoder, compressed video can be efficiently reconstructed using the modified GPSR (gradient projection for sparse reconstruction) algorithm. With the assistance of the proposed initialization and stopping criteria for GRSR, derived from statistical dependencies among successive video frames, our modified GPSR algorithm can terminate faster and reconstruct better video quality. The performance of our DCVS method is demonstrated via simulations to outperform three known CS reconstruction algorithms.

In this paper, a new image hashing scheme satisfying robustness and security is proposed. We exploit the property of dimensionality reduction inherent in compressive sensing/sampling (CS) for image hash design. The gained benefits include (1) the hash size can be kept small and (2) the CS-based hash is computationally secure. We study the use of visual information fidelity (VIF) for hash comparison under Stirmark attacks. We further derive the relationships between the hash of an image and both of its MSE distortion and visual quality measured by VIF, respectively. Hence, based on hash comparisons, both the distortion and visual quality of a query image can be approximately estimated without accessing its original version. We also derive the minimum distortion for manipulating an image to be unauthentic to measure the security of our scheme.

Wireless visual sensor networks are potentially applicable for several emerging applications. Since the resource-limited restriction for a visual sensor node (VSN), efficient resource allocation for video compression is challenging. In this paper, a power-rate-distortion (PRD) model-based low-complexity multiview video codec is proposed. Our PRD model is used to characterize the relationship between the available resources and the RD performance of our video codec. More specifically, an RD function in terms of the percentages for different coding modes of blocks and the target bit rate under the available resource constraints is derived for optimal coding mode decision. Analytic and simulation results are provided to verify the resource scalability and accuracy of our PRD model in facilitating our low-complexity multiview video codec to achieve good performance.

To meet the requirements of resource-limited video sensors, low-complexity video encoding technique is highly desired. In this paper, a low-complexity multi-view distributed video encoding scheme by using the correlations among video frames from adjacent video sensor nodes (VSNs) via robust media hashing at encoder and the global motion parameters estimated and fed back from the decoder is proposed. The frames from adjacent VSNs are warped into the same view-direction based on the global motion parameters. Then, the significant differences between the warped key frame and the non-key frame from adjacent VSNs are efficiently extracted based on robust media hashing for non-key frame compression. The key is that few data (hash information) exchanges among adjacent VSNs are allowed to efficiently exploit the correlations among VSNs. The coding performance and energy consumption of the proposed encoder have been verified through simulations and comparisons with existing low-complexity video encoders.

In distributed video coding, individual frames are encoded independently but decoded conditionally. The Wyner-Ziv theorem-based source coding with side information only available at the decoder states that an intraframe encoder with interframe decoder system can approach the efficiency of a conventional interframe encoder and decoder system. In this paper, a new block discrete cosine transform (DCT)-based Wyner-Ziv video codec with coding mode-aided motion compensation at the decoder is proposed. The key is that for each block, a large amount of candidate blocks are evaluated based on some criteria derived from Reed-Solomon (RS) decoding and best neighborhood matching to find the best candidate block as the side information. Another characteristic is that error correction code (ECC) decoding is proposed to participate in generating side information. Compared with some known Wyner-Ziv video coding systems, in the proposed video codec, no extra information should be transmitted and feedback channel is unnecessary. The coding performance of our method has been verified through simulations.

To meet the requirement of distributed video coding in resource-limited sensor networks, Wyner-Ziv theorem-based source coding with side information available at the decoder states that an intraframe encoder with interframe decoder system can achieve comparable coding efficiency of a conventional video codec. Most existing Wyner-Ziv video coding systems are with light encoder and heavy decoder. In this paper, a new content-aware media hash-based Wyner-Ziv video codec with light encoder and light decoder is proposed. The key is that the significant differences between a video frame and its reference frame are efficiently extracted and used for frame recovery based on robust image hashing without needing to perform motion estimation. The particular contribution of our method is its low complexity in both the encoder and decoder sides. Simulation results demonstrate the achievable coding efficiency of our method in particular for videos with small and middle motions

For entropy-coded H.264 video frames, a transmission error in a codeword will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received video frames. In this study, an error resilient coding scheme for H.264 video transmission is proposed. At the encoder, for an H.264 I frame, the important data for each macroblock (MB) are extracted and embedded into the next frame by the proposed MB-interleaving slice-based data embedding scheme for I frames. For an H.264 P frame, two types of important data for each MB are extracted and embedded into the next frame by the proposed MB-interleaving slice-based data embedding scheme for P frames. At the decoder, if the important data for a corrupted MB can be correctly extracted, the extracted important data for the corrupted MB will facilitate the employed error concealment scheme to conceal the corrupted MB; otherwise, the employed error concealment scheme is used to conceal the corrupted MB. Based on the simulation results obtained in this study, the proposed scheme can recover high-quality H.264 video frames from the corresponding corrupted video frames up to a video packet loss rate of 20%.

For entropy-coded MPEG-2 video frames, a transmission error will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received video frames. In this study, a hybrid error concealment (EC) scheme for MPEG-2 video transmission is proposed. Based on the fitness function employed for evaluating the candidate concealed blocks of a corrupted block, a corrupted block in an intra-coded I frame is concealed by either the spatial EC algorithm in H.264 or the proposed fast best neighborhood matching (BNM) algorithm. A corrupted block in an inter-coded P or B frame is concealed by the proposed fast motion-compensated BNM algorithm. Based on the simulation results obtained in this study, the proposed scheme can recover high-quality MPEG-2 video frames from the corresponding corrupted video frames up to a video packet loss rate (VPLR) of 20%.

For an entropy-coded wavelet-based image, such as a JPEG-2000 image, a transmission error in a codeword will not only affects the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received image. In this study, two error resilient coding schemes for wavelet-based image transmission based on data embedding and genetic algorithms (GAs) are proposed. In this study, as JPEG-2000, an image is decomposed into 6 wavelet levels (levels 0-5). For level 0, an error resilient coding scheme based on data embedding and side-match vector quantization is employed. For level 1, the proposed basic scheme uses the embedded important data for a corrupted code block to conceal the corrupted code block, whereas the proposed enhanced scheme uses a GA-based error concealment scheme to conceal the corrupted code block. For level 2, the embedded important data for a corrupted code block are used to conceal the corrupted code block. For levels 3-5, the wavelet coefficients of each corrupted code block are simply replaced by zeros. Based on the simulation results obtained in this study, the performances of the two proposed schemes are better than that of the three existing approaches for comparison. The proposed schemes can recover high-quality wavelet-based JPEG-2000 images from the corresponding corrupted images up to a bit error rate of 0.5%.