Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scenelanguage models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes.

With the rapid proliferation of AI-generated content (AIGC) on multimedia platforms, efficient and reliable video forgery detection has become increasingly important. Existing approaches often rely on either visual artifacts or semantic inconsistencies, but suffer from high computational costs, limiting their deployment at scale. In this work, we propose SVD-Det, a lightweight and efficient pipeline that leverages both Semantic and Visual Defect cues to detect forged videos. SVD-Det fuses spatiotemporal representations from raw RGB frames and compression-induced distortions using a 3D-Swin Transformer, and augments semantic understanding via CLIP-based embeddings. To integrate these heterogeneous modalities, we introduce Domain-Query Attention (DoQA), a novel attention mechanism that hierarchically aggregates spatial and temporal features. Experiments across seven video generation domains demonstrate that SVD-Det not only achieves state-of-the-art detection performance but also reduces model size and inference time by over 97% and 98%, respectively, compared to LMM-based baselines. Our results highlight the practicality and robustness of SVD-Det for scalable AIGC detection in real-world scenarios.

Colored point cloud becomes a fundamental representation in the realm of 3D vision. Effective Point Cloud Compression (PCC) is urgently needed due to the huge amount of data. In this paper, we propose an end-to-end Deep Joint Geometry and Attribute Compression (Deep-JGAC) method for dense colored point clouds. First, we propose a flexible Deep-JGAC framework, where the geometry and attribute encoders are compatible with either learning or non-learning encoders. Second, we propose an end-to-end deep residual self-attention-based geometry encoder to improve geometry coding efficiency, where a Hybrid Residual Self-attention Module (HRSM) is proposed to enhance geometry representation by considering its geometrical importance. Third, to solve the mismatch between the point cloud geometry and attribute caused by the geometry compression distortion, we propose an optimized re-colorization module to attach attribute to the geometrically distorted point cloud for attribute coding, which lowers the computational complexity. Extensive experimental results demonstrate that, in terms of the geometry quality metric D1-PSNR, the proposed Deep-JGAC achieves average Bjøntegaard Delta Bit Rate (BDBR) of -82.96%, -44.63%, -36.46%, -41.72%, and -31.16% compared to the G-PCC (Octree), G-PCC (Trisoup), V-PCC, GRASP, and PCGCv2, respectively. For the perceptual joint quality metric MS-GraphSIM, Deep-JGAC achieves an average BDBR of - 48.72%, -57.14%, -14.67% and -13.37% against G-PCC(Octree), IT-DL-PCC, V-PCC, and DeepPCC, respectively. In addition, the costs of encoding/decoding time are reduced by 32.8%/30.8%, 80.1%/81.8%, 97.2%/35.7%, 98.4%/92.3%, and 96.4%/99.6% on average compared to G-PCC (Octree), G-PCC (Trisoup), V-PCC, IT-DL-PCC and DeepPCC. The code and pre-trained models are available at https://github.com/SYSU-Video/Deep-JGAC.

Video Camouflaged Object Detection (VCOD) aims to segment objects visually indistinguishable from their background in unconstrained video sequences. This task is challenging due to the extremely low appearance contrast and motion ambiguity caused by camera shifts, occlusion, and background clutter. Although motion cues provide crucial signals to break camouflage, existing methods typically adopt either implicit motion modeling based on initial semantic representations or explicit modeling based on low-level pixel motion, both of which suffer from information loss or noise sensitivity. To this end, we propose IDM-VCOD, an integrated dual-motion VCOD framework that unifies two complementary motion modeling strategies through a dual-motion integrated design. The implicit path aggregates temporal context via spatio-temporal prediction cubes for temporal neighborhood refinement, while the explicit path explicitly aligns multi-frame backgrounds to highlight foreground motion. A selective activation mechanism adaptively triggers the explicit branch only when implicit predictions are unreliable, enhancing accuracy and efficiency. Extensive experiments on the MoCA-Mask and CAD datasets demonstrate that IDM-VCOD achieves superior detection accuracy and generalization compared to state-of-the-art VCOD methods, while significantly reducing model size and inference cost. Comprehensive ablation studies further validate the effectiveness of the dual-motion design and its robustness in challenging camouflage scenarios.

This work proposes novel beam tracking methods to significantly reduce training overhead and combat harsh channel environments that lead to tracking failures, including blockages, multipaths, user movements, and rotations. The proposed beam tracking system is designed using a new low-carbon-footprint machine learning technique called Green Learning (GL). We introduce a GL-based prediction model (GLPM) along with an anti-blockage method (ABM) and an anti-rotation method (ARM) to enhance robustness. In a 5G NR beam management scenario utilizing a near-realistic ray-tracing channel, the proposed integrated beam tracking scheme significantly reduces the number of Synchronization Signal Blocks (SSBs) by 46–57 and outperforms traditional beam sweeping methods, achieving 1.3 to 4 times higher data rates compared to sweeping with 64 SSBs. Additionally, the proposed approach delivers superior accuracy while maintaining lower computational complexity than deep learning-based methods. Moreover, when examining instantaneous tracking ability, the proposed beam tracking scheme can closely follow ideal cases and rescue frequent tracking failures in conventional beam sweeping caused by harsh channel environments. This rescue is more pronounced in low SNR regions, where conventional beam sweeping can frequently lose connection entirely, while the proposed scheme maintains satisfactory link quality thanks to its accurate beam tracking that provides sufficient received power. From the experiments, the GLPM accounts for 70% to 93% of the overall operations, resulting in low latency and computational complexity. The ABM and ARM do not need to operate frequently but effectively address the shortcomings of GLPM in specific environments, significantly enhancing overall performance. The superior tracking ability and reduction in training redundancy demonstrate that the proposed integrated beam tracking technique is a promising solution for future 5G/6G beam management.

In this paper, a green learning (GL)-based precoding framework is proposed for simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-aided millimeter-wave (mmWave) MIMO broadcasting systems. Motivated by the growing emphasis on environmental sustainability in future 6G networks, this work adopts a broadcasting transmission architecture for scenarios where multiple users share identical information, improving spectral efficiency and reducing redundant transmissions and power consumption. Different from conventional optimization methods, such as block coordinate descent (BCD) that require perfect channel state information (CSI) and iterative computation, the proposed GL framework operates directly on received uplink pilot signals without explicit CSI estimation. Unlike deep learning (DL) approaches that require CSI-based labels for training, the proposed GL approach also avoids deep neural networks and backpropagation, leading to a more lightweight design. Although the proposed GL framework is trained with supervision generated by BCD under full CSI, inference is performed in a fully CSI-free manner. The proposed GL integrates subspace approximation with adjusted bias (Saab), relevant feature test (RFT)-based supervised feature selection, and eXtreme gradient boosting (XGBoost)-based decision learning to jointly predict the STAR-RIS coefficients and transmit precoder. Simulation results show that the proposed GL approach achieves competitive spectral efficiency compared to BCD and DL-based models, while reducing floating-point operations (FLOPs) by over four orders of magnitude. These advantages make the proposed GL approach highly suitable for real-time deployment in energy-and hardware-constrained broadcasting scenarios.

Magnetic Resonance Imaging (MRI) is one of the primary steps for screening for prostate cancer (PCa), one of the most frequent types of cancer in men. Although it provides significant clinical information, MRI is acquired at low-resolution images, thus making its readout more challenging for radiologists or AI-based tools for PCa detection. Generative Artificial Intelligence (GenAI) and Super Resolution (SR) have achieved impressive results in improving image quality. This work proposes SR-MR-Hop, an SR method to enhance the MRI input, meant to improve the PCa detection performance of AI-based detectors. SR-MR-Hop method provides a novel, lightweight and transparent feature extraction methodology for super-resolving the input MRI sequences. Moreover, a novel tumor-aware loss function is introduced to boost learning from the tumor regions. Also, a fusion model is proposed to enhance the MRI sequence, based on the cross-modal features from the different sequences of MRI, leveraging their complementary properties. The experiments on the publicly available PI-CAI dataset, show that SR-MR-Hop can provide superior MRI enhancement results. Notably, by combining the SR-MR-Hop as a preprocessing method with AI-based PCa detection models, the Area Under The Curve (AUC) increases from 0.811 to 0.834.

This work proposes a green learning (GL) approach to restore medical images. Without loss of generality, we use low-dose computed tomography (LDCT) images as examples. LDCT images are susceptible to noise and artifacts, where the imaging process introduces distortion. LDCT image restoration is an important preprocessing step for further medical analysis. Deep learning (DL) methods have been developed to solve this problem. We examine an alternative solution using the Green Learning (GL) methodology. The new restoration method is characterized by mathematical transparency, computational and memory efficiency, and high performance. Experiments show that our GL method offers state-of-the-art restoration performance at a smaller model size and with lower inference complexity.

An energy-efficient high-performance static IR-drop estimation method based on green learning called Green IR Drop (GIRD) is proposed in this work. GIRD processes the IC design input in three steps. First, the input netlist data are converted to multichannel maps. Their joint spatial–spectral representations are determined with PixelHop. Next, discriminant features are selected using the relevant feature test (RFT). Finally, the selected features are fed to the eXtreme Gradient Boosting trees regressor. Both PixelHop and RFT are green learning tools. GIRD yields a low carbon footprint due to its smaller model sizes and lower computational complexity. Besides, its performance scales well with small training datasets. Experiments on synthetic and real circuits are given to demonstrate the superior performance of GIRD. The model size and the complexity, measured by the floating point operations (FLOPs) of GIRD, are only$10^{-3}$and$10^{-2}$of deep-learning methods, respectively.

Owing to the proliferation of user-generated videos on the Internet, blind video quality assessment (BVQA) at the edge attracts growing attention. The usage of deep-learning-based methods is restricted to be applied at the edge due to their large model sizes and high computational complexity. In light of this, a novel lightweight BVQA method called GreenBVQA is proposed in this work. GreenBVQA features a small model size, low computational complexity, and high performance. Its processing pipeline includes: video data cropping, unsupervised representation generation, supervised feature selection, and mean-opinion-score (MOS) regression and ensembles. We conduct experimental evaluations on three BVQA datasets and show that GreenBVQA can offer state-of-the-art performance in the Pearson Linear Correlation Coefficient (PLCC) and the Spearman Rank Order Correlation Coefficient (SROCC) metrics while demanding significantly smaller model sizes and lower computational complexity. Thus, GreenBVQA is well-suited for edge devices.

Demosaicking is a critical process in the digital imaging pipeline, tasked with reconstructing full-color images from sampled data captured by R/G/B color sensors. The challenge arises from two-thirds of the pixel data being missing, which complicates the task of accurate reconstruction. Recent deep learning-based solutions have yielded considerable advancements in demosaicking performance. However, they are computationally intensive and rely on large model architectures, rendering them unsuitable for edge-devices deployment. This work introduces a novel demosaicking method to address these challenges based on green learning (GL), named green image demosaicking (GID). GID offers model transparency while significantly reducing the model size and computational complexity against deep learning methods. Notably, GID does not utilize neural networks. Instead, it is built upon unsupervised representation learning and supervised feature dimension reduction. GID effectively addresses the challenges of big data in vision applications and enhances predictive accuracy during decision-making. GID is engineered for rapid execution with parallel training, making it well-suited for real-time vision tasks on resource-constrained devices.

The discriminant ability of features plays a central role in classification tasks. Deep learning (DL) utilizes back-propagation to yield discriminant features with an end-to-end depth structure, which lacks interpretability. A new learning paradigm, Green Learning (GL), has been proposed to address the weaknesses of DL, including high carbon footprints, large model sizes, and mathematical opaqueness. Following this idea, a statistics-based feature generation (SFG) method is proposed to boost system performance further. SFG consists of two modules: 1) feature subset selection and 2) discriminant feature generation via linear combination. In the first module, SFG traces the splitting features along individual tree paths within auxiliary XGBoost classifiers, which form the desired feature subsets for each binary label subspace. In the second module, SFG determines the weights of features in each subspace by formulating a least-square normal equation supervised by its corresponding binary sub-labels. SFG enables the adaptive assessment of feature importance at global and various local scales. Besides, it reduces the computational requirement by working in low-dimensional feature sub-spaces. Extensive experiments are conducted on classical image classification tasks to demonstrate the effectiveness and efficiency of SFG.

A green-learning-based spoofed speech detector that decides whether an input speech sample is bona fide (genuine) or spoofed in an automatic speaker verification (ASV) system, is proposed in this work. The proposed solution, called the green ASVspoof detector (GAD), adopts Wav2vec (version 2.0) speech representations as its front-end model. We partition an input speech sample into temporal segments and adopt the Wav2vec representation for each segment. Then, GAD is a 3-stage decision process comprising one XGBoost classifier in each stage. It offers an interpretable design. It is shown by experimental results that GAD achieves competitive performance in ASVspoof detection. At the same time, it has a smaller model size and significantly lower computational complexity, thus positioning it as an effective and green solution.

Knowledge graph entity typing (KGET) aims to predict the missing entity types in knowledge graphs (KG). The relationship between entities and their corresponding types is often expressed using a single relation, hasType. However, hasType has a limited capability for modeling diverse entity-type relationships in the embedding space. In this paper, we first introduce multiple auxiliary relations to model the complex entity-type relationship. We propose an efficient and robust algorithm to group similar entity types together and assign a unique auxiliary relation to each group. Then, with the auxiliary relations, we propose an Asynchronous representation learning framework for KGET, named AsyncET, where entity and type embeddings are updated alternatively. Consequently, the quality of entity embeddings is gradually improved during training by infusing type information. In addition, entity types with different granularities and semantics can be properly modeled in the embedding space. Experimental results show that AsyncET can substantially improve the performance of embedding-based methods on the KGET task and has a significant advantage over state-of-the-art neural network-based methods in terms of model sizes and inference time.

This work proposes a novel machine learning (ML) based beam tracking scheme. The proposed scheme is inspired by a new learning method, called Green Learning (GL), which has several nice properties including lightweight, low complexity, and logical transparency. GL has demonstrated outstanding performance in the field of image/video processing, and this work continues showing its great potentials in beam tracking problem. In a setting of 5G NR (New Radio) beam tracking protocol, the proposed GL beam tracking scheme demonstrates several advantages over conventional ML methods including an improved accuracy in tracking angle of arrival, less sensitive to channel signal-to-noise ratio, an extended time duration to lose tracking, and lower computational complexity in both training and inference. These effective and efficient properties make the proposed GL beam tracking suitable for future communications, especially with critical demand of low carbon footprint.

To attain diverse performance benchmarks for 6G systems, multi-input-multi-output (MIMO) systems assisted by reconfigurable intelligent surface (RIS) have been investigated intensively. Various deep learning (DL) based methods have been adopted to perform the non-convex optimization of such systems, which leads to significant enhancement in the spectral efficiency. However, the DL-based methods rely on the operations of neural network, which demands high computational cost and usually requires perfect channel state information (CSI). To tackle the challenge, we develop a green learning (GL) based method to optimize the system with the input of the received pilot signals. Bypassing the channel estimations, the proposed algorithm first extract the representations from the received pilot signals, following by selecting the most relevant representations for the precoding optimization. The model size of the proposed method can be greatly reduced by the analysis of the dominant subspace as well as relevant feature selection. It shows through simulation that the proposed method slightly outperforms the DL-based method with simply $1.3 \sim 1.8 \%$ floating-point operations (FLOPs).

Three-dimensional (3D) positioning technology plays an important role in millimeter wave (mmWave) non-terrestrial or integrated sensing and communication (ISAC) networks in sixth-generation (6G) systems. However, the complexity of joint range and orientation estimation in mmWave hybrid beamforming (HBF) systems forms a technical hurdle to its practical realization. In view of the recent advancement in green learning (GL) technology, a low-complexity GL architecture is developed herein for 3D positioning in mmWave HBF systems. The entire architecture only consists of one layer of unsupervised representation learning, followed by a supervised feature learning stage and a regression layer for parameter estimation. Compared to the typical deep learning method, the complexity of the proposed method is at least 3 order lower, while the performance is comparable to that of the maximum likelihood estimations of individual parameters given perfect knowledge of the other parameters. This presents the potential of GL in ISAC for future$6G$systems.

Unsupervised image-to-image translation learns cross-domain image mapping that transfers input from the source domain to output in the target domain while preserving its semantics. One challenge is that different semantic statistics in source and target domains result in content discrepancy known as semantic distortion. To address this problem, a novel I2I method that maintains semantic consistency in translation is proposed and named SemST in this work. SemST reduces semantic distortion by employing contrastive learning and aligning the structural and textural properties of input and output by maximizing their mutual information. Furthermore, a multi-scale approach is introduced to enhance translation performance, thereby enabling the applicability of SemST to domain adaptation in high-resolution images. Experiments show that SemST effectively mitigates semantic distortion and achieves state-of-the-art performance. Also, the application of SemST to domain adaptation is explored. It is demonstrated by preliminary experiments that SemST can be utilized as a beneficial pre-training for the semantic segmentation task.

Recovering unknown, missing, damaged, distorted, or lost information in DCT coefficients is a common task in multiple applications of digital image processing, including image compression, selective image encryption, and image communication. This paper investigates the recovery of sign bits in DCT coefficients of digital images, by proposing two different approximation methods to solve a mixed integer linear programming (MILP) problem, which is NP-hard in general. One method is a relaxation of the MILP problem to a linear programming (LP) problem, and the other splits the original MILP problem into some smaller MILP problems and an LP problem. We considered how the proposed methods can be applied to JPEG-encoded images and conducted extensive experiments to validate their performances. The experimental results showed that the proposed methods outperformed other existing methods by a substantial margin, both according to objective quality metrics and our subjective evaluation.

Satellite images are highly susceptible to forgery due to various editing techniques. Traditional forgery detection methods, designed for natural images, often fail when applied to satellite images because of differences in sensing technology and processing protocols. The rise of generative models, such as diffusion models, has further complicated the detection of forgeries in satellite images. This study tackles these challenges from both methodological and data perspectives. We introduce a multitask forgery localization and detection collaborative framework (FLDCF), comprising a multiview forgery localization network (M-FLnet) and a forgery detection network. The M-FLnet, leveraging a content-based prior, generates forgery masks that serve as auxiliary information to improve the detection network’s accuracy. Conversely, the detection network refines these masks, reducing noise for authentic images. Furthermore, two novel forgery datasets, namely, Fake-Vaihingen and Fake-LoveDA, are derived from the Vaihingen and LoveDA satellite image sets, respectively, by exploiting the latest generative models. These datasets represent the first open-source datasets for forgery localization and detection in remote sensing. Extensive experimental results on Fake-Vaihingen and Fake-LoveDA demonstrate that the proposed FLDCF can effectively detect sophisticated forgeries in satellite imagery. The source code and datasets in this work are available athttps://github.com/littlebeen/Forgery-localization-for-remote-sensing.

Blind video quality assessment (VQA) has become an increasingly demanding problem in automatically assessing the quality of ever-growing in-the-wild videos. Although efforts have been made to measure temporal distortions, the core to distinguish between VQA and image quality assessment (IQA), the lack of modeling of how the human visual system (HVS) relates to the temporal quality of videos hinders the precise mapping of predicted temporal scores to the human perception. Inspired by the recent discovery of the temporal straightness law of natural videos in the HVS, this paper intends to model the complex temporal distortions of in-the-wild videos in a simple and uniform representation by describing the geometric properties of videos in the visual perceptual domain. A novel videolet, with perceptual representation embedding of a few consecutive frames, is designed as the basic quality measurement unit to quantify temporal distortions by measuring the angular and linear displacements from the straightness law. By combining the predicted score on each videolet, a perceptually temporal quality evaluator (PTQE) is formed to measure the temporal quality of the entire video. Experimental results demonstrate that the perceptual representation in the HVS is an efficient way of predicting subjective temporal quality. Moreover, when combined with spatial quality metrics, PTQE achieves top performance over popular in-the-wild video datasets. More importantly, PTQE requires no additional information beyond the video being assessed, making it applicable to any dataset without parameter tuning. Additionally, the generalizability of PTQE is evaluated on video frame interpolation tasks, demonstrating its potential to benefit temporal-related enhancement tasks.

The multilayer perceptron (MLP) neural network is interpreted from the geometrical viewpoint in this work, that is, an MLP partition an input feature space into multiple nonoverlapping subspaces using a set of hyperplanes, where the great majority of samples in a subspace belongs to one object class. Based on this high-level idea, we propose a three-layer feedforward MLP (FF-MLP) architecture for its implementation. In the first layer, the input feature space is split into multiple subspaces by a set of partitioning hyperplanes and rectified linear unit (ReLU) activation, which is implemented by the classical two-class linear discriminant analysis (LDA). In the second layer, each neuron activates one of the subspaces formed by the partitioning hyperplanes with specially designed weights. In the third layer, all subspaces of the same class are connected to an output node that represents the object class. The proposed design determines all MLP parameters in a feedforward one-pass fashion analytically without backpropagation. Experiments are conducted to compare the performance of the traditional backpropagation-based MLP (BP-MLP) and the new FF-MLP. It is observed that the FF-MLP outperforms the BP-MLP in terms of design time, training time, and classification performance in several benchmarking datasets. Our source code is available at https://colab.research.google.com/drive/1Gz0L8AnT4ijrUchrhEXXsnaacrFdenn?usp = sharing.

Unsupervised domain adaptation (UDA) has been successfully applied to transfer knowledge from a labeled source domain to target domains without their labels. Recently introduced transferable prototypical networks (TPNs) further address class-wise conditional alignment. In TPN, while the closeness of class centers between source and target domains is explicitly enforced in a latent space, the underlying fine-grained subtype structure and the cross-domain within-class compactness have not been fully investigated. To counter this, we propose a new approach to adaptively perform a fine-grained subtype-aware alignment to improve the performance in the target domain without the subtype label in both domains. The insight of our approach is that the unlabeled subtypes in a class have the local proximity within a subtype while exhibiting disparate characteristics because of different conditional and label shifts. Specifically, we propose to simultaneously enforce subtype-wise compactness and class-wise separation, by utilizing intermediate pseudo-labels. In addition, we systematically investigate various scenarios with and without prior knowledge of subtype numbers and propose to exploit the underlying subtype structure. Furthermore, a dynamic queue framework is developed to evolve the subtype cluster centroids steadily using an alternative processing scheme. Experimental results, carried out with multiview congenital heart disease data and VisDA and DomainNet, show the effectiveness and validity of our subtype-aware UDA, compared with state-of-the-art UDA methods.

We offer a method for one-shot mask-guided image synthesis that allows controlling manipulations of a single image by inverting a quasi-robust classifier equipped with strong regularizers. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier to better preserve the input semantics while preserving its classification accuracy, thereby guaranteeing credibility in the synthesis. Unlike current methods that use complex primitives to supervise the process or use attention maps as a weak supervisory signal, MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior. MAGIC implements a series of manipulations with a single framework achieving shape and location control, intense non-rigid shape deformations, and copy/move operations in the presence of repeating objects and gives users firm control over the synthesis by requiring to simply specify binary guide masks. Our study and findings are supported by various qualitative comparisons with the state-of-the-art on the same images sampled from ImageNet and quantitative analysis using machine perception along with a user survey of 100+ participants that endorse our synthesis quality.

Geometric transformations including translation, rotation, and scaling are commonly used operations in image processing.Besides, some of them are successfully used in developing effective knowledge graph embedding (KGE).Inspired by the synergy, we propose a new KGE model by leveraging all three operations in this work.Since translation, rotation, and scaling operations are cascaded to form a composite one, the new model is named Com-poundE.By casting CompoundE in the framework of group theory, we show that quite a few distanced-based KGE models are special cases of CompoundE.CompoundE extends the simple distance-based scoring functions to relation-dependent compound operations on head and/or tail entities.To demonstrate the effectiveness of CompoundE, we perform three prevalent KG prediction tasks including link prediction, path query answering, and entity typing, on a range of datasets.CompoundE outperforms extant models consistently, demonstrating its effectiveness and flexibility.

Knowledge graph completion (KGC) aims to discover missing relationships between entities in knowledge graphs (KGs).Most prior KGC work focuses on learning embeddings for entities and relations through a simple scoring function.Yet, a higher-dimensional embedding space is usually required for a better reasoning capability, which leads to a larger model size and hinders applicability to real-world problems (e.g., large-scale KGs or mobile/edge computing).A lightweight modularized KGC solution, called GreenKGC, is proposed in this work to address this issue.GreenKGC consists of three modules: representation learning, feature pruning, and decision learning, to extract discriminant KG features and make accurate predictions on missing relationships using classifiers and negative sampling.Experimental results demonstrate that, in low dimensions, GreenKGC can outperform SOTA methods in most datasets.In addition, low-dimensional GreenKGC can achieve competitive or even better performance against high-dimensional models with a much smaller model size.We make our code publicly available.1

AI algorithms at the edge demand smaller model sizes and lower computational complexity. To achieve these objectives, we adopt a green learning (GL) paradigm rather than the deep learning paradigm. GL has three modules: 1) unsupervised representation learning, 2) supervised feature learning, and 3) supervised decision learning. We focus on the second module in this work. In particular, we derive new discriminant features from proper linear combinations of input features, denoted by x, obtained in the first module. They are called complementary and raw features, respectively. Along this line, we present a novel supervised learning method to generate highly discriminant complementary features based on the least-squares normal transform (LNT). LNT consists of two steps. First, we convert a C-class classification problem to a binary classification problem. The two classes are assigned with 0 and 1, respectively. Next, we formulate a least-squares regression problem from the N-dimensional (N-D) feature space to the 1-D output space, and solve the least-squares normal equation to obtain one N-D normal vector, denoted by a1. Since one normal vector is yielded by one binary split, we can obtain M normal vectors with M splits. Then, Ax is called an LNT of x, where transform matrix A$\in R^{M\times N}$ by stacking $\mathrm{a}_{J}^{T}$, j=1,, M, and the LNT, Ax, can generate M new features. The newly generated complementary features are shown to be more discriminant than the raw features. Experiments show that the classification performance can be improved by these new features.

Inspired by the decision learning process of multilayer per-ceptron (MLP) and decision tree (DT), a new classification model, named the subspace learning machine (SLM), is proposed in this work. SLM first identifies a discriminant subspace, S0, by examining the discriminant power of each input feature. Then, it learns projections of features in S0to yield 1D subspaces and finds the optimal partition for each. A criterion is developed to choose the best q partitions that yield 2qpartitioned subspaces. The partitioning process is recursively applied at each child node to build an SLM tree. When the samples at a child node are sufficiently pure, the partitioning process stops, and each leaf node makes a prediction. The ensembles of SLM trees can yield a stronger predictor. Extensive experiments are conducted for performance benchmarking among SLM trees, ensembles and classical classifiers.

Many point cloud classification methods are developed under the assumption that all point clouds in the dataset are well aligned with the canonical axes so that the 3D Cartesian point coordinates can be employed to learn features. When input point clouds are not aligned, the classification performance drops significantly. In this work, we focus on a mathematically transparent point cloud classification method called PointHop, analyze its reason for failure due to pose variations, and solve the problem by replacing its pose dependent modules with rotation invariant counterparts. The pro-posed method is named SO(3)-Invariant PointHop (or S3I-PointHop in short). We also significantly simplify the PointHop pipeline using only one single hop along with multiple spatial aggregation techniques. The idea of exploiting more spatial information is novel. Experiments on the ModelNet40 dataset demonstrate the superiority of S3I-PointHop over traditional PointHop-like methods.

A light-weight super-resolution (LSR) method from a single image targeting mobile applications is proposed in this work. LSR predicts the residual image between the interpolated low-resolution (ILR) and high-resolution (HR) images using a self-supervised framework. To lower the computational complexity, LSR does not adopt the end-to-end optimization deep networks. It consists of three modules: 1) generation of a pool of rich and diversified representations in the neighborhood of a target pixel via unsupervised learning, 2) selecting a subset from the representation pool that is most relevant to the underlying super-resolution task automatically via supervised learning, 3) predicting the residual of the target pixel via regression. LSR has low computational complexity and reasonable model size so that it can be implemented on mobile/edge platforms conveniently. Besides, it offers better visual quality than classical exemplar-based methods in terms of PSNR/SSIM measures.

Despite prolific work on evaluating generative models, little research has been done on the quality evaluation of an individual generated sample. To address this problem, a lightweight generated sample quality evaluation (LGSQE) method is proposed in this work. In the training stage of LGSQE, a binary classifier is trained on real and synthetic samples, where real and synthetic data are labeled by 0 and 1, respectively. In the inference stage, the classifier assigns soft labels (ranging from 0 to 1) to each generated sample. The value of the soft label indicates the quality level; namely, the quality is better if its soft label is closer to 0. LGSQE can serve as a post-processing module for quality control. Furthermore, LGSQE can be used to evaluate the performance of generative models, such as accuracy, AUC, precision, and recall, by aggregating sample-level quality. Experiments are conducted on several datasets and generative models to demonstrate that LGSQE can preserve the same performance rank order as that predicted by the Fréchet Inception Distance (FID) but with significantly lower complexity.

A low-complexity point cloud compression method called the Green Point Cloud Geometry Codec (GPCGC), is proposed to encode the 3D spatial coordinates of static point clouds efficiently. GPCGC consists of two modules. In the first module, point coordinates of input point clouds are hierarchically organized into an octree structure. Points at each leaf node are projected along one of three axes to yield image maps. In the second module, the occupancy map is clustered into 9 modes while the depth map is coded by a low-complexity high-efficiency image codec, called the green image codec (GIC). GIC is a multi-resolution codec based on vector quantization (VQ). Its complexity is significantly lower than HEVC-Intra. Furthermore, the rate-distortion optimization (RDO) technique is used to select the optimal coding parameters. GPCGC is a progressive codec, and it offers a coding performance competitive with MPEG's V-PCC and G-PCC standards at significantly lower complexity.

Subspace partitioning in a high-dimensional feature space plays a fundamental role in the design of effective classifiers. A novel subspace learning machine (SLM) that projects high-dimensional feature vectors into a 1D feature subspace and partitions it into two disjoint sets was recently proposed. As an extension, SLM with soft partitioning, denoted by SLM/SP, is proposed in this work. SLM/SP adopts the soft decision tree (SDT) data structure for decision learning. It starts by learning an adaptive tree structure by using local greedy subspace partitioning. Once the stopping criteria are met for all child nodes and the tree structure is determined, all projection vectors are updated globally. This methodology enables efficient training, high classification accuracy, and a small model size. It is shown by experimental results that an SLM/SP tree offers a lightweight and high performance classification method.

Rapid advances in artificial intelligence (AI) in the last decade have been largely built upon the wide applications of deep learning (DL). However, the high carbon footprint yielded by larger and larger DL networks has become a concern for sustainability . Furthermore, DL decision mechanism is somewhat obscure in that it can only be verified by test data. Green learning (GL) is being proposed as an alternative paradigm to address these concerns. GL is characterized by low carbon footprints, lightweight model, low computational complexity , and logical transparency. It offers energy-efficient solutions in cloud centers as well as mobile/edge devices. GL also provides a more transparent, logical decision-making process which is essential to gaining people’s trust. Several statistical tools such as unsupervised representation learning , supervised feature learning, and supervised decision learning, have been developed to achieve this goal in recent years. We have seen a few successful GL examples with performance comparable with state-of-the-art DL solutions. This paper introduces the key characteristics of GL, its demonstrated applications, and future outlook.

An enhanced label propagation (LP) method called GraphHop was proposed recently. It outperforms graph convolutional networks (GCNs) in the semi-supervised node classification task on various networks. Although the performance of GraphHop was explained intuitively with joint node attribute and label signal smoothening, its rigorous mathematical treatment is lacking. In this paper, we propose a label efficient regularization and propagation (LERP) framework for graph node classification, and present an alternate optimization procedure for its solution. Furthermore, we show that GraphHop only offers an approximate solution to this framework and has two drawbacks. First, it includes all nodes in the classifier training without taking the reliability of pseudo-labeled nodes into account in the label update step. Second, it provides a rough approximation to the optimum of a subproblem in the label aggregation step. Based on the LERP framework, we propose a new method, named the LERP method, to solve these two shortcomings. LERP determines reliable pseudo-labels adaptively during the alternate optimization and provides a better approximation to the optimum with computational efficiency. Theoretical convergence of LERP is guaranteed. Extensive experiments are conducted to demonstrate the effectiveness and efficiency of LERP. That is, LERP outperforms all benchmarking methods, including GraphHop, consistently on five common test datasets, two large-scale networks, and an object recognition task at extremely low label rates (i.e., 1, 2, 4, 8, 16, and 20 labeled samples per class).

Recent deep image compression methods have achieved prominent progress by using nonlinear modeling and powerful representation capabilities of neural networks. However, most existing learning-based image compression approaches employ customized convolutional neural network (CNN) to utilize visual features by treating all pixels equally, neglecting the effect of local key features. Meanwhile, the convolutional filters in CNN usually express the local spatial relationship within the receptive field and seldom consider the long-range dependencies from distant locations. This results in the long-range dependencies of latent representations not being fully compressed. To address these issues, an end-to-end image compression method is proposed by integrating graph attention and asymmetric convolutional neural network (ACNN). Specifically, ACNN is used to strengthen the effect of local key features and reduce the cost of model training. Graph attention is introduced into image compression to address the bottleneck problem of CNN in modeling long-range dependencies. Meanwhile, regarding the limitation that existing attention mechanisms for image compression hardly share information, we propose a self-attention approach which allows information flow to achieve reasonable bit allocation. The proposed self-attention approach is in compliance with the perceptual characteristics of human visual system, as information can interact with each other via attention modules. Moreover, the proposed self-attention approach takes into account channel-level relationship and positional information to promote the compression effect of rich-texture regions. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performances after being optimized by MS-SSIM compared to recent deep compression models on the benchmark datasets of Kodak and Tecnick. The project page with the source code can be found inhttps://mic.tongji.edu.cn.

Dynamic point cloud is a volumetric visual data representing realistic 3D scenes for virtual reality and augmented reality applications. However, its large data volume has been the bottleneck of data processing, transmission, and storage, which requires effective compression. In this paper, we propose a Perceptually Weighted Rate-Distortion Optimization (PWRDO) scheme for Video-based Point Cloud Compression (V-PCC), which aims to minimize the perceptual distortion of reconstructed point cloud at the given bit rate. Firstly, we propose a general framework of perceptually optimized V-PCC to exploit visual redundancies in point clouds. Secondly, a multi-scale Projection based Point Cloud quality Metric (PPCM) is proposed to measure the perceptual quality of 3D point cloud. The PPCM model comprises 3D-to-2D patch projection, multi-scale structural distortion measurement, and fusion model. Approximations and simplifications of the proposed PPCM are also presented for both V-PCC integration and low complexity. Thirdly, based on the simplified PPCM model, we propose a PWRDO scheme with Lagrange multiplier adaptation, which is incorporated into the V-PCC to enhance the coding efficiency. Experimental results show that the proposed PPCM models can be used as standalone quality metrics, and they are able to achieve higher consistency with the human subjective scores than the state-of-the-art objective visual quality metrics. Also, compared with the latest V-PCC reference model, the proposed PWRDO-based V-PCC scheme achieves an average bit rate reduction of 13.52%, 8.16%, 10.56% and 9.54%, respectively, in terms of four objective visual quality metrics for point clouds. It is significantly superior to the state-of-the-art coding algorithms. The computational complexity of the proposed PWRDO increases by 1.71% and 0.05% on average to the V-PCC encoder and decoder, respectively, which is negligible. The source codes of the PPCM and PWRDO schemes are available at https://github.com/VVCodec/PPCM-PWRDO.

Learned video compression has developed rapidly and achieved impressive progress in recent years. Despite efficient compression performance, existing signal fidelity oriented or semantic fidelity oriented video compression methods limit the capability to meet the requirements of both machine and human vision. To address this problem, a task-driven video compression framework is proposed to flexibly support vision tasks for both human vision and machine vision. Specifically, to improve the compression performance, the backbone of the video compression framework is optimized by using three novel modules, including multi-scale motion estimation, multi-frame feature fusion, and reference based in-loop filters. Then, based on the proposed efficient compression backbone, a task-driven optimization approach is designed to achieve the trade-off between signal fidelity oriented compression and semantic fidelity oriented compression. Moreover, a post-filter module is employed for the framework to further improve the performance of the human vision branch. Finally, rate-distortion performance, rate-accuracy performance, and subjective quality are employed as the evaluation metrics, and experimental results show the superiority of the proposed framework for both human vision and machine vision. The source code of this work can be found inhttps://mic.tongji.edu.cn.

Learning low-dimensional representations of bipartite graphs enables e-commerce applications, such as recommendation, classification, and link prediction. A layerwise-trained bipartite graph neural network (L-BGNN) embedding method, which is unsupervised, efficient, and scalable, is proposed in this work. To aggregate the information across and within two partitions of a bipartite graph, a customized interdomain message passing (IDMP) operation and an intradomain alignment (IDA) operation are adopted by the proposed L-BGNN method. Furthermore, we develop a layerwise training algorithm for L-BGNN to capture the multihop relationship of large bipartite networks and improve training efficiency. We conduct extensive experiments on several datasets and downstream tasks of various scales to demonstrate the effectiveness and efficiency of the L-BGNN method as compared with state-of-the-art methods. Our codes are publicly available at https://github.com/TianXieUSC/L-BGNN.

A scalable semisupervised node classification method on graph-structured data, called GraphHop, is proposed in this work. The graph contains all nodes' attributes and link connections but labels of only a subset of nodes. Graph convolutional networks (GCNs) have provided superior performance in node label classification over the traditional label propagation (LP) methods for this problem. Nevertheless, current GCN algorithms suffer from a considerable amount of labels for training because of high model complexity or cannot be easily generalized to large-scale graphs due to the expensive cost of loading the entire graph and node embeddings. Besides, nonlinearity makes the optimization process a mystery. To this end, an enhanced LP method, called GraphHop, is proposed to tackle these problems. GraphHop can be viewed as a smoothening LP algorithm, in which each propagation alternates between two steps: label aggregation and label update. In the label aggregation step, multihop neighbor embeddings are aggregated to the center node. In the label update step, new embeddings are learned and predicted for each node based on aggregated results from the previous step. The two-step iteration improves the graph signal smoothening capacity. Furthermore, to encode attributes, links, and labels on graphs effectively under one framework, we adopt a two-stage training process, i.e., the initialization stage and the iteration stage. Thus, the smooth attribute information extracted from the initialization stage is consistently imposed in the propagation process in the iteration stage. Experimental results show that GraphHop outperforms state-of-the-art graph learning methods on a wide range of tasks in graphs of various sizes (e.g., multilabel and multiclass classification on citation networks, social graphs, and commodity consumption graphs).

Word and sentence embeddings are useful feature representations in natural language processing.However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade.Word and sentence similarity tasks have become the de facto evaluation method.It leads models to overfit to such evaluations, negatively impacting embedding models' development.This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations.Further, we propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks.Extensive experiments are conducted based on 60+ models and popular datasets to certify our judgments.Finally, the practical evaluation toolkit is released for future benchmarking purposes.1

A novel method for detecting CNN-generated images, called Attentive PixelHop (or A-PixelHop), is proposed in this work. It has three advantages: 1) low computational complexity and a small model size, 2) high detection performance against a wide range of generative models, and 3) mathematical transparency. A-PixelHop is designed under the assumption that it is difficult to synthesize high-quality, high-frequency components in local regions. It contains four building modules: 1) selecting edge/texture blocks that contain significant high-frequency components, 2) applying multiple filter banks to them to obtain rich sets of spatial-spectral responses as features, 3) feeding features to multiple binary classifiers to obtain a set of soft decisions, 4) developing an effective ensemble scheme to fuse the soft decisions into the final decision. Experimental results show that A-PixelHop outperforms state-of-the-art methods in detecting CycleGAN-generated images. Furthermore, it can generalize well to unseen generative models and datasets.

An unsupervised point cloud object retrieval and pose estimation method, called PCRP, is proposed in this work. It is assumed that there exists a gallery point cloud set that contains point cloud objects with given pose orientation information. PCRP attempts to register the unknown point cloud object with those in the gallery set so as to achieve content-based object retrieval and pose estimation jointly, where the point cloud registration task is built upon an enhanced version of the unsupervised R-PointHop method. Experiments on the ModelNet40 dataset demonstrate the superior performance of PCRP in comparison with traditional and learning based methods.

An unsupervised nuclei segmentation method for histology images is proposed in this work. It consists of three modules applied to each of non-overlapping blocks: 1) data-driven color transform for dimension reduction, 2) fully-automated adaptive binarization, and 3) incorporation of geometric priors with morphological processing. The method is called CBM, which comes from the first letter of the three modules – "Color transform", "Binarization" and "Morphological processing". Experiments on the MoNuSeg dataset validate the effectiveness of the proposed CBM method. It outperforms all other unsupervised methods and offers a competitive standing among supervised models based on the Aggregated Jaccard Index (AJI) met-ric.

Language modality within the vision language pre-training framework is innately discretized, endowing each word in the language vocabulary a semantic meaning. In contrast, visual modality is inherently continuous and high-dimensional, which potentially prohibits the alignment as well as fusion between vision and language modalities. We therefore propose to "discretize" the visual representation by joint learning a codebook that imbues each visual token a semantic. We then utilize these discretized visual semantics as self-supervised ground-truths for building our Masked Image Modeling objective, a counterpart of Masked Language Modeling which proves successful for language models. To optimize the codebook, we extend the formulation of VQ-VAE which gives a theoretic guarantee. Experiments validate the effectiveness of our approach across common vision-language benchmarks.

A new method, called PSL-DefakeHop, is proposed to detect fake satellite images based on the parallel subspace learning (PSL) framework in this work. The DefakeHop method was developed previously for detection of deepfake generated faces under the successive subspace learning (SSL) framework. PSL is proposed to extract features from responses of multiple single-stage filter banks (or called PixelHops), which operate in parallel, and it improves SSL that extracts features from multi-stage cascaded filter banks. PSL has two advantages. First, PSL preserves discriminant features often lie in high-frequency channels, which are however ignored by SSL. Second, decisions from multiple filter banks can be ensembled to further improve detection accuracy. To demonstrate the effectiveness of the proposed PSL-DefakeHop method, we evaluate it on the UW Fake Satellite Image dataset and observe perfect classification performance (i.e., 100% F1 score, precision and recall).

Being different from deep-learning-based (DL-based) image generation methods, a new image generative model built upon successive subspace learning principle is proposed and named GenHop (an acronym of Generative PixelHop) in this work. GenHop consists of three modules: 1) high-to-low dimension reduction, 2) seed image generation, and 3) low-to-high dimension expansion. In the first module, it builds a sequence of high-to-low dimensional subspaces through a sequence of whitening processes, each of which contains samples of joint-spatial-spectral representation. In the second module, it generates samples in the lowest dimensional subspace. In the third module, it finds a proper high-dimensional sample for a seed image by adding details back via locally linear embedding (LLE) and a sequence of coloring processes. Experiments show that GenHop can generate visually pleasant images whose FID scores are comparable or even better than those of DL-based generative models for MNIST, Fashion-MNIST and CelebA datasets.

Normalizing flow models have been used successfully for generative image super-resolution (SR) by approximating complex distribution of natural images to simple tractable distribution in latent space through Invertible Neural Networks (INN). These models can generate multiple realistic SR images from one low-resolution (LR) input using randomly sampled points in the latent space, simulating the ill-posed nature of image upscaling where multiple high-resolution (HR) images correspond to the same LR. Lately, the invertible process in INN has also been used successfully by bidirectional image rescaling models like IRN and HCFlow for joint optimization of downscaling and inverse upscaling, resulting in significant improvements in upscaled image quality. While they are optimized for image downscaling too, the ill-posed nature of image downscaling, where one HR image could be downsized to multiple LR images depending on different interpolation kernels and resampling methods, is not considered. A new downscaling latent variable, in addition to the original one representing uncertainties in image upscaling, is introduced to model variations in the image downscaling process. This dual latent variable enhancement is applicable to different image rescaling models and it is shown in extensive experiments that it can improve image upscaling accuracy consistently without sacrificing image quality in downscaled LR images. It is also shown to be effective in enhancing other INN-based models for image restoration applications like image hiding.

Visual odometry aims to track the incremental motion of an object using the information captured by visual sensors. In this work, we study the point cloud odometry problem, where only the point cloud scans obtained by the LiDAR (Light Detection And Ranging) are used to estimate object's motion trajectory. A lightweight point cloud odometry solution is proposed and named the green point cloud odometry (GreenPCO) method. GreenPCO is an unsupervised learning method that predicts object motion by matching features of consecutive point cloud scans. It consists of three steps. First, a geometry-aware point sampling scheme is used to select discriminant points from the large point cloud. Second, the view is partitioned into four regions surrounding the object, and the PointHop++ method is used to extract point features. Third, point correspondences are established to estimate object motion between two consecutive scans. Experiments on the KITTI dataset are conducted to demonstrate the effectiveness of the GreenPCO method. It is observed that GreenPCO outperforms benchmarking deep learning methods in accuracy while it has a significantly smaller model size and less training time.

Deep neural networks (DNNs) achieve great success in blind image quality assessment (BIQA) with large pre-trained models in recent years. Their solutions cannot be easily deployed at mobile or edge devices, and a lightweight solution is desired. In this work, we propose a novel BIQA model, called GreenBIQA, that aims at high performance, low computational complexity and a small model size. GreenBIQA adopts an unsupervised feature generation method and a supervised feature selection method to extract quality-aware features. Then, it trains an XGBoost regressor to predict quality scores of test images. We conduct experiments on four popular IQA datasets, which include two synthetic-distortion and two authentic-distortion datasets. Experimental results show that GreenBIQA is competitive in performance against state-of-the-art DNNs with lower complexity and smaller model sizes. Code is available at: https://github.com/zhanxuanm/GreenBIQA

Supervised and unsupervised deep trackers that rely on deep learning technologies are popular in recent years. Yet, they demand high computational complexity and a high memory cost. A green unsupervised single-object tracker, called GUSOT, that aims at object tracking for long videos under a resource-constrained environment is proposed in this work. Built upon a baseline tracker, UHP-SOT++, which works well for short-term tracking, GUSOT contains two additional new modules: 1) lost object recovery, and 2) color-saliency-based shape proposal. They help resolve the tracking loss problem and offer a more flexible object proposal, respectively. Thus, they enable GUSOT to achieve higher tracking accuracy in the long run. We conduct experiments on the large-scale dataset LaSOT with long video sequences, and show that GUSOT offers a lightweight high-performance tracking solution that finds applications in mobile and edge computing platforms.

Deep learning recognition approaches can potentially perform better if we can extract a discriminative representation that controllably separates nuisance factors. In this paper, we propose a novel approach to explicitly enforce the extracted discriminative representation d, extracted latent variation l (e,g., background, unlabeled nuisance attributes), and semantic variation label vector s (e.g., labeled expressions/pose) to be independent and complementary to each other. We can cast this problem as an adversarial game in the latent space of an auto-encoder. Specifically, with the to-be-disentangled s, we propose to equip an end-to-end conditional adversarial network with the ability to decompose an input sample into d and l. However, we argue that maximizing the cross-entropy loss of semantic variation prediction from d is not sufficient to remove the impact of s from d, and that the uniform-target and entropy regularization are necessary. A collaborative mutual information regularization framework is further proposed to avoid unstable adversarial training. It is able to minimize the differentiable mutual information between the variables to enforce independence. The proposed discriminative representation inherits the desired tolerance property guided by prior knowledge of the task. Our proposed framework achieves top performance on diverse recognition tasks, including digits classification, large-scale face recognition on LFW and IJB-A datasets, and face recognition tolerant to changes in lighting, makeup, disguise, etc.

In this work, we propose a new data visualization and clustering technique for discovering discriminative structures in high-dimensional data. This technique, referred to as cPCA++, is motivated by the fact that the interesting features of a “target” dataset may be obscured by high variance components during traditional PCA. By analyzing what is referred to as a “background” dataset (i.e., one that exhibits the high variance principal components but not the interesting structures), our technique is capable of efficiently highlighting the structures that are unique to the “target” dataset. Similar to another recently proposed algorithm called “contrastive PCA” (cPCA), the proposed cPCA++ method identifies important dataset-specific patterns that are not detected by traditional PCA in a wide variety of settings. However, unlike cPCA, the proposed cPCA++ method does not require a parameter sweep, and as a result, it is significantly more efficient. Several experiments were conducted in order to compare the proposed method to state-of-the-art methods. These experiments show that the proposed method achieves performance that is similar to or better than that of the other methods, while being more efficient.

Human visual system has a limitation of sensitivity in detecting small distortion in an image/video and the minimum perceptual threshold is so called Just Noticeable Difference (JND). JND modelling is challenging since it highly depends on visual contents and perceptual factors are not fully understood. In this paper, we propose deep learning based JND and perceptual quality prediction models, which are able to predict the Satisfied User Ratio (SUR) and Video Wise JND (VWJND) of compressed videos with different resolutions and coding parameters. Firstly, the SUR prediction is modeled as a regression problem that fits deep learning tools. Then, Video Wise Spatial SUR method (VW-SSUR) is proposed to predict the SUR value for compressed video, which mainly considers the spatial distortion. Thirdly, we further propose Video Wise Spatial-Temporal SUR (VW-STSUR) method to improve the SUR prediction accuracy by considering the spatial and temporal information. Two fusion schemes that fuse the spatial and temporal information in quality score level and in feature level, respectively, are investigated. Finally, key factors including key frame and patch selections, cross resolution prediction and complexity are analyzed. Experimental results demonstrate the proposed VW-SSUR method outperforms in both SUR and VWJND prediction as compared with the state-of-the-art schemes. Moreover, the proposed VW-STSUR further improves the accuracy as compared with the VW-SSUR and the conventional JND models, where the mean SUR prediction error is 0.049, and mean VWJND prediction error is 1.69 in quantization parameter and 0.84 dB in peak signal-to-noise ratio.

Inspired by the recent PointHop classification method, an unsupervised 3D point cloud registration method, called R-PointHop, is proposed in this work. R-PointHop first determines a local reference frame (LRF) for every point using its nearest neighbors and finds local attributes. Next, R-PointHop obtains local-to-global hierarchical features by point downsampling, neighborhood expansion, attribute construction and dimensionality reduction steps. Thus, point correspondences are built in hierarchical feature space using the nearest neighbor rule. Afterwards, a subset of salient points with good correspondence is selected to estimate the 3D transformation. The use of the LRF allows for invariance of the hierarchical features of points with respect to rotation and translation, thus making R-PointHop more robust at building point correspondence, even when the rotation angles are large. Experiments are conducted on the 3DMatch, ModelNet40, and Stanford Bunny datasets, which demonstrate the effectiveness of R-PointHop for 3D point cloud registration. R-PointHop's model size and training time are an order of magnitude smaller than those of deep learning methods, and its registration errors are smaller, making it a green and accurate solution. Our codes are available on GitHub (https://github.com/pranavkdm/R-PointHop).

Deep learning has great potential for accurate detection and classification of diseases with medical imaging data, but the performance is often limited by the number of training datasets and memory requirements. In addition, many deep learning models are considered a "black-box," thereby often limiting their adoption in clinical applications. To address this, we present a successive subspace learning model, termed VoxelHop, for accurate classification of Amyotrophic Lateral Sclerosis (ALS) using T2-weighted structural MRI data. Compared with popular convolutional neural network (CNN) architectures, VoxelHop has modular and transparent structures with fewer parameters without any backpropagation, so it is well-suited to small dataset size and 3D imaging data. Our VoxelHop has four key components, including (1) sequential expansion of near-to-far neighborhood for multi-channel 3D data; (2) subspace approximation for unsupervised dimension reduction; (3) label-assisted regression for supervised dimension reduction; and (4) concatenation of features and classification between controls and patients. Our experimental results demonstrate that our framework using a total of 20 controls and 26 patients achieves an accuracy of 93.48 % and an AUC score of 0.9394 in differentiating patients from controls, even with a relatively small number of datasets, showing its robustness and effectiveness. Our thorough evaluations also show its validity and superiority to the state-of-the-art 3D CNN classification approaches. Our framework can easily be generalized to other classification tasks using different imaging modalities.

Modeling statistical properties of anatomical structures using magnetic resonance imaging is essential for revealing common information of a target population and unique properties of specific subjects. In brain imaging, a statistical brain atlas is often constructed using a number of healthy subjects. When tumors are present, however, it is difficult to either provide a common space for various subjects or align their imaging data due to the unpredictable distribution of lesions. Here we propose a deep learning-based image inpainting method to replace the tumor regions with normal tissue intensities using only a patient population. Our framework has three major innovations: 1) incompletely distributed datasets with random tumor locations can be used for training; 2) irregularly-shaped tumor regions are properly learned, identified, and corrected; and 3) a symmetry constraint between the two brain hemispheres is applied to regularize inpainted regions. Henceforth, regular atlas construction and image registration methods can be applied using inpainted data to obtain tissue deformation, thereby achieving group-specific statistical atlases and patient-to-atlas registration. Our framework was tested using the public database from the Multimodal Brain Tumor Segmentation challenge. Results showed increased similarity scores as well as reduced reconstruction errors compared with three existing image inpainting methods. Patient-to-atlas registration also yielded better results with improved normalized cross-correlation and mutual information and a reduced amount of deformation over the tumor regions.

Recent advances in unsupervised domain adaptation (UDA) show that transferable prototypical learning presents a powerful means for class conditional alignment, which encourages the closeness of cross-domain class centroids. However, the cross-domain inner-class compactness and the underlying fine-grained subtype structure remained largely underexplored. In this work, we propose to adaptively carry out the fine-grained subtype-aware alignment by explicitly enforcing the class-wise separation and subtype-wise compactness with intermediate pseudo labels. Our key insight is that the unlabeled subtypes of a class can be divergent to one another with different conditional and label shifts, while inheriting the local proximity within a subtype. The cases with or without the prior information on subtype numbers are investigated to discover the underlying subtype structure in an online fashion. The proposed subtype-aware dynamic UDA achieves promising results on a medical diagnosis task.

Most existing distance metric learning approaches use fully labeled data to learn the sample similarities in an embedding space. We present a self-training framework, SLADE, to improve retrieval performance by leveraging additional unlabeled data. We first train a teacher model on the labeled data and use it to generate pseudo labels for the unlabeled data. We then train a student model on both labels and pseudo labels to generate final feature embeddings. We use self-supervised representation learning to initialize the teacher model. To better deal with noisy pseudo labels generated by the teacher network, we design a new feature basis learning component for the student network, which learns basis functions of feature representations for unlabeled data. The learned basis vectors better measure the pairwise similarity and are used to select high-confident samples for training the student network. We evaluate our method on standard retrieval benchmarks: CUB-200, Cars-196 and In-shop. Experimental results demonstrate that with additional unlabeled data, our approach significantly improves the performance over the state-of-the-art methods.

Targeting both computing and coding efficiencies, we propose in this work a novel hierarchical bit-wise differential coding scheme to compress point cloud attributes. The encoder firstly quantizes and organizes the points into an octree structure and, for each internal node, picks its attribute(s) from a child named source child. Next, the encoder conducts a top-down scanning of the hierarchy. For each node with more than one child, it computes the bit-wise attribute difference between the current node and each non-source child by exclusive-OR and encodes the difference with an arithmetic coder. Further, a table look-up approach is proposed to accelerate the online source child identification. The proposed scheme produces superior computing and coding efficiencies for lossless point cloud attribute compression, outperforming the MPEG benchmark coders by large margins in our experiments.

In this work, we propose an adversarial unsupervised domain adaptation (UDA) method under inherent conditional and label shifts, in which we aim to align the distributions w.r.t. both p(x|y) and p(y). Since labels are inaccessible in a target domain, conventional adversarial UDA methods assume that p(y) is invariant across domains and rely on aligning p(x) as an alternative to the p(x|y) alignment. To address this, we provide a thorough theoretical and empirical analysis of the conventional adversarial UDA methods under both conditional and label shifts, and propose a novel and practical alternative optimization scheme for adversarial UDA. Specifically, we infer the marginal p(y) and align p(x|y) iteratively at the training stage, and precisely align the posterior p(y|x) at the testing stage. Our experimental results demonstrate its effectiveness on both classification and segmentation UDA and partial UDA.

A high quality disparity remapping method that preserves 2D shapes and 3D structures, and adjusts disparities of important objects in stereo image pairs is proposed. It is formulated as a constrained optimization problem, whose solution is challenging, since we need to meet multiple requirements of disparity remapping simultaneously. The one-stage optimization process either degrades the quality of important objects or introduces serious distortions in background regions. To address this challenge, we propose a two-stage warping process to solve it. In the first stage, we develop a warping model that finds the optimal warping grids for important objects to fulfill multiple requirements of disparity remapping. In the second stage, we derive another warping model to refine warping results in less important regions by eliminating serious distortions in shape, disparity and 3D structure. The superior performance of the proposed method is demonstrated by experimental results.

A light-weight high-performance Deepfake detection method, called DefakeHop, is proposed in this work. State-of-the-art Deepfake detection methods are built upon deep neural networks. DefakeHop uses the successive subspace learning (SSL) principle to extracts features automatically from various parts of face images. The features are extracted by channel-wise (c/w) Saab transform and further processed by our feature distillation module using spatial dimension re-duction and soft classification for each channel to get a more concise description of the face. Extensive experiments are conducted to demonstrate the effectiveness of the proposed DefakeHop method. With a small model size of 42,845 parameters, DefakeHop achieves state-of-the-art performance with the area under the ROC curve (AUC) of 100%, 94.95%, and 90.56% on UADFV, Celeb-DF v1, and Celeb-DF v2 datasets, respectively. Our codes are available on GitHub1.

Commonsense knowledge graph (CKG) is a special type of knowledge graph (KG), where entities are composed of free-form text. Existing CKG completion methods focus on transductive learning setting, where all the entities are present during training. Here, we propose the first inductive learning setting for CKG completion, where unseen entities may appear at test time. We emphasize that the inductive learning setting is crucial for CKGs, because unseen entities are frequently introduced due to the fact that CKGs are dynamic and highly sparse. We propose InductivE as the first framework targeted at the inductive CKG completion task. InductivE first ensures the inductive learning capability by directly computing entity embeddings from raw entity attributes. Second, a graph neural network with novel densification process is proposed to further enhance unseen entity representation with neighboring structural information. Experimental results show that InductivE performs especially well on inductive scenarios where it achieves above 48% improvement over previous methods while also outperforms state-of-the-art baselines in transductive settings.

An image anomaly localization method based on the successive subspace learning (SSL) framework, called Anomaly-Hop, is proposed in this work. AnomalyHop consists of three modules: 1) feature extraction via successive subspace learning (SSL), 2) normality feature distributions modeling via Gaussian models, and 3) anomaly map generation and fusion. Comparing with state-of-the-art image anomaly localization methods based on deep neural networks (DNNs), AnomalyHop is mathematically transparent, easy to train, and fast in its inference speed. Besides, its area under the ROC curve (ROC-AUC) performance on the MVTec AD dataset is 95.9%, which is among the best of several benchmarking methods.

An unsupervised online object tracking method that exploits both foreground and background correlations is proposed and named UHP-SOT (Unsupervised High-Performance Single Object Tracker) in this work. UHP-SOT consists of three modules: 1) appearance model update, 2) background motion modeling, and 3) trajectory-based box prediction. A state-of-the-art discriminative correlation filters (DCF) based tracker is adopted by UHP-SOT as the first module. We point out shortcomings of using the first module alone such as failure in recovering from tracking loss and inflexibility in object box adaptation and then propose the second and third modules to overcome them. Both are novel in single object tracking (SOT). We test UHP-SOT on two popular object tracking benchmarks, TB-50 and TB-100, and show that it outperforms all previous unsupervised SOT methods, achieves a performance comparable with the best supervised deep-learning-based SOT methods, and operates at a fast speed (i.e. 22.7-32.0 FPS on a CPU).

The relationship between a multilayer perceptron (MLP) regressor and a piecewise polynomial approximator is investigated in this work. We propose an MLP construction method, including the choice of activation, the specification of neuron numbers and filter weights. Through the construction, a one-to-one correspondence between an MLP and a piecewise polynomial is established. Especially, we point out that the form of nonlinear activation is related to the polynomial order. Since the approximation capability of piecewise polynomials is well understood, our study sheds new light on the universal approximation capability of an MLP.

A point cloud is a set of 3D points that can be used to represent a 3D surface. Each point has a spatial position (x, y, z) and a vector of attributes, such as colors, material reflection, or normal. As point clouds are capable of reconstructing 3D objects or scenes, they have the potential to be widely used in various applications such as auto-driving and 6-degree virtual reality. However, the following properties of point cloud make the point cloud compression and processing become rather challenging. 1) Unstructured. The point cloud is a series of non-uniform sampled points. On the one hand, it makes the correlations among various points difficult to be utilized for compression. On the other hand, the convolutional neural network that is widely used in image/video processing cannot be applied to the point cloud processing. 2) Unordered. Unlike images and videos, the point cloud is a set of points without a specific order. Therefore, both the point cloud processing and compression algorithms need to be invariant to any permutations of the input point clouds.

Image quality assessment is a critical problem for image compression, which can be utilized as a guidance for image compression and codec evaluation. In this paper, we propose a full reference image quality assessment (IQA) algorithm to measure the perceptual quality of compressed images. The proposed IQA model utilizes a data-driven transform, multi-stage Karhunen-Loeve Transform (MS-KLT), as a feature extractor to decompose both reference and compressed images into feature domain, where the importance of feature distortions in different spectral components to human visual system (HVS) is easy to distinguish. Accordingly, an efficient weighting strategy is proposed to reflect the importance of feature distortions based on the energy of transformed coefficients. Considering HVS characteristics, weighted spatial masking effect is derived from both local and global perspectives. In addition, to avoid influences of random noises, a local adaptive low-pass filtering process is applied as a pre-processing operation. Extensive experimental results on popular datasets show that our proposed method correlates better with the subjective results compared with the state-of-the-art algorithms. Moreover, the proposed method behaves more robustly compared with existing methods, and achieves top-ranking performance on different IQA datasets.

Viewing various stereo images under different viewing conditions has escalated the need for effective object-level remapping techniques. In this paper, we propose a new object spatial mapping scheme, which adjusts the depth and size of the selected object to match user preference and viewing conditions. Existing warping-based methods often distort the shape of important objects or cannot faithfully adjust the depth/size of the selected object due to improper warping such as local rotations. In this paper, by explicitly reducing the transformation freedom degree of warping, we propose an optimization model based on axis-aligned warping for object spatial remapping. The proposed axis-aligned warping based optimization model can simultaneously adjust the depths and sizes of selected objects to their target values without introducing severe shape distortions. Moreover, we propose object consistency constraints to ensure the size/shape of parts inside a selected object to be consistently adjusted. Such constraints improve the size/shape adjustment performance while remaining robust to some extent to incomplete object extraction. Experimental results demonstrate that the proposed method achieves high flexibility and effectiveness in adjusting the size and depth of objects compared with existing methods.

It has been recognized that videos have to be encoded in a rate-distortion optimized manner for high coding performance. Therefore, operational coding methods have been developed for conventional distortion metrics such as Sum of Squared Error (SSE). Nowadays, with the rapid development of machine learning, the state-of-the-art learning based metric Video Multimethod Assessment Fusion (VMAF) has been proven to outperform conventional ones in terms of the correlation with human perception, and thus deserves integration into the coding framework. However, unlike conventional metrics, VMAF has no specific computational formulas and may be frequently updated by new training data, which invalidates the existing coding methods and makes it highly desired to develop a rate-distortion optimized method for VMAF. Moreover, VMAF is designed to operate at the frame level, which leads to further difficulties in its application to today's block based coding. In this paper, we propose a VMAF oriented perceptual coding method based on piecewise metric coupling. Firstly, we explore the correlation between VMAF and SSE in the neighborhood of a benchmark distortion. Then a rate-distortion optimization model is formulated based on the correlation, and an optimized block based coding method is presented for VMAF. Experimental results show that 3.61% and 2.67% bit saving on average can be achieved for VMAF under the low_delay_p and the random_access_main configurations of HEVC coding respectively.

Blind image quality assessment (BIQA) plays an important role in image services as independent of the reference image. Herein, the perceptual relevant feature design is the core of BIQA methods, but their performance is still not satisfied at present. In this work, we propose an unsupervised feature extraction approach for BIQA based on Karhunen-Loéve transform (KLT). Specifically, a normalization operation is firstly applied to the test image by calculating its mean subtracted contrast normalized (MSCN) coefficient. Then, KLT is employed as a data-driven feature extraction approach to extract image structural features, wherein kernels with different sizes are utilized to perform multi-scale analysis. Finally, generalized Gaussian distribution (GGD) is employed to model the KLT coefficients distribution in different spectral components as quality relevant features. Extensive experiments conducted on four widely utilized IQA databases have demonstrated that the proposed Multi-scale KLT (MsKLT) BIQA metric compares favorably with existing BIQA methods in terms of high accordance with human subjective scores on both common and uncommon distortion types.

A block-level perceptual image compression framework is proposed in this work, including a block-level just noticeable difference (JND) prediction model and a preprocessing scheme. Specifically speaking, block-level JND values are first deduced by utilizing the OTSU method based on the variation of block-level structural similarity values between two adjacent picture-level JND values in the MCL-JCI dataset. After the JND value for each image block is generated, a convolutional neural network–based prediction model is designed to forecast block-level JND values for a given target image. Then, a preprocessing scheme is devised to modify the discrete cosine transform coefficients during JPEG compression on the basis of the distribution of block-level JND values of the target test image. Finally, the test image is compressed by the max JND value across all of its image blocks in the light of the initial quality factor setting. The experimental results demonstrate that the proposed block-level perceptual image compression method is able to achieve 16.75% bit saving as compared to the state-of-the-art method with similar subjective quality. The project page can be found at https://mic.tongji.edu.cn/43/3f/c9778a148287/page.htm.

The successive subspace learning (SSL) principle was developed and used to design an interpretable learning model, known as the PixelHop method, for image classification in our prior work. Here, we propose an improved PixelHop method and call it Pixel-Hop++. First, to make the PixelHop model size smaller, we decouple a joint spatial-spectral input tensor to multiple spatial tensors (one for each spectral component) under the spatial-spectral separability assumption and perform the Saab transform in a channel-wise manner, called the channel-wise (c/w) Saab transform. Second, by performing this operation from one hop to another successively, we construct a channel-decomposed feature tree whose leaf nodes contain features of one dimension (1D). Third, these 1D features are ranked according to their cross-entropy values, which allows us to select a subset of discriminant features for image classification. In Pixel-Hop++, one can control the learning model size of fine-granularity, offering a flexible tradeoff between the model size and the classification performance. We demonstrate the flexibility of Pixel-Hop++ on MNIST, Fashion MNIST, and CIFAR-10 three datasets.

The PointHop method was recently proposed by Zhang et al. for 3D point cloud classification with unsupervised feature extraction. It has an extremely low training complexity while achieving state-of-the-art classification performance. In this work, we improve the PointHop method furthermore in two aspects: 1) reducing its model complexity in terms of the model parameter number and 2) ordering discriminant features automatically based on the cross-entropy criterion. The resulting method is called PointHop++. The first improvement is essential for wearable and mobile computing while the second improvement bridges statistics-based and optimization-based machine learning methodologies. With experiments conducted on the ModelNet40 benchmark dataset, we show that the PointHop++ method performs on par with deep neural network (DNN) solutions and surpasses other unsupervised feature extraction methods.

A novel sentence embedding method built upon semantic subspace analysis, called semantic subspace sentence embedding (S3E), is proposed in this work. Given the fact that word embeddings can capture semantic relationship while semantically similar words tend to form semantic groups in a high-dimensional embedding space, we develop a sentence representation scheme by analyzing semantic subspaces of its constituent words. Specifically, we construct a sentence model from two aspects. First, we represent words that lie in the same semantic group using the intra-group descriptor. Second, we characterize the interaction between multiple semantic groups with the inter-group descriptor. The proposed S3E method is evaluated on both textual similarity tasks and supervised tasks. Experimental results show that it offers comparable or better performance than the state-of-the-art. The complexity of our S3E method is also much lower than other parameterized models.

Categorical speech emotion recognition is typically performed as a sequence-to-label problem, i.e., to determine the discrete emotion label of the input utterance as a whole. One of the main challenges in practice is that most of the existing emotion corpora do not give ground truth labels for each segment; instead, we only have labels for whole utterances. To extract segment-level emotional information from such weakly labeled emotion corpora, we propose using multiple instance learning (MIL) to learn segment embeddings in a weakly supervised manner. Also, for a sufficiently long utterance, not all of the segments contain relevant emotional information. In this regard, three attention-based neural network models are then applied to the learned segment embeddings to attend the most salient part of a speech utterance. Experiments on the CASIA corpus and the IEMOCAP database show better or highly competitive results than other state-of-the-art approaches.

A simple and effective low-light image enhancement method based on a noise-aware texture-preserving retinex model is proposed in this work. The new method, called NATLE, attempts to strike a balance between noise removal and natural texture preservation through a lowcomplexity solution. Its cost function includes an estimated piece-wise smooth illumination map and a noise-free texture-preserving reflectance map. Afterwards, illumination is adjusted to form the enhanced image together with the reflectance map. Extensive experiments are conducted on common low-light image enhancement datasets to demonstrate the superior performance of NATLE.

Inspired by the recently proposed successive subspace learning (SSL) principles, we develop a successive subspace graph transform (SSGT) to address point cloud attribute compression in this work. The octree geometry structure is utilized to partition the point cloud, where every node of the octree represents a point cloud subspace with a certain spatial size. We design a weighted graph with self-loop to describe the subspace and define a graph Fourier transform based on the normalized graph Laplacian. The transforms are applied to large point clouds from the leaf nodes to the root node of the octree recursively, while the represented subspace is expanded from the smallest one to the whole point cloud successively. It is shown by experimental results that the proposed SSGT method offers better R-D performances than the previous Region Adaptive Haar Transform (RAHT) method.

An unsupervised point cloud registration method, called salient points analysis (SPA), is proposed in this work. The proposed SPA method can register two point clouds effectively using only a small subset of salient points. It first applies the PointHop++ method to point clouds, finds corresponding salient points in two point clouds based on the local surface characteristics of points and performs registration by matching the corresponding salient points. The SPA method offers several advantages over the recent deep learning based solutions for registration. Deep learning methods such as PointNetLK and DCP train end-to-end networks and rely on full supervision (namely, ground truth transformation matrix and class label). In contrast, the SPA is completely unsupervised. Furthermore, SPA's training time and model size are much less. The effectiveness of the SPA method is demonstrated by experiments on seen and unseen classes and noisy point clouds from the ModelNet-40 dataset.

3D point cloud analysis and processing find numerous applications in computer-aided design, 3D printing, autonomous driving, etc. Most state-of-the-art point cloud processing methods are based on convolutional neural networks (CNNs). Although they outperform traditional methods in terms of accuracy, they demand heavy supervision and higher training complexity. Besides, they lack mathematical transparency. In this talk, I will present three interpretable and effective machine learning methods for 3D point cloud registration, classification and segmentation, respectively. First, an unsupervised registration method that extracts salient points for matching is presented. Second, an unambiguous way to order points sequentially in a point cloud set is developed. Then, their spatial coordinates can be treated as geometric attributes of 1 D data array. This idea facilitates the classification task. Third, for the segmentation task, we show how to leverage prior knowledge on point clouds to derive an intuitive and effective segmentation method. Extensive experiments are conducted to demonstrate the performance of the three new methods. I will also provide performance benchmarking between these interpretable methods and deep learning methods.

In contrast to supervised backpropagation-based feature learning in deep neural networks (DNNs), an unsupervised feedforward feature (UFF) learning scheme for joint classification and segmentation of 3D point clouds is proposed in this work. The UFF method exploits statistical correlations of points in a point cloud set to learn shape and point features in a one-pass feedforward manner through a cascaded encoder-decoder architecture. It learns global shape features through the encoder and local point features through the concatenated encoder-decoder architecture. The extracted features of an input point cloud are fed to classifiers f or shape classification and part segmentation. Experiments are conducted to evaluate the performance of the UFF method. For shape classification, the UFF is superior to existing unsupervised methods and on par with state-of-the-art DNNs. For part segmentation, the UFF outperforms semi-supervised methods and performs slightly worse than DNNs.

Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) — an abrupt degradation of performance on the original set of classes when the training objective is adapted to a newly added set of classes. Existing IL approaches tend to produce a model that is biased towards either the old classes or new classes, unless with the help of exemplars of the old data. To address this issue, we propose a class-incremental learning paradigm called Deep Model Consolidation (DMC), which works well even when the original training data is not available. The idea is to first train a separate model only for the new classes, and then combine the two individual models trained on data of two distinct set of classes (old classes and new classes) via a novel double distillation training objective. The two existing models are consolidated by exploiting publicly available unlabeled auxiliary data. This overcomes the potential difficulties due to unavailability of original training data. Compared to the state-of-the-art techniques, DMC demonstrates significantly better performance in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL VOC 2007) in the single-headed IL setting.

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fuse information across layers to find better sentence representations. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentence-level probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available.

High Efficiency Video Coding (HEVC) has doubled the video compression ratio with equivalent subjective quality as compared to its predecessor H.264/AVC. The significant coding efficiency improvement is attributed to many new techniques. Inter-frame coding is one of the most powerful yet complicated techniques therein and has posed high computational burden thus main obstacle in HEVC-based real-time applications. Recently, plenty of research has been done to optimize the inter-frame coding, either to reduce the complexity for real-time applications, or to further enhance the encoding efficiency. In this paper, we provide a comprehensive review of the state-of-the-art techniques for HEVC inter-frame coding from three aspects, namely fast inter coding solutions, implementation on different hardware platforms as well as advanced inter coding techniques. More specifically, different algorithms in each aspect are further subdivided into sub-categories and compared in terms of pros, cons, coding efficiency and coding complexity. To the best of our knowledge, this is the first such comprehensive review of the recent advances of the inter-frame coding for HEVC and hopefully it would help the improvement, implementation and applications of HEVC as well as the ongoing development of the next generation video coding standard.

Stereo video retargeting aims at minimizing shape and depth distortions with temporal coherence in resizing a stereo video content to a desired size. Existing methods extend stereo image retargeting schemes to stereo video retargeting by adding additional temporal constraints that demand temporal coherence in all corresponding regions. However, such a straightforward extension incurs conflicts among multiple requirements (i.e., shape and depth preservation and their temporal coherence), thus failing to meet one or more of these requirements satisfactorily. To mitigate conflicts among depth, shape, and temporal constraints and avoid degrading temporal coherence perceptually, we relax temporal constraints for non-paired regions at frame boundaries, derive new temporal constraints to improve human viewing experience of a 3D scene, and propose an efficient grid-based implementation for stereo video retargeting. Experimental results demonstrate that our method achieves superior visual quality over existing methods.

Picture Wise Just Noticeable Difference (PW-JND), which accounts for the minimum difference of a picture that human visual system can perceive, can be widely used in perception-oriented image and video processing. However, the conventional Just Noticeable Difference (JND) models calculate the JND threshold for each pixel or sub-band separately, which may not reflect the total masking effect of a picture accurately. In this paper, we propose a deep learning based PW-JND prediction model for image compression. Firstly, we formulate the task of predicting PW-JND as a multi-class classification problem, and propose a framework to transform the multi-class classification problem to a binary classification problem solved by just one binary classifier. Secondly, we construct a deep learning based binary classifier named perceptually lossy/lossless predictor which can predict whether an image is perceptually lossy to another or not. Finally, we propose a sliding window based search strategy to predict PW-JND based on the prediction results of the perceptually lossy/lossless predictor. Experimental results show that the mean accuracy of the perceptually lossy/lossless predictor reaches 92%, and the absolute prediction error of the proposed PW-JND model is 0.79 dB on average, which shows the superiority of the proposed PW-JND model to the conventional JND models.

Image compression has always been an important topic in the last decades due to the explosive increase of images. The popular image compression formats are based on different transforms which convert images from the spatial domain into compact frequency domain to remove the spatial correlation. In this paper, we focus on the exploration of data-driven transform, Karhunen-Loéve transform (KLT), the kernels of which are derived from specific images via Principal Component Analysis (PCA), and design a high efficient KLT based image compression algorithm with variable transform sizes. To explore the optimal compression performance, the multiple transform sizes and categories are utilized and determined adaptively according to their rate-distortion (RD) costs. Moreover, comprehensive analyses on the transform coefficients are provided and a band-adaptive quantization scheme is proposed based on the coefficient RD performance. Extensive experiments are performed on several class-specific images as well as general images, and the proposed method achieves significant coding gain over the popular image compression standards including JPEG, JPEG 2000, and the state-of-the-art dictionary learning based methods.

With explosive increase of internet video services, perceptual modeling for video quality has attracted more attentions to provide high quality-of-experience (QoE) for end-users subject to bandwidth constraints, especially for compressed video quality. In this paper, a novel perceptual model for satisfied-user-ratio (SUR) on compressed video quality is proposed by exploiting compressed video bitrate changes and spatial-temporal statistical characteristics extracted from both uncompressed original video and reference video. In the proposed method, an efficient video feature set is explored and established to model SUR curves against bitrate variations by leveraging the Gaussian Processes Regression (GPR) framework. In particular, the proposed model is based on the recently released large-scale video quality dataset, VideoSet, and takes both spatial and temporal masking effects into consideration. To make it more practical, we further optimize the proposed method from three aspects including feature source simplification, computation complexity reduction and video codec adaption. Based on experimental results on VideoSet, the proposed method can accurately model SUR curves for various video contents and predict their required bitrates at given SUR values. Subjective experiments are conducted to further verify the generalization ability of the proposed SUR model.

An explainable machine learning method for point cloud classification, called the PointHop method, is proposed in this work. The PointHop method consists of two stages: 1) local-to-global attribute building through iterative one-hop information exchange and 2) classification and ensembles. In the attribute building stage, we address the problem of unordered point cloud data using a space partitioning procedure and developing a robust descriptor that characterizes the relationship between a point and its one-hop neighbor in a PointHop unit. When we put multiple PointHop units in cascade, the attributes of a point will grow by taking its relationship with one-hop neighbor points into account iteratively. Furthermore, to control the rapid dimension growth of the attribute vector associated with a point, we use the Saab transform to reduce the attribute dimension in each PointHop unit. In the classification and ensemble stage, we feed the feature vector obtained from multiple PointHop units to a classifier. We explore ensemble methods to improve the classification performance furthermore. It is shown by experimental results that the PointHop method offers classification performance that is comparable with state-of-the-art methods while demanding much lower training complexity.

Unsupervised neural machine translation (UNMT) has recently achieved remarkable results \cite{lample2018phrase} with only large monolingual corpora in each language. However, the uncertainty of associating target with source sentences makes UNMT theoretically an ill-posed problem. This work investigates the possibility of utilizing images for disambiguation to improve the performance of UNMT. Our assumption is intuitively based on the invariant property of image, i.e., the description of the same visual content by different languages should be approximately similar. We propose an unsupervised multi-modal machine translation (UMNMT) framework based on the language translation cycle consistency loss conditional on the image, targeting to learn the bidirectional multi-modal translation simultaneously. Through an alternate training between multi-modal and uni-modal, our inference model can translate with or without the image. On the widely used Multi30K dataset, the experimental results of our approach are significantly better than those of the text-only UNMT on the 2016 test dataset.

High Efficiency Video Coding (HEVC) achieves significant coding performance over H.264. However, the performance gain is achieved at the cost of substantially higher encoding complexity, in which the coding tree unit (CTU) partition is one of the most time-consuming parts due to the rate-distortion optimization-based ergodic search of all possible quad-tree partitions. To address this problem, this paper proposes a texture-classification accelerated convolutional neural network (CNN)-based fast intra CU partition scheme to reduce the encoding complexity for intra-coding in HEVC, by taking into consideration of the heterogeneous texture characteristics into the CNN-based classification. First, a threshold-based texture classification model is developed to identify the heterogeneous and homogeneous CTUs, through jointly consideration of the CU depth, quantization parameter and texture complexity. Second, three different CNN structures are designed and trained to predict the CU partition mode for each CU layer in the heterogeneous CTUs. Finally, extensive experimental results show that the proposed scheme can reduce intra-mode encoding time by 62.13% with negligible BD-rate loss of 2.01%, consistently outperforming two state-of-the-art CNN-based schemes in terms of both coding performance and complexity reduction.

Language graph learning is an important task with many applications such as text classification, link prediction and community detection. One of the challenges in this domain is finding an efficient way to learn and encode graph into a low dimensional embedding. In this paper, a novel DeepWalk-assisted Graph PCA (DGPCA) method is proposed for processing language network data represented by graphs. This method can generate a precise text representation for nodes (or vertices) in language networks. Unlike other existing work, our learned low dimensional vector representations add flexibility in exploring vertices' neighborhood information, while reducing noise contained in the original data. To demonstrate the effectiveness, we use DGPCA to classify vertices that contain text information in three language networks. Experimentally, DGPCA is shown to perform well on the language datasets in comparison to several state-of-the-art benchmarking methods.

Features that capture textural patterns of a certain class of images are crucial for texture segmentation tasks. This paper introduces a data-centric approach to efficiently extract and represent textural information, which adapts to a wide variety of textures. Based on the strong self-similarities and quasi-periodicity in texture images, the proposed method first constructs a representative texture pattern set for the given image by leveraging the patch clustering strategy. Then, pixel-wise texture features are designed according to the similarities between local patches and the representative textural patterns. Moreover, the proposed feature is generic and flexible, and can perform segmentation task by integrating it into various segmentation approaches easily. Extensive experimental results on both textural and natural image segmentation show that the segmentation method using the proposed features achieves very competitive or even better performance compared with the stat-of-the-art methods.

An ensemble method that fuses the output decision vectors of multiple feedforward-designed convolutional neural networks (FF-CNNs) to solve the image classification problem is proposed in this work. To enhance the performance of the ensemble system, it is critical to increase the diversity of FF-CNN models. To achieve this objective, we introduce diversities by adopting three strategies: 1) different parameter settings in convolutional layers, 2) flexible feature subsets fed into the Fully-connected (FC) layers, and 3) multiple image embeddings of the same input source. Furthermore, we partition input samples into easy and hard ones based on their decision confidence scores. As a result, we can develop a new ensemble system tailored to hard samples to further boost classification accuracy. Experiments are conducted on the MNIST and CIFAR-10 datasets to demonstrate the effectiveness of the ensemble method.

A semi-supervised learning framework using the feedforward-designed convolutional neural networks (FF-CNNs) is proposed for image classification in this work. One unique property of FF-CNNs is that no backpropagation is used in model parameters determination. Since unlabeled data may not always enhance semi-supervised learning [1], we define an effective quality score and use it to select a subset of unlabeled data in the training process. We conduct experiments on the MNIST, SVHN, and CIFAR-10 datasets, and show that the proposed semi-supervised FF-CNN solution outperforms the CNN trained by backpropagation (BP-CNN) when the amount of labeled data is reduced. Furthermore, we develop an ensemble system that combines the output decision vectors of different semi-supervised FF-CNNs to boost classification accuracy. The ensemble systems can achieve further performance gains on all three benchmarking datasets.

Viewing various stereo images under different viewing conditions has escalated the need for efficient and effective depth mapping techniques for adjusting the depths and sizes of objects to match user preference. Existing methods mainly alter the depth of an object through non-uniform region warping, which, however, often cause severe depth or shape distortions, due to improper warping such as local rotations. In this paper, we propose a new object depth mapping scheme based on axis-aligned warping. The proposed axis-aligned-warping based optimization model can simultaneously adjust the depths and sizes of selected objects to their target values without introducing severe shape distortions. Experimental results demonstrate that our method achieves high flexibility and effectiveness in adjusting the size and depth of object compared with existing methods.

Image classification is vulnerable to adversarial attacks. This work investigates the robustness of Saak transform against adversarial attacks towards high performance image classification. We develop a complete image classification system based on multi-stage Saak transform. In the Saak transform domain, clean and adversarial images demonstrate different distributions at different spectral dimensions. Selection of the spectral dimensions at every stage can be viewed as an automatic denoising process. Motivated by this observation, we carefully design strategies of feature extraction, representation and classification that increase adversarial robustness. The performances with well-known datasets and attacks are demonstrated by extensive experimental evaluations.

Kinship verification aims to identify the kin relation between two given face images. It is a very challenging problem due to the lack of training data and facial similarity variations between kinship pairs. In this work, we build a novel appearance and shape based deep learning pipeline. First we adopt the knowledge learned from general face recognition network to learn general facial features. Afterwards, we learn kinship oriented appearance and shape features from kinship pairs and combine them for the final prediction. We have evaluated the model performance on a widely used popular benchmark and demonstrated the superiority over the state-of-the-art.

A hierarchical spatial-spectral correlation (HSSC) method is proposed for texture analysis in this work. The HSSC method first applies a multi-stage spatial-spectral transform to input texture patches, which is known as the Saak transform. Then, it conducts a correlation analysis on Saak transform coefficients to obtain texture features of high discriminant power. To demonstrate the effectiveness of the HSSC method, we conduct extensive experiments on texture classification and show that it offers very competitive results comparing with state-of-the-art methods.

Compressed image quality assessment plays an important role in image services, especially in image compression applications, which can be utilized as a guidance to optimize image processing algorithms. In this paper, we propose an objective image quality assessment algorithm to measure the quality of compressed images. The proposed method utilizes a data-driven transform, Saak (Subspace approximation with augmented kernels), to decompose images into hierarchical structural feature space. We measure the distortions of Saak features and accumulate these distortions according to the feature importance to human visual system. Compared with the state-of-the-art image quality assessment methods on widely utilized datasets, the proposed method correlates better with the subjective results. In addition, the proposed methods achieves more robust results on different datasets.

An interpretable generative model for handwritten digits synthesis is proposed in this work. Modern image generative models such as the variational autoencoder (VAE) are trained by backpropagation (BP). The training process is complex, and its underlying mechanism is not transparent. Here, we present an explainable generative model using a feedforward design methodology without BP. Being similar to VAEs, it has an encoder and a decoder. For the encoder design, we derive principal-component-analysis-based (PCA-based) transform kernels using the covariance of its inputs. This process converts input images of correlated pixels to uncorrelated spectral components, which play the same role as latent variables in a VAE system. For the decoder design, we convert randomly generated spectral components to synthesized images through the inverse PCA transform. A subject test is conducted to compare the quality of digits generated using the proposed method and the VAE method. They offer comparable perceptual quality yet our model can be obtained at much lower complexity.

Language processing becomes more and more important in multimedia processing. Although embedded vector representations of words offer impressive performance on many natural language processing (NLP) applications, the information of ordered input sequences is lost to some extent if only context-based samples are used in the training. For further performance improvement, two new post-processing techniques, called post-processing via variance normalization (PVN) and post-processing via dynamic embedding (PDE), are proposed in this work. The PVN method normalizes the variance of principal components of word vectors, while the PDE method learns orthogonal latent variables from ordered input sequences. The PVN and the PDE methods can be integrated to achieve better performance. We apply these post-processing techniques to several popular word embedding methods to yield their post-processed representations. Extensive experiments are conducted to demonstrate the effectiveness of the proposed post-processing techniques.

The key challenge of generative Visual Dialogue (VD) systems is to respond to human queries with informative answers in natural and contiguous conversation flow. Traditional Maximum Likelihood Estimation-based methods only learn from positive responses but ignore the negative responses, and consequently tend to yield safe or generic responses. To address this issue, we propose a novel training scheme in conjunction with weighted likelihood estimation method. Furthermore, an adaptive multi-modal reasoning module is designed, to accommodate various dialogue scenarios automatically and select relevant information accordingly. The experimental results on the VisDial benchmark demonstrate the superiority of our proposed algorithm over other state-of-the-art approaches, with an improvement of 5.81% on recall@10.

Much work in robotics has focused on “humanin-the-loop” learning techniques that improve the efficiency of the learning process. However, these algorithms have made the strong assumption of a cooperating human supervisor that assists the robot. In reality, human observers tend to also act in an adversarial manner towards deployed robotic systems. We show that this can in fact improve the robustness of the learned models by proposing a physical framework that leverages perturbations applied by a human adversary, guiding the robot towards more robust models. In a manipulation task, we show that grasping success improves significantly when the robot trains with a human adversary as compared to training in a self-supervised manner.

Mobile devices such as smart phones are ubiquitously being used to take photos and videos, thus increasing the importance of image deblurring. This study introduces a novel deep learning approach that can automatically and progressively achieve the task via adversarial blurred region mining and refining (adversarial BRMR). Starting with a collaborative mechanism of two coupled conditional generative adversarial networks (CGANs), our method first learns the image-scale CGAN, denoted as iGAN, to globally generate a deblurred image and locally uncover its still blurred regions through an adversarial mining process. Then, we construct the patch-scale CGAN, denoted as pGAN, to further improve sharpness of the most blurred region in each iteration. Owing to such complementary designs, the adversarial BRMR indeed functions as a bridge between iGAN and pGAN, and yields the performance synergy in better solving blind image deblurring. The overall formulation is self-explanatory and effective to globally and locally restore an underlying sharp image. Experimental results on benchmark datasets demonstrate that the proposed method outperforms the current state-of-the-art technique for blind image deblurring both quantitatively and qualitatively.

Domain Adaptation (DA) approaches achieved significant improvements in a wide range of machine learning and computer vision tasks (i.e., classification, detection, and segmentation). However, as far as we are aware, there are few methods yet to achieve domain adaptation directly on 3D point cloud data. The unique challenge of point cloud data lies in its abundant spatial geometric information, and the semantics of the whole object is contributed by including regional geometric structures. Specifically, most general-purpose DA methods that struggle for global feature alignment and ignore local geometric information are not suitable for 3D domain alignment. In this paper, we propose a novel 3D Domain Adaptation Network for point cloud data (PointDAN). PointDAN jointly aligns the global and local features in multi-level. For local alignment, we propose Self-Adaptive (SA) node module with an adjusted receptive field to model the discriminative local structures for aligning domains. To represent hierarchically scaled features, node-attention module is further introduced to weight the relationship of SA nodes across objects and domains. For global alignment, an adversarial-training strategy is employed to learn and align global features across domains. Since there is no common evaluation benchmark for 3D point cloud DA scenario, we build a general benchmark (i.e., PointDA-10) extracted from three popular 3D object/scene datasets (i.e., ModelNet, ShapeNet and ScanNet) for cross-domain 3D objects classification fashion. Extensive experiments on PointDA-10 illustrate the superiority of our model over the state-of-the-art general-purpose DA methods.

Image translation between two domains is a class of problems aiming to learn mapping from an input image in the source domain to an output image in the target domain. It has been applied to numerous applications, such as data augmentation, domain adaptation, and unsupervised training. When paired training data is not accessible, image translation becomes an ill-posed problem. We constrain the problem with the assumption that the translated image needs to be perceptually similar to the original image and also appears to be drawn from the new domain, and propose a simple yet effective image translation model consisting of a single generator trained with a self-regularization term and an adversarial term. We further notice that the existing image translation techniques are agnostic to the subjects of interest and often introduce unwanted changes or artifacts to the input. Thus, we propose to add an attention module to predict an attention map to guide the image translation process. The module learns to attend to key parts of the image while keeping everything else unaltered, essentially avoiding undesired artifacts or changes. Extensive experiments and evaluations show that our model while being simpler, achieves significantly better performance than existing image translation methods.

To improve the parallel processing capability of video coding, the emerging high efficiency video coding (HEVC) standard introduces two parallel techniques, i.e., Wavefront Parallel Processing (WPP) andTiles, to make it much more parallel-friendly than its predecessors. However, these two techniques are designed to explore coarse-grained parallelism in HEVC encoding on multicore Central Processing Unit (CPU) platforms. As the computing architecture undergoes a trend toward heterogeneity in the last decade, multi-grained parallel computing methods can be designed to accelerate HEVC encoding on heterogeneous systems. In this paper, a multi-grained parallel solution (MPS) is proposed to optimize HEVC encoding on a typical heterogeneous platform. A massively parallel motion estimation algorithm is employed by MPS to parallelize part of HEVC encoding on Graphic Processing Unit (GPU). Meanwhile, several other HEVC encoding modules are accelerated on CPU through the cooperation of WPP and an adaptive parallel mode decision algorithm. The parallelism between CPU and GPU is well designed and implemented to guarantee an efficient concurrent execution of HEVC encoding on multi-grained parallel levels. The effectiveness of the proposed MPS for HEVC encoding is verified on a number of experiments.

As compared with traditional video retargeting, stereo video retargeting poses new challenges because stereo video contains the depth information of salient objects and its time dynamics. In this work, we propose a depth-aware stereo video retargeting method by imposing the depth fidelity constraint. The proposed depth-aware retargeting method reconstructs the 3D scene to obtain the depth information of salient objects. We cast it as a constrained optimization problem, where the total cost function includes the shape, temporal and depth distortions of salient objects. As a result, the solution can preserve the shape, temporal and depth fidelity of salient objects simultaneously. It is demonstrated by experimental results that the depth-aware retargeting method achieves higher retargeting quality and provides better user experience.

We propose a method for unsupervised video object segmentation by transferring the knowledge encapsulated in image-based instance embedding networks. The instance embedding network produces an embedding vector for each pixel that enables identifying all pixels belonging to the same object. Though trained on static images, the instance embeddings are stable over consecutive video frames, which allows us to link objects together over time. Thus, we adapt the instance networks trained on static images to video object segmentation and incorporate the embeddings with objectness and optical flow features, without model retraining or online fine-tuning. The proposed method outperforms state-of-the-art unsupervised segmentation methods in the DAVIS dataset and the FBMS dataset.

A large-scale video quality dataset called the VideoSet has been constructed recently to measure human subjective experience of H.264 coded video in terms of the just-noticeable-difference (JND). It measures the first three JND points of 5-second video of resolution 1080p, 720p, 540p and 360p. Based on the VideoSet, we propose a method to predict the satisfied-user-ratio (SUR) curves using a machine learning framework. First, we partition a video clip into local spatial-temporal segments and evaluate the quality of each segment using the VMAF quality index. Then, we aggregate these local VMAF measures to derive a global one. Finally, the masking effect is incorporated and the support vector regression (SVR) is used to predict the SUR curves, from which the JND points can be derived. Experimental results are given to demonstrate the performance of the proposed SUR prediction method.

A domain adaptation method for urban scene segmentation is proposed in this work. We develop a fully convolutional tri-branch network, where two branches assign pseudo labels to images in the unlabeled target domain while the third branch is trained with supervision based on images in the pseudo-labeled target domain. The re-labeling and re-training processes alternate. With this design, the tri-branch network learns target-specific discriminative representations progressively and, as a result, the cross-domain capability of the segmenter improves. We evaluate the proposed network on large-scale domain adaptation experiments using both synthetic (GTA) and real (Cityscapes) images. It is shown that our solution achieves the state-of-the-art performance and it outperforms previous methods by a significant margin.

We propose a method to extract representative features for fashion analysis by utilizing weakly annotated online fashion images in this work. The proposed system consists of two stages. In the first stage, we attempt to detect clothing items in a fashion image: the top clothes (t), bottom clothes (b) and one-pieces (o). In the second stage, we extract discriminative features from detected regions for various applications of interest. Unlike previous work that heavily relies on well-annotated fashion data, we propose a way to collect fashion images from online resources and conduct automatic annotation on them. Based on this methodology, we create a new fashion dataset, called the Web Attributes, to train our feature extractor. It is shown by experiments that extracted regional features can capture local characteristics of fashion images well and offer better performance than previous works.

Face detection is a widely studied problem over the past few decades. Recently, significant improvements have been achieved via the deep neural network, however, it is still challenging to directly apply these techniques to mobile devices for its limited computational power and memory. In this work, we present a proposal generation acceleration framework for real-time face detection. More specifically, we adopt a popular cascaded convolutional neural network (CNN) as the basis, then apply our acceleration approach on the basic framework to speed up the model inference time. We are motivated by the observation that the computation bottleneck of this framework arises from the proposal generation stage, where each level of the dense image pyramid has to go through the network. In this work, we reduce the number of image pyramid levels by utilizing both global and local facial characteristics (i.e., global face and facial parts). Experimental results on public benchmarks WIDER-face and FDDB demonstrate the satisfactory performance and faster speed compared to the state-of-the-arts.

One fundamental problem of the convolutional neural network (CNN) is catastrophic forgetting, which occurs when new object classes and data are added while the original dataset is not available any more. Training the network only using the new dataset deteriorates the performance with respect to the old dataset. To overcome this problem, we propose an expanded network architecture, called the ExpandNet, to enhance the CNN incremental learning capability. Our solution keeps filters of the original networks on one hand, yet adds additional filters to the convolutional layers as well as the fully connected layers on the other hand. The proposed new architecture does not need any information of the original dataset, and it is trained using the new dataset only. Extensive evaluations based on the CIFAR -10 and the CIFAR -100 datasets show that the proposed method has a slower forgetting rate as compared to several existing incremental learning networks.

A novel text data dimension reduction technique, called the tree-structured multi-linear principal component analysis (TMPCA), is proposed in this work. Being different from traditional text dimension reduction methods that deal with the word-level representation, the TMPCA technique reduces the dimension of input sequences and sentences to simplify the following text classification tasks. It is shown mathematically and experimentally that the TMPCA tool demands much lower complexity (and, hence, less computing power) than the ordinary principal component analysis (PCA). Furthermore, it is demonstrated by experimental results that the support vector machine (SVM) method applied to the TMPCA-processed data achieves commensurable or better performance than the state-of-the-art recurrent neural network (RNN) approach.

Stereo video retargeting aims to avoid shape and depth distortions while maintaining temporal coherence of shape and depth while resizing a stereo video to a desired size. Existing methods resort to extending stereo image retargeting schemes to stereo video retargeting by imposing temporal constraints to consistently resize all corresponding regions so as to maintain temporal coherence. However, such a direct extension often incurs conflicts among the requirements for preserving shape information and depth information and maintaining their temporal coherence, thereby failing to meet one or more of these requirements. We find that properly relaxing temporal constraints for non-paired regions at frame boundaries can effectively mitigate conflicts among depth, shape, and temporal constraints without severely degrading temporal coherence perceptually. Based on this new finding, we derive effective temporal constraints to improve the viewing experience of a 3D scene for stereo video retargeting. Accordingly, we propose an efficient grid-based implementation for our method. Experimental results show that our method achieves superior visual quality over existing methods.

An efficient, scalable and robust approach to the handwritten digits recognition problem based on the Saak transform is proposed in this work. First, multi-stage Saak transforms are used to extract a family of joint spatial-spectral representations of input images. Then, the Saak coefficients are used as features and fed into the SVM classifier for the classification task. In order to control the size of Saak coefficients, we adopt a lossy Saak transform that uses the principal component analysis (PCA) to select a smaller set of transform kernels. The handwritten digits recognition problem is well solved by the convolutional neural network (CNN) such as the LeNet-5. We conduct a comparative study on the performance of the LeNet-5 and the Saak-transform-based solutions in terms of scalability and robustness as well as the efficiency of lossless and lossy Saak transforms under a comparable accuracy level.

The just-noticeable-difference (JND) visual perception property has received much attention in characterizing human subjective viewing experience of compressed video. In this work, we quantity the JND-based video quality assessment model using the satisfied user ratio (SUR) curve, and show that the SUR model can be greatly simplified since the JND points of multiple subjects for the same content in the VideoSet can be well modeled by the normal distribution. Then, we design an SUR prediction method with video quality degradation features and masking features and use them to predict the first, second and the third JND points and their corresponding SUR curves. Finally, we verify the performance of the proposed SUR prediction method with different configurations on the VideoSet. The experimental results demonstrate that the proposed SUR prediction method achieves good performance in various resolutions with the mean absolute error (MAE) of the SUR smaller than 0.05 on average.

A novel graph-to-tree conversion mechanism called the deep-tree generation (DTG) algorithm is first proposed to predict text data represented by graphs. The DTG method can generate a richer and more accurate representation for nodes (or vertices) in graphs. It adds flexibility in exploring the vertex neighborhood information to better reflect the second order proximity and homophily equivalence in a graph. Then, a Deep-Tree Recursive Neural Network (DTRNN) method is presented and used to classify vertices that contains text data in graphs. To demonstrate the effectiveness of the DTRNN method, we apply it to three real-world graph datasets and show that the DTRNN method outperforms several state-of-the-art benchmarking methods.

With the development of Fully Convolutional Neural Network (FCN), there have been progressive advances in the field of semantic segmentation in recent years. The FCN-based solutions are able to summarize features across training images and generate matching templates for the desired object classes, yet they overlook intra-class difference (ICD) among multiple instances in the same class. In this work, we present a novel fine-to-coarse learning (FCL) procedure, which first guides the network with designed 'finer' sub-class labels, whose decisions are mapped to the original 'coarse' object category through end-to-end learning. A sub-class labeling strategy is designed with unsupervised clustering upon deep convolutional features, and the proposed FCL procedure enables a balance between the fine-scale (i.e. sub-class) and the coarse-scale (i.e. class) knowledge. We conduct extensive experiments on several popular datasets, including PASCAL VOC, Context, Person-Part and NYUDepth-v2 to demonstrate the advantage of learning finer sub-classes and the potential to guide the learning of deep networks with unsupervised clustering.

As the number of object categories continues to increase in object classification, it is inevitable to have certain categories that are more confusing than others due to the proximity of their samples in the feature space. In this work, we conduct a detail analysis on confusing categories and propose a confusing categories identification and resolution (CCIR) scheme, which can be applied to any CNNbased object classification baseline method to further improve its performance. In the CCIR scheme, we first present a procedure to cluster confusing object categories together to form a confusion set automatically. Then, a binary-treestructured (BTS) clustering method is adopted to split a confusion set into multiple subsets. A classifier is subsequently learned within each subset to enhance its performance. Experimental results on the ImageNet ILSVRC2012 dataset show that the proposed CCIR scheme can offer a significant performance gain over the AlexNet and the VGG16.

segmentation, cardiac model construction and finite element software is available as part of the SimVascular open source project at www.simvascular.org. The registration toolkit is available at https://sites.google.

A contour-like artifact called false contour is often observed in large smooth areas of decoded images and video. Without loss of generality, we focus on detection and removal of false contours resulting from the state-of-the-art High Efficiency Video Coding codec. First, we identify the cause of false contours by explaining the human perceptual experiences on them with specific experiments. Next, we propose a precise pixel-based false contour detection method based on the evolution of a false contour candidate (FCC) map. The number of points in the FCC map becomes fewer by imposing more constraints step by step. Special attention is paid to separating false contours from real contours such as edges and textures in the video source. Then, a decontour method is designed to remove false contours in the exact contour position while preserving edge/texture details. Extensive experimental results are provided to demonstrate the superior performance of the proposed false contour detection and removal method in both compressed images and videos.

In order to meet the high computational demand to achieve superior coding efficiency and to explore the parallelism of parallel processing architectures, the emerging high efficiency video coding (HEVC) standard has been designed to be more parallelizable than previous video coding standards. However, it is still desirable to design an efficient parallel HEVC encoder to fully exploit the parallelism of the increasingly powerful multicore platforms, especially when considering the amount of parallelism, the scalability of parallelization, and the coding efficiency. In this work, a performance model of HEVC encoding is first introduced to investigate the speedup and the limitations of the technique of wavefront parallel processing (WPP) under various conditions. Then, a collaborative scheduling-based parallel solution (CSPS) for HEVC encoding is proposed, which includes adaptive parallel mode decision, asynchronous frame-level pixel interpolation, and multigrained task scheduling. The goal of the proposed CSPS is to defeat the disadvantages of WPP and further improve the parallelization of HEVC encoding on multicore platforms. Extensive experimental results demonstrate the efficiency of the proposed CSPS for parallelizing HEVC encoding as the computing resources of multicore architectures can be fully utilized.

Evaluation of coding efficiency is traditionally modeled as a continuous rate-distortion (R-D) function, where the peak signal-to-noise ratio (PSNR) is adopted as the quality measure. Although the PSNR-versus-bitrate curve offers some useful tradeoff information between video quality and coding bit-rates, it does not take human perceptual experience into account. In this work, by following the recent image/video quality assessment framework based on the just-noticeable-difference (JND) notion, we conduct a subjective test for HEVC (High Efficiency Video Codec) video to measure the QP value that lies in the boundary of perceptually lossless and lossy coded bit streams for each human subject. This is also known as the first JND point. It is observed that the statistics of the first JND points of 30 subjects follows the normal distribution for a great majority of test sequences. Finally, a machine-learning approach is proposed to predict the mean of the group-based JND distribution based on extracted video features. It is shown by experimental results that the mean JND point can be predicted accurately.

In this paper, we propose the use of recurrent neural networks (RNNs) to develop an effective solution to two problems in electrocardiogram (ECG)-based biometrics: identification/classification and authentication. Different RNN architectures with various parameter settings were evaluated, including traditional, long short-term memory (LSTM), gated recurrent unit (GRU), unidirectional, and bidirectional networks. Unlike many existing methods, the RNN-based method does not require any feature extraction. The method was evaluated using two publicly available datasets: ECG-ID and MIT-BIH Arrhythmia (MITDB). For the identification problem, nearly 100% classification accuracy on the ECG-ID dataset was achieved, and similar results were observed for the MITDB dataset. For the authentication problem, an RNN was trained and the hidden state at the final time step was extracted to make a decision. We evaluated the effect of the training size on the equal error rate (EER), and showed that the EER drops from approximately 3.5% to nearly 0% as we increased the percentage of subjects used for training from approximately 15% to 80%.

We target at solving the problem of automatic image segmentation. Although 1D contour and 2D surface cues have been widely utilized in existing work, 3D depth information of an image, a necessary cue according to human visual perception, is however overlooked in automatic image segmentation. In this paper, we study how to fully utilize 1D contour, 2D surface, and 3D depth cues for image segmentation. First, three elementary segmentation modules are developed for these cues respectively. The proposed 3D depth cue is able to segment different textured regions even with similar color, and also merge similar textured areas, which cannot be achieved using state-of-the-art approaches. Then, a content-dependent spectral (CDS) graph is proposed for layered affinity models to produce the final segmentation. CDS is designed to build a more reliable relationship between neighboring surface nodes based on the three elementary cues in the spectral graph. Extensive experiments not only show the superior performance of the proposed algorithm over state-of-the-art approaches, but also verify the necessities of these three cues in image segmentation.

Object proposal generation has been an important preprocessing step for object detectors in general and the convolutional neural network (CNN) detectors in particular. Recently, people start to use the CNN to generate object proposals but most of these methods suffer from the localization bias problem, like other objectness-based methods. Since contours offer a powerful cue for accurate localization, we propose a box refinement method by searching for the optimal contour for each initial bounding box that minimizes the contour cost. Experiments on the PASCAL VOC2007 test dataset show that our box refinement method can significantly improve the object recall at a high overlapping threshold while maintaining a similar recall at a loose one. Given 1000 proposals, the average recall of multiple existing methods is increased by more than 5% with our box refinement process integrated.

A boosted convolutional neural network (BCNN) system is proposed to enhance the pedestrian detection performance in this work. Being inspired by the classic boosting idea, we develop a weighted loss function that emphasizes challenging samples in training a convolutional neural network (CNN). Two types of samples are considered challenging: 1) samples with detection scores falling in the decision boundary, and 2) temporally associated samples with inconsistent scores. A weighting scheme is designed for each of them. Finally, we train a boosted fusion layer to benefit from the integration of these two weighting schemes. We use the Fast-RCNN as the baseline, and test the corresponding BCNN on the Caltech pedestrian dataset in the experiment, and show a significant performance gain of the BCNN over its baseline.

• A large-scale JND-based coded video quality dataset is presented. • The VideoSet contains 220 5-s sequences in four resolutions coded by H.264/AVC. • The subjective test procedure, JND data cleaning and properties are described. • The significance and implications of the VideoSet are discussed. • This work points out a clear path to data-driven perceptual coding. A new methodology to measure coded image/video quality using the just-noticeable-difference (JND) idea was proposed in Lin et al. (2015). Several small JND-based image/video quality datasets were released by the Media Communications Lab at the University of Southern California in Jin et al. (2016) and Wang et al. (2016) [3]. In this work, we present an effort to build a large-scale JND-based coded video quality dataset. The dataset consists of 220 5-s sequences in four resolutions (i.e., 1920 × 1080 , 1280 × 720 , 960 × 540 and 640 × 360 ). For each of the 880 video clips, we encode it using the H.264/AVC codec with QP = 1 , … , 51 and measure the first three JND points with 30 + subjects. The dataset is called the “VideoSet”, which is an acronym for “Video Subject Evaluation Test (SET)”. This work describes the subjective test procedure, detection and removal of outlying measured data, and the properties of collected JND data. Finally, the significance and implications of the VideoSet to future video coding research and standardization efforts are pointed out. All source/coded video clips as well as measured JND data included in the VideoSet are available to the public in the IEEE DataPort (Wang et al., 2016 [4]).

Textual data such as tags, sentence descriptions are combined with visual cues to reduce the semantic gap for image retrieval applications in today's Multimodal Image Retrieval (MIR) systems. However, all tags are treated as equally important in these systems, which may result in misalignment between visual and textual modalities during MIR training. This will further lead to degenerated retrieval performance at query time. To address this issue, we investigate the problem of tag importance prediction, where the goal is to automatically predict the tag importance and use it in image retrieval. To achieve this, we first propose a method to measure the relative importance of object and scene tags from image sentence descriptions. Using this as the ground truth, we present a tag importance prediction model to jointly exploit visual, semantic and context cues. The Structural Support Vector Machine (SSVM) formulation is adopted to ensure efficient training of the prediction model. Then, the Canonical Correlation Analysis (CCA) is employed to learn the relation between the image visual feature and tag importance to obtain robust retrieval performance. Experimental results on three real-world datasets show a significant performance improvement of the proposed MIR with Tag Importance Prediction (MIR/TIP) system over other MIR systems.

Blood flow and mechanical forces in the ventricle are implicated in cardiac development and trabeculation. However, the mechanisms of mechanotransduction remain elusive. This is due in part to the challenges associated with accurately quantifying mechanical forces in the developing heart. We present a novel computational framework to simulate cardiac hemodynamics in developing zebrafish embryos by coupling 4-D light sheet imaging with a stabilized finite element flow solver, and extract time-dependent mechanical stimuli data. We employ deformable image registration methods to segment the motion of the ventricle from high resolution 4-D light sheet image data. This results in a robust and efficient workflow, as segmentation need only be performed at one cardiac phase, while wall position in the other cardiac phases is found by image registration. Ventricular hemodynamics are then quantified by numerically solving the Navier-Stokes equations in the moving wall domain with our validated flow solver. We demonstrate the applicability of the workflow in wild type zebrafish and three treated fish types that disrupt trabeculation: (a) chemical treatment using AG1478, an ErbB2 signaling inhibitor that inhibits proliferation and differentiation of cardiac trabeculation; (b) injection of gata1a morpholino oligomer (gata1aMO) suppressing hematopoiesis and resulting in attenuated trabeculation; (c) weak-atriumm58 mutant (wea) with inhibited atrial contraction leading to a highly undeveloped ventricle and poor cardiac function. Our simulations reveal elevated wall shear stress (WSS) in wild type and AG1478 compared to gata1aMO and wea. High oscillatory shear index (OSI) in the grooves between trabeculae, compared to lower values on the ridges, in the wild type suggest oscillatory forces as a possible regulatory mechanism of cardiac trabeculation development. The framework has broad applicability for future cardiac developmental studies focused on quantitatively investigating the role of hemodynamic forces and mechanotransduction during morphogenesis.

Steganalysis of the quantization index modulation (QIM) steganography in a low-bit-rate encoded speech stream is conducted in this research. According to the speech generation theory and the phoneme distribution properties in language, we first point out that the correlation characteristics of split vector quantization (VQ) codewords of linear predictive coding filter coefficients are changed after the QIM steganography. Based on this observation, we construct a model called the Quantization codeword correlation network (QCCN) based on split VQ codeword from adjacent speech frames. Furthermore, the QCCN model is pruned to yield a stronger correlation network. After quantifying the correlation characteristics of vertices in the pruned correlation network, we obtain feature vectors that are sensitive to steganalysis. Finally, we build a high-performance detector using the support vector machine (SVM) classifier. It is shown by experimental results that the proposed QCCN steganalysis method can effectively detect the QIM steganography in encoded speech stream when it is applied to low-bit-rate speech codec such as G.723.1 and G.729.

Object quality assessment for compressed video is critical to various video compression systems that are essential in the video delivery and storage. Although mean squared error (MSE) is computationally simple, it may not be accurate to reflect the perceptual quality of compressed videos, which are also affected dramatically by the characteristics of the human visual system (HVS), such as contrast sensitivity, visual attention, and masking effect. In this paper, a video quality metric is proposed based on perceptually weighted MSE. A low-pass filter is designed to model the contrast sensitivity of the HVS with the consideration of visual attention. The imperceptible distortion is adaptively removed in the salient and nonsalient regions. To quantitatively measure the masking effect, the randomness of video content is proposed in both the spatial and temporal domains. Since the masking effect highly depends on the regularity of structure and motion in the spatial and temporal directions, the video signal is modeled as a linear dynamic system, and the prediction error of future frames from previous frames is used as randomness to measure the significance of masking. The relation is investigated between MSE and perceptual quality scores across various contents, and a masking modulation model is proposed to compensate the impact of the masking effect on the MSE. The performance of the proposed quality metric is validated on three video databases with various compression distortions. The experimental results demonstrate that the proposed algorithm outperforms other benchmark quality metrics.

Image quality assessment (IQA) is a fundamental yet constantly developing task for computer vision and image processing. Most IQA evaluation mechanisms are based on the pertinence of subjective and objective estimation. Each image distortion type has its own property correlated with human perception. However, this intrinsic property may not be fully exploited by existing IQA methods. In this paper, we make two main contributions to the IQA field. First, a novel IQA method is developed based on a local linear model that examines the distortion between the reference and the distorted images for better alignment with human visual experience. Second, a distortion-specific compensation strategy is proposed to offset the negative effect on IQA modeling caused by different image distortion types. These score offsets are learned from several known distortion types. Furthermore, for an image with an unknown distortion type, a convolutional neural network-based method is proposed to compute the score offset automatically. Finally, an integrated IQA metric is proposed by combining the aforementioned two ideas. Extensive experiments are performed to verify the proposed IQA metric, which demonstrate that the local linear model is useful in human perception modeling, especially for individual image distortion, and the overall IQA method outperforms several state-of-the-art IQA approaches.

The just noticeable difference (JND) in an image, which reveals the visibility limitation of the human visual system (HVS), is widely used for visual redundancy estimation in signal processing. To determine the JND threshold with the current schemes, the spatial masking effect is estimated as the contrast masking, and this cannot accurately account for the complicated interaction among visual contents. Research on cognitive science indicates that the HVS is highly adapted to extract the repeated patterns for visual content representation. Inspired by this, we formulate the pattern complexity as another factor to determine the total masking effect: the interaction is relatively straightforward with a limited masking effect in a regular pattern, and is complicated with a strong masking effect in an irregular pattern. From the orientation selectivity mechanism in the primary visual cortex, the response of each local receptive field can be considered as a pattern; therefore, in this paper, the orientation that each pixel presents is regarded as the fundamental element of a pattern, and the pattern complexity is calculated as the diversity of the orientation in a local region. Finally, considering both pattern complexity and luminance contrast, a novel spatial masking estimation function is deduced, and an improved JND estimation model is built. Experimental results on comparing with the latest JND models demonstrate the effectiveness of the proposed model, which performs highly consistent with the human perception. The source code of the proposed model is publicly available at http://web.xidian.edu.cn/wjj/en/index.html.

An ensemble method for full-reference image quality assessment (IQA) based on the parallel boosting (ParaBoost) idea is proposed in this paper. We first extract features from existing image quality metrics and train them to form basic image quality scorers (BIQSs). Then, we select additional features to address specific distortion types and train them to construct auxiliary image quality scorers (AIQSs). Both BIQSs and AIQSs are trained on small image subsets of certain distortion types and, as a result, they are weak performers with respect to a wide variety of distortions. Finally, we adopt the ParaBoost framework, which is a statistical scorer selection scheme for support vector regression (SVR), to fuse the scores of BIQSs and AIQSs to evaluate the images containing a wide range of distortion types. This ParaBoost methodology can be easily extended to images of new distortion types. Extensive experiments are conducted to demonstrate the superior performance of the ParaBoost method, which outperforms existing IQA methods by a significant margin. Specifically, the Spearman rank order correlation coefficients (SROCCs) of the ParaBoost method with respect to the LIVE, CSIQ, TID2008, and TID2013 image quality databases are 0.98, 0.97, 0.98, and 0.96, respectively.

In wireless sensor networks, data collection (or gathering) is the task of transmitting rounds of measurements of physical phenomena from the sensor nodes to a sink node. We study how to increase the efficiency of data collection via supervised in-network classification of rounds of measurements. We assume that the end users of the data are interested only in rounds characterized by certain patterns. Hence the wireless sensor network uses classification to select the rounds of measurements that are transmitted to the base station. The energy consumption is potentially reduced by avoiding the transmission of rounds of measurements that are not of interest to the end users. In-network classification requires distributed feature extraction and transmission. Such tasks can be less or more energy expensive than the transmission of measurements without classification. We provide analytical results and simulations on real data to show requirements and key trade-offs for the design of in-network data classification systems that can improve the collection efficiency. Besides, we study the impact of spatial subsampling of the sensor data (a way to further decrease energy consumption) on the classification performance.

Based on the notion of just noticeable differences (JND), a stair quality function (SQF) was recently proposed to model human perception on JPEG images. Furthermore, a k-means clustering algorithm was adopted to aggregate JND data collected from multiple subjects to generate a single SQF. In this work, we propose a new method to derive the SQF using the Gaussian Mixture Model (GMM). The newly derived SQF can be interpreted as a way to characterize the mean viewer experience. Furthermore, it has a lower information criterion (BIC) value than the previous one, indicating that it offers a better model. A specific example is given to demonstrate the advantages of the new approach.

Just noticeable difference (JND) reveals the visibility of our human visual system (HVS), below which changes cannot be perceived by the human. Though dozens of JND estimation models have been introduced during the past decade, how to accurately estimate the JND thresholds for different content regions (e.g., edge and texture region) is still an open problem. Research on cognitive science indicates that the HVS is adaptive to extract the visual regularities from an input scene for content perception and understanding. Thus, we analyze the effect of content regularity on visual sensitivity, and suggest that the visual regularity is another important factor that determines the JND threshold. According to the orientation distributions of local regions, the content regularities are firstly calculated. Then, by considering the effect from content regularity, luminance adaptation, and contrast masking, a novel JND model is proposed. Experimental results demonstrate that the proposed model can effectively estimate the JND thresholds of regions with different visual contents.

A compressed video quality assessment dataset based on the just noticeable difference (JND) model, called MCL-JCV, is recently constructed and released. In this work, we explain its design objectives, selected video content and subject test procedures. Then, we conduct statistical analysis on collected JND data. We compute the difference between every two adjacent JND points and propose an outlier detection algorithm to remove unreliable data. We also show that each JND difference group can be well approximated by a normal distribution so that we can adopt the Gaussian mixture model (GMM) to characterize the distribution of multiple JND points. Finally, it is demonstrated by experimental results that the proposed JND analysis performed in the difference domain, called the D-method, achieves a lower BIC (Bayesian information criteria) value than the previously proposed G-method.

A two stages car detection method using deformable part models with composite feature sets (DPM/CF) is proposed to recognize cars of various types and from multiple viewing angles. In the first stage, a HOG template is matched to detect the bounding box of the entire car of a certain type and viewed from a certain angle (called a t/a pair), which yields a region of interest (ROI). In the second stage, various part detectors using either HOG or the convolution neural network (CNN) features are applied to the ROI for validation. An optimization procedure based on latent logistic regression is adopted to select the optimal part detector's location, window size, and feature to use. Extensive experimental results indicate the proposed DPM/CF system can strike a balance between detection accuracy and training complexity.

A large number of saliency models, each based on a different hypothesis, have been proposed over the past 20 years. In practice, while subscribing to one hypothesis or computational principle makes a model that performs well on some types of images, it hinders the general performance of a model on arbitrary images and large-scale data sets. One natural approach to improve overall saliency detection accuracy would then be fusing different types of models. In this paper, inspired by the success of late-fusion strategies in semantic analysis and multi-modal biometrics, we propose to fuse the state-of-the-art saliency models at the score level in a para-boosting learning fashion. First, saliency maps generated by several models are used as confidence scores. Then, these scores are fed into our para-boosting learner (i.e., support vector machine, adaptive boosting, or probability density estimator) to generate the final saliency map. In order to explore the strength of para-boosting learners, traditional transformation-based fusion strategies, such as Sum, Min, and Max, are also explored and compared in this paper. To further reduce the computation cost of fusing too many models, only a few of them are considered in the next step. Experimental results show that score-level fusion outperforms each individual model and can further reduce the performance gap between the current models and the human inter-observer model.

Effective feature selection plays a vital role in anterior segment imaging for determining the mechanism involved in angle-closure glaucoma (ACG) diagnosis. This research focuses on the use of redundant features for complex disease diagnosis such as ACG using anterior segment optical coherence tomography images. Both supervised [minimum redundancy maximum relevance (MRMR)] and unsupervised [Laplacian score (L-score)] feature selection algorithms have been cross-examined with different ACG mechanisms. An AdaBoost machine learning classifier is then used for classifying the five various classes of ACG mechanism such as iris roll, lens, pupil block, plateau iris, and no mechanism using both feature selection methods. The overall accuracy has shown that the usefulness of redundant features by L-score method in improved ACG diagnosis compared to minimum redundant features by MRMR method.

This paper proposes secrecy-enhancing signaling schemes to exploit the advantages of having only channel state information (CSI) at the transmitter, but no (or limited) CSI at the receiver and the eavesdropper in fast-fading wiretap channels and also in wiretap channels with Gaussian interference. With only CSI at the transmitter, the transmit signaling can be designed to precompensate for the distortion or the interference on the main channel to facilitate decoding at the receiver while leaving the eavesdropper confused by the uncertainties of its own channel. In fading wiretap channels, truncated channel-inversion and phase-compensation schemes are proposed to achieve this task. Here, truncation is used so that transmission occurs only when the main channel is sufficiently reliable. In wiretap channels with Gaussian interference, a truncated interference-nulling scheme is proposed to enable interference precancellation at the transmitter. The proposed signaling schemes exploit the advantages of having only CSI at the transmitter without employing more complex encoding schemes. The achievable secrecy rates are derived using Gaussian input, and approximate expressions are proposed for the optimization of system parameters. The effectiveness of the proposed transmission schemes and the advantages of having only CSI at the transmitter are demonstrated through numerical simulations.

The contour-guided color palette (CCP) is proposed for robust image segmentation. It efficiently integrates contour and color cues of an image. To find representative colors of an image, color samples along long contours between regions, similar in spirit to machine learning methodology that focus on samples near decision boundaries, are collected followed by the mean-shift (MS) algorithm in the sampled color space to achieve an image-dependent color palette. This color palette provides a preliminary segmentation in the spatial domain, which is further fine-tuned by post-processing techniques such as leakage avoidance, fake boundary removal, and small region mergence. Segmentation performances of CCP and MS are compared and analyzed. While CCP offers an acceptable standalone segmentation result, it can be further integrated into the framework of layered spectral segmentation to produce a more robust segmentation. The superior performance of CCP-based segmentation algorithm is demonstrated by experiments on the Berkeley Segmentation Dataset.

The Netflix ingest and encoding pipeline is a cloud-based platform that generates video encodes for the Netflix streaming service. Due to the large throughput of the system, automated video quality assessment of the source videos and the generated encodes is essential in ensuring the quality of experience of viewers. This paper discusses the motivations for integrating video quality assessment in the production pipeline, outlines currently deployed solutions and presents the technical challenges in improving the system.

Abstract Motivation: The deconvolution of isoform expression from RNA-seq remains challenging because of non-uniform read sampling and subtle differences among isoforms. Results: We present a weighted-log-likelihood expectation maximization method on isoform quantification (WemIQ). WemIQ integrates an effective bias removal with a weighted expectation maximization (EM) algorithm to distribute reads among isoforms efficiently. The weight represents the oversampling or undersampling of sequence reads and is estimated through a generalized Poisson model without any presumption on the bias sources and formats. WemIQ significantly improves the quantification of isoform and gene expression as well as the derived exon inclusion rates. It provides robust expression estimates across different laboratories and protocols, which is valuable for the integrative analysis of RNA-seq. For the recent single-cell RNA-seq data, WemIQ also provides the opportunity to distinguish bias heterogeneity from true biological heterogeneity and uncovers smaller cell-to-cell expression variability. Availability and implementation: WemIQ can be downloaded from http://www-rcf.usc.edu/~liangche/software.html Contact: [email protected] Supplementary information: Supplementary data are available at Bioinformatics online.

This paper describes a recently created image database, TID2013, intended for evaluation of full-reference visual quality assessment metrics. With respect to TID2008, the new database contains a larger number (3000) of test images obtained from 25 reference images, 24 types of distortions for each reference image, and 5 levels for each type of distortion. Motivations for introducing 7 new types of distortions and one additional level of distortions are given; examples of distorted images are presented. Mean opinion scores (MOS) for the new database have been collected by performing 985 subjective experiments with volunteers (observers) from five countries (Finland, France, Italy, Ukraine, and USA). The availability of MOS allows the use of the designed database as a fundamental tool for assessing the effectiveness of visual quality. Furthermore, existing visual quality metrics have been tested with the proposed database and the collected results have been analyzed using rank order correlation coefficients between MOS and considered metrics. These correlation indices have been obtained both considering the full set of distorted images and specific image subsets, for highlighting advantages and drawbacks of existing, state of the art, quality metrics. Approaches to thorough performance analysis for a given metric are presented to detect practical situations or distortion types for which this metric is not adequate enough to human perception. The created image database and the collected MOS values are freely available for downloading and utilization for scientific purposes.

A non-collaborative coexistence mechanism for wireless-fidelity (Wi-Fi) and bluetooth (BT) systems based on dynamic packet fragmentation is proposed in this work. The basic idea is to adapt the packet length of Wi-Fi in the MAC layer such that the fragmented packet has a better chance to survive the interference from the nearby BT devices. We first develop an analytical model that specifies the information required by the Wi-Fi MAC layer to decide the best fragmentation strategy. Then, this model is extended to analyze the throughput and transmission delay of the Wi-Fi device. The analytical model is validated by computer simulation. Furthermore, it is demonstrated by simulation results that the proposed coexistence mechanism improves the performance of Wi-Fi in throughput and transmission delay significantly while relatively smaller performance improvement is observed for BT.

We present a novel multistage learning system, called grouping estimation fusion (GEF), for human age estimation via facial images. The GEF consists of three stages: (1) age grouping; (2) age estimation within age groups; and (3) decision fusion for final age estimation. In the first stage, faces are classified into different groups, where each group has a different age range. In the second stage, three methods are adopted to extract global features from the whole face and local features from facial components (e.g., eyes, nose, and mouth). Each global or local feature is individually utilized for age estimation in each group. Thus, several decisions (i.e., estimation results) are derived. In the third stage, we obtain the final estimated age by fusing the diverse decisions from the second stage. To create diverse decisions for fusion, we investigate multiple age grouping systems in the first stage, where each system has a different number of groups and different age ranges. Thus, various decisions can be made from the second stage, and will be delivered to the third stage for fusion. Totally, six fusion schemes (i.e., intra-system fusion, inter-system fusion, intra-inter fusion, inter-intra fusion, maximum-diversity fusion, and composite fusion) are developed and compared. The performance of the GEF system is evaluated on the Face and Gesture Recognition Research Network and the MORPH-II databases, and it outperforms the existing state-of-the-art age estimation methods by a significant margin. That is, the mean absolute errors of age estimation are reduced from 4.48 to 2.81 years on FG-NET and 3.82 to 2.97 years on MORPH-II.

Objective quality assessment for compressed images is critical to various image compression systems that are essential in image delivery and storage. Although the mean squared error (MSE) is computationally simple, it may not be accurate to reflect the perceptual quality of compressed images, which is also affected dramatically by the characteristics of human visual system (HVS), such as masking effect. In this paper, an image quality metric (IQM) is proposed based on perceptually weighted distortion in terms of the MSE. To capture the characteristics of HVS, a randomness map is proposed to measure the masking effect and a preprocessing scheme is proposed to simulate the processing that occurs in the initial part of HVS. Since the masking effect highly depends on the structural randomness, the prediction error from neighborhood with a statistical model is used to measure the significance of masking. Meanwhile, the imperceptible signal with high frequency could be removed by preprocessing with low-pass filters. The relation is investigated between the distortions before and after masking effect, and a masking modulation model is proposed to simulate the masking effect after preprocessing. The performance of the proposed IQM is validated on six image databases with various compression distortions. The experimental results show that the proposed algorithm outperforms other benchmark IQMs.

The quality assessment for synthesized video with texture/depth compression distortion is important for the design, optimization, and evaluation of the multi-view video plus depth (MVD)-based 3D video system. In this paper, the subjective and objective studies for synthesized view assessment are both conducted. First, a synthesized video quality database with texture/depth compression distortion is presented with subjective scores given by 56 subjects. The 140 videos are synthesized from ten MVD sequences with different texture/depth quantization combinations. Second, a full reference objective video quality assessment (VQA) method is proposed concerning about the annoying temporal flicker distortion and the change of spatio-temporal activity in the synthesized video. The proposed VQA algorithm has a good performance evaluated on the entire synthesized video quality database, and is particularly prominent on the subsets which have significant temporal flicker distortion induced by depth compression and view synthesis process.

Identifying interpretable discriminative high-order feature interactions given limited training data in high dimensions is challenging in both machine learning and data mining. In this paper, we propose a factorization based sparse learning framework termed FHIM for identifying high-order feature interactions in linear and logistic regression models, and study several optimization methods for solving them. Unlike previous sparse learning methods, our model FHIM recovers both the main effects and the interaction terms accurately without imposing tree-structured hierarchical constraints. Furthermore, we show that FHIM has oracle properties when extended to generalized linear regression models with pairwise interactions. Experiments on simulated data show that FHIM outperforms the state-of-the-art sparse lear-ning techniques. Further experiments on our experimentally generated data from patient blood samples using a novel SOMAmer (Slow Off-rate Modified Aptamer) technology show that, FHIM performs blood-based cancer diagnosis and bio-marker discovery for Renal Cell Carcinoma much better than other competing methods, and it identifies interpretable block-wise high-order gene interactions predictive of cancer stages of samples. A literature survey shows that the interactions identified by FHIM play important roles in cancer development.

Content-aware image resizing (or image retargeting) is a technique that resizes images for optimum display on devices with different resolutions and aspect ratios. Traditional objective quality of experience (QoE) assessment methods are not applicable to retargeted images because the size of a retargeted image is different from its source. In this work, three determining factors for humans visual QoE on retargeted images are analyzed. They are global structural dis- tortion (G), local region distortion (L) and loss of salient information (S). Different features are selected to quantify their respective distortion degrees. Then, an objective quality assessment index, called GLS, is proposed to predict viewers' QoE by fusing selected features into one single quality score. Several regression models used for feature fusion are discussed and compared. Experimental results demonstrate that the proposed GLS quality index has stronger correlation with human QoE than other existing objective metrics in retargeted image quality assessment.

In this paper, we present a structured fusion method for facial age group classification. To utilize the structured fusion of shape features and surface features, we introduced the region of certainty (ROC) to not only control the classification accuracy for shape feature based system but also reduce the classification needs on surface feature based system. In the first stage, we design two shape features, which can be used to classify frontal faces with high accuracies. In the second stage, a surface feature is adopted and then selected by a statistical method. The statistical selected surface features combined with a SVM classifier can offer high classification rates. With properly adjusting the ROC by a single non-sensitive parameter, the structured fusion of two stages can provide a performance improvement. In the experiments, we use face images in the public available FG-NET and MORPH databases and partition them into three pre-defined age groups. It is observed that the proposed method offers a correct classification rate of 95.1% in FG-NET and 93.7% in MORPH, which outperforms state-of-the-art methods by a significant margin.

His research interests include multimedia data compression, communication and networking, multimedia content analysis and modeling,

Depth video is used as the geometrical information of 3D world scenes in 3D view synthesis. Due to the mismatch between the number of depth levels and disparity levels in the view synthesis, the relationship between depth distortion and rendering position error can be modeled as a many-to-one mapping function, in which different depth distortion values might be projected to the same geometrical distortion in the synthesized virtual view image. Based on this property, we present an allowable depth distortion (ADD) model for 3D depth map coding. Then, an ADD-based rate-distortion model is proposed for mode decision and motion/disparity estimation modules aiming at minimizing view synthesis distortion at a given bit rate constraint. In addition, an ADD-based depth bit reduction algorithm is proposed to further reduce the depth bit rate while maintaining the qualities of the synthesized images. Experimental results in intra depth coding show that the proposed overall algorithm achieves Bjontegaard delta peak signal-to-noise ratio gains of 1.58 and 2.68 dB on average for half and integer-pixel rendering precisions, respectively. In addition, the proposed algorithms are also highly efficient for inter depth coding when evaluated with different metrics.

In this paper, we present a compressed-domain video retargeting solution that operates without compromising the resizing quality. Existing video retargeting methods operate in the spatial (or pixel) domain. Such a solution is not practical if it is implemented in mobile devices due to its large memory requirement. In the proposed solution, each component of the retargeting system is designed to exploit the low-level compressed domain features extracted from the coded bit stream. For example, motion information is obtained directly from motion vectors. An efficient column shape mesh deformation is employed to solve the difficulty of sophisticated quad-shape mesh deformation in the compressed domain. The proposed solution achieves comparable (or slightly better) visual quality performance as compared with several state-of-the-art pixel-domain retargeting methods at lower computational and memory costs, making content-aware video resizing both scalable and practical in real-world applications.

Automatic classification of human motion capture (mocap) data has many commercial, biomechanical, and medical applications and is the principal focus of this paper. First, we propose a multi-resolution string representation scheme based on the tree-structured vector quantization (TSVQ) to transform the time-series of human poses into codeword sequences. Then, we take the temporal variations of human poses into account via codeword sequence matching. Furthermore, we develop a family of pose-histogram-based classifiers to examine the spatial distribution of human poses. We analyze the performance of the temporal and spatial classifiers separately. To achieve a higher classification rate, we merge their decisions and soft scores using novel fusion methods. The proposed fusion solutions are tested on a wide variety of sequences from the CMU mocap database using five-fold cross validation, and a correct classification rate of 99.6% is achieved.

In this work, a new coding tool called Edge Mode is proposed for HEVC intra coding, aimed at improving coding efficiency for screen content video. A set of edge modes that correspond to edge positions are identified based upon intra prediction directions. Then, a simplified scheme is developed to select the best edge mode. To avoid applying a transform over strong edges, directional 2D separable transforms are applied to blocks partitioned using these edge modes. Experimental results show that HEVC with edge modes (HEVC/EM) can achieve up to an 18% reduction in bit-rate as compared to unmodified HEVC, with an average reduction of 10.4% for screen content video sequences.

A new video quality index, called the distortion/content-dependent video quality index (DCVQI), is proposed in this paper. The distortion and content effects on video quality assessment are taken into account explicitly in the DCVQI. First, we classify distortions into two types (local and global distortions) with a distortion classification scheme. Then, for a given distortion type, we observe a linear relation model between the perceptual quality and SSIM for a given video clip, and the parameters of the linear model are determined by video contents. These parameters can be obtained through a machine learning process. It is shown by experimental results the proposed DCVQI offers a PLCC (Pearson Linear Correlation Coefficient) value of 0.869 when tested on the LIVE database, which outperforms all state-of-the-art video quality assessment methods, including MOVIE and STMAD.

We investigate dependent rate/distortion (R/D) modeling techniques for H.264/SVC videos. We introduce a self-domain (S-domain) analysis method for characterizing the dependent R/D behaviors, where the R/D characteristics of a base layer are employed as the observation domain for those of dependent layers. Based onS-domain observations, we propose empirical dependent R/D models and analyze physical implications of the proposed models. As an application of the proposed R/D models, we examine a joint temporal-quality layer bit allocation algorithm formulated as a Lagrange optimization problem. The proposed R/D models enable us to derive an analytical solution to the joint optimization problem. Finally, it is demonstrated by experimental results that our bit allocation algorithm outperforms JSVM benchmark by a significant margin (10%-20%) at various bit rates.

A new image decomposition scheme, called the adaptive directional total variation (ADTV) model, is proposed to achieve effective segmentation and enhancement for latent fingerprint images in this work. The proposed model is inspired by the classical total variation models, but it differentiates itself by integrating two unique features of fingerprints; namely, scale and orientation. The proposed ADTV model decomposes a latent fingerprint image into two layers: cartoon and texture. The cartoon layer contains unwanted components (e.g., structured noise) while the texture layer mainly consists of the latent fingerprint. This cartoon-texture decomposition facilitates the process of segmentation, as the region of interest can be easily detected from the texture layer using traditional segmentation methods. The effectiveness of the proposed scheme is validated through experimental results on the entire NIST SD27 latent fingerprint database. The proposed scheme achieves accurate segmentation and enhancement results, leading to improved feature detection and latent matching performance.

In this paper, a novel rate control scheme with optimized bits allocation for the 3-D video coding is proposed. First, we investigate the R-D characteristics of the texture and depth map of the coded view, as well as the quality dependency between the virtual view and the coded view. Second, an optimal bit allocation scheme is developed to allocate target bits for both the texture and depth maps of different views. Meanwhile, a simplified model parameter estimation scheme is adopted to speed up the coding process. Finally, the experimental results on various 3-D video sequences demonstrate that the proposed algorithm achieves excellent R-D efficiency and bit rate accuracy compared to benchmark algorithms.

In this paper, we propose a cascaded sparse/DCT (S/DCT) two-layer representation of prediction residuals, and implement this idea on top of the state-of-the-art high efficiency video coding (HEVC) standard. First, a dictionary is adaptively trained to contain featured patterns of residual signals so that a high portion of energy in a structured residual can be efficiently coded via sparse coding. It is observed that the sparse representation alone is less effective in the R-D performance due to the side information overhead at higher bit rates. To overcome this problem, the DCT representation is cascaded at the second stage. It is applied to the remaining signal to improve coding efficiency. The two representations successfully complement each other. It is demonstrated by experimental results that the proposed algorithm outperforms the HEVC reference codec HM5.0 in the Common Test Condition.

A new methodology for objective image quality assessment (IQA) with multi-method fusion (MMF) is presented in this paper. The research is motivated by the observation that there is no single method that can give the best performance in all situations. To achieve MMF, we adopt a regression approach. The new MMF score is set to be the nonlinear combination of scores from multiple methods with suitable weights obtained by a training process. In order to improve the regression results further, we divide distorted images into three to five groups based on the distortion types and perform regression within each group, which is called "context-dependent MMF" (CD-MMF). One task in CD-MMF is to determine the context automatically, which is achieved by a machine learning approach. To further reduce the complexity of MMF, we perform algorithms to select a small subset from the candidate method set. The result is very good even if only three quality assessment methods are included in the fusion process. The proposed MMF method using support vector regression is shown to outperform a large number of existing IQA methods by a significant margin when being tested in six representative databases.

In this paper, we propose a view synthesis distortion model (VSDM) that establishes the relationship between depth distortion and view synthesis distortion for the regions with different characteristics: color texture area corresponding depth (CTAD) region and color smooth area corresponding depth (CSAD), respectively. With this VSDM, we propose regional bit allocation (RBA) and rate distortion optimization (RDO) algorithms for multiview depth video coding (MDVC) by allocating more bits on CTAD for rendering quality and fewer bits on CSAD for compression efficiency. Experimental results show that the proposed VSDM based RBA and RDO can improve the coding efficiency significantly for the test sequences. In addition, for the proposed overall MDVC algorithm that integrates VSDM based RBA and RDO, it achieves 9.99% and 14.51% bit rate reduction on average for the high and low bit rate, respectively. It can improve virtual view image quality 0.22 and 0.24 dB on average at the high and low bit rate, respectively, when compared with the original joint multiview video coding model. The RD performance comparisons using five different metrics also validate the effectiveness of the proposed overall algorithm. In addition, the proposed algorithms can be applied to both INTRA and INTER frames.

In a generic decision process, optimal stopping theory aims to achieve a good tradeoff between decision performance and time consumed, with the advantages of theoretical decision-making and predictable decision performance. In this paper, optimal stopping theory is employed to develop an effective hybrid model for the mode decision problem, which aims to theoretically achieve a good tradeoff between the two interrelated measurements in mode decision, as computational complexity reduction and rate-distortion degradation. The proposed hybrid model is implemented and examined with a multiview encoder. To support the model and further promote coding performance, the multiview coding mode characteristics, including predicted mode probability and estimated coding time, are jointly investigated with inter-view correlations. Exhaustive experimental results with a wide range of video resolutions reveal the efficiency and robustness of our method, with high decision accuracy, negligible computational overhead, and almost intact rate-distortion performance compared to the original encoder.

ABSTRACT A graph‐based framework is proposed in this paper to implement dynamic fractional frequency reuse (D‐FFR) in a multicell Orthogonal Frequency Division Multiple Access (OFDMA) network. FFR is a promising resource‐allocation technique that can effectively mitigate intercell interference (ICI) in OFDMA networks. The proposed D‐FFR scheme enhances the conventional FFR by enabling adaptive spectral sharing as per cell‐load conditions. Such adaptation has significant benefits in practical systems where traffic loads in different cells are usually unequal and time‐varying. The dynamic adaptation is accomplished via a graph framework in which the resource‐allocation problem is solved in two phases: (1) constructing an interference graph that matches the specific realization of FFR and the network topology and (2) coloring the graph by use of a heuristic algorithm. Various realizations of FFR can easily be incorporated in the framework by manipulating the first phase. The performance improvement enabled by the proposed D‐FFR scheme is demonstrated by computer simulation for a 19‐cell network with equal and unequal cell loads. In the unequal‐load scenario, the proposed D‐FFR scheme offers significant performance improvement in terms of cell throughput and service rate as compared to conventional FFR and previous interference management schemes. Copyright © 2011 John Wiley & Sons, Ltd.

A new block-based multi-metric fusion (BMMF) approach is proposed for perceptual image quality assessment. The proposed BMMF scheme automatically detects image content and distortion types in a block via machine learning, which is motivated by the observation that the performance of an image quality metric is highly influenced by these factors. Locally, image block content is classified into three types; namely, smooth, edge and texture. Image distortion is detected and grouped into five types. An appropriate image quality metric is adopted for each block by considering its content and distortion types, and then all block-based quality metrics are fused to result in one final score. Furthermore, a corrected version of BMMF is derived for a specific group of distortions based on image complexity analysis. The proposed BMMF scheme is tested on TID database with its Spearman Correlation equal to 0.9471, which outperforms today's state-of-the-art image quality metrics.

In this paper, we analyze the effect of texture regularity on the performance of image resizing (or called image retargeting), and then propose an efficient texture-aware resizing algorithm. Being perceived as unimportant due to spatial homogeneity, textured patterns are largely deformed (or warped) in existing image resizing algorithms. However, arbitrary warping without considering the specific texture property tends to lead to noticeable visual artifacts. To address this issue, we exploit region features, including the scale and the shape information, to preserve both local and global structures. Guided by region and contour information, we propose a mesh-based resizing technique, which is formulated as a nonlinear least squares optimization problem and solved by an iterative Gauss- Newton method. Texture redundancy is effectively reduced through texture regularity analysis and real-time texture synthesis. The superior performance of the proposed image resizing technique is demonstrated by experimental results.

A JPEG-based perceptual image coder is proposed in this work, where a block-based image quality metric is used to optimize the rate-quality (RQ) performance. Under this framework, the quality of each image block is measured using a local quality metric while the overall image quality is evaluated by summing up all local quality metrics. A rate-quality optimization (RQO) problem is formulated in each macroblock of size 16×16 based on its associated empirical RQ curve. Then, to achieve the best perceptual image quality under a given bit budget constraint, the set of optimal quantization parameters (QPs) for image blocks is solved using the Lagrangian approach. It is demonstrated that the proposed perceptual image codec offers a significant improvement over the JPEG baseline in both subjective and objective evaluations.

Latent fingerprint detection and segmentation play a critical role in image forensics for law enforcement. Being collected from crime scenes, a latent fingerprint is often mixed with other components such as structured noise or other fingerprints. Existing fingerprint recognition algorithms fail to work properly for latent fingerprint images, since they are mostly applicable under the assumption that the image is already properly segmented and there is no overlap between the target fingerprint and other components. In this work, we present a novel directional total variation (DTV) model to achieve effective latent fingerprint detection and segmentation. As compared with existing total variation models, the proposed DTV model differentiates itself by considering spatial-dependent texture orientations in the TV computation, which is particularly suitable for images with oriented textures. We demonstrate the superior performance of the proposed DTV technique using images from the NIST SD27 latent fingerprint database.

With the continually growing popularity of online video sharing, energy consumption in video sharing servers has become an pivotal issue. Energy savings in video servers can be achieved by utilizing low power modes in disks, but this causes enhanced delay which is not desirable for video sharing services. In this paper, we present a new energy management scheme that jointly optimizes energy and delay for the video storage system. There are two salient characteristics of the proposed scheme. First, it exploits the unique workload characteristics of video sharing services. Second, it includes a new model that efficiently facilitates the selection of power modes for disks. Experiments show that our scheme achieves considerably more energy savings under the same delay level compared to the traditional threshold-based energy management scheme.

Block-based image steganalysis, which conducts steganalysis on smaller homogenous blocks of a given test image, was proposed to improve the performance of steganalysis. However, the computational complexity of block-based image steganalysis is high when the feature size is large. To reduce the complexity, we develop a content-dependent feature selection scheme for a binary classifier. The main idea is to select important features depending on block types, which explains the term of “content-dependent” selection. This choice enables us to obtain better performance with a smaller number of features and reduce computational complexity at the same time. Experimental results are conducted to demonstrate the performance improvement of content-dependent feature selection with high detection accuracy.

Fast mode decision algorithms have been widely used in the video encoder implementation to reduce encoding complexity yet without much sacrifice in the coding performance. Optimal stopping theory, which addresses early termination for a generic class of decision problems, is adopted in this paper to achieve fast mode decision for the H.264/Scalable Video Coding standard. A constrained model is developed with optimal stopping, and the solutions to this model are employed to initialize the candidate mode list and predict the early termination. Comprehensive simulation results are conducted to demonstrate that the proposed method strikes a good balance between low encoding complexity and high coding efficiency.

The use of opportunistic multicast scheduling (OMS) for systems with multiple multicast groups is examined in this work. Here, a general downlink scenario where a single basestation is to transmit independent data streams to multiple groups of users is considered. Within each group, common information is transmitted, but between groups, the source information is independent. In the literature, the OMS scheme has been proposed for the case where all users belong to a single multicast group. This scheme allows the base-station to transmit at a higher rate in each time slot by scheduling the transmission to only a subset of users in the group. By encoding the data stream using fountain codes, each user will be able to decode the message whenever a sufficient number of data bits are received, regardless of which specific time slots the user was able to receive in. In this work, the use of OMS is extended to systems with multiple multicast groups and the so-called multicast throughput region is defined to characterize the performance of the multigroup OMS scheme. The analytical results based on extreme value theory are utilized to accurately predict the optimal multicast group-sizes and the optimal power allocation policy when maximizing the weighted sum throughput. By choosing the weights appropriately, it is shown that the method can be further utilized to ensure proportional fairness among the multiple multicast groups. The efficacy of the proposed OMS schemes is shown through numerical simulations.

Information theoretic security, also called secrecy analysis, provides theoretical limit for secret data transmission even in wireless networking environment, and it is more focused in these days because of increasing number of private information transmission through wireless channel and its inherent broadcasting characteristic. However, many studies in secrecy analysis assume that there exists only one eavesdropper or there is no cooperation among multiple eavesdroppers, which makes research results difficult to be applied in real situations. In this paper, we conduct a game theoretic analysis of eavesdropper cooperation in MISO wireless communication system where a secret information sender, Alice, has a counter-strategy against eavesdropping. We assume that Alice has a method for preventing eavesdroppers from overhearing. It is triggered by eavesdropper detection and degrades eavesdropping channel by forwarding artificial noise. For this reason, eavesdroppers' cooperation can pose a high risk to them and we utilize game theory to investigate eavesdropper behavior based on mutual information I (X; Yeve) between the input X from Alice and the output Yeveto an eavesdropper. We derive that non-cooperation is Nash Equilibrium in one shot eavesdropper cooperation game and show necessary conditions for eavesdropper cooperation in infinitely repeated game.

A first-order-residual/second-order-residual (FOR/SOR) based video coding algorithm that incorporates the super macroblock (SMB) and the inter-frame stripe prediction (ISP) technique is proposed for high definition (HD) video coding in this work. We first examine the limitation of the SMB in high-bit-rate coding, and show that a simple extension of the block-size is not sufficient to get a significant improvement in coding performance. Then, we introduce the FOR/SOR coding technique to resolve this issue. For the FOR coding, we apply a larger quantization stepize and motion-predictive coding to remove the temporal correlation. Then, for the SOR coding, we propose an efficient prediction method, called the inter frame stripe prediction (ISP) technique, to remove the structured residuals after the FOR coding to achieve a higher coding gain. It is demonstrated by experimental results that the proposed FOR/SOR algorithm outperforms H.264/AVC by a significant margin; namely, about 21.57% in the bit rate saving or 1.09dB in the PSNR improvement.

The FOR/SOR algorithm was initially proposed in [1] and shown to outperform H.264/AVC by a significant margin for high definition (HD) video coding when the bit rate is equal to 20Mbps or higher. In this work, we propose an improved FOR/SOR coding scheme which offers a superior coding performance in a bit rate range of practical interest, which ranges from 1-15 Mbps. To enhance the coding performance of the proposed algorithm, we incorporate several new coding tools in the basic FOR/SOR coding algorithm in [1]. Furthermore, we attempt to give an intuitive explanation of the superior performance of the FOR/SOR coding scheme using arguments based on the R-D analysis. Finally, it is shown by experimental results that the proposed algorithm outperforms H.264/AVC significantly for HD video in the bit rate range of common interest.

A general method for recovering missing DCT coefficients in DCT-transformed images is presented in this work. We model the DCT coefficients recovery problem as an optimization problem and recover all missing DCT coefficients via linear programming. The visual quality of the recovered image gradually decreases as the number of missing DCT coefficients increases. For some images, the quality is surprisingly good even when more than 10 most significant DCT coefficients are missing. When only the DC coefficient is missing, the proposed algorithm outperforms existing methods according to experimental results conducted on 200 test images. The proposed recovery method can be used for cryptanalysis of DCT based selective encryption schemes and other applications.

Social networking becomes increasingly important due to the recent surge in online interaction. Social network analysis can be used to study the functioning of computer networks, information flow patterns in communities, and emergent behavior of physical and biological systems. In this paper, the mathematical formulation and computational models for security and privacy of social network data are discussed. Several possible ways for an attacker to attack are presented so that the mathematical formulation can take them into account. The metrics for measuring the amount of security and privacy in an online social network (OSN) are discussed so that we have an idea of how good a model is. Based on these current techniques and attack strategies, future directions of research are discussed.

This paper re-evaluates the security of a secure Lempel-Ziv-Welch (LZW) algorithm proposed at ICME'2008. A chosen-plaintext attack is proposed to break all ciphertext indices corresponding to single-symbol dictionary entries. For short plaintexts the chosen-plaintext attack works well because string-symbol strings appear very frequently. The number of required chosen plaintexts is at the order of the alphabet size. The complexity of the chosen-plaintext attack is O(ML), where M is the number of chosen plaintexts and L is the size of the ciphertext. The chosen-plaintext attack can also be generalized to chosen-ciphertext attack. In addition to the security problem, we point out that the secure LZW algorithm has a lower compression efficiency compared with the original LZW algorithm. Finally we propose several enhancements to the secure LZW algorithm under study.

Block-based image steganalysis, which uses smaller homogenous blocks from a given test image, was previously proposed to improve the steganalysis performance. Performance study on block-based image steganalysis in terms of block sizes and block numbers is conducted in this research. First, we analyze the dependence of the steganalysis performance on one of these two factors, and show that a larger block size and a larger block number will lead to better steganalysis performance. Our study is verified by experimental results. For a given test image, there exists a trade-off between the block size and the block number, and it is shown that a balance between the block size and the block number can result in better steganalysis performance.

In this paper, we propose a novel dictionary based video coding technique with adaptive construction of over complete dictionaries and advanced coding methods tailored to sparse signal representations. A set of dictionaries is trained off-line using inter or intra predicted residual samples and is applied for encoding. New coding tools are developed so that the encoder can more compactly represent the residual signal. The same set of dictionary elements can be reused for neighboring blocks, and the optimal number of dictionary elements can be decided using rate-distortion optimization. Experimental results demonstrate that the proposed algorithm yields both improved coding performance and improved perceptual quality at low bit rates.

In this work, we conduct an in-depth survey on recent multimedia storage security research activities in association with cloud computing. After an overview of the cloud storage system and its security problem, we focus on four hot research topics. They are data integrity, data confidentiality, access control, and data manipulation in the encrypted domain. We describe several key ideas and solutions proposed in the current literature and point out possible extensions and futuristic research opportunities. Our research objective is to offer a state-of-the-art knowledge to new researchers who would like to enter this exciting new field.

The denoising performance of the Non-Local Means (NLM) method decreases as the variance of additive white Gaussian noise becomes higher. In this paper, we explain this phenomenon and propose a modified version of the Non-Local Means (NLM) method, called the Enhanced-Weights NLM (EWNLM) algorithm, to denoise highly noisy images. The EWNLM algorithm evaluates weights from a pre-filtered image using the Gaussian kernel, which in turn result in more robust weight contributions from similar pixels in the search window. Experimental results are given to demonstrate the superior performance of the EWNLM scheme when the standard deviation of the additive white Gaussian noise (AWGN) is greater than 20.

We propose a source-specific dictionary approach to efficient music representation, and apply it to separation of music signals that coexist with background noise such as speech or environmental sounds. The basic idea is to determine a set of elementary functions, called atoms, that efficiently capture music signal characteristics. There are three steps in the construction of a source-specific dictionary. First, we decompose basic components of musical signals (e.g., musical notes) into a set of source-independent atoms (i.e., Gabor atoms). Then, we prioritize these Gabor atoms according to their approximation capability to music signals of interest. Third, we use the prioritized Gabor atoms to synthesize new atoms to build a compact dictionary. The number of atoms needed to represent music signals using the source-specific dictionary is much less than that of the Gabor dictionary, resulting in a sparse music representation. For the single-channel music signal separation, we project the mixture signal onto source-specific atoms. Experimental results are given to demonstrate the efficiency and applications of the proposed approach.

A novel frame-level rate control (RC) algorithm is presented in this paper for temporal scalability of scalable video coding. First, by introducing a linear quality dependency model, the quality dependency between a coding frame and its references is investigated for the hierarchical B-picture prediction structure. Second, linear rate-quantization (R-Q) and distortion-quantization (D-Q) models are introduced based on different characteristics of temporal layers. Third, according to the proposed quality dependency model and R-Q and D-Q models for each temporal layer, adaptive weighting factors are derived to allocate bits efficiently among temporal layers. Experimental results on not only traditional quarter common intermediate format/common intermediate format but also standard definition and high definition sequences demonstrate that the proposed algorithm achieves excellent coding efficiency as compared to other benchmark RC schemes.

A lossless image coding method, known as the two-layered residual coding (TRC) scheme, is proposed in this letter. After the H.264/AVC lossy intra prediction, we propose an advanced scheme for residual coding, which consists of two residual coders in cascade. The first-layer residual coder is conducted via transform and quantization with a coarser quantization parameter. The second-layer residual coder is a bit-plane coding method. It is shown experimentally that the TRC scheme outperforms the H.264/AVC lossless intra coding with an averaged bit rate saving of about 24%.

The addition of a new component to the traditional synthesis-based texture video coding algorithm is investigated in this paper. That is, we add the side information in the form of low-quality video to enhance the texture video synthesis performance with reducing the unpleasant mismatch between analyzed and synthesized regions. As compared with the conventional synthesis algorithm, our algorithm is more flexible since the behavior and quality of the output texture can be adjusted by the amount of the side information, which is determined by the user. To this end, we develop an area-adaptive side information selection scheme that chooses the proper amount of the side information for a given bit budget. Furthermore, we propose a texture decomposition scheme that extracts the non-synthesizable illumination component from the source video for separate coding so as to maximize the synthesis functionality. The superior performance of the proposed texture video synthesis technique is demonstrated by several coding examples.

We investigate methods to enable robust and secure sharing among trusted peers in wireless erasure networks based on network coding (NC). The proposed communication model allows an eavesdropper to take advantage of the broadcast medium to successfully overhear messages transmitted along a min-cut. In situations where there is no secure capacity, we give a convolutional NC scheme that achieves weak secrecy to decrease the number of symmetric keys required. Furthermore, we demonstrate the fundamental duality between robustness and secrecy enforced by the wireless environment.

Physical layer multicasting with opportunistic user selection and optimized space-time transmission is examined in this work. We consider a multi-antenna downlink scenario where a multi-antenna base-station transmits common information to a given set of users, each with a single receive antenna. In the past, most existing multi-antenna multicasting solutions have been restricted to spatial multiplexing and transmit beamforming. Here, we adopt the opportunistic multicast scheduling (OMS) scheme, where messages are encoded using fountain codes and an optimal subset of users is targeted for reception in each time slot. Capitalizing on extreme value theory, we derive analytical expressions for the average system throughput and utilize this result to obtain the optimal user selection ratio for both spatial multiplexing and transmit beamforming scenarios. To better utilize the channel state information available at the transmitter, we further consider an optimized space-time transmission (OST) scheme where the statistics of the space-time codeword is chosen to maximize the rate of the worst selected user. For reduced complexity, we utilize a semi-orthogonal user selection algorithm to determine the spatial dimensions of the signal and derive optimal power allocation across these dimensions. The proposed OST is a generalization of transmit beamforming and spatial multiplexing and, thus, outperforms both conventional schemes.

An adaptive search range (SR) selection algorithm in motion estimation for video encoding is proposed in this work with an objective to achieve excellent coding performance with reduced memory access bandwidth. In modern fast motion search algorithms, a motion vector predictor (MVP) is obtained from the motion vectors (MVs) of spatially and temporally neighboring blocks, which is called the predictive MV set. By exploiting the relationship between the variance of the predictive MV set and the SR, we develop an adaptive SR selection algorithm. That is, a larger variance implies lower accuracy of the MVP and, thus, a larger SR. The superior performance of the proposed SR selection algorithm is demonstrated by experimental results.

Motivated by the work of Uehara et al. [1], an improved method to recover DC coefficients from AC coefficients of DCT-transformed images is investigated in this work, which finds applications in cryptanalysis of selective multimedia encryption. The proposed under/over-flow rate minimization (FRM) method employs an optimization process to get a statistically more accurate estimation of unknown DC coefficients, thus achieving a better recovery performance. It was shown by experimental results based on 200 test images that the proposed DC recovery method significantly improves the quality of most recovered images in terms of the PSNR values and several state-of-the-art objective image quality assessment (IQA) metrics such as SSIM and MS-SSIM.

Traditional image steganalysis techniques for classification of steganograhic algorithms are conducted with respect to the entire image. In this work, we aim to design a multi-classifier which classifies stego images depending on their steganographic algorithms in addition to distinguishing stego images from cover images. This classification is based on steganalysis results of decomposed image blocks. As a natural image often consists of heterogeneous regions, its decomposition will lead to smaller image blocks, each of which is more homogeneous. We classify these image blocks into multiple classes and find a classifier for each class to decide whether a block is from a cover image or a stego image with a specific steganographic algorithm. Consequently, the steganalysis of the whole image can be conducted by fusing weighted steganalysis results of all image blocks through a voting process. Experimental results will be given to show the advantage of using the proposed block-based image steganalysis for a multi-classifier.

We propose a tree model simplification method based on a hybrid polygon/billboard approach in this work. The objective is to create a simplified tree model with lower polygon counts while preserving the visual appearance of the original model. The result is suitable for human viewing from different angles around the tree as well as close up views. In the simplification process, a subset of polygons is carefully chosen to speed up the rendering process, and textures with a different number of leaves are mapped to scaled polygons based on the density of polygon's position on the tree. For objective performance evaluation, saliency maps of rendered images of simplified and original models are generated and the mean-squared-error (MSE) between them is calculated to quantify their resemblance. Furthermore, it is shown that feature maps (e.g., the intensity, orientation, and color maps of an image) can be utilized to simplify the tree model more aggressively since they provide means to highlight human visual attention. Finally, a subjective test is conducted to rate different simplification algorithms. Experimental results indicate that the proposed hybrid polygon/billboard method has better performance than other benchmarking methods in both objective and subjective evaluations.

Traditional image steganalysis techniques are conducted with respect to the entire image. In this work, we aim to differentiate a stego image from its cover image based on steganalysis results of decomposed image blocks. As a natural image often consists of heterogeneous regions, its decomposition will lead to smaller image blocks, each of which is more homogeneous. We classify these image blocks into multiple classes and find a classifier for each class to decide whether a block is from a cover or stego image. Consequently, the steganalysis of the whole image can be conducted by fusing steganalysis results of all image blocks through a voting process. Experimental results will be given to show the advantage of the proposed block-based image steganalysis approach.

A novel multi-order-residual (MOR) coding approach is proposed for high-bit-rate video compression in this work. We show that a significant amount of short and medium-range correlations due to the use of a fine quantization parameter cannot be easily removed by the well known motion compensated prediction process in the high-bit- rate coding scenario. Consequently, the video coding performance degrades rapidly as quality increases. To address this issue, we propose a Multi-Order-Residual (MOR) coding scheme that encodes prediction residuals in multiple stages according to their correlation characteristics. Then, we adopt different compensation and coding schemes in different stages. Experimental results are given to show the proposed MOR scheme outperforms H.264/AVC by a significant margin.

Abstract A feature‐oriented generic progressive lossless mesh coder (FOLProM) is proposed to encode triangular meshes with arbitrarily complex geometry and topology. In this work, a sequence of levels of detail (LODs) are generated through iterative vertex set split and bounding volume subdivision. The incremental geometry and connectivity updates associated with each vertex set split and/or bounding volume subdivision are entropy coded. Due to the visual importance of sharp geometric features, the whole geometry coding process is optimized for a better presentation of geometric features, especially at low coding bitrates. Feature‐oriented optimization in FOLProM is performed in hierarchy control and adaptive quantization. Efficient coordinate representation and prediction schemes are employed to reduce the entropy of data significantly. Furthermore, a simple yet efficient connectivity coding scheme is proposed. It is shown that FOLProM offers a significant rate‐distortion (R‐D) gain over the prior art, which is especially obvious at low bitrates.

The sparse representation of music sounds that consist of a single note at a time was examined in. Here, we extend the results to a more generic setting where music sounds may contain multiple notes (or chords) at the same time. The basic idea is to determine a set of elementary functions, called source-specific atoms, that efficiently capture music signal characteristics. We first decompose basic components of musical signals (i.e,, musical notes) into a set of Gabor atoms. Then, these Gabor atoms are prioritized according to their approximation capability to music signals of interest, and the prioritized Gabor atoms are used to synthesize source-specific atoms. To find a sparse representation for musical chords, we generate new atoms by regrouping source-specific atoms. This technique is applied to the approximation of real piano recordings, and its effectiveness in terms of good approximation capability and low computational complexity is demonstrated by experiments.

A speed up technique for the non-local means (NLM) image denoising algorithm based on probabilistic early termination (PET) is proposed. A significant amount of computation in the NLM scheme is dedicated to the distortion calculation between pixel neighborhoods. The proposed PET scheme adopts a probability model to achieve early termination. Specifically, the distortion computation can be terminated and the corresponding contributing pixel can be rejected earlier, if the expected distortion value is too high to be of significance in weighted averaging. Performance comparative with several fast NLM schemes is provided to demonstrate the effectiveness of the proposed algorithm.

H.264/advanced video coding (AVC) decoding complexity modeling and applications are examined in this paper. We focus on complex modeling of spatial and temporal compensations achieved by a rich set of inter and intra-prediction modes of H.264/AVC. We study the relationship between motion vectors (MV), frame sizes, and distributions of selected reference frames, which are highly related to cache management efficiency and, thus, the decoding complexity. An H.264/AVC encoder equipped with the complexity models can choose the best inter or intra-prediction mode and the best MV or intra-prediction direction while meeting the decoding complexity constraint of the target receiver platform. The proposed complexity models are accurate. The application of complexity models to the H.264/AVC decoding complexity reduction is demonstrated.

We propose a model-based spatial layer bit allocation algorithm for H.264/scalable video coding (SVC) in this paper. The challenge of this problem lies in the fact that the rate-distortion (R-D) behavior of an enhancement layer is dependent on its preceding layers because of inter-layer prediction. To solve it, we first focus on the case of two spatial layers, derive the distortion and rate models of the dependent layer analytically, and develop a low-complexity bit allocation algorithm. It is shown by experimental results that the proposed two-layer bit allocation algorithm can achieve the coding performance close to the optimal R-D performance based on the full search method. Then, we extend this result to multilayer bit allocation by performing the two-layer allocation scheme recursively. Finally, we compare the performance of group of pictures-based and frame-based spatial layer bit allocation schemes at a fixed temporal resolution. The superior performance of the proposed spatial layer bit allocation algorithm is demonstrated using Joint Scalable Video Model reference software algorithm and two prior H.264/SVC rate control algorithms as the benchmarks.

An adaptive motion search range (SR) prediction algorithm for video coding is proposed in this letter. To determine a proper SR size for motion vectors (MVs) is important to video encoding, since a good choice helps reduce the memory access bandwidth while maintaining the rate-distortion coding performance. To achieve this objective, we first obtain a motion vector predictor (MVP) for a target block based on MVs of its spatially and temporally neighboring blocks, which form a MV prediction set. Then, we relate the variance of the MV prediction set to the SR. That is, a larger variance implies lower accuracy of the MVP and, thus, a larger SR. Finally, we derive a probability model for the motion vector prediction difference, the difference between the optimal MV and the MVP, to quantify the probability for a chosen SR to contain the optimal MV. The superior performance of the proposed SR selection algorithm is demonstrated by experimental results.

Direct techniques for the optimal resolution estimation and position prediction of subpel motion vectors (MVs) based on integer-pel MVs are investigated in this paper. Although it is common to determine the optimal MV position by fitting a local error surface using integer-pel MVs, the characteristics of the error surface have not been thoroughly studied in the past. Here, we use an approximate condition number of the Hessian matrix of the error surface to characterize its shape in a local region. By exploiting this shape information, we propose a block-based subpel MV resolution estimation method that allows each block to choose its optimal subpel MV resolution for the optimal rate-distortion (R-D) performance adaptively. Furthermore, we propose two MV position prediction schemes for ill and well-conditioned error surfaces, respectively. All proposed techniques are direct methods, where no iteration is required. Experimental results are given to show the R-D performance of the proposed subpel MV resolution estimation and position prediction schemes.

This paper proposes an efficient algorithm for fast mode decision in H.264/advanced video coding by adaptively predicting the optimal mode for each macroblock (MB) to be coded. Firstly, encoding modes are projected as points onto a 2-D map, and an optimal 2-D point of the MB to be coded is predicted based on the encoding information of spatial-temporal neighboring blocks. Then, a priority-based mode candidate list with a descending order to be the best mode is constructed based on the optimal 2-D point. Finally, mode decision is performed according to the priority-based mode candidate list in the checking order, from the most important mode to the least one, with early termination conditions. Extensive experimental results demonstrate that the proposed algorithm is superior to three recent fast mode decision algorithms, with the entire encoding time being reduced by about 60% for quarter common intermediate format/common intermediate format/standard-definition sequences on average and the rate distortion performance being kept almost intact.

A precoding technique applied to symbols transmitted in an ultra-wideband (UWB) system can concentrate the signal power at the receiver for a higher data detection rate. The codeword design problem for optimal UWB precoding is investigated in this work. After the problem formulation, we examine its solution in the full-rank and the reduced-rank subspaces to obtain different tradeoffs between the detection performance and the amount of feedback messages. Furthermore, we propose two subspace selection schemes, each of which corresponds to a code design method. The resultant optimal codewords improve the performance of previous work significantly at the expense of a slightly increased amount of feedback messages. Finally, the performance of a precoded UWB system with various codewords is compared via computer simulation.

Optimized opportunistic multicast scheduling (OMS) is studied for cellular networks, where the problem of efficiently transmitting a common set of fountain-encoded data from a single base station to multiple users over quasi-static fading channels is examined. The proposed OMS scheme better balances the tradeoff between multiuser diversity and multicast gain by transmitting to a subset of users in each time slot using the maximal data rate that ensures successful decoding by these users. We first analyze the system delay in homogeneous networks by capitalizing on extreme value theory and derive the optimal selection ratio (i.e., the portion of users that are selected in each time slot) that minimizes the delay. Then, we extend results to heterogeneous networks where users are subject to different channel statistics. By partitioning users into multiple approximately homogeneous rings, we turn a heterogeneous network into a composite of smaller homogeneous networks and derive the optimal selection ratio for the heterogeneous network. Computer simulations confirm theoretical results and illustrate that the proposed OMS can achieve significant performance gains in both homogeneous and heterogeneous networks as compared with the conventional unicast and broadcast scheduling.

We present a cross-layer approach to the design of a wireless video streaming system, which includes the development of an adaptive radio link buffer and congestion control algorithm and the adoption of the H.264/SVC scalable video format. The proposed system performs a group-of-pictures (GOP) based post-channel adaptation, which does not require the channel status information at the media server. Video packets are prioritized in a GOP-by-GOP manner based on the dependence relation within a GOP. We employ H.264/SVC for the video stream preparation to provide the flexibility and adaptability of compressed video data. The priority information is conveyed in the header extension of the network abstraction layer unit (NALU) of H.264/SVC, which is parsed by a media aware network element (MANE) to realize an efficient transmission and congestion control. The performance of the proposed algorithm is verified by the computer simulation.

An anti-collusion fingerprinting system is developed to protect media files against time-varying collusion attacks based on the scalar Costa scheme (SCS) and colluder weight recovery. We treat the host signal as a parallel Gaussian channel and fingerprints as transmitted user signals. We decompose the Gaussian channel into multiple independent subchannels, and assign different user messages to different subchannels. Then, colluder weights in collusion attacks can be estimated using pilot symbols at the decoder, and all weights can be estimated and compensated. As a result, the decoding region on the parametric space can be recovered as an original format. It is shown by experimental results that the proposed fingerprinting system has excellent performance in colluder detection.

We propose a technique to separate audio sources from their anechoic mixtures with long delay in an underdetermined setting (i.e., the number of audio sensors is smaller than that of sources). It consists of two stages: 1) to estimate anechoic mixing parameters of attenuation and arrival delay and 2) to recover original audio sources based on estimated mixing parameters. When delay is longer than one sample, previous algorithms perform poorly. To address this shortcoming, we estimate the maximum delay and use it to find a proper frequency range that produces no phase ambiguity. Then, we determine mixing parameters with time-frequency points in this range. Finally, mathematical tools are used to solve the underdetermined linear system to recover original audio sources. The effectiveness of the proposed technique on various mixing scenarios with noisy observation of mixtures and different types of sounds is demonstrated by experimental results.

The bit allocation problem for hierarchical B-pictures in H.264/SVC is studied with a GOP-based dependent distortion model in this work. Inter-dependency between temporal layers of H.264/SVC is often neglected because of the complexity involved, which often leads to poorer rate control performance. To address this shortcoming, we propose a distortion model that takes inter-dependency into consideration while preserving the low complexity of the encoding process. It is demonstrated by experimental results that the new distortion model results in a highly efficient bit allocation scheme, which outperforms the rate control algorithm in the JSVM 9.12 reference codec by a significant margin.

We present a framework for audio background modeling of complex and unstructured audio environments. The determination of background audio is important for understanding and predicting the ambient context surrounding an agent, both human and machine. Our method extends the online adaptive Gaussian Mixture model technique to model variations in the background audio. We propose a method for learning the initial background model using a semi-supervised learning approach. This information is then integrated into the online background determination process, providing us with a more complete background model. We show that we can utilize both labeled and unlabeled data to improve audio classification performance. By incorporating prediction models in the determination process, we can improve the background detection performance even further. Experimental results on real data sets demonstrate the effectiveness of our proposed method.

Optimized opportunistic multicast scheduling (OMS) has been studied previously by the authors for homogeneous cellular networks, where the problem of efficiently transmitting a common set of data from a single base station to multiple users that have identical channel statistics was examined. It has been demonstrated that OMS can achieve significant performance improvement by exploiting the optimal tradeoff between multiuser diversity and multicast gain. In this work, we extend our studies to heterogeneous networks with users subject to different channel statistics. Specifically, we consider a single cell wireless network with users uniformly distributed in a circular region around the base station. Since users with low SNR are the ones that hinder system throughput, we argue that system performance may be predicted by the behavior of users in the outmost ring of the cell, which are approximately homogeneous. Using extreme value theory and results obtained from the homogeneous case, we determine the optimal user selection ratio for a homogeneous ring of users near the edge of the cell and then use it to derive the optimal selection ratio over the entire heterogeneous network. Simulations confirm theoretical results and illustrate the effectiveness of the proposed scheme.

To address the multiplicity and copyright issues on file sharing social networks, we propose a fast video copy detection algorithm using the suffix array data structure in this work. The proposed algorithm consists of two steps. In the first step, we extract robust features which are discriminative yet insensitive to various attacks. Specifically, we develop a compact one-dimensional signature based on the shot change position of video files. Unlike images and audio, the size of a video file is usually large, which makes it computationally expensive to match two long signature sequences. Thus, in the second step, we adopt an efficient matching technique based on the suffix array data structure. The proposed system can perform the sequence matching in linear time while the complexity of conventional duplicate video detection algorithms grows at least quadratically with the video length.

A graph-based framework for dynamic fractional frequency reuse (FFR) in multi-cell OFDMA networks is proposed in this work. FFR is a promising resource allocation technique that can effectively mitigate inter-cell interference (ICI) in OFDMA networks. The proposed scheme enhances the conventional FFR by enabling adaptive spectral sharing per cell load conditions. Such adaptation has significant benefits in a practical environment where traffic load in different cells may be asymmetric and time-varying. The dynamic feature is accomplished via a graph approach in which the resource allocation problem is translated to a graph coloring problem. Specifically, in order to incorporate various versions of FFR in our framework, we construct a graph that matches the specific version of FFR and then color the graph using the corresponding graph algorithm. The performance improvement enabled by the proposed dynamic FFR scheme is further demonstrated by computer simulation for a 19-cell network with asymmetric cell load. For instance, the proposed dynamic FFR scheme can achieve a 12% and 33% gain in cell throughput and service rate over conventional FFR, and render a 70% and 107% gain in cell throughput and service rate with respect to the reuse-3 system.

A 2D intra prediction with geometrical manipulations (2DIP) was proposed in [Yai, et al; 2008 ] to enhance the intra coding performance of H.264/AVC. However, the complexity of 2DIP is high if the full search scheme is used. In this work, we investigate ways to speed up the intra prediction mode decision for 2DIP. To achieve this, we first conduct a block correlation analysis that reveals the unique characteristics of 2DIP. Based on the analysis, a fast 2DIP search scheme is proposed to reduce the encoder complexity, where a block classification method is used to identify blocks using LIP only, a good initial search position is predicted and a block pruning technique is introduced. It is shown by experimental results that the fast 2DIP search can reduce the complexity of the full search significantly with little loss in the RD performance.

A fast duplicate video detection system based on camera transitional behavior and the suffix array data structure is proposed in this work. The main idea is to match video clips according to their temporal structures, and frames corresponding to unique events are marked as anchor frames. To simplify the detection process, we use the camera transitional behavior to indicate unique events. Specifically, shot boundaries and the beginning and the ending points of camera panning and tilting movements are detected as anchor frames. The length between neighboring anchor frames is computed to form an one-dimensional sequence, called the gap sequence, which serves as the signature of the video. An efficient gap sequence matching algorithm based on the suffix array data structure is adopted to match two given video signatures. Experiment results show that the proposed framework is not only very fast but also has high accuracy in identifying duplicate video pairs.

A technique to predict the optimal sub-pel motion vector (MV) position based on integer-pel MVs is investigated in this work. Although it is possible to find the optimal MV position by fitting a local error surface using integer-pel MVs, the minimum of the error surface may fall outside of the region of interest. By analyzing the behavior of the error surface, we propose a classification scheme that divides blocks into regular and irregular types. For a regular block, we adopt the traditional error surface model and solve its minimum as the predicted sub-pel MV position. For an irregular block, we developed a novel prediction scheme to predict the optimal sub-pel MV position. Experimental results are given to show the low complexity and good RD performance of the proposed sub-pel MV prediction scheme.

The effectiveness of 3D tree model simplification techniques and their objective and subjective performance evaluations are examined in this work. The simplification techniques developed in were based on pixel-based metrics, which did not consider the tree model's leaf density. For performance improvement, we perform simplification based on the tree leaf density of rendered images viewed from multiple angles. Furthermore, objective performance analysis is conducted to evaluate how well different algorithms are able to simplify tree models that appear as close to the original tree model with a given budget on the number of tree leaves in the model. To this end, a performance metric based on the Gabor filter is developed to analyze the orientation and spatial relationship within the rendered tree models. Finally, subjective evaluation is conducted by a group of 23 people. Both the objective and the subject evaluations reach a consistent conclusion; namely, the newly proposed density-based simplification technique offers the best results.

A texture interpolation technique based on the locally piecewise auto-regressive (PAR) model and the non-local (NL) training procedure is investigated in this work. The proposed PAR/NL scheme selects model parameters adaptively based on local image properties with an objective to improve the interpolation performance of nonadaptive models, e.g., the bicubic algorithm. To determine model parameters for stochastic texture, we use the non-local (NL) learning algorithm to update and refine these local model parameters under the assumption that the PAR model parameters are self-regular. As compared to previous interpolation algorithms, the proposed PAR/NL scheme boosts texture details, and eliminates blurring artifacts perceptually. Experimental results are given to demonstrate the performance of the proposed technique.

In this work, we propose a modified Hilbert-Huang transform (HHT) method to detect low pitch musical signals within a short analysis temporal window for real-time fundamental frequency estimation. HHT is a non-linear method which is suitable for the analysis of non-stationary AM/FM like data. However, to apply HHT directly to music signals encounters several problems. Here, we modify HHT so that it can be tailored to the short-window pitch analysis. It is shown by experimental results that the proposed HHT method performs significantly better than several benchmark schemes.

In this paper, we attempt to shed light on Benford's law from the viewpoint of probability theory and point out its limitation in image forensic applications. First, we consider a generalized form of Benford's law and relate it to a random variable of a certain probability density function such as the generalized Gaussian or Laplacian function. Then, we examine the application of the generalized Benford's law to image forensic applications and point out such an application is vulnerable to the histogram manipulation attack.

In this paper, we propose a cost-effective scheme for robust wireless multi-party video conferencing based on network coding (NC). The main idea is the adoption of a NC scheme to enhance robust transmission, to simplify the erasure protection procedure, and to reduce the downlink bandwidth by leveraging the properties of opportunistic NC and wireless broadcasting. We design a pipelining schedule to meet the delay requirement for real-time video conferencing. The proposed NC method outperforms the opportunistic network coding method by a significant margin and reduces the downlink bandwidth of the overall video bit rate.

We analyze time-varying collusion attacks for a fingerprinting system using concepts of capacity and throughput in this work. Continuous media provide a limited resource for data embedding, which is analogous to the capacity of a wireless communication channel. Here, we characterize the capacity of a host media using the just-noticeable-distortion (JND) of the human visual system (HVS). Furthermore, the collusion attack can be interpreted as a channel shared by multiple users. Based on this analogy, the colluder detection performance can be understood from the viewpoint of throughput. Specifically, we show how to determine instantaneous throughput using the fingerprint-to-interference-plus-noise ratio (FINR), and extend it to the total throughput and the averaged throughput over a time interval. Our analysis provides a good framework to the understanding of collusion attacks and ways to enhance the traitor tracing performance of a fingerprinting system.

Nintendo's Wii system uses motion acceleration sensors to provide an intuitive user interface with limited motion recognition capabilities. In this work, we investigate these limitations, and propose a new system called WiiStick to enhance the motion recognition performance. One main limitation of Wii is that the direction of movement cannot be recognized accurately if the motion involves both accelerating and rotating. To overcome this difficulty, we propose to add an elastic stick and an IR emitter to the Wiimote. Besides, we develop a motion recognition algorithm based on signal analysis and pattern recognition techniques. As a result, motion can be recognized more effectively. It is shown by experimental results that the motion recognition capability of the Wii system can be greatly improved by the proposed WiiStick system.

A joint quality-temporal (Q-T) bit allocation scheme is proposed for H.264/SVC in this work. First, rate and distortion (R-D) models for dependent quality layers are derived, where the complex inter-layer dependency is considered. Then, the joint Q-T bit allocation problem is formulated as an optimization problem using the Lagrange method and solved numerically with the derived R-D models. As a result, we develop a low-complexity bit allocation scheme for the joint Q-T scalability of H.264/SVC. It is demonstrated by experimental results that the new R-D models results in a highly efficient bit allocation scheme, which outperforms the JSVM benchmark by a significant margin.

An intra prediction (IP) scheme based on 2D geometrical manipulation, called 2DIP, was proposed in to improve the line-based intra prediction (LIP) in H.264/AVC. In this work, we propose ways to reduce the complexity and enhance the efficiency of 2DIP and call the resultant scheme 2D enhanced intra prediction (2DEIP). These enhancements are achieved via block classification and translation vector prediction. It is shown by experimental results that the proposed 2DEIP method allows better prediction for images with complex textures. The average PSNR gain over the H.264/AVC intra prediction can be as high as 0.6 dB.

We investigate efficient on-line lossless coding of motion capture (mocap) data in this work. The mocap data consist of a set of curves as the temporal trajectory of variables known as the degrees of freedom (dofs). To achieve compression, we sample each dof's curve temporally, interpolate the curve by a piece-wise Hermite spline as the prediction, and encode residuals with DPCM and Huffman coding. We compare the fixed-interval and the adaptive-interval sampling methods. Both schemes can achieve an averaged bit rate saving of around 80% (or, equivalently, a compression ratio of 20%) with real-time encoding/decoding.

A lossless video compression scheme is proposed based on residual image prediction and coding (RIPC) in this work. We first study different prediction schemes used in lossless coding, and observe that prediction residuals are not white but in form of granular noise. This study reveals a need to treat residual images with granular noise carefully. Then, we propose a new scheme, called RIPC, to encode the residual image effectively. Experimental results show that the proposed RIPC scheme outperforms the H.264 lossless intra prediction (H.264-LS) and JPEG-LS with a bit rate saving of 20% and 12%, respectively.

The paper considers the task of recognizing environmental sounds for the understanding of a scene or context surrounding an audio sensor. A variety of features have been proposed for audio recognition, including the popular Mel-frequency cepstral coefficients (MFCCs) which describe the audio spectral shape. Environmental sounds, such as chirpings of insects and sounds of rain which are typically noise-like with a broad flat spectrum, may include strong temporal domain signatures. However, only few temporal-domain features have been developed to characterize such diverse audio signals previously. Here, we perform an empirical feature analysis for audio environment characterization and propose to use the matching pursuit (MP) algorithm to obtain effective time-frequency features. The MP-based method utilizes a dictionary of atoms for feature selection, resulting in a flexible, intuitive and physically interpretable set of features. The MP-based feature is adopted to supplement the MFCC features to yield higher recognition accuracy for environmental sounds. Extensive experiments are conducted to demonstrate the effectiveness of these joint features for unstructured environmental sound classification, including listening tests to study human recognition capabilities. Our recognition system has shown to produce comparable performance as human listeners.

This paper proposes a novel approach to early determination of zero-quantized 8 × 8 discrete cosine transform (DCT) coefficients for fast video encoding. First, with the dynamic range analysis of DCT coefficients at different frequency positions, several sufficient conditions are derived to early determine whether a prediction error block (8 × 8) is an all-zero or a partial-zero block, i.e., the DCT coefficients within the block are all or partially zero-quantized. Being different from traditional methods that utilize the sum of absolute difference (SAD) of the entire prediction error block, the sufficient conditions are derived based on the SAD of each row of the prediction error block. For partial-zero blocks, fast DCT/IDCT algorithms are further developed by pruning conventional 8-point butterfly-based DCT/IDCT algorithms. Experimental results exhibit that the proposed early determination algorithm greatly reduces computational complexity in terms of DCT/IDCT, quantization, and inverse quantization, as compared with existing algorithms.

A technique for film grain noise extraction, modeling and synthesis is studied and applied to high-definition video coding in this paper. Film grain noise enhances the natural appearance of pictures in high-definition video and should be preserved in coded video. However, the coding of video contents with film grain noise is expensive. In previous art, it was proposed to enhance the coding performance by extracting film grain noise from the input video at the encoder as a preprocessing step, and by resynthesizing and adding it back to the decoded video at the decoder as a postprocessing step. In a novel implementation of this approach, we first remove film grain noise from image/video with a variational denoising approach without distorting its original content. Then, we present a parametric model (consisting of a small set of parameters) to generate film grain noise that is close to the actual one in terms of a couple of observed statistical properties, such as the power spectral density and the crosschannel spectral correlation. Under this framework, the coding gain of denoised video is higher while the visual quality of the final reconstructed video is well preserved. Experimental results are provided to demonstrate the superior performance of the proposed approach.

The design of robust fingerprinting systems for traitor tracing against time-varying collusion attacks in protecting continuous media, such as audio and video, is investigated in this research. We first show that it can be formulated as a multiuser detection problem in a wireless communication system with a time-varying channel response. Being inspired by the multicarrier code-division multiaccess technique, we propose a fingerprinting system that consists of three modules: 1) codeword generation with a multicarrier approach, 2) colluder weight estimation (CWE), and 3) advanced message symbol detection. We construct embedding codes with code spreading followed by multicarrier modulation. For CWE, we show that the weight estimation is analogous to channel response estimation, which can be solved by inserting pilot signals in the embedded fingerprint. As to advanced message symbol detection, we replace the traditional correlation-based detector with the maximal ratio combining detector and the parallel interference cancellation multiuser detector. The superior performance of the proposed fingerprinting system in terms of number of users/identified colluders and the bit-error probability of symbol detection is demonstrated by representative audio and video examples.

Scalable video representation of the state-of-art scalable extension of the H.264/MPEG-4 AVC (SVC) and its combined 3d scalability feature offer an excellent solution to flexible video multicast over IP network. In order to relieve the impact of the severe bandwidth fluctuations and packet loss to the reconstructed quality in heterogeneous networks like the internet, a GOP-adaptive layer-based priority ordering algorithm for SVC is proposed in this paper. The method ordered the SVC bit stream according to the rate distortion contribution of different layer to the whole performance within a GOP, which makes the transmission more efficient and robust under the same bandwidth condition. Simulation results of different sequences are given to demonstrate that the proposed algorithms offer better performance in video quality as compared the default SVC ordering method and the SNR-based ordering method.

We propose to use the Rake receiver to improve the system performance by collecting more signal power in an ultra-wideband (UWB) system precoded by the channel phase information, called the CPP-UWB system [1], in this work. The performance of the proposed system is determined by the selected channel taps and the corresponding combining schemes for those Rake fingers. Three combing schemes, Le., the maximum ratio combining (MRC), the minimized mean square error (MMSE) and the zero-forcing (ZF), are proposed. When the MRC scheme is applied to the Rake receiver with M fingers, M channel taps can be easily found to maximize the averaged signal power at the receiver output by exploiting the property of the channel- phase-precoded channel. It is shown by simulation results that the proposed scheme significantly outperforms the preceding system using one matched filter as proposed in [1].

A novel, practical and low-complexity multi-cell OFDMA downlink channel assignment method using a graph- based approach is proposed in this work. The inter-cell interference (ICI) information is obtained through inference from the diversity set of mobile stations (MSs) and presented in the form of an interference graph. The proposed downlink channel assignment method consists of two phases. The task of ICI reduction is mapped to the MAX fe-CUT problem in graph theory and solved in the first phase. Then, channel assignment is conducted by taking into account instantaneous channel conditions in the second phase. State-of-the-art ICI management techniques such as ICI coordination (ICIC) and base station cooperation (BSC) are incorporated in our framework. Heuristic algorithms are proposed to solve both phases of the problem efficiently. Simulation is conducted to demonstrate the effectiveness of the proposed solution, where the SINR improvement can be as high as 4.5 dB. The proposed solution can be used in next generation cellular systems such as 3 GPP long term evolution (LTE) and IEEE 802.16 m.

The performance of several unstructured peer-to- peer (P2P) content discovery techniques over ad hoc networks was analyzed in this work. They include: query flooding, expanding ring search, random walk and Bloom filter(BF)-based probabilistic routing. The chosen performance metrics are the query success rate, the route stretch and the search cost. Mathematic analysis is conducted to predict their behavior in static ad hoc networks. Finally, extensive computer simulations is performed to validate our analytical results in the ad hoc network. It is concluded that the BF-based probabilistic routing outperforms flooding-based and random walk schemes in finding a good balance among various performance metrics. Its only potential disadvantage is that the control packet size increases as the number of shared objects increases, which may not impose a severe constraint on a middle-sized ad hoc network.

The design of optimal opportunistic multicast scheduling (OMS) that maximizes the throughput of cellular networks is investigated. In a cellular network, the base station (BS) transmits the same information to multiple users within a cell. Recently, Gopala et al. proposed a median-user static OMS scheme in which a BS selects and broadcasts to the best 50% users in each transmission until all users are served. This scheduling scheme has been shown to exploit both the multiuser diversity and the multicasting gain simultaneously. In this work, we examine the optimal user selection ratio in each transmission. Based on extreme value theory, we first derive the limiting distribution of the maximum decodable rate of the M-th best user and the corresponding average number of transmissions required to serve all users. Then, the optimal selection ratio is established by maximizing the average network throughput. Furthermore, a dynamic selection algorithm is proposed to adjust the user selection ratio adaptively in each transmission. It is shown by simulation results that the OMS scheme with optimized static and dynamic selected ratios outperform the conventional unicast, multicast and the median-user static OMS schemes.

Defining suitable features for environmental sounds is an important problem in an automatic acoustic scene recognition system. As with most pattern recognition problems, extracting the right feature set is the key to effective performance. A variety of features have been proposed for audio recognition, but the vast majority of the past work utilizes features that are well-known for structured data, such as speech and music, and assumes this association will transfer naturally well to unstructured sounds. In this paper, we propose a novel method based on matching pursuit (MP) to analyze environment sounds for their feature extraction. The proposed MP-based method utilizes a dictionary from which to select features, resulting in a representation that is flexible, yet intuitive and physically interpretable. We will show that these features are less sensitive to noise and are capable of effectively representing sounds that originate from different sources and different frequency ranges. The MP- based feature can be used to supplement another well-known audio feature, i.e. MFCC, to yield higher recognition accuracy for environmental sounds.

Previous work demonstrated that the improvements in spectral efficiency and link reliability can be obtained in wireless relay networks employing terminals with MIMO capability. In order to take the advantage of MIMO relay networks while mitigate the complexity of decoding procedure, we propose a scheme that applies ZF relaying at the relay terminals and utilizes MMSE V-BLAST receiver at the destination terminal for two- hop half-duplex coherent MIMO relay networks. Considering the channel estimation is often imperfect due to the noise and interference, this paper extends the V-BLAST receiver design to the imperfect channel state information (CSI), assuming the channel estimation errors are independent complex Gaussian with zero mean and known second-order statistics. A robust MMSE nulling operator and its corresponding symbol detection ordering criterion is proposed for the MMSE V-BLAST receiver, which aims at providing, in each layer, the minimum average mean square error (MSE), over the random channel errors. The proposed ordering rule achieves global average MSE optimization which results in better performance than the previously proposed robust ZF V-BLAST. It has been shown that the proposed design exploits the characteristics of MIMO relay networks to provide good performance with respect to imperfect CSI while keeping a low complexity of the decoding procedure.

We propose to use carrier interferometry (CI) codes to reduce the complexity of the maximum- likelihood multiuser-detection (ML-MUD) receiver in MC-CDMA systems with carrier frequency offsets (CFOs). Complexity reduction is achieved by exploiting the sparsity of the cross-correlation matrix of CI codes. It is shown that, for a fully-loaded MC-CDMA with spread gain N over a multipath channel with length L, the complexity of the ML-MUD receiver grows exponentially with 2L-1 (instead of N), where L << N in a typical setting. Moreover, an upper bound for the minimum error probability is derived. Finally, simulation results are given to corroborate theoretical results.

A novel block-based intra-prediction scheme is proposed for efficient image (or intra-frame) coding, where we apply various 2D geometrical manipulations to reference image blocks to enrich the pool of prediction blocks for a given target block. As compared with the traditional line-based intra prediction in H.264/AVC, the new scheme offers a significant coding gain (about 0.24∓1.23dB in the PSNR value with the same bit rate) at the cost of higher complexity. Several techniques to reduce the search complexity are also discussed.

A robust technique for latent fingerprint image segmentation and enhancement is presented. In contrast with most state-of-the-art methods, our approach does not rely on the information of local gradients, which are sensitive to structured and unstructured background noise. Thus, the proposed technique is robust against gradient deviations. It also provides robust estimates to orientations and frequencies of fingerprints in a local region to allow effective Gabor filtering for fingerprint ridge/valley pattern enhancement. Results of latent fingerprint segmentation and enhancement along with orientation estimations on real data are presented to demonstrate the superior performance of the proposed technique.

Complexity modeling of spatial-temporal compensations in H.264/AVC decoding is performed by examining a rich set of inter- and intra-prediction modes. Specifically, we study the relationship between motion vectors, frame sizes and properties of reference frames since they are related to cache management efficiency. The proposed models provide good estimation results for test video bit streams over a wide range of bit rates. As an application, an H.264/AVC encoder equipped with these models estimates the decoding complexity and chooses the best inter- or intra-prediction mode to meet the decoding complexity constraint of a target decoding platform. The decoding complexity of the resultant bit stream is reduced at the cost of small PSNR loss.

This paper presents a synthesis-based texture coding technique that uses low-quality video as side information to control the output texture for video compression. As compared with the current pure synthesis algorithm, the proposed algorithm is generic, in the sense that the behavior and quality of the output texture can be adjusted by the amount of the side information and determined by the user. We develop an area-adaptive side information assignment technique to improve coding efficiency. Additionally, we present a fast-searching algorithm to reduce computational complexity. Simulations demonstrate the performance of the proposed technique.

In this work, we examine a family of anti-collusion codes using the principle of "orthogonal spreading followed by the inverse Fourier transform (OSIFT)" as introduced in (B.-H. Cha and C.-C. Jay Juo, 2007). The Hadamard-Walsh (HW) codes were adopted as the orthogonal spreading codes in (B.-H. Cha and C.-C. Jay Juo, 2007). Here, we introduce another type of orthogonal spreading codes; namely, the carrier interferometry (CI) codes, which are comparable to HW codes. To detect OSIFT hiding codes against weighted collusion attacks, we propose advanced detectors that can identify a large number of colluders with weighted collusion attacks. The proposed detection schemes include the maximal ratio combining (MRC) scheme for colluder detection in the same group and the multiuser detection (MUD) scheme for colluder detection in different groups. It is demonstrated that HW- and Cl-based OSIFT codes have robust performance with advanced detection schemes against weighted collusion attacks compared with conventional matched filter (MF) in an exemplary audio watermarking system.

An efficient music representation based on the matching pursuit (MP) technique with content-adaptive dictionaries (CADs) is investigated in this work. The Gabor atoms are commonly adopted in the MP-based signal representation due to their excellent time-frequency localization property. However, the Gabor dictionary may not yield a concise representation for music signals. Music signals have special characteristics which are specified by pitches and durations of music notes. In this work, we exploit music characteristics to create a set of content-adaptive elementary functions called atoms that efficiently capture the inherent structure of music signals. As a result, we are able to project musical signals onto a subspace spanned by atoms from CAD for a concise and efficient representation. The proposed CAD representation technique is applied to music enhancement with noisy background to demonstrate the power of the proposed representation.

A complexity model for context-based adaptive variable length coding (CAVLC) and universal variable length coding (UVLC) in the H.264/AVC decoder is proposed. CAVLC and UVLC are used to code source data, i.e., quantized transformed coefficients (QTCs), and header data, respectively. CAVLC and UVLC are important in the baseline profile of the H.264/AVC codec since this profile targets at low complexity and low cost applications so that only CAVLC and UVLC are allowed to be the entropy coding tools. This model is verified experimentally. Possible applications of this model are also described. Since the entropy decoding complexity becomes significant in the high bit rate video due to more non-zero QTCs and motion vectors (MVs), one application scenario is to integrate this model into the H.264 encoder so that the encoder can estimate the possible decoding complexity and then select proper coding modes to generate the decoder-friendly bit streams, which are easy to decode for a particular decoding platform.

We study the problem of automatic musical beat tracking from acoustic data, i.e., finding locations of beats of a music piece by computers on-the-fly, in this work. An on-line musical beat tracking algorithm based on Kalman filtering (KF) with an enhanced probability data association (EPDA) method is proposed. The beat tracking algorithm is built upon a linear dynamic model of beat progression, to which the Kalman filtering technique can be conveniently applied. The beat tracking performance can be seriously degraded by noisy measurements in the Kalman filtering process. Three methods are presented for noisy measurements selection. They are the local maximum (LM) method, the probabilistic data association (PDA) method and the enhanced PDA (EPDA) method. We see that the performance of EPDA outperforms that of LM and PDA significantly.

Robust H.264/SVC video multicast with joint network coding (NC) and application-layer forward error correction (AL-FEC) is presented in this work. Unlike traditional AL-FEC that protects the video source only, the NC scheme provides another level of protection by allowing the mixing of transmitted packets in the intermediate nodes of the network. The joint NC/AL-FEC solution simplifies the construction and maintenance of multicast trees greatly, and enables efficient video delivery to heterogeneous receivers. It is shown by computer simulation that the proposed joint NC/AL-FEC scheme outperforms the traditional store-and-forward (S/F) network with AL-FEC only by a large margin.

We present a set of techniques to simplify tree models for faster rendering while retaining their visual resemblance to the original model. This goal is achieved by setting a budget constraint of how many leaves shall be rendered, and we select leaves with higher possibilities to be visible to generate a simplified model for rendering. We first examine how leaf objects can be prioritized to provide a tree model suitable for a single viewing angle. The camerapsilas projected screen space is partitioned into several small regions to ensure that each region is guaranteed to be filled with a certain number of leaves. Each region selects higher priority leaves by utilizing the leaf objectspsila spatial relationship to the camera. Then, an iterative algorithm is proposed to allocate leaf objects at different viewing angles. The algorithm chooses different viewing positions for leaf allocation at each iteration with the goal of maximizing the resemblance to the original tree model at all possible angles. Three criteria taken in consideration for viewing angle selection are: distance, leaf normal, and unfilled pixel. Experimental results are provided to demonstrate time efficiency of the proposed techniques at the cost of little visual degradation.

We propose a model-based spatial layer bit allocation algorithm for H.264/SVC in this work. The spatial scalability of H.264/SVC is achieved by a multi-layer approach, where an enhancement layer is bound by the dependency on its preceding layers. The inter-layer dependency is decoupled in our analysis by a careful examination of the signal flow in the H.264/SVC encoder. We show that the rate and the distortion (R-D) characteristics of a dependent layer can be represented by a number of independent functions with a group of pictures (GOP) as a basic coding unit. Finally, a low complexity spatial layer bit allocation scheme is developed using the proposed GOP-based R-D models. It is shown by experimental results that our proposed bit allocation algorithm can achieve the coding performance close to the optimal R-D performance of full search and is significantly improved from current reference software JSVM.

In this paper, we examine the uplink of a cooperative CDMA network, where users cooperate by relaying each other's messages to the base station. When spreading waveforms are not orthogonal, multiple access interference (MAI) exists at the relays and the destination, causing cooperative diversity gains to diminish. To address this issue, we adopt the multiuser detection (MUD) technique to mitigate MAI in achieving the full advantages of cooperation. Specifically, the relay-assisted decorrelating multiuser detector (RAD-MUD) is proposed to separate interfering signals at the destination with the help of preceding at the relays along with pre-whitening at the destination. Unlike the conventional zero-forcing (ZF) precoder or the decorrelating MUD, the proposed RAD-MUD experiences neither power expansion at the relays nor noise amplification at the destination. Three cooperative transmission strategies are considered on top of RAD-MUD; namely, transmit beamforming, selective relaying and distributed space-time coding. Since the reliability of each source-relay and/or relay-destination links are different, relay transmissions are weighted accordingly in our schemes to further combat MAI. The advantages of RAD-MUD over ZF precoding and other existing cooperative MUD schemes are shown through computer simulations.

An error resilient video streaming scheme that transmits offline coded H.264/AVC video through erroneous channels, called the alternative macroblock coding (AMC) scheme, is proposed in this work. In the AMC scheme, each macroblock can be reconstructed from one default and several alternative versions coded using different predictions. During the transmission, the sender tracks ACK or NACK messages from the receiver to detect transmission errors. When the reference used in the default version of a macroblock that has not been transmitted is corrupted by errors, one of its alternative versions that has the correctly received reference is selected to replace the default version in the output bit stream to stop error propagation. The AMC scheme is designed to ensure that the bit stream replacement during streaming will not cause significant mismatch at the decoder end and neither introduce large bit rate overhead to the transmitted bit stream. Furthermore, adaptive quantization selection and bit stream replacement methods are developed to improve the rate-distortion performance of received video. It is demonstrated by experimental results that the AMC scheme is effective in reducing error propagation in offline coded H.264/AVC video.

In this research, we first investigate the cross-layer interaction between TCP and routing protocols in the IEEE 802.11 ad hoc network. On-demand ad hoc routing protocols respond to network events such as channel noise, mobility, and congestion in the same manner, which, in association with TCP, deteriorates the quality of an existing end-to-end connection. The poor end-to-end connectivity deteriorates TCP's performance in turn. Based on the well-known TCP-friendly equation, we conduct a quantitative study on the TCP operation range using static routing and long-lived TCP flows and show that the additive-increase, multiplicative-decrease (AIMD) behavior of the TCP window mechanism is aggressive for a typical multihop IEEE 802.11 network with a low-bandwidth-delay product. Then, to address these problems, we propose two complementary mechanisms, that is, the TCP fractional window increment (FeW) scheme and the Route-failure notification using BUIk-losS Trigger (ROBUST) policy. The TCP FeW scheme is a preventive solution used to reduce the congestion-driven wireless link loss. The ROBUST policy is a corrective solution that enables on-demand routing protocols to suppress overreactions induced by the aggressive TCP behavior. It is shown by computer simulation that these two mechanisms result in a significant improvement of TCP throughput without modifying the basic TCP window or the wireless MAC mechanisms.

In this paper, we propose a generic point cloud encoder that provides a unified framework for compressing different attributes of point samples corresponding to 3D objects with arbitrary topology. In the proposed scheme, the coding process is led by an iterative octree cell subdivision of the object space. At each level of subdivision, positions of point samples are approximated by the geometry centers of all tree-front cells while normals and colors are approximated by their statistical average within each of tree-front cells. With this framework, we employ attribute-dependent encoding techniques to exploit different characteristics of various attributes. All of these have led to significant improvement in the rate-distortion (R-D) performance and a computational advantage over the state of the art. Furthermore, given sufficient levels of octree expansion, normal space partitioning and resolution of color quantization, the proposed point cloud encoder can be potentially used for lossless coding of 3D point clouds.

Two new techniques are proposed to improve stereo matching performance in this work. First, to address the disparity discontinuity problem in occluded regions, we present a disparity estimation procedure, which consists of two steps; namely, a greedy disparity filling algorithm and a least-squared-errors (LSE) fitting method. Second, it is observed that the existing fronto-parallel model with color segmentation is built upon the piecewise constant surface approximation, which is however not efficient in approximating slanted or curved objects. We use a piecewise linear surface model to represent 3-dimensional (3D) geometric structure for better surface modeling. The proposed stereo matching system with these two new components is evaluated with Middlebury data sets with excellent quantitative and qualitative results.

A distributed medium access control (MAC) algorithm for uplink OFDMA networks under the IEEE 802.16 framework is proposed and analyzed in this work. We present a simple yet efficient algorithm to enhance the system throughput by integrating opportunistic medium access and collision resolution through random subchannel backoff. Consequently, the resulting algorithm is called the opportunistic access with random subchannel backoff (OARSB) scheme. OARSB not only achieves distributed coordination among users but also reduces the amount of information exchange between the base station and users. The throughput and delay performance analysis of OARSB is conducted using a Markov chain model. The superior performance of OARSB over an existing scheme is demonstrated by analysis as well as computer simulation.

The channel phase precoding (CPP) technique was recently proposed for the ultra-wideband (UWB) communication system in [1] to save the feedback overhead and the computational complexity as compared with the time-reversal prefllter (TRP) technique [2]. Two ideal assumptions have been made in both systems; namely, the availability of accurate channel information and perfect synchronization of transmitted pulses. In this work, we examine the impact of timing jitter on the performance of TRP and CPP and show that the CPP-UWB system is more robust against the timing jitter than the TRP- UWB svstem.

Previous work demonstrated that the improvements in spectral efficiency and link reliability can be obtained in wireless relay networks employing terminals with MIMO capability. We propose to apply the non layered and layered ZF (V-BLAST) receiver to the relay and destination terminals in coherent MIMO relay networks under two-hop half-duplex relaying. Considering the channel estimation is often imperfect due to the noise and interference, this paper assumes that no perfect channel state information (CSI) is available for both backward and forward channels. The V-BLAST receiver is extended to the imperfect CSI under the assumption that the channel estimation errors are independent complex Gaussian random variables with zero mean and known second-order statistics. A robust symbol detection ordering criterion is proposed for applying ZF V-BLAST to the MIMO relay networks. This new ordering rule aims at providing, in each layer, the maximum average signal to interference- plus-noise ratio (SINR), over the random channel errors. Our simulations show that aided by the new ordering criterion, the suboptimal ZF operators, directly resulted from the imperfect CSI, is capable of alleviating the effect of CSI errors and improving the robustness of V-BLAST system.

In this paper, we present an enhanced adaptive frequency hopping (EAFH) mechanism for improving the performance of frequency hopping-based wireless personal area networks (WPANs) under frequency-static and frequency-dynamic interference. The proposed mechanism monitors the overall packet error rate (PER) of the system to determine the right number of channels to be excluded from the hopset. Then based on the PER of individual channel, it decides whether to exclude a certain channel or not. Finally, proper packet length is associated with those channels remaining in the hopset. These decisions, which pertain to hopset size and packet length, are made so as to optimize the performance of the hopping system in a coexistence environment. We developed an analytical model to justify the behavior and performance of the proposed mechanism. Simulations are conducted under an environment of some collocated Bluetooth (BT) piconets and a Wi-Fi network to validate the developed model and show the superiority of EAFH. Simulation results show that, compared with those existing mechanisms including orthogonal hopset-based mechanisms, EAFH could provide much higher throughput while still maintaining reasonably good channel occupancy.

The use of multi-user detection (MUD) in a cooperative CDMA network is investigated for the uplink in synchronous CDMA systems. Suppose that, at any instant in time, part of the users serve as sources while the others serve as relays. The proposed MUD scheme decorrelates the sources' messages at the destination with the help of precoding at the relays. Three cooperation methods are considered: (1) transmit beamforming, (2) selective relaying and (3) distributed space- time coding. The optimal weighting factors of each method are determined by taking the quality of the source-to-relay and/or the relay-to-destination links into account. We show that significant improvements in terms of the spatial diversity and multiple-access interference (MAI) mitigation can be attained when precoding is employed at the relays to aid the decorrelation at the destination. The advantages are even more pronounced when selective relaying is combined with the other two schemes.

The optimal deployment of sinks in a sensor region for power efficient data gathering of a physical phenomenon is investigated in this work. In the system of consideration, nodes perform lossless distributed coding of the sensed data and the spatial statistics of the monitored phenomenon are possibly nonstationary due to heterogeneity in the sensing environment (e.g. variations in the altimetric profile). Non-stationary spatial statistics lead to uneven spatial profiles of the bit rates, unlike stationary statistics. The properties of rate profiles and the consequent optimal sink locations for a broad class of spatially non-stationary covariance models are studied by mathematical analysis and numerical examples.

A robust and flexible wireless video multicast system with the H.264 scalable video coding (SVC) coding format and network coding (NC) is proposed in this work. The system enables efficient video streaming to heterogeneous receivers in packet erasure channels. NC is used to simplify erasure protection as well as the complexity of multicast tree construction and maintenance. It is shown by simulation results that the proposed solution improves PSNR over the traditional video multicast with error correction codes (ECC) in store-and-forward networks.

A new methodology to design collusion-free hiding codes using the multi-access-interference-free (MAI-free) principle is proposed. A precoding technique was recently introduced in orthogonal frequency division multiplexing (OFDM) wireless communication systems to result in MAI-free codes, which simplifies the receiver design greatly. Being motivated by this work, we propose a new class of collusion-free codes called the OSIFT (orthogonal spreading followed by the inverse Fourier transform) codes. It is demonstrated by computer simulation that OSIFT codes have more robust performance against collusion attacks than existing collusion-free codes in an exemplary audio watermarking system.

Three power control schemes for the space-time coded amplify-and-forward (AF) relaying scheme targeting at wireless sensor network applications are examined and compared. The opportunistic scheme performs the best by considering the signal-to-noise-ratio (SNR) of the received signal. However, if the power for the relay is limited, the performance of the opportunistic scheme degrades due to the loss of active relay nodes that have better channel conditions. Since the battery lifetime of nodes for wireless sensor networks is limited and the loss of relay nodes is critical to system performance, we propose an SNR-constrained power reduction scheme to prolong the relay lifetime for the opportunistic scheme. It is demonstrated by computer simulation that the opportunistic scheme with SNR-constrained power reduction is power efficient and the relay lifetime of dense relay networks can be significantly prolonged.

A modified Kalman filtering approach to on-line music beat tracking is proposed in this work. The proposed algorithm first detects onset signals from the acoustic waveform. Then, it tracks the beat and the tempo using the Kalman filter. Two techniques (i.e., observation smoothing and lock detection) are adopted to improve the tracking performance furthermore. It is shown by experimental results that the proposed modified Kalman filtering approach provides satisfactory beat tracking performance.

A rate-distortion (R-D) model is derived to investigate the potential coding gain from the use of a larger macroblock size for high definition (HD) video coding in this work. Coding rates of using different block sizes for HD video coding are compared with the R-D model, and limitations of the current coding techniques that might impede the additional R-D gain of using enlarged MB for HD coding are explained. Experiments show that an average bit rate gain of 27 % across QP range can be achieved by using enlarged MB and extended motion search range.

A complexity model for motion compensation in the H.264/AVC decoder is proposed. This model considers a rich set of inter prediction modes allowed by H.264 as well as the relationship of motion vectors (MVs), which are related to cache miss performance. This model is verified by experimental results. Possible applications of the complexity model are also described. One application scenario is to integrate it in an H.264 encoder so that the encoder can estimate the decoding complexity and choose the best inter prediction mode to meet the complexity constraint of the target decoding platform.

A view-dependent visibility estimation technique for tree models is proposed in this work. While most previous work focused on visibility estimation of large objects in architectural walkthroughs, we consider the visibility estimation problem of tree leaves in complex natural scenes such as forests. Exact calculation of how each leaf object blocks another from a given viewpoint is difficult due to the large amount of computational cost. To address this issue, we propose three simple yet effective methods for efficient visibility estimation of tree leaves: the distance-based, the normal-based, and the layer-plus-normal-based methods. Experimental results show that the distance-based method always produces the best analytical performance, the normal-based method works better in preserving volumetric visual quality at low budget constraints, and the layer-plus-normal-based method preserves volumetric visual quality while yielding excellent analytical performance.

A new approach to musical onset detection by detecting the significant change of joint energy and phase features is proposed in this work. We modify the phase deviation method in (Duxbury et al., 2003) to yield a more robust phase feature. To capture the energy deviation of an audio waveform along time, we compute the error of a forward linear prediction filter as the energy feature. Several ways to fuse obtained energy and phase features are discussed and compared. It is demonstrated by experimental results that a proper scheme to integrate energy and phase features can provide excellent onset detection results.

A rate-distortion (RD) model for subpel motion vector (MV) selection is proposed for power-efficient video coding in this work. To reduce the encoder complexity, the block texture feature is used in the proposed RD model to predict the potential subpel MV gain for a given block. If the gain is not sufficiently large, we may choose integer or half-pel MV to save the search and interpolation cost at the encoder. Quantization noise and external noise are included in the model to improve the prediction accuracy furthermore. The resultant model is shown to be reliable in predicting the RD gain for different subpel resolutions. It is shown by experimental results that the proposed subpel MV selection scheme can maintain the RD performance similar to full subpel MV while reducing encoder complexity significantly.

Video encoding to yield a decoder-friendly H.264 bit stream that consumes less decoding power yet with little coding efficiency degradation is investigated in this work. The energy saving of the decoder relies on the use of adaptive deblocking filters (ADF). We first propose a power consumption model for the deblocking filter. Then, the encoder performs the rate-distortion-decoder complexity optimization (RDC) to save the decoder power needed for deblocking filter operations, which is called the decoder-friendly adaptive deblocking filter (DF-ADF) mode decision. The RDC optimization framework presents a way to balance coding efficiency and the ADF decoding cost in the mode decision process. The effectiveness of the proposed DF-ADF algorithm is demonstrated by experiments with diverse video contents and bit rates.

A new scheme for disparity vector (DV) estimation and virtual view synthesis to generate 3D video display from a pair of stereo video inputs is investigated in this work. Two performance metrics are considered for the algorithmic evaluation; i.e. quality and complexity. To enhance the overall performance, a two-stage algorithm for accurate and fast DV estimation and occlusion handling is first presented. Then, a preprocessing algorithm and a synthesis method are described. The proposed preprocessing algorithm can remove false matched regions for DV refinement effectively. The new synthesis method can reduce blurring and ghostly effects greatly. It is demonstrated by experimental results that the proposed algorithm offers better virtual view quality at a much lower complexity than existing methods.

In this work, a simplified rate control algorithm is proposed for non-conversational H.264 video applications. When a GOP (group of pictures) structure is imposed on an encoded video, a bit rate control and a bit allocation among frames are two important components of a rate control algorithm. We employ a GOP rate model and the monotonicity property to address the bit rate control and the frame-layer bit allocation problem, respectively. Since the proposed algorithm employs only a GOP rate model, the rate control mechanism can be greatly simplified while it achieves better video quality as compared with that in the H.264 JM8.6 reference encoder.

An improved motion compensation decoding complexity model is proposed and its application to H.264/AVC decoding complexity reduction is examined in this work. This complexity model considers a rich set of inter prediction modes of H.264 as well as the relationship between motion vectors and frame sizes, which turn out to be highly related to the cache management efficiency. An H.264 encoder equipped with the complexity model can estimate the decoding complexity and choose the best inter prediction mode to meet the decoding complexity constraint of the target receiver platform. The performance of the proposed complexity model and its application to video decoding complexity reduction are demonstrated experimentally.

A high definition video coding technique using super-macroblocks is investigated in this work. Our research is motivated by the observation that the macroblock-based partition in H.264/AVC may not be efficient for high definition video since the maximum macroblock size of 16 x 16 is relatively small against the whole image size. In the proposed super-macboblock based video coding scheme, the original block size MxN in H.264 is scaled to 2Mx2N. Along with the super-macroblock prediction framework, a low-complexity 16 x 16 discrete cosine transform (DCT) is proposed. As compared with the 1D 8 x 8 DCT, only 16 additions are added for a 1D 16 points 16 x 16 DCT. Furthermore, an adaptive scheme is proposed for the selection the best coding mode and best transform size. It is shown by experimental results that the super-macroblock coding scheme can achieve a higher coding gain.

A new technique for film grain noise extraction, modeling and synthesis is proposed and applied to the coding of high definition video in this work. The film grain noise is viewed as a part of artistic presentation by people in the movie industry. On one hand, since the film grain noise can boost the natural appearance of pictures in high definition video, it should be preserved in high-fidelity video processing systems. On the other hand, video coding with film grain noise is expensive. It is desirable to extract film grain noise from the input video as a pre-processing step at the encoder and re-synthesize the film grain noise and add it back to the decoded video as a post-processing step at the decoder. Under this framework, the coding gain of the denoised video is higher while the quality of the final reconstructed video can still be well preserved. Following this idea, we present a method to remove film grain noise from image/video without distorting its original content. Besides, we describe a parametric model containing a small set of parameters to represent the extracted film grain noise. The proposed model generates the film grain noise that is close to the real one in terms of power spectral density and cross-channel spectral correlation. Experimental results are shown to demonstrate the efficiency of the proposed scheme.

Discrete power allocation strategies for amplify- and-forward cooperative networks are proposed based on selective relaying methods. The goal of power allocation is to maximize the network lifetime, which is defined as the duration of time for which the outage probability at the destination can be maintained above a certain level. The discrete power levels enable a low cost implementation and a close integration with high speed digital circuits. We propose three power allocation strategies that take into consideration both the channel state information (CSI) and the residual energy information (REI) at each node. By modeling the residual energy of each node as the states of a Markov Chain, we are able to derive the network lifetime analytically by computing the expected number of transitions to the absorbing states, i.e., the energy states for which the outage probability is no longer achievable. The performance of the three strategies are compared through numerical simulations and a significant improvement in network lifetime is shown, when compared with the case considering only the local CSI.

The throughput performance of adaptive multi-input multi-output orthogonal frequency division multiplexing (MIMO-OFDM) system without channel station information (CSI) in priori is investigated in this research. First, throughput scaling factor (pre-log factor) of an nttransmit antennas, nrreceive antennas MIMO-OFDM with total N subcarriers is asymptotically equal to n* (1- n* L/(MN)), where n* = min(nt, nr) and M is the number of OFDM blocks experienced a constant channel. Second, feedback directed adaptive schemes can achieve the pre-log factor of the lower bound. Third, diversity-multiplexing gain tradeoff of variable-rate systems has been shown to have the same behavior as MIMO systems employed with random Gaussian codewords. Finally, it is shown that throughput of OFDMA with opportunistic scheduling performs better than traditional OFDM-TDMA scheduling with constant enhancement independent of SNR in multiuser downlink environment.

The performance analysis and transceiver design of an adaptive bit-interleaved-coded multi-input multi-output orthogonal frequency division multiplexing (BIC-MIMO-OFDM) system without channel station information (CSI) in priori are investigated in this research. First, the noncoherent capacity of a MIMO-OFDM system with TV subcarriers, nttransmit antennas and nrreceive antennas is analyzed. It is shown that the lower bound of noncoherent capacity in the high signal-to-noise ratio (SNR) region is asymptotically equal to n*(1 - n*L/N) log rho, where n* = min(nl,nr), L is the number of channel taps and rho is the SNR value. Then, an adaptive BIC-MIMO-OFDM system based on transmitter beamforming and variable-rate-variable-power (VRVP) QAM is proposed. The throughput of adaptive scheme can achieve the noncoherent capacity, however, the gap to the coherent capacity cannot be totally compensated due to the overhead of training. The effect of imperfect channel state information (CSI) feedback is studied. It is demonstrated by numerical simulation that the proposed BIC-MIMO-OFDM scheme has better performance than adaptive trellis coded modulation (TCM) when outdated channel estimation is used.

The MAC protocol of a cognitive radio (CR) device should allow it to access unused or under-utilized spectrum without (or with minimal) interference to primary users dynamically. To fulfill such a goal, we propose a cognitive MAC protocol using statistical channel allocation and call it SCA-MAC in this work. SCA-MAC is a CSMA/CA-based protocol, which exploits statistics of spectrum usage for decision making on channel access. For each transmission, the sender negotiates with the receiver on transmission parameters through the control channel. A model is developed for CR devices to evaluate the successful rate of transmission. A CR device should pass the threshold of the successful transmission rate via negotiation before it can begin a valid transmission on data channels. The operating range and channel aggregation are two control parameters introduced to maintain the MAC performance. To validate our ideas, we conducted theoretical analysis and simulations to show that SCA-MAC does improve the throughput performance and guarantee the interference to incumbents to be bounded by a predetermined acceptable rate. The proposed MAC protocol does not need a centralized controller, as the negotiation between the sender and the receiver is performed using the CSMA/CA-based algorithm.

Power allocation strategies are devised to maximize the network lifetime of amplify-and-forward (AF) cooperative networks. The paper considers the scenario where one source and multiple partners cooperate to transmit messages to the destination. The powers emitted by the users are subject to the SNR requirement at the destination. First, the power allocation strategy that demands the minimum instantaneous aggregate transmit power of all cooperating partners is described and analyzed. The optimal solution results in a form of selective relaying; namely, the user with the best channel condition is selected to help in relaying the message. However, this instantaneous power minimization strategy does not necessarily maximize the lifetime of battery-limited systems. Then, three AF cooperative schemes were proposed to exploit the channel state information (CSI), the residual battery energy and the QoS requirement. It is shown that the network lifetime can be extended considerably by taking all these three factors into account.

A cross-layer approach to contour nodes inference of monitored physical phenomena with data fusion in wireless sensor networks is investigated in this work. The authors first analyze three sources of signal distortion: sensing noise, data quantization error and data communication noise. The sensing noise is often negligible. We need to choose the proper data quantization levels to balance the communication cost and the desired signal quality. Besides, data fusion at a local fusion center can be used to mitigate the communication noise due to the poor wireless channel. Instead of making a hard binary decision, the probability for a sensor node to be a contour node is calculated at the local fusion center. An adaptive data fusion scheme is proposed to avoid excessive packet retransmissions. Simulation results are given to show the effects of different system parameters on the overall system performance.

Performance analysis of multiuser orthogonal frequency division multiplexing (OFDM-TDMA) and orthogonal frequency division multiple access (OFDMA) networks in support of multimedia transmission is conducted in this work. We take a cross-layer approach and analyze several quality-of-service (QoS) measures that incilude the bit rate and the bit error rate (BER) in the physical layer, and packet average throughput/delay and packet maximum delay in the link layer. The authors adopt a cross-layer QoS framework similar to that in IEEE 802.16, where service classification, flow control and opportunistic scheduling with different subcarrier/bit allocation schemes are implemented. In the analysis, the Rayleigh fading channel in the link layer is modeled by a finite-state Markov chain, and the channel state information (CSI) is assumed to be available at the base station. With the M/G/1 queueing model and flow control results, the analysis provides important insights into the performance difference of these two multiaccess systems. The derived analytical results are verified by extensive computer simulation. It is demonstrated by analysis and simulation that OFDMA outperforms OFDM-TDMA in QoS metrics of interest. Thus, OFDMA has higher potential than OFDM-TDMA in supporting multimedia services.

An efficient anonymous communication protocol, called MANET Anonymous Peer-to-peer Communication Protocol (MAPCP), for P2P applications over mobile ad-hoc networks (MANETs) is proposed in this work. MAPCP employs broadcasts with probabilistic-based flooding control to establish multiple anonymous paths between communication peers. It requires no hop-by-hop encrypt ion/decryption along anonymous paths and, hence, demands lower computational complexity and power consumption than those MANET anonymous routing protocols. Since MAPCP builds multiple paths to multiple peers within a single query phase without using an extra route discovery process, it is more efficient in P2P applications. Through analysis and extensive simulations, we demonstrate that MAPCP always maintains a higher degree of anonymity than a MANET anonymous single-path routing protocol in a hostile environment. Simulation results also show that MAPCP is resilient to passive attacks.

Orthogonal frequency division multiple access (OFDMA) has recently attracted vast research attention from both academia and industry and has become part of new emerging standards for broadband wireless access. Even though the OFDMA concept is simple in its basic principle, the design of a practical OFDMA system is far from being a trivial task. Synchronization represents one of the most challenging issues and plays a major role in the physical layer design. The goal of this paper is to provide a comprehensive survey of the latest results in the field of synchronization for OFDMA systems, with tutorial objectives foremost. After quantifying the effects of synchronization errors on the system performance, we review some common methods to achieve timing and frequency alignment in a downlink transmission. We then consider the uplink case, where synchronization is made particularly difficult by the fact that each user's signal is characterized by different timing and frequency errors, and the base station has thus to estimate a relatively large number of unknown parameters. A second difficulty is related to how the estimated parameters must be employed to correct the uplink timing and frequency errors. The paper concludes with a comparison of the reviewed synchronization schemes in an OFDMA scenario inspired by the IEEE 802.16 standard for wireless metropolitan area networks.

The performance of a multiuser orthogonal frequency-division multiplexing (OFDM) transceiver, called the precoded multiuser OFDM (PMU-OFDM), with time asynchronous access is investigated in this paper. It is shown that multiaccess interference (MAI) due to time asynchronism is asymptotically zero as the number of parallel input symbolsNbecomes large in the PMU-OFDM system. Then, we show that PMU-OFDM with even or odd Hadamard-Walsh codewords can significantly suppress the MAI effect due to time asynchronism. For the half-loaded case, no sophisticated signal processing technique is needed by PMU-OFDM for MAI suppression. For the fully loaded situation, PMU-OFDM demands less complexity than does conventional OFDMA for interference suppression since PMU-OFDM deals with only one half of interferers.

A fast feature-based intra/inter coding mode selection scheme for the H.264/AVC video coding standard is proposed in this paper. First, three features are extracted from a macroblock to form a feature space. Then, the feature space is partitioned into three regions, i.e., risk-free, risk-tolerable, and risk-intolerable regions, where the risk is calculated using the rate-distortion (RD) performance loss due to wrong mode decision as well as the probability distribution of inter/intra modes in the feature space obtained from an off-line training process. Depending on the region where the feature vector of a macroblock is located, we can apply mechanisms of different complexity for final mode decision. To calculate the likelihood function of the risk, both parametric and nonparametric density estimation schemes are developed to offer different rate-distortion-complexity tradeoffs. It is demonstrated by experimental results that the proposed algorithm can save approximately 20%-32% of the total encoding time of H.264 (JM7.3a) with little degradation in the rate-distortion performance

A new rate control scheme for H.264 video encoding with enhanced rate and distortion models is proposed in this work. Compared with existing H.264 rate control schemes, our scheme has offered several new features. First, the inter-dependency between rate-distortion optimization (RDO) and rate control in H.264 is resolved via quantization parameter estimation and update. Second, since the bits of the header information may occupy a larger portion of the total bit budget, which is especially true when being coded at low bit rates, a rate model for the header information is developed to estimate header bits more accurately. The number of header bits is modeled as a function of the number of nonzero motion vector (MV) elements and the number of MVs. Third, a new source rate model and a distortion model are proposed. For this purpose, coded 4 times 4 blocks are identified and the number of source bits and distortion are modeled as functions of the quantization stepsize and the complexity of coded 4 times 4 blocks. Finally, a R-D optimized bit allocation scheme among macroblocks (MBs) is proposed to improve picture quality. Built upon the above ideas, a rate control algorithm is developed for the H.264 baseline-profile encoder under the constant bit rate constraint. It is shown by experimental results that the new algorithm can control bit rates accurately with the R-D performance significantly better than that of the rate control algorithm implemented in the H.264 software encoder JM8.1a

A recursive algorithm for estimating and updating the effective carrier frequency offset (CFO) vector in a multicarrier code-division multiple-access (MC-CDMA) system is proposed in this work. The recursive relation is derived based on the expectation maximization (EM) algorithm with a quadratic constraint. This new approach enables the use of linear estimation theory to tackle the CFO estimation problem with or without training data, which leads to an analytic CFO estimate in closed form. Furthermore, the multiple access interference (MAI) is mitigated using the second order statistics of the interference-plus-noise vector, which is updated in a recursive manner under the EM formulation, too. When reaching a converged estimate, a fixed-norm quadratic constraint is imposed so that the final CFO estimate is robust to an imprecise covariance matrix estimate caused by insufficient data samples. It is demonstrated by computer simulation that the performance of an MC-CDMA system without the CFO information can be restored by the proposed scheme in the sense that its bit error probability (BEP) performance is close to that with perfect CFO knowledge

The problem of frequency synchronization, channel estimation, and data detection for all active users in the uplink of an OFDMA system is investigated in this work. Since the exact maximum likelihood (ML) solution to this problem turns out to be too complex for practical purposes, we derive an alternative scheme that operates in an iterative fashion. At each step, the superimposed signals arriving at the base station (BS) are separated by means of the space-alternating generalized expectation-maximization (SAGE) algorithm. Each separated signal is then passed to an expectation-conditional maximization (ECM)-based processor that updates frequency estimates and performs channel estimation and data detection for each user. The resulting architecture is reminiscent of the parallel interference cancellation (PIC) receiver, where interference is generated and removed from the received signal to improve the system performance. Simulations indicate that the proposed scheme outperforms other benchmark solutions at the price of increased computational complexity

Low-complexity low-rate super-orthogonal turbo codes (SOTC) are proposed in this work to replace the implicit repetition code (RC) in ultra-wide band impulse radio (UWB-IR) systems to improve the transmission range and system throughput. Various receivers, including the matched-filter and the RAKE receivers, are examined for data detection in the additive white Gaussian noise (AWGN) channel and the indoor IEEE 802.15.3a channels. The performance of SOTC-coded UWB with perfect and imperfect timing and channel information is analyzed and corroborated by computer simulation. It is demonstrated that the SOTC-based UWB-IR system can achieve significant performance improvement over the conventional direct-sequence UWB (DS-UWB) system encoded by RC over both ISI and ISI-free channels.

Recognition of shapes in images is an important problem in computer vision with application in various medical problems, including robotic surgery and cell analysis. The similarity measures for such purpose must be robust to various transformations and modest occlusions. Transformations, such as scaling and translation can be handled easily by techniques through data representations or similarity measures. Rotation invariance is an inherently more difficult problem and can be handled through data representation, but at the expense of poor discrimination. Approaches which provide excellent discrimination require a complexity of O(n3) for each shape comparison. In this paper, we present a framework that provides a speedup over the slow but accurate approaches. The algorithm is inspired by the iterative deepening framework in artificial intelligence, by examining the data at increasingly fine levels of approximation until it is either considered irrelevant or submits to the full calculations. Although we examine the data several times at different levels of abstractions, because the time required for the last iteration dwarfs all others, this apparent redundancy is inconsequential. We show that our method provides at least a 3-4 orders of magnitude in speedup without generating any false dismissals

The delay analysis and comparison of OFDM- TDMA and OFDMA using a flow control scheme under the QoS framework of IEEE 802.16 are conducted in this work. We investigate the maximum packet delay performance by deriving delay bounds for several multiaccess modes with different subcarrier/time-slot assignment and bit allocation schemes. The analytical delay bounds provide insights into the performance difference among different multiaccess schemes, and demonstrate each scheme's capability of supporting real-time, delay-sensitive multimedia applications. The derived analytical results are verified by computer simulation.

A preceding technique applied to symbols transmitted in an ultra-wideband (UWB) system was proposed, which can concentrate the signal power at the receiver to facilitate data detection. The design of the optimal codeword for preceding is investigated in this work. After the problem is formulated, we examine its solution in the full-rank space as well as the reduced-rank subspace to get different tradeoff between detection performance and the amount of feedback messages. Furthermore, we propose two subspace selection schemes that are suitable for different channel conditions. The codeword obtained from this work improves the performance of the result significantly at the price of a larger amount of feedback messages.

An efficient anonymous communication protocol, called MANET anonymous peer-to-peer communication protocol (MAPCP), for P2P applications over mobile ad-hoc networks (MANETs) is proposed in this work. MAPCP employs broadcasts with probabilistic flooding control to establish multiple anonymous paths between communication peers. It requires no hop-by-hop encryption/decryption along anonymous paths and, hence, demands lower complexity of computation and power consumption than other anonymous routing protocols for MANETs. Since MAPCP builds multiple paths to multiple peers within a single query phase without using an extra route discovery process, it is more efficient in P2P applications. Through analysis and extensive simulations, we demonstrate that MAPCP always maintains a higher degree of anonymity than a MANET anonymous single-path routing protocol in a hostile environment. Simulation results also show that MAPCP is resilient to passive attacks in data forwarding for both one-to-one and one-to-many communications.

Differentiated QoS provision in wireless environments via multicarrier energy allocation is examined in this work. Within this framework, differentiated QoS is applied to transmitted data classes, such as text, audio and video, simultaneously. QoS targets, or constraints, are defined by specifying distortion targets for each class, and energy is allocated to carriers of different classes to meet those targets/constraints accordingly. QoS differentiation is achieved by a concept called QoS scaling. We develop a methodology to find the set of scalars that allows each service constraint to be met. While earlier work can be viewed to offer solutions to the scalar-to-distortion mapping problem, this research solves the more challenging inverse mapping problem; namely, from distortion requirements to selection of scalars. We present two algorithms for differentiated QoS that depend upon the amount of constrained classes relative to the total amount of classes. Finally, to provide a clearer understanding of the effect of QoS scaling, we analyze the effect on distortion by varying the total number of scalars.

Low-complexity low-rate super-orthogonal turbo codes (SOTC) are proposed in this work to replace the implicit repetition code (RC) in an ultra-wide band impulse radio (UWB-IR) system to improve BER, transmission range and system throughput. Various receivers including matched-filter and RAKE receivers are adopted for data detection in the additive white Gaussian noise (AWGN) channel and the indoor IEEE 802.15.3a channel. The performance of SOTC-encoded UWB is analyzed and validated by computer simulation. It is shown that the SOTC-based UWB-IR system can achieve significant performance improvement over the conventional DS- UWB system encoded by RC.

The performance of MC-CDMA in the presence of carrier frequency offsets (CFO) can be severely degraded due to multiaccess interference (MAI). It is shown in this work that a properly chosen subset of real Hadamard-Walsh or exponential codes can achieve zero MAI in a CFO environment. For a channel of length L and a number G of power of 2 with G ges L, we prove that 1 + log2(iV/G) Hadamard-Walsh codewords or N/G exponential codewords can achieve zero MAI under any CFO level. Simulation results are given to corroborate derived theoretical results and evaluate the performance of MC-CDMA with a variable number of users in a CFO environment.

Channel characterization and modeling is essential to the wireless communication system design. A model that optimally represents a fading channel with a variable length Markov chain (VLMC) is proposed in this work. VLMC offers a general class of Markov chains whose structure has a variable order and a parsimonious number of transition probabilities. The proposed model consists of two main components: 1) the optimal fading partition under the constraint of a transmission policy and 2) the derivation of the best VLMC representation. The fading partition is used to discretize a continuous fading channel gain. The optimal discretization criterion is developed based on the cost function of fading channel statistics and the transmission policy used in the system. Once a continuous fading channel gain is discretized, a VLMC is then used to model the channel. To obtain the optimal VLMC representation, we use the Kullback-Liebler (K-L) distance as the optimization criterion. Finally, we show simulation results that demonstrate the accuracy of the proposed fading channel representation in modeling the Rayleigh fading as well as the log-normal fading.

A fast multiple reference assignment scheme is proposed to reduce the motion search complexity in the H.264 video standard in this work. The relationship between the temporal search range and the block size is exploited to develop a new algorithm, called the block-size adaptive referencing (BAR) scheme, that assigns a different number of references in a block-size adaptive manner. The expected rate-distortion (RD) coding loss due to the use of fewer references in the BAR scheme is first analyzed. Furthermore, it is found that the RD coding loss of BAR is related to the RD gradient through a model. Thus, a proper BAR scheme can be chosen to minimize the reference frame numbers while keeping the expected RD loss under a target level. Finally, fast motion search is performed within selected references in a block-size adaptive manner. Experimental results show that the proposed BAR scheme can save approximately 25-70% of the encoding complexity in H.264 (JM10.1) motion estimation with negligible degradation in the RD performance.

The frame-layer bit allocation problem for non-conversational H.264 video rate control is studied in this work. For non-conversational applications, we first impose a GOP (group of pictures) structure on the underlying H.264 video. Then, under the assumption of frame's independence, a one-pass algorithm is developed using the Lagrange optimization framework and GOP-based rate modeling. Instead of estimating the R-D data of future frames to determine the Lagrange multiplier, we estimate the Lagrange multiplier directly from GOP rate models that specify the number of bits consumed in a GOP. It is shown by experimental results that the proposed bit allocation algorithm achieves higher picture quality than that used in the H.264 JM8.1a reference code.

Automatic recognition of unstructured environments is an important problem for mobile robots. We focus on using audio features to recognize different auditory environments, where they are characterized by different types of sounds. The use of audio information provides a complementary means of scene recognition that can effectively augment visual information. In particular, audio can be used toward both the analysis and characterization of the environment at a higher level of abstraction. We begin our investigation of recognizing different auditory environments with the audio information. In this paper, we utilize low-level audio features from a mobile robot and investigate using high-level features based on spectral analysis for scene characterization, and a recognition system was built to discriminate between different environments based on these audio features found.

A joint rate-distortion-complexity H.264 motion search framework is proposed to balance the encoder's coding efficiency and complexity in an embedded system environment. Under our framework, the complexity of H.264 motion search is primarily measured by the execution time of the sum of absolute differences (SAD) calculation. Two Lagrange parameters are used to terminate the complexity-inefficient motion search rounds and skip redundant motion search of small block modes, respectively. Then, the relationship between the weighted complexity and the Lagrange parameters is explored to allocate the complexity cost among different coding units. It is demonstrated by experimental results that the proposed method can reduce the complexity without much sacrifice in coding efficiency

A new rate control algorithm for the H.264 encoder is developed in this work. There are several unique features in the proposed rate control scheme. First, a two-stage encoding mechanism is used to resolve the problem of inter-dependency between R-D optimization (RDO) and rate control (RC) in H.264. Second, it is emphasized that the bits of the header information may occupy a larger portion of the total bit budget especially when encoding at low bit rates in H.264. To address this issue, a rate model for the header information is established so as to estimate header bits more accurately. Third, a new source rate model is proposed to estimate the quantization parameter (QP) from the residual signal. Built upon the above ideas, a rate control algorithm is developed for the H.264 baseline-profile encoder under the constant-bit-rate (CBR) constraint. It is shown by experimental results that the proposed scheme can control bit rates accurately with the R-D performance better than that of the rate control algorithm implemented in H.264 JM8.1a

A new musical onset detection technique based on adaptive linear prediction theory is proposed in this work. We decompose a music signal into multiple sub-bands, and then apply a forward linear prediction error filter (LPEF) to model the narrow-band signal in each band, respectively. To enhance the modeling performance, the coefficients of the LPEF are updated with the least-mean-squares (LMS) algorithm. Under this framework, the onset detection problem can be formulated as the peak-error location problem. Peak selection algorithms are applied to prediction errors to locate the onset time. It is shown by experimental results that the proposed algorithm outperforms several well known existing methods for onset detection

A new scheme that classifies the Internet traffic according to their application types for scalable QoS provision is proposed in this work. The traditional port-based classification method does not yield satisfactory performance, since the same port can be shared by multiple applications. Furthermore, asymmetric routing and errors of modern measurement tools such as PCF and NetFlow degrades the classification performance. To address these issues, the proposed classification process consists of two steps: feature selection and classification. Candidate features that can be obtained easily by ISP are examined. Then, we perform feature reduction so as to balance the performance and complexity. As to classification, the REPTree and the bagging schemes are adopted and compared. It is demonstrated by simulations with real data that the proposed classification scheme outperforms existing techniques

The analysis of audio signals of popular and rock songs of the verse-chorus form to reconstruct its original musical structures is investigated in this work. We first compute the similarity degree between any two measures in a song based on selected features and represent these numbers in a measure-based similarity matrix. Then, we study the similarity across a sequence of consecutive measures, which is revealed by straight segments in parallel with the diagonal line of the similarity matrix. Generally, chorus parts have higher similarity values while verse parts have lower similarity values. As a result, the verse parts are difficult to detect in the presence of the chorus parts. To tackle this problem systematically, the Viterbi Algorithm is adopted to find optimal paths in the lower-triangular similarity matrix, which represent repetitive segments of both choruses and verses. Finally, several post-processing steps are developed to decode the music structure into the verse, the chorus and other non-repetitive parts. Experimental results obtained from several musical audio data are shown to demonstrate the performance of the proposed method.

A quad-tree decomposition approach is proposed for cartoon image compression in this work. The proposed algorithm achieves excellent coding performance by using a unique quad-tree decomposition and shape coding method along with a GIF like color indexing technique to efficiently encode large areas of the same color, which appear in a cartoon-type image commonly. To reduce complexity, the input image is partitioned into small blocks and the quad-tree decomposition is independently applied to each block instead of the entire image. The LZW entropy coding method can be performed as a postprocessing step to further reduce the coded file size. It is demonstrated by experimental results that the proposed method outperforms several well-known lossless image compression techniques for cartoon images that contain 256 colors or less

An adaptive reference selection (ARS) scheme is proposed to enhance error resilience performance of H.264 video in this work, where multiple prediction paths can be created in the compressed video stream at the macroblock level without a large amount of bit rate overhead. We first develop a method to measure the expected distortion at the decoder when the H.264 video is transmitted through erroneous channels. Then, we use an updated rate-distortion cost function to incorporate this measurement into the mode decision process. The best prediction for each macroblock is selected with the objective to achieve the highest expected rate-distortion performance of the GOP in the received video stream. It is shown by experimental results that error propagation is largely reduced and the quality of received video stream is improved significantly by the proposed scheme

Performance comparison of multiuser OFDM-TDMA and OFDMA systems by considering physical and link cross-layer behavior is conducted in this work. We provide a new framework for performance evaluation that centers on "scheduling" with several different subcarrier/bit allocation schemes. The performance metrics include: the bit rate, the bit error rate (BER), packet throughput and delay. The instantaneous channel conditions of users are assumed to be available at the base station, which can be achieved by channel estimation at the mobile terminal (MT), and then fed back to the base station through a control channel. To perform the link layer comparison, we model the Rayleigh fading channel with a finite-state Markov chain, and adopt the M/G/1 queueing model as the analytical tool. Computer simulation is performed to verify conducted analytical results.

The optimal codeword length that generates the maximum output signal to interference ratio (SIR) for the channel-phase-precoded ultra-wideband (CPPUWB) system proposed in [1] is studied in this work to mitigate the interference in high data rate transmission. As compared with the proposed MMSE receiver in [1], this code length optimization technique demands no additional training symbols, which sacrifices the actual data rate, and maintains a simple receiver structure. This optimization problem is highly nonlinear in nature, and a fast search algorithm is developed to speed up the optimization process. The signal to interference plus noise ratio (SINR) performance of the proposed scheme degrades slightly when the input SNR is low, and it will converge to the maximum SINR as the input SNR becomes large.

A statistical approach to distributed edge region detection in wireless sensor networks, which is optimized under the Neyman-Pearson (NP) criterion, is proposed in this work. The concept of edge nodes is adopted to label the defined edge region. Even though statistical methods have been proposed to detect edge nodes, a rigorous way to select the threshold value is lacking. Based on the NP criterion, a decision-fusion approach is developed to address the problem of threshold selection. Performance comparison of the proposed approach and the classifier-based approach is conducted. Simulation results show that the proposed approach is more stable and outperforms the classifier-based approach when there is a location error.

A frequency-domain pre-equalization scheme for single-carrier space-division multiple-access (SC-SDMA) downlink transmissions is proposed in this work. In the system of our interest, both the base station (BS) and mobile terminals (MTs) are equipped with multiple antennas. It operates in a time-division-duplex (TDD) mode and the channel reciprocity between alternative uplink and downlink transmissions is exploited to feed the channel state information (CSI) back to the transmit side. Pre-filtering coefficients are designed so as to minimize the sum of mean-squared-errors at all MTs under an overall transmit power constraint. Several strategies are investigated for linearly combining the received signals at each MT in the frequency domain. Simulation results are provided to demonstrate the effectiveness of the proposed scheme.