Real-time video communication is critically hampered by packet loss. Building on the neural video codec (NVC) and exploiting the statistics of its latent representations, we propose a channel importance-aware redundancy framework that eliminates insignificant channels and reallocates high redundancy to protect important channels for loss-resilient video coding. We first develop a channel-wise importance evaluation method based on the in-distribution (IND) region length criterion [1] for latent representations of prediction residuals and motion vectors produced by NVCs. Consequently, we derive a heuristic that adapts the number of protected channels to the packet loss rate (PLR). More channels are retained at low PLRs to improve video quality, while only a few critical channels are kept and assigned higher redundancy for enhanced robustness at high PLRs. Experimental results show that the proposed method provides more graceful quality degradation and maintains significant performance advantages, particularly under high PLRs, as illustrated in Figure 1.

Despite strong generalization ability, large language models (LLMs) are resource-intensive to deploy. Post-training quantization (PTQ) effectively compresses pretrained LLMs without retraining. Although recent studies target extremely low-bit quantization, most methods rely on extra parameters and the effective bit-width usually exceeds 2 bits. We introduce a saliency-guided mixed-precision PTQ framework that achieves an average bit-width of 1.70 bits while maintaining competitive performance. The proposed method allocates bit-widths across weight groups under a strict total budget, guided by a cross-precision saliency metric. The key innovation is a learnable affine transformation consisting of rotation, scaling, and shifting to align quantization grids with local weight distributions to minimize error. Experiments show that the proposed method matches state-of-the-art low-bit PTQ performance, while reducing the average bit-width by approximately 0.55 bits in Table 1.

Task-adapted compressed sensing magnetic resonance imaging (CS-MRI) is emerging to address the specific demands of downstream clinical tasks with significantly fewer k-space measurements than required by Nyquist sampling. However, existing task-adapted CS-MRI methods suffer from the uncertainty problem for medical diagnosis and cannot achieve adaptive sampling in end-to-end optimization with reconstruction or clinical tasks. To address these limitations, we propose the first task-adapted CS-MRI from the information-theoretic perspective to simultaneously achieve probabilistic inference for uncertainty prediction and adapt to arbitrary sampling ratios and versatile clinical applications. Specifically, we formalize the task-adapted CS-MRI optimization problem by maximizing the mutual information between undersampled k-space measurements and clinical tasks to enable probabilistic inference for addressing the uncertainty problem. We leverage amortized optimization and construct tractable variational bounds for mutual information to jointly optimize sampling, reconstruction, and task-inference models, which enables flexible sampling ratio control using a single end-to-end trained model. Furthermore, the proposed framework addresses two kinds of distinct clinical scenarios within a unified approach, i.e., i) joint task and reconstruction, where reconstruction serves as an auxiliary process to enhance task performance; and ii) task implementation with suppressed reconstruction, applicable for privacy protection. Extensive experiments on large-scale MRI datasets demonstrate that the proposed framework achieves highly competitive performance on standard metrics like Dice compared to deterministic counterpart but provides better distribution matching to the ground-truth posterior distribution as measured by the generalized energy distance (GED).

Pooling and unpooling are indispensable in constructing hierarchical spherical convolutional neural networks (HS-CNNs). Most existing models employ simple downsampling-based pooling, which ignores the sampling theorem and cannot adapt to different spherical signals (with different spectra) and tasks (dependent on different frequency components), thus suffering a significant information loss. Besides, signals reconstructed by the widely-adopted padding-based unpooling may also change unwantedly the spectra of original signals. To address these, we propose a novel framework of HS-CNNs with lifting structures to learn adaptive spherical wavelets for pooling and unpooling, named LiftHS-CNNs. Specifically, we learn spherical wavelets with a lifting structure to adaptively partition the input signal into low- and high-frequency sub-bands, with the down-scaled representations for pooling generated to preserve more information in the low-frequency sub-band. The lifting structure consists of learnable update and predict operators parameterized with graph attention to jointly consider the signal's characteristics and underlying geometries. We then propose an unpooling operation invertible to the lifting-based pooling for restoring the up-scaled representations, which can well preserve spectral characteristics of the original signal. Particular properties (i.e., spatial locality, vanishing moments, and stability) of the learned wavelets and the information preserving ability of the proposed pooling and unpooling are further studied. Experiments on benchmark spherical datasets for a wide range of tasks verify the superiority of our LiftHS-CNNs.

Adaptive bitrate (ABR) streaming is a popular technique used to improve the quality of experience (QoE) for users who watch videos online, which, for example, can provide a smoother video playback by dynamically adjusting the requested video quality with associated bitrate according to the constrained yet diverse network conditions. Recently, learning-based ABR algorithms have achieved a notable performance gain with lower inference overhead than the conventional heuristic or model-based baselines. However, their performance may degrade significantly in an unseen network environment with time-varying and heterogeneous throughput dynamics. For a better generalization, in this paper, we propose a meta-reinforcement learning (meta-RL)-based neural ABR algorithm that is able to quickly adapt its policy to these unseen throughput dynamics. Specifically, we propose a model-free system framework comprising an inference network and a policy network. The inference network infers distribution of the latent representation for underlying dynamics based on the recent throughout context, while the policy network is trained to quickly adapt to the changing throughout dynamics with the sampled latent representation. To effectively learn the inference network and meta-policy on mixed dynamics of the practical ABR scenarios, we further design a variational information bottleneck theory-based loss function for training the inference and policy networks, whose objective is to strike a trade-off between brevity of the latent representation and expressiveness of the meta-policy. We also derive a theoretically necessary condition for the bitrate versions that yield higher long-term QoE, based on which a dynamic action pruning strategy is further developed for practical implementation. This pruning strategy can not only prevent unsafe policy outputs in midst of unseen throughput dynamics, but may also reduce the computational complexity of model-based ABR algorithms. Finally, the meta-training and meta-adaptation procedures of our proposed algorithm are implemented across a range of throughput dynamics. The empirical evaluations on various datasets containing real-world network traces verify that our algorithm surpasses the state-of-the-art ABR algorithms, particularly in terms of the average chunk QoE and fast adaptation across out-of-distribution throughput traces.

Point cloud attribute compression is challenged by fitting the attribute signals living on irregular geometric structures. Existing methods cannot achieve compact multiscale representation for high-fidelity reconstruction using the handcrafted transforms or deep learning-based techniques. In this paper, we propose a novel geometry-aware lifting-based multiscale network via spatial-channel lifting scheme for point cloud attribute compression. The proposed network cascades geometry-aware spatial lifting to reduce spatial redundancy by adaptively capturing irregular geometric structures and progressive channel lifting to progressively reduce channel-wise redundancy in multiscale representation. Furthermore, we design the split, predict, and update operations for geometry-aware spatial lifting to fully exploit the geometry information representing irregular structures. We develop geometry-aware adaptive split to equally split input points with significance scores indicating their dependencies, and propose geometry-aware cross-attention filtering for the predict and update operations for decorrelation based on geometry information. To our best knowledge, this paper achieves the first lifting-based learned transform for point cloud compression that enjoys reversibility guarantees of multiscale representation to enhance rate-distortion performance. Experimental results show that the proposed framework achieves state-of-the-art performance on extensive point cloud datasets, and outperforms latest MPEG G-PCC standard and most recent deep learning based methods.

Conventional image signal processing (ISP) control algorithms based on human visual perception are insufficient for the demands of modern machine vision systems. Although recent learning-based methods have attempted to adapt ISP hyperparameters for specific vision tasks, their high latency and hardware cost of iterative optimization hinder deployment on edge devices such as those used in autonomous driving. To address these limitations, this paper proposes a real-time machine vision system through algorithm–hardware co-design. First, we introduce a one-iteration learning framework to avoid iterative optimization, significantly reducing latency for real-time use. Second, we propose a hardware-friendly controller, RasterNet, specifically tailored for raster-scanning sensor dataflow, eliminating redundant computation. Third, we present a pipelined ISP controller architecture incorporating branch and chroma time division multiplexing techniques to minimize the number of processing elements, achieving a compact and efficient design. Experiments demonstrate that the proposed system achieves superior object detection accuracy on resource-constrained platforms, with real-time performance reaching 70 FPS on FPGA and 224 FPS on ASIC implementations.

Sparse representation via dedicated dictionary learning facilitates image restoration by simultaneously preserving significant representation and removing trivial perturbations. However, existing dictionary learning under strict ℓ0-norm based sparsity constraint lacks of convergence guarantees and suffers from the dilemma between block artifacts and degraded quality in restoring fine details, and could hamper the consequent deep unfolded networks (DUNs). To address these problems, in this paper, we propose the first ℓ0-regularized dictionary learning with a theoretical guarantee of global convergence that achieves globally consistent sparse approximation for image restoration. Specifically, we first formulate an ℓ0-norm based optimization for dictionary learning that simultaneously achieves sparse approximation of local patches and ensures global consistency between the recovered and observed images. Consequently, we develop ℓ0-regularized global-local dictionary learning (ℓ0-GLDL) that alternately achieves sparse code pursuit, local dictionary update, and global image reconstruction based on alternating proximal linearized method. We demonstrate in theory that ℓ0-GLDL converges globally to a critical point. Furthermore, we design a deep unfolded network by unrolling ℓ0 -GLDL into diverse subblocks to strictly realize alternating optimization for stable image restoration. Experimental results demonstrate that the proposed methods achieve state-of-the-art performance and outperform existing deep learning based methods in image restoration tasks such as image denoising, image deblurring, and image inpainting.

Image denoising remains a challenging problem due to the ill-posed nature of recovering clean images from noisy observations. Prior-based modeling plays a vital role in addressing this challenge. Low-rank priors effectively exploit non-local self-similarity by modeling correlations across similar images, but often over-smooth fine textures. In contrast, sparse priors, especially in the form of Convolutional Sparse Coding (CSC), excel at preserving high-frequency texture details, yet typically neglect the structure across images. Recent unfolded CSC networks have improved denoising performance by combining CSC with deep networks, but most of them rely solely on sparsity, overlooking the complementary low-rank structure information. To overcome these limitations, we propose UCSC-LR, the first interpretable unfolded CSC network that jointly leverages both sparse and low-rank priors within a unified architecture. UCSC-LR unrolls both the Iterative Shrinkage-Thresholding Algorithm (ISTA) and Singular Value Thresholding (SVT) to strictly implement alternating minimization, enabling simultaneous pursuit of sparse representations and low-rank reconstructions. A dedicated Fusion Net is introduced to adaptively integrate features from the two priors, allowing the model to preserve both texture and structural content. To further enhance performance on color image denoising, UCSC-LR incorporates a lightweight channel attention mechanism to capture inter-channel dependencies. With shared, pre-learned convolutional dictionaries and efficient parameterization, UCSC-LR achieves state-of-the-art performance with low model complexity. Extensive experiments on grayscale, blind, and color denoising tasks validate the effectiveness of the proposed method, consistently surpassing existing CSC-based and deep learning-based denoisers.

Existing 2D-to-3D human pose estimation (HPE) methods struggle with the occlusion issue by enriching information like temporal and visual cues in the lifting stage. In this paper, we argue that these methods ignore the limitation of the sparse skeleton 2D input representation, which fundamentally restricts the 2D-to-3D lifting and worsens the occlusion issue. To address these, we propose a novel two-stage generative densification method, named Hierarchical Pose AutoRegressive Transformer (HiPART), to generate hierarchical 2D dense poses from the original sparse 2D pose. Specifically, we first develop a multi-scale skeleton tokenization module to quantize the highly dense 2D pose into hierarchical tokens and propose a Skeleton-aware Alignment to strengthen token connections. We then develop a Hierarchical AutoRegressive Modeling scheme for hierarchical 2D pose generation. With generated hierarchical poses as inputs for 2D-to-3D lifting, the proposed method shows strong robustness in occluded scenarios and achieves state-of-the-art performance on the single-frame-based 3D HPE. Moreover, it outperforms numerous multi-frame methods while reducing parameter and computational complexity and can also complement them to further enhance performance and robustness.

Controllability of video generation has been recently concerned in addition to the quality of generated videos. The main challenge to controllable video generation is to synthesize videos based on user-specified instance spatial locations and movement trajectories. However, existing methods suffer from a dilemma between the resource consumption, generation quality, and user controllability. As an efficient alternative to prohibitive training-based video generation, existing zero-shot video generation methods cannot generate high-quality and motion-consistent videos under the control of layouts and movement trajectories. In this paper, we propose a novel zero-shot method named IM-Zero that ameliorates instance-level motion controllable video generation with enhanced control accuracy, motion consistency, and richness of details to address this problem. Specifically, we first present a motion generation stage that extracts motion and textural guidance from keyframe candidates from pre-trained grounded text-to-image model to generate the desired coarse motion video. Subsequently, we develop a video refinement stage that injects the motion priors of pre-trained text-to-video models and detail priors of pre-trained text-to-image models into the latents of coarse motion videos to further enhance video motion consistency and richness of details. To our best knowledge, IM-Zero is the first to simultaneously achieve high-quality video generation and allow to control both layouts and movement trajectories in a zero-shot manner. Extensive experiments demonstrate that IM-Zero outperforms existing methods in terms of video quality, inter-frame consistency, and the alignment of location and trajectory. Furthermore, compared with existing methods, IM-Zero enjoys extra advantages of versatility in video generation, including motion control of subparts within instances, finer control of specifying instance shapes via masks, and more difficult tasks of motion transfer for customizing fine-grained motion patterns through reference videos and high-quality text-to-video generation.

Autofocus is a crucial component of modern digital cameras. While recent learning-based methods achieve state-of-the-art in focus prediction accuracy, they unfortunately ignore the potential focus hunting phenomenon of back-and-forth lens movement in the multi-step focusing procedure. To address this, in this paper, we propose an expert regularized deep reinforcement learning (DRL)-based approach for autofocus, which can utilize the sequential information of lens movement trajectory to both enhance the multi-step in-focus prediction accuracy and reduce the chance of focus hunting. Our method generally follows an actor-critic framework. To accelerate the DRL’s training with a higher sample efficiency, we initialize the policy with a pre-trained single-step prediction network, where the network is further improved by modifying the output of absolute in-focus position distribution to the relative lens movement distribution to establish a better mapping between input images and lens movement. To further stabilize DRL’s training with a lower occurrence of focus hunting in the resulting lens movement trajectory, we generate some offline trajectories based on prior knowledge to avoid focus hunting, which are then leveraged as an offline dataset of expert trajectories to regularize the actor network’s training. Empirical evaluations show that our method outperforms those learning-based methods on public benchmarks, with higher single- and multi-step prediction accuracy, and a significant reduction of focus hunting rate.

Parameter-efficient transfer learning (PETL) has emerged as a promising solution to adapt large-scale pre-trained models to downstream tasks. Nevertheless, these methods have not thoroughly explored the characteristics of PETL methods to optimize the fine-tuning performance with miminal volume of parameters. In this paper, we first reveal that, compared to pre-trained models, PETL tends to generate similar features via homogeneous feature transformations across different layers. Subsequently, we propose a Global Share Local Transform Mixture-of-Experts framework, namely GLEAM, that decomposes the adapter into a shared component and layer-specific local components to simultaneously reduce the redundancy in layer-wise parameter matrices for homogeneous feature transformations and fine-tune the locally specific parameters for minimizing performance loss. Specifically, we develop a shared mixture of convolution that introduces shared multi-scale sparse MoE to enable diverse transformations for suppressing the homogeneity issue of feature transformations in PETL. GLEAM is evaluated on more than 20 datasets for image classification and few-shot learning. Extensive experimental results demonstrate that it achieves comparable performance with existing PETL methods like LoRA with only 3% of its parameters and further yields competitive performance using only 0.07M parameters.

Generative diffusion models have shown empirical successes in point cloud resampling, generating a denser and more uniform distribution of points from sparse or noisy 3D point clouds by progressively refining noise into structure. However, existing diffusion models employ manually predefined schemes, which often fail to recover the underlying point cloud structure due to the rigid and disruptive nature of the geometric degradation. To address this issue, we propose a novel learnable heat diffusion framework for point cloud resampling, which directly parameterizes the marginal distribution for the forward process by learning the adaptive heat diffusion schedules and local filtering scales of the time-varying heat kernel, and consequently, generates an adaptive conditional prior for the reverse process. Unlike previous diffusion models with a fixed prior, the adaptive conditional prior selectively preserves geometric features of the point cloud by minimizing a refined variational lower bound, guiding the points to evolve towards the underlying surface during the reverse process. Extensive experimental results demonstrate that the proposed point cloud resampling achieves state-of-the-art performance in representative reconstruction tasks including point cloud denoising and upsampling.

Learned image compression (LIC) has demonstrated superior rate-distortion (R-D) performance compared to traditional codecs, but is challenged by training inefficiency that could incur more than two weeks to train a state-of-the-art model from scratch. Existing LIC methods overlook the slow convergence caused by compacting energy in learning nonlinear transforms. In this paper, we first reveal that such energy compaction consists of two components, \emph{i.e.}, feature decorrelation and uneven energy modulation. On such basis, we propose a linear auxiliary transform (AuxT) to disentangle energy compaction in training nonlinear transforms. The proposed AuxT obtains coarse approximation to achieve efficient energy compaction such that distribution fitting with the nonlinear transforms can be simplified to fine details. We then develop wavelet-based linear shortcuts (WLSs) for AuxT that leverages wavelet-based downsampling and orthogonal linear projection for feature decorrelation and subband-aware scaling for uneven energy modulation. AuxT is lightweight and plug-and-play to be integrated into diverse LIC models to address the slow convergence issue. Experimental results demonstrate that the proposed approach can accelerate training of LIC models by 2 times and simultaneously achieves an average 1\% BD-rate reduction. To our best knowledge, this is one of the first successful attempt that can significantly improve the convergence of LIC with comparable or superior rate-distortion performance.

In real-time communication (RTC) systems, a congestion control (CC) is indispensable to guarantee the end user’s quality of experience (QoE), the key of which is to predict the bottleneck link capacity and thus set the target bitrate for the sender. Existing deep reinforcement learning (DRL)-based bandwidth prediction methods, though improving the bandwidth prediction performance based on the pre-collected offline dataset, are usually designed for a single RTC scenario with poor generalization ability to other scenarios. What’s worse, the policy learned by these DRL methods can hardly capture the complex multi-modal distribution of underlying optimal policy. To address this, in this paper, we propose an offline DRL-based meta-diffusion policy, where we pre-train an encoder to extract the effective feature embedding of network dynamics, and further learn an expressive meta-policy based on the diffusion model to predict the bandwidth conditioned on the observed network states and extracted network dynamics features. Experiments demonstrate the enhanced accuracy and generalization of our meta-diffusion policy over existing landmark schemes, with a performance gain of 7.84% and 27.12%, in terms of the bandwidth prediction error rate and overestimate rate, respectively.

Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50\% BitOPs per client on average for FL with less than 2\% accuracy loss compared to low-bit training with INT8 precision.

The significant communication overhead and client data heterogeneity have posed an important challenge to current federated learning (FL) paradigm. Existing compression-based and optimization-based FL algorithms typically focus on addressing either the model compression challenge or the data heterogeneity issue individually, rather than tackling both of them. In this paper, we observe that by symbolizing the client model updates to be uploaded (i.e., normalizing the magnitude for each model parameter at local clients), the model heterogeneity, essentially stemmed from data heterogeneity, can be mitigated, and thereby helping improve the overall generalization performance of the globally aggregated model at the server. Inspired with this observation, and further motivated by the success of Lion optimizer in achieving the optimal performance on most tasks in the centralized learning, we propose a new FL algorithm, called FedSMU, which simultaneously reduces the communication overhead and alleviates the data heterogeneity issue. Specifically, FedSMU splits the standard Lion optimizer into the local updates and global execution, where only the symbol of client model updates commutes between the client and server. We theoretically prove the convergence of FedSMU for the general non-convex settings. Through extensive experimental evaluations on several benchmark datasets, we demonstrate that our FedSMU algorithm not only reduces the communication overhead, but also achieves a better generalization performance than the other compression-based and optimization-based baselines.

We propose Noise Conditional Variational Score Distillation (NCVSD), a novel method for distilling pretrained diffusion models into generative denoisers. We achieve this by revealing that the unconditional score function implicitly characterizes the score function of denoising posterior distributions. By integrating this insight into the Variational Score Distillation (VSD) framework, we enable scalable learning of generative denoisers capable of approximating samples from the denoising posterior distribution across a wide range of noise levels. The proposed generative denoisers exhibit desirable properties that allow fast generation while preserve the benefit of iterative refinement: (1) fast one-step generation through sampling from pure Gaussian noise at high noise levels; (2) improved sample quality by scaling the test-time compute with multi-step sampling; and (3) zero-shot probabilistic inference for flexible and controllable sampling. We evaluate NCVSD through extensive experiments, including class-conditional image generation and inverse problem solving. By scaling the test-time compute, our method outperforms teacher diffusion models and is on par with consistency models of larger sizes. Additionally, with significantly fewer NFEs than diffusion-based methods, we achieve record-breaking LPIPS on inverse problems.

Fourier phase retrieval (FPR) is a longstanding ill-posed non-convex inverse problem suffering from failure in reconstruction. Existing methods usually mitigate the issue in a trial-and-error fashion and cannot identify the failure to flexibly adjust the FPR process. In this paper, we propose FA-FPR, a novel failure aware method that guides FPR with diffusion priors by failure detection. We first discover that unsuccessful reconstructions can be identified as fake images due to their distinct structures from true signals, and then leverage a plug-and-play fake image detector to automatically determine the approximation to true signals. Furthermore, we develop a hybrid decoupling scheme that adaptively combines Gaussian noise and predicted noise by score function based on the failure detection. Extensive experiments under various oversampling ratios with noisy and noiseless measurements demonstrate that FA-FPR requires less than 50% number of repeated experiments on average to achieve the state-of-the-art performance.

Source-free domain adaptation (SFDA) is a challenging task that tackles domain shifts using only a pre-trained source model and unlabeled target data. Existing SFDA methods are restricted by the fundamental limitation of source-target domain discrepancy. Non-generation SFDA methods suffer from unreliable pseudo-labels in challenging scenarios with large domain discrepancies, while generation-based SFDA methods are evidently degraded due to enlarged domain discrepancies in creating pseudo-source data. To address this limitation, we propose a novel generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) that leverages unlabeled target data as references to reliably generate and progressively refine a pseudo-target domain for SFDA. Specifically, we divide the target samples into a trust set and a non-trust set based on the reliability of pseudo-labels to sufficiently and reliably exploit their information. For samples from the non-trust set, we develop a manipulation strategy to semantically transform them into the newly assigned categories, while simultaneously maintaining them in the target distribution via a latent diffusion model. Furthermore, we design a progressive refinement mechanism that progressively reduces the domain discrepancy between the pseudo-target domain and the real target domain via iterative refinement. Experimental results demonstrate that DPTM outperforms existing methods by a large margin and achieves state-of-the-art performance on four prevailing SFDA benchmark datasets with different scales. Remarkably, DPTM can significantly enhance the performance by up to 18.6\% in scenarios with large source-target gaps.

In low-latency live streaming (LLLS), the adaptive bitrate algorithm plays a critical role in optimizing encoding bitrates to meet the strict end-to-end latency requirement, which is mostly several seconds or less. Existing methods usually rely on a fixed preset latency target, limiting their generalization capabilities across the heterogeneous scenarios. To address this, we propose an offline meta-reinforcement learning-based bitrate decision framework for LLLS, which incorporates the expertise of multiple state-of-the-art LLLS algorithms from their collected experience and constructs a universal bitrate selection policy via the offline training. Specifically, we establish an RL-based policy that adaptively adjusts the network throughput measurements and selects the encoding bitrates for LLLS based on the network conditions. The policy’s performance is enhanced by improving the robustness of short-term throughput estimations. To leverage the expertise of current LLLS algorithms, we collect their decision-making trajectories, and directly train the policy on these offline trajectories with implicit Q-learning. In addition, a meta-RL paradigm is further adopted to learn a universal policy that performs uniformly well across the heterogeneous network conditions and varying target latencies. Experimental results demonstrate that, compared to the other baselines, the proposed method achieves at least a 5.8% performance gain in terms of the overall quality of experience (QoE), across the diverse target-latency scenarios in the real-world throughput traces.

Progressive coding adapts to reliable image and video transmission over unstable network with fluctuating bandwidth with truncatable bitstreams produced by layer-wise conditional entropy coding. However, recent learned progressive image and video compression methods solely leverage table lookup or real-time computation for conditional entropy coding, and suffer from dramatically increased storage and computational overheads due to the expansion of codewords. To address this problem, in this paper, we propose an efficient conditional entropy coding scheme for learned progressive coding that incorporates the fast lookup-table and low-memory real-time computation based on statistical distribution of the codewords. Specifically, we build a smaller lookup table that stores only the symbols with high occurrence frequency and achieve real-time computation of those symbols with low occurrence frequency. Furthermore, we seamlessly integrate the proposed scheme into existing learned image and video compression frameworks for enhanced coding efficiency. Experimental results show that the proposed scheme accelerates both encoding and decoding processes for learned progressive image and video compression. Remarkably, we for the first time successfully extend the proposed scheme to learned video compression to demonstrate its feasibility.

Cross-layer feature fusion critically influences the performance of deep neural networks, where mainstream approaches like additive fusion of residual connections struggle to model high-order nonlinear interactions, limiting representational capacity, while concatenative fusion of dense connections incurs significant memory and computational overhead. Although recent attention-based feature fusion methods refine feature representations and strengthen hierarchical interactions, they often struggle to balance expressiveness and efficiency. To address these limitations, we propose the Chebyshev Fusion Unit (CFU), a lightweight yet effective cross-layer fusion mechanism. Specifically, CFU computes high-order Chebyshev polynomial terms between residual features and current-layer features, each of which is aggregated through learnable scalar weights, forming enhanced fusion features. This design enables explicit modeling of complex cross-layer dependencies with minimal additional parameters. Extensive experiments across various commonly used base models on image classification, medical image segmentation, and physical law learning tasks demonstrate its superior performance.

Existing methods for integerized training speed up deep learning by using low-bitwidth integerized weights, activations, gradients, and optimizer buffers. However, they overlook the issue of full-precision latent weights, which consume excessive memory to accumulate gradient-based updates for optimizing the integerized weights. In this paper, we propose the first latent weight quantization schema for general integerized training, which minimizes quantization perturbation to training process via residual quantization with optimized dual quantizer. We leverage residual quantization to eliminate the correlation between latent weight and integerized weight for suppressing quantization noise. We further propose dual quantizer with optimal nonuniform codebook to avoid frozen weight and ensure statistically unbiased training trajectory as full-precision latent weight. The codebook is optimized to minimize the disturbance on weight update under importance guidance and achieved with a three-segment polyline approximation for hardware-friendly implementation. Extensive experiments show that the proposed schema allows integerized training with lowest 4-bit latent weight for various architectures including ResNets, MobileNetV2, and Transformers, and yields negligible performance loss in image classification and text generation. Furthermore, we successfully fine-tune Large Language Models with up to 13 billion parameters on one single GPU using the proposed schema.

Recent advances in self-supervised learning have witnessed great achievements, especially with the introduction of contrastive learning, where the goal is to maximize the mutual information between different augmentations of the same image, i.e., positive pairs. However, such optimization does not necessarily correspond to optimal representation due to noisy samples, thus inevitably being over-confident in the relevance between views. As a result, the learned model would capture spurious correlation and retain superfluous information that deteriorates representations. In this paper, we facilitate contrastive learning by reducing superfluous relevance between positive views. To this end, we introduce the representation entropy minimization regularization over the objective of vanilla contrastive learning, which forces representations to retain possibly the least information, thus alleviating superfluous relevance from irrelevant views. Then, we derive the analytical expression of the proposed objective by converting it to an information bottleneck problem and solving via variation approximation, which leads to a novel contrastive learning framework, termed as CLIMB, short for Contrastive Learning via variational InforMation Bottleneck. Experiments over multiple benchmarks demonstrate that CLIMB brings consistent improvement. Notably, using DINO as an instantiation, CLIMB achieves 4.5% and 3.5% gain under the k-NN classification metric with EfficientNet-B0 and ResNet-50 as backbones, respectively.

Qualifying the discrepancy between 3D geometric models, which could be represented with either point clouds or triangle meshes, is a pivotal issue with board applications. Existing methods mainly focus on directly establishing the correspondence between two models and then aggregating point-wise distance between corresponding points, resulting in them being either inefficient or ineffective. In this paper, we propose DDM, an efficient, effective, robust, and differentiable distance metric for 3D geometry data. Specifically, we construct DDM based on the proposed implicit representation of 3D models, namely directional distance field (DDF), which defines the directional distances of 3D points to a model to capture its local surface geometry. We then transfer the discrepancy between two 3D geometric models as the discrepancy between their DDFs defined on an identical domain, naturally establishing model correspondence. To demonstrate the advantage of our DDM, we explore various distance metric-driven 3D geometric modeling tasks, including template surface fitting, rigid registration, non-rigid registration, scene flow estimation and human pose optimization. Extensive experiments show that our DDM achieves significantly higher accuracy under all tasks. As a generic distance metric, DDM has the potential to advance the field of 3D geometric modeling.

Federated learning (FL) commonly encourages the clients to perform multiple local updates before the global aggregation, thus avoiding frequent model exchanges and relieving the communication bottleneck between the server and clients. Though empirically effective, the negative impact of multiple local updates on the stability of FL is not thoroughly studied, which may result in a globally unstable and slow convergence. Based on sensitivity analysis, we define in this paper a local-update stability index for the general FL, as measured by the maximum inter-client model discrepancy after the multiple local updates that mainly stems from the data heterogeneity. It enables to determine how much the variation of client's models with multiple local updates may influence the global model, and can also be linked with the convergence and generalization. We theoretically derive the proposed local-update stability for current state-of-the-art FL methods, providing possible insight to understanding their motivation and limitation from a new perspective of stability. For example, naively executing the parallel acceleration locally at clients would harm the local-update stability. Motivated by this, we then propose a novel accelerated yet stabilized FL algorithm (named FedANAG) based on the server- and client-level Nesterov accelerated gradient (NAG). In FedANAG, the global and local momenta are elaborately designed and alternatively updated, while the stability of local update is enhanced with help of the global momentum. We prove the convergence of FedANAG for strongly convex, general convex and non-convex settings. We then conduct evaluations on both the synthetic and real-world datasets to first validate our proposed local-update stability. The results further show that across various data heterogeneity and client participation ratios, FedANAG not only accelerates the global convergence by reducing the required number of communication rounds to a target accuracy, but converges to an eventually higher accuracy.

End-to-end optimization via deep neural networks has facilitated lossy image compression. Existing neural network-based entropy models for end-to-end optimized image compression are limited by parameterized Gaussian distributions with deterministic mean and variance and cannot achieve accurate rate estimation for bottleneck representation with varying statistics. In this paper, we propose a novel entropy model based on deep Gaussian process regression (DGPR) to address this problem. Specifically, the proposed entropy model leverages autoregressive DGPR to flexibly predict the channel-wise posterior distributions of high-dimensional bottleneck representation for entropy coding. Consequently, we develop a well-established bit-rate estimation scheme via posterior inference of DGPR using the learned probabilistic distribution. Furthermore, scalable training is achieved via tensor train decomposition and Monte Carlo sampling to enable tractable variational inference of DGPR. To our best knowledge, this paper is the first attempt to develop the learnable probabilistic model for flexible parameter estimation in entropy modeling. Experimental results show that the proposed model outperforms conventional image compression methods (e.g., JPEG2000 and BPG) as well as recent end-to-end optimized methods on the Kodak and Tecnick datasets in terms of rate-distortion performance.

Entropy modeling is the core component of learned image compression (LIC) that models the distribution of latent representation learned from input images via neural networks for bit-rate estimation. However, existing entropy models employ presumed parameterized distributions such as Gaussian models and are limited for the learned latent representation characterized by complex distributions. To address this problem, in this paper, we for the first time achieve generative probabilistic entropy modeling of latent representation based on conditional diffusion models. Specifically, we propose a conditional diffusion-based probabilistic entropy model (CDPEM) to parameterize the latent representation with distributions of arbitrary forms that are generated by well designed training-test consistent denoising diffusion implicit model (TC-DDIM) without introducing any presumption. TC-DDIM is designed to leverage ancestral sampling to gradually approximate the distribution of latent representation with guaranteed consistency in generation for training and test. Furthermore, we develop a hierarchical spatial-channel context model to incorporate with TC-DDIM to sufficiently exploit spatial correlations with the approximate contextual information produced by ancestral sampling and channel-wise correlations using channel-wise information aggregation with reweighted training loss. Experimental results demonstrate that the proposed entropy model achieves state-of-the-art performance on the Kodak, CLIC, and Tecnick datasets compared to existing LIC methods. Remarkably, when incorporated with recent baselines, the proposed model outperforms latest VVC standard by an evident gain in R-D performance.

Variable-rate coding is challenging but indispensable for learned image compression (LIC) that is in nature characterized by nonlinear transform coding (NTC). Existing methods for variable-rate LIC are restricted by the non-smooth quantization process with zero gradients almost everywhere, and consequently, suffer from training-test gap and degraded rate-distortion (R-D) performance. To address this problem, in this paper, we propose sampling-based optimization for training NTC models along with non-uniform quantizers. Different from gradient-based optimization, the proposed sampling-based optimization first randomly samples the parameters from Gaussian distributions with progressively reduced variance and then selects the optimal parameters with a R-D indicator. On the basis of sampling-based optimization, we develop a learnable non-uniform dead-zone quantizer by adaptively refining the quantization steps for variable-rate coding with nonlinear transforms. Furthermore, we incorporate the learnable dead-zone quantizer to achieve a variable-rate LIC model with enhanced R-D performance and design rate and distortion control algorithms to adapt to dynamic network conditions. Experimental results show that the proposed method achieves state-of-the-art R-D performance in variable-rate image compression. It obtains an average 8.82% BD-rate reduction compared to latest versatile video compression (VVC) standard, and simultaneously achieves precise rate and distortion control with an average variation of 0.0087 bpp in bit-rates and 0.1265 dB in distortion on the Kodak dataset.

Image compression for both human and machine vision has become prevailing to accommodate to rising demands for machine-machine and human-machine communications. Scalable human-machine image compression is recently emerging as an efficient alternative to simultaneously achieve high accuracy for machine vision in the base layer and obtain high-fidelity reconstruction for human vision in the enhancement layer. However, existing methods achieve scalable coding with heuristic mechanisms, which cannot fully exploit the inter-layer correlations and evidently sacrifice rate-distortion performance. In this paper, we propose task-adapted learnable embedded quantization to address this problem in an analytically optimized fashion. We first reveal the relationship between the latent representations for machine and human vision and demonstrate that optimal representation for machine vision can be approximated with post-training optimization on the learned representation for human vision. On such basis, we propose task-adapted learnable embedded quantization that leverages learnable step predictor to adaptively determine the optimal quantization step for diverse machine vision tasks such that inter-layer correlations between representations for human and machine vision are sufficiently exploited using embedded quantization. Furthermore, we develop a human-machine scalable coding framework by incorporating the proposed embedded quantization into pre-trained learned image compression models. Experimental results demonstrate that the proposed framework achieves state-of-the-art performance on machine vision tasks like object detection, instance segmentation, and panoptic segmentation with negligible loss in rate-distortion performance for human vision.

Existing pretraining methods for semantic segmentation are hampered by the task gap between global image -level pretraining and local pixel-level finetuning. Joint dense-level pretraining is a promising alternative to exploit off-the-shelf annotations from diverse segmentation datasets but suffers from low-quality class embeddings and inconsistent data and supervision signals across multiple datasets by directly employing CLIP. To overcome these challenges, we propose a novelMulti-datasEt harmoNized pretraining framework forSemantic sEgmentation (MENSA). MENSA incorporates high-quality language embeddings and momentum-updated visual embeddings to effectively model the class relationships in the embedding space and thereby provide reliable supervision information for each category. To further adapt to multiple datasets, we achieve one-to-many pixel-embedding pairing with cross-dataset multi-label mapping through cross-modal information exchange to mitigate inconsistent supervision signals and introduce region-level and pixel-level cross-dataset mixing for varying data distribution. Experimental results demonstrate that MENSA is a powerful foundation segmentation model that consistently outperforms popular supervised or unsupervised ImageNet pretrained models for various benchmarks under standard fine-tuning. Furthermore, MENSA is shown to significantly benefit frozen-backbone fine-tuning and zero-shot learning by endowing pixel-level distinctiveness to learned representations.

In federated learning (FL), the heterogeneity of data and asynchronous participation of clients have been observed to induce the local client's model discrepancy with high variance, leading to a slow and unstable convergence globally at the server. In this article, motivated by the usefulness of stale client updates, we first propose a general framework, named FedVR, to address this issue. In FedVR, we design an aggregate of both fresh and stale local model updates without additional communication overhead, which is computed at the server as a control variate to reduce the client variance incurred by data heterogeneity and client sampling. In order to further reduce the model discrepancy between local clients, we therefore propose FedMDVR, which broadcasts the designed control variate to all the active clients to help correct their local update directions toward the global optimum, i.e., stationary point of the global objective function. While in the global update at server, the client variance is also decreased as inherited from the variance reduction nature of FedVR. We theoretically prove the convergence of FedVR and FedMDVR in the general nonconvex settings. Through extensive experimental evaluations on several benchmark datasets, we also demonstrate that our proposed FedVR and FedMDVR not only accelerate the convergence by reducing the number of communication rounds required to achieve a certain target accuracy, but more importantly, can converge to a higher accuracy than the baseline algorithms.

Network planning is crucial to facilitate network service under limited network operation costs. However, adapting the network topology (i.e., connections and capacities for physical and IP links) to time-varying and stochastic network states is challenging due to high computational overhead of problem solving. Existing deep learning based methods suffer from prohibitive computational complexity due to simultaneously solving IP link capacities and allocated traffics or iteratively exploring search space with excessive samples via heuristic rules. In this paper, we propose GDPlan, the first generative framework that leverages conditional graph diffusion model to address this challenge. To achieve high solving efficiency, GDPlan decouples network planning into two separate stages, i.e., generation of diverse high-quality solutions to IP link capacity and refinement of the solutions under feasibility constraints. Specifically, we develop generation-oriented graph signal modeling that reformulates solving IP link capacities as generation of graph topology conditioned on the graph signals related to traffic demands, physical links and network costs. Consequently, energy-guided controllable graph generation is achieved to capture significant structural patterns of graph topology with score-based graph diffusion model and produce diverse solutions with a guarantee of feasibility under specific objectives or constraints. GDPlan successfully achieves graph generation based solution to network planning with an order of magnitude higher solving speed than the commercial solver Gurobi. Experimental results demonstrate that, compared with Gurobi, the proposed GDPlan obtains lowest 3.7% average gap with only about 7.5% average running time.

Learned image compression (LIC) has recently emerged as a promising alternative to realize transform-based coding. Contrary to widely studied transform and entropy modeling in LIC, most LIC methods are limited to uniform scalar quantization and existing vector quantization (VQ) methods are restricted by the soft-to-hard quantization strategy. In this paper, we propose a novel entropy relaxed lattice vector quantization (ERLVQ) scheme for LIC. ERLVQ achieves a tractable dimension-wise approximation to the probability mass function for the complex Voronoi regions in a general class of lattices. Instead of directly performing probabilistic estimation on the lattices, ERLVQ equivalently aligns complex Voronoi regions with a set of complete and non-overlapping hypercubes in a projected parametric domain. Space covering technique is then developed to generating hypercubes with their centers fixed to integer projected lattice points and their volumes equaling the Voronoi regions to enable separable dimension-wise estimation of multi-variate probability mass functions. The proposed ERLVQ is the first to adapt to arbitrary lattices for VQ based on LIC and can be embedded into existing LIC methods. Experimental results demonstrate that the proposed ERLVQ is superior to the uniform scalar quantization (USQ) and existing lattice VQ methods [1] on prevailing LIC baselines, as shown in Table 1 .

Network quantization is promising to alleviate the storage and deployment burden for deep neural networks. However, existing methods focus on directly quantizing the network parameters but ignore the correlations between the initialized parameters (generated by random seeds) and the optimized parameters (obtained when the network training converges). In this paper, we reveal the redundancy between the initialized and optimized parameters and propose a novel quantization scheme for network parameters with rescaled initialization. We develop a novel quantization scheme for quantizing the delta parameters, i.e. , the difference between the optimized parameters and the rescaled initialization, as shown in Figure 1 . The scale is learned to adapt the weight regularization in training by minimizing the rounding error and penalizing the long-tailed distribution of the delta parameter. Adaptive rounding is further presented to calibrate the rounding direction of the delta parameter via a learnable bias. The proposed method can achieve high-efficiency prediction of network parameters and consistent performance gains with a trivial overhead of single random seed. Extensive experimental results demonstrate its superior performance on various popular network architectures, as exemplified in Table 1 .

State-of-the-art learning-based stability control methods for nonlinear robotic systems suffer from the issue of reality gap, which stems from discrepancy of the system dynamics between training and target (test) environments. To mitigate this gap, we propose an adversarially robust neural Lyapunov control (ARNLC) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. Specifically, inspired by adversarial learning, we introduce an adversary to simulate the dynamics discrepancy, which is learned through deep reinforcement learning to generate the worst-case perturbations during the controller’s training. By alternatively updating the controller to minimize the perturbed Lyapunov risk and the adversary to deviate the controller from its objective, the learned control policy enjoys a theoretical guarantee of stability. Empirical evaluations on five stability control tasks with the uniform and worst-case perturbations demonstrate that ARNLC not only accelerates the convergence to asymptotic stability, but can generalize better in the entire perturbation space.

Existing diffusion models for inverse problems have demonstrated impressive performance but suffer from prohibitive sampling complexity due to lengthy iterative sampling procedures. While introducing pre-trained consistency models (CMs) as priors holds promise for fast and high-quality sampling, theoretical disparities between CMs and diffusion models remain, hindering the application of CMs in solving inverse problems. To address this issue, we propose a novel framework, named Dual Data Consistency Guided Consistency Model (DCCM), that for the first time to solve inverse problems with pretrained CM priors. We establish a denoising interpretation of CMs to set up the equivalence between CMs and denoisers and incorporate CM in a theoretically sound fashion. Consequently, we develop refined data consistency to facilitate optimization with CM priors and avoid local minima caused by the nonlinearity of degradation operators. Furthermore, we introduce the data consistency shortcut that leverages the manifold hypothesis to approximate refined data consistency and bypass backpropagation for enhanced sampling speed without reconstruction quality loss. Extensive experiments demonstrate DCCM achieves state-of-the-art performance in terms of reconstruction quality and sampling speed in a wide range tasks of image deblurring, super-resolution, and inpainting.

Learned image compression (LIC) has gained traction as an effective solution for image storage and transmission in recent years. However, existing LIC methods are redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these challenges, we propose a novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LIC. The FAT block comprises frequency-decomposition window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Additionally, we introduce frequency-modulation feed-forward network (FMFFN) to adaptively modulate different frequency components, improving rate-distortion performance. Furthermore, we present a transformer-based channel-wise autoregressive (T-CA) model that effectively exploits channel dependencies. Experiments show that our method achieves state-of-the-art rate-distortion performance compared to existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 by 14.5\%, 15.1\%, 13.0\% in BD-rate on the Kodak, Tecnick, and CLIC datasets.

As two prominent strategies for representation learning, Contrastive Learning (CL) and Masked Image Modeling (MIM) have witnessed significant progress. Previous studies have demonstrated the advantages of each approach in specific scenarios. CL, resembling supervised pre-training, excels at capturing longer-range global patterns and enhancing feature discrimination, while MIM is adept at introducing local and diverse attention across transformer layers. Considering the respective strengths, previous studies utilize feature distillation to inherit both discrimination and diversity. In this paper, we thoroughly examine previous feature distillation methods and observe that the increase in diversity mainly stems from asymmetric designs, which may in turn compromise the discrimination ability. To strike a balance between the two properties, we propose a simple yet effective strategy termed Hybrid Distill, which leverages both the CL and MIM teachers to jointly guide the student model. Hybrid Distill emulates the token relations of the MIM teacher at intermediate layers for diversity, while simultaneously distilling the final features of the CL teacher to enhance discrimination. A progressive redundant token masking strategy is employed to reduce the expenses associated with distillation and aid in preventing the model from converging to local optima. Experimental results demonstrate that Hybrid Distill achieves superior performance on various benchmark datasets.

Pre-training followed by full fine-tuning has gradually been substituted by Parameter-Efficient Tuning (PET) in the field of computer vision. PET has gained popularity, especially in the context of large-scale models, due to its ability to reduce transfer learning costs and conserve hardware resources. However, existing PET approaches primarily focus on recognition tasks and typically support uni-modal optimization, while neglecting dense prediction tasks and vision language interactions. To address this limitation, we propose a novel PET framework called **B**i-direction**a**l Inte**r**twined Vision **L**anguage Effici**e**nt Tuning for **R**eferring **I**mage Segment**a**tion (**BarLeRIa**), which leverages bi-directional intertwined vision language adapters to fully exploit the frozen pre-trained models' potential in cross-modal dense prediction tasks. In BarLeRIa, two different tuning modules are employed for efficient attention, one for global, and the other for local, along with an intertwined vision language tuning module for efficient modal fusion. Extensive experiments conducted on RIS benchmarks demonstrate the superiority of BarLeRIa over prior PET methods with a significant margin, i.e., achieving an average improvement of 5.6\%. Remarkably, without requiring additional training datasets, BarLeRIa even surpasses SOTA full fine-tuning approaches. The code is available at https://github.com/NastrondAd/BarLeRIa.

Low-bit integer training emerges as a promising approach to mitigate the heavy burden during network training by quantizing the weights, activations, and gradients. However, existing methods cannot well achieve mixed-precision quantization for low-bit training and are commonly limited to INT8 precision. In this paper, we propose a novel low-bit integer training framework that, for the first time, achieves adaptive mixed-precision allocation (AMPA) for weights, activations, and gradients, and pushes the boundaries to a precision level below INT8. We develop a novel magnitude-based sensitivity measurement with regard to the quantization losses of weight, activation, and gradient quantization and the average gradient magnitudes, which is demonstrated as an upper bound of quantization influence in theory. We further design a layer-wise precision update strategy under observations on the quantization losses and their effects on model performance in low-bit training. Extensive experiments on different backbones and datasets show that, compared to INT8 quantization, the proposed method can achieve more than 38% BitOPs reduction with a tolerable loss below 2% in image classification, image segmentation, and language modeling.

Recent diffusion models provide a promising zero-shot solution to noisy linear inverse problems without retraining for specific inverse problems. In this paper, we reveal that recent methods can be uniformly interpreted as employing a Gaussian approximation with hand-crafted isotropic covariance for the intractable denoising posterior to approximate the conditional posterior mean. Inspired by this finding, we propose to improve recent methods by using more principled covariance determined by maximum likelihood estimation. To achieve posterior covariance optimization without retraining, we provide general plug-and-play solutions based on two approaches specifically designed for leveraging pre-trained models with and without reverse covariance. We further propose a scalable method for learning posterior covariance prediction based on representation with orthonormal basis. Experimental results demonstrate that the proposed methods significantly enhance reconstruction performance without requiring hyperparameter tuning.

Existing self-supervised methods for gaze estimation using the dominant streams of contrastive and generative approaches are restricted to eye images and could fail in general full-face settings. In this paper, we reveal that contrastive methods are ineffective in data augmentation for self-supervised full-face gaze estimation, while generative methods are prone to trivial solutions due to the absence of explicit regularization on semantic representations. To address this challenge, we propose a novel approach called **B**ootstrap auto-**e**ncoders with **C**ontrastive p**a**radigm (**BeCa**), which combines the strengths of both generative and contrastive methods. Specifically, we revisit the Auto-Encoder used in generative approaches and incorporate the contrastive paradigm to introduce explicit regularization on gaze representation. Furthermore, we design the InfoMSE loss as an alternative to the vanilla MSE loss for Auto-Encoder to mitigate the inconsistency between reconstruction and representation learning. Experimental results demonstrate that the proposed approaches outperform state-of-the-art unsupervised gaze approaches on extensive datasets (including wild scenes) under both within-dataset and cross-dataset protocols.

Point cloud upsampling is crucial for 3D reconstruction, with recent research significantly benefitting from the advances in deep learning technologies. The majority of existing methods, which focus on a sequence of processes including feature extraction, augmentation, and the reconstruction of coordinates, encounter significant challenges in interpreting the geometric attributes they uncover, particularly with respect to the intricacies of transitioning feature dimensionality. In this paper, we delve deeper into modeling Partial Differential Equations (PDEs) specifically tailored for the inverse heat dissipation process in dense point clouds. Our goal is to detect gradients within the dense point cloud data distribution and refine the accuracy of interpolated points' positions along with their complex geometric nuances through a systematic iterative approximation method. Simultaneously, we adopt multivectors from geometric algebra as the primary tool for representing the geometric characteristics of point clouds, moving beyond the conventional vector space representations. The use of geometric products of multivectors enables us to capture the complex relationships between scalars, vectors, and their components more effectively. This methodology not only offers a robust framework for depicting the geometric features of point clouds but also enhances our modeling capabilities for inverse heat dissipation PDEs. Through both qualitative and quantitative assessments, we demonstrate that our results significantly outperform existing state-of-the-art techniques in terms of widely recognized point cloud evaluation metrics and 3D visual reconstruction fidelity.

Image compression for machine vision exhibits various rate-accuracy performance across different downstream tasks and content types. An efficient utilization of constrained network resource for achieving an optimal overall task performance has thus recently attracted a growing attention. In this paper, we propose Tombo, a task-oriented image compression and transmission framework that efficiently identifies the optimal encoding bitrate and routing scheme for multiple image bitstreams delivered simultaneously for different downstream tasks. Specifically, we study the characteristics of image rate-accuracy performance for different machine vision tasks, and formulate the task-oriented joint bitrate and routing optimization problem for multi-bitstreams as a multi-commodity network flow problem with the time-expanded network modeling. To ensure consistency between the encoding bitrate and routing optimization, we also propose an augmented network that incorporates the encoding bitrate variables into the routing variables. To improve computational efficiency, we further convert the original optimization problem to a multi-marginal optimal transport problem, and adopt a Sinkhorn iteration-based algorithm to quickly obtain the near-optimal solution. Finally, we adapt Tombo to efficiently deal with the dynamic network scenario where link capacities may fluctuate over time. Empirical evaluations on three typical machine vision tasks and four real-world network topologies demonstrate that Tombo achieves a comparable performance to the optimal one solved by the off-the-shelf solver Gurobi, with a 5x ~ 114× speedup.

Most of existing federated learning (FL) formulation is treated as a point-estimate of models, inherently prone to overfitting on scarce client-side data with overconfident decisions. Though Bayesian inference can alleviate this issue, a direct posterior inference at clients may result in biased local posterior estimates due to data heterogeneity, leading to a sub-optimal global posterior. From an information-theoretic perspective, we propose FedMDMI, a federated posterior inference framework based on model-data mutual information (MI). Specifically, a global model-data MI term is introduced as regularization to enforce the global model to learn essential information from the heterogeneous local data, alleviating the bias caused by data heterogeneity and hence enhancing generalization. To make this global MI tractable, we decompose it into local MI terms at the clients, converting the global objective with MI regularization into several locally optimizable objectives based on local data. For these local objectives, we further show that the optimal local posterior is a Gibbs posterior, which can be efficiently sampled with stochastic gradient Langevin dynamics methods. Finally, at the server, we approximate sampling from the global Gibbs posterior by simply averaging samples from the local posteriors. Theoretical analysis provides a generalization bound for FL w.r.t. the model-data MI, which, at different levels of regularization, represents a federated version of the bias-variance trade-off. Experimental results demonstrate a better generalization behavior with better calibrated uncertainty estimates of FedMDMI.

Diffusion Transformer (DiT) is emerging as a cutting-edge trend in the landscape of generative diffusion models for image generation. Recently, masked-reconstruction strategies have been considered to improve the efficiency and semantic consistency in training DiT but suffer from deficiency in contextual information extraction. In this paper, we provide a new insight to reveal that noisy-to-noisy masked-reconstruction harms sufficient utilization of contextual information. We further demonstrate the insight with theoretical analysis and empirical study on the mutual information between unmasked and masked patches. Guided by such insight, we propose a novel training paradigm named MC-DiT for fully learning contextual information via diffusion denoising at different noise variances with clean-to-clean mask-reconstruction. Moreover, to avoid model collapse, we design two complementary branches of DiT decoders for enhancing the use of noisy patches and mitigating excessive reliance on clean patches in reconstruction. Extensive experimental results on 256$\times$256 and 512$\times$512 image generation on the ImageNet dataset demonstrate that the proposed MC-DiT achieves state-of-the-art performance in unconditional and conditional image generation with enhanced convergence speed.

Autofocus (AF) is a fundamental task in digital image capturing, yet current approaches often exhibit a poor performance in terms of speed or accuracy. In this paper, we propose a deep reinforcement learning (DRL)-based framework for camera AF, by incorporating a pre-trained MobileNet-v2 encoder for image feature extraction and a Gaussian process regression (GPR) module for in-focus prediction. Specially, we formulate AF as a multi-step process, in which the lens movement trajectory is dynamically generated from a DRL-based policy. By leveraging image features extracted from the MobileNet-v2 encoder, the DRL-based policy can efficiently get close to the in-focus position, which enables the focus search in a fast speed, thus overcoming the drawback of slow focusing procedure suffered by traditional search-based AF methods. A GPR-based in-focus predictor is further designed to terminate the DRL’s exploration once it is able to provide a reliable prediction on the in-focus position, by exploiting the sharpness metric values of image patches captured at the DRL-generated trajectory of lens positions. In further contrast with the one-step prediction-based methods, we leverage the sequential information brought additionally by the multi-step exploration, enabling to gain a higher prediction accuracy. Experimental results demonstrate that our approach provides a significant improvement compared to both the search-based and prediction-based AF baselines.

By introducing randomness on the environments, domain randomization (DR) imposes diversity to the policy training of deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance incurred by trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing a low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. Based on our theory, we further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity for the practical implementation. By dividing the entire space of environments into some subspaces and then estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of variance reduction and faster convergence than the state-dependent baselines. Empirical evaluations on six robot control tasks with randomized dynamics demonstrate that VRDR not only accelerates the convergence of policy training, but can consistently achieve a better eventual policy with improved training stability.

3-D point clouds facilitate 3-D visual applications with detailed information of objects and scenes but bring about enormous challenges to design efficient compression technologies. The irregular signal statistics and high-order geometric structures of 3-D point clouds cannot be fully exploited by existing sparse representation and deep learning based point cloud attribute compression schemes and graph dictionary learning paradigms. In this paper, we propose a novel p-Laplacian embedding graph dictionary learning framework that jointly exploits the varying signal statistics and high-order geometric structures for 3-D point cloud attribute compression. The proposed framework formulates a nonconvex minimization constrained by p-Laplacian embedding regularization to learn a graph dictionary varying smoothly along the high-order geometric structures. An efficient alternating optimization paradigm is developed by harnessing ADMM to solve the nonconvex minimization. To our best knowledge, this paper proposes the first graph dictionary learning framework for point cloud compression. Furthermore, we devise an efficient layered compression scheme that integrates the proposed framework to exploit the correlations of 3-D point clouds in a structured fashion. Experimental results demonstrate that the proposed framework is superior to state-of-the-art transform-based methods in M-term approximation and point cloud attribute compression and outperforms recent MPEG G-PCC reference software.

Source-free domain adaptation (SFDA) shows the potential to improve the generalizability of deep learning-based face anti-spoofing (FAS) while preserving the privacy and security of sensitive human faces. However, existing SFDA methods are significantly degraded without accessing source data due to the inability to mitigate domain and identity bias in FAS. In this paper, we propose a novel Source-free Domain Adaptation framework for FAS (SDA-FAS) that systematically addresses the challenges of source model pre-training, source knowledge adaptation, and target data exploration under the source-free setting. Specifically, we develop a generalized method for source model pre-training that leverages a causality-inspired PatchMix data augmentation to diminish domain bias and designs the patch-wise contrastive loss to alleviate identity bias. For source knowledge adaptation, we propose a contrastive domain alignment module to align conditional distribution across domains with a theoretical equivalence to adaptation based on source data. Furthermore, target data exploration is achieved via self-supervised learning with patch shuffle augmentation to identify unseen attack types, which is ignored in existing SFDA methods. To our best knowledge, this paper provides the first full-stack privacy-preserving framework to address the generalization problem in FAS. Extensive experiments on nineteen cross-dataset scenarios show our framework considerably outperforms state-of-the-art methods.

Object pose estimation constitutes a critical area within the domain of 3D vision. While contemporary state-of-the-art methods that leverage real-world pose annotations have demonstrated commendable performance, the procurement of such real training data incurs substantial costs. This paper focuses on a specific setting wherein only 3D CAD models are utilized as a priori knowledge, devoid of any background or clutter information. We introduce a novel method, CPPF++, designed for sim-to-real category-level pose estimation. This method builds upon the foundational point-pair voting scheme of CPPF, reformulating it through a probabilistic view. To address the challenge posed by vote collision, we propose a novel approach that involves modeling the voting uncertainty by estimating the probabilistic distribution of each point pair within the canonical space. Furthermore, we augment the contextual information provided by each voting unit through the introduction of N-point tuples. To enhance the robustness and accuracy of the model, we incorporate several innovative modules, including noisy pair filtering, online alignment optimization, and a tuple feature ensemble. Alongside these methodological advancements, we introduce a new category-level pose estimation dataset, named DiversePose 300. Empirical evidence demonstrates that our method significantly surpasses previous sim-to-real approaches and achieves comparable or superior performance on novel datasets.

Recent learned image compression models surpass manually designed methods in rate-distortion performance by introducing nonlinear transforms and end-to-end optimization. However, there still lack quantitative measurements that efficiently evaluate the latent representations inferred by learned image compression models. To address this problem, we develop novel measurements on robustness and importance of the latent representations. We first propose an admissible range that can be efficiently estimated via gradient ascent and descent for establishing the empirical distribution of latent representations. Consequently, the in-distribution region within the admissible range is derived to measure the robustness and channel importance of latent representations of natural images. Visualization demonstrates the statistics of latent representations are significantly distinguishing in the properties of robustness and linearity within and outside the in-distribution region. To our best knowledge, this paper proposes the first statistically meaningful measurements for learned image compression and successfully applies the measurements in corruption alleviation during successive image compression and post-training pruning in a training-free fashion. Compared with existing methods, the shrunk in-distribution constraint derived from the in-distribution region achieves superior robustness and rate-distortion performance in successive compression. The channel importance allows post-training pruning to achieve comparable rate-distortion performance with a reduction of up to 60% entropy coding time.

Predicting the saliency map of a 360-degree video is the key for various downstream tasks, such as saliency-based compression and tile-based adaptive streaming. Besides static salient objects, the moving target will also contribute to the saliency map. Therefore, the joint exploitation of spherical spatio-temporal information is necessary for an accurate saliency prediction. The spherical spatial feature extraction, however, is hindered by the non-Euclidean geometric nature of spherical data, which imposes difficulty on direct extraction of the spatial features with traditional convolutional neural networks (CNNs). While the efficient exploitation of temporal correlation between these spherical spatial features remains another challenge, which requires the extraction of spherical optical flows for explicit motion information. To address these, in this paper, we first propose a spherical graph-based Farneback algorithm to extract the spherical optical flows directly in the sphere domain, by leveraging the GICOPix uniform sampling scheme. We then design a 3D separable graph convolutional network-based saliency prediction framework, named 360Spred, by taking both the spherical frames and spherical optical flows as input. The proposed 360Spred framework is based on the U-Net structure, with a 3D separable graph convolution (3DSGC) operator that directly extracts the visual and motion features in the sphere domain and exploits temporal correlation of both the high-level and low-level spatial features. Experimental results on two public datasets show that 360Spred can achieve a better performance than other baseline models in terms of the saliency prediction accuracy for 360-degree videos.

In adaptive video streaming, the design of an adaptive bitrate (ABR) strategy is critical for the quality-of-experience (QoE) perceived by users. Though current learning-based ABR algorithms achieve state-of-the-art performance for users with a given QoE metric setting for training, they may unfortunately suffer the poor generalization issue for other users with different QoE preferences. Besides, how to quantitatively characterize the distinct QoE preference for a user has also not been extensively studied yet. In this paper, we propose STEER, a successor feature-based transfer reinforcement learning framework for fast learning the ABR strategies on heterogeneous QoE preferences. Specifically, we first develop a QoE preference analysis scheme to infer the personal QoE preference of a single user based on the user's actual viewing history. We then formulate the personalized QoE maximization problem as a reinforcement learning (RL) task, which optimizes the ABR strategy to maximize the overall QoE perceived by the user. Further, we model the QoE maximization problem for multiple users with heterogeneous QoE preferences as a multi-task RL problem, with each task distinguished by the user-distinct QoE preference. To efficiently address this problem, the proposed STEER solves for each RL-based ABR task by learning its optimal successor feature (SF) function, which can be exploited as shared knowledge across tasks to facilitate the transfer between tasks. With SF functions, STEER can quickly evaluate the optimal policies of previously learned tasks on a new task, and further use the generalized policy improvement operation to obtain a jumpstart policy. Both theoretically and empirically, we show that this jumpstart policy is a good initial policy with a performance guarantee for better generalization in the new task, and can also lead to a faster convergence to the optimal policy of the new task.

Viewers of 360-degree videos are provided with both visual modality to characterize their surrounding views and audio modality to indicate the sound direction. Though both modalities are important for saliency prediction, little work has been done by jointly exploiting them, which is mainly due to the lack of audio-visual saliency datasets and insufficient exploitation of the multi-modality. In this article, we first construct an audio-visual saliency dataset with 57 360-degree videos watched by 63 viewers. Through a deep analysis of the constructed dataset, we find that the human gaze can be attracted by the auditory cues, resulting in a more concentrated saliency map if the sound source's location is further provided. To jointly exploit the visual and audio features and their correlation, we further design a saliency prediction network for 360-degree videos (SVGC-AVA) based on spherical vector-based graph convolution and audio-visual attention. The proposed spherical vector-based graph convolution can process visual and audio features directly in the sphere domain, thus avoiding projection distortion incurred by traditional CNN-based predictors. In addition, the audio-visual attention scheme explores self-modal and cross-modal correlation for both modalities, which are further hierarchically processed with the U-Net's multi-scale structure of SVGC-AVA. Evaluations on both our and public datasets validate that SVGC-AVA can achieve higher prediction accuracy, both qualitatively and subjectively.

Message passing has evolved as an effective tool for designing graph neural networks (GNNs). However, most existing methods for message passing simply sum or average all the neighboring features to update node representations. They are restricted by two problems: 1) lack of interpretability to identify node features significant to the prediction of GNNs and 2) feature overmixing that leads to the oversmoothing issue in capturing long-range dependencies and inability to handle graphs under heterophily or low homophily. In this article, we propose a node-level capsule graph neural network (NCGNN) to address these problems with an improved message passing scheme. Specifically, NCGNN represents nodes as groups of node-level capsules, in which each capsule extracts distinctive features of its corresponding node. For each node-level capsule, a novel dynamic routing procedure is developed to adaptively select appropriate capsules for aggregation from a subgraph identified by the designed graph filter. NCGNN aggregates only the advantageous capsules and restrains irrelevant messages to avoid overmixing features of interacting nodes. Therefore, it can relieve the oversmoothing issue and learn effective node representations over graphs with homophily or heterophily. Furthermore, our proposed message passing scheme is inherently interpretable and exempt from complex post hoc explanations, as the graph filter and the dynamic routing procedure identify a subset of node features that are most significant to the model prediction from the extracted subgraph. Extensive experiments on synthetic as well as real-world graphs demonstrate that NCGNN can well address the oversmoothing issue and produce better node representations for semisupervised node classification. It outperforms the state of the arts under both homophily and heterophily.

The scene graph is a novel data structure describing objects and their pairwise relationship within image scenes. As the size of scene graphs in vision and multimedia applications increases, the need for lossless storage and transmission of such data becomes more critical. However, the compression of scene graphs is less studied because of the complicated data structures involved and complex distributions. Existing solutions usually involve general-purpose compressors or graph structure compression methods, which are weak at reducing the redundancy in scene graph data. This article introduces a novel lossless compression framework with adaptive predictors for the joint compression of objects and relations in scene graph data. The proposed framework comprises a unified prior extractor and specialized element predictors to adapt to different data elements. Furthermore, to exploit the context information within and between graph elements, Graph Context Convolution is proposed to support different graph context modeling schemes for different graph elements. Finally, an overarching framework incorporates the learned distribution model to predict numerical data under complicated conditional constraints. Experiments conducted on labeled or generated scene graphs demonstrate the effectiveness of the proposed framework for scene graph lossless compression.

The rapid advancement of multimedia and imaging technologies has resulted in increasingly diverse visual and semantic data. A large range of applications such as remote-assisted driving requires the amalgamated storage and transmission of various visual and semantic data. However, existing works suffer from the limitation of insufficiently exploiting the redundancy between different types of data. In this article, we propose a unified framework to jointly compress a diverse spectrum of visual and semantic data, including images, point clouds, segmentation maps, object attributes, and relations. We develop a unifying process that embeds the representations of these data into a joint embedding graph according to their categories, which enables flexible handling of joint compression tasks for various visual and semantic data. To fully leverage the redundancy between different data types, we further introduce an embedding-based adaptive joint encoding process and a Semantic Adaptation Module to efficiently encode diverse data based on the learned embeddings in the joint embedding graph. Experiments on the Cityscapes, MSCOCO, and KITTI datasets demonstrate the superiority of our framework, highlighting promising steps toward scalable multimedia processing.

There has been a recent surge of interest in introducing transformers to 3D human pose estimation (HPE) due to their powerful capabilities in modeling long-term dependencies. However, existing transformer-based methods treat body joints as equally important inputs and ignore the prior knowledge of human skeleton topology in the self-attention mechanism. To tackle this issue, in this paper, we propose a Pose-Oriented Transformer (POT) with uncertainty guided refinement for 3D HPE. Specifically, we first develop novel pose-oriented self-attention mechanism and distance-related position embedding for POT to explicitly exploit the human skeleton topology. The pose-oriented self-attention mechanism explicitly models the topological interactions between body joints, whereas the distance-related position embedding encodes the distance of joints to the root joint to distinguish groups of joints with different difficulties in regression. Furthermore, we present an Uncertainty-Guided Refinement Network (UGRN) to refine pose predictions from POT, especially for the difficult joints, by considering the estimated uncertainty of each joint with uncertainty-guided sampling strategy and self-attention mechanism. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art methods with reduced model parameters on 3D HPE benchmarks such as Human3.6M and MPI-INF-3DHP.

Domain Generalization (DG) has achieved great success in generalizing knowledge from source domains to unseen target domains. However, current DG methods rely heavily on labeled source data, which are usually costly and unavailable. Since unlabeled data are far more accessible, we study a more practical unsupervised domain generalization (UDG) problem. Learning invariant visual representation from different views, i.e., contrastive learning, promises well semantic features for in-domain unsupervised learning. However, it fails in cross-domain scenarios. In this paper, we first delve into the failure of vanilla contrastive learning and point out that semantic connectivity is the key to UDG. Specifically, suppressing the intra-domain connectiv-ity and encouraging the intra-class connectivity help to learn the domain-invariant semantic information. Then, we propose a novel unsupervised domain generalization approach, namely Dual Nearest Neighbors contrastive learning with strong Augmentation (DN2A). Our DN2A leverages strong augmentations to suppress the intra-domain connectivity and proposes a novel dual nearest neighbors search strategy to find trustworthy cross domain neighbors along with in-domain neighbors to encourage the intra-class connectivity. Experimental results demonstrate that our DN2A outperforms the state-of-the-art by a large margin, e.g., 12.01% and 13.11 % accuracy gain with only 1% labels for linear evaluation on PACS and DomainNet, respectively.

Pretraining followed by fine-tuning has proven to be effective in visual recognition tasks. However, fine-tuning all parameters can be computationally expensive, particularly for large-scale models. To mitigate the computational and storage demands, recent research has explored Parameter-Efficient Fine-Tuning (PEFT), which focuses on tuning a minimal number of parameters for efficient adaptation. Existing methods, however, fail to analyze the impact of the additional parameters on the model, resulting in an unclear and suboptimal tuning process. In this paper, we introduce a novel and effective PEFT paradigm, named SNF (Shortcut adaptation via Normalization Flow), which utilizes normalizing flows to adjust the shortcut layers. We highlight that layers without Lipschitz constraints can lead to error propagation when adapting to downstream datasets. Since modifying the over-parameterized residual connections in these layers is expensive, we focus on adjusting the cheap yet crucial shortcuts. Moreover, learning new information with few parameters in PEFT can be challenging, and information loss can result in label information degradation. To address this issue, we propose an information-preserving normalizing flow. Experimental results demonstrate the effectiveness of SNF. Specifically, with only 0.036M parameters, SNF surpasses previous approaches on both the FGVC and VTAB-1k benchmarks using ViT/B-16 as the backbone. The code is available at https://github.com/Wang-Yaoming/SNF

End-to-end learned image compression (LIC) has become promising alternatives for lossy image compression. However, deployments of LIC models are limited in real-world applications due to excessive network parameters and high computational complexity. Existing LIC models realized with integer networks are significantly de-graded in rate-distortion (R-D) performance. In this paper, we propose a novel fully integerized LIC model that simultaneously achieves channel-wise weight and 8-bit activation quantization for alleviating the loss of R-D performance. For weight quantization, we develop an internal bit width increment (IBWI) via nonlinear logarithmic mapping to convert the stored low-precision integer weights to high bit-width weights for inference. Moreover, outlier channel splitting (OCS) [1] is employed to address large outliers of weight distribution by duplicating channels and constrain the integer weights within the scope of INT8. For activation quantization, we leverage activation equalization to balance the channel-wise distribution of activations. Experimental results demonstrate that the proposed method achieves a reduction of 75% storage cost with subtle performance loss compared to full-precision pre-trained models, and outperforms existing integer-only networks, as shown in Table 1.

Quantizing one single deep neural network into multiple compression rates (precisions) has been recently considered for flexible deployments in real-world scenarios. In this paper, we propose a novel scheme that achieves progressive bit-width allocation and joint training to simultaneously optimize mixed-precision quantized networks under multiple compression rates. Specifically, we develop a progressive bit-width allocation with switchable quantization step size to enable mixed-precision quantization based on analytic sensitivity of network layers under multiple compression rates. Furthermore, we achieve joint training for quantized networks under different compression rates via knowledge distillation to exploit their correlations based on the shared network structure. Experimental results show that the proposed scheme outperforms AdaBits [1] in various networks on CIFAR-10 and ImageNet.

Generalization ability of face anti-spoofing has been widely concerned in recent years. Existing domain generalization methods use adversarial learning or metric learning to extract invariant features across domains but are proved to be flawed from causal views. The learned domain-invariant features cannot generalize well to unseen real scenarios in both theoretical and practical senses. In this paper, we propose a novel method that learns Causal Representations for Face Anti-Spoofing (CRFAS). We first model the data generation process of face anti-spoofing via Structural Casual Model (SCM) and reveal that only the causal feature is capable of generalizing to unseen domains. On such basis, we extract causal features via back- door adjustment without prior assumptions rather than learn domain-invariant features as existing methods. Furthermore, we employ the supervised contrastive loss to generate more realistic counterfactual features for backdoor adjustment and improve the generalization ability of learned causal features. Extensive experiments on six cross-dataset testing scenarios demonstrate that CRFAS outperforms the state-of-the-art face anti-spoofing methods in terms of HTER and AUC in generalizing to unseen domains.

Generalizable face anti-spoofing (FAS) based on domain generalization (DG) has gained growing attention due to its robustness in real-world applications. However, these DG methods rely heavily on labeled source data, which are usually costly and hard to access. Comparably, unlabeled face data are far more accessible in various scenarios. In this paper, we propose the first Unsupervised Domain Generalization framework for Face Anti-Spoofing, namely UDG-FAS, which could exploit large amounts of easily accessible unlabeled data to learn generalizable features for enhancing the low-data regime of FAS. Yet without supervision signals, learning intrinsic live/spoof features from complicated facial information is challenging, which is even tougher in cross-domain scenarios due to domain shift. Existing unsupervised learning methods tend to learn identity-biased and domain-biased features as shortcuts, and fail to specify spoof cues. To this end, we propose a novel Split-Rotation-Merge module to build identity-agnostic local representations for mining intrinsic spoof cues and search the nearest neighbors in the same domain as positives for mitigating the identity bias. Moreover, we propose to search cross-domain neighbors with domain-specific normalization and merged local features to learn a domain-invariant feature space. To our best knowledge, this is the first attempt to learn generalized FAS features in a fully unsupervised way. Extensive experiments show that UDG-FAS significantly outperforms state-of-the-art methods on six diverse practical protocols.

Transformers have become the primary backbone of the computer vision community due to their impressive performance. However, the unfriendly computation cost impedes their potential in the video recognition domain. To optimize the speed-accuracy trade-off, we propose Semanticaware Temporal Accumulation score (STA) to prune spatiotemporal tokens integrally. STA score considers two critical factors: temporal redundancy and semantic importance. The former depicts a specific region based on whether it is a new occurrence or a seen entity by aggregating token-to-token similarity in consecutive frames while the latter evaluates each token based on its contribution to the overall prediction. As a result, tokens with higher scores of STA carry more temporal redundancy as well as lower semantics thus being pruned. Based on the STA score, we are able to progressively prune the tokens without introducing any additional parameters or requiring further re-training. We directly apply the STA module to off-the-shelf ViT and VideoSwin backbones, and the empirical results on Kinetics-400 and Something-Something V2 achieve over 30% computation reduction with a negligible ~ 0.2% accuracy drop. The code is released at https://github.com/Mark12Ding/STA.

Point cloud upsampling is a challenging task to generate dense and uniform point clouds from sparse and irregular inputs. Existing deep learning based methods usually leverage graph convolution networks (GCNs) to learn features for expansion and reconstruction but suffer from the over-smoothing issue by discarding the high-frequency components. In this paper, we design a novel dynamic graph scattering transform (GST) to achieve multi-scale feature extraction for point cloud upsampling. The dynamic GST can accommodate to the K-nearest neighbor (KNN)-based feature graph dynamically constructed for point clouds to achieve multi-resolution band-pass representations. Consequently, a feature extractor with dense connection is developed based on the dynamic GST such that low-frequency and high-frequency components can be sufficiently combined for subsequent feature expansion and reconstruction. To our best knowledge, this paper is the first to introduce graph scattering transform for representation learning with dynamic feature graphs in point cloud upsampling. Experimental results on the PU1K dataset show that the proposed network based on dynamic GST achieve promising performance under various metrics especially for sparse point clouds.

Deep convolutional neural networks have achieved great progress in image denoising tasks. However, their complicated architectures and heavy computational cost hinder their deployments on mobile devices. Some recent efforts in designing lightweight denoising networks focus on reducing either FLOPs (floating-point operations) or the number of parameters. However, these metrics are not directly correlated with the on-device latency. In this paper, we identify the real bottlenecks that affect the CNN-based models’ runtime performance on mobile devices: memory access cost and NPU-incompatible operations, and build the model based on these. To further improve the denoising performance, the mobile-friendly attention module MFA and the model reparameterization module RepConv are proposed, which enjoy both low latency and excellent denoising performance. To this end, we propose a mobile-friendly denoising network, namely MFDNet. The experiments show that MFDNet achieves state-of-the-art performance on real-world denoising benchmarks SIDD and DND under real-time latency on mobile devices. The code and pre-trained models will be released.

Recent 2D-to-3D human pose estimation (HPE) utilizes temporal consistency across sequences to alleviate the depth ambiguity problem but ignore the action related prior knowledge hidden in the pose sequence. In this paper, we propose a plug-and-play module named Action Prompt Module (APM) that effectively mines different kinds of action clues for 3D HPE. The highlight is that, the mining scheme of APM can be widely adapted to different frameworks and bring consistent benefits. Specifically, we first present a novel Action-related Text Prompt module (ATP) that directly embeds action labels and transfers the rich language information in the label to the pose sequence. Besides, we further introduce Action-specific Pose Prompt module (APP) to mine the position-aware pose pattern of each action, and exploit the correlation between the mined patterns and input pose sequence for further pose refinement. Experiments show that APM can improve the performance of most video-based 2D-to-3D HPE frameworks by a large margin.

In personalized federated learning (PFL), multiple clients train customized models to fulfill their personal objectives, which, however, are prone to overfitting to local data due to the heterogeneity and scarcity of local data. To address this, we propose from the information-theoretic perspective a personalized federated learning framework based on the common representation learned across clients, named FedCR. Specifically, we introduce to the local client update a regularizer that aims at minimizing the discrepancy between local and global conditional mutual information (CMI), such that clients are encouraged to learn and exploit the common representation. Upon this, each client learns individually a customized predictor (head), while the extractor (body) remains to be aggregated by the server. Our CMI regularizer leads to a theoretically sound alignment between the local and global stochastic feature distributions in terms of their Kullback-Leibler (KL) divergence. More importantly, by modeling the global joint feature distribution as a product of multiple local feature distributions, clients can efficiently extract diverse information from the global data but without need of the raw data from other clients. We further show that noise injection via feature alignment and ensemble of local predictors in FedCR would help enhance its generalization capability. Experiments on benchmark datasets demonstrate a consistent performance gain and better generalization behavior of FedCR.

Learning a rotation-invariant (RI) representation is of significant importance for real-world point cloud segmentation that is perturbed by arbitrary rotations. Recent principal component analysis (PCA)-based methods provide an effective alternative to align point clouds and produce the RI representation under the preservation of global information. However, conventional PCA with 3-D coordinates cannot fully represent high-dimensional geometric structures like surfaces and curves and cannot uniformly align these structures for learning RI representation. In this paper, we propose a novel rotation-invariant method for point cloud segmentation, which leverages kernel PCA (KPCA) for aligning point clouds in a projected high-dimensional space via non-linear mapping and develops a Geometry-based Weighted Convolution (GWConv) to distinguish part boundaries during segmentation. Specifically, the KPCA produces a RI representation with polynomial kernels for effectively representing complicated geometric structures in point clouds. Moreover, the GWConv incorporates geometric structures into convolution and enhances neighboring points with similar geometry for fine-grained segmentation based on the RI representation. Experimental results demonstrate that the proposed method can achieve competitive performance with the state-of-the-arts and outperforms existing PCA-based methods in part segmentation on ShapeNet. Furthermore, it achieves evident performance gains on complicated 3-D shapes such as Earphone and Car and facilitates segmentation around the part boundaries.

Parameter-Efficient Tuning (PET) has emerged as a leading advancement in both Natural Language Processing and Computer Vision, enabling efficient accommodation of downstream tasks without costly fine-tuning. However, most existing PET approaches are limited to uni-modal tuning, even for vision-language models like CLIP. We investigate this limitation and demonstrate that simultaneous tuning of the two modalities in such models leads to multi-modal forgetting and catastrophic performance degradation, particularly when generalizing to new classes. To address this issue, we propose a novel PET approach called VioLET (Vision Language Efficient Tuning) that utilizes collaborative multi-modal gradients to unlock the full potential of both modalities. Specifically, we incorporate an additional visual encoder without learnable parameters and use these two visual encoders to compute the gradients of the context parameters separately. When conflicts arise, we replace the original gradient with an orthogonal gradient. Extensive experiments are conducted on few-shot recognition and unseen class generalization tasks using ResNet-50 or ViT/B-16 as the backbone. VioLET consistently outperforms several state-of-the-art methods on 11 datasets, showcasing its superiority over existing PET approaches. The code is available at https://github.com/Wang-Yaoming/VioLET.

Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.

Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. However, their complexity dramatically increases when handling long token sequences, particularly for dense prediction tasks that require high-resolution input. Notably, dense prediction tasks, such as semantic segmentation or object detection, emphasize more on the contours or shapes of objects, while the texture inside objects is less informative. Motivated by this observation, we propose to apply adaptive resolution for different regions in the image according to their importance. Specifically, at the intermediate layer of the ViT, we select anchors from the token sequence using the proposed spatial-aware density-based clustering algorithm. Tokens that are adjacent to anchors are merged to form low-resolution regions, while others are preserved independently as high-resolution. This strategy could significantly reduce the number of tokens, and the following layers only handle the reduced token sequence for acceleration. At the output end, the resolution of the feature map is recovered by unfolding merged tokens for task prediction. Consequently, we can considerably accelerate ViTs for dense prediction tasks. The proposed method is evaluated across three different datasets and demonstrates promising performance. For instance, "Segmenter ViT-L" can be accelerated by 48\% FPS without fine-tuning, while maintaining the performance. Moreover, our method can also be applied to accelerate fine-tuning. Experiments indicate that we can save 52\% training time while accelerating 2.46$\times$ FPS with only a 0.09\% performance drop.

Deep learning models have greatly improved the accuracy of point cloud classification. Nevertheless, existing deep learning models are vulnerable to data corruptions such as Gaussian noise and outliers which are inevitable in real-world point cloud collection. To address this challenge, we develop a novel point cloud classification model that is robust to data corruptions, with permutohedral lattice-based representation and density-improved hierarchical feature extraction. Specifically, raw noisy point clouds are firstly projected into a regular per-mutohedral lattice space to obtain the quantized representations. Subsequently, we propose to leverage the local density to improve the farthest point sampling (FPS) and spectral graph convolution for robust hierarchical feature extraction, inspired by that the local point density implicitly reveals the reliability and semantic information of each point. The local density can be efficiently calculated from the permutohedral lattice-based representation. Extensive experiments on the benchmark dataset (i.e., ModelNet-C) verify the robustness of the proposed model. In comparison to the state-of-the-art methods, the proposed model achieves superior performance on the corrupted dataset while maintaining competitive classification accuracy on the clean dataset.

Entropy modeling plays an important role in estimating the rates of latent representations and optimizing the rate-distortion performance for learned image compression. Autoregression modules are demonstrated to eliminate spatial/channel-wise redundancy of latent representations in fixed-rate learned image compression. However, it cannot be efficiently achieved in progressive coding due to the high computational complexity raised by element-wise probability prediction. In this paper, we propose a learned progressive image compression method that enables spatial autoregression for entropy modeling. Specifically, we develop a novel codeword alignment scheme to prevent coding redundancy and achieve efficient autoregression of latent representations in different quality layers. Consequently, conditional probability estimation for the latent prediction can be achieved based on spatial autoregression in a layer-wise manner. We further extend the proposed method by dead-zone quantizers to obtain promoted rate-distortion performance. The proposed method is a successful attempt to enable spatial autoregression in learned progressive coding and further bridge the performance gap with fixed-rate models. Experimental results show that it outperforms traditional methods such as JPEG and BPG, as well as recent fine-grained learned progressive coding models DPICT and PLONQ in terms of rate-distortion performance.

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned confidence maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission. The code will be publicly available at https://github.com/jingjin25/LFhybridSR-Fusion.

Batch normalization (BN) is a fundamental unit in modern deep neural networks. However, BN and its variants focus on normalization statistics but neglect the recovery step that uses linear transformation to improve the capacity of fitting complex data distributions. In this paper, we demonstrate that the recovery step can be improved by aggregating the neighborhood of each neuron rather than just considering a single neuron. Specifically, we propose a simple yet effective method named batch normalization with enhanced linear transformation (BNET) to embed spatial contextual information and improve representation ability. BNET can be easily implemented using the depth-wise convolution and seamlessly transplanted into existing architectures with BN. To our best knowledge, BNET is the first attempt to enhance the recovery step for BN. Furthermore, BN is interpreted as a special case of BNET from both spatial and spectral views. Experimental results demonstrate that BNET achieves consistent performance gains based on various backbones in a wide range of visual tasks. Moreover, BNET can accelerate the convergence of network training and enhance spatial information by assigning important neurons with large weights accordingly.

Self-supervised learning based on instance discrimination has shown remarkable progress. In particular, contrastive learning, which regards each image as well as its augmentations as an individual class and tries to distinguish them from all other images, has been verified effective for representation learning. However, conventional contrastive learning does not model the relation between semantically similar samples explicitly. In this paper, we propose a general module that considers the semantic similarity among images. This is achieved by expanding the views generated by a single image to Cross-Samples and Multi-Levels, and modeling the invariance to semantically similar images in a hierarchical way. Specifically, the cross-samples are generated by a data mixing operation, which is constrained within samples that are semantically similar, while the multi-level samples are expanded at the intermediate layers of a network. In this way, the contrastive loss is extended to allow for multiple positives per anchor, and explicitly pulling semantically similar images together at different layers of the network. Our method, termed as CSML, has the ability to integrate multi-level representations across samples in a robust way. CSML is applicable to current contrastive based methods and consistently improves the performance. Notably, using MoCo v2 as an instantiation, CSML achieves 76.6% top-1 accuracy with linear evaluation using ResNet-50 as backbone, 66.7% and 75.1% top-1 accuracy with only 1% and 10% labels, respectively. All these numbers set the new state-of-the-art. The code is available at https://github.com/haohang96/CSML.

It is promising to solve linear inverse problems by unfolding iterative algorithms (e.g., iterative shrinkage thresholding algorithm (ISTA)) as deep neural networks (DNNs) with learnable parameters. However, existing ISTA-based unfolded algorithms restrict the network architectures for iterative updates with the partial weight coupling structure to guarantee convergence. In this paper, we propose hybrid ISTA to unfold ISTA with both pre-computed and learned parameters by incorporating free-form DNNs (i.e., DNNs with arbitrary feasible and reasonable network architectures), while ensuring theoretical convergence. We first develop HCISTA to improve the efficiency and flexibility of classical ISTA (with pre-computed parameters) without compromising the convergence rate in theory. Furthermore, the DNN-based hybrid algorithm is generalized to popular variants of learned ISTA, dubbed HLISTA, to enable a free architecture of learned parameters with a guarantee of linear convergence. To our best knowledge, this paper is the first to provide a convergence-provable framework that enables free-form DNNs in ISTA-based unfolded algorithms. This framework is general to endow arbitrary DNNs for solving linear inverse problems with convergence guarantees. Extensive experiments demonstrate that hybrid ISTA can reduce the reconstruction error with an improved convergence rate in the tasks of sparse recovery and compressive sensing.

Progressive coding is essential to the practical deployment of learned image compression over heterogeneous networks and clients. Existing methods for learned progressive image compression require complex and empirical design to achieve near-optimal rate-distortion performance over a wide range of bit-rates. However, these methods are limited by the implicit learned mechanism based on neural networks and introduction of uniform quantizers. In this paper, we propose generalized learned progressive image compression with analytic rate-distortion optimization using dead-zone quantizers on the latent representation. Specifically, we reveal that dead-zone quantizers, as a general case of uniform quantizers, are equivalent to uniform quantizers in fixed-rate nonlinear transform coding and can prevent extra redundancy in embedded quantization for progressive coding. Consequently, we propose rate-distortion optimized learned progressive coding by approximating the optimal quantizer in the source spaces using dead-zone quantizers in an analytic manner on the Laplacian source. To our best knowledge, this paper is the first to achieve general learned progressive coding from the perspective of optimal quantizers. The proposed method achieves theoretically sound and practically efficient embedded quantization and learned progressive coding of latent representations with improved rate-distortion performance. It can also enable embedded quantization with diverse assignments of truncation points and support flexible configuration of quality layers of varying numbers and at varying target bit-rates. Furthermore, we successfully incorporate the proposed method into existing pre-trained fixed-rate models to realize progressive learned image compression without re-training. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performance in learned progressive image compression compared with traditional codecs and recent learned methods.

Endobronchial ultrasound (EBUS) elastography videos have shown great potential to supplement intrathoracic lymph node diagnosis. However, it is laborious and subjective for the specialists to select the representative frames from the tedious videos and make a diagnosis, and there lacks a framework for automatic representative frame selection and diagnosis. To this end, we propose a novel deep learning framework that achieves reliable diagnosis by explicitly selecting sparse representative frames and guaranteeing the invariance of diagnostic results to the permutations of video frames. Specifically, we develop a differentiable sparse graph attention mechanism that jointly considers frame-level features and the interactions across frames to select sparse representative frames and exclude disturbed frames. Furthermore, instead of adopting deep learning-based frame-level features, we introduce the normalized color histogram that considers the domain knowledge of EBUS elastography images and achieves superior performance. To our best knowledge, the proposed framework is the first to simultaneously achieve automatic representative frame selection and diagnosis with EBUS elastography videos. Experimental results demonstrate that it achieves an average accuracy of 81.29% and area under the receiver operating characteristic curve (AUC) of 0.8749 on the collected dataset of 727 EBUS elastography videos, which is comparable to the performance of the expert-based clinical methods based on manually-selected representative frames.

With increasing popularity of virtual reality and augmented reality, application of point clouds is in critical demand as it enables users to freely navigate in an immersive scene with six degrees of freedom. However, point clouds usually comprise large amounts of data, and are thus difficult to stream in bandwidth-constrained networks. It is therefore important, yet challenging, to efficiently stream the resource-intensive point clouds, such that the user’s quality of experience (QoE) is guaranteed on a high-level but with a low bandwidth consumption. To this end, we propose a QoE-driven adaptive streaming approach for the tile-based point cloud transmission, to maximize the user’s QoE while reducing the transmission redundancy. By exploiting the perspective projection, we specifically model the QoE of a 3D tile as a function of the bitrate of its representation, user’s view frustum and spatial position, occlusion between tiles, and the resolution of rendering device. Based on this QoE model, we then formulate the QoE-optimized rate adaptation problem as a multiple-choice knapsack problem, which allocates bitrates for different tiles under a given transmission capacity. It is equivalently converted to a submodular function maximization problem subject to knapsack constraints, and solved by a practical greedy-based algorithm with a theoretical worst-case performance guarantee. The proposed algorithm is able to achieve a near-optimal performance, but with a very low computational complexity. Experimental results further demonstrate superiority of the proposed rate adaptation algorithm over existing schemes, in terms of both user’s visual quality and transmission efficiency.

Light field (LF) data are widely used in the immersive representations of the 3D world. To record the light rays along with different directions, an LF requires much larger storage space and transmission bandwidth than a conventional 2D image with similar spatial dimension. In this paper, we propose a novel framework for light field image compression that leverages graph learning and dictionary learning to remove structural redundancies between different views. Specifically, to significantly reduce the bit-rates, only a few key views are sampled and encoded, whereas the remaining non-key views are reconstructed via the graph adjacency matrix learned from the angular patch. Furthermore, dictionary-guided sparse coding is developed to compress the graph adjacency matrices and reduce the coding overheads. To our best knowledge, this paper is the first to achieve compact representation of cross-view structural information via adaptive learning on graphs. Experimental results demonstrate that the proposed framework achieves better performance than the standardized HEVC-based codec.

In light of the success of contrastive learning in the image domain, current self-supervised video representation learning methods usually employ contrastive loss to facilitate video representation learning. When naively pulling two augmented views of a video closer, the model however tends to learn the common static background as a shortcut but fails to capture the motion information, a phenomenon dubbed as background bias. Such bias makes the model suffer from weak generalization ability, leading to worse performance on downstream tasks such as action recognition. To alleviate such bias, we propose Foreground-background Merging (FAME) to deliberately compose the moving foreground region of the selected video onto the static background of others. Specifically, without any off-the-shelf detector, we extract the moving fore-ground out of background regions via the frame difference and color statistics, and shuffle the background regions among the videos. By leveraging the semantic consistency between the original clips and the fused ones, the model focuses more on the motion patterns and is debiased from the background shortcut. Extensive experiments demonstrate that FAME can effectively resist background cheating and thus achieve the state-of-the-art performance on downstream tasks across UCF101, HMDB51, and Diving48 datasets. The code and configurations are released at https://github.com/Mark12Ding/FAME.

Appearance-based Gaze Estimation leverages deep neural networks to regress the gaze direction from monocular images and achieve impressive performance. However, its success depends on expensive and cumbersome annotation capture. When lacking precise annotation, the large domain gap hinders the performance of trained models on new domains. In this paper, we propose a novel gaze adaptation approach, namely Contrastive Regression Gaze Adaptation (CRGA), for generalizing gaze estimation on the target domain in an unsupervised manner. CRGA leverages the Contrastive Domain Generalization (CDG) module to learn the stable representation from the source domain and leverages the Contrastive Self-training Adaptation (CSA) module to learn from the pseudo labels on the target domain. The core of both CDG and CSA is the Contrastive Regression (CR) loss, a novel contrastive loss for regression by pulling features with closer gaze directions closer together while pushing features with farther gaze directions farther apart. Experimentally, we choose ETH-XGAZE and Gaze-360 as the source domain and test the domain generalization and adaptation performance on MPIIGAZE, RT-GENE, Gaze-Capture, EyeDiap respectively. The results demonstrate that our CRGA achieves remarkable performance improvement compared with the baseline models and also outperforms the state-of-the-art domain adaptation approaches on gaze adaptation tasks.

Existing entropy models in learned image compression are cumbersome to generate fixed mean and variance for estimating Gaussian distributions for latent representation. In this paper, we propose a novel entropy model based on Gaussian process regression (GPR) that flexibly predicts the mean of Gaussians with posterior distributions characterized by covari-ance functions spanned in the high-dimensional feature space. Furthermore, we develop the rate-distortion optimization based on the proposed entropy model by approximating the bitrates with the evidence lower bound (ELBO) derived via variational inference for GPR. The proposed model can be seamlessly integrated into existing end-to-end optimized frame-works by substituting the masked convolution based autoregressive models. Experimental results demonstrate that the proposed model outperforms conventional image compression methods such as JPEG2000 and BPG, as well as recent learning based methods on the Kodak dataset in terms of rate-distortion performance.

Existing methods for cross-component prediction focus on the chroma intra pre-diction but neglect the chroma prediction across multiple frames. In this paper, we propose a novel compression framework that leverages chroma frame sampling and inter-frame chroma prediction to address this problem. Specifically, chroma frame sampling discards the chroma components in inter-predicted frames to further reduce the bit consumption, whereas inter-frame chroma prediction recovers the discarded chroma components with optical flow estimation and post-training optimization for a guaranteed fidelity. To our best knowledge, this paper is the first attempt to achieve deep learning-based inter prediction of chroma components. To accommodate the proposed framework, we redesign the HEVC codec to enable hybrid coding of luma and chroma components. Experimental results show that the proposed framework achieves up to 0.76% BD-rate reduction when compared to standard HEVC.

3-D point clouds rendering solid representations of scenes or objects often carry a tremendous amount of points, compulsorily requesting high-efficiency compression for storage and transmission. In this paper, we propose a novel$p$-Laplacian embedding graph dictionary learning algorithm for 3-D point cloud attribute compression. The proposed method integrates the underlying graph topology to the learned graph dictionary capitalizing on$p$-Laplacian eigenfunctions and leads to parsimonious representations of 3-D point clouds. We further devise alternating optimization with the help of ADMM to efficiently solve the resulting non-convex minimization problem.

We propose a structured pruning method to achieve a light-weighted decoder of learned image compression to accommodate various terminals. The structured pruning method identifies the effectiveness of each channel of decoder via gradient ascent and gradient descent while maintaining the encoder and entropy model. To our best knowledge, this paper is the first attempt to design a structured pruning method for universal pretrained learned image compression. Experimental results demonstrate that the proposed method can reduce about 40% parameters and save 25% inference time at the cost of 0.23 dB BD-PSNR and 4.33% BD-rate change.

Graph convolutional networks (GCNs) are widely adopted for spherical data processing, striking a balance between rotation equivariance and computation efficiency. However, current GCN-based approaches either apply radial filters with a fixed convolution kernel, or adopt attention schemes to learn a variable kernel based only on node features, which lack expressiveness and may suffer over-smoothing problem. To address this, in this paper, we propose a global-local attention-based spherical graph convolution (GlasGC) for spherical data representation. Specifically, the designed graph convolution includes a structure-enhanced local attention module to estimate the structure correlation efficiently based on topological characteristics of the given spherical graph. The structure correlation is then incorporated with the feature correlation to form the local attention that determines importance of a neighbor node to the center node. We also design a global attention module to obtain the importance scores of different nodes based on both node features and graph topology, which acts as a message-passing enhancer for informative nodes and can relieve the over-smoothing problem. Based on our proposed GlasGC, we design specific GlasGCNs for classification and semantic segmentation of spherical data. Empirical evaluations on these tasks demonstrate that our proposed GlasGCNs can generally achieve better classification and segmentation performance.

Generalizable face anti-spoofing (FAS) has drawn growing attention due to its robustness to unseen real scenarios. Existing domain generalization methods leverage adversarial learning or meta-learning to mitigate the domain bias and improve generalizability. However, these methods are heuristic and suffer from complicated min-max problems or cumbersome meta-updates. In this paper, we propose a simple yet effective Causal Intervention method for generalizable Face Anti-Spoofing, namely CIFAS. Firstly, we figure out the generalizability is undermined by a domain-aware confounder based on the structural causal model. Instantiating the confounder as the domain-specific factor, a domain embedding module is employed with Dirichlet mixup to obtain representative domain features. Consequently, we propose a novel backdoor adjustment model for causal intervention to capture the true causality and learn a robust FAS model. Our CIFAS is the first attempt to introduce causal learning into FAS. Extensive experiments on seven cross-dataset tests demonstrate that CIFAS outperforms the state-of-the-art methods.

Network quantization can facilitate the practical deployment of U-Net in various medical applications, especially medical image segmentation. However, existing quantization methods for U-Net do not sufficiently exploit the skip connection between its encoder and decoder structures. In this paper, we propose a novel mixed-precision quantization scheme for U-Net that leverages split convolution and skip-supervised quantization aware training to address this problem. Specifically, split convolution decouples the feature maps obtained by skip connection and up-sampling to enable fine-grained bitwidth allocation for mixed-precision quantization. Furthermore, we introduce supervision on skip connection for quantization aware training to compensate gradients in back propagation and enhance the skip features in finetuning. To our best knowledge, this is the first attempt to design mixed-precision quantization specifically for medical image segmentation with U-Net. Experimental results on four medical image datasets demonstrate that the proposed scheme outperforms existing U-Net quantization strategy and recent general-purpose mixed-precision quantizaton methods in medical image segementation.

Recent video colorization methods optimize correspondence estimation and information propagation in an end-to-end manner. However, they usually suffer from loss of fidelity due to the inaccurate inference of correspondence measurement. In this paper, we propose a post-training optimization (PTO) strategy to refine correspondence measurement in the end-to-end optimized framework. The proposed PTO strategy introduces a pseudo loss function to well approximate the target loss and guide the direction of updates. We further develop a video colorization method that incorporates PTO and optical flow to guarantee high-fidelity colorized frames in theory. Experimental results demonstrate that the proposed method achieves state-of-the-art PSNR performance in video colorization on the DAVIS dataset and common test sequences for video coding. Furthermore, the proposed method can be employed into video compression and achieves competitive rate-distortion performance with the recent High Efficiency Video Coding (HEVC) standard.

Differentiable architecture search (DARTS) has achieved success in searching powerful network architectures but suffers from unstable search, high search cost from repetitive attempts and unawareness of computational cost. In this paper, we propose a novel approach, namely Efficient and Stable Differentiable Architecture Search (ES-DARTS), that leverages decoupled search strategy and variational proxy pruning to achieve efficient search for efficient neural networks. Specifically, the decoupled search strategy stabilizes the search via bridging the gap between search and evaluation, and achieves acceleration with decoupled optimization, while the variational proxy pruning introduces structural pruning into DARTS to accommodate varying constraints on computational cost. ES-DARTS can be a plug-and-play module for DARTS-based approaches to achieve improved performance with reduced search cost and FLOPS. Experimental results demonstrate that ES-DARTS reduces top-1 error rate to 2.71% with $10 \times$ acceleration of DARTS on CIFAR10. ES-DARTS finds an architecture of 2.80% top-1 error rate with only 2.27 MB parameters on CIFAR-10 and of 29.4% top-1 error rate with only 322M FLOPS when transferred to ImageNet.

Contrastive learning has shown promising potential in self-supervised spatio-temporal representation learning. Most works naively sample different clips to construct positive and negative pairs. However, we observe that this formulation inclines the model towards the background scene bias. The underlying reasons are twofold. First, the scene difference is usually more noticeable and easier to discriminate than the motion difference. Second, the clips sampled from the same video often share similar backgrounds but have distinct motions. Simply regarding them as positive pairs will draw the model to the static background rather than the motion pattern. To tackle this challenge, this paper presents a novel dual contrastive formulation. Concretely, we decouple the input RGB video sequence into two complementary modes, static scene and dynamic motion. Then, the original RGB features are pulled closer to the static features and the aligned dynamic features, respectively. In this way, the static scene and the dynamic motion are simultaneously encoded into the compact RGB representation. We further conduct the feature space decoupling via activation maps to distill static- and dynamic-related features. We term our method as Dual Contrastive Learning for spatio-temporal Representation (DCLR). Extensive experiments demonstrate that DCLR learns effective spatio-temporal representations and obtains state-of-the-art or comparable performance on UCF-101, HMDB-51, and Diving-48 datasets.

In this paper, we present a meta reinforcement learning (Meta-RL)-based neural adaptive bitrate streaming (ABR) algorithm that is able to rapidly adapt its control policy to the changing network throughput dynamics. Specifically, to allow rapid adaptation, we discuss the necessity of detaching the inference of throughput dynamics with the universal control mechanism that is in essence shared by all potential throughput dynamics for neural ABR algorithms. To meta-learn the ABR policy, we then build up a model-free system framework, composed of a probabilistic latent encoder that infers the underlying dynamics from the recent throughput context, and a policy network that is conditioned on latent variable and learns to quickly adapt to new environments. Additionally, to address the difficulties caused by training the policy on mixed dynamics, on-policy RL (or imitation learning) algorithms are suggested for policy training, with a mutual information-based regularization to make the latent variable more informative about the policy. Finally, we implement our algorithm's meta-training and meta-adaptation procedures under a variety of throughput dynamics. Empirical evaluations on different QoE metrics and multiple datasets containing real-world network traces demonstrate that our algorithm outperforms state-of-the-art ABR algorithms, in terms of the performance on the average chunk QoE, consistency and fast adaptation across a wide range of throughput patterns.

Entropy coding is fundamental for reducing the coding redundancy in image compression. However, existing entropy models for learned image compression are restricted by independent or autoregressive modeling based on presumed distribution generated from the family of Gaussian functions. In this paper, we propose a novel tensor network-based entropy model that can explicitly infer the joint distribution of image representation for learned image compression. We utilize a tree tensor network (TTN) to enable exact computation for probabilities of image representation. Specifically, we produce efficient tensor representation for entropy modeling based on bit planes slicing with gray code. Furthermore, we leverage tensor contraction to accurately calculate the partition function and jointly predict the entire bit plane for entropy coding. To our best knowledge, this paper is the first to directly infer the joint distribution of image representation in learned image compression without any presumed condition. Experimental results demonstrate that the proposed method is competitive with the state-of-the-art in both lossless and lossy image compression.

360-degree videos pose a challenge to their transmission due to the ultra-high resolution. To achieve efficient streaming, current tile-based schemes divide the 2D projection of spherical frames into small rectangular tiles, which are treated as individual sources for encoding and then delivered according to user's actual viewport. However, the trade-off between transmission and encoding efficiency, as controlled by encoding parameters such as tile sizes and quantization steps, has not been fully studied yet. In this paper, we propose a tile-based encoding scheme to strike this trade-off, by selecting the optimal encoding parameters that minimize the distortion of video displayed for user's viewport under a given transmission capacity. Specifically, we quantify the transmission efficiency by estimating the average number of pixels required for transmission to cover the user's viewport, while a CNN-based predictor is trained to predict the encoding efficiency based on video content characteristics, expressed as number of bits needed for encoding one pixel. We then formulate a rate-distortion optimization problem for tile-based encoding, which is solved efficiently by decreasing the feasible set of encoding parameters. Simulation results demonstrate that the proposed optimal tile-based encoding outperforms both fixed tiling and state-of-the-art adaptive tiling schemes in terms of rate-distortion behavior.

Semi-supervised learning, which assigns pseudo labels with models trained using limited labeled data, has been widely used in object detection to reduce the labeling cost. However, the provided pseudo annotations inevitably suffer noise since the initial model is not perfect. To address this issue, this paper introduces contrastive learning into semi-supervised object detection, and we claim that contrastive loss, which inherently relies on data augmentations, is much more robust than traditional softmax regression for noisy labels. To take full advantage of it in the detection task, we incorporate labels prior to contrastive loss and leverage plenty of region proposals to enhance diversity, which is crucial for contrastive learning. In this way, the model is optimized to make the region-level features with the same class be translation and scale invariant. Furthermore, we redesign the negative memory bank in contrastive learning to make the training more efficient. As far as we know, we are the first attempt that introduces contrastive learning in semi-supervised object detection. Experimental results on detection benchmarks demonstrate the superiority of our method. Notably, our method achieves 79.9% accuracy on VOC, which is 6.2% better than the supervised baseline and 0.7% improvement compared with the state-of-the-art method.

In this paper, we propose the K-Shot Contrastive Learning (KSCL) of visual features by applying multiple augmentations to investigate the sample variations within individual instances. It aims to combine the advantages of inter-instance discrimination by learning discriminative features to distinguish between different instances, as well as intra-instance variations by matching queries against the variants of augmented samples over instances. Particularly, for each instance, it constructs an instance subspace to model the configuration of how the significant factors of variations in K-shot augmentations can be combined to form the variants of augmentations. Given a query, the most relevant variant of instances is then retrieved by projecting the query onto their subspaces to predict the positive instance class. This generalizes the existing contrastive learning that can be viewed as a special one-shot case. An eigenvalue decomposition is performed to configure instance subspaces, and the embedding network can be trained end-to-end through the differentiable subspace configuration. Experiment results demonstrate the proposed K-shot contrastive learning achieves superior performances to the state-of-the-art unsupervised methods.

Tile-based rate adaption can improve the quality of experience (QoE) for adaptive 360-degree video streaming under constrained network conditions, which, however, is a challenging problem due to the requirements of accurate prediction for users’ viewports and optimal bitrate allocation for tiles. In this paper, we propose a strategy that deploys reinforcement learning-based Rate Adaptation with adaptive Prediction and Tiling for 360-degree video streaming, named RAPT360, to address these challenges. Specifically, to improve the accuracy of the state-of-the-art viewport prediction approaches, we fit the time-varying Laplace distribution-based probability density function of the prediction error for different prediction lengths. On the basis of that, we develop a viewport identification method to determine the viewport area of a user depending on the buffer occupancy, where the obtained viewport can cover the real viewport with any given probability confidence level. We then propose a viewport-aware adaptive tiling scheme to improve the bandwidth efficiency, where three types of tile granularities are allocated according to the shape and position of the 2-D projection of that viewport. By establishing an adaptive streaming model and QoE metric specific to 360-degree videos, we finally formulate the rate adaptation problem for tile-based 360-degree video streaming as a non-linear discrete optimization problem that targets at maximizing the long-term user QoE under a bandwidth-constrained network. To efficiently solve this problem, we model the rate adaptation logic as a Markov decision process (MDP) and employ the deep reinforcement learning (DRL)-based algorithm to dynamically learn the optimal bitrate allocation of tiles. Extensive experimental results show that RAPT360 achieves a performance gain of at least 1.47 dB on average chunk QoE, including a video quality improvement of at least 1.33 dB, in comparison to the existing strategies for tile-based adaptive 360-degree video streaming.

The Cox proportional hazards model is a popular semi-parametric model for survival analysis. In this paper, we aim at developing a federated algorithm for the Cox proportional hazards model over vertically partitioned data (i.e., data from the same patient are stored at different institutions). We propose a novel algorithm, namely VERTICOX, to obtain the global model parameters in a distributed fashion based on the Alternating Direction Method of Multipliers (ADMM) framework. The proposed model computes intermediary statistics and exchanges them to calculate the global model without collecting individual patient-level data. We demonstrate that our algorithm achieves equivalent accuracy for the estimation of model parameters and statistics to that of its centralized realization. The proposed algorithm converges linearly under the ADMM framework. Its computational complexity and communication costs are polynomially and linearly associated with the number of subjects, respectively. Experimental results show that VERTICOX can achieve accurate model parameter estimation to support federated survival analysis over vertically distributed data by saving bandwidth and avoiding exchange of information about individual patients. The source code for VERTICOX is available at: https://github.com/daiwenrui/VERTICOX.

Model quantization is essential to deploy deep convolutional neural networks (DCNNs) on resource-constrained devices. In this article, we propose a general bitwidth assignment algorithm based on theoretical analysis for efficient layerwise weight and activation quantization of DCNNs. The proposed algorithm develops a prediction model to explicitly estimate the loss of classification accuracy led by weight quantization with a geometrical approach. Consequently, dynamic programming is adopted to achieve optimal bitwidth assignment on weights based on the estimated error. Furthermore, we optimize bitwidth assignment for activations by considering the signal-to-quantization-noise ratio (SQNR) between weight and activation quantization. The proposed algorithm is general to reveal the tradeoff between classification accuracy and model size for various network architectures. Extensive experiments demonstrate the efficacy of the proposed bitwidth assignment algorithm and the error rate prediction model. Furthermore, the proposed algorithm is shown to be well extended to object detection.

With the advent of data science, the analysis of network or graph data has become a very timely research problem. A variety of recent works have been proposed to generalize neural networks to graphs, either from a spectral graph theory or a spatial perspective. The majority of these works, however, focus on adapting the convolution operator to graph representation. At the same time, the pooling operator also plays an important role in distilling multiscale and hierarchical representations, but it has been mostly overlooked so far. In this article, we propose a parameter-free pooling operator, called iPool, that permits to retain the most informative features in arbitrary graphs. With the argument that informative nodes dominantly characterize graph signals, we propose a criterion to evaluate the amount of information of each node given its neighbors and theoretically demonstrate its relationship to neighborhood conditional entropy. This new criterion determines how nodes are selected and coarsened graphs are constructed in the pooling layer. The resulting hierarchical structure yields an effective isomorphism-invariant representation of networked data on arbitrary topologies. The proposed strategy achieves superior or competitive performance in graph classification on a collection of public graph benchmark data sets and superpixel-induced image graph data sets.

Unsupervised training of deep representations has demonstrated remarkable potentials in mitigating the prohibitive expenses on annotating labeled data recently. Among them is predicting transformations as a pretext task to self-train representations, which has shown great potentials for unsupervised learning. However, existing approaches in this category learn representations by either treating a discrete set of transformations as separate classes, or using the Euclidean distance as the metric to minimize the errors between transformations. None of them has been dedicated to revealing the vital role of the geometry of transformation groups in learning representations. Indeed, an image must continuously transform along the curved manifold of a transformation group rather than through a straight line in the forbidden ambient Euclidean space. This suggests the use of geodesic distance to minimize the errors between the estimated and groundtruth transformations. Particularly, we focus on homographies, a general group of planar transformations containing the Euclidean, similarity and affine transformations as its special cases. To avoid an explicit computing of intractable Riemannian logarithm, we project homographies onto an alternative group of rotation transformations SR(3) with a tractable form of geodesic distance. Experiments demonstrate the proposed approach to Auto-Encoding Transformations exhibits superior performances on a variety of recognition problems.

Cloud service has been emerging as a promising alternative to handle massive volumes of video sequences triggered by increasing demands on video service, especially surveillance and entertainment. Lossless compression of encoded video bitstreams can further eliminate the redundancies without altering the contents and facilitate the efficiency of cloud storage. In this paper, we propose a novel lossless compression scheme to further compress the video bitstreams generated by the state-of-the-art hybrid coding frameworks like H.264/AVC and HEVC. Different from transcoding, the proposed scheme develops frequency prediction and Macro Block (MB) merging to eliminate the redundancies remained in the intra-and inter-predicted frames with a strict guarantee of video fidelity. To our best knowledge, this paper is the first attempt to realize lossless compression of video bitstreams generated by advanced coding standards H.264/AVC and HEVC. Experimental results demonstrate that the proposed scheme can achieve compression gains of 17.38% and 10.85% on standard test sequences and surveillance videos, respectively.

Deep learning has been widely adopted in compression sensing (CS) to achieve superior reconstruction quality, but is restricted by the black-box architecture in network design and lack of interpretability. In this paper, we propose a novel deep network-based CS framework via unfolding the$\ell_{0}$-constrained convolutional sparse coding (CSC). The proposed method incorporates deep neural networks (DNNs) into the state-of-the-art optimization paradigm of alternating direction method of multipliers (ADMM). CS reconstruction can be improved with the compact and shift-invariant sparse representation under the merits of CSC and DNNs with learnable parameters. Convergence analysis demonstrates that the proposed method is guaranteed to converge, when the network parameters are reused for all the building blocks. Experimental results demonstrate that the proposed method is competitive or outperform the state-of-the-art DNN-based methods in CS of natural images and magnetic resonance imaging (MRI).

Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation system consisting of one master and$N$workers characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with$L$model parameters. We show that it is sufficient to provide at most$N$levels of redundancies for tolerating$0,1, \cdots, N-1$stragglers, respectively. Consequently, we propose an optimal block coordinate gradient coding scheme based on a stochastic optimization problem that optimizes the partition of the$L$coordinates into$N$blocks, each with identical redundancy, to minimize the expected overall runtime for collaboratively computing the gradient. We obtain an optimal solution using a stochastic projected subgradient method and propose two low-complexity approximate solutions with closed-from expressions, for the stochastic optimization problem. We also show that under a shifted-exponential distribution, for any$L$, the expected overall runtimes of the two approximate solutions and the minimum overall runtime have sub-linear multiplicative gaps in$N$. To the best of our knowledge, this is the first work that optimizes the redundancies of gradient coding introduced across the coordinates of gradients.

Application of point clouds is in critical demand, which, however, are composed of large amounts of data and difficult to stream in bandwidth-constrained networks. To address this, we propose a QoE-driven and tile-based adaptive streaming approach for point clouds, to reduce transmission redundancy and maximize user’s QoE. Specifically, by utilizing the perspective projection, we model the QoE of a 3D tile as a function of the bitrate of its representation, user’s view frustum and spatial position, occlusion between tiles, and the resolution of rendering device. We then formulate the QoE-optimized rate adaptation problem as a multiple-choice knapsack problem that allocates bitrates for different tiles under a given transmission capacity. We equivalently convert it as a submodular function maximization problem subject to knapsack constraints, and develop a practical greedy algorithm with a theoretical performance guarantee. Experimental results further demonstrate superiority of the proposed rate adaptation algorithm over existing schemes, in terms of both user’s visual quality and transmission efficiency.

Existing differentiable neural architecture search approaches simply assume the architectural distribution on each edge is independent of each other, which conflicts with the intrinsic properties of architecture. In this paper, we view the architectural distribution as the latent representation of specific data points. Then we propose Variational Information Maximization Neural Architecture Search (VIM-NAS) to leverage a simple yet effective convolutional neural network to model the latent representation, and optimize for a tractable variational lower bound to the mutual information between the data points and the latent representations. VIM-NAS automatically learns a nearly one-hot distribution from a continuous distribution with extremely fast convergence speed, e.g., converging with one epoch. Experimental results demonstrate VIM-NAS achieves state-of-the-art performance on various search spaces, including DARTS search space, NAS-Bench-1shot1, NAS-Bench-201, and simplified search spaces S1-S4. Specifically, VIM-NAS achieves a top-1 error rate of 2.45% and 15.80% within 10 minutes on CIFAR-10 and CIFAR-100, respectively, and a top-1 error rate of 24.0% when transferred to ImageNet.

Learned image compression has achieved improved rate-distortion performance with end-to-end optimized framework based on deep neural networks. However, context-based entropy modeling for learned image compression cannot simultaneously achieve enhanced efficiency and sufficiently exploiting the channel-wise correlations. In this paper, we propose a novel framework for learned image compression with channel-wise grouped context modeling. The proposed framework presents channel-wise grouping to explicitly exploit the channel-wise correlations and develop a grouped 3-D context model to achieve efficient entropy coding with a guarantee of rate-distortion performance. The proposed framework achieves competitive performance with a significantly reduced decoding complexity in comparison to 3-D context models.

State-of-the-art deep reinforcement learning (DRL) algorithms tend to overfit due to the model discrepancy between source and target environments. Though applying domain randomization during training can improve the average performance by randomly generating a sufficient diversity of environments in simulator, the worst-case environment is still neglected without any performance guarantee. Since the average and worst-case performance are both important for generalization in RL, in this paper, we propose a policy optimization approach for concurrently improving the policy’s performance in the average and worst-case environment. We theoretically derive a lower bound for the worst-case performance of a given policy by relating it to the expected performance. Guided by this lower bound, we formulate an optimization problem to jointly optimize the policy and sampling distribution, and prove that by iteratively solving it the worst-case performance is monotonically improved. We then develop a practical algorithm, named monotonic robust policy optimization (MRPO). Experimental evaluations in several robot control tasks demonstrate that MRPO can generally improve both the average and worst-case performance in the source environments for training, and facilitate in all cases the learned policy with a better generalization capability in some unseen testing environments.

In this paper, we propose BayesMPC, an uncertainty-aware robust adaptive bitrate (ABR) algorithm on the basis of Bayesian neural network (BNN) and model predictive control (MPC). Specifically, to improve the capacity of learning transition probability of the network throughput, we adopt a BNN-based predictor that is able to predict the statistical distribution of future throughput from the past throughput by not only considering the aleatoric uncertainty (e.g., noise), but also capturing the epistemic uncertainty incurred by lack of adequate training samples. We further show that by using the negative log-likelihood loss function to train this BNN-based throughput predictor, the generalization error can be minimized with the guarantee of PAC-Bayesian theorem. Rather than a point estimate, the learnt uncertainty can contribute to a confidence region for the future throughput, the lower bound of which then leads to an uncertainty-aware robust MPC strategy to maximize the worst-case user quality-of-experience (QoE) w.r.t. this confidence region. Finally, experimental results on three real-world network trace datasets validate the efficiency of both the proposed BNN-based predictor and uncertainty-aware robust MPC strategy, and demonstrate the superior performance compared to other baselines, in terms of both the overall QoE performance and generalization across all ranges of heterogeneous network and user conditions.

Mixed-precision quantization with adaptive bitwidth allocation for neural network has achieved higher compression rate and accuracy in classification task. However, it has not been well explored for object detection networks. In this paper, we propose a novel mixed-precision quantization scheme with dynamical Hessian matrix for object detection networks. We iteratively select a layer with the lowest sensitivity based on the Hessian matrix and downgrade its precision to reach the required compression ratio. The L-BFGS algorithm is utilized for updating the Hessian matrix in each quantization iteration. Moreover, we specifically design the loss function for objection detection networks by jointly considering the quantization effects on classification and regression loss. Experimental results on RetinaNet and Faster R-CNN show that the proposed DHMQ achieves state-of-the-art performance for quantized object detec-tors.

Differentiable architecture search (DARTS) enables effective neural architecture search (NAS) using gradient descent, but suffers from high memory and computational costs. In this paper, we propose a novel approach, namely Partially-Connected DARTS (PC-DARTS), to achieve efficient and stable neural architecture search by reducing the channel and spatial redundancies of the super-network. In the channel level, partial channel connection is presented to randomly sample a small subset of channels for operation selection to accelerate the search process and suppress the over-fitting of the super-network. Side operation is introduced for bypassing (non-sampled) channels to guarantee the performance of searched architectures under extremely low sampling rates. In the spatial level, input features are down-sampled to eliminate spatial redundancy and enhance the efficiency of the mixed computation for operation selection. Furthermore, edge normalization is developed to maintain the consistency of edge selection based on channel sampling with the architectural parameters for edges. Theoretical analysis shows that partial channel connection and parameterized side operation are equivalent to regularizing the super-network on the weights and architectural parameters during bilevel optimization. Experimental results demonstrate that the proposed approach achieves higher search speed and training stability than DARTS. PC-DARTS obtains a top-1 error rate of 2.55 percent on CIFAR-10 with 0.07 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.1 percent on ImageNet (under the mobile setting) within 2.8 GPU-days.

Existing 3-D point cloud attribute compression methods are rigid for complex signals defined in spatially irregular domain, as they solely depend on geometric information or adopt pre-defined analytic bases for transform. In this article, we propose a multi-scale structured dictionary learning algorithm for 3-D point cloud attribute compression. The proposed multi-scale dictionary leverages hierarchical sparsity within signals by adapting atoms in an arborescent structure. It supports hierarchical sparse transform for encoding voxelized point cloud attributes with progressive refinement of high-frequency details from coarser to finer scales. To represent spatial blocks with varying amounts of occupied voxels, a non-uniform binary weight matrix is incorporated in dictionary learning to characterize the dimensional irregularity of signals. To guarantee optimal approximation with improved efficiency, alternating optimizations including Alternating Direction Method of Multipliers andGauss-Seideliterations are adopted for the weighted mixed$\ell _{2}/\ell _{1}$-minimization. Furthermore, we develop a lossy attribute compression framework by integrating sparse transform based prediction with dedicated quantization and entropy coding routines. Experimental results demonstrate that the proposed framework outperforms the state-of-the-art transform-based coding methods and is competitive with the most recent MPEG PCC test models in point cloud attribute compression.

In this paper, we investigate the adaptive video delivery for multiple users over time-varying and mutually interfering multi-cell wireless networks. The key research challenge is to jointly design the physical-layer resource allocation scheme and application-layer rate adaptation logic, such that the users' long-term fair quality of experience (QoE) can be maximized. Due to the timescale mismatch between these two layers and the asynchrony of user requests, however, it is difficult to directly model the cross-layer stochastic control problem by using a reinforcement learning framework. To address this difficulty, we propose a novel two-level decision framework where an optimization-based beamforming scheme (performed at the base stations) and a deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals) are, respectively, developed, such that a highly complex long-term multi-user QoE fairness problem is decomposed into some relatively simple problems and solved effectively. Our strategy represents a significant departure from the existing schemes with consideration of either a short-term multi-user QoE maximization or a long-term single-user point-to-point QoE maximization. Extensive simulations demonstrate that the proposed cross-layer design is effective and promising.

The task of reidentifying groups of people under different camera views is an important yet less-studied problem. Group reidentification (Re-ID) is a very challenging task since it is not only adversely affected by common issues in traditional single-object Re-ID problems, such as viewpoint and human pose variations, but also suffers from changes in group layout and group membership. In this paper, we propose a novel concept of group granularity by characterizing a group image by multigrained objects: individual people and subgroups of two and three people within a group. To achieve robust group Re-ID, we first introduce multigrained representations which can be extracted via the development of two separate schemes, that is, one with handcrafted descriptors and another with deep neural networks. The proposed representation seeks to characterize both appearance and spatial relations of multigrained objects, and is further equipped with importance weights which capture variations in intragroup dynamics. Optimal group-wise matching is facilitated by a multiorder matching process which, in turn, dynamically updates the importance weights in iterative fashion. We evaluated three multicamera group datasets containing complex scenarios and large dynamics, with experimental results demonstrating the effectiveness of our approach.

IoT devices produce a wealth of data desired for learning models to empower more intelligent applications. However, such data is often privacy sensitive making data owners reluctant upload their data to a central server for learning purposes. Federated learning provides a promising privacy-preserving learning approach, which decouples the model training from the need of accessing to the sensitive data. However, realizing a deployed, dependable federated learning system faces critical challenges, such as frequent dropouts of learning workers, heterogeneity of workers computation, and limited communication. In this paper, we focus on the systems aspects to advance federated learning and contribute a highly efficient and reliable distributed federated learning framework, FedMax, aiming to tackle these challenges. In designing FedMax, we contribute new techniques in light of the properties of a real federated learning setting, including a relaxed synchronization communication scheme and a similarity-based worker selection approach. We have implemented a prototype of FedMax and evaluated FedMax upon multiple popular machine learning models and datasets, showing that FedMax significantly increases the robustness of a federated learning system, speeds up the convergence rate by 25%, and increases the system efficiency by 50%, in comparison with state-of-the-art approaches.

Convolutional neural networks (CNNs) designed for low-dimensional regular grids will unfortunately lead to non-optimal solutions for analyzing spherical images, due to their different geometrical properties from planar images. In this paper, we generalize the grid-based CNNs to a non-Euclidean space by taking into account the geometry of spherical surfaces and propose a Spherical Graph Convolutional Network (SGCN) to encode rotation equivariant representations. Specifically, we propose a spherical graph construction criterion showing that a graph needs to be regular by evenly covering the spherical surfaces in order to design a rotation equivariant graph convolutional layer. For the practical case where the perfectly regular graph does not exist, we design two quantitative measures to evaluate the degree of irregularity for a spherical graph. The Geodesic ICOsahedral Pixelation (GICOPix) is adopted to construct spherical graphs with the minimum degree of irregularity compared to the current popular pixelation schemes. In addition, we design a hierarchical pooling layer to keep the rotation-equivariance, followed by a transition layer to enforce the invariance to the rotations for spherical image classification. We evaluate the proposed graph convolutional layers with different pixelations schemes in terms of equivariance errors. We also assess the effectiveness of the proposed SGCN in fulfilling rotation-invariance by the invariance error of the transition layers and recognizing the spherical images and 3D objects.

Generative model has emerged as a disruptive alternative for lossy compression of natural images, but suffers from the low-fidelity reconstruction. In this paper, we propose a noise-to-compression variational antoencoder (NC-VAE) to achieve efficient rate-distortion optimization (RDO) for end-to-end optimized image compression with a guarantee of fidelity. The proposed NC-VAE improves rate-distortion performance by adaptively adjusting the distribution of latent variables with trainable noise perturbation. Consequently, high-efficiency RDO is developed based on the distribution of latent variables for simplified decoder. Furthermore, robust end-to-end learning is developed over the corrupted inputs to suppress the deformation and color drift in standard VAE based generative models. Experimental results show that NC-VAE outperforms the state-of-the-art lossy image coders and recent end-to-end optimized compression methods in low bit-rate region, i.e., below 0.2 bits per pixel (bpp).

Reversible residual network naturally extends the linear lifting scheme with no theoretic guarantee. In this paper, we propose a reversible autoencoder (Rev-AE) with this extended non-linear lifting scheme to improve image reconstruction. Nonlinear prediction and update operators are designed based on shallow convolutional neural networks to model multilayer non-linearities. Different from existing autoencoders, Rev-AE support efficient image reconstruction with parameters reusable for the symmetric encoder and decoder. Rev-AE forms a set of related frames to guarantee perfect reconstruction with the non-linear extension of classic lifting scheme. Lower and upper bounds are developed for the set of frames to relate with the singular values for each non-linear operator. Furthermore, we employ Rev-AE into lossy image compression to evaluate its effectiveness on image reconstruction. Experimental results show that Rev-AE achieves competitive performance in comparison to the state-of-the-art.

Existing deep learning based methods for 3D human pose estimation cannot be deployed on resource-constrained devices. In this paper, we propose a Tiny-HourglassNet for 3D human pose estimation that improves the efficiency of Stacked Hourglass Networks with a guarantee of estimation performance. To be concrete, we develop an efficient Tiny-Hourglass backbone with two different types of ShuffleNet V2 blocks. At the meantime, we present a Simple Estimation Head (SEH) to reduce the resource utility for depth prediction. Furthermore, we develop two feature enhancement modules, namely Feature Enhancement Module (FEM) and Intermediate Enhancement Module (IEM), to improve the feature representation ability of Tiny-HourglassNet without evident increment of resource usage and computational complexity. Experimental results demonstrate that Tiny-HourglassNet achieves equivalent accuracy with a reduction of 80% complexity (FLOPs) in comparison to the baselines.

The k-nearest neighbors (KNN) algorithm has been widely adopted to construct graph convolutional networks (GCNs) for point cloud segmentation. However, the ℓ2norm cannot discriminate the multi-dimensional structures within a point cloud. In this paper, we propose a novel structure-aware graph construction for point clouds that compensates the ℓ2norm with per-dimension differences of the signal. The proposed method dynamically calculates the similarity ratio to determine the dimension-based proximity of the pair of points. Consequently, it improves both the spatial and spectral GCNs with the capability of aggregating information from relevant neighbors for point cloud segmentation. As a model-agnostic method, it can be seamlessly embedded into arbitrary GCN architectures during the graph construction phase. Experimental results demonstrate that the proposed method can improve classification accuracy around the joint areas of objects.

Bayesian methods have improved the interpretability and stability of neural architecture search (NAS). In this paper, we propose a novel probabilistic approach, namely Semi-Implicit Variational Dropout one-shot Neural Architecture Search (SI-VDNAS), that leverages semi-implicit variational dropout to support architecture search with variable operations and edges. SI-VDNAS achieves stable training that would not be affected by the over-selection of skip-connect operation. Experimental results demonstrate that SI-VDNAS finds a convergent architecture with only 2.7 MB parameters within 0.8 GPU-days and can achieve 2.60% top-1 error rate on CIFAR-10. The convergent architecture can obtain a top-1 error rate of 16.20% and 25.6% when transferred to CIFAR-100 and ImageNet (mobile setting).

To enable DNNs on edge devices like mobile phones, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pre-trained model by low-rank decomposition; however, small approximation errors in parameters can ripple over a large prediction loss. As a result, performance usually drops significantly and a sophisticated effort on fine-tuning is required to recover accuracy. Apparently, it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training process. We propose Trained Rank Pruning (TRP), which alternates between low rank approximation and training. TRP maintains the capacity of the original network while imposing low-rank constraints during training. A nuclear regularization optimized by stochastic sub-gradient descent is utilized to further promote low rank in TRP. The TRP trained network inherently has a low-rank structure, and is approximated with negligible performance loss, thus eliminating the fine-tuning process after low rank decomposition. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression methods using low rank approximation.

Federated learning has successfully mitigated the privacy concerns for deep learning over distributed data, but suffers from performance decay led by the non-identical distribution of data. In this paper, we propose a novel framework, namely FedExg, to improve federated learning for non-IID data with model exchange. Two strategies, i.e., FedExg-S and FedExg-A, are developed to realize model exchange in a privacy-preserving fashion. FedExg-S randomly shuffles the models for broadcasting, while FedExg-A improves FedExg-S with partial aggregation to avoid model inversion attack. Theoretical analysis demonstrates that FedExg achieves a reduced upper bound of average regret in comparison to traditional model averaging methods. Experimental results show that FedExg improves image classification tasks on MNIST and CIFAR-10 dataset.

Recently, deep learning based compressive sensing (CS) methods show superior reconstruction performance. However, these approaches are restricted for compressive video sampling in practical design of imaging devices. In this paper, we propose a novel deep network-based CS framework for efficient optical implementation. The proposed framework, dubbed DSCR-Net, consists of a diffractive sensing network employing light diffraction for efficient sampling and a complex-valued neural network for reconstruction, respectively. The diffractive sensing network can achieve real-time sampling at the speed of light. Furthermore, complex-valued neural network is developed to facilitate the reconstruction quality from the complex-valued measurements by jointly considering their real and imaginary parts. Extensive experiments demonstrate that DSCR-Net outperforms the state-of-the-art CS methods in the low sampling-rate region with a potential of immediate implementation of imaging device.

The non-Euclidean geometry characteristic poses a challenge to the saliency prediction for 360-degree images. Since spherical data cannot be projected onto a single plane without distortion, existing saliency prediction methods based on traditional CNNs are inefficient. In this paper, we propose a saliency prediction framework for 360-degree images based on graph convolutional networks (SalGCN), which directly applies to the spherical graph signals. Specifically, we adopt the Geodesic ICOsahedral Pixelation (GICOPix) to construct a spherical graph signal from a spherical image in equirectangular projection (ERP) format. We then propose a graph saliency prediction network to directly extract the spherical features and generate the spherical graph saliency map, where we design an unpooling method suitable for spherical graph signals based on linear interpolation. The network training process is realized by modeling the node regression problem of the input and output spherical graph signals, where we further design a Kullback-Leibler (KL) divergence loss with sparse consistency to make the sparseness of the saliency map closer to the ground truth. Eventually, to obtain the ERP format saliency map for evaluation, we further propose a spherical crown-based (SCB) interpolation method to convert the output spherical graph saliency map into a saliency map in ERP format. Experiments show that our SalGCN can achieve comparable or even better saliency prediction performance both subjectively and objectively, with a much lower computation complexity.

End-to-end optimized image compression has emerged as a disruptive technique to reduce the spatial redundancies with an improved reconstruction quality. However, existing entropy model for latent representations cannot sufficiently exploit their spatial and channel-wise correlations. In this paper, we propose a novel entropy model based on spatial-channel contexts for end-to-end optimized image compression. The proposed model jointly leverages spatial structural dependencies and channel-wise correlations to improve the probabilistic estimation of latent representations. Instead of complex autoregressive hyperprior network, shallow artificial neural networks (ANNs) incorporating 3-D masks are developed to efficiently realize the entropy model with a guarantee of causality. Experimental results demonstrate that the proposed model achieves competitive rate-distortion performance and reduces model complexity in comparison to recent end-to-end optimized methods for image compression.

Collecting fine-grained labels usually requires expert-level domain knowledge and is prohibitive to scale up. In this paper, we propose Attribute Mix, a data augmentation strategy at attribute level to expand the fine-grained samples. The principle lies in that attribute features are shared among fine-grained sub-categories, and can be seamlessly transferred among images. Toward this goal, we propose an automatic attribute mining approach to discover attributes that belong to the same supercategory, and Attribute Mix is operated by mixing semantically meaningful attribute features from two images. Attribute Mix is a simple but effective data augmentation strategy that can significantly improve the recognition performance without increasing the inference budgets. Extensive experiments and ablation studies show that the proposed method consistently outperforms the state-of-the-art methods on challenging benchmarks including CUB-200-2011, FGVC-Aircraft, and Stanford Cars.

Existing decentralized coded caching solutions cannot guarantee small loads in the general scenario with arbitrary file sizes and cache sizes. In this paper, we propose an optimization framework for decentralized coded caching in the general scenario to minimize the worst-case load and average load (under an arbitrary file popularity), respectively. Specifically, we first propose a class of decentralized coded caching schemes for the general scenario, which are specified by a general caching parameter and include several known schemes as special cases. Then, we optimize the caching parameter to minimize the worst-case load and average load, respectively. Each of the two optimization problems is a challenging nonconvex problem with a nondifferentiable objective function. For each optimization problem, we develop an iterative algorithm to obtain a stationary point using techniques for solving Complementary Geometric Programming (GP). We also obtain a low-complexity approximate solution by solving an approximate problem with a differentiable objective function which is an upper bound on the original nondifferentiable one, and characterize the performance loss caused by the approximation. Finally, we present two information-theoretic converse bounds on the worst-case load and average load (under an arbitrary file popularity) in the general scenario, respectively. To the best of our knowledge, this is the first work that provides optimization-based decentralized coded caching schemes and information-theoretic converse bounds for the general scenario.

Conventional scalable compressive video sampling cannot sufficiently exploit the structured sparsity within video sequences. This paper proposes a novel quality scalable structured compressive video sampling (SS-CVS) framework with hierarchical subspace learning to support video transmission over heterogeneous network conditions. The proposed framework incorporates the union of data-driven subspaces (UoDS) model to introduce structured sparsity into sensing matrix for progressive acquisition of measurements. Hierarchical subspace learning is developed to generate the subspaces with their bases based on adaptive subspace clustering in a progressive fashion. Consequently, two hierarchical structures are derived to enable progressive mapping of subspaces and bases on the structured sensing matrices to support quality scalability. To guarantee the convergence of hierarchical subspace learning, the training set is updated with the adaptive group set from the reconstructed reference frames. It is demonstrated that SS-CVS can guarantee stable recovery for each quality layer under the constraints of block restricted isometric property (RIP). Furthermore, the proposed SS-CVS model is generalized to tensor subspaces for scalable compressive sampling of high-dimensional signals. Experimental results demonstrate that the proposed algorithm can achieve quality scalabilities with an improved reconstructed performance in comparison to the state-of-the-art scalable compressive video sampling methods.

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions.

This paper introduces a new model for Weakly Supervised Object Localization (WSOL) problems where only image-level supervision is provided. The key to solve such problems is to infer the object locations accurately. Previous methods usually model the missing object locations as latent variables, and alternate between updating their estimates and learning a detector accordingly. However, the performance of such alternative optimization is sensitive to the quality of the initial latent variables and the resulted localization model is prone to overfitting to improper localizations. To address these issues, we develop a Prior-induced Multi-view Learning Localization Network (PML-LocNet) which exploits both view diversity and sample diversity to improve object localization. In particular, the view diversity is imposed by a two-phase multi-view learning strategy, with which the complementarity among learned features from different views and the consensus among localized instances from each view are leveraged to benefit localization. The sample diversity is pursued by harnessing coarse-to-fine priors at both image and instance levels. With these priors, more emphasis would go to the reliable samples and the contributions of the unreliable ones would be decreased, such that the intrinsic characteristics of each sample can be exploited to make the model more robust during network learning. PML-LocNet can be easily combined with existing WSOL models to further improve the localization accuracy. Its effectiveness has been proved experimentally. Notably, it achieves 69.3% CorLoc and 50.4% mAP on PASCAL VOC 2007, surpassing the state-of-the-arts by a large margin.

This article addresses neural network based post-processing for the state-of-the-art video coding standard, High Efficiency Video Coding (HEVC). We first propose a partition-aware convolution neural network (CNN) that utilizes the partition information produced by the encoder to assist in the post-processing. In contrast to existing CNN-based approaches, which only take the decoded frame as input, the proposed approach considers the coding unit (CU) size information and combines it with the distorted decoded frame such that the artifacts introduced by HEVC are efficiently reduced. We further introduce an adaptive-switching neural network (ASN) that consists of multiple independent CNNs to adaptively handle the variations in content and distortion within compressed-video frames, providing further reduction in visual artifacts. Additionally, an iterative training procedure is proposed to train these independent CNNs attentively on different local patch-wise classes. Experiments on benchmark sequences demonstrate the effectiveness of our partition-aware and adaptive-switching neural networks.

Network quantization offers an effective solution to deep neural network compression for practical usage. Existing network quantization methods cannot theoretically guarantee the convergence. This paper proposes a novel iterative framework for network quantization with arbitrary bit-widths. We present two Lipschitz constraint based quantization strategies, namely width-level network quantization (WLQ) and multi-level network quantization (MLQ), for high-bit and extremely low-bit (ternary) quantization, respectively. In WLQ, Lipschitz based partition is developed to divide parameters in each layer into two groups: one for quantization and the other for re-training to eliminate the quantization loss. WLQ is further extended to MLQ by introducing layer partition to suppress the quantization loss for extremely low bit-widths. The Lipschitz based partition is proven to guarantee the convergence of the quantized networks. Moreover, the proposed framework is complementary to network compression methods such as activation quantization, pruning and efficient network architectures. The proposed framework is evaluated over extensive state-of-the-art deep neural networks, i.e., AlexNet, VGG-16, GoogleNet and ResNet18. Experimental results show that the proposed framework improves the performance of tasks like classification, object detection and semantic segmentation.

Filter bank based convolutional networks (FBCNs) enable efficient separable multiscale and multidirectional decomposition with a convolutional cascade of 1-D radial and directional filter banks. In this paper, we propose a two-stage subband coding framework for FBCN analysis coefficients using a SPIHT-like algorithm and subsequent primitive-based adaptive arithmetic coding (AAC). The SPIHT-like algorithm extends spatial orientation tree to exploit inter-subband dependency between subbands of different scales and directions. Mutual information is estimated for information-theoretical measurement to formulate such dependencies. Various primitives are designed adaptively encode the generated bitstream by fitting its varying lists and passes. Neural networks are leveraged to improve probability estimation for AAC, where nonlinear prediction is made based on contexts regarding scales, directions, locations and significance of analysis coefficients. Experimental results show that the proposed framework improves the lossy coding performance for FBCN analysis coefficients in comparison to the state-of-the-arts subband coding schemes SPIHT.

In this paper, we propose a Dynamic Network Quantization (DNQ) framework. Unlike most existing quantization methods that use a universal quantization bit-width for the whole network, we utilize policy gradient [1] to train an agent to learn the bit-width of each layer by the bit-width controller.

Many communication tasks have been formulated as optimization problems that can be solved by iterative algorithms. However, these algorithms are usually computationally intensive. To enable real-time processing of communication algorithms, in this paper, we propose a new deep neural network (DNN) architecture for algorithm approximation. Based on the idea of deep unfolding, we expand the iterative algorithm into a cascade network structure of basic blocks, with each network block corresponding to an iteration. Then, instead of directly approximating each iteration through network layers, we introduce the multivariate polynomial regression (MPR) as a bridge between the iterations and layers for constructing the unfolded network blocks. Specifically, we use a multivariate polynomial to approximate an iteration of the original iterative algorithm, and further construct a shallow neural network with one hidden layer to realize the polynomial, where the order of the polynomial provides theoretical guidance for determining the size of the network and controls the tradeoff between the approximation accuracy and computation speed. As empirical justifications, we apply the proposed DNN architecture in approximating the classic WMMSE algorithm for wireless interference management, showing that the proposed approach can achieve a good approximation accuracy with a much faster computation speed.

In this paper, we propose a deep reinforcement learning (DRL)-based rate adaptation algorithm for adaptive 360-degree video streaming, which is able to maximize the quality of experience of viewers by adapting the transmitted video quality to the time-varying network conditions. Specifically, to reduce the possible switching latency of the field of view (FoV), we design a new QoE metric by introducing a penalty term for the large buffer occupancy. A scalable FoV method is further proposed to alleviate the combinatorial explosion of the action space in the DRL formulation. Then, we model the rate adaptation logic as a Markov decision process and employ the DRL-based algorithm to dynamically learn the optimal video transmission rate. Simulation results show the superior performance of the proposed algorithm compared to the existing algorithms.

Object attention maps generated by image classifiers are usually used as priors for weakly-supervised segmentation approaches. However, normal image classifiers produce attention only at the most discriminative object parts, which limits the performance of weakly-supervised segmentation task. Therefore, how to effectively identify entire object regions in a weakly-supervised manner has always been a challenging and meaningful problem. We observe that the attention maps produced by a classification network continuously focus on different object parts during training. In order to accumulate the discovered different object parts, we propose an online attention accumulation (OAA) strategy which maintains a cumulative attention map for each target category in each training image so that the integral object regions can be gradually promoted as the training goes. These cumulative attention maps, in turn, serve as the pixel-level supervision, which can further assist the network in discovering more integral object regions. Our method (OAA) can be plugged into any classification network and progressively accumulate the discriminative regions into integral objects as the training process goes. Despite its simplicity, when applying the resulting attention maps to the weakly-supervised semantic segmentation task, our approach improves the existing state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark, achieving a mIoU score of 66.4% on the test set. Code is available at https://mmcheng.net/oaa/.

The head-mounted display (HMD) for 360-degree videos cannot be shared by multiple users, which significantly increases the bandwidth consumption when multiple users request the same video content simultaneously. This paper proposes a server-side hybrid multicast-unicast strategy for multi-user adaptive 360-degree video streaming, aiming to ensure the overall quality of experience (QoE) of the users in a bandwidth-constrained environment. A framework is established to realise this strategy, through which the delivery mode (multicast or unicast) and the corresponding bitrate for each tile can be jointly adapted to the dynamics of users' throughputs and field of views (FoVs). To maximize the overall QoE, a user clustering method is proposed to classify the users into several multicast clusters. Based on these clusters, we then formulate the joint delivery mode selection and rate adaptation problem as a non-linear integer programming problem which can be solved using the steepest ascent gradient algorithm. Finally, experiments are carried out to verify the performance of the proposed strategy.

Reinforcement Learning (RL) based methods optimize evaluation metric directly in image captioning task. In these methods, metric scores of captions are regard as rewards for examples. However, existing methods suffer from inferior performances on hard examples. In this paper, we propose an adaptive hard example mining method with additional supervised training for image captioning. Beam search algorithm is leveraged to estimate score expectation for each example. Examples whose caption scores are lower than expectation are selected automatically. For the selected hard examples, we propose an additional reward policy for high-scoring captions to force model learning from them. The proposed method is hyper-parameter free without tuning. Experimental results on MSCOCO dataset validate effectiveness of the proposed method.

In this paper, we propose a deformable 3D shape descriptor learning approach that takes into account the spatial correlations among local shape descriptors. By constructing a weighted graph that connects salient points on a shape surface, local feature descriptors on salient points are considered as signals defined on graph vertices, incorporating local surface information into the global graph structure. We then learn a graph structure-aware dictionary for each category of shapes with the multi-graph dictionary learning strategy, capturing similar spectral properties among graph signals generated from different shapes of the same category. Experiments conducted on representative 3D shape benchmark datasets demonstrate that our method improves over the state-of-the-art.

Sending only the viewport of interest provides a solution for 360-degree video streaming under the current bandwidth-constrained infrastructure. To this end, the user viewport requires to be prefetched in advance by conducting viewport prediction. To more accurately capture the nonlinear and long-term dependent relation between the future and past viewports, we develop a single viewport prediction model using convolutional neural network (CNN), in which the pooling layers are dropped and more convolutional layers are added for stronger nonlinear fitting ability. Further, we design a viewport trajectory prediction model based on recurrent neural network (RNN) which learns long-term dependency in sequential viewports. Specially, it is capable to estimate future viewport trajectory and support variable-size prediction window with low complexity. Finally, a correlation filter-based viewport tracker (CFVT) is proposed to perform content-aware viewport prediction. The combination of the RNN and the CFVT through a fusion model enables them to complement each other which is validated by significant improvement in prediction accuracy.

Quantization is known as an effective compression method for deploying neural networks on mobile devices. However, most existing works train from scratch a quantized network with universal bitwidth for all layers, making it hard to find the optimal trade-off between compression ratio and inference accuracy. In this paper, we propose a novel post-training quantization approach which derives a flexible bitwidth scheme. Our algorithm progressively downgrades bitwidth of chosen layer in the network and performs feature map alignment with pre-trained model. The algorithm comprises a meter of layer sensitivity and an iterative quantizer. Specifically, the meter dynamically estimates for every layer the error of quantization on its output feature map, meanwhile the error serves as an objective function to be minimized by the quantizer. Extensive experiments on CIFAR-10 and ImageNet ILSVRC2012 datasets demonstrate that the proposed approach achieves impressive results for mainstream neural networks.

The high-definition and panorama characteristics of 360-degree video make its transmission in existing networks more challenging. To alleviate this problem, this paper leverages transcoding-enabled edge caching in tile-granularity and makes joint caching, transcoding and delivery decisions among multiple edge servers for adaptive 360-degree video streaming. We formulate the joint caching, transcoding and delivery decision-making problem as a multivariate nonlinear integer programming problem, aiming at minimizing the aggregate network operational cost and balancing the server load, subject to storage and computation capacity constraints of the edge servers. Through decomposition and relaxation approach, the original intractable optimization problem is solved by an effective iterative algorithm. Finally, simulation results show that the proposed iterative algorithm outperforms the existing schemes in terms of resource utilization, cost reduction and server coordination.

In this paper, we propose a cross-layer decision framework for multiuser adaptive video delivery over time-varying and mutually interfering wireless cellular network. The key idea is to synthetically design the physical-layer optimization-based beamforming scheme (performed at the base stations) and the application-layer deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals), so that a very complex multi-user overall fair long-term quality of experience (QoE) maximization problem can be decomposed to two layers and solved effectively. Extensive simulations show that the proposed cross-layer design is effective and promising.

Object segmentation from 3D point clouds is an important topic in real-world applications. However, due to the sparsity and irregularity of point clouds, instance segmentation often suffers unsatisfying performance in various scenes such as autonomous driving. There are two main difficulties. First, in a wild scene, background noise often occupies a majority of the entire point set. Second, small-scale objects are often difficult to be recognized due to larger uncertainty. In this paper, we propose Det2Seg, a two-stage approach which alleviates the above issues towards more accurate instance segmentation of road-objects. In the first stage, we adopt Pointpillars [1] to detect the regions-of- interest that can localize and classify objects in a coarse level; in the second stage, we extract points from the detected regions into pillars, encode them into a new data format and feed it into a 2D convolutional neural network to perform fine-grained, domain-specific instance segmentation. We evaluate our approach on raw LiDAR (Light Detection And Ranging) data from the KITTI dataset [2]. The experimental results show that our approach largely outperforms the prior researches. In particular, our approach stands out for its significant ability on recognizing and segmenting small-scale objects, i.e., an improvement of over 20% in terms of Intersection over Union (IoU), beyond state-of- the-arts, is obtained for the cyclist class.

Caching can effectively smooth the temporal traffic variability and decrease the redundant data transmission in mobile video delivery. In this paper, we consider a video caching system consisting of a video provider (VP), a mobile network operator (MNO) with a set of cache-enabled base stations (BSs), and multiple mobile users. The VP leases some popular videos to the MNO, while the MNO places these rented videos in local caches of its BSs to save expensive backhaul transmission cost. However, in such a two-sided market, these two entities are competing with each other for their own profit due to their opposite expectation on the video pricing. To address this, we model the competition between the two entities using the framework of Stackelberg games and propose a joint video pricing and cache placement strategy by considering the heterogeneity of video file sizes and exploiting the classic law of demand from the field of economics. The proposed optimization problem is able to jointly maximize the profit of the VP and the MNO by the optimal selection of the video pricing and the cache placement strategy given that price, for both noncooperative BS caching and cooperative BS caching cases. We then develop iterative algorithms based on dynamic programming and gradient ascent, respectively, for these two cases to find the Stackelberg equilibrium (SE). The simulation results further show that the proposed joint optimization formulation follows the law of demand in economics, and the proposed algorithms for both cases can efficiently converge to the SE point that jointly maximizes the profit for both the VP and the MNO.

Due to the popularity of 3D videos, 2D-to-3D video conversion has become a hot research topic for the past few years. The most critical issue in 3D video synthesis is the estimation of depth maps for the video frames. Numerous efforts have been devoted in fully automatic and semi-automatic depth estimation approaches, although the discontinuity of depth field and the ambiguity of motion boundary are still the main challenges in depth estimation. This paper proposes a semi-automatic structure-aware sparse-to-dense depth estimation method, which leverages the tensor voting at two different levels to propagate depth across frames. In the first level, a 4D tensor voting is performed to remove outliers caused by inaccurate motion estimation. Noticing that the 4D tensors of correctly matched points should lie on the smooth layer in the manifold, we utilize the variety saliency defined by the eigen-system of the tensor for outlier removal. In the second level, a high-dimensional tensor voting algorithm, incorporating spatial location, motion, and color into the tensor representation, is devised to propagate the depth from the sparse points to the entire image domain. By projecting the input feature into the tangent space, the relation between the location, motion, color, and the depth can be established by voting process. Extensive experiments on public data set validate the effectiveness of the proposed method in comparison with state-of-the-art depth estimation approaches.

This paper aims at learning discriminative part detectors with only image-level labels. To this end, we need to develop effective technologies for both pattern mining and detection learning. Different from previous methods, which train part detectors in one step, we divide the detector learning process into two stages and formulate it as a weak to strong learning framework. In particular, we first learn exemplar detectors from the unaligned patterns and perform a detector-based spectral clustering to produce weak detectors that are only responsible for a few discriminative patterns. In this way, the weak detectors are able to offer right initial patterns for strong detector learning. Second, we learn strong detectors with patterns discovered from the weak detectors, which we formulate as a confidence-loss sparse multiple instance learning (cls-MIL) task. The cls-MIL considers the diversity of positive samples while avoiding drifting away from the well localized ones by assigning a confidence value to each positive sample. The responses of the learned detectors produce an effective mid-level image representation for both image classification and object localization. Experiments conducted on benchmark data sets well demonstrate the superiority of our method over existing approaches.

The cognitive radio technique allows secondary users (SUs) to share the spectrum with primary users (PUs) in an exclusive or opportunistic manner. This paper studies spectrum pricing conducted by spectrum owners, that is, primary operators (POs), and SU decision-making strategies for three kinds of duopoly markets. The single-band exclusive use market considers two POs with each providing a single band dedicated to SUs. A pre-emptive resume priority (PRP) M/M/1 queueing model is presented, based on which SUs decide to join which PO and which queue. We prove the existence of a unique Wardrop equilibrium for the decision-making process, and a unique Nash equilibrium for the proposed parallel pricing strategy. In a single-band mixed use market, the competition of two POs is represented by a Stackelberg game. We formulate the spectrum sharing among PU and SUs with a 3-level PRP M/M/1 queueing structure, and derive the close form expressions of SUs' queueing delay. In a multiband exclusive use market, where POs have to determine how many bands they will rent as well as the admission price, we define the problem as a mixed integer linear programming problem and propose a global particle swarm optimization algorithm to find the global optimum. Finally, we study a generalized scenario with multiple POs and multiple priority queues.

Dimension reduction is widely regarded as an effective way for decreasing the computation, storage and communication loads of data-driven intelligent systems, leading to a growing demand for statistical methods that allow analysis (e.g., clustering) of compressed data. We therefore study in this paper a novel problem called compressive robust subspace clustering, which is to perform robust subspace clustering with the compressed data, and which is generated by projecting the original high-dimensional data onto a lower-dimensional subspace chosen at random. Given only the compressed data and sensing matrix, the proposed method, row space pursuit (RSP), recovers the authentic row space that gives correct clustering results under certain conditions. Extensive experiments show that RSP is distinctly better than the competing methods, in terms of both clustering accuracy and computational efficiency.

Action recognition is an important yet challenging task in computer vision. In this paper, we propose a novel deep-based framework for action recognition, which improves the recognition accuracy by: 1) deriving more precise features for representing actions, and 2) reducing the asynchrony between different information streams. We first introduce a coarse-to-fine network which extracts shared deep features at different action class granularities and progressively integrates them to obtain a more accurate feature representation for input actions. We further introduce an asynchronous fusion network. It fuses information from different streams by asynchronously integrating stream-wise features at different time points, hence better leveraging the complementary information in different streams. Experimental results on action recognition benchmarks demonstrate that our approach achieves the state-of-the-art performance.

Network quantization is an effective solution to compress deep neural networks for practical usage. Existing network quantization methods cannot sufficiently exploit the depth information to generate low-bit compressed network. In this paper, we propose two novel network quantization approaches, single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ) for extremely low-bit quantization (ternary). We are the first to consider the network quantization from both width and depth level. In the width level, parameters are divided into two parts: one for quantization and the other for re-training to eliminate the quantization loss. SLQ leverages the distribution of the parameters to improve the width level. In the depth level, we introduce incremental layer compensation to quantize layers iteratively which decreases the quantization loss in each iteration. The proposed approaches are validated with extensive experiments based on the state-of-the-art neural networks including AlexNet, VGG-16, GoogleNet and ResNet-18. Both SLQ and MLQ achieve impressive results.

This paper addresses weakly supervised object detection with only image-level supervision at training stage. Previous approaches train detection models with entire images all at once, making the models prone to being trapped in sub-optimums due to the introduced false positive examples. Unlike them, we propose a zigzag learning strategy to simultaneously discover reliable object instances and prevent the model from overfitting initial seeds. Towards this goal, we first develop a criterion named mean Energy Accumulation Scores (mEAS) to automatically measure and rank localization difficulty of an image containing the target object, and accordingly learn the detector progressively by feeding examples with increasing difficulty. In this way, the model can be well prepared by training on easy examples for learning from more difficult ones and thus gain a stronger detection ability more efficiently. Furthermore, we introduce a novel masking regularization strategy over the high level convolutional feature maps to avoid overfitting initial samples. These two modules formulate a zigzag learning process, where progressive learning endeavors to discover reliable object instances, and masking regularization increases the difficulty of finding object instances properly. We achieve 47.6% mAP on PASCAL VOC 2007, surpassing the state-of-the-arts by a large margin.

How to balance the tradeoff between the user experience and bandwidth utilization emerges a critical challenge for multi-user 360-degree video adaptive streaming. This paper studies the server-side rate adaptation strategy for multiple users which are competing for the server bandwidth capacity. A tile visibility probability model is established, by which the tiles are classified into predicted, marginal and invisible types. A fine-grained rate adaptation problem is formulated as a nonlinear integer programming (NIP) problem, which aims at maximizing the video quality and navigation smoothness for multiple users. Thereafter, a steepest ascent algorithm with feasible starting point is developed to solve the proposed NIP problem in polynomial time. Finally, simulation results verify the performance of the proposed rate adaptation strategy.

A number of models have been proposed to capture rich semantic representation in Visual Question Answering (VQA). In this paper, we illustrate the compositionality of general cognitive ability in VQA and take the linguistic structure of language into consideration in semantic representation. We decompose the question into several components by the semantic tree and apply a tree-structured model to distill the sentence representation. In addition, we exploit the complementary image of the new dataset and optimize the classifier used to predict answers. We design a dual path network for the new VQA 2.0 dataset in training process to lead the model to effectively take advantage of the property of the dataset. Experiments show that our method could obtain more useful information and improve the performance.

Most state-of-the-art methods for representation learning are supervised, which require a large number of labeled data. This paper explores a novel unsupervised approach for learning visual representation. We introduce an image-wise discrimination criterion in addition to a pixel-wise reconstruction criterion to model both individual images and the difference between original images and reconstructed ones during neural network training. These criteria induce networks to focus on not only local features but also global high-level representations, so as to provide a competitive alternative to supervised representation learning methods, especially in the case of limited labeled data. We further introduce a competition mechanism to drive each component to increase its capability to win its adversary. In this way, the identity of representations and the likeness of reconstructed images to original ones are alternately improved. Experimental results on several tasks demonstrate the effectiveness of our approach.

This paper addresses an important yet less-studied problem: re-identifying groups of people in different camera views. Group re-identification (Re-ID) is very challenging since it is not only interfered by view-point and human pose variations in the traditional single-object Re-ID tasks, but also suffers from group layout and group member variations. To handle these issues, we propose to leverage the information of multi-grain objects: individual person and subgroups of two and three people inside a group image. We compute multi-grain representations to characterize the appearance and spatial features of multi-grain objects and evaluate the importance weight of each object for group Re-ID, so as to handle the interferences from group dynamics. We compute the optimal group-wise matching by using a multi-order matching process based on the multi-grain representation and importance weights. Furthermore, we dynamically update the importance weights according to the current matching results and then compute a new optimal group-wise matching. The two steps are iteratively conducted, yielding the final matching results.Experimental results on various datasets demonstrate the effectiveness of our approach.

Two-stream convolutional networks have shown excellent performance in video action recognition in recent years. However, it remains unclear how to model the correlation between the temporal and spatial streams more effectively. First, the spatial stream and temporal stream pay attention to different aspects, which can lead to different recognition results. Second, the variety in the length of optical flow fields tends to have a great impact on the classification results. In this paper, we propose a novel multi-scale spatiotemporal information fusion network to fuse the spatial and temporal features. Specifically, our network takes advantage of multi-scale temporal information to better utilize the motion cues. Considering the complementary relationship between the spatial and temporal features, we take the hierarchical fusion strategies and asynchronous fusion method to fuse the two-stream features. Experimental results on two benchmark datasets (UCF101 and HMDB51) show that the proposed network achieves competitive performance.

Given a set of labeled data with semantic descriptions, zero-shot learning aims at recognizing objects from unseen classes, where no instances of the classes are used during training. Most existing methods solve this problem via embedding images and labels into an embedding space and computing similarity across different information sources. However, the similarity calculation could be unreliable when the training and testing data distributions are inconsistent. In this paper, we propose a novel zero-shot learning model that forms a neighborhood-preserving structure in the semantic embedding space and utilize it to predict classifiers for unseen classes. By constructing a locally connected graph for class embeddings, we exploit the structural constraint of embeddings of similar classes and retain the global structure in the semantic embedding space to obtain an effective representation of semantic information. Experiment results on three benchmark datasets demonstrate that the proposed method generates effective semantic representations and out-performs state-of-the-art methods.

The nonlinear recovery is not promising in accuracy and speed, which limits the practical usage of compressed sensing (CS). This paper proposes a deep learning-based CS framework which leverages a deep convolutional auto-encoder for image sensing and recovery. The utilized auto-encoder architecture consists of three components: the fully convolutional network acts as an adaptive measurement matrix generator in the encoder; while in the decoder, the deconvolution network and refined reconstruction network are learned for intermediate and final recovery, respectively. Different from most previous work focusing on the block-wise manner to reduce implementation cost but result in blocky artifacts, our adaptive measurement matrix is applicable to any size of scene image and the decoder network reconstructs the whole image efficiently without any blocky artifacts. Moreover, dense connectivity is leveraged to combine multi-level features and alleviate the vanishing-gradient problem in the refined reconstruction network which boosts the performance on image recovery. Compared to the state-of-the-art methods, our algorithm improves more than 0.8 dB in average PSNR.

Dynamic adaptive streaming addresses user heterogeneity by providing multiple encoded representations at different rates and/or resolutions for the same video content. For delay-sensitive applications, such as live streaming, there is however a stringent requirement on the encoding delay, and usually the encoding power (or rate) budget is also limited by the computational (or storage) capacity of the server. It is therefore important, yet challenging, to optimally select the source coding parameters for each encoded representation in order to minimize the resource consumption while maintaining a high quality of experience for the users. To address this, we propose an optimization framework with an optimal representation selection problem for delay, power, and rate constrained adaptive video streaming. Then, by the optimal selection of source coding parameters for each selected representation, we maximize the overall expected user satisfaction, subject not only to the encoding rate constraint, but also to the delay and power constraints at the server. We formulate the proposed optimization problem as an integer linear program formulation to provide the performance upper bound, and as a submodular maximization problem with two knapsack constraints to develop a practically feasible algorithm. Simulation results show that the proposed weighted rate and power cost benefit greedy algorithm is able to achieve a near-optimal performance with very low time complexity. In addition, it can strike the best tradeoff both between the rate and power cost, and between the algorithm's performance and the delay requirements proposed by delay sensitive applications.

Caching at mobile edge servers can smooth temporal traffic variability and reduce the service load of base stations in mobile video delivery. However, the assignment of multiple video representations to distributed servers is still a challenging question in the context of adaptive streaming, since any two representations from different videos or even from the same video will compete for the limited caching storage. Therefore, it is important, yet challenging, to optimally select the cached representations for each edge server in order to effectively reduce the service load of base station while maintaining a high quality of experience (QoE) for users. To address this, we study a QoE-driven mobile edge caching placement optimization problem for dynamic adaptive video streaming that properly takes into account the different rate-distortion (R-D) characteristics of videos and the coordination among distributed edge servers. Then, by the optimal caching placement of representations for multiple videos, we maximize the aggregate average video distortion reduction of all users while minimizing the additional cost of representation downloading from the base station, subject not only to the storage capacity constraints of the edge servers, but also to the transmission and initial startup delay constraints of the users. We formulate the proposed optimization problem as an integer linear program to provide the performance upper bound, and as a submodular maximization problem with a set of knapsack constraints to develop a practically feasible cost benefit greedy algorithm. The proposed algorithm has polynomial computational complexity and a theoretical lower bound on its performance. Simulation results further show that the proposed algorithm is able to achieve a near-optimal performance with very low time complexity. Therefore, the proposed optimization framework reveals the caching performance upper bound for general adaptive video streaming systems, while the proposed algorithm provides some design guidelines for the edge servers to select the cached representations in practice based on both the video popularity and content information.

In this paper we consider the time-varying characteristics of practical wireless networks and propose a joint dynamic rate allocation and transmission scheduling optimization scheme for scalable video multirate multicast based on opportunistic routing (OR) and network coding. With OR the decision of optimal routes for scalable video coding layered streaming is integrated into the joint optimization formulation. The network throughput is also increased by taking advantage of the broadcast nature of the wireless shared medium and by network coding operations in intermediate nodes. To maximize the overall video reception quality among all destinations the proposed scheme can jointly optimize the video reception rate the associated routes to different destinations and the time fraction scheduling of transmitter sets that are concurrently transmitting in the shared wireless medium. By using dual decomposition and primal-dual update approach we develop a cross-layer algorithm in a fully distributed manner. Simulation results demonstrate significant network multicast throughput improvement and adaptation to dynamic network changes relative to existing optimization schemes.

Demanding for efficient compression and storage of DNA sequences has been rising with the rapid growth of DNA sequencing technologies. Existing reference-based algorithms map all patterns to regions found in the reference sequence, which lead to redundancy of incomplete similarity. This paper proposes an efficient reference-based method for DNA sequence compression that integrates FM-index and complementary context models to improve compression performance. The proposed method introduces FM-index to represent the full-text matching for exact repeats between the target and reference sequences. For unmatched symbols, complementary context models are leveraged to make weighted estimation conditioned on variable-order contexts. Reversed reference index is used to guarantee the longest match of variable-length substrings. Experimental results show that the proposed method can achieve a 213-fold compression ratio when tested on the first Korean personal genome sequence data set.

Tensor-based compressive sensing (CS) can preserve the intrinsic multidimensional structures with a reduced computational complexity. However, its recovery performance is degraded by simple sparsity. This paper proposes an efficient recovery algorithm GT-ADMM to solve a nonconvex optimization problem for tensor-based CS with group sparsity. Group-sparse representations of tensors is derived based on the Kronecker product of adaptively trained basis. The proposed algorithm incorporates alternating direction method of multipliers (ADMM) to obtain desirable recovery quality with improved efficiency. It requires few degrees of freedom to achieve tensor recovery with a guarantee of minimized approximation error. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art tensor-based methods in compressive video sampling.

This paper proposes an extended conjugate polar Fourier transform (ECPFT), to design iterated radial filter bank (RFB) and directional filter bank (DFB) convenient for accurate multiscale and multidirectional decomposition in discretization over a convolution network. With conjugated symmetric form, ECPFT would convert complex directional wavelets in original spatial domain to real ones in the inverse Fourier domain of ECPFT. Furthermore, it can contribute to changing the design of nonseparable RFB and DFB to decomposition in scales and directions with 1-D filter bank in each dimension separably in the ECPFT domain. The generated properties of conjugate symmetry and periodicity of 2π in both angular and radial dimensions, also guarantee convolution and sampling operations in both dimensions within the inverse Fourier domain of ECPFT. The fast algorithm for discretization is developed to reduce complexity.

Conventional image classification and object detection methods depend on manual annotations, such as image-level labels and bounding boxes. However, the acquisition of such annotations for millions of images is trivial. This paper addresses the problem of webly-supervised visual concept learning, and develops an automatic algorithm using parallel text and visual corpora to discover informative visual patterns from the web images. Based on the mined patterns, a cardinality-guided multiple instance learning algorithm is designed to establish the link between the image patterns and the literal concepts. Furthermore, due to the diversity of visual concepts, we perform clustered multitask refinement on the learned concept classifiers to enhance their generalization capability via a clustered regularization. Experiments demonstrate the superiority of the proposed method over traditional approaches.

In this paper, we propose a multiscale dictionary learning framework for hierarchical sparse representation of natural images. The proposed framework leverages an adaptive quadtree decomposition to represent structured sparsity in different scales. In dictionary learning, a tree-structured regularized optimization is formulated to distinguish and represent high-frequency details based on varying local statistics and group low-frequency components for local smoothness and structural consistency. In comparison to traditional proximal gradient method, block-coordinate descent is adopted to improve the efficiency of dictionary learning with a guarantee of recovery performance. The proposed framework enables hierarchical sparse representation by naturally organizing the trained dictionary atoms in a prespecified arborescent structure with descending scales from root to leaves. Consequently, the approximation of high-frequency details can be improved with progressive refinement from coarser to finer scales. Employed into image denoising, the proposed framework is demonstrated to be competitive with the state-of-the-art methods in terms of objective and visual restoration quality.

Real-time video services are usually delay sensitive and have strict constraints on the transmission reliability, which poses challenges to live video streaming over multi-hop wireless networks, since the unpredictable packet losses and network congestions caused by time-varying wireless channels greatly degrade the received video quality. To address this, in this paper, we propose a reinforcement learning (RL)-based opportunistic routing (OR) scheme for wireless video streaming with high-reliability and low-delay requirements. It can exploit the broadcast nature of the wireless shared medium and path diversity through OR to improve the transmission reliability, and find the low-delay paths between the source-destination pair dynamically for video packets through the RL module embedded in each relay node. Specifically, we design for the OR a new path-cost metric called the expected anypath delay (EAD), to estimate the end-to-end delay of a packet between the current relay node and the destination. The EAD is dynamically measured and updated over time, thereby reflecting the changes of link quality and the congestion level at the relay node. Moreover, we utilize the ACK message to piggyback the EAD of each relay node to its previous-hop node. Based on the local communication of the EADs from the neighbors, each node in the network can iteratively and independently run the RL module to update its own EAD value. Then, the next-hop forwarder node on a low delay route can be determined by assigning higher relay priority to the candidate forwarder nodes with lower EADs in OR. Simulation results show that the proposed RLOR algorithm can achieve a proper tradeoff between the transmission reliability and latency, so as to support the low-delay transmission of wireless video streams with high received video quality.

Perception is the interaction interface between an intelligent system and the real world. Without sophisticated and flexible perceptual capabilities, it is impossible to create advanced artificial intelligence (AI) systems. For the next-generation AI, called ‘AI 2.0’, one of the most significant features will be that AI is empowered with intelligent perceptual capabilities, which can simulate human brain’s mechanisms and are likely to surpass human brain in terms of performance. In this paper, we briefly review the state-of-the-art advances across different areas of perception, including visual perception, auditory perception, speech perception, and perceptual information processing and learning engines. On this basis, we envision several R&D trends in intelligent perception for the forthcoming era of AI 2.0, including: (1) human-like and transhuman active vision; (2) auditory perception and computation in an actual auditory setting; (3) speech perception and computation in a natural interaction setting; (4) autonomous learning of perceptual information; (5) large-scale perceptual information processing and learning platforms; and (6) urban omnidirectional intelligent perception and reasoning engines. We believe these research directions should be highlighted in the future plans for AI 2.0.

This paper presents a transductive multicomponent video segmentation algorithm, which is capable of segmenting the predefined object of interest in the frames of a video sequence. To ensure temporal consistency, a temporal coherent parametric min-cut algorithm is developed to generate segmentation hypotheses based on visual cues and motion cues. Furthermore, each hypothesis is evaluated by an energy function from foreground resemblance, foreground/background divergence, boundary strength, and visual saliency. In particular, the state-of-the-art R-convolutional neural network descriptor is leveraged to encode the visual appearance of the foreground object. Finally, the optimal segmentation of the frame can be attained by assembling the segmentation hypotheses through the Monte Carlo approximation. In particular, multiple foreground components are built to capture the variances of the foreground object in shapes and poses. To group the frames into different components, a tree-structured graphical model named temporal tree is designed, where visually similar and temporally coherent frames are arranged in branches. The temporal tree can be constructed by iteratively adding frames to the active nodes by probabilistic clustering. In addition, each component, consisting of frames in the same branch, is characterized by a support vector machine classifier, which is learned in a transductive fashion by jointly maximizing the margin over the labeled frames and the unlabeled frames. As the frames from the same video sequence follow the same distribution, the transductive classifiers achieve stronger generalization capability than inductive ones. Experimental results on the public benchmarks demonstrate the effectiveness of the proposed method in comparison with other state-of-the-art supervised and unsupervised video segmentation methods.

Dictionary learning has emerged as a promising alternative to the conventional hybrid coding framework. However, the rigid structure of sequential training and prediction degrades its performance in scalable video coding. This paper proposes a progressive dictionary learning framework with hierarchical predictive structure for scalable video coding, especially in low bitrate region. For pyramidal layers, sparse representation based on spatio-temporal dictionary is adopted to improve the coding efficiency of enhancement layers with a guarantee of reconstruction performance. The overcomplete dictionary is trained to adaptively capture local structures along motion trajectories as well as exploit the correlations between the neighboring layers of resolutions. Furthermore, progressive dictionary learning is developed to enable the scalability in temporal domain and restrict the error propagation in a closed-loop predictor. Under the hierarchical predictive structure, online learning is leveraged to guarantee the training and prediction performance with an improved convergence rate. To accommodate with the state-of-the-art scalable extension of H.264/AVC and latest High Efficiency Video Coding (HEVC), standardized codec cores are utilized to encode the base and enhancement layers. Experimental results show that the proposed method outperforms the latest scalable extension of HEVC and HEVC simulcast over extensive test sequences with various resolutions.

Recent methods for object co-segmentation focus on discovering single co-occurring relation of candidate regions representing the foreground of multiple images. However, region extraction based only on low and middle level information often occupies a large area of background without the help of semantic context. In addition, seeking single matching solution very likely leads to discover local parts of common objects. To cope with these deficiencies, we present a new object co-segmentation framework, which takes advantages of semantic information and globally explores multiple co-occurring matching cliques based on an N-partite graph structure. To this end, we first propose to incorporate candidate generation with semantic context. Based on the regions extracted from semantic segmentation of each image, we design a merging mechanism to hierarchically generate candidates with high semantic responses. Second, all candidates are taken into consideration to globally formulate multiple maximum weighted matching cliques, which complement the discovery of part of the common objects induced by a single clique. To facilitate the discovery of multiple matching cliques, an N-partite graph, which inherently excludes intralinks between candidates from the same image, is constructed to separate multiple cliques without additional constraints. Further, we augment the graph with an additional virtual node in each part to handle irrelevant matches when the similarity between the two candidates is too small. Finally, with the explored multiple cliques, we statistically compute pixel-wise co-occurrence map for each image. Experimental results on two benchmark data sets, i.e., iCoseg and MSRC data sets achieve desirable performance and demonstrate the effectiveness of our proposed framework.

It is a challenging task to recognize fine-grained subcategories due to the highly localized and subtle differences among them. Different from most previous methods that rely on object/part annotations, this paper proposes an automatic fine-grained recognition approach, which is free of any object/part annotation at both training and testing stages. The key idea includes two steps of picking neural activations computed from the convolutional neural networks, one for localization, and the other for description. The first picking step is to find distinctive neurons that are sensitive to specific patterns significantly and consistently. Based on these picked neurons, we initialize positive samples and formulate the localization as a regularized multiple instance learning task, which aims at refining the detectors via iteratively alternating between new positive sample mining and part model retraining. The second picking step is to pool deep neural activations via a spatially weighted combination of Fisher Vectors coding. We conditionally select activations to encode them into the final representation, which considers the importance of each activation. Integrating the above techniques produces a powerful framework, and experiments conducted on several extensive fine-grained benchmarks demonstrate the superiority of our proposed algorithm over the existing methods.

Recognizing fine-grained sub-categories such as birds and dogs is extremely challenging due to the highly localized and subtle differences in some specific parts. Most previous works rely on object / part level annotations to build part-based representation, which is demanding in practical applications. This paper proposes an automatic fine-grained recognition approach which is free of any object / part annotation at both training and testing stages. Our method explores a unified framework based on two steps of deep filter response picking. The first picking step is to find distinctive filters which respond to specific patterns significantly and consistently, and learn a set of part detectors via iteratively alternating between new positive sample mining and part model retraining. The second picking step is to pool deep filter responses via spatially weighted combination of Fisher Vectors. We conditionally pick deep filter responses to encode them into the final representation, which considers the importance of filter responses themselves. Integrating all these techniques produces a much more powerful framework, and experiments conducted on CUB-200-2011 and Stanford Dogs demonstrate the superiority of our proposed algorithm over the existing methods.

Conventional Compressive Sensing (CS) obscures the intrinsic structures of multidimensional signals with the vectorized representation. Although tensor-based CS methods can preserve the intrinsic multidimensional structures with reduced computational complexity, their sampling efficiency and recovery performance are degraded with the assumption of standard/simple sparsity. This paper proposes a general and adaptive model that incorporates structured sparsity into tensor representation to fit the varying nonstationary statistics of multidimensional signals. To guarantee the block sparsity, subspace clustering is adopted to adaptively generate the union of tensor subspaces with its basis of each tensor subspace learned for optimized representation. For sampled tensors, the stable recovery algorithm is developed to achieve desirable recovery performance using fewer degrees of freedom. Moreover, the proposed model inherits the merit from tensor-based CS to alleviate the computational and storage burden in sampling and recovery. Experimental results demonstrate that the proposed model can achieve better recovery performance in video sampling in comparison to the state-of-the-art tensor-based method.

Recently, 2D-to-3D video conversion has raised great interest because of the increasing demand of 3D video contents. A critical component in semi-automatic methods is to estimate the depth information of non-key frames from key frames. To overcome the sensitiveness to occlusion in existing schemes, a depth propagation method is proposed in this paper where tensor voting is leveraged to eliminate occlusion points, thereby derive accurate sparse depth. Furthermore, high dimensional tensor is constructed with coordinate, motion and color features to represent points in the sparse depth. A dense depth can be generated via tensor-voting based interpolation. It is robust against object occlusion and inaccurate motion estimation because tensor voting can efficiently make denoising and capture the structures. Experiment results demonstrate that the proposed method outperforms state-of-the-art semi-automatic techniques.

Efficient image representation methods are of great significance in image processing. This paper proposes an invertible deep convolutional network where the entire architecture is constructed by a decomposition process and a frequency recombination process. The decomposition allows frequency division in both the lower and higher part at the same time on each layer. It is implemented by using iterated directional filter banks. Perfect reconstruction is available if we adopt biorthogonal qincunx filter banks. The remaining frequency recombination process is adjoined to the architecture to transform the uniform frequency partition to nonuniform one and thereby increase the efficiency and flexibility. Numerical experiments reveal that the underlying network has good performance especially for images with large amount of fine-grained information.

This paper proposes a hybrid wavelet convolution network (HWCN) which is composed of a scattering convolution component and a convolution neural component. The hierarchical end-to-end network implements sparse-coding and high-dimensional reconstruction for inverse problem through cascade convolutions. With the pre-defined scattering convolutions from nonlinear operators, the network can be tailored in accordance with the frequency property to provide sparse code candidates, and the convolution neural component could automatically select and weight these candidates for sparse coding. Given a tiny dataset, HWCN could train complex deep network with better generalization by regularization from scattering convolutions, and thereby is a competitive alternative to convolutional neural networks (CNN). Moreover, we further demonstrate that HWCN is a superior selection of sparse-coding based image super-resolution and achieves state-of-the-art performance.

To enable robust multiview video streaming over current wireless broadcast infrastructures, this paper proposes a joint source and channel coding optimization framework for video streaming based upon multiview video coding technique. It utilizes random linear coding (RLC) with ladder-shaped global coding coefficient matrix (GCM) for unequal erasure protection. An accurate expression of the decoding probability for RLC with ladder-shaped GCM is derived. A fast two-layer local search algorithm is developed to solve the source and channel coding rate allocation problem, where the source and channel rate of each layer can be adjusted independently for the overall video quality. Simulation results demonstrate significant video quality improvement to existing rate allocation schemes.

This paper proposes a scene depth map generation method based on lens-based light field cameras. In particular, it achieves the functions of the incident rays and the corresponding directional features, which can favor determining the coordinates of candidate space points. The light rays behind the aperture in the 4D light field would be converted into the rays before the aperture with their directions known. The light rays through the aperture center are denoted as reference light rays and keep their directions. With the probability (cost) of each reference light ray in each depth value, we obtain an initial depth map by selecting the depth value with minimum cost. It would be refined via multi-label optimization and weighted median filtering. Experimental results demonstrate the accuracy of the depth map estimated by the proposed method.

This paper addresses the problem of automatic scattering feature selection for signal classification. While features derived from group invariant scattering networks are quite effective for signal classification. We argue that scattering networks are not always the appropriate choice as they are not learned for the objective at hand. In this paper, we explore jointly learning a deep scattering convolution network with a support vector machine by casting the problem as a multiple kernel learning problem. The convolution paths of the network are kernelized respectively to be selected in a large-margin context. We deduce scattering paths from the corresponding kernels after solving the kernel learning problem. Experiments on several datasets demonstrate the effectiveness of the proposed method over state-of-the-art techniques.

Convolutional neural networks (CNN) have achieved prominent performance in facial landmark detection in recent years. However, the training of such deep network is non-trivial due to the over-fitting problem caused by the insufficient training data and the diminishing gradients problem occurred in the back-propagation. To address these problems, we propose a multi-task learning framework with supervised neural networks to jointly detect facial landmarks with a set of related tasks. On the one hand, to handle the over-fitting problem, the proposed method takes the advantage of additional task labels to train the model in a multi-task learning fashion to generate a shared feature representation for high-level recognition tasks. On the other hand, in order to tackle the transparency and diminishing gradients problem, the proposed method enforces supervision to the intermediate layers of the network, augmenting the gradient signal propagated from the final layer. Experiments on public benchmarks validate the effectiveness of the proposed method.

Caching at edge servers can smooth the temporal traffic variability and reduce the service load of base stations in wireless streaming. However, the assignment of the cached content for possibly multiple versions of different video sequences is still a challenging question in the context of adaptive streaming. In this paper, we propose a wireless video caching placement optimization problem for dynamic adaptive video streaming that properly takes into account the different rate-distortion (R-D) characteristics of the video sequences. Our objective is to minimize the expected video distortion by the optimal caching of compressed video sequences, such that the clients can effectively access video content storage and bandwidth constraints. We prove that our optimization problem is a submodular maximization problem subject to a knapsack constraint. A cost benefit greedy algorithm is developed to obtain an approximate solution with polynomial time complexity and theoretical approximation guarantees. Simulation results demonstrate significant video distortion reduction relative to different baseline caching placement schemes.

In this paper, we propose a representation selection optimization problem for complexity constrained adaptive video streaming that properly takes into account the different complexity-rate-distortion (C-R-D) characteristics of the videos when implementing rate control for desired representations. Our objective is to maximize the expected video distortion reduction of users, subject not only to encoding rate constraints, but also to complexity constraints. We prove that our optimization problem is a submodular maximization problem with two knapsack constraints. A weighted rate and complexity cost benefit greedy algorithm is then developed to obtain an approximate solution with polynomial time complexity and good approximation performance in simulations.

MOTIVATION: Genome-wide association studies (GWAS) have been widely used in discovering the association between genotypes and phenotypes. Human genome data contain valuable but highly sensitive information. Unprotected disclosure of such information might put individual's privacy at risk. It is important to protect human genome data. Exact logistic regression is a bias-reduction method based on a penalized likelihood to discover rare variants that are associated with disease susceptibility. We propose the HEALER framework to facilitate secure rare variants analysis with a small sample size. RESULTS: We target at the algorithm design aiming at reducing the computational and storage costs to learn a homomorphic exact logistic regression model (i.e. evaluate P-values of coefficients), where the circuit depth is proportional to the logarithmic scale of data size. We evaluate the algorithm performance using rare Kawasaki Disease datasets. AVAILABILITY AND IMPLEMENTATION: Download HEALER at http://research.ucsd-dbmi.org/HEALER/ CONTACT: [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

OBJECTIVE: To develop an accurate logistic regression (LR) algorithm to support federated data analysis of vertically partitioned distributed data sets. MATERIAL AND METHODS: We propose a novel technique that solves the binary LR problem by dual optimization to obtain a global solution for vertically partitioned data. We evaluated this new method, VERTIcal Grid lOgistic regression (VERTIGO), in artificial and real-world medical classification problems in terms of the area under the receiver operating characteristic curve, calibration, and computational complexity. We assumed that the institutions could "align" patient records (through patient identifiers or hashed "privacy-protecting" identifiers), and also that they both had access to the values for the dependent variable in the LR model (eg, that if the model predicts death, both institutions would have the same information about death). RESULTS: The solution derived by VERTIGO has the same estimated parameters as the solution derived by applying classical LR. The same is true for discrimination and calibration over both simulated and real data sets. In addition, the computational cost of VERTIGO is not prohibitive in practice. DISCUSSION: There is a technical challenge in scaling up federated LR for vertically partitioned data. When the number of patients m is large, our algorithm has to invert a large Hessian matrix. This is an expensive operation of time complexity O(m(3)) that may require large amounts of memory for storage and exchange of information. The algorithm may also not work well when the number of observations in each class is highly imbalanced. CONCLUSION: The proposed VERTIGO algorithm can generate accurate global models to support federated data analysis of vertically partitioned data.

Fine-grained visual categorization is an emerging research area and has been attracting growing attention recently. Due to the large inter-class similarity and intra-class variance, it is extremely challenging to recognize objects in fine-grained domains. A traditional spatial pyramid matching model could obtain desirable results for the basic-level category classification by weak alignment, but may easily fail in fine-grained domains, since the discriminative features are extremely localized. This paper proposes a new framework for fine-grained visual categorization. First, an efficient part localization method incorporates semantic prior into geometric alignment. It detects the less deformable parts, such as the head of birds with a template-based model, and localizes other highly deformable parts with simple geometric alignment. Second, we learn one-vs-all features, which are simple and transplantable. The learned mid-level features are dimension friendly and more robust to outlier instances. Furthermore, in view that some subcategories are too similar to tell them apart easily, we fuse the subcategories iteratively according to their similarities, and learn fused one-vs-all features. Experimental results show the superior performance of our algorithms over the existing methods.

Classical dictionary learning methods for video coding suffer from high computational complexity and interfered coding efficiency by disregarding its underlying distribution. This paper proposes a spatio-temporal online dictionary learning (STOL) algorithm to speed up the convergence rate of dictionary learning with a guarantee of approximation error. The proposed algorithm incorporates stochastic gradient descents to form a dictionary of pairs of 3D low-frequency and high-frequency spatio-temporal volumes. In each iteration of the learning process, it randomly selects one sample volume and updates the atoms of dictionary by minimizing the expected cost, rather than optimizes empirical cost over the complete training data, such as batch learning methods, e.g., K-SVD. Since the selected volumes are supposed to be independent identically distributed samples from the underlying distribution, decomposition coefficients attained from the trained dictionary are desirable for sparse representation. Theoretically, it is proved that the proposed STOL could achieve better approximation for sparse representation than K-SVD and maintain both structured sparsity and hierarchical sparsity. It is shown to outperform batch gradient descent methods (K-SVD) in the sense of convergence speed and computational complexity, and its upper bound for prediction error is asymptotically equal to the training error. With lower computational complexity, extensive experiments validate that the STOL-based coding scheme achieves performance improvements than H.264/AVC or High Efficiency Video Coding as well as existing super-resolution-based methods in rate-distortion performance and visual quality.

In the past few decades, we have witnessed the success of bag-of-features (BoF) models in scene classification, object detection, and image segmentation. Whereas it is also well acknowledged that the limitation of BoF-based methods lies in the low-level feature encoding and coarse feature pooling. This paper proposes a novel scene classification method, which leverages several semantic codebooks learned in a multitask fashion for robust feature encoding, and designs a context-aware image representation for efficient feature pooling. Apart from conventional universal codebook learning approaches, the proposed method encodes each class of local features with a unique semantic codebook, which captures the distinct distribution of different semantic classes more effectively. Instead of learning each semantic codebook separately, we learn a compact global codebook, of which each semantic codebook is a sparse subset, with a two-stage iterative multitask learning algorithm. While minimizing the clustering divergence, the semantic codeword assignment is solved by submodular optimization simultaneously. Built upon the global and semantic codebooks, a context-aware image representation is further developed to encode both global and semantic features in image representation via contextual quantization, semantic response computation, and semantic pooling. Extensive experiments have been conducted to validate the effectiveness of the proposed method on various public benchmarks with several popular local features.

The rapid growth of web images presents new challenges as well as opportunities to the task of image understanding. Conventional approaches rely heavily on fine-grained annotations, such as bounding boxes and semantic segmentations, which are not available for web-scale images. In general, images over the Internet are accompanied with descriptive texts, which are relevant to their contents. To bridge the gap between textual and visual analysis for image understanding, this paper presents an algorithm to learn the relations between scenes, objects, and texts with the help of image-level annotations. In particular, the relation between the texts and objects is modeled as the matching probability between the nouns and the object classes, which can be solved via a constrained bipartite matching problem. On the other hand, the relations between the scenes and objects/texts are modeled as the conditional distributions of their co-occurrence. Built upon the learned cross-domain relations, an integrated model brings together scenes, objects, and texts for joint image understanding, including scene classification, object classification and localization, and the prediction of object cardinalities. The proposed cross-domain learning algorithm and the integrated model elevate the performance of image understanding for web images in the context of textual descriptions. Experimental results show that the proposed algorithm significantly outperforms conventional methods in various computer vision tasks.

Summary form only given. This paper proposes a bi-directional context modeling (BCM) technique for reference-free genome sequence compression, which constructs its contexts by combining arbitrary predicted symbols in two directions corresponding to approximate repeats and non-repeat regions. Thus, BCM can sequentially predict DNA sequences with weighted conditional probabilities that simultaneously exploit the correlations among matched approximate repeats and fit the variable-order statistics in non-repeat regions. Moreover, BCM eliminates the overhead of pointer information for specifying approximate repeats, as it is synchronized in both encoder and decoder. In theory, we show that upper bounds of excess model redundancy led by BCM vanish with the growth of sequence size. Experimental results show that BCM outperforms the state-of-the-art reference-free compressors like FCM and CTW+LZ.

Existing sparse representation with subspace learning is hampered by the intersection of subspaces of bases. With structured sparsity to enable the prior knowledge of signal statistics, this paper proposes a novel compressive video sampling by subspace learning to minimize the intersection of subspaces. As the measurement, the block coherence is optimized with the regularized learning to generate a class of independent bases associated with the subspaces. Thus, the proposed framework can make a compact block sparse representation based on the derived basis in an efficient and adaptive manner. The block-based recovery of video sequences is demonstrated to be stable under the constraint of block restricted isometric property (RIP). Experimental results show that the proposed method outperforms existing compressive video sampling schemes.

This paper proposes an adaptive dictionary learning approach based on sub modular optimization. A candidate atom set is constructed based on multiple bases from the combination of analytic and trained dictionaries. With the low-frequency components by the analytic DCT atoms, high-resolution dictionaries can be inferred through online learning to make efficient approximation with rapid convergence. It is formulated as a combinatorial optimization for approximate sub modularity, which is suitable for sparse representation based on dictionaries with arbitrary structures. In single-image super-resolution, the proposed scheme has been demonstrated to improve the reconstruction performance in comparison with double sparsity dictionary in terms of both objective and subjective restoration quality.

To enable learning-based video coding for transmission over heterogenous networks, this paper proposes a scalable video coding framework by progressive dictionary learning. With the hierarchical B-picture prediction structure, the inter-predicted frames would be reconstructed in terms of the spatio-temporal dictionary in a successive sense. Within the progressive dictionary learning, the training set is enriched with the samples from the reconstructed frames in the coarse layer. Through minimizing the expected cost, the stochastic gradient descent is leveraged to update the dictionary for practical coding. It is demonstrated that the learning-based scalable framework can effectively guarantee the consistency of motion trajectory with the well-designed spatio-temporal dictionary.

This paper proposes a novel multi-component video co-segmentation approach to simultaneously separate the foreground from the background in the video frames. To capture the variance of appearance of the foreground object, a multi-component foreground model is developed. Each component of the model characterizes a specific viewpoint/pose/appearance of the foreground object. To learn the parameters of the multi-component model, a transductive learning algorithm is leveraged to “transfer” the information of the labeled frames to the unlabeled frames in a tree-structured model, namely, temporal tree. Each branch of the temporal tree consists of the exemplars of a foreground component, and a transductive support vector regressor is capable of being trained. Experiments show that the proposed method outperforms quite a few state-of-the-art video segmentation algorithms in public benchmark.

This paper proposes a multiview video scheduling scheme through asymmetric Nash bargaining for a tradeoff between user satisfaction and view fairness. The interactions of view streams queueing at the scheduler are modeled as a bargaining problem, where packets of different views share the transmission opportunities via negotiation. A utility function based upon the viewing angle is defined to indicate whether a view is desired by a user, then a network utility maximization problem is solved by the Nash bargaining solution (NBS). The impacts of the bargaining powers on both network performance and view fairness are analytically investigated. Simulation results are provided to compare the performances of three scheduling schemes including fair scheduling, symmetric bargaining, and proportional scheduling, which can be achieved by adjusting the bargaining powers.

2D to 3D video conversion has become a popular manner to produce 3D contents, in which the key task is to estimate the depth for the images. However, the main limitation of conventional motion-based depth propagation methods is that they are susceptible to inaccurate motion estimation and occlusion. To address this problem, we propose a two-stage structure-aware depth propagation method for semi-automatic 2D-to-3D conversion. In the first stage, an initial depth map is generated by shifted bilateral filtering over the temporally consistent region, which is determined by validating the forward and backward optical flow fields. In the second stage, the depth of the temporally inconsistent region will be estimated by solving a multi-label graph inference problem. In particular, an efficient priority belief propagation algorithm is developed, in which the priority of nodes to propagate messages depends on the structure saliency from tensor voting. Experimental results show that the proposed method outperforms existing depth estimation methods in public benchmark.

In addition to the rate-distortion (R-D) behavior, in a real-time wireless video communication system, the end-to-end delay would also significantly affect the overall video reception quality. To analyze, control, and optimize the R-D behavior under the end-to-end delay constraint, in this paper we extend the traditional R-D optimization (RDO) for the wireless video communication system and formulate a novel delay-RDO-based rate control problem, by investigating the allocation of end-to-end delay to different delay components. It aims at minimizing the average total end-to-end distortion under the transmission rate and end-to-end delay constraints, by a joint selection of both the source coding and the channel coding parameters. The wireless channel is represented by a finite-state Markov channel model characterizing the time-varying process and predicting the future channel condition. As applicable solutions, a practical algorithm based on the Lagrange multiplier approaches, Karush-Kuhn-Tucker conditions, and sequential quadratic programming methods is developed. The experimental results demonstrate the superiority of the proposed algorithm over the existing schemes.

LS-based adaptation cannot fully exploit high-dimensional correlations in image signals, as linear prediction model in the input space of supports is undesirable to capture higher order statistics. This paper proposes Gaussian process regression for prediction in lossless image coding. Incorporating kernel functions, the prediction support is projected into a high-dimensional feature space to fit the anisotropic and nonlinear image statistics. Instead of directly conditioned on the support, Gaussian process regression is leveraged to make prediction in the feature space. The model parameters are optimized by measuring the similarities based on the training set, which is evaluated by combined kernel function in the sense of translation and rotation invariance among supports mapped in the feature space. Experimental results show that the proposed predictor outperforms most benchmark predictors reported.

The standard Compressive Sensing (CS) theory indicates that robust signals recovery can be obtained from just a few collection of incoherent projections. To further decrease the necessary measurements, an alternative to the generic CS framework assumes that signals lie on a union of subspaces (UoS). However, UoS model is limited to the specific type of signal regularity. This paper considers a more general and adaptive model which presumes that signals lie on a union of data-driven subspaces (UoDS). The UoDS model inherits the merit from UoS that signals have structural sparse representation. Meanwhile, it allows to recover signals using fewer degrees of freedom for a desirable recovery quality than UoS. To construct the UoDS model, a subspace clustering method is utilized to form an adaptive group set. The corresponding adaptive basis is learned by applying a linear subspace learning (LSL) method to each group. A corresponding recovery algorithm with provable performance is also given. Experiment results demonstrate that the proposed model for video sampling is valid and applicable.

Summary form only given. The multi-shot Coded Aperture Snapshot Spectral Imaging system (CASSI) is an imaging architecture that senses the spectral imaging information of a 3D cube using a 2D focal plane array snapshot. Modeled as the summation of coded and shifted versions of different spectral voxels, the compressive CASSI measurements are difficult to be further compressed due to little correlation and redundancy. This paper is dedicated to retaining the correlations in the original data cube by an embedded transform coding on the compressive measurements. Through statistical modeling, the transformed measurements prove to be more approximate to the distribution of original spectral images than the compressive measurements. Meanwhile, bit-plane coding for the transformed measurements is applied by taking advantage of the known code aperture. The significant performance of this embedded coder on random compressive measurements is evaluated for different number of snapshots.

Summary form only given. This paper proposes a novel multiscale online dictionary learning algorithm with double sparsity structure for scalable video coding. Along hierarchical structures on the feature set by wavelet transform, the search space of online learning is optimized to sub-blocks for hierarchical sparsity. The group sparsity is exploited on lowest sub-band in the base layer to obtain the low-frequency sub-dictionary and sparse coefficient. We also designed cross-scale decomposition and reconstruction, for which the recovery error can be bounded. The dictionary is updated by stochastic gradient descent to optimize the expected cost. Hierarchical high-frequency information is predicted from a pre-learned corresponding sub-dictionary pairs for scalable coding. We demonstrated that the proposed algorithm can achieve scalable signal to noise ratio (SNR).

A depth image represents three-dimensional scene information and is commonly used for depth image based rendering (DIBR) to support 3-D video applications. Reconstruction errors around depth edges lead to image blurring in synthesized views. This paper proposes a new coding method to improve the depth edge preserving effect. It makes use of both edge regularity and grayscale regularity to model the depth map, where the edge regularity is modeled by multiscale beamlet representation and the grayscale regularity is modeled by edge adaptive wavelet transform.

Cosegmentation has achieved great success in exploiting inter-image segmentation consistency to segment a group of images simultaneously. To enforces non-local temporal coherence across all the frames by high-order object-level appearance/semantic correspondence with a compensation to the short-time window motion coherence cue, this paper cosegments the video frames together with a novel interframe segmentation consistency term. A direct application of existing cosegmentation algorithms to video frames encounters the following challenges: the high correlation of adjacent frames which makes the segmentation ambiguous and a large number of video frames which makes the computation expensive. To tackle them, we formulate the cosegmentation in a transductive learning framework to iteratively learn the inter-frame consistency term from all the video frames. The proposed algorithm is evaluated on the standard SegTrack dataset and promising results are obtained.

In this paper, we extend the traditional rate-distortion optimization (RDO) for the end-to-end wireless video communication system and develop a novel delay-rate-distortion optimization (dRDO) based rate control (RC) algorithm, by investigating the allocation of end-to-end delay to different delay components. Tradeoffs regarding, respectively, source coding delay versus buffering delay and available source coding rate versus redundant rate incurred by channel coding, are coupled in the proposed dRDO algorithm. It targets at minimizing the average total end-to-end distortion under the transmission rate and end-to-end delay constraints, by joint selection of search range and quantization step size in source coding and channel code rate in channel coding. Experimental results demonstrate the superiority of the proposed dRDO algorithm over existing schemes.

As an emerging research topic, fine-grained visual categorization has been attracting growing attentions in recent years. Due to the large inter-class similarity and intra-class variance, recognizing objects in fine-grained domains is extremely challenging, and sometimes even humans can not recognize them accurately. Traditional bag-of-words model could obtain desirable results for basic-level category classification by weak alignment using spatial pyramid matching model, but may easily fail in fine-grained domains since the discriminative features are not only subtle but also extremely localized. The fine differences often get swamped by those irrelevant features, and it is virtually impossible to distinguish them. To address the problems above, we propose a new framework for fine-grained visual categorization. We strengthen the spatial correspondence among parts by including foreground segmentation and part localization. Based on the part representations of the images, we learn a large set of mid-level features which are more suitable for fine-grained tasks. Comparing with the low level features directly extracted from the images, the learned one-vs-all mid-level features enjoy the following advantages. First, the dimension of the mid-level features is relatively small. In order to obtain high classification accuracy, the dimension of the low level features usually reaches several thousand to tens of thousand, and becomes even larger when introducing spatial pyramid model. However, the dimension of our mid-level features is related to the number of classes, which is far less. Second, each entry of the proposed mid-level features is meaningful, which forms a more compact representation of the image. Third, the mid-level features are more robust than the low level ones, which is helpful for classification. Fourth, the learning process of the mid-level features is independent and can be easily combined with other techniques to boost the performance. We evaluate the proposed approach on the extensive fine-grained dataset CUB 200-2011 and Stanford Dogs, by learning the mid-level features based on the popular Fisher vectors and convolutional neural network, we boost the classification accuracy by a considerable margin and advance the state-of-the-art performance in fine-grained visual categorization.

View synthesis is dedicated to generating arbitrary views of the same scene from given inputs. As an alternative to depth-image-based rendering (DIBR), image warping based view synthesis approaches could automatically generate visually plausible virtual views in real-time. Recognizing that existing techniques would lead to temporal incoherence and shape distortions in synthesized videos, this paper proposes a novel video warping algorithm which motion saliency map and global motion from reference views are incorporated into motion-aware constraints to maintain temporal coherence in virtual views. Furthermore, a salient curve based disparity constraint is imposed to prevent shape deformations and avoid possible artifacts. Extensive experiments are validated by visual comparison, which demonstrates that the proposed algorithm outperforms existing warping-based methods.

BACKGROUND AND OBJECTIVE: Short-read sequencing is becoming the standard of practice for the study of structural variants associated with disease. However, with the growth of sequence data largely surpassing reasonable storage capability, the biomedical community is challenged with the management, transfer, archiving, and storage of sequence data. METHODS: We developed Hierarchical mUlti-reference Genome cOmpression (HUGO), a novel compression algorithm for aligned reads in the sorted Sequence Alignment/Map (SAM) format. We first aligned short reads against a reference genome and stored exactly mapped reads for compression. For the inexact mapped or unmapped reads, we realigned them against different reference genomes using an adaptive scheme by gradually shortening the read length. Regarding the base quality value, we offer lossy and lossless compression mechanisms. The lossy compression mechanism for the base quality values uses k-means clustering, where a user can adjust the balance between decompression quality and compression rate. The lossless compression can be produced by setting k (the number of clusters) to the number of different quality values. RESULTS: The proposed method produced a compression ratio in the range 0.5-0.65, which corresponds to 35-50% storage savings based on experimental datasets. The proposed approach achieved 15% more storage savings over CRAM and comparable compression ratio with Samcomp (CRAM and Samcomp are two of the state-of-the-art genome compression algorithms). The software is freely available at https://sourceforge.net/projects/hierachicaldnac/with a General Public License (GPL) license. LIMITATION: Our method requires having different reference genomes and prolongs the execution time for additional alignments. CONCLUSIONS: The proposed multi-reference-based compression algorithm for aligned reads outperforms existing single-reference based algorithms.

Smart mobile phones are capable of performing video coding and streaming over wireless networks, but are often constrained by the end-to-end delay requirement and energy supply. To achieve optimal performance under the delay and energy constraints, in this paper we extend the traditional rate-distortion (R-D) model and the previously proposed delay R-D model to a novel delay-power-rate-distortion (d-P-R-D) model by including another two dimensions (the encoding time and encoder power consumption), which quantifies the relationship among source encoding delay, rate, distortion, and power consumption for IPPPP coding mode in H.264/AVC. We have verified the accuracy of our proposed d-P-R-D model through experiments. Based on the proposed d-P-R-D model, we develop a novel rate-control (RC) algorithm, which minimizes the encoding distortion under the constraints of rate, delay, and power. The experimental results demonstrate the superiority of the proposed RC algorithm over the existing scheme. Therefore, the d-P-R-D model and the model-based RC provide a theoretical basis and a practical guideline for the cross-layer system design and performance optimization in wireless video communication under delay and energy constraints.

Inherent statistical correlation for context-based prediction and structural interdependencies for local coherence is not fully exploited in existing lossless image coding schemes. This paper proposes a novel prediction model where the optimal correlated prediction for a set of pixels is obtained in the sense of the least code length. It not only exploits the spatial statistical correlations for the optimal prediction directly based on 2D contexts, but also formulates the data-driven structural interdependencies to make the prediction error coherent with the underlying probability distribution for coding. Under the joint constraints for local coherence, max-margin Markov networks are incorporated to combine support vector machines structurally to make max-margin estimation for a correlated region. Specifically, it aims to produce multiple predictions in the blocks with the model parameters learned in such a way that the distinction between the actual pixel and all possible estimations is maximized. It is proved that, with the growth of sample size, the prediction error is asymptotically upper bounded by the training error under the decomposable loss function. Incorporated into the lossless image coding framework, the proposed model outperforms most prediction schemes reported.

Detecting generic object categories in images and videos are a fundamental issue in computer vision. However, it faces the challenges from inter and intraclass diversity, as well as distortions caused by viewpoints, poses, deformations, and so on. To solve object variations, this paper constructs a structure kernel and proposes a multiscale part-based model incorporating the discriminative power of kernels. The structure kernel would measure the resemblance of part-based objects in three aspects: 1) the global similarity term to measure the resemblance of the global visual appearance of relevant objects; 2) the part similarity term to measure the resemblance of the visual appearance of distinctive parts; and 3) the spatial similarity term to measure the resemblance of the spatial layout of parts. In essence, the deformation of parts in the structure kernel is penalized in a multiscale space with respect to horizontal displacement, vertical displacement, and scale difference. Part similarities are combined with different weights, which are optimized efficiently to maximize the intraclass similarities and minimize the interclass similarities by the normalized stochastic gradient ascent algorithm. In addition, the parameters of the structure kernel are learned during the training process with regard to the distribution of the data in a more discriminative way. With flexible part sizes on scale and displacement, it can be more robust to the intraclass variations, poses, and viewpoints. Theoretical analysis and experimental evaluations demonstrate that the proposed multiscale part-based representation model with structure kernel exhibits accurate and robust performance, and outperforms state-of-the-art object classification approaches.

In the existing cooperative cognitive radio (CR), qualities of service (e.g., rate or outage probability) of primary users can be improved by cooperative transmission from secondary users. However, as owners of the spectrum, primary users' traffic demands are relatively easy to satisfy. The rationale is that they would be more interested in the benefits of other formats (e.g., revenue), rather than the enhanced rate. In this paper, we propose a new cooperative CR framework, where primary users assist in the transmissions of secondary users. In exchange for this concession, primary users receive payments from secondary users for the spectrum and cooperative transmit power being used in cooperation. An auction-theoretic model with multiple auctioneers, multiple bidders, and multiple commodities is developed for a joint spectrum and cooperative power allocation. Since the spectrum and power are two heterogeneous but correlated commodities, their properties are specifically considered in auction strategy design. Finally, we mathematically prove the convergence of the proposed auction game, and show with numerical results, that the proposed auction is beneficial to both primary and secondary users.

Previous reference-based compression on DNA sequences do not fully exploit the intrinsic statistics by merely concerning the approximate matches. In this paper, an adaptive difference distribution-based coding framework is proposed by the fragments of nucleotides with a hierarchical tree structure. To keep the distribution of difference sequence from the reference and target sequences concentrated, the sub-fragment size and matching offset for predicting are flexible to the stepped size structure. The matching with approximate repeats in reference will be imposed with the Hamming-like weighted distance measure function in a local region closed to the current fragment, such that the accuracy of matching and the overhead of describing matching offset can be balanced. A well-designed coding scheme will make compact both the difference sequence and the additional parameters, e.g. sub-fragment size and matching offset. Experimental results show that the proposed scheme achieves 150% compression improvement in comparison with the best reference-based compressor GReEn.

To speed up the convergence rate of learning dictionary in low bit-rate video coding, this paper proposes a spatio-temporal online dictionary learning (STOL) algorithm to improve the original adaptive regularized dictionary learning with K-SVD which involves a high computational complexity and interfere with the coding efficiency. Considering the intrinsic dimensionality of the primitives in training each series of 2-D sub dictionaries is low, the 3-D low-frequency and high-frequency dictionary pair would be formed by the online dictionary learning to update the atoms for optimal sparse representation and convergence. Instead of classical first-order stochastic gradient descent on the constraint set, e.g. K-SVD, the online algorithm would exploit the structure of sparse coding in the design of an optimization procedure in terms of stochastic approximations. It depends on low memory consumption and lower computational cost without the need of explicit learning rate tuning. Through drawing a cubic from i.i.d. samples of a distribution in each inner loop and alternating classical sparse coding steps for computing the decomposition coefficient of the cubic over previous dictionary, the dictionary update problem is converted to solve the expected cost instead of the empirical cost. For dynamic training data over time, online dictionary learning behaves faster than second-order iteration batch alternatives, e.g. K-SVD. Through experiments, the super-resolution reconstruction based on STOL obviously reduces the computational complexity to 40% to 50% of the K-SVD learning-based schemes with a guaranteed accuracy.

In this paper, we develop a novel genome compression framework based on distributed source coding (DSC)[3], which is specially tailored to the need of miniaturized devices. At the encoder side, subsequences with adaptive code length can be compressed flexibly through either low complexity DSC based syndrome coding or hash coding with the decision determined by the existence of variations between source and reference known from the decoder feedback. Moreover, to tackle the variations between source and reference at the decoder, we carefully designed a factor graph based low-density parity-check (LDPC) decoder, which automatically detects insertion, deletion and substitution.

A multi-view video system would capture the same scene from different viewpoints, and suffer from significant illumination variations due to inaccurate camera calibration and varying light conditions. It deteriorates the inter-view correlations and the quality of synthesized views at decoder side. By converting the problem of illumination compensation to noise removing, this paper proposes a low-rank matrix completion algorithm to reduce the effect of illumination variation. Diverged from the existing work which chooses a central view as reference and keeps views consistent, it is dedicated to compensating all the views to match the low-rank structure of views. The discrepancies among views are regarded as mixed noise, and constructed as an incomplete matrix with low-rank. It is solved by a stable matrix completion and obtains a mapping function for color correction. It is robust to outliers since only plausible corresponding points are involved with completion. Experimental results show that the proposed algorithm can increase coding efficiency by up to 0.7 dB for luminance component and up to 2.1 dB for chrominance components.

Video retargeting is a technique that resizes the aspect ratio and resolution of a video in a content aware manner. There are three main challenges in the video retargeting: visual saliency preservation, deformation prevention and temporal consistency preservation. Existing video retargeting methods can be roughly classified into seam carving based methods and warping based methods. These methods effectively protect important regions in video, but can hardly deal with deformations and temporal incoherence, especially for videos whose foreground and background are both in motion. In this paper, we propose a novel shape preserving video retargeting method to reduce deformations and maintain temporal consistency via matching salient curves in the video frames. Our method transforms the anti-deformation and temporal consistency problems in video retargeting into a curve matching cost minimization problem. By incorporating a deformation cost and temporal inconsistency cost into the seam carving framework, the quality of the retargeted videos can be significantly improved. Experimental results also show that the proposed method achieves better performance compared with other two state-of-the-art methods.

Recently, compressive sampling (CS) is an active research field of signal processing. To further decrease the necessary measurements and get more efficient recovery of a signal x, recent approaches assume that x lives in a union of subspaces (UoS). Unlike previous approaches, this paper proposes a novel method to sample and recover an unknown signal from a union of data-driven subspaces (UoDS). Instead of a fix set of supports, this UoDS is learned from classified signal series which are uniquely formed by block matching. The basis of these data-driven subspaces is regularized after dimensionality reduction by principal component extraction. A corresponding recovery solution with provable performance guarantees is also given, which takes full advantage of block-sparsity structure and improves the recovery efficiency. In practice, the proposed scheme is fulfilled to sample and recover frames in video sequences. The experimental results demonstrate that the proposed video sampling behaves better performance in sampling and recovery than the classical CS.

This paper proposes a novel model on intra coding for High Efficiency Video Coding (HEVC), which simultaneously predicts blocks of pixels with optimal rate distortion. It utilizes the spatial statistical correlation for the optimal prediction based on 2-D contexts, in addition to formulating the data-driven structural interdependences to make the prediction error coherent with the probability distribution, which is desirable for successful transform and coding. The structured set prediction model incorporates a max-margin Markov network (M3N) to regulate and optimize multiple block predictions. The model parameters are learned by discriminating the actual pixel value from other possible estimates to maximize the margin (i.e., decision boundary bandwidth). Compared to existing methods that focus on minimizing prediction error, the M3N-based model adaptively maintains the coherence for a set of predictions. Specifically, the proposed model concurrently optimizes a set of predictions by associating the loss for individual blocks to the joint distribution of succeeding discrete cosine transform coefficients. When the sample size grows, the prediction error is asymptotically upper bounded by the training error under the decomposable loss function. As an internal step, we optimize the underlying Markov network structure to find states that achieve the maximal energy using expectation propagation. For validation, we integrate the proposed model into HEVC for optimal mode selection on rate-distortion optimization. The proposed prediction model obtains up to 2.85% bit rate reduction and achieves better visual quality in comparison to the HEVC intra coding.

For promising vision-based video coding on low-quality data, this paper proposes a sparse spatio-temporal representation with adaptive regularized dictionary learning and develops a low bit-rate video coding scheme. In a reversed-complexity Wyner-Ziv coding manner, it selects a subset of key frames to code at original resolution, while the rest are down sampled and reconstructed by a sparse spatio-temporal approximation using key frames as a training dataset. Since primitive patches (geometry) are of low dimensionality and can be well learned from the primitive patches across frames in a scale space, a video frame is divided into three layers: a primitive layer, a nonprimitive coarse layer, and a nonprimitive smooth layer. The multiscale differential feature representations are invertible to facilitate reconstruction with dictionary learning, and the target is formulated as an optimization problem by constructing a sparse representation of 2-D patches and 3-D volumes over adaptive regularized dictionaries, a set of 2-D subdictionary pairs trained from primitive patches, and a 3-D dictionary trained from nonprimitive volumes. Specifically, the nonprimitive layer is constructed as volumes in to order keep it consistent along the motion trajectory, which enables sparse representations over a learned 3-D spatio-temporal dictionary. Through hierarchical bidirectional motion estimation and adaptive overlapped block motion compensation, the 3-D low-frequency and high-frequency dictionary pair is designed by the K-SVD algorithm to update the atoms for optimal sparse representation and convergence. In reconstruction, the lost high-frequency information of the down-sampled frames can be synthesized from the sparse spatio-temporal representation over the adaptive regularized dictionaries. Extensive experiments validate the compression efficiency of the proposed scheme versus H.264/AVC in terms of both objective and subjective comparisons.

Detecting multiple targets and obtaining a record of trajectories of identical targets that interact mutually infer countless applications in a large number of fields. However, it presents a significant challenge to the technology of object tracking. This paper describes a novel structured learning-based graph matching approach to track a variable number of interacting objects in complicated environments. Different from previous approaches, the proposed method takes full advantage of neighboring relationships as the edge feature in a structured graph, which performs better than using the node feature only. Therefore, a structured graph matching model is established, and the problem is regarded as structured node and edge matching between graphs generated from successive frames. In essence, it is formulated as the maximum weighted bipartite matching problem to be solved using the dynamic Hungarian algorithm, which is applicable to optimally solving the assignment problem in situations with changing edge costs or weights. In the proposed graph matching model, the parameters of the structured graph matching model are determined in a stochastic learning process. In order to improve the tracking performance, bilateral tracking is also used. Finally, extensive experimental results on Dynamic Cell, Football, and Car sequences demonstrate that the new approach effectively deals with complicated target interactions.

In this paper, we propose a sparse representation learning with adaptive regularized dictionaries and develop a low bit-rate video coding scheme. In a reversed-complexity manner, it select a subset of key frames to encode at original resolution, while the rest are down-sampled and super-resolution reconstructed by a sparse super-resolution estimations using key frames as training set. Since primitive patches are of low dimensionality and can be well learned from the primitive patches across different images, video frame is divided into three layers: a primitive layer, a non-primitive coarse layer, and a non-primitive smooth layer. The non-primitive layer is constructed as volumes to keep consistent along the motion trajectory, which enables sparse representations over a learned 3-D spatio-temporal dictionary. Correspondingly, the target is formulated as an optimization problem by constructing a sparse representation of low-resolution frame patches or volumes over adaptive regularized dictionaries: a set of 2-D sub dictionary pairs trained from 2-D primitive patches and a 3-D dictionary trained from non-primitive volumes. In reconstruction, the lost high-frequency information of the down-sampled frames can be synthesized from the sparse spatio-temporal representation over the adaptive regularized dictionaries. Experimental results validate the compression efficiency of the proposed scheme versus the H.264/AVC in terms of both objective and subjective comparison.

In this paper, we proposes a highly compressed video sampling scheme that optimally utilizes the redundancy spanning all the dimensions of the signal. Considering that we further compress the redundant measurements of the existing Kronecker compressive sensing (KCS), the proposed scheme will significantly improve the sampling efficiency. The contribution of our work is two fold. First, a holo-compressive sampling spanning both temporal and spatial dimensions is proposed to overcome the drawback of KCS, and named HKCS. This new scheme keeps on employing Kronecker product to design the sensing matrix. However, it is worth emphasizing that the original identity matrix, a factor of the conventional Kronecker product sensing matrix, is replaced by an ill-posed matrix which enables compressive samplings along the temporal dimension. The advantages of HKCS include: samplings of video along the temporal and spatial dimensions are simultaneously compressed and the necessary measurements for exact recovery are signihcantly reduced; moreover, the Kronecker product sensing matrix retains the block structure, which ensures a feasible distributed sampling scheme for practical application. We prove that, with the same sampling rate, matrices of HKCS have relatively smaller mutual coherence.

Object recognition is a fundamental problem in computer vision. Part-based models offer a sparse, flexible representation of objects, but suffer from difficulties in training and often use standard kernels. In this paper, we propose a positive definite kernel called "structure kernel", which measures the similarity of two part-based represented objects. The structure kernel has three terms: 1) the global term that measures the global visual similarity of two objects; 2) the part term that measures the visual similarity of corresponding parts; 3) the spatial term that measures the spatial similarity of geometric configuration of parts. The contribution of this paper is to generalize the discriminant capability of local kernels to complex part-based object models. Experimental results show that the proposed kernel exhibit higher accuracy than state-of-art approaches using standard kernels.

This paper proposes a novel Slepian-Wolf decoding algorithm for distributed video coding by exploiting not only the statistical correlation between the side-information and source but also the spatio-temporal consistency constraint of video sequences. The proposed algorithm models the log-likelihood-ratio (LLR) information for Slepian-Wolf decoding as an anisotropic MRF model and solving the inference by iteratively performing conventional probabilistic Slepian-Wolf decoding, which imposes the global bit-wise constraint from the Slepian-Wolf encoding process, and MRF optimization with belief propagation (BP) to enforce the local geometric regularity constraint of video frames. Experimental results demonstrate a considerable performance gain beyond existing Slepian-Wolf decoding algorithms in literature.

In this paper, we investigate and propose a novel prediction model for lossless image coding in which the optimal correlated prediction for block of pixels are simultaneously obtained in the sense of the least code length. It not only utilizes the spatial statistical correlation for the optimal prediction directly based on 2-D contexts, but also formulates the data-driven structural interdependencies to make the prediction error coherent with the underlying probability distribution for coding. Besides the discriminative adaptive pixel-wise prediction, the Markov network is adaptively derived to maintain the coherence of prediction in the blocks and seek the concurrent optimization of set of prediction by relating the loss function to actual code length. The prediction error is shown to be asymptotically upper bounded by the training error under the decomposable loss function. For validation, we apply the proposed model into lossless image coding and experimental results show that the proposed scheme outperforms the best prediction scheme reported.

This paper proposes a novel model on intra-coding for high efficiency video coding (HEVC), which can simultaneously make the set of prediction for block of pixels in an optimal rate-distortion sense. It not only utilizes the spatial statistical correlation for the optimal prediction based on 2-D contexts, but also formulates the data-driven structural interdependencies to make the prediction error coherent with the probability distribution which is favorable for subsequent transform and coding. The so-called structured set prediction model incorporates max-margin Markov network to regulate and reason the multiple prediction in the blocks. The model parameters are learned by discriminating the actual pixel value from the other possible estimates to the maximal margin. Distinguished from the existing methods concerning the minimal prediction error, the Markov network is adaptively derived to maintain the coherence of set of prediction. To be concrete, the proposed model seeks the concurrent optimization of the set of prediction by relating the loss function to the probability distribution of subsequent DCT coefficients. The prediction error is demonstrated to be asymptotically upper bounded by the training error under the decomposable loss function. For validation, we integrate the proposed model into HEVC intra coding and experimental results show obvious improvement of coding performance in terms of BD-rate.

Variational method is a well-established technique that solves for a dense field, which is widely adopted in the estimation of optical flow field and remains the most accurate technique to date. However, one of the problems in variational method lies in that it is optimized in an iterative manner towards a single objective, but local details may be compromised owing to the “big picture”. In this paper, we address this problem in an optical flow framework by introducing two sparse local rectifications to the global numerical scheme, i.e., motion-based weight selection and keypoint matching. The selection of the weighting parameter in a self-adaptive and content-aware manner provides a more accurate estimation of the optical flow field near motion boundaries, and motion details and small structures are preserved in the optical flow field by keypoint matching in the initialization of the optical flow field. Experimental results using the Middlebury dataset show that the proposed algorithm achieves higher accuracy compared to the original TV-ℓ1optical flow algorithm and many state-of-the-art methods.

This paper addresses the problem of advanced silhouette tracking with no prior information, and proposes shape-oriented segmentation together with graph matching corroboration. In terms of unified energy minimization, the shape-oriented graph cut in segmentation exploits the shape information by penalizing the feature points in alignment with shape-oriented map of adjacent frames. While reducing the temporal inconsistencies and improve the accuracy of segmentation, the energy model of graph matching is further designed to compensate the validity of segmentation. To be concrete, it is involved with structural matching cost and unmatched penalty cost to deal with occlusion during tracking. The effectiveness of the proposed scheme is shown with experiments on challenging real-world image sequences.

To analyze the tubular structure correctly and obtain a record of the centerlines has become significantly more challenging and infers countless applications in a large amount of fields. Hence, a robust and automated technique for extracting the centerlines of the tubular structure is required. To address complicated 3D tubular objects, a novel kernel-based modeling approach with regard to minimizing tracking energy is presented in this paper. The 3D tubular structure can be demonstrated as a kernel-based superellipsoid model with non-uniform weights. To improve the performance, Gaussian process is also introduced to update the parameters of the kernel-based model, especially for the complicated structure with cross sections, varying radii, and complicated branches. At last, the extensive experimental results on 3D tubular data demonstrate that our proposed method deals effectively with complicated tubular structure.

This paper proposes a distributed robust optimization scheme to jointly optimize overall video quality and traffic performance for scalable video multirate multicast over practical wireless networks. In order to guarantee layered utility maximization, the initial nominal joint source and network optimization is defined, where each scalable layer is tailored in an incremental order and finds jointly optimal multicast paths and associated rates with network coding. To enhance the robustness of the nominal convex optimization formulation with nonlinear constraints, we reserve partial bandwidth for backup paths disjoint from the primal paths. It considers the path-overlapping allocation of backup paths for different receivers to take advantage of network coding, and takes into account the robust multipath rate-control and bandwidth reservation problem for scalable video multicast streaming when possible link failures of primary paths exist. Specifically, an uncertainty set of the wireless medium capacity is introduced to represent the uncertain and time-varying property of parameters related to the wireless channel. The targeted uncertainty in the robust optimization problem is studied in a form of protection functions with nonlinear constraints, to analyze the tradeoff between robustness and distributedness. Using the dual decomposition and primal-dual update approach, we develop a fully decentralized algorithm with regard to communication overhead. Through extensive experimental results under critical performance factors, the proposed algorithm could converge to the optimal steady-state more quickly, and adapt the dynamic network changes in an optimal tradeoff between optimization performance and robustness than existing optimization schemes.

In this paper, we propose the bitwise structured prediction model for lossless image coding, especially for the oscillatory regions. The learning-based model utilizes the regular features obtained from the predicted local data. At first, the pixel-wise prediction is decomposed into the bitwise ones. In each bit plane, the prediction of the current bit is simplified to the max margin estimation for the 0/1 prediction problem and obtained directly conditioned on the neighboring predicted bits. Furthermore, since the decreasing dependencies of neighboring bits in lower bit plane lead to the turbulence of predictive results, the structured prediction is proposed to establish the Markov network to constrain the outputs of the bit planes, and suppress the prediction errors with a well-defined loss function. Consequently, the min-max formulation is proposed for the concurrent optimization for maximizing the 0/1 margin of all the bit planes.

Summary form only given. In this paper, we propose an optimized in painting-based macro block(MB) prediction mode (IP-mode) in the state-of-the-art H.264/AVC video compression engine, and belief propagation (BP) is applied to achieve the global spatio-temporal consistency between the predicted content and the co-located known region. To decrease the computing complexity of the iterative BP algorithm, we explore structure and motion features by tensor votes projected from the decoded regions, to assign the priority of message scheduling and prune the intolerable labels. No side information is need to be coded into the bit stream, while the structure and motion information is estimated from the decoded region at decoder side. Compared with the existing prediction modes in H.264/AVC, the proposed IP-mode only encode the macro block header and residual data, where the residual is lighter in homogeneous texture regions by the optimized BP algorithm with label pruning. Experiments validate that the proposed video compression scheme can achieve a better R-D performance, and the computing complexity is largely reduced through the inference of structure and motion features.

The contourlet transform provides a flexible directional image decomposition by employing Laplacian pyramid and uniform directional filter banks. Although it is able to efficiently capture the 2-D piecewise smooth functions with line or curve discontinuities, its major drawbacks of 4/3 redundancy and non-ideal filter banks in Laplacian pyramid are the main obstacles for high performance image/video coding. In this paper, we propose a video coding scheme based on the sharp frequency localized contourlet transform (contourlet/SFL) under the sparse representation framework, in which the iterative thresholding algorithm is applied to get an l1-norm sparser version of the transform coefficients. In the meantime, considering the fact that strong inter-band and inter-scale dependencies exist in the contourlet/SFL coefficients, a directional embedded image coding system is proposed to propagate the significance status by using the neighbor, cousin and parent significant coefficients as seeds. Experimental results show that the coding performance and visual quality of the sparse contourlet/SFL based video compression scheme are superior to the wavelet-based one, especially on those sequences full of directional structures at low bit rates.

In this paper, we address the problem of access to global information from any single point in wireless sensor networks. To this end, we propose a distributed in-network data acquisition approach, on the basis of compressive sensing, in which sparse random projections and randomized gossiping are jointly designed. In the context of unreliable distributed wireless settings, a simple random gossip algorithm is adopted. It allows the sensor to forward the linear combined data (readings) to randomly chosen neighbors. Also, each transmission path of the combined data is mapped to one row of the projection matrix, forming low-cost sparse random projections. Moreover, a theoretical model is developed that exactly characterizes the relationship of the number of sparse projections, the degree of the sparsity, and the error probability. Finally, simulation results show that, compared with the conventional approach, the proposed algorithm can achieve better reconstruction quality with less communication overhead.

This paper studies the problem of correlated data gathering in wireless sensor networks. For a maximum network utility, a efficient data capture and transmission framework is proposed for correlated sources, where localized S-W source coding, network coding based flow control and opportunistic routing are jointly optimized. To increase the throughput and guarantee the decodability simultaneously, a dynamic network coding strategy is proposed, with which the intermediate node can easily decide whether to make a combination among the incoming flows. Also, an opportunistic routing approach is presented, which adopts a new metric (minimum congestion price) for forwarding node selection and results in a maximum utility benefit for the sensor nodes. Through the Lagrange dual and gradient approach, a fully distributed algorithm is represented. And the convergence and performance are validated by the numerical results.

Thanks to recent advances in hierarchical video coding and overlay network, layered overlay multicast has emerged as an important solution for video streaming over heterogeneous network. In this paper, we formulate it into a joint network flow control and performance optimization problem where adaptive layer rates are determined through a greedy algorithm close to optimal. The overlay network is constrained by a practical two-level hierarchical overlay model where the bandwidth sharing is imposed on underlying edge-bottleneck links. To maintain less computational complexity and avoidance of global information, the M-layer maximization problem is transformed into multiple one-layer minimization subproblems. In turn, a fully distributed algorithm is developed by decomposing the one-layer minimization problem into both inequality constrained transportation subproblem and the shortest path subproblem by using Lagrangian duality. We demonstrate that the distributed algorithm converges quickly and attains effective performance in both static and dynamic networks.

To correctly detect dynamic targets and obtain a record of the trajectories of identical targets in appearance over time, has become significantly more challenging and infers countless applications in biomedicine. In this paper, we propose a novel structured learning-based graph matching algorithm to track a variable number of interacting objects in dynamic environments. Different from previous approaches, the proposed method takes full advantage of neighboring relationships as edge feature in the structured graph. The target problem is regarded as structured node and edge matching between graphs generated from successive frames. In essence, it is formulated as the maximum weighted bipartite matching problem which is solved by dynamic Hungarian algorithm. The parameters of the structured graph matching model can be acquired in a stochastic graduated learning step in different dynamic environments. The extensive experiments on dynamic cell and football sequences demonstrate that the resulting approach deals effectively with complicated target interactions.

With the ever increasing concern of vision-based video analysis and coding over resource-limited systems, this paper proposes a novel video coding scheme that exploits low- quality video data and formulates as an inverse learning based video reconstruction from online training by diverse stochastic processes. Given a sparsely sampled incomplete data, the intrinsic nonlocal and spatio-temporal geometric regularity related to online training examples in the key frames are considered as a state-dependent uncertainty estimation problem using Gaussian Process (GP) regression. Unlike non-parametric or exemplar- based sampling methods, we consider non-parametric system models for sequential state estimation by using the Unscented Kalman Filter (UKF) as the state estimator. It inherits the unscented transform for linearization to the transition function and the observation function. Once an approximate motion and observation model is available, it can naturally be incorporated to make a further performance improvement.

This paper proposes a generic video compression framework with low-quality video data and a learning-based approach, which is rooted in sparse representation for the ill-posed problem of video super-resolution reconstruction. It is regularized by the prior manifold only on the “primitive patches”, and each primitive patch is modeled by a sparse representation concerning an over-complete dictionary of trained set. Due to low intrinsic dimensionality of primitives, the number of samples in the dictionary can be greatly reduced. Considering the similar geometry of the manifolds of the feature spaces from the low-frequency and the high-frequency primitives, we hypothesize that the low-frequency and its corresponding high-frequency primitive patches share the same sparse representation structure. In this sense, high-resolution frame primitives are divided into low-frequency and high-frequency frame primitives, and high-frequency frame primitive patches can be synthesized from both the high-frequency primitive patch dictionary and the sparse structure of the corresponding low-frequency frame primitive patches. It does not involve with explicit motion estimation and any assistant information, and decomposes the original video sequence into key frames and low-resolution frames with low entropy. The corresponding high-resolution frames would be reconstructed by combining the high-frequency and the low-frequency patches with smoothness constraints and the backpro-jection process. Experimental results demonstrate the objective and subjective efficiency in comparison with H.264/AVC and existing super-resolution reconstruction approaches.

Variational methods are among the most accurate techniques of optical flow computation. TV-L1 optical flow, which is based on L1-norm data fidelity term and total variation (TV) regularization term, preserves discontinuities in the flow field and also can deal with large displacements. However, the TV-L1 optical flow method is inaccurate near edges and computationally intensive. In this paper, we proposed a technique, called Edge-based Image Decomposition (EID), to improve the accuracy in the edge areas and also accelerate the original TV-L1 method. EID improves the performance by decomposing image into edge regions and flat regions, and also assigns computing power discriminatively. We evaluated our algorithm on Middlebury datasets and proved that by applying EID, 30% of run-time can be saved with no loss in accuracy, and with same run-time, 7% of accuracy can be promoted. In addition, we implemented our EID-enhanced TV-L1 optical flow algorithm on mobile phone with Android operating system. Our application calculates the optical flow field between two images and can be used to generate the disparity map and reconstruct 3D scenes.

This paper presents a new temporal and inter-view side-information fusion algorithm for distributed multi-view video coding (DMVC). Unlike existing fusion algorithms in DMVC schemes that produce the fusion mask by finding the motion vector outliers, it introduces conditional random fields (CRF) to exploit the intrinsic geometric regularity and temporal consistency constraint in multi-view video sequences. Specifically, Wyner-Ziv (WZ) frames are modeled by CRF with the temporal and the inter-view side-information as two observations. The observation distribution models the local accuracy of the temporal and the inter-view side-information. The transition distribution of the CRF model represents the local geometric regularity, e.g., the edge directions and the local smoothness of the WZ frame. Its parameters are trained from previously decoded WZ frames, and the inference is made on trained weights to generate fused side-information. The accurate modeling is validated to show a significant performance gain over the existing fusion algorithms by experiments.

This paper aims to optimize the overall video quality and traffic performance for multi-rate video multicast over hybrid wired/wireless networks. In order to perform layered utility maximization over tiered networks, we propose a joint source-network flow optimization scheme where individual layers of the scalable video stream are imposed on their optimal multicast paths and associated rates for the highest sustainable layered video quality with minimum costs. It sufficiently guarantees that each destination node accesses progressive layered stream in an incremental order, considers network coding across overlapping paths to destination nodes for decent multicast capacity, and addresses the link contention problem during wireless transmission. We formulate the problem into convex programming with the objective to minimize the total rate-distortion variations between layers. Using primal decomposition and the primal-dual approach, we develop a decentralized algorithm with two levels of optimization. The numerical and packet-level results compare extensive performance under different control conditions over coded and non-coded hybrid networks. It demonstrates that the proposed algorithm could actually achieve the max-flow throughput and provide better video quality with optimal layered access for heterogeneous receivers.

This paper studies a joint coding/routing optimization between network lifetime and video distortion by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding [distributed video coding and network coding (NC)] from both combinatorial optimization and information theory could make significant progress toward the performance limit and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps the total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Also, a generalized power consumption model for distributed video sources is developed, in which the coding complexity of Key frames and Wyner-Ziv frames is measured by translating specific coding behavior into energy consumption. On the basis of the distributed multiview video coding and NC-based multipath routing, the balance problem between lifetime (costs) and distortion (capacity) is modeled as an optimization formulation with a fully distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved by the gradient algorithm. The low-level optimization problem is further decomposed into a secondary master dual problem with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. The implementation of the distributed algorithm is discussed with regard to the communication overhead and dynamic network change. Simulation results validate the convergence and performance of the proposed algorithm.

This paper presents a new video compression framework over H.264/AVC scheme that integrates our proposed structured priority belief propagation (BP)-based inpainting prediction (IP) to exploit the intrinsic nonlocal and geometric regularity in video samples. Unlike the existing edge-based inpainting adopted in lossy image coding, the optimal predictor could maintain the pixel-wise fidelity and the robust error resilience without any assistant information. Beyond the local prediction limitation of traditional intra and inter-modes, the priority BP with regularized structure priors of a spatio-temporal Markov random field is imposed on the predictor in an adaptive and more convergent sense. Specifically, the structured sparsity of the predicted macroblock region is inferred by tensor voting projected from the co-located decoded regions. In turn, the priority and visiting order of nodes are assigned according to the sets of updated beliefs as the propagation of messages. Through relatively few iterations of forward and backward process, the sparse inference of priority BP would ensure a stable marginal belief distribution on the structure and texture through updating local messages and beliefs. Within the optimal mode selection on rate-distortion optimization (RDO), the IP-mode with structured priority BP outperforms the existing vision-based approaches, and specially achieves a better objective rate-distortion performance besides visual quality. The IP-mode with structured priority BP can be applied to both I and P frames to generate low entropy residue, e.g., homogeneous visual patterns, and the computation complexity is also competitive with one iteration of sparse inference. Moreover, it behaves more resilient with an intrinsic probabilistic inference than the intra and inter-modes.

Although wavelet-based scalable video coding becomes the state-of-the-art video compression engine for its adaptability to heterogeneous networks and clients, a large number of attempts have been made to integrate local directionality onto discrete wavelet transform to explore the intrinsic geometrical structures. Taking into consideration that the contours and textures scattered in different scales change their directional resolutions as their curvatures change, we investigate adaptive directional resolutions along scales to achieve the dual (scale and orientation) multiresolution transform. This paper proposes nonuniform directional frequency decompositions for video representation and approximation, and exploits the nonuniformity of orientation multiresolution distribution and designs nonuniform directional filter banks to make the geometrical transform more sparse and efficient. The nonuniform directional frequency decomposition under arbitrary scales is fulfilled by a non-symmetric binary tree (NSBT) topology structure with nonuniform directional filterbank design. In turn, the proposed scalable video coding framework, called DMSVC, is enriched with the dual multiresolution transform. Each temporal subband through motion compensated temporal filtering is further decomposed into multiscale subbands, and the highpass wavelet subspaces are divided into an arbitrary number of directional subspaces in alignment with the orientation distribution via phase congruency to establish NSBT. The paraunitary perfect reconstruction condition is provided through a polyphase identical form of filter bank. Comparing with the isolated wavelet basis, our transform provides a greater correlated set of localized and anisotropic basis functions. The spatio-temporal subband coefficients are coded by a 3-D ESCOT entropy coding algorithm which is adopted to match the structure of NSBT. Experimental results show that the reconstructed video frames DMSVC in the proposed DMSVC scheme have better visual quality than existing scalable video coding schemes. It could produce higher compression ratio on video sequences full of directional edges and textures.

In this paper, we propose a generic error resilient video coding (ERVC) scheme using embedded Wyner-Ziv (WZ) description. At the encoder side, a joint source-channel R-D optimized mode selection (JSC-RDO-MS) algorithm with WZ-coded anchor frames is statistically studied and developed. Given a stationary first-order Markov Gaussian source, the proposed mode optimization is justified by an analysis of the RD impact on the WZ bit-rate. JSC-RDO-MS involves in the estimation of expected rate and distortion of WZ coding with the unavailable side information, and the WZ bit-rate of each coding mode is determined based on the error correction capability of the specific WZ codec. At the decoder side, an online correlation noise model between the source and the side-information is proposed with a mixture of Laplacians whose parameters are attained to reflect the coherence of the motion field of successive frames and the energy of prediction residual. Each mixture component represents the statistical distribution of prediction residuals, and the mixing coefficients represent the amount of errors in motion compensation. The proposed scheme achieves the so-called classification gain by exploiting the spatially non-stationary characteristics of the motion field and texture. Extensive experimental results show that the proposed WZ-ERVC scheme achieves a better overall RD performance than existing ERVC schemes, and the proposed modeling algorithm also significantly outperforms the conventional Laplacian model by up to 2 dB.

Within the existing reconstruction process of distributed video coding (DVC), there are two major approaches: the maximum probability reconstruction and the minimum mean square error (MMSE) reconstruction. Both of them assume that each node, a pixel in pixel domain DVC or a coefficient in transform domain DVC, is i.i.d., and reconstruct the value of each node independently by only exploiting statistical correlation between source and side-information. These kinds of models produce considerable amount of artifacts in decoded Wyner-Ziv (WZ) frames and degrade the objective performance. In this paper, we propose a context-adaptive Markov random field (MRF) reconstruction algorithm which exploits both the statistical correlation and the spatio-temporal consistency by modeling the corresponding MRF of a generic DVC architecture, and solve the inference by finding its MRF-based maximum a posteriori (MAP) estimate. The energy function of the MRF model consists of two terms: a data term measuring the statistical correlation, and a geometric regularity term enforcing local spatio-temporal structure consistency which is modeled by optical flow estimation with regard to the critical parameters under a wide variety of DVC scenarios. In case the unreliability of the derived local structure, a confidence parameter is introduced to prevent inappropriate penalizing. To find the reconstructed patch assignment with the largest expected probability in the context-adaptive MRF, the energy minimization for the MRF-based MAP estimate of the WZ frames is solved by global optimization and greedy strategies. Compared to the existing maximum probability and MMSE reconstruction with i.i.d. model, a better subjective and objective performance is validated by extensive experiments.

In this paper, we study performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Content priority of video coding layers is considered during the flow routing to determine the optimal multicast paths and associated data rates for each layer. Our optimization scheme attempts to find content distribution meshes with minimum path costs for each video coding layer while satisfying the inter-layer dependency during scalable video coding. Based on primal decomposition and primal-dual analysis, we develop a decentralized algorithm with two optimization levels to solve the performance optimization problem. We also prove the stability and convergence of the proposed iterative algorithm using Lyapunov theories. Extensive experimental results demonstrate that the proposed algorithm not only achieves the max-flow throughput using network coding, but also provides better video quality with balanced layered access for heterogeneous receivers.

This paper develops a complete framework for perceptual video coding with anisotropic diffusion-based abstraction and completion under the spatio-temporal variation regularity. A soft clustering method is applied to retrieve transferable semantic implications of sampled frames in a sparser way. The restoration inference process as a learning equivalent optimization problem from a given set of sparse data, serves as non-parametric or exemplar-based sampling method by taking 3-D spatio-temporal similarity into consideration. Aside from pixel-wise color-based matching, two patch-based similarity metrics on motion-based pixel-wise similarity and semantic coherence are introduced for the exemplar-based reconstruction under complex situations. For each pixel in the abstracted frames, a set of matched patches will provide inference and its color is predicted with multi-hypothesis via weighted averaging. A pixel-wise confidence map based on spatio-temporal feature is also provided for important point selection so as to reduce the computation cost to an acceptable level. We validate both compression efficiency and restoration performance from coding gain, SSIM, optical flow consistency, and just-noticeable distortion (JND) on a variety of sources.

This paper aims to optimize joint overall video quality and traffic performance of multirate multicast of scalable video streaming accessing hybrid wired/wireless paths. In order to guarantee layered utility maximization over tiered wired/wireless coded networks, we propose a joint source and network flow optimization scheme where each scalable layer is tailored in an incremental order and finds jointly optimal multicast paths and associated rates. It not only achieves the highest sustainable layered video quality, but also takes into account both path-overlapping allocation for different receivers which takes advantage of network coding and the wireless link contention in a shared transmission medium. The convex programming is imposed on the total variation minimization of layered rate-distortion. Using primal decomposition and the primal-dual approach, we develop a decentralized algorithm with two levels of optimization. Experimental results demonstrate that the proposed algorithm not only provides better video quality with optimal layered access for heterogeneous receivers, but benefits the asymptotic capacity of a hierarchical "hybrid" wired/wireless architecture in a scalable sense.

This paper studies the power consumption performance and resource allocation optimization in wireless video sensor networks. Network coding based multipath routing, network flow control and video encoding bit rate are jointly optimized, aiming to maximize the network lifetime at a given power budget and video quality requirement. Importantly, to concretely measure the network coding power utilized on error recovery, a generalized power consumption model for network coding is first developed in this paper. Through the Lagrange dual and subgradient approach, a fully decentralized algorithm is proposed to solve the target convex optimization problem. Numerical results validate the convergence and performance of the proposed algorithm.

In this paper, we propose an optimized inpainting-based macroblock prediction mode in the state-of-the-art H.264/AVC video engine. Parallel with the existing intra- and inter- modes, it is regularized by the global tempo-spatial consistency between the current MB and the co-located decoded region. The target is formulated as a local optimization problem by minimizing the energy of Markov Random Field (MRF). An ordered belief propagation (BP) is developed to solve the optimization problem with spatio-temporal consistency regularity and achieve the patch assignment of largest probability. It ensures a stable marginal belief distribution through updating local messages via iterative forward and backward process, to impose a prioritized inference on the structure. Rate-distortion optimization is used to evaluate the mode selection within the inpainting-based prediction mode, intra- and inter modes. It has been implemented into H.264/AVC, and achieves bit-rate saving and higher PSNR performance, especially in low bit-rate.

In this paper, we propose an optimized inpainting-based macroblock (MB) prediction mode (IP-mode) in the state-of-the-art H.264/AVC video compression engine, and investigate a natural extension of structured sparsity over the ordered Belief Propagation (BP) inference in inpainting-based prediction. The IP-mode is regularized by a global spatio-temporal consistency between the predicted content and the co-located known texture, and could be adopted in both Intra and Inter frames without redundant assistant information. It is solved by an optimization problem under Markov Random Field (MRF), and the structured sparsity of the predicted macroblock region is inferred by tensor voting projected from the decoded regions to tune the priority of message scheduling in BP with a more convergent manner. Rate-distortion optimization is maintained to select the optimal mode among the inpainting-based prediction (IP-), the intra-, and inter-modes. Compared to the existing prediction modes in H.264/AVC, the proposed inpainting-based prediction scheme is validated to achieve a better R-D performance for homogeneous visual patterns and behave a more robust error resilience capability with an intrinsic probabilistic inference.

Regarding orientation resolution as an isolated variable from scale, this paper proposes a dual (scale and orientation) multiresolution transform into scalable video coding (SVC) framework. By projecting 2D signals (textures and edges) onto nonuniformly divided orientation subspaces, the dual multiresolution SVC (DMSVC) can capture 2-D curve smoothness with less coefficients, and provide more flexible spatial decomposition structures than traditional wavelet-based SVC (WSVC). In the spatial decomposition module of DMSVC, the nonuniform directional distribution along scale of each frame is detected by phase congruency in the overcomplete wavelet domain. The corresponding orientational multi-resolution is achieved by nonuniform directional filter banks (NUDFB) which is fulfilled via a non-symmetric binary tree (NSBT) structured frequency division. The wavelet basis function in each scale is converted to an adaptive set of nonuniform directional basis by employing nonuniform directional filter banks. Experimental results validate a superior coding performance and visual quality over WSVC especially on those sequences full of directional edges and textures.

In this paper, we address the joint optimization problem of rate allocation and distortion control for scalable video coding (SVC) multicast networks. Firstly, we construct video distribution meshes by coupling network coding and multipath routing with multi-rate control to seek optimal routing paths and associated transmission rates. Secondly, a particular minimum bandwidth consumption scheme is presented for each video layer to reduce video distortion, where the content priority of the base layer and the minimum bandwidth consumption at higher layers are taken into account. Finally, a convex mathematic model combining these considerations is proposed. Through decomposition and dual approach, the target convex optimization problem is solved by a fully decentralized algorithm which is decomposed into a two-level optimization procedure. Simulation results validate the convergence behavior and performance of the proposed algorithm.

We focus on the problem of single image super-resolution in this paper. Given a low-resolution image, we seek to synthesize its underlying high-resolution details using a learning based method. Inspired by recent progress in compressive sensing, we use sparse representation prior to regularize this ill-posed problem. On the other hand, with natural image statistics taken into consideration, we enforce the prior only on those image patches associated with image primitives rather than on arbitrary ones. Specifically, each patch from primitive layer of the lowresolution image, which can be viewed as a low-dimensional projection of a high-resolution primitive patch, is conjectured to have a sparse representation concerning an over-complete dictionary. Under mild conditions, the sparse representation can be correctly restored from the low-dimensional projection according to the theory of compressive sensing. We also construct a dictionary using image primitive patches which works well on generic input images. Experiment results show the efficiency of our method by outperforming other learning-based methods both subjectively and objectively.

In this paper, we study the optimal bandwidth allocation for scalable video coding (SVC) streaming in multiple overlays. We model the whole bandwidth request and distribution process as a set of decentralized auction games between the competing peers. For the upstream peer, a bandwidth allocation mechanism is introduced to maximize the aggregate revenue. For the downstream peer, a dynamic bidding strategy is proposed. It achieves maximum utility and efficient resource usage by collaborating with a content-aware layer dropping/adding strategy. Also, the convergence of the proposed auction games is theoretically proved. Experimental results show that the auction strategies can adapt to dynamic join of competing peers and video layers.

An important issue in Wyner-Ziv video coding is the reconstruction of Wyner-Ziv frames with decoded bit-planes. So far, there are two major approaches: the Maximum a Posteriori (MAP) reconstruction and the Minimum Mean Square Error (MMSE) reconstruction algorithms. However, these approaches do not exploit smoothness constraints in natural images. In this paper, we model a Wyner-Ziv frame by Markov random fields (MRFs), and produce reconstruction results by finding an MAP estimation of the MRF model. In the MRF model, the energy function consists of two terms: a data term, MSE distortion metric in this paper, measuring the statistical correlation between side-information and the source, and a smoothness term enforcing spatial coherence. In order to better describe the spatial constraints of images, we propose a context-adaptive smoothness term by analyzing the correspondence between the output of Slepian-Wolf decoding and successive frames available at decoders. The significance of the smoothness term varies in accordance with the spatial variation within different regions. To some extent, the proposed approach is an extension to the MAP and MMSE approaches by exploiting the intrinsic smoothness characteristic of natural images. Experimental results demonstrate a considerable performance gain compared with the MAP and MMSE approaches.

This paper studies a joint coding/routing optimization between network lifetime and rate-distortion, by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding (distributed source coding and network coding) from both combinatorial optimization and information theory could make significant progress towards the performance limit of information networks and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps a total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Based on the localized Slepian-Wolf coding and network coding-based multipath routing, the balance problem between distortion (capacity) and lifetime (costs) is modeled as an optimization formulation with a distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved with the gradient algorithm. The low-level optimization problem is decomposed into a secondary master dual problem (encoding, energy, and congestion prices update) with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. Numerical results validate the convergence and performance of the proposed algorithm.

This paper presents a generic video coding framework with the texture abstraction and completion, inspired by a strong grouping bias of local elements in Gestalt psychology. Abstracting imagery by grouping perceptual salience from anisotropic diffusion, it decomposes video images into two layers composing of semantic components and residual detail. The similarity between textures of abstraction layer is motivated to infer the restoration of missing detail, under the spatio-temporal variation regularity. Through a motion and spatial context of moton, hence, a group of pictures (GOP) is divided into key frames and abstracted frames to form the final compressed data. An abstraction refinement is tuned to improve matching of detail restoration based on bilateral filtering. The proposed approach is more generic without incurring any specific side information, and achieves up to 20% bit saving versus standard H.264 at similar visual quality levels.

In this paper, we adopt constrained relaxation for distributed multi-view video coding (DMVC). The novel framework integrates the graph-based segmentation and matching to generate inter-view correlated side information without knowing the camera parameters. Moreover, graph-based representations of multi-view images are incorporated to form more distinctive feature constraints. The sparse data as a good hypothesis space aim for a best matching optimization of inter-view side information with compact syndromes, from inferred relaxed coset. The plausible filling-in from a priori feature constraints between neighboring views could reinforce a promising compensation to inter-view side information generation for joint multi-view decoding. In order to find distinctive feature matching with a more stable approximation, PCA-SIFT and TPS (thin plate spline) are adopted to reduce the dimension of SIFT descriptors and construct a more accurate inter-view motion model. The experimental results validate the high estimation precision and the rate-distortion improvements.

This paper addresses the performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Importantly, contextual priors of scalable video layers are imposed on the flow routing optimization problem, seeking to guarantee the transmission cost for each layer in an incremental order and find jointly optimal multicast paths and associated rates. Through a primal decomposition and the primal-dual approach, a decentralized algorithm with two-level optimization update is developed to solve the target convex optimization problem. Numerical and simulation results validate the convergence and network performance of the proposed algorithm.

Most of the Wyner-Ziv (WZ) video coding schemes in literature model the correlation noise (CN) between original frame and side information (SI) by a given distribution whose parameters are estimated in an offline process. In this paper, an online CN modeling algorithm is proposed towards a more practical WZ-based error resilient video coding (WZ-ERVC). In ERVC scenario, the side-information is typically generated from the error concealed picture instead of bi-directional motion prediction. The proposed online CN modeling algorithm achieves the so-called classification gain by exploiting the spatially non-stationary characteristics of the motion field and texture. The CN between the source and error concealed SI is modeled by a Laplacian mixture model, where each mixture component represents the statistical distribution of prediction residuals and the mixing coefficients portray the motion vectors estimation error. Experimental results demonstrate significant performance gains both in rate and distortion versus the conventional Laplacian model.

Neuron axon analysis through confocal microscopic image stack is dedicated in visualizing the geometrical features and topological characteristics of the 3D tubular biological objects, to ascertain the morphological properties and reconstruct the connectivity of neurons. This paper proposes a new curvilinear tracking algorithm which initializes a superellipsoid kernel into the tube by fitting the intensity energy distribution with multiple scales of steps and radii other than Hessian kernel. It is herein solved as an energy optimization in a graphical model with maximum likelihood, which preserves an equilibrium distribution across all the nodes with an attenuation penalty of orientation transition. Local potential energy diffusion of different axons is tracked by dynamic priority and pruning inference, to solve the cross-over problem. The centerline could be semi-automatically extracted following the selected initial point. Experimental results on 3D axon volumes verify that the proposed approach can handle complicated axon structures without elaborate segmentation.

In this paper, we seek the optimal overlay multicast performance for scalable video streaming. To adapt to layered multicast with multiple paths, a new metric layer stretch is introduced, measuring video layer's dissemination latency along the overlay. Multi-path routing, network coding based layer subscription and congest control are jointly optimized, formulating a linear programming to minimize the average layer stretch over all receivers. It not only solves the issue of inter-layer dependency of scalable video coding, but allows each receiver to determine the required video quality. Moreover, a heuristic algorithm that can be implemented in a distributed manner is provided to approximate the optimal solution. Simulation results validate the network performance of the proposed scheme.

The sequential context modeling framework is generalized to a non-sequential one by context relaxation from consecutive suffix of the subsequences of symbols to the permutation of the preceding symbols as result of considering complex context structures in such sources as video and program binaries. Context weighting tree is also extended to a series of context trees which are built according to the "model tree", in which the descendent relationship in the formation of non-sequential context sets is described. Model redundancy and maximum a posteriori model in the framework are discussed and compared. A decision method based on the greedy algorithm is proposed to customize sets of models fitting the concrete sources. Brief description of application to executable data files incorporating with the semantics and syntax constraints are given and experiment are made accordingly as a validation.

Apart from the existing temporal and inter-view interpolation technique in distributed multi-view video coding (DMVC), this paper is dedicated to not only investigating multiple side information implication at the decoder, but also preserving the constrained relaxation with high-level features matching. We present a novel feature-based Wyner-Ziv coding framework (FWZC) for DMVC, which devotes scale-invariant features extraction and matching to generate inter-view correlated side information without knowing the camera parameters and has a more significant improvement rate-distortion performance. The scale-invariant local features are identified as the most resistant to image deformations and affine distortion between different views of an object or scene. The plausible filling-in from a priori distinctive feature constraints between neighboring views could make a promising compensation to inter-view side information generation for joint multi-view decoding. The experimental results show high precision for objects with high motion and quite significant improvement in the rate-distortion performance.

The fundamental problem associating error resilience with vision-related technique is how to generate a plausible substitute for the unknown regions with complex and semantic structure, implicitly or explicitly by adding significant context redundancy to the coded video, subject to local similarity and global consistence in tempo-spatial domain. Following the structure-aware inpainting for image coding, an adaptive error resilience algorithm using redundant structure coding with multiscale B-spline based feature localization is proposed in this paper. In the proposed scheme, the curvature-based curve representation information extracted from original video pictures is coded and encapsulated into redundant slice of the H.264/AVC standard. To meet the required channel bandwidth and conditions, the underlying redundant structure coding allows for different representations with an optimal coding strategy from corresponding texture compressed version in the primary slice. With the proposed curvature and distortion measure, we can observe that acceptable compressed rate is achieved while progressive reconstruction accuracy is ensured.

This paper aims to attain sparser representation of a 2-D signal by introducing orientation resolution as a second multiresolution besides multiscale, which is formulated to achieve a dual multiresolution decomposition framework by nonuniform directional frequency decompositions (NUDFB) under arbitrary scales. In this scheme, NUDFB is fulfilled by changing the topology structure of a non-symmetric binary tree (NSBT). Through this nonuniform division, we can get arbitrary orientation resolution r at a direction of c2-runder a target scale. Every two-channel filter bank on each node of this NSBT is designed to be a paraunitary perfect reconstruction filter bank, so NUDFB is an orthogonal filter bank. This dual multiresolution decomposition will definitely have bright prospect in its application, such as texture analysis, image processing or video coding. A potential application is presented by applying NUDFB in wavelet domain.

In most existing in-band wavelet video coding schemes, over-complete wavelet transform is used for the motion-compensated temporal filtering (MCTF) of each spatial subband. It can overcome the shift-variance of critical sampling wavelet transform and improve the coding efficiency of the in-band scheme. However, a dilemma exists in the current implementations of in-band MCTF (IBMCTF), which is whether or not to exploit the spatial highpass subbands in motion compensation of the spatial lowpass subband. The absence of the spatial highpass subbands will result in significant quality loss in the reconstructed full-resolution video, whereas the presence of the spatial highpass subbands may bring serious mismatch error in the decoded low-resolution video since the corresponding highpass subbands may be unavailable at the decoder. In this paper, we first analyze the mismatch error propagation in decoding the low-resolution video. Based on our analysis, we then propose a frame-based cross-resolution leaky prediction scheme for IBMCTF. It can make a good tradeoff between alleviating the low-resolution mismatch and improving the full-resolution coding efficiency. Experimental results show that the proposed scheme can dramatically reduce the mismatch error by 0.3-2.5 dB for low resolution, while the performance loss is marginal for high resolution.

Neuron axon analysis is a fundamental mean to investigate nervous system in neurobiology and often requires collecting a great amount of statistical information. Automated extraction of axons axis in 3D microscopic images posts a key problem in the field of neuron axon analysis. To address tortuous axons with several touchings in 3D volumes, a novel simulation based approach with regard to volume modeling and scenario mapping is presented. Given the simulation model, the properties of dataset and the nature of axon (e.g., continuity) are converted to physics laws including the principle of minimum potential energy, law of buoyancy, and law of inertia. Experimental results on axon volumes show the proposed algorithm can effectively handle axon touching problem and actually extract multiple axon axes.

A novel cell nucleus tracking approach is raised in this paper, developed from applied graph theory that simultaneously handles major challenges of cells touching, splitting, disappearing and emerging. The key point of our work lies in the graphs with robust structures that constructed from large cells groups. Then the problem of nucleus tracking is simplified to that of vertex matching between graphs generated from successive frames. Single cells are modeled as vertices in the graph and edges are built between neighboring ones, through which cells in large quantities are connected altogether. The algorithm works through a process of graph saturation following vertices with maximal entering belief-propagation flows. Feasibility of our approach is validated by so-called 'Crowds Model' that explains motional features of multi-cells groups. In processing CT image sources with high cell intensities as well as complex variations in structures, robustness of our approach is confirmed in that both successful tracking rates and segmentation rates are kept above 96%.

In the case of delivering real-time video over the 3G cellular networks, burst frame losses may be inevitable and unpredictable, which may cause severe quality degradation. Based on cross-layer frame discarding (CLFD), this paper proposes an enhanced error-resilient video coding scheme for cellular video communication. By using unequal retransmission at the radio link (RL) layer, a base station can provide reliable transmission for the relatively important frames in one video sequence. Relying on the unequal protection at the RL layer, the encoder at the application (APP) layer can actively discard a certain number of frames according to the received acknowledgement messages. Thus, unpredictable burst frame losses during transmission can be transformed into selective frame discarding at the encoder. Experiments results show that the proposed scheme can enhance the error resilience of the cellular video communication significantly.

This paper proposes an image compression approach, in which we incorporate primal sketch based learning into the mainstream image compression framework. The key idea of our approach is to use primal sketch information to enhance the quality of distorted images. With this method, we only encode the down-sampled image and use the primal sketch based learning to recover the high frequency information which has been removed by down-sampling. Experimental results demonstrate that our scheme achieves better objective visual quality as well as subjective quality compared with JPEG2000 at the same bit-rates.

Neuron axon analysis is an important means to investigate disease mechanisms and signaling pathways in neurobiology and often requires collecting a great amount of statistical information and phenomena. Automated extraction of axons in 3D microscopic images posts a key problem in the field of neuron axon analysis. To address tortuous axons in 3D volumes, a self-adjust region growing approach referring to surface modeling and self-adjustment which takes advantage of the nature of axon (e.g., continuity), is presented. Experimental results on axon volumes show that the proposed scheme provides a reliable solution to axon retrieving and overcomes several common drawbacks from other existing methods.

In this paper, we propose an error recovery scheme to cope with the frame loss problem in large end-to-end delay scenario. Because Wyner-Ziv coding can produce deterministic output with nondeterministic inputs, we adopt the decoder's error-corrupted reference picture as side information to exploit its correlation with source picture, and thus improve the coding efficiency. To prevent retransmission request, we propose a feasible encoder-driven rate estimation scheme by only storing MVs and part of the error pattern at encoder. Furthermore, a new Laplacian parameter computing method is proposed based on discrete Laplacian PDF. The experimental results show that the proposed estimation schemes have quite high precision, and the error recovery scheme outperforms INTRA refresh scheme up to 5 dB.

Various error control schemes have been addressed in wireless video stream transmission. By combining an adaptive binary arithmetic coding technique with context modeling, CABAC as a normative part of H.264/AVC has achieved a high degree of adaptation and redundancy reduction. However, error propagation still remains a problem because of the property of arithmetic coding. The presented scheme compares the various error detection methods, and proposes an efficient error detection technique based on CABAC semantics, which is achieved by inserting detective markers denoting by syntax elements. The misdetection probability versus stream size expansion can be easily handled. In addition, placements of markers can vary with regard to specific video content, thus efficiency within this scheme is enhanced. Comparison with other detection scheme is also presented

A new postprocessing method based on adjusted contourlet transform is introduced in this paper for suppressing blocking artifacts (BA) in block-based discrete cosine transform (BDCT) compressed images. To our best knowledge, this is the first time contourlet is applied to this field. By exploiting scale space edge detector (ss-edge detector), our algorithm can extract and protect blocking map (BM) and edge map (EM) in the compressed image respectively in the same time. By transforming the compressed image into adjusted contourlet domain, the adaptive thresholds are obtained according to BM. According to the adaptive thresholds, the contourlet coefficients in different subbands are filtered. Experimental results show that our deblocking algorithm achieves better performance than the other iterative and noniterative methods reported in the literature

This paper carves out a way to image compression that is motivated by the recent advancement in image inpainting. An image coding approach is proposed in which a number of regions of the input image are skipped at the encoder and are recovered through the inpainting process at the decoder. Furthermore, a structure-aware inpainting (SAI) method is developed to restore the skipped structural regions by taking advantage of the available portion of the decoded image. A binary structure map is extracted and compressed into the generated bit-stream to indicate the skipped regions with salient structures. By making use of the decoded texture information together with the structure map, the SAI method can recover the skipped structural regions as well as the non-structural ones effectively at the decoder. Compared with JPEG, our proposed image compression scheme allows smaller file, with the potential of up to 50% bit-saving capability, at similar visual quality levels

A major challenge in video encoding and transmission over wireless networks is that the channel is error-prone while the compressed video data is highly sensitive to errors. The transmission errors will cause decoding failure at the receiver side. More importantly, the transmission errors introduced in one video frame will propagate to its subsequent frames along the motion prediction path and significantly degrade video presentation quality. This type of picture distortion is called transmission distortion. In this work, we propose a control system approach to transmission distortion modeling. More specifically, we consider the wireless video transmission and decoding system as a linear system with the transmission errors as system input and the transmission distortion as output. We study the fading behavior of the impulse transmission distortion. We analyze the low-pass filtering behavior of video encoders and develop a scheme to estimate the instantaneous transmission distortion. Our extensive simulation results demonstrate that the proposed transmission distortion model is accurate and robust. In addition, it has very low computational complexity. More importantly, because it is a predictive model, it allows the encoder to predict the transmission distortion even before the video is compressed and transmitted. This type of transmission distortion model has important applications in resource allocation and performance optimization in real-time wireless video communication

In video transmission over wireless networks, the channel is error-prone and the compressed video data is highly sensitive to errors. The transmission errors causes decoding failure and distort the pictures. This type of picture distortion is called transmission distortion. In this work, we develop a predictive approach for transmission modeling so that the encoder is able to estimate the average transmission distortion of the decoded video data. Our simulation results demonstrate that the proposed model is robust and accurate. Using this type of predictive transmission distortion model, the encoder is able to intelligently allocate its resources, such as bits and energy, to maximize the decoded video quality at receiver side.

A lifting implementation of the discrete wavelet transform applied along motion trajectories has recently gained a lot of attention in the video community as strong candidates in incoming scalable video coders. We generalize the coding scheme for classical lifting-based motion compensation temporal filtering and permit the codec to choose adaptively between the original reference frames and new fading-compensated reference frames to predict residuals while maintaining the invertibility of the inter-frame transform. Experimental results show that the proposed algorithm not only significantly improves subjective visual quality of the temporal low-pass frames, but also has 0.15-0.3 dB gain in PSNR performance compared with the normal (5, 3) lifting schemes.

The objective of joint rate control is to dynamically distribute the channel capacity among video sequences according to their respective complexities, thus a more uniform picture quality and a more efficient utilization of channel capacity are achieved. Most existing approaches are based on MPEG2 coding platform. This paper presents a novel joint rate control scheme for multiple video sequences coding based on H.264 standard. A novel complexity measure that adapts to the characteristics of H.264 video coding is proposed. Experimental results show that the proposed scheme maintains a good balance in picture quality among the sequences as well as within a sequence

This paper presents an improved update operator for H.264 scalable extension, which is a new joint standardization effort of ISO/IEC and ITU-T. Energy distributed update (EDU) is utilized to reduce decoder complexity. The relationship between EDU and the optimum update in B. Girod et al., (2005) is revealed. Moreover, for the constrained motion-compensated lifting structure employed in H.264 scalable extension, we derive a set of new update coefficients to improve coding performance based on distortion analysis. Experimental results show that the proposed update operator can improve coding efficiency with less complexity and meanwhile, frame-wise quality fluctuation is also reduced