Real-time video communication is critically hampered by packet loss. Building on the neural video codec (NVC) and exploiting the statistics of its latent representations, we propose a channel importance-aware redundancy framework that eliminates insignificant channels and reallocates high redundancy to protect important channels for loss-resilient video coding. We first develop a channel-wise importance evaluation method based on the in-distribution (IND) region length criterion [1] for latent representations of prediction residuals and motion vectors produced by NVCs. Consequently, we derive a heuristic that adapts the number of protected channels to the packet loss rate (PLR). More channels are retained at low PLRs to improve video quality, while only a few critical channels are kept and assigned higher redundancy for enhanced robustness at high PLRs. Experimental results show that the proposed method provides more graceful quality degradation and maintains significant performance advantages, particularly under high PLRs, as illustrated in Figure 1.

Despite strong generalization ability, large language models (LLMs) are resource-intensive to deploy. Post-training quantization (PTQ) effectively compresses pretrained LLMs without retraining. Although recent studies target extremely low-bit quantization, most methods rely on extra parameters and the effective bit-width usually exceeds 2 bits. We introduce a saliency-guided mixed-precision PTQ framework that achieves an average bit-width of 1.70 bits while maintaining competitive performance. The proposed method allocates bit-widths across weight groups under a strict total budget, guided by a cross-precision saliency metric. The key innovation is a learnable affine transformation consisting of rotation, scaling, and shifting to align quantization grids with local weight distributions to minimize error. Experiments show that the proposed method matches state-of-the-art low-bit PTQ performance, while reducing the average bit-width by approximately 0.55 bits in Table 1.

Task-adapted compressed sensing magnetic resonance imaging (CS-MRI) is emerging to address the specific demands of downstream clinical tasks with significantly fewer k-space measurements than required by Nyquist sampling. However, existing task-adapted CS-MRI methods suffer from the uncertainty problem for medical diagnosis and cannot achieve adaptive sampling in end-to-end optimization with reconstruction or clinical tasks. To address these limitations, we propose the first task-adapted CS-MRI from the information-theoretic perspective to simultaneously achieve probabilistic inference for uncertainty prediction and adapt to arbitrary sampling ratios and versatile clinical applications. Specifically, we formalize the task-adapted CS-MRI optimization problem by maximizing the mutual information between undersampled k-space measurements and clinical tasks to enable probabilistic inference for addressing the uncertainty problem. We leverage amortized optimization and construct tractable variational bounds for mutual information to jointly optimize sampling, reconstruction, and task-inference models, which enables flexible sampling ratio control using a single end-to-end trained model. Furthermore, the proposed framework addresses two kinds of distinct clinical scenarios within a unified approach, i.e., i) joint task and reconstruction, where reconstruction serves as an auxiliary process to enhance task performance; and ii) task implementation with suppressed reconstruction, applicable for privacy protection. Extensive experiments on large-scale MRI datasets demonstrate that the proposed framework achieves highly competitive performance on standard metrics like Dice compared to deterministic counterpart but provides better distribution matching to the ground-truth posterior distribution as measured by the generalized energy distance (GED).

Pooling and unpooling are indispensable in constructing hierarchical spherical convolutional neural networks (HS-CNNs). Most existing models employ simple downsampling-based pooling, which ignores the sampling theorem and cannot adapt to different spherical signals (with different spectra) and tasks (dependent on different frequency components), thus suffering a significant information loss. Besides, signals reconstructed by the widely-adopted padding-based unpooling may also change unwantedly the spectra of original signals. To address these, we propose a novel framework of HS-CNNs with lifting structures to learn adaptive spherical wavelets for pooling and unpooling, named LiftHS-CNNs. Specifically, we learn spherical wavelets with a lifting structure to adaptively partition the input signal into low- and high-frequency sub-bands, with the down-scaled representations for pooling generated to preserve more information in the low-frequency sub-band. The lifting structure consists of learnable update and predict operators parameterized with graph attention to jointly consider the signal's characteristics and underlying geometries. We then propose an unpooling operation invertible to the lifting-based pooling for restoring the up-scaled representations, which can well preserve spectral characteristics of the original signal. Particular properties (i.e., spatial locality, vanishing moments, and stability) of the learned wavelets and the information preserving ability of the proposed pooling and unpooling are further studied. Experiments on benchmark spherical datasets for a wide range of tasks verify the superiority of our LiftHS-CNNs.

Adaptive bitrate (ABR) streaming is a popular technique used to improve the quality of experience (QoE) for users who watch videos online, which, for example, can provide a smoother video playback by dynamically adjusting the requested video quality with associated bitrate according to the constrained yet diverse network conditions. Recently, learning-based ABR algorithms have achieved a notable performance gain with lower inference overhead than the conventional heuristic or model-based baselines. However, their performance may degrade significantly in an unseen network environment with time-varying and heterogeneous throughput dynamics. For a better generalization, in this paper, we propose a meta-reinforcement learning (meta-RL)-based neural ABR algorithm that is able to quickly adapt its policy to these unseen throughput dynamics. Specifically, we propose a model-free system framework comprising an inference network and a policy network. The inference network infers distribution of the latent representation for underlying dynamics based on the recent throughout context, while the policy network is trained to quickly adapt to the changing throughout dynamics with the sampled latent representation. To effectively learn the inference network and meta-policy on mixed dynamics of the practical ABR scenarios, we further design a variational information bottleneck theory-based loss function for training the inference and policy networks, whose objective is to strike a trade-off between brevity of the latent representation and expressiveness of the meta-policy. We also derive a theoretically necessary condition for the bitrate versions that yield higher long-term QoE, based on which a dynamic action pruning strategy is further developed for practical implementation. This pruning strategy can not only prevent unsafe policy outputs in midst of unseen throughput dynamics, but may also reduce the computational complexity of model-based ABR algorithms. Finally, the meta-training and meta-adaptation procedures of our proposed algorithm are implemented across a range of throughput dynamics. The empirical evaluations on various datasets containing real-world network traces verify that our algorithm surpasses the state-of-the-art ABR algorithms, particularly in terms of the average chunk QoE and fast adaptation across out-of-distribution throughput traces.

Point cloud attribute compression is challenged by fitting the attribute signals living on irregular geometric structures. Existing methods cannot achieve compact multiscale representation for high-fidelity reconstruction using the handcrafted transforms or deep learning-based techniques. In this paper, we propose a novel geometry-aware lifting-based multiscale network via spatial-channel lifting scheme for point cloud attribute compression. The proposed network cascades geometry-aware spatial lifting to reduce spatial redundancy by adaptively capturing irregular geometric structures and progressive channel lifting to progressively reduce channel-wise redundancy in multiscale representation. Furthermore, we design the split, predict, and update operations for geometry-aware spatial lifting to fully exploit the geometry information representing irregular structures. We develop geometry-aware adaptive split to equally split input points with significance scores indicating their dependencies, and propose geometry-aware cross-attention filtering for the predict and update operations for decorrelation based on geometry information. To our best knowledge, this paper achieves the first lifting-based learned transform for point cloud compression that enjoys reversibility guarantees of multiscale representation to enhance rate-distortion performance. Experimental results show that the proposed framework achieves state-of-the-art performance on extensive point cloud datasets, and outperforms latest MPEG G-PCC standard and most recent deep learning based methods.

Sparse representation via dedicated dictionary learning facilitates image restoration by simultaneously preserving significant representation and removing trivial perturbations. However, existing dictionary learning under strict ℓ0-norm based sparsity constraint lacks of convergence guarantees and suffers from the dilemma between block artifacts and degraded quality in restoring fine details, and could hamper the consequent deep unfolded networks (DUNs). To address these problems, in this paper, we propose the first ℓ0-regularized dictionary learning with a theoretical guarantee of global convergence that achieves globally consistent sparse approximation for image restoration. Specifically, we first formulate an ℓ0-norm based optimization for dictionary learning that simultaneously achieves sparse approximation of local patches and ensures global consistency between the recovered and observed images. Consequently, we develop ℓ0-regularized global-local dictionary learning (ℓ0-GLDL) that alternately achieves sparse code pursuit, local dictionary update, and global image reconstruction based on alternating proximal linearized method. We demonstrate in theory that ℓ0-GLDL converges globally to a critical point. Furthermore, we design a deep unfolded network by unrolling ℓ0 -GLDL into diverse subblocks to strictly realize alternating optimization for stable image restoration. Experimental results demonstrate that the proposed methods achieve state-of-the-art performance and outperform existing deep learning based methods in image restoration tasks such as image denoising, image deblurring, and image inpainting.

Image denoising remains a challenging problem due to the ill-posed nature of recovering clean images from noisy observations. Prior-based modeling plays a vital role in addressing this challenge. Low-rank priors effectively exploit non-local self-similarity by modeling correlations across similar images, but often over-smooth fine textures. In contrast, sparse priors, especially in the form of Convolutional Sparse Coding (CSC), excel at preserving high-frequency texture details, yet typically neglect the structure across images. Recent unfolded CSC networks have improved denoising performance by combining CSC with deep networks, but most of them rely solely on sparsity, overlooking the complementary low-rank structure information. To overcome these limitations, we propose UCSC-LR, the first interpretable unfolded CSC network that jointly leverages both sparse and low-rank priors within a unified architecture. UCSC-LR unrolls both the Iterative Shrinkage-Thresholding Algorithm (ISTA) and Singular Value Thresholding (SVT) to strictly implement alternating minimization, enabling simultaneous pursuit of sparse representations and low-rank reconstructions. A dedicated Fusion Net is introduced to adaptively integrate features from the two priors, allowing the model to preserve both texture and structural content. To further enhance performance on color image denoising, UCSC-LR incorporates a lightweight channel attention mechanism to capture inter-channel dependencies. With shared, pre-learned convolutional dictionaries and efficient parameterization, UCSC-LR achieves state-of-the-art performance with low model complexity. Extensive experiments on grayscale, blind, and color denoising tasks validate the effectiveness of the proposed method, consistently surpassing existing CSC-based and deep learning-based denoisers.

Existing 2D-to-3D human pose estimation (HPE) methods struggle with the occlusion issue by enriching information like temporal and visual cues in the lifting stage. In this paper, we argue that these methods ignore the limitation of the sparse skeleton 2D input representation, which fundamentally restricts the 2D-to-3D lifting and worsens the occlusion issue. To address these, we propose a novel two-stage generative densification method, named Hierarchical Pose AutoRegressive Transformer (HiPART), to generate hierarchical 2D dense poses from the original sparse 2D pose. Specifically, we first develop a multi-scale skeleton tokenization module to quantize the highly dense 2D pose into hierarchical tokens and propose a Skeleton-aware Alignment to strengthen token connections. We then develop a Hierarchical AutoRegressive Modeling scheme for hierarchical 2D pose generation. With generated hierarchical poses as inputs for 2D-to-3D lifting, the proposed method shows strong robustness in occluded scenarios and achieves state-of-the-art performance on the single-frame-based 3D HPE. Moreover, it outperforms numerous multi-frame methods while reducing parameter and computational complexity and can also complement them to further enhance performance and robustness.

Controllability of video generation has been recently concerned in addition to the quality of generated videos. The main challenge to controllable video generation is to synthesize videos based on user-specified instance spatial locations and movement trajectories. However, existing methods suffer from a dilemma between the resource consumption, generation quality, and user controllability. As an efficient alternative to prohibitive training-based video generation, existing zero-shot video generation methods cannot generate high-quality and motion-consistent videos under the control of layouts and movement trajectories. In this paper, we propose a novel zero-shot method named IM-Zero that ameliorates instance-level motion controllable video generation with enhanced control accuracy, motion consistency, and richness of details to address this problem. Specifically, we first present a motion generation stage that extracts motion and textural guidance from keyframe candidates from pre-trained grounded text-to-image model to generate the desired coarse motion video. Subsequently, we develop a video refinement stage that injects the motion priors of pre-trained text-to-video models and detail priors of pre-trained text-to-image models into the latents of coarse motion videos to further enhance video motion consistency and richness of details. To our best knowledge, IM-Zero is the first to simultaneously achieve high-quality video generation and allow to control both layouts and movement trajectories in a zero-shot manner. Extensive experiments demonstrate that IM-Zero outperforms existing methods in terms of video quality, inter-frame consistency, and the alignment of location and trajectory. Furthermore, compared with existing methods, IM-Zero enjoys extra advantages of versatility in video generation, including motion control of subparts within instances, finer control of specifying instance shapes via masks, and more difficult tasks of motion transfer for customizing fine-grained motion patterns through reference videos and high-quality text-to-video generation.

Autofocus is a crucial component of modern digital cameras. While recent learning-based methods achieve state-of-the-art in focus prediction accuracy, they unfortunately ignore the potential focus hunting phenomenon of back-and-forth lens movement in the multi-step focusing procedure. To address this, in this paper, we propose an expert regularized deep reinforcement learning (DRL)-based approach for autofocus, which can utilize the sequential information of lens movement trajectory to both enhance the multi-step in-focus prediction accuracy and reduce the chance of focus hunting. Our method generally follows an actor-critic framework. To accelerate the DRL’s training with a higher sample efficiency, we initialize the policy with a pre-trained single-step prediction network, where the network is further improved by modifying the output of absolute in-focus position distribution to the relative lens movement distribution to establish a better mapping between input images and lens movement. To further stabilize DRL’s training with a lower occurrence of focus hunting in the resulting lens movement trajectory, we generate some offline trajectories based on prior knowledge to avoid focus hunting, which are then leveraged as an offline dataset of expert trajectories to regularize the actor network’s training. Empirical evaluations show that our method outperforms those learning-based methods on public benchmarks, with higher single- and multi-step prediction accuracy, and a significant reduction of focus hunting rate.

Parameter-efficient transfer learning (PETL) has emerged as a promising solution to adapt large-scale pre-trained models to downstream tasks. Nevertheless, these methods have not thoroughly explored the characteristics of PETL methods to optimize the fine-tuning performance with miminal volume of parameters. In this paper, we first reveal that, compared to pre-trained models, PETL tends to generate similar features via homogeneous feature transformations across different layers. Subsequently, we propose a Global Share Local Transform Mixture-of-Experts framework, namely GLEAM, that decomposes the adapter into a shared component and layer-specific local components to simultaneously reduce the redundancy in layer-wise parameter matrices for homogeneous feature transformations and fine-tune the locally specific parameters for minimizing performance loss. Specifically, we develop a shared mixture of convolution that introduces shared multi-scale sparse MoE to enable diverse transformations for suppressing the homogeneity issue of feature transformations in PETL. GLEAM is evaluated on more than 20 datasets for image classification and few-shot learning. Extensive experimental results demonstrate that it achieves comparable performance with existing PETL methods like LoRA with only 3% of its parameters and further yields competitive performance using only 0.07M parameters.

Generative diffusion models have shown empirical successes in point cloud resampling, generating a denser and more uniform distribution of points from sparse or noisy 3D point clouds by progressively refining noise into structure. However, existing diffusion models employ manually predefined schemes, which often fail to recover the underlying point cloud structure due to the rigid and disruptive nature of the geometric degradation. To address this issue, we propose a novel learnable heat diffusion framework for point cloud resampling, which directly parameterizes the marginal distribution for the forward process by learning the adaptive heat diffusion schedules and local filtering scales of the time-varying heat kernel, and consequently, generates an adaptive conditional prior for the reverse process. Unlike previous diffusion models with a fixed prior, the adaptive conditional prior selectively preserves geometric features of the point cloud by minimizing a refined variational lower bound, guiding the points to evolve towards the underlying surface during the reverse process. Extensive experimental results demonstrate that the proposed point cloud resampling achieves state-of-the-art performance in representative reconstruction tasks including point cloud denoising and upsampling.

Learned image compression (LIC) has demonstrated superior rate-distortion (R-D) performance compared to traditional codecs, but is challenged by training inefficiency that could incur more than two weeks to train a state-of-the-art model from scratch. Existing LIC methods overlook the slow convergence caused by compacting energy in learning nonlinear transforms. In this paper, we first reveal that such energy compaction consists of two components, \emph{i.e.}, feature decorrelation and uneven energy modulation. On such basis, we propose a linear auxiliary transform (AuxT) to disentangle energy compaction in training nonlinear transforms. The proposed AuxT obtains coarse approximation to achieve efficient energy compaction such that distribution fitting with the nonlinear transforms can be simplified to fine details. We then develop wavelet-based linear shortcuts (WLSs) for AuxT that leverages wavelet-based downsampling and orthogonal linear projection for feature decorrelation and subband-aware scaling for uneven energy modulation. AuxT is lightweight and plug-and-play to be integrated into diverse LIC models to address the slow convergence issue. Experimental results demonstrate that the proposed approach can accelerate training of LIC models by 2 times and simultaneously achieves an average 1\% BD-rate reduction. To our best knowledge, this is one of the first successful attempt that can significantly improve the convergence of LIC with comparable or superior rate-distortion performance.

In real-time communication (RTC) systems, a congestion control (CC) is indispensable to guarantee the end user’s quality of experience (QoE), the key of which is to predict the bottleneck link capacity and thus set the target bitrate for the sender. Existing deep reinforcement learning (DRL)-based bandwidth prediction methods, though improving the bandwidth prediction performance based on the pre-collected offline dataset, are usually designed for a single RTC scenario with poor generalization ability to other scenarios. What’s worse, the policy learned by these DRL methods can hardly capture the complex multi-modal distribution of underlying optimal policy. To address this, in this paper, we propose an offline DRL-based meta-diffusion policy, where we pre-train an encoder to extract the effective feature embedding of network dynamics, and further learn an expressive meta-policy based on the diffusion model to predict the bandwidth conditioned on the observed network states and extracted network dynamics features. Experiments demonstrate the enhanced accuracy and generalization of our meta-diffusion policy over existing landmark schemes, with a performance gain of 7.84% and 27.12%, in terms of the bandwidth prediction error rate and overestimate rate, respectively.

Federated learning (FL) is greatly challenged by the communication bottleneck and computation limitation on clients. Existing methods based on quantization for FL cannot simultaneously reduce the uplink and downlink communication cost and mitigate the computation burden on clients. To address this problem, in this paper, we propose the first low-bit integerized federated learning (LBI-FL) framework that quantizes the weights, activations, and gradients to lower than INT8 precision to evidently reduce the communication and computational costs. Specifically, we achieve dynamical temporal bit-width allocation for weights, activations, and gradients along the training trajectory via reinforcement learning. An agent is trained to determine bit-width allocation by comprehensively considering the states like current bit-width, training stage, and quantization loss as the state. The agent efficiently trained on small-scale datasets can be well generalized to train varying network architectures on non-independent and identically distributed datasets. Furthermore, we demonstrated in theory that federated learning with gradient quantization achieves an equivalent convergence rate to FedAvg. The proposed LBI-FL can reduce the communication costs by 8 times compared to full-precision FL. Extensive experiments show that the proposed LBI-FL achieves a reduction of more than 50\% BitOPs per client on average for FL with less than 2\% accuracy loss compared to low-bit training with INT8 precision.

The significant communication overhead and client data heterogeneity have posed an important challenge to current federated learning (FL) paradigm. Existing compression-based and optimization-based FL algorithms typically focus on addressing either the model compression challenge or the data heterogeneity issue individually, rather than tackling both of them. In this paper, we observe that by symbolizing the client model updates to be uploaded (i.e., normalizing the magnitude for each model parameter at local clients), the model heterogeneity, essentially stemmed from data heterogeneity, can be mitigated, and thereby helping improve the overall generalization performance of the globally aggregated model at the server. Inspired with this observation, and further motivated by the success of Lion optimizer in achieving the optimal performance on most tasks in the centralized learning, we propose a new FL algorithm, called FedSMU, which simultaneously reduces the communication overhead and alleviates the data heterogeneity issue. Specifically, FedSMU splits the standard Lion optimizer into the local updates and global execution, where only the symbol of client model updates commutes between the client and server. We theoretically prove the convergence of FedSMU for the general non-convex settings. Through extensive experimental evaluations on several benchmark datasets, we demonstrate that our FedSMU algorithm not only reduces the communication overhead, but also achieves a better generalization performance than the other compression-based and optimization-based baselines.

We propose Noise Conditional Variational Score Distillation (NCVSD), a novel method for distilling pretrained diffusion models into generative denoisers. We achieve this by revealing that the unconditional score function implicitly characterizes the score function of denoising posterior distributions. By integrating this insight into the Variational Score Distillation (VSD) framework, we enable scalable learning of generative denoisers capable of approximating samples from the denoising posterior distribution across a wide range of noise levels. The proposed generative denoisers exhibit desirable properties that allow fast generation while preserve the benefit of iterative refinement: (1) fast one-step generation through sampling from pure Gaussian noise at high noise levels; (2) improved sample quality by scaling the test-time compute with multi-step sampling; and (3) zero-shot probabilistic inference for flexible and controllable sampling. We evaluate NCVSD through extensive experiments, including class-conditional image generation and inverse problem solving. By scaling the test-time compute, our method outperforms teacher diffusion models and is on par with consistency models of larger sizes. Additionally, with significantly fewer NFEs than diffusion-based methods, we achieve record-breaking LPIPS on inverse problems.

Fourier phase retrieval (FPR) is a longstanding ill-posed non-convex inverse problem suffering from failure in reconstruction. Existing methods usually mitigate the issue in a trial-and-error fashion and cannot identify the failure to flexibly adjust the FPR process. In this paper, we propose FA-FPR, a novel failure aware method that guides FPR with diffusion priors by failure detection. We first discover that unsuccessful reconstructions can be identified as fake images due to their distinct structures from true signals, and then leverage a plug-and-play fake image detector to automatically determine the approximation to true signals. Furthermore, we develop a hybrid decoupling scheme that adaptively combines Gaussian noise and predicted noise by score function based on the failure detection. Extensive experiments under various oversampling ratios with noisy and noiseless measurements demonstrate that FA-FPR requires less than 50% number of repeated experiments on average to achieve the state-of-the-art performance.

Source-free domain adaptation (SFDA) is a challenging task that tackles domain shifts using only a pre-trained source model and unlabeled target data. Existing SFDA methods are restricted by the fundamental limitation of source-target domain discrepancy. Non-generation SFDA methods suffer from unreliable pseudo-labels in challenging scenarios with large domain discrepancies, while generation-based SFDA methods are evidently degraded due to enlarged domain discrepancies in creating pseudo-source data. To address this limitation, we propose a novel generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) that leverages unlabeled target data as references to reliably generate and progressively refine a pseudo-target domain for SFDA. Specifically, we divide the target samples into a trust set and a non-trust set based on the reliability of pseudo-labels to sufficiently and reliably exploit their information. For samples from the non-trust set, we develop a manipulation strategy to semantically transform them into the newly assigned categories, while simultaneously maintaining them in the target distribution via a latent diffusion model. Furthermore, we design a progressive refinement mechanism that progressively reduces the domain discrepancy between the pseudo-target domain and the real target domain via iterative refinement. Experimental results demonstrate that DPTM outperforms existing methods by a large margin and achieves state-of-the-art performance on four prevailing SFDA benchmark datasets with different scales. Remarkably, DPTM can significantly enhance the performance by up to 18.6\% in scenarios with large source-target gaps.

In low-latency live streaming (LLLS), the adaptive bitrate algorithm plays a critical role in optimizing encoding bitrates to meet the strict end-to-end latency requirement, which is mostly several seconds or less. Existing methods usually rely on a fixed preset latency target, limiting their generalization capabilities across the heterogeneous scenarios. To address this, we propose an offline meta-reinforcement learning-based bitrate decision framework for LLLS, which incorporates the expertise of multiple state-of-the-art LLLS algorithms from their collected experience and constructs a universal bitrate selection policy via the offline training. Specifically, we establish an RL-based policy that adaptively adjusts the network throughput measurements and selects the encoding bitrates for LLLS based on the network conditions. The policy’s performance is enhanced by improving the robustness of short-term throughput estimations. To leverage the expertise of current LLLS algorithms, we collect their decision-making trajectories, and directly train the policy on these offline trajectories with implicit Q-learning. In addition, a meta-RL paradigm is further adopted to learn a universal policy that performs uniformly well across the heterogeneous network conditions and varying target latencies. Experimental results demonstrate that, compared to the other baselines, the proposed method achieves at least a 5.8% performance gain in terms of the overall quality of experience (QoE), across the diverse target-latency scenarios in the real-world throughput traces.

Progressive coding adapts to reliable image and video transmission over unstable network with fluctuating bandwidth with truncatable bitstreams produced by layer-wise conditional entropy coding. However, recent learned progressive image and video compression methods solely leverage table lookup or real-time computation for conditional entropy coding, and suffer from dramatically increased storage and computational overheads due to the expansion of codewords. To address this problem, in this paper, we propose an efficient conditional entropy coding scheme for learned progressive coding that incorporates the fast lookup-table and low-memory real-time computation based on statistical distribution of the codewords. Specifically, we build a smaller lookup table that stores only the symbols with high occurrence frequency and achieve real-time computation of those symbols with low occurrence frequency. Furthermore, we seamlessly integrate the proposed scheme into existing learned image and video compression frameworks for enhanced coding efficiency. Experimental results show that the proposed scheme accelerates both encoding and decoding processes for learned progressive image and video compression. Remarkably, we for the first time successfully extend the proposed scheme to learned video compression to demonstrate its feasibility.

Cross-layer feature fusion critically influences the performance of deep neural networks, where mainstream approaches like additive fusion of residual connections struggle to model high-order nonlinear interactions, limiting representational capacity, while concatenative fusion of dense connections incurs significant memory and computational overhead. Although recent attention-based feature fusion methods refine feature representations and strengthen hierarchical interactions, they often struggle to balance expressiveness and efficiency. To address these limitations, we propose the Chebyshev Fusion Unit (CFU), a lightweight yet effective cross-layer fusion mechanism. Specifically, CFU computes high-order Chebyshev polynomial terms between residual features and current-layer features, each of which is aggregated through learnable scalar weights, forming enhanced fusion features. This design enables explicit modeling of complex cross-layer dependencies with minimal additional parameters. Extensive experiments across various commonly used base models on image classification, medical image segmentation, and physical law learning tasks demonstrate its superior performance.

Existing methods for integerized training speed up deep learning by using low-bitwidth integerized weights, activations, gradients, and optimizer buffers. However, they overlook the issue of full-precision latent weights, which consume excessive memory to accumulate gradient-based updates for optimizing the integerized weights. In this paper, we propose the first latent weight quantization schema for general integerized training, which minimizes quantization perturbation to training process via residual quantization with optimized dual quantizer. We leverage residual quantization to eliminate the correlation between latent weight and integerized weight for suppressing quantization noise. We further propose dual quantizer with optimal nonuniform codebook to avoid frozen weight and ensure statistically unbiased training trajectory as full-precision latent weight. The codebook is optimized to minimize the disturbance on weight update under importance guidance and achieved with a three-segment polyline approximation for hardware-friendly implementation. Extensive experiments show that the proposed schema allows integerized training with lowest 4-bit latent weight for various architectures including ResNets, MobileNetV2, and Transformers, and yields negligible performance loss in image classification and text generation. Furthermore, we successfully fine-tune Large Language Models with up to 13 billion parameters on one single GPU using the proposed schema.

Federated learning (FL) commonly encourages the clients to perform multiple local updates before the global aggregation, thus avoiding frequent model exchanges and relieving the communication bottleneck between the server and clients. Though empirically effective, the negative impact of multiple local updates on the stability of FL is not thoroughly studied, which may result in a globally unstable and slow convergence. Based on sensitivity analysis, we define in this paper a local-update stability index for the general FL, as measured by the maximum inter-client model discrepancy after the multiple local updates that mainly stems from the data heterogeneity. It enables to determine how much the variation of client's models with multiple local updates may influence the global model, and can also be linked with the convergence and generalization. We theoretically derive the proposed local-update stability for current state-of-the-art FL methods, providing possible insight to understanding their motivation and limitation from a new perspective of stability. For example, naively executing the parallel acceleration locally at clients would harm the local-update stability. Motivated by this, we then propose a novel accelerated yet stabilized FL algorithm (named FedANAG) based on the server- and client-level Nesterov accelerated gradient (NAG). In FedANAG, the global and local momenta are elaborately designed and alternatively updated, while the stability of local update is enhanced with help of the global momentum. We prove the convergence of FedANAG for strongly convex, general convex and non-convex settings. We then conduct evaluations on both the synthetic and real-world datasets to first validate our proposed local-update stability. The results further show that across various data heterogeneity and client participation ratios, FedANAG not only accelerates the global convergence by reducing the required number of communication rounds to a target accuracy, but converges to an eventually higher accuracy.

End-to-end optimization via deep neural networks has facilitated lossy image compression. Existing neural network-based entropy models for end-to-end optimized image compression are limited by parameterized Gaussian distributions with deterministic mean and variance and cannot achieve accurate rate estimation for bottleneck representation with varying statistics. In this paper, we propose a novel entropy model based on deep Gaussian process regression (DGPR) to address this problem. Specifically, the proposed entropy model leverages autoregressive DGPR to flexibly predict the channel-wise posterior distributions of high-dimensional bottleneck representation for entropy coding. Consequently, we develop a well-established bit-rate estimation scheme via posterior inference of DGPR using the learned probabilistic distribution. Furthermore, scalable training is achieved via tensor train decomposition and Monte Carlo sampling to enable tractable variational inference of DGPR. To our best knowledge, this paper is the first attempt to develop the learnable probabilistic model for flexible parameter estimation in entropy modeling. Experimental results show that the proposed model outperforms conventional image compression methods (e.g., JPEG2000 and BPG) as well as recent end-to-end optimized methods on the Kodak and Tecnick datasets in terms of rate-distortion performance.

Entropy modeling is the core component of learned image compression (LIC) that models the distribution of latent representation learned from input images via neural networks for bit-rate estimation. However, existing entropy models employ presumed parameterized distributions such as Gaussian models and are limited for the learned latent representation characterized by complex distributions. To address this problem, in this paper, we for the first time achieve generative probabilistic entropy modeling of latent representation based on conditional diffusion models. Specifically, we propose a conditional diffusion-based probabilistic entropy model (CDPEM) to parameterize the latent representation with distributions of arbitrary forms that are generated by well designed training-test consistent denoising diffusion implicit model (TC-DDIM) without introducing any presumption. TC-DDIM is designed to leverage ancestral sampling to gradually approximate the distribution of latent representation with guaranteed consistency in generation for training and test. Furthermore, we develop a hierarchical spatial-channel context model to incorporate with TC-DDIM to sufficiently exploit spatial correlations with the approximate contextual information produced by ancestral sampling and channel-wise correlations using channel-wise information aggregation with reweighted training loss. Experimental results demonstrate that the proposed entropy model achieves state-of-the-art performance on the Kodak, CLIC, and Tecnick datasets compared to existing LIC methods. Remarkably, when incorporated with recent baselines, the proposed model outperforms latest VVC standard by an evident gain in R-D performance.

Variable-rate coding is challenging but indispensable for learned image compression (LIC) that is in nature characterized by nonlinear transform coding (NTC). Existing methods for variable-rate LIC are restricted by the non-smooth quantization process with zero gradients almost everywhere, and consequently, suffer from training-test gap and degraded rate-distortion (R-D) performance. To address this problem, in this paper, we propose sampling-based optimization for training NTC models along with non-uniform quantizers. Different from gradient-based optimization, the proposed sampling-based optimization first randomly samples the parameters from Gaussian distributions with progressively reduced variance and then selects the optimal parameters with a R-D indicator. On the basis of sampling-based optimization, we develop a learnable non-uniform dead-zone quantizer by adaptively refining the quantization steps for variable-rate coding with nonlinear transforms. Furthermore, we incorporate the learnable dead-zone quantizer to achieve a variable-rate LIC model with enhanced R-D performance and design rate and distortion control algorithms to adapt to dynamic network conditions. Experimental results show that the proposed method achieves state-of-the-art R-D performance in variable-rate image compression. It obtains an average 8.82% BD-rate reduction compared to latest versatile video compression (VVC) standard, and simultaneously achieves precise rate and distortion control with an average variation of 0.0087 bpp in bit-rates and 0.1265 dB in distortion on the Kodak dataset.

Image compression for both human and machine vision has become prevailing to accommodate to rising demands for machine-machine and human-machine communications. Scalable human-machine image compression is recently emerging as an efficient alternative to simultaneously achieve high accuracy for machine vision in the base layer and obtain high-fidelity reconstruction for human vision in the enhancement layer. However, existing methods achieve scalable coding with heuristic mechanisms, which cannot fully exploit the inter-layer correlations and evidently sacrifice rate-distortion performance. In this paper, we propose task-adapted learnable embedded quantization to address this problem in an analytically optimized fashion. We first reveal the relationship between the latent representations for machine and human vision and demonstrate that optimal representation for machine vision can be approximated with post-training optimization on the learned representation for human vision. On such basis, we propose task-adapted learnable embedded quantization that leverages learnable step predictor to adaptively determine the optimal quantization step for diverse machine vision tasks such that inter-layer correlations between representations for human and machine vision are sufficiently exploited using embedded quantization. Furthermore, we develop a human-machine scalable coding framework by incorporating the proposed embedded quantization into pre-trained learned image compression models. Experimental results demonstrate that the proposed framework achieves state-of-the-art performance on machine vision tasks like object detection, instance segmentation, and panoptic segmentation with negligible loss in rate-distortion performance for human vision.

Existing pretraining methods for semantic segmentation are hampered by the task gap between global image -level pretraining and local pixel-level finetuning. Joint dense-level pretraining is a promising alternative to exploit off-the-shelf annotations from diverse segmentation datasets but suffers from low-quality class embeddings and inconsistent data and supervision signals across multiple datasets by directly employing CLIP. To overcome these challenges, we propose a novelMulti-datasEt harmoNized pretraining framework forSemantic sEgmentation (MENSA). MENSA incorporates high-quality language embeddings and momentum-updated visual embeddings to effectively model the class relationships in the embedding space and thereby provide reliable supervision information for each category. To further adapt to multiple datasets, we achieve one-to-many pixel-embedding pairing with cross-dataset multi-label mapping through cross-modal information exchange to mitigate inconsistent supervision signals and introduce region-level and pixel-level cross-dataset mixing for varying data distribution. Experimental results demonstrate that MENSA is a powerful foundation segmentation model that consistently outperforms popular supervised or unsupervised ImageNet pretrained models for various benchmarks under standard fine-tuning. Furthermore, MENSA is shown to significantly benefit frozen-backbone fine-tuning and zero-shot learning by endowing pixel-level distinctiveness to learned representations.

In federated learning (FL), the heterogeneity of data and asynchronous participation of clients have been observed to induce the local client's model discrepancy with high variance, leading to a slow and unstable convergence globally at the server. In this article, motivated by the usefulness of stale client updates, we first propose a general framework, named FedVR, to address this issue. In FedVR, we design an aggregate of both fresh and stale local model updates without additional communication overhead, which is computed at the server as a control variate to reduce the client variance incurred by data heterogeneity and client sampling. In order to further reduce the model discrepancy between local clients, we therefore propose FedMDVR, which broadcasts the designed control variate to all the active clients to help correct their local update directions toward the global optimum, i.e., stationary point of the global objective function. While in the global update at server, the client variance is also decreased as inherited from the variance reduction nature of FedVR. We theoretically prove the convergence of FedVR and FedMDVR in the general nonconvex settings. Through extensive experimental evaluations on several benchmark datasets, we also demonstrate that our proposed FedVR and FedMDVR not only accelerate the convergence by reducing the number of communication rounds required to achieve a certain target accuracy, but more importantly, can converge to a higher accuracy than the baseline algorithms.

Network planning is crucial to facilitate network service under limited network operation costs. However, adapting the network topology (i.e., connections and capacities for physical and IP links) to time-varying and stochastic network states is challenging due to high computational overhead of problem solving. Existing deep learning based methods suffer from prohibitive computational complexity due to simultaneously solving IP link capacities and allocated traffics or iteratively exploring search space with excessive samples via heuristic rules. In this paper, we propose GDPlan, the first generative framework that leverages conditional graph diffusion model to address this challenge. To achieve high solving efficiency, GDPlan decouples network planning into two separate stages, i.e., generation of diverse high-quality solutions to IP link capacity and refinement of the solutions under feasibility constraints. Specifically, we develop generation-oriented graph signal modeling that reformulates solving IP link capacities as generation of graph topology conditioned on the graph signals related to traffic demands, physical links and network costs. Consequently, energy-guided controllable graph generation is achieved to capture significant structural patterns of graph topology with score-based graph diffusion model and produce diverse solutions with a guarantee of feasibility under specific objectives or constraints. GDPlan successfully achieves graph generation based solution to network planning with an order of magnitude higher solving speed than the commercial solver Gurobi. Experimental results demonstrate that, compared with Gurobi, the proposed GDPlan obtains lowest 3.7% average gap with only about 7.5% average running time.

Learned image compression (LIC) has recently emerged as a promising alternative to realize transform-based coding. Contrary to widely studied transform and entropy modeling in LIC, most LIC methods are limited to uniform scalar quantization and existing vector quantization (VQ) methods are restricted by the soft-to-hard quantization strategy. In this paper, we propose a novel entropy relaxed lattice vector quantization (ERLVQ) scheme for LIC. ERLVQ achieves a tractable dimension-wise approximation to the probability mass function for the complex Voronoi regions in a general class of lattices. Instead of directly performing probabilistic estimation on the lattices, ERLVQ equivalently aligns complex Voronoi regions with a set of complete and non-overlapping hypercubes in a projected parametric domain. Space covering technique is then developed to generating hypercubes with their centers fixed to integer projected lattice points and their volumes equaling the Voronoi regions to enable separable dimension-wise estimation of multi-variate probability mass functions. The proposed ERLVQ is the first to adapt to arbitrary lattices for VQ based on LIC and can be embedded into existing LIC methods. Experimental results demonstrate that the proposed ERLVQ is superior to the uniform scalar quantization (USQ) and existing lattice VQ methods [1] on prevailing LIC baselines, as shown in Table 1 .

Network quantization is promising to alleviate the storage and deployment burden for deep neural networks. However, existing methods focus on directly quantizing the network parameters but ignore the correlations between the initialized parameters (generated by random seeds) and the optimized parameters (obtained when the network training converges). In this paper, we reveal the redundancy between the initialized and optimized parameters and propose a novel quantization scheme for network parameters with rescaled initialization. We develop a novel quantization scheme for quantizing the delta parameters, i.e. , the difference between the optimized parameters and the rescaled initialization, as shown in Figure 1 . The scale is learned to adapt the weight regularization in training by minimizing the rounding error and penalizing the long-tailed distribution of the delta parameter. Adaptive rounding is further presented to calibrate the rounding direction of the delta parameter via a learnable bias. The proposed method can achieve high-efficiency prediction of network parameters and consistent performance gains with a trivial overhead of single random seed. Extensive experimental results demonstrate its superior performance on various popular network architectures, as exemplified in Table 1 .

State-of-the-art learning-based stability control methods for nonlinear robotic systems suffer from the issue of reality gap, which stems from discrepancy of the system dynamics between training and target (test) environments. To mitigate this gap, we propose an adversarially robust neural Lyapunov control (ARNLC) method to improve the robustness and generalization capabilities for Lyapunov theory-based stability control. Specifically, inspired by adversarial learning, we introduce an adversary to simulate the dynamics discrepancy, which is learned through deep reinforcement learning to generate the worst-case perturbations during the controller’s training. By alternatively updating the controller to minimize the perturbed Lyapunov risk and the adversary to deviate the controller from its objective, the learned control policy enjoys a theoretical guarantee of stability. Empirical evaluations on five stability control tasks with the uniform and worst-case perturbations demonstrate that ARNLC not only accelerates the convergence to asymptotic stability, but can generalize better in the entire perturbation space.

Learned image compression (LIC) has gained traction as an effective solution for image storage and transmission in recent years. However, existing LIC methods are redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these challenges, we propose a novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LIC. The FAT block comprises frequency-decomposition window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Additionally, we introduce frequency-modulation feed-forward network (FMFFN) to adaptively modulate different frequency components, improving rate-distortion performance. Furthermore, we present a transformer-based channel-wise autoregressive (T-CA) model that effectively exploits channel dependencies. Experiments show that our method achieves state-of-the-art rate-distortion performance compared to existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 by 14.5\%, 15.1\%, 13.0\% in BD-rate on the Kodak, Tecnick, and CLIC datasets.

As two prominent strategies for representation learning, Contrastive Learning (CL) and Masked Image Modeling (MIM) have witnessed significant progress. Previous studies have demonstrated the advantages of each approach in specific scenarios. CL, resembling supervised pre-training, excels at capturing longer-range global patterns and enhancing feature discrimination, while MIM is adept at introducing local and diverse attention across transformer layers. Considering the respective strengths, previous studies utilize feature distillation to inherit both discrimination and diversity. In this paper, we thoroughly examine previous feature distillation methods and observe that the increase in diversity mainly stems from asymmetric designs, which may in turn compromise the discrimination ability. To strike a balance between the two properties, we propose a simple yet effective strategy termed Hybrid Distill, which leverages both the CL and MIM teachers to jointly guide the student model. Hybrid Distill emulates the token relations of the MIM teacher at intermediate layers for diversity, while simultaneously distilling the final features of the CL teacher to enhance discrimination. A progressive redundant token masking strategy is employed to reduce the expenses associated with distillation and aid in preventing the model from converging to local optima. Experimental results demonstrate that Hybrid Distill achieves superior performance on various benchmark datasets.

Low-bit integer training emerges as a promising approach to mitigate the heavy burden during network training by quantizing the weights, activations, and gradients. However, existing methods cannot well achieve mixed-precision quantization for low-bit training and are commonly limited to INT8 precision. In this paper, we propose a novel low-bit integer training framework that, for the first time, achieves adaptive mixed-precision allocation (AMPA) for weights, activations, and gradients, and pushes the boundaries to a precision level below INT8. We develop a novel magnitude-based sensitivity measurement with regard to the quantization losses of weight, activation, and gradient quantization and the average gradient magnitudes, which is demonstrated as an upper bound of quantization influence in theory. We further design a layer-wise precision update strategy under observations on the quantization losses and their effects on model performance in low-bit training. Extensive experiments on different backbones and datasets show that, compared to INT8 quantization, the proposed method can achieve more than 38% BitOPs reduction with a tolerable loss below 2% in image classification, image segmentation, and language modeling.

Recent diffusion models provide a promising zero-shot solution to noisy linear inverse problems without retraining for specific inverse problems. In this paper, we reveal that recent methods can be uniformly interpreted as employing a Gaussian approximation with hand-crafted isotropic covariance for the intractable denoising posterior to approximate the conditional posterior mean. Inspired by this finding, we propose to improve recent methods by using more principled covariance determined by maximum likelihood estimation. To achieve posterior covariance optimization without retraining, we provide general plug-and-play solutions based on two approaches specifically designed for leveraging pre-trained models with and without reverse covariance. We further propose a scalable method for learning posterior covariance prediction based on representation with orthonormal basis. Experimental results demonstrate that the proposed methods significantly enhance reconstruction performance without requiring hyperparameter tuning.

Point cloud upsampling is crucial for 3D reconstruction, with recent research significantly benefitting from the advances in deep learning technologies. The majority of existing methods, which focus on a sequence of processes including feature extraction, augmentation, and the reconstruction of coordinates, encounter significant challenges in interpreting the geometric attributes they uncover, particularly with respect to the intricacies of transitioning feature dimensionality. In this paper, we delve deeper into modeling Partial Differential Equations (PDEs) specifically tailored for the inverse heat dissipation process in dense point clouds. Our goal is to detect gradients within the dense point cloud data distribution and refine the accuracy of interpolated points' positions along with their complex geometric nuances through a systematic iterative approximation method. Simultaneously, we adopt multivectors from geometric algebra as the primary tool for representing the geometric characteristics of point clouds, moving beyond the conventional vector space representations. The use of geometric products of multivectors enables us to capture the complex relationships between scalars, vectors, and their components more effectively. This methodology not only offers a robust framework for depicting the geometric features of point clouds but also enhances our modeling capabilities for inverse heat dissipation PDEs. Through both qualitative and quantitative assessments, we demonstrate that our results significantly outperform existing state-of-the-art techniques in terms of widely recognized point cloud evaluation metrics and 3D visual reconstruction fidelity.

Image compression for machine vision exhibits various rate-accuracy performance across different downstream tasks and content types. An efficient utilization of constrained network resource for achieving an optimal overall task performance has thus recently attracted a growing attention. In this paper, we propose Tombo, a task-oriented image compression and transmission framework that efficiently identifies the optimal encoding bitrate and routing scheme for multiple image bitstreams delivered simultaneously for different downstream tasks. Specifically, we study the characteristics of image rate-accuracy performance for different machine vision tasks, and formulate the task-oriented joint bitrate and routing optimization problem for multi-bitstreams as a multi-commodity network flow problem with the time-expanded network modeling. To ensure consistency between the encoding bitrate and routing optimization, we also propose an augmented network that incorporates the encoding bitrate variables into the routing variables. To improve computational efficiency, we further convert the original optimization problem to a multi-marginal optimal transport problem, and adopt a Sinkhorn iteration-based algorithm to quickly obtain the near-optimal solution. Finally, we adapt Tombo to efficiently deal with the dynamic network scenario where link capacities may fluctuate over time. Empirical evaluations on three typical machine vision tasks and four real-world network topologies demonstrate that Tombo achieves a comparable performance to the optimal one solved by the off-the-shelf solver Gurobi, with a 5x ~ 114× speedup.

Most of existing federated learning (FL) formulation is treated as a point-estimate of models, inherently prone to overfitting on scarce client-side data with overconfident decisions. Though Bayesian inference can alleviate this issue, a direct posterior inference at clients may result in biased local posterior estimates due to data heterogeneity, leading to a sub-optimal global posterior. From an information-theoretic perspective, we propose FedMDMI, a federated posterior inference framework based on model-data mutual information (MI). Specifically, a global model-data MI term is introduced as regularization to enforce the global model to learn essential information from the heterogeneous local data, alleviating the bias caused by data heterogeneity and hence enhancing generalization. To make this global MI tractable, we decompose it into local MI terms at the clients, converting the global objective with MI regularization into several locally optimizable objectives based on local data. For these local objectives, we further show that the optimal local posterior is a Gibbs posterior, which can be efficiently sampled with stochastic gradient Langevin dynamics methods. Finally, at the server, we approximate sampling from the global Gibbs posterior by simply averaging samples from the local posteriors. Theoretical analysis provides a generalization bound for FL w.r.t. the model-data MI, which, at different levels of regularization, represents a federated version of the bias-variance trade-off. Experimental results demonstrate a better generalization behavior with better calibrated uncertainty estimates of FedMDMI.

Diffusion Transformer (DiT) is emerging as a cutting-edge trend in the landscape of generative diffusion models for image generation. Recently, masked-reconstruction strategies have been considered to improve the efficiency and semantic consistency in training DiT but suffer from deficiency in contextual information extraction. In this paper, we provide a new insight to reveal that noisy-to-noisy masked-reconstruction harms sufficient utilization of contextual information. We further demonstrate the insight with theoretical analysis and empirical study on the mutual information between unmasked and masked patches. Guided by such insight, we propose a novel training paradigm named MC-DiT for fully learning contextual information via diffusion denoising at different noise variances with clean-to-clean mask-reconstruction. Moreover, to avoid model collapse, we design two complementary branches of DiT decoders for enhancing the use of noisy patches and mitigating excessive reliance on clean patches in reconstruction. Extensive experimental results on 256$\times$256 and 512$\times$512 image generation on the ImageNet dataset demonstrate that the proposed MC-DiT achieves state-of-the-art performance in unconditional and conditional image generation with enhanced convergence speed.

Autofocus (AF) is a fundamental task in digital image capturing, yet current approaches often exhibit a poor performance in terms of speed or accuracy. In this paper, we propose a deep reinforcement learning (DRL)-based framework for camera AF, by incorporating a pre-trained MobileNet-v2 encoder for image feature extraction and a Gaussian process regression (GPR) module for in-focus prediction. Specially, we formulate AF as a multi-step process, in which the lens movement trajectory is dynamically generated from a DRL-based policy. By leveraging image features extracted from the MobileNet-v2 encoder, the DRL-based policy can efficiently get close to the in-focus position, which enables the focus search in a fast speed, thus overcoming the drawback of slow focusing procedure suffered by traditional search-based AF methods. A GPR-based in-focus predictor is further designed to terminate the DRL’s exploration once it is able to provide a reliable prediction on the in-focus position, by exploiting the sharpness metric values of image patches captured at the DRL-generated trajectory of lens positions. In further contrast with the one-step prediction-based methods, we leverage the sequential information brought additionally by the multi-step exploration, enabling to gain a higher prediction accuracy. Experimental results demonstrate that our approach provides a significant improvement compared to both the search-based and prediction-based AF baselines.

By introducing randomness on the environments, domain randomization (DR) imposes diversity to the policy training of deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance incurred by trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing a low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. Based on our theory, we further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity for the practical implementation. By dividing the entire space of environments into some subspaces and then estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of variance reduction and faster convergence than the state-dependent baselines. Empirical evaluations on six robot control tasks with randomized dynamics demonstrate that VRDR not only accelerates the convergence of policy training, but can consistently achieve a better eventual policy with improved training stability.

3-D point clouds facilitate 3-D visual applications with detailed information of objects and scenes but bring about enormous challenges to design efficient compression technologies. The irregular signal statistics and high-order geometric structures of 3-D point clouds cannot be fully exploited by existing sparse representation and deep learning based point cloud attribute compression schemes and graph dictionary learning paradigms. In this paper, we propose a novel p-Laplacian embedding graph dictionary learning framework that jointly exploits the varying signal statistics and high-order geometric structures for 3-D point cloud attribute compression. The proposed framework formulates a nonconvex minimization constrained by p-Laplacian embedding regularization to learn a graph dictionary varying smoothly along the high-order geometric structures. An efficient alternating optimization paradigm is developed by harnessing ADMM to solve the nonconvex minimization. To our best knowledge, this paper proposes the first graph dictionary learning framework for point cloud compression. Furthermore, we devise an efficient layered compression scheme that integrates the proposed framework to exploit the correlations of 3-D point clouds in a structured fashion. Experimental results demonstrate that the proposed framework is superior to state-of-the-art transform-based methods in M-term approximation and point cloud attribute compression and outperforms recent MPEG G-PCC reference software.

Predicting the saliency map of a 360-degree video is the key for various downstream tasks, such as saliency-based compression and tile-based adaptive streaming. Besides static salient objects, the moving target will also contribute to the saliency map. Therefore, the joint exploitation of spherical spatio-temporal information is necessary for an accurate saliency prediction. The spherical spatial feature extraction, however, is hindered by the non-Euclidean geometric nature of spherical data, which imposes difficulty on direct extraction of the spatial features with traditional convolutional neural networks (CNNs). While the efficient exploitation of temporal correlation between these spherical spatial features remains another challenge, which requires the extraction of spherical optical flows for explicit motion information. To address these, in this paper, we first propose a spherical graph-based Farneback algorithm to extract the spherical optical flows directly in the sphere domain, by leveraging the GICOPix uniform sampling scheme. We then design a 3D separable graph convolutional network-based saliency prediction framework, named 360Spred, by taking both the spherical frames and spherical optical flows as input. The proposed 360Spred framework is based on the U-Net structure, with a 3D separable graph convolution (3DSGC) operator that directly extracts the visual and motion features in the sphere domain and exploits temporal correlation of both the high-level and low-level spatial features. Experimental results on two public datasets show that 360Spred can achieve a better performance than other baseline models in terms of the saliency prediction accuracy for 360-degree videos.

In adaptive video streaming, the design of an adaptive bitrate (ABR) strategy is critical for the quality-of-experience (QoE) perceived by users. Though current learning-based ABR algorithms achieve state-of-the-art performance for users with a given QoE metric setting for training, they may unfortunately suffer the poor generalization issue for other users with different QoE preferences. Besides, how to quantitatively characterize the distinct QoE preference for a user has also not been extensively studied yet. In this paper, we propose STEER, a successor feature-based transfer reinforcement learning framework for fast learning the ABR strategies on heterogeneous QoE preferences. Specifically, we first develop a QoE preference analysis scheme to infer the personal QoE preference of a single user based on the user's actual viewing history. We then formulate the personalized QoE maximization problem as a reinforcement learning (RL) task, which optimizes the ABR strategy to maximize the overall QoE perceived by the user. Further, we model the QoE maximization problem for multiple users with heterogeneous QoE preferences as a multi-task RL problem, with each task distinguished by the user-distinct QoE preference. To efficiently address this problem, the proposed STEER solves for each RL-based ABR task by learning its optimal successor feature (SF) function, which can be exploited as shared knowledge across tasks to facilitate the transfer between tasks. With SF functions, STEER can quickly evaluate the optimal policies of previously learned tasks on a new task, and further use the generalized policy improvement operation to obtain a jumpstart policy. Both theoretically and empirically, we show that this jumpstart policy is a good initial policy with a performance guarantee for better generalization in the new task, and can also lead to a faster convergence to the optimal policy of the new task.

Viewers of 360-degree videos are provided with both visual modality to characterize their surrounding views and audio modality to indicate the sound direction. Though both modalities are important for saliency prediction, little work has been done by jointly exploiting them, which is mainly due to the lack of audio-visual saliency datasets and insufficient exploitation of the multi-modality. In this article, we first construct an audio-visual saliency dataset with 57 360-degree videos watched by 63 viewers. Through a deep analysis of the constructed dataset, we find that the human gaze can be attracted by the auditory cues, resulting in a more concentrated saliency map if the sound source's location is further provided. To jointly exploit the visual and audio features and their correlation, we further design a saliency prediction network for 360-degree videos (SVGC-AVA) based on spherical vector-based graph convolution and audio-visual attention. The proposed spherical vector-based graph convolution can process visual and audio features directly in the sphere domain, thus avoiding projection distortion incurred by traditional CNN-based predictors. In addition, the audio-visual attention scheme explores self-modal and cross-modal correlation for both modalities, which are further hierarchically processed with the U-Net's multi-scale structure of SVGC-AVA. Evaluations on both our and public datasets validate that SVGC-AVA can achieve higher prediction accuracy, both qualitatively and subjectively.

Message passing has evolved as an effective tool for designing graph neural networks (GNNs). However, most existing methods for message passing simply sum or average all the neighboring features to update node representations. They are restricted by two problems: 1) lack of interpretability to identify node features significant to the prediction of GNNs and 2) feature overmixing that leads to the oversmoothing issue in capturing long-range dependencies and inability to handle graphs under heterophily or low homophily. In this article, we propose a node-level capsule graph neural network (NCGNN) to address these problems with an improved message passing scheme. Specifically, NCGNN represents nodes as groups of node-level capsules, in which each capsule extracts distinctive features of its corresponding node. For each node-level capsule, a novel dynamic routing procedure is developed to adaptively select appropriate capsules for aggregation from a subgraph identified by the designed graph filter. NCGNN aggregates only the advantageous capsules and restrains irrelevant messages to avoid overmixing features of interacting nodes. Therefore, it can relieve the oversmoothing issue and learn effective node representations over graphs with homophily or heterophily. Furthermore, our proposed message passing scheme is inherently interpretable and exempt from complex post hoc explanations, as the graph filter and the dynamic routing procedure identify a subset of node features that are most significant to the model prediction from the extracted subgraph. Extensive experiments on synthetic as well as real-world graphs demonstrate that NCGNN can well address the oversmoothing issue and produce better node representations for semisupervised node classification. It outperforms the state of the arts under both homophily and heterophily.

There has been a recent surge of interest in introducing transformers to 3D human pose estimation (HPE) due to their powerful capabilities in modeling long-term dependencies. However, existing transformer-based methods treat body joints as equally important inputs and ignore the prior knowledge of human skeleton topology in the self-attention mechanism. To tackle this issue, in this paper, we propose a Pose-Oriented Transformer (POT) with uncertainty guided refinement for 3D HPE. Specifically, we first develop novel pose-oriented self-attention mechanism and distance-related position embedding for POT to explicitly exploit the human skeleton topology. The pose-oriented self-attention mechanism explicitly models the topological interactions between body joints, whereas the distance-related position embedding encodes the distance of joints to the root joint to distinguish groups of joints with different difficulties in regression. Furthermore, we present an Uncertainty-Guided Refinement Network (UGRN) to refine pose predictions from POT, especially for the difficult joints, by considering the estimated uncertainty of each joint with uncertainty-guided sampling strategy and self-attention mechanism. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art methods with reduced model parameters on 3D HPE benchmarks such as Human3.6M and MPI-INF-3DHP.

Domain Generalization (DG) has achieved great success in generalizing knowledge from source domains to unseen target domains. However, current DG methods rely heavily on labeled source data, which are usually costly and unavailable. Since unlabeled data are far more accessible, we study a more practical unsupervised domain generalization (UDG) problem. Learning invariant visual representation from different views, i.e., contrastive learning, promises well semantic features for in-domain unsupervised learning. However, it fails in cross-domain scenarios. In this paper, we first delve into the failure of vanilla contrastive learning and point out that semantic connectivity is the key to UDG. Specifically, suppressing the intra-domain connectiv-ity and encouraging the intra-class connectivity help to learn the domain-invariant semantic information. Then, we propose a novel unsupervised domain generalization approach, namely Dual Nearest Neighbors contrastive learning with strong Augmentation (DN2A). Our DN2A leverages strong augmentations to suppress the intra-domain connectivity and proposes a novel dual nearest neighbors search strategy to find trustworthy cross domain neighbors along with in-domain neighbors to encourage the intra-class connectivity. Experimental results demonstrate that our DN2A outperforms the state-of-the-art by a large margin, e.g., 12.01% and 13.11 % accuracy gain with only 1% labels for linear evaluation on PACS and DomainNet, respectively.

End-to-end learned image compression (LIC) has become promising alternatives for lossy image compression. However, deployments of LIC models are limited in real-world applications due to excessive network parameters and high computational complexity. Existing LIC models realized with integer networks are significantly de-graded in rate-distortion (R-D) performance. In this paper, we propose a novel fully integerized LIC model that simultaneously achieves channel-wise weight and 8-bit activation quantization for alleviating the loss of R-D performance. For weight quantization, we develop an internal bit width increment (IBWI) via nonlinear logarithmic mapping to convert the stored low-precision integer weights to high bit-width weights for inference. Moreover, outlier channel splitting (OCS) [1] is employed to address large outliers of weight distribution by duplicating channels and constrain the integer weights within the scope of INT8. For activation quantization, we leverage activation equalization to balance the channel-wise distribution of activations. Experimental results demonstrate that the proposed method achieves a reduction of 75% storage cost with subtle performance loss compared to full-precision pre-trained models, and outperforms existing integer-only networks, as shown in Table 1.

Quantizing one single deep neural network into multiple compression rates (precisions) has been recently considered for flexible deployments in real-world scenarios. In this paper, we propose a novel scheme that achieves progressive bit-width allocation and joint training to simultaneously optimize mixed-precision quantized networks under multiple compression rates. Specifically, we develop a progressive bit-width allocation with switchable quantization step size to enable mixed-precision quantization based on analytic sensitivity of network layers under multiple compression rates. Furthermore, we achieve joint training for quantized networks under different compression rates via knowledge distillation to exploit their correlations based on the shared network structure. Experimental results show that the proposed scheme outperforms AdaBits [1] in various networks on CIFAR-10 and ImageNet.

Generalization ability of face anti-spoofing has been widely concerned in recent years. Existing domain generalization methods use adversarial learning or metric learning to extract invariant features across domains but are proved to be flawed from causal views. The learned domain-invariant features cannot generalize well to unseen real scenarios in both theoretical and practical senses. In this paper, we propose a novel method that learns Causal Representations for Face Anti-Spoofing (CRFAS). We first model the data generation process of face anti-spoofing via Structural Casual Model (SCM) and reveal that only the causal feature is capable of generalizing to unseen domains. On such basis, we extract causal features via back- door adjustment without prior assumptions rather than learn domain-invariant features as existing methods. Furthermore, we employ the supervised contrastive loss to generate more realistic counterfactual features for backdoor adjustment and improve the generalization ability of learned causal features. Extensive experiments on six cross-dataset testing scenarios demonstrate that CRFAS outperforms the state-of-the-art face anti-spoofing methods in terms of HTER and AUC in generalizing to unseen domains.

Point cloud upsampling is a challenging task to generate dense and uniform point clouds from sparse and irregular inputs. Existing deep learning based methods usually leverage graph convolution networks (GCNs) to learn features for expansion and reconstruction but suffer from the over-smoothing issue by discarding the high-frequency components. In this paper, we design a novel dynamic graph scattering transform (GST) to achieve multi-scale feature extraction for point cloud upsampling. The dynamic GST can accommodate to the K-nearest neighbor (KNN)-based feature graph dynamically constructed for point clouds to achieve multi-resolution band-pass representations. Consequently, a feature extractor with dense connection is developed based on the dynamic GST such that low-frequency and high-frequency components can be sufficiently combined for subsequent feature expansion and reconstruction. To our best knowledge, this paper is the first to introduce graph scattering transform for representation learning with dynamic feature graphs in point cloud upsampling. Experimental results on the PU1K dataset show that the proposed network based on dynamic GST achieve promising performance under various metrics especially for sparse point clouds.

In personalized federated learning (PFL), multiple clients train customized models to fulfill their personal objectives, which, however, are prone to overfitting to local data due to the heterogeneity and scarcity of local data. To address this, we propose from the information-theoretic perspective a personalized federated learning framework based on the common representation learned across clients, named FedCR. Specifically, we introduce to the local client update a regularizer that aims at minimizing the discrepancy between local and global conditional mutual information (CMI), such that clients are encouraged to learn and exploit the common representation. Upon this, each client learns individually a customized predictor (head), while the extractor (body) remains to be aggregated by the server. Our CMI regularizer leads to a theoretically sound alignment between the local and global stochastic feature distributions in terms of their Kullback-Leibler (KL) divergence. More importantly, by modeling the global joint feature distribution as a product of multiple local feature distributions, clients can efficiently extract diverse information from the global data but without need of the raw data from other clients. We further show that noise injection via feature alignment and ensemble of local predictors in FedCR would help enhance its generalization capability. Experiments on benchmark datasets demonstrate a consistent performance gain and better generalization behavior of FedCR.

Learning a rotation-invariant (RI) representation is of significant importance for real-world point cloud segmentation that is perturbed by arbitrary rotations. Recent principal component analysis (PCA)-based methods provide an effective alternative to align point clouds and produce the RI representation under the preservation of global information. However, conventional PCA with 3-D coordinates cannot fully represent high-dimensional geometric structures like surfaces and curves and cannot uniformly align these structures for learning RI representation. In this paper, we propose a novel rotation-invariant method for point cloud segmentation, which leverages kernel PCA (KPCA) for aligning point clouds in a projected high-dimensional space via non-linear mapping and develops a Geometry-based Weighted Convolution (GWConv) to distinguish part boundaries during segmentation. Specifically, the KPCA produces a RI representation with polynomial kernels for effectively representing complicated geometric structures in point clouds. Moreover, the GWConv incorporates geometric structures into convolution and enhances neighboring points with similar geometry for fine-grained segmentation based on the RI representation. Experimental results demonstrate that the proposed method can achieve competitive performance with the state-of-the-arts and outperforms existing PCA-based methods in part segmentation on ShapeNet. Furthermore, it achieves evident performance gains on complicated 3-D shapes such as Earphone and Car and facilitates segmentation around the part boundaries.

Meta-reinforcement learning (Meta-RL), though enabling a fast adaptation to learn new skills by exploiting the common structure shared among different tasks, suffers performance degradation in the sparse-reward setting. Current hindsight-based sample transfer approaches can alleviate this issue by transferring relabeled trajectories from other tasks to a new task so as to provide informative experience for the target reward function, but are unfortunately constrained with the unrealistic assumption that tasks differ only in reward functions. In this paper, we propose a doubly robust augmented transfer (DRaT) approach, aiming at addressing the more general sparse reward meta-RL scenario with both dynamics mismatches and varying reward functions across tasks. Specifically, we design a doubly robust augmented estimator for efficient value-function evaluation, which tackles dynamics mismatches with the optimal importance weight of transition distributions achieved by minimizing the theoretically derived upper bound of mean squared error (MSE) between the estimated values of transferred samples and their true values in the target task. Due to its intractability, we then propose an interval-based approximation to this optimal importance weight, which is guaranteed to cover the optimum with a constrained and sample-independent upper bound on the MSE approximation error. Based on our theoretical findings, we finally develop a DRaT algorithm for transferring informative samples across tasks during the training of meta-RL. We implement DRaT on an off-policy meta-RL baseline, and empirically show that it significantly outperforms other hindsight-based approaches on various sparse-reward MuJoCo locomotion tasks with varying dynamics and reward functions.

It is promising to solve linear inverse problems by unfolding iterative algorithms (e.g., iterative shrinkage thresholding algorithm (ISTA)) as deep neural networks (DNNs) with learnable parameters. However, existing ISTA-based unfolded algorithms restrict the network architectures for iterative updates with the partial weight coupling structure to guarantee convergence. In this paper, we propose hybrid ISTA to unfold ISTA with both pre-computed and learned parameters by incorporating free-form DNNs (i.e., DNNs with arbitrary feasible and reasonable network architectures), while ensuring theoretical convergence. We first develop HCISTA to improve the efficiency and flexibility of classical ISTA (with pre-computed parameters) without compromising the convergence rate in theory. Furthermore, the DNN-based hybrid algorithm is generalized to popular variants of learned ISTA, dubbed HLISTA, to enable a free architecture of learned parameters with a guarantee of linear convergence. To our best knowledge, this paper is the first to provide a convergence-provable framework that enables free-form DNNs in ISTA-based unfolded algorithms. This framework is general to endow arbitrary DNNs for solving linear inverse problems with convergence guarantees. Extensive experiments demonstrate that hybrid ISTA can reduce the reconstruction error with an improved convergence rate in the tasks of sparse recovery and compressive sensing.

Progressive coding is essential to the practical deployment of learned image compression over heterogeneous networks and clients. Existing methods for learned progressive image compression require complex and empirical design to achieve near-optimal rate-distortion performance over a wide range of bit-rates. However, these methods are limited by the implicit learned mechanism based on neural networks and introduction of uniform quantizers. In this paper, we propose generalized learned progressive image compression with analytic rate-distortion optimization using dead-zone quantizers on the latent representation. Specifically, we reveal that dead-zone quantizers, as a general case of uniform quantizers, are equivalent to uniform quantizers in fixed-rate nonlinear transform coding and can prevent extra redundancy in embedded quantization for progressive coding. Consequently, we propose rate-distortion optimized learned progressive coding by approximating the optimal quantizer in the source spaces using dead-zone quantizers in an analytic manner on the Laplacian source. To our best knowledge, this paper is the first to achieve general learned progressive coding from the perspective of optimal quantizers. The proposed method achieves theoretically sound and practically efficient embedded quantization and learned progressive coding of latent representations with improved rate-distortion performance. It can also enable embedded quantization with diverse assignments of truncation points and support flexible configuration of quality layers of varying numbers and at varying target bit-rates. Furthermore, we successfully incorporate the proposed method into existing pre-trained fixed-rate models to realize progressive learned image compression without re-training. Experimental results demonstrate that the proposed method achieves state-of-the-art rate-distortion performance in learned progressive image compression compared with traditional codecs and recent learned methods.

With increasing popularity of virtual reality and augmented reality, application of point clouds is in critical demand as it enables users to freely navigate in an immersive scene with six degrees of freedom. However, point clouds usually comprise large amounts of data, and are thus difficult to stream in bandwidth-constrained networks. It is therefore important, yet challenging, to efficiently stream the resource-intensive point clouds, such that the user’s quality of experience (QoE) is guaranteed on a high-level but with a low bandwidth consumption. To this end, we propose a QoE-driven adaptive streaming approach for the tile-based point cloud transmission, to maximize the user’s QoE while reducing the transmission redundancy. By exploiting the perspective projection, we specifically model the QoE of a 3D tile as a function of the bitrate of its representation, user’s view frustum and spatial position, occlusion between tiles, and the resolution of rendering device. Based on this QoE model, we then formulate the QoE-optimized rate adaptation problem as a multiple-choice knapsack problem, which allocates bitrates for different tiles under a given transmission capacity. It is equivalently converted to a submodular function maximization problem subject to knapsack constraints, and solved by a practical greedy-based algorithm with a theoretical worst-case performance guarantee. The proposed algorithm is able to achieve a near-optimal performance, but with a very low computational complexity. Experimental results further demonstrate superiority of the proposed rate adaptation algorithm over existing schemes, in terms of both user’s visual quality and transmission efficiency.

Light field (LF) data are widely used in the immersive representations of the 3D world. To record the light rays along with different directions, an LF requires much larger storage space and transmission bandwidth than a conventional 2D image with similar spatial dimension. In this paper, we propose a novel framework for light field image compression that leverages graph learning and dictionary learning to remove structural redundancies between different views. Specifically, to significantly reduce the bit-rates, only a few key views are sampled and encoded, whereas the remaining non-key views are reconstructed via the graph adjacency matrix learned from the angular patch. Furthermore, dictionary-guided sparse coding is developed to compress the graph adjacency matrices and reduce the coding overheads. To our best knowledge, this paper is the first to achieve compact representation of cross-view structural information via adaptive learning on graphs. Experimental results demonstrate that the proposed framework achieves better performance than the standardized HEVC-based codec.

Existing entropy models in learned image compression are cumbersome to generate fixed mean and variance for estimating Gaussian distributions for latent representation. In this paper, we propose a novel entropy model based on Gaussian process regression (GPR) that flexibly predicts the mean of Gaussians with posterior distributions characterized by covari-ance functions spanned in the high-dimensional feature space. Furthermore, we develop the rate-distortion optimization based on the proposed entropy model by approximating the bitrates with the evidence lower bound (ELBO) derived via variational inference for GPR. The proposed model can be seamlessly integrated into existing end-to-end optimized frame-works by substituting the masked convolution based autoregressive models. Experimental results demonstrate that the proposed model outperforms conventional image compression methods such as JPEG2000 and BPG, as well as recent learning based methods on the Kodak dataset in terms of rate-distortion performance.

Existing methods for cross-component prediction focus on the chroma intra pre-diction but neglect the chroma prediction across multiple frames. In this paper, we propose a novel compression framework that leverages chroma frame sampling and inter-frame chroma prediction to address this problem. Specifically, chroma frame sampling discards the chroma components in inter-predicted frames to further reduce the bit consumption, whereas inter-frame chroma prediction recovers the discarded chroma components with optical flow estimation and post-training optimization for a guaranteed fidelity. To our best knowledge, this paper is the first attempt to achieve deep learning-based inter prediction of chroma components. To accommodate the proposed framework, we redesign the HEVC codec to enable hybrid coding of luma and chroma components. Experimental results show that the proposed framework achieves up to 0.76% BD-rate reduction when compared to standard HEVC.

3-D point clouds rendering solid representations of scenes or objects often carry a tremendous amount of points, compulsorily requesting high-efficiency compression for storage and transmission. In this paper, we propose a novel$p$-Laplacian embedding graph dictionary learning algorithm for 3-D point cloud attribute compression. The proposed method integrates the underlying graph topology to the learned graph dictionary capitalizing on$p$-Laplacian eigenfunctions and leads to parsimonious representations of 3-D point clouds. We further devise alternating optimization with the help of ADMM to efficiently solve the resulting non-convex minimization problem.

We propose a structured pruning method to achieve a light-weighted decoder of learned image compression to accommodate various terminals. The structured pruning method identifies the effectiveness of each channel of decoder via gradient ascent and gradient descent while maintaining the encoder and entropy model. To our best knowledge, this paper is the first attempt to design a structured pruning method for universal pretrained learned image compression. Experimental results demonstrate that the proposed method can reduce about 40% parameters and save 25% inference time at the cost of 0.23 dB BD-PSNR and 4.33% BD-rate change.

Graph convolutional networks (GCNs) are widely adopted for spherical data processing, striking a balance between rotation equivariance and computation efficiency. However, current GCN-based approaches either apply radial filters with a fixed convolution kernel, or adopt attention schemes to learn a variable kernel based only on node features, which lack expressiveness and may suffer over-smoothing problem. To address this, in this paper, we propose a global-local attention-based spherical graph convolution (GlasGC) for spherical data representation. Specifically, the designed graph convolution includes a structure-enhanced local attention module to estimate the structure correlation efficiently based on topological characteristics of the given spherical graph. The structure correlation is then incorporated with the feature correlation to form the local attention that determines importance of a neighbor node to the center node. We also design a global attention module to obtain the importance scores of different nodes based on both node features and graph topology, which acts as a message-passing enhancer for informative nodes and can relieve the over-smoothing problem. Based on our proposed GlasGC, we design specific GlasGCNs for classification and semantic segmentation of spherical data. Empirical evaluations on these tasks demonstrate that our proposed GlasGCNs can generally achieve better classification and segmentation performance.

Generalizable face anti-spoofing (FAS) has drawn growing attention due to its robustness to unseen real scenarios. Existing domain generalization methods leverage adversarial learning or meta-learning to mitigate the domain bias and improve generalizability. However, these methods are heuristic and suffer from complicated min-max problems or cumbersome meta-updates. In this paper, we propose a simple yet effective Causal Intervention method for generalizable Face Anti-Spoofing, namely CIFAS. Firstly, we figure out the generalizability is undermined by a domain-aware confounder based on the structural causal model. Instantiating the confounder as the domain-specific factor, a domain embedding module is employed with Dirichlet mixup to obtain representative domain features. Consequently, we propose a novel backdoor adjustment model for causal intervention to capture the true causality and learn a robust FAS model. Our CIFAS is the first attempt to introduce causal learning into FAS. Extensive experiments on seven cross-dataset tests demonstrate that CIFAS outperforms the state-of-the-art methods.

Network quantization can facilitate the practical deployment of U-Net in various medical applications, especially medical image segmentation. However, existing quantization methods for U-Net do not sufficiently exploit the skip connection between its encoder and decoder structures. In this paper, we propose a novel mixed-precision quantization scheme for U-Net that leverages split convolution and skip-supervised quantization aware training to address this problem. Specifically, split convolution decouples the feature maps obtained by skip connection and up-sampling to enable fine-grained bitwidth allocation for mixed-precision quantization. Furthermore, we introduce supervision on skip connection for quantization aware training to compensate gradients in back propagation and enhance the skip features in finetuning. To our best knowledge, this is the first attempt to design mixed-precision quantization specifically for medical image segmentation with U-Net. Experimental results on four medical image datasets demonstrate that the proposed scheme outperforms existing U-Net quantization strategy and recent general-purpose mixed-precision quantizaton methods in medical image segementation.

Recent video colorization methods optimize correspondence estimation and information propagation in an end-to-end manner. However, they usually suffer from loss of fidelity due to the inaccurate inference of correspondence measurement. In this paper, we propose a post-training optimization (PTO) strategy to refine correspondence measurement in the end-to-end optimized framework. The proposed PTO strategy introduces a pseudo loss function to well approximate the target loss and guide the direction of updates. We further develop a video colorization method that incorporates PTO and optical flow to guarantee high-fidelity colorized frames in theory. Experimental results demonstrate that the proposed method achieves state-of-the-art PSNR performance in video colorization on the DAVIS dataset and common test sequences for video coding. Furthermore, the proposed method can be employed into video compression and achieves competitive rate-distortion performance with the recent High Efficiency Video Coding (HEVC) standard.

Differentiable architecture search (DARTS) has achieved success in searching powerful network architectures but suffers from unstable search, high search cost from repetitive attempts and unawareness of computational cost. In this paper, we propose a novel approach, namely Efficient and Stable Differentiable Architecture Search (ES-DARTS), that leverages decoupled search strategy and variational proxy pruning to achieve efficient search for efficient neural networks. Specifically, the decoupled search strategy stabilizes the search via bridging the gap between search and evaluation, and achieves acceleration with decoupled optimization, while the variational proxy pruning introduces structural pruning into DARTS to accommodate varying constraints on computational cost. ES-DARTS can be a plug-and-play module for DARTS-based approaches to achieve improved performance with reduced search cost and FLOPS. Experimental results demonstrate that ES-DARTS reduces top-1 error rate to 2.71% with $10 \times$ acceleration of DARTS on CIFAR10. ES-DARTS finds an architecture of 2.80% top-1 error rate with only 2.27 MB parameters on CIFAR-10 and of 29.4% top-1 error rate with only 322M FLOPS when transferred to ImageNet.

In this paper, we present a meta reinforcement learning (Meta-RL)-based neural adaptive bitrate streaming (ABR) algorithm that is able to rapidly adapt its control policy to the changing network throughput dynamics. Specifically, to allow rapid adaptation, we discuss the necessity of detaching the inference of throughput dynamics with the universal control mechanism that is in essence shared by all potential throughput dynamics for neural ABR algorithms. To meta-learn the ABR policy, we then build up a model-free system framework, composed of a probabilistic latent encoder that infers the underlying dynamics from the recent throughput context, and a policy network that is conditioned on latent variable and learns to quickly adapt to new environments. Additionally, to address the difficulties caused by training the policy on mixed dynamics, on-policy RL (or imitation learning) algorithms are suggested for policy training, with a mutual information-based regularization to make the latent variable more informative about the policy. Finally, we implement our algorithm's meta-training and meta-adaptation procedures under a variety of throughput dynamics. Empirical evaluations on different QoE metrics and multiple datasets containing real-world network traces demonstrate that our algorithm outperforms state-of-the-art ABR algorithms, in terms of the performance on the average chunk QoE, consistency and fast adaptation across a wide range of throughput patterns.

Entropy coding is fundamental for reducing the coding redundancy in image compression. However, existing entropy models for learned image compression are restricted by independent or autoregressive modeling based on presumed distribution generated from the family of Gaussian functions. In this paper, we propose a novel tensor network-based entropy model that can explicitly infer the joint distribution of image representation for learned image compression. We utilize a tree tensor network (TTN) to enable exact computation for probabilities of image representation. Specifically, we produce efficient tensor representation for entropy modeling based on bit planes slicing with gray code. Furthermore, we leverage tensor contraction to accurately calculate the partition function and jointly predict the entire bit plane for entropy coding. To our best knowledge, this paper is the first to directly infer the joint distribution of image representation in learned image compression without any presumed condition. Experimental results demonstrate that the proposed method is competitive with the state-of-the-art in both lossless and lossy image compression.

360-degree videos pose a challenge to their transmission due to the ultra-high resolution. To achieve efficient streaming, current tile-based schemes divide the 2D projection of spherical frames into small rectangular tiles, which are treated as individual sources for encoding and then delivered according to user's actual viewport. However, the trade-off between transmission and encoding efficiency, as controlled by encoding parameters such as tile sizes and quantization steps, has not been fully studied yet. In this paper, we propose a tile-based encoding scheme to strike this trade-off, by selecting the optimal encoding parameters that minimize the distortion of video displayed for user's viewport under a given transmission capacity. Specifically, we quantify the transmission efficiency by estimating the average number of pixels required for transmission to cover the user's viewport, while a CNN-based predictor is trained to predict the encoding efficiency based on video content characteristics, expressed as number of bits needed for encoding one pixel. We then formulate a rate-distortion optimization problem for tile-based encoding, which is solved efficiently by decreasing the feasible set of encoding parameters. Simulation results demonstrate that the proposed optimal tile-based encoding outperforms both fixed tiling and state-of-the-art adaptive tiling schemes in terms of rate-distortion behavior.

Tile-based rate adaption can improve the quality of experience (QoE) for adaptive 360-degree video streaming under constrained network conditions, which, however, is a challenging problem due to the requirements of accurate prediction for users’ viewports and optimal bitrate allocation for tiles. In this paper, we propose a strategy that deploys reinforcement learning-based Rate Adaptation with adaptive Prediction and Tiling for 360-degree video streaming, named RAPT360, to address these challenges. Specifically, to improve the accuracy of the state-of-the-art viewport prediction approaches, we fit the time-varying Laplace distribution-based probability density function of the prediction error for different prediction lengths. On the basis of that, we develop a viewport identification method to determine the viewport area of a user depending on the buffer occupancy, where the obtained viewport can cover the real viewport with any given probability confidence level. We then propose a viewport-aware adaptive tiling scheme to improve the bandwidth efficiency, where three types of tile granularities are allocated according to the shape and position of the 2-D projection of that viewport. By establishing an adaptive streaming model and QoE metric specific to 360-degree videos, we finally formulate the rate adaptation problem for tile-based 360-degree video streaming as a non-linear discrete optimization problem that targets at maximizing the long-term user QoE under a bandwidth-constrained network. To efficiently solve this problem, we model the rate adaptation logic as a Markov decision process (MDP) and employ the deep reinforcement learning (DRL)-based algorithm to dynamically learn the optimal bitrate allocation of tiles. Extensive experimental results show that RAPT360 achieves a performance gain of at least 1.47 dB on average chunk QoE, including a video quality improvement of at least 1.33 dB, in comparison to the existing strategies for tile-based adaptive 360-degree video streaming.

Model quantization is essential to deploy deep convolutional neural networks (DCNNs) on resource-constrained devices. In this article, we propose a general bitwidth assignment algorithm based on theoretical analysis for efficient layerwise weight and activation quantization of DCNNs. The proposed algorithm develops a prediction model to explicitly estimate the loss of classification accuracy led by weight quantization with a geometrical approach. Consequently, dynamic programming is adopted to achieve optimal bitwidth assignment on weights based on the estimated error. Furthermore, we optimize bitwidth assignment for activations by considering the signal-to-quantization-noise ratio (SQNR) between weight and activation quantization. The proposed algorithm is general to reveal the tradeoff between classification accuracy and model size for various network architectures. Extensive experiments demonstrate the efficacy of the proposed bitwidth assignment algorithm and the error rate prediction model. Furthermore, the proposed algorithm is shown to be well extended to object detection.

Cloud service has been emerging as a promising alternative to handle massive volumes of video sequences triggered by increasing demands on video service, especially surveillance and entertainment. Lossless compression of encoded video bitstreams can further eliminate the redundancies without altering the contents and facilitate the efficiency of cloud storage. In this paper, we propose a novel lossless compression scheme to further compress the video bitstreams generated by the state-of-the-art hybrid coding frameworks like H.264/AVC and HEVC. Different from transcoding, the proposed scheme develops frequency prediction and Macro Block (MB) merging to eliminate the redundancies remained in the intra-and inter-predicted frames with a strict guarantee of video fidelity. To our best knowledge, this paper is the first attempt to realize lossless compression of video bitstreams generated by advanced coding standards H.264/AVC and HEVC. Experimental results demonstrate that the proposed scheme can achieve compression gains of 17.38% and 10.85% on standard test sequences and surveillance videos, respectively.

Deep learning has been widely adopted in compression sensing (CS) to achieve superior reconstruction quality, but is restricted by the black-box architecture in network design and lack of interpretability. In this paper, we propose a novel deep network-based CS framework via unfolding the$\ell_{0}$-constrained convolutional sparse coding (CSC). The proposed method incorporates deep neural networks (DNNs) into the state-of-the-art optimization paradigm of alternating direction method of multipliers (ADMM). CS reconstruction can be improved with the compact and shift-invariant sparse representation under the merits of CSC and DNNs with learnable parameters. Convergence analysis demonstrates that the proposed method is guaranteed to converge, when the network parameters are reused for all the building blocks. Experimental results demonstrate that the proposed method is competitive or outperform the state-of-the-art DNN-based methods in CS of natural images and magnetic resonance imaging (MRI).

Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation system consisting of one master and$N$workers characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with$L$model parameters. We show that it is sufficient to provide at most$N$levels of redundancies for tolerating$0,1, \cdots, N-1$stragglers, respectively. Consequently, we propose an optimal block coordinate gradient coding scheme based on a stochastic optimization problem that optimizes the partition of the$L$coordinates into$N$blocks, each with identical redundancy, to minimize the expected overall runtime for collaboratively computing the gradient. We obtain an optimal solution using a stochastic projected subgradient method and propose two low-complexity approximate solutions with closed-from expressions, for the stochastic optimization problem. We also show that under a shifted-exponential distribution, for any$L$, the expected overall runtimes of the two approximate solutions and the minimum overall runtime have sub-linear multiplicative gaps in$N$. To the best of our knowledge, this is the first work that optimizes the redundancies of gradient coding introduced across the coordinates of gradients.

Application of point clouds is in critical demand, which, however, are composed of large amounts of data and difficult to stream in bandwidth-constrained networks. To address this, we propose a QoE-driven and tile-based adaptive streaming approach for point clouds, to reduce transmission redundancy and maximize user’s QoE. Specifically, by utilizing the perspective projection, we model the QoE of a 3D tile as a function of the bitrate of its representation, user’s view frustum and spatial position, occlusion between tiles, and the resolution of rendering device. We then formulate the QoE-optimized rate adaptation problem as a multiple-choice knapsack problem that allocates bitrates for different tiles under a given transmission capacity. We equivalently convert it as a submodular function maximization problem subject to knapsack constraints, and develop a practical greedy algorithm with a theoretical performance guarantee. Experimental results further demonstrate superiority of the proposed rate adaptation algorithm over existing schemes, in terms of both user’s visual quality and transmission efficiency.

Existing differentiable neural architecture search approaches simply assume the architectural distribution on each edge is independent of each other, which conflicts with the intrinsic properties of architecture. In this paper, we view the architectural distribution as the latent representation of specific data points. Then we propose Variational Information Maximization Neural Architecture Search (VIM-NAS) to leverage a simple yet effective convolutional neural network to model the latent representation, and optimize for a tractable variational lower bound to the mutual information between the data points and the latent representations. VIM-NAS automatically learns a nearly one-hot distribution from a continuous distribution with extremely fast convergence speed, e.g., converging with one epoch. Experimental results demonstrate VIM-NAS achieves state-of-the-art performance on various search spaces, including DARTS search space, NAS-Bench-1shot1, NAS-Bench-201, and simplified search spaces S1-S4. Specifically, VIM-NAS achieves a top-1 error rate of 2.45% and 15.80% within 10 minutes on CIFAR-10 and CIFAR-100, respectively, and a top-1 error rate of 24.0% when transferred to ImageNet.

Learned image compression has achieved improved rate-distortion performance with end-to-end optimized framework based on deep neural networks. However, context-based entropy modeling for learned image compression cannot simultaneously achieve enhanced efficiency and sufficiently exploiting the channel-wise correlations. In this paper, we propose a novel framework for learned image compression with channel-wise grouped context modeling. The proposed framework presents channel-wise grouping to explicitly exploit the channel-wise correlations and develop a grouped 3-D context model to achieve efficient entropy coding with a guarantee of rate-distortion performance. The proposed framework achieves competitive performance with a significantly reduced decoding complexity in comparison to 3-D context models.

State-of-the-art deep reinforcement learning (DRL) algorithms tend to overfit due to the model discrepancy between source and target environments. Though applying domain randomization during training can improve the average performance by randomly generating a sufficient diversity of environments in simulator, the worst-case environment is still neglected without any performance guarantee. Since the average and worst-case performance are both important for generalization in RL, in this paper, we propose a policy optimization approach for concurrently improving the policy’s performance in the average and worst-case environment. We theoretically derive a lower bound for the worst-case performance of a given policy by relating it to the expected performance. Guided by this lower bound, we formulate an optimization problem to jointly optimize the policy and sampling distribution, and prove that by iteratively solving it the worst-case performance is monotonically improved. We then develop a practical algorithm, named monotonic robust policy optimization (MRPO). Experimental evaluations in several robot control tasks demonstrate that MRPO can generally improve both the average and worst-case performance in the source environments for training, and facilitate in all cases the learned policy with a better generalization capability in some unseen testing environments.

In this paper, we propose BayesMPC, an uncertainty-aware robust adaptive bitrate (ABR) algorithm on the basis of Bayesian neural network (BNN) and model predictive control (MPC). Specifically, to improve the capacity of learning transition probability of the network throughput, we adopt a BNN-based predictor that is able to predict the statistical distribution of future throughput from the past throughput by not only considering the aleatoric uncertainty (e.g., noise), but also capturing the epistemic uncertainty incurred by lack of adequate training samples. We further show that by using the negative log-likelihood loss function to train this BNN-based throughput predictor, the generalization error can be minimized with the guarantee of PAC-Bayesian theorem. Rather than a point estimate, the learnt uncertainty can contribute to a confidence region for the future throughput, the lower bound of which then leads to an uncertainty-aware robust MPC strategy to maximize the worst-case user quality-of-experience (QoE) w.r.t. this confidence region. Finally, experimental results on three real-world network trace datasets validate the efficiency of both the proposed BNN-based predictor and uncertainty-aware robust MPC strategy, and demonstrate the superior performance compared to other baselines, in terms of both the overall QoE performance and generalization across all ranges of heterogeneous network and user conditions.

Mixed-precision quantization with adaptive bitwidth allocation for neural network has achieved higher compression rate and accuracy in classification task. However, it has not been well explored for object detection networks. In this paper, we propose a novel mixed-precision quantization scheme with dynamical Hessian matrix for object detection networks. We iteratively select a layer with the lowest sensitivity based on the Hessian matrix and downgrade its precision to reach the required compression ratio. The L-BFGS algorithm is utilized for updating the Hessian matrix in each quantization iteration. Moreover, we specifically design the loss function for objection detection networks by jointly considering the quantization effects on classification and regression loss. Experimental results on RetinaNet and Faster R-CNN show that the proposed DHMQ achieves state-of-the-art performance for quantized object detec-tors.

One-stage object detectors are trained by optimizing classification-loss and localization-loss simultaneously, with the former suffering much from extreme foreground-background class imbalance issue due to the large number of anchors. This paper alleviates this issue by proposing a novel framework to replace the classification task in one-stage detectors with a ranking task, and adopting the average-precision loss (AP-loss) for the ranking problem. Due to its non-differentiability and non-convexity, the AP-loss cannot be optimized directly. For this purpose, we develop a novel optimization algorithm, which seamlessly combines the error-driven update scheme in perceptron learning and backpropagation algorithm in deep networks. We provide in-depth analyses on the good convergence property and computational complexity of the proposed algorithm, both theoretically and empirically. Experimental results demonstrate notable improvement in addressing the imbalance issue in object detection over existing AP-based optimization algorithms. An improved state-of-the-art performance is achieved in one-stage detectors based on AP-loss over detectors using classification-losses on various standard benchmarks. The proposed framework is also highly versatile in accommodating different network architectures. Code is available at https://github.com/cccorn/AP-loss.

Existing 3-D point cloud attribute compression methods are rigid for complex signals defined in spatially irregular domain, as they solely depend on geometric information or adopt pre-defined analytic bases for transform. In this article, we propose a multi-scale structured dictionary learning algorithm for 3-D point cloud attribute compression. The proposed multi-scale dictionary leverages hierarchical sparsity within signals by adapting atoms in an arborescent structure. It supports hierarchical sparse transform for encoding voxelized point cloud attributes with progressive refinement of high-frequency details from coarser to finer scales. To represent spatial blocks with varying amounts of occupied voxels, a non-uniform binary weight matrix is incorporated in dictionary learning to characterize the dimensional irregularity of signals. To guarantee optimal approximation with improved efficiency, alternating optimizations including Alternating Direction Method of Multipliers andGauss-Seideliterations are adopted for the weighted mixed$\ell _{2}/\ell _{1}$-minimization. Furthermore, we develop a lossy attribute compression framework by integrating sparse transform based prediction with dedicated quantization and entropy coding routines. Experimental results demonstrate that the proposed framework outperforms the state-of-the-art transform-based coding methods and is competitive with the most recent MPEG PCC test models in point cloud attribute compression.

In this paper, we investigate the adaptive video delivery for multiple users over time-varying and mutually interfering multi-cell wireless networks. The key research challenge is to jointly design the physical-layer resource allocation scheme and application-layer rate adaptation logic, such that the users' long-term fair quality of experience (QoE) can be maximized. Due to the timescale mismatch between these two layers and the asynchrony of user requests, however, it is difficult to directly model the cross-layer stochastic control problem by using a reinforcement learning framework. To address this difficulty, we propose a novel two-level decision framework where an optimization-based beamforming scheme (performed at the base stations) and a deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals) are, respectively, developed, such that a highly complex long-term multi-user QoE fairness problem is decomposed into some relatively simple problems and solved effectively. Our strategy represents a significant departure from the existing schemes with consideration of either a short-term multi-user QoE maximization or a long-term single-user point-to-point QoE maximization. Extensive simulations demonstrate that the proposed cross-layer design is effective and promising.

The task of reidentifying groups of people under different camera views is an important yet less-studied problem. Group reidentification (Re-ID) is a very challenging task since it is not only adversely affected by common issues in traditional single-object Re-ID problems, such as viewpoint and human pose variations, but also suffers from changes in group layout and group membership. In this paper, we propose a novel concept of group granularity by characterizing a group image by multigrained objects: individual people and subgroups of two and three people within a group. To achieve robust group Re-ID, we first introduce multigrained representations which can be extracted via the development of two separate schemes, that is, one with handcrafted descriptors and another with deep neural networks. The proposed representation seeks to characterize both appearance and spatial relations of multigrained objects, and is further equipped with importance weights which capture variations in intragroup dynamics. Optimal group-wise matching is facilitated by a multiorder matching process which, in turn, dynamically updates the importance weights in iterative fashion. We evaluated three multicamera group datasets containing complex scenarios and large dynamics, with experimental results demonstrating the effectiveness of our approach.

Convolutional neural networks (CNNs) designed for low-dimensional regular grids will unfortunately lead to non-optimal solutions for analyzing spherical images, due to their different geometrical properties from planar images. In this paper, we generalize the grid-based CNNs to a non-Euclidean space by taking into account the geometry of spherical surfaces and propose a Spherical Graph Convolutional Network (SGCN) to encode rotation equivariant representations. Specifically, we propose a spherical graph construction criterion showing that a graph needs to be regular by evenly covering the spherical surfaces in order to design a rotation equivariant graph convolutional layer. For the practical case where the perfectly regular graph does not exist, we design two quantitative measures to evaluate the degree of irregularity for a spherical graph. The Geodesic ICOsahedral Pixelation (GICOPix) is adopted to construct spherical graphs with the minimum degree of irregularity compared to the current popular pixelation schemes. In addition, we design a hierarchical pooling layer to keep the rotation-equivariance, followed by a transition layer to enforce the invariance to the rotations for spherical image classification. We evaluate the proposed graph convolutional layers with different pixelations schemes in terms of equivariance errors. We also assess the effectiveness of the proposed SGCN in fulfilling rotation-invariance by the invariance error of the transition layers and recognizing the spherical images and 3D objects.

Reversible residual network naturally extends the linear lifting scheme with no theoretic guarantee. In this paper, we propose a reversible autoencoder (Rev-AE) with this extended non-linear lifting scheme to improve image reconstruction. Nonlinear prediction and update operators are designed based on shallow convolutional neural networks to model multilayer non-linearities. Different from existing autoencoders, Rev-AE support efficient image reconstruction with parameters reusable for the symmetric encoder and decoder. Rev-AE forms a set of related frames to guarantee perfect reconstruction with the non-linear extension of classic lifting scheme. Lower and upper bounds are developed for the set of frames to relate with the singular values for each non-linear operator. Furthermore, we employ Rev-AE into lossy image compression to evaluate its effectiveness on image reconstruction. Experimental results show that Rev-AE achieves competitive performance in comparison to the state-of-the-art.

Existing deep learning based methods for 3D human pose estimation cannot be deployed on resource-constrained devices. In this paper, we propose a Tiny-HourglassNet for 3D human pose estimation that improves the efficiency of Stacked Hourglass Networks with a guarantee of estimation performance. To be concrete, we develop an efficient Tiny-Hourglass backbone with two different types of ShuffleNet V2 blocks. At the meantime, we present a Simple Estimation Head (SEH) to reduce the resource utility for depth prediction. Furthermore, we develop two feature enhancement modules, namely Feature Enhancement Module (FEM) and Intermediate Enhancement Module (IEM), to improve the feature representation ability of Tiny-HourglassNet without evident increment of resource usage and computational complexity. Experimental results demonstrate that Tiny-HourglassNet achieves equivalent accuracy with a reduction of 80% complexity (FLOPs) in comparison to the baselines.

The k-nearest neighbors (KNN) algorithm has been widely adopted to construct graph convolutional networks (GCNs) for point cloud segmentation. However, the ℓ2norm cannot discriminate the multi-dimensional structures within a point cloud. In this paper, we propose a novel structure-aware graph construction for point clouds that compensates the ℓ2norm with per-dimension differences of the signal. The proposed method dynamically calculates the similarity ratio to determine the dimension-based proximity of the pair of points. Consequently, it improves both the spatial and spectral GCNs with the capability of aggregating information from relevant neighbors for point cloud segmentation. As a model-agnostic method, it can be seamlessly embedded into arbitrary GCN architectures during the graph construction phase. Experimental results demonstrate that the proposed method can improve classification accuracy around the joint areas of objects.

Bayesian methods have improved the interpretability and stability of neural architecture search (NAS). In this paper, we propose a novel probabilistic approach, namely Semi-Implicit Variational Dropout one-shot Neural Architecture Search (SI-VDNAS), that leverages semi-implicit variational dropout to support architecture search with variable operations and edges. SI-VDNAS achieves stable training that would not be affected by the over-selection of skip-connect operation. Experimental results demonstrate that SI-VDNAS finds a convergent architecture with only 2.7 MB parameters within 0.8 GPU-days and can achieve 2.60% top-1 error rate on CIFAR-10. The convergent architecture can obtain a top-1 error rate of 16.20% and 25.6% when transferred to CIFAR-100 and ImageNet (mobile setting).

Federated learning has successfully mitigated the privacy concerns for deep learning over distributed data, but suffers from performance decay led by the non-identical distribution of data. In this paper, we propose a novel framework, namely FedExg, to improve federated learning for non-IID data with model exchange. Two strategies, i.e., FedExg-S and FedExg-A, are developed to realize model exchange in a privacy-preserving fashion. FedExg-S randomly shuffles the models for broadcasting, while FedExg-A improves FedExg-S with partial aggregation to avoid model inversion attack. Theoretical analysis demonstrates that FedExg achieves a reduced upper bound of average regret in comparison to traditional model averaging methods. Experimental results show that FedExg improves image classification tasks on MNIST and CIFAR-10 dataset.

Recently, deep learning based compressive sensing (CS) methods show superior reconstruction performance. However, these approaches are restricted for compressive video sampling in practical design of imaging devices. In this paper, we propose a novel deep network-based CS framework for efficient optical implementation. The proposed framework, dubbed DSCR-Net, consists of a diffractive sensing network employing light diffraction for efficient sampling and a complex-valued neural network for reconstruction, respectively. The diffractive sensing network can achieve real-time sampling at the speed of light. Furthermore, complex-valued neural network is developed to facilitate the reconstruction quality from the complex-valued measurements by jointly considering their real and imaginary parts. Extensive experiments demonstrate that DSCR-Net outperforms the state-of-the-art CS methods in the low sampling-rate region with a potential of immediate implementation of imaging device.

The non-Euclidean geometry characteristic poses a challenge to the saliency prediction for 360-degree images. Since spherical data cannot be projected onto a single plane without distortion, existing saliency prediction methods based on traditional CNNs are inefficient. In this paper, we propose a saliency prediction framework for 360-degree images based on graph convolutional networks (SalGCN), which directly applies to the spherical graph signals. Specifically, we adopt the Geodesic ICOsahedral Pixelation (GICOPix) to construct a spherical graph signal from a spherical image in equirectangular projection (ERP) format. We then propose a graph saliency prediction network to directly extract the spherical features and generate the spherical graph saliency map, where we design an unpooling method suitable for spherical graph signals based on linear interpolation. The network training process is realized by modeling the node regression problem of the input and output spherical graph signals, where we further design a Kullback-Leibler (KL) divergence loss with sparse consistency to make the sparseness of the saliency map closer to the ground truth. Eventually, to obtain the ERP format saliency map for evaluation, we further propose a spherical crown-based (SCB) interpolation method to convert the output spherical graph saliency map into a saliency map in ERP format. Experiments show that our SalGCN can achieve comparable or even better saliency prediction performance both subjectively and objectively, with a much lower computation complexity.

End-to-end optimized image compression has emerged as a disruptive technique to reduce the spatial redundancies with an improved reconstruction quality. However, existing entropy model for latent representations cannot sufficiently exploit their spatial and channel-wise correlations. In this paper, we propose a novel entropy model based on spatial-channel contexts for end-to-end optimized image compression. The proposed model jointly leverages spatial structural dependencies and channel-wise correlations to improve the probabilistic estimation of latent representations. Instead of complex autoregressive hyperprior network, shallow artificial neural networks (ANNs) incorporating 3-D masks are developed to efficiently realize the entropy model with a guarantee of causality. Experimental results demonstrate that the proposed model achieves competitive rate-distortion performance and reduces model complexity in comparison to recent end-to-end optimized methods for image compression.

Existing decentralized coded caching solutions cannot guarantee small loads in the general scenario with arbitrary file sizes and cache sizes. In this paper, we propose an optimization framework for decentralized coded caching in the general scenario to minimize the worst-case load and average load (under an arbitrary file popularity), respectively. Specifically, we first propose a class of decentralized coded caching schemes for the general scenario, which are specified by a general caching parameter and include several known schemes as special cases. Then, we optimize the caching parameter to minimize the worst-case load and average load, respectively. Each of the two optimization problems is a challenging nonconvex problem with a nondifferentiable objective function. For each optimization problem, we develop an iterative algorithm to obtain a stationary point using techniques for solving Complementary Geometric Programming (GP). We also obtain a low-complexity approximate solution by solving an approximate problem with a differentiable objective function which is an upper bound on the original nondifferentiable one, and characterize the performance loss caused by the approximation. Finally, we present two information-theoretic converse bounds on the worst-case load and average load (under an arbitrary file popularity) in the general scenario, respectively. To the best of our knowledge, this is the first work that provides optimization-based decentralized coded caching schemes and information-theoretic converse bounds for the general scenario.

Conventional scalable compressive video sampling cannot sufficiently exploit the structured sparsity within video sequences. This paper proposes a novel quality scalable structured compressive video sampling (SS-CVS) framework with hierarchical subspace learning to support video transmission over heterogeneous network conditions. The proposed framework incorporates the union of data-driven subspaces (UoDS) model to introduce structured sparsity into sensing matrix for progressive acquisition of measurements. Hierarchical subspace learning is developed to generate the subspaces with their bases based on adaptive subspace clustering in a progressive fashion. Consequently, two hierarchical structures are derived to enable progressive mapping of subspaces and bases on the structured sensing matrices to support quality scalability. To guarantee the convergence of hierarchical subspace learning, the training set is updated with the adaptive group set from the reconstructed reference frames. It is demonstrated that SS-CVS can guarantee stable recovery for each quality layer under the constraints of block restricted isometric property (RIP). Furthermore, the proposed SS-CVS model is generalized to tensor subspaces for scalable compressive sampling of high-dimensional signals. Experimental results demonstrate that the proposed algorithm can achieve quality scalabilities with an improved reconstructed performance in comparison to the state-of-the-art scalable compressive video sampling methods.

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions.

This article addresses neural network based post-processing for the state-of-the-art video coding standard, High Efficiency Video Coding (HEVC). We first propose a partition-aware convolution neural network (CNN) that utilizes the partition information produced by the encoder to assist in the post-processing. In contrast to existing CNN-based approaches, which only take the decoded frame as input, the proposed approach considers the coding unit (CU) size information and combines it with the distorted decoded frame such that the artifacts introduced by HEVC are efficiently reduced. We further introduce an adaptive-switching neural network (ASN) that consists of multiple independent CNNs to adaptively handle the variations in content and distortion within compressed-video frames, providing further reduction in visual artifacts. Additionally, an iterative training procedure is proposed to train these independent CNNs attentively on different local patch-wise classes. Experiments on benchmark sequences demonstrate the effectiveness of our partition-aware and adaptive-switching neural networks.

Network quantization offers an effective solution to deep neural network compression for practical usage. Existing network quantization methods cannot theoretically guarantee the convergence. This paper proposes a novel iterative framework for network quantization with arbitrary bit-widths. We present two Lipschitz constraint based quantization strategies, namely width-level network quantization (WLQ) and multi-level network quantization (MLQ), for high-bit and extremely low-bit (ternary) quantization, respectively. In WLQ, Lipschitz based partition is developed to divide parameters in each layer into two groups: one for quantization and the other for re-training to eliminate the quantization loss. WLQ is further extended to MLQ by introducing layer partition to suppress the quantization loss for extremely low bit-widths. The Lipschitz based partition is proven to guarantee the convergence of the quantized networks. Moreover, the proposed framework is complementary to network compression methods such as activation quantization, pruning and efficient network architectures. The proposed framework is evaluated over extensive state-of-the-art deep neural networks, i.e., AlexNet, VGG-16, GoogleNet and ResNet18. Experimental results show that the proposed framework improves the performance of tasks like classification, object detection and semantic segmentation.

One-stage object detectors are trained by optimizing classification-loss and localization-loss simultaneously, with the former suffering much from extreme foreground-background class imbalance issue due to the large number of anchors. This paper alleviates this issue by proposing a novel framework to replace the classification task in one-stage detectors with a ranking task, and adopting the Average-Precision loss (AP-loss) for the ranking problem. Due to its non-differentiability and non-convexity, the AP-loss cannot be optimized directly. For this purpose, we develop a novel optimization algorithm, which seamlessly combines the error-driven update scheme in perceptron learning and backpropagation algorithm in deep networks. We verify good convergence property of the proposed algorithm theoretically and empirically. Experimental results demonstrate notable performance improvement in state-of-the-art one-stage detectors based on AP-loss over different kinds of classification-losses on various benchmarks, without changing the network architectures.

Many communication tasks have been formulated as optimization problems that can be solved by iterative algorithms. However, these algorithms are usually computationally intensive. To enable real-time processing of communication algorithms, in this paper, we propose a new deep neural network (DNN) architecture for algorithm approximation. Based on the idea of deep unfolding, we expand the iterative algorithm into a cascade network structure of basic blocks, with each network block corresponding to an iteration. Then, instead of directly approximating each iteration through network layers, we introduce the multivariate polynomial regression (MPR) as a bridge between the iterations and layers for constructing the unfolded network blocks. Specifically, we use a multivariate polynomial to approximate an iteration of the original iterative algorithm, and further construct a shallow neural network with one hidden layer to realize the polynomial, where the order of the polynomial provides theoretical guidance for determining the size of the network and controls the tradeoff between the approximation accuracy and computation speed. As empirical justifications, we apply the proposed DNN architecture in approximating the classic WMMSE algorithm for wireless interference management, showing that the proposed approach can achieve a good approximation accuracy with a much faster computation speed.

In this paper, we propose a deep reinforcement learning (DRL)-based rate adaptation algorithm for adaptive 360-degree video streaming, which is able to maximize the quality of experience of viewers by adapting the transmitted video quality to the time-varying network conditions. Specifically, to reduce the possible switching latency of the field of view (FoV), we design a new QoE metric by introducing a penalty term for the large buffer occupancy. A scalable FoV method is further proposed to alleviate the combinatorial explosion of the action space in the DRL formulation. Then, we model the rate adaptation logic as a Markov decision process and employ the DRL-based algorithm to dynamically learn the optimal video transmission rate. Simulation results show the superior performance of the proposed algorithm compared to the existing algorithms.

The head-mounted display (HMD) for 360-degree videos cannot be shared by multiple users, which significantly increases the bandwidth consumption when multiple users request the same video content simultaneously. This paper proposes a server-side hybrid multicast-unicast strategy for multi-user adaptive 360-degree video streaming, aiming to ensure the overall quality of experience (QoE) of the users in a bandwidth-constrained environment. A framework is established to realise this strategy, through which the delivery mode (multicast or unicast) and the corresponding bitrate for each tile can be jointly adapted to the dynamics of users' throughputs and field of views (FoVs). To maximize the overall QoE, a user clustering method is proposed to classify the users into several multicast clusters. Based on these clusters, we then formulate the joint delivery mode selection and rate adaptation problem as a non-linear integer programming problem which can be solved using the steepest ascent gradient algorithm. Finally, experiments are carried out to verify the performance of the proposed strategy.

Sending only the viewport of interest provides a solution for 360-degree video streaming under the current bandwidth-constrained infrastructure. To this end, the user viewport requires to be prefetched in advance by conducting viewport prediction. To more accurately capture the nonlinear and long-term dependent relation between the future and past viewports, we develop a single viewport prediction model using convolutional neural network (CNN), in which the pooling layers are dropped and more convolutional layers are added for stronger nonlinear fitting ability. Further, we design a viewport trajectory prediction model based on recurrent neural network (RNN) which learns long-term dependency in sequential viewports. Specially, it is capable to estimate future viewport trajectory and support variable-size prediction window with low complexity. Finally, a correlation filter-based viewport tracker (CFVT) is proposed to perform content-aware viewport prediction. The combination of the RNN and the CFVT through a fusion model enables them to complement each other which is validated by significant improvement in prediction accuracy.

The high-definition and panorama characteristics of 360-degree video make its transmission in existing networks more challenging. To alleviate this problem, this paper leverages transcoding-enabled edge caching in tile-granularity and makes joint caching, transcoding and delivery decisions among multiple edge servers for adaptive 360-degree video streaming. We formulate the joint caching, transcoding and delivery decision-making problem as a multivariate nonlinear integer programming problem, aiming at minimizing the aggregate network operational cost and balancing the server load, subject to storage and computation capacity constraints of the edge servers. Through decomposition and relaxation approach, the original intractable optimization problem is solved by an effective iterative algorithm. Finally, simulation results show that the proposed iterative algorithm outperforms the existing schemes in terms of resource utilization, cost reduction and server coordination.

In this paper, we propose a cross-layer decision framework for multiuser adaptive video delivery over time-varying and mutually interfering wireless cellular network. The key idea is to synthetically design the physical-layer optimization-based beamforming scheme (performed at the base stations) and the application-layer deep reinforcement learning (DRL)-based rate adaptation scheme (performed at the user terminals), so that a very complex multi-user overall fair long-term quality of experience (QoE) maximization problem can be decomposed to two layers and solved effectively. Extensive simulations show that the proposed cross-layer design is effective and promising.

Caching can effectively smooth the temporal traffic variability and decrease the redundant data transmission in mobile video delivery. In this paper, we consider a video caching system consisting of a video provider (VP), a mobile network operator (MNO) with a set of cache-enabled base stations (BSs), and multiple mobile users. The VP leases some popular videos to the MNO, while the MNO places these rented videos in local caches of its BSs to save expensive backhaul transmission cost. However, in such a two-sided market, these two entities are competing with each other for their own profit due to their opposite expectation on the video pricing. To address this, we model the competition between the two entities using the framework of Stackelberg games and propose a joint video pricing and cache placement strategy by considering the heterogeneity of video file sizes and exploiting the classic law of demand from the field of economics. The proposed optimization problem is able to jointly maximize the profit of the VP and the MNO by the optimal selection of the video pricing and the cache placement strategy given that price, for both noncooperative BS caching and cooperative BS caching cases. We then develop iterative algorithms based on dynamic programming and gradient ascent, respectively, for these two cases to find the Stackelberg equilibrium (SE). The simulation results further show that the proposed joint optimization formulation follows the law of demand in economics, and the proposed algorithms for both cases can efficiently converge to the SE point that jointly maximizes the profit for both the VP and the MNO.

Due to the popularity of 3D videos, 2D-to-3D video conversion has become a hot research topic for the past few years. The most critical issue in 3D video synthesis is the estimation of depth maps for the video frames. Numerous efforts have been devoted in fully automatic and semi-automatic depth estimation approaches, although the discontinuity of depth field and the ambiguity of motion boundary are still the main challenges in depth estimation. This paper proposes a semi-automatic structure-aware sparse-to-dense depth estimation method, which leverages the tensor voting at two different levels to propagate depth across frames. In the first level, a 4D tensor voting is performed to remove outliers caused by inaccurate motion estimation. Noticing that the 4D tensors of correctly matched points should lie on the smooth layer in the manifold, we utilize the variety saliency defined by the eigen-system of the tensor for outlier removal. In the second level, a high-dimensional tensor voting algorithm, incorporating spatial location, motion, and color into the tensor representation, is devised to propagate the depth from the sparse points to the entire image domain. By projecting the input feature into the tangent space, the relation between the location, motion, color, and the depth can be established by voting process. Extensive experiments on public data set validate the effectiveness of the proposed method in comparison with state-of-the-art depth estimation approaches.

The cognitive radio technique allows secondary users (SUs) to share the spectrum with primary users (PUs) in an exclusive or opportunistic manner. This paper studies spectrum pricing conducted by spectrum owners, that is, primary operators (POs), and SU decision-making strategies for three kinds of duopoly markets. The single-band exclusive use market considers two POs with each providing a single band dedicated to SUs. A pre-emptive resume priority (PRP) M/M/1 queueing model is presented, based on which SUs decide to join which PO and which queue. We prove the existence of a unique Wardrop equilibrium for the decision-making process, and a unique Nash equilibrium for the proposed parallel pricing strategy. In a single-band mixed use market, the competition of two POs is represented by a Stackelberg game. We formulate the spectrum sharing among PU and SUs with a 3-level PRP M/M/1 queueing structure, and derive the close form expressions of SUs' queueing delay. In a multiband exclusive use market, where POs have to determine how many bands they will rent as well as the admission price, we define the problem as a mixed integer linear programming problem and propose a global particle swarm optimization algorithm to find the global optimum. Finally, we study a generalized scenario with multiple POs and multiple priority queues.

Multi-channel data broadcast attracts increasingly focus in recent years. For the complex and large multimedia data like images, audios, and videos etc, multi-channel data broadcast is a promising approach to mitigate various limitations of data dissemination in mobile environment, such as narrow bandwidth, unreliable connections, and battery limitation. However, existing data broadcast schemes are inefficient for employing multiple channels. In this paper, we present five novel multimedia data broadcast schemes (SDAA, ISDAA, LDAA, AEA, and COA) specifically designed for wireless multichannel communications. The key idea behind those schemes is to utilize SVC (Scalable Video Coding) to generate data segments with different qualities and implement indexing and channel assignment to minimize the expected waiting time for clients. In terms of waiting time minimization, we prove theoretically that SDAA is a 4/3-approximation algorithm and ISDAA is a 6/5-approximation algorithm for global allocation, and AEA can achieve locally optimal solution. We show that integrating SDAA and AEA form a best schedule for practical applications. We provide numerical experiments by using a SVC dataset and a YouTube dataset to evaluate the system performance and prove the efficiency of our schemes. We also propose a multiple video broadcast framework and analyze its performance.

How to balance the tradeoff between the user experience and bandwidth utilization emerges a critical challenge for multi-user 360-degree video adaptive streaming. This paper studies the server-side rate adaptation strategy for multiple users which are competing for the server bandwidth capacity. A tile visibility probability model is established, by which the tiles are classified into predicted, marginal and invisible types. A fine-grained rate adaptation problem is formulated as a nonlinear integer programming (NIP) problem, which aims at maximizing the video quality and navigation smoothness for multiple users. Thereafter, a steepest ascent algorithm with feasible starting point is developed to solve the proposed NIP problem in polynomial time. Finally, simulation results verify the performance of the proposed rate adaptation strategy.

Dynamic adaptive streaming addresses user heterogeneity by providing multiple encoded representations at different rates and/or resolutions for the same video content. For delay-sensitive applications, such as live streaming, there is however a stringent requirement on the encoding delay, and usually the encoding power (or rate) budget is also limited by the computational (or storage) capacity of the server. It is therefore important, yet challenging, to optimally select the source coding parameters for each encoded representation in order to minimize the resource consumption while maintaining a high quality of experience for the users. To address this, we propose an optimization framework with an optimal representation selection problem for delay, power, and rate constrained adaptive video streaming. Then, by the optimal selection of source coding parameters for each selected representation, we maximize the overall expected user satisfaction, subject not only to the encoding rate constraint, but also to the delay and power constraints at the server. We formulate the proposed optimization problem as an integer linear program formulation to provide the performance upper bound, and as a submodular maximization problem with two knapsack constraints to develop a practically feasible algorithm. Simulation results show that the proposed weighted rate and power cost benefit greedy algorithm is able to achieve a near-optimal performance with very low time complexity. In addition, it can strike the best tradeoff both between the rate and power cost, and between the algorithm's performance and the delay requirements proposed by delay sensitive applications.

Caching at mobile edge servers can smooth temporal traffic variability and reduce the service load of base stations in mobile video delivery. However, the assignment of multiple video representations to distributed servers is still a challenging question in the context of adaptive streaming, since any two representations from different videos or even from the same video will compete for the limited caching storage. Therefore, it is important, yet challenging, to optimally select the cached representations for each edge server in order to effectively reduce the service load of base station while maintaining a high quality of experience (QoE) for users. To address this, we study a QoE-driven mobile edge caching placement optimization problem for dynamic adaptive video streaming that properly takes into account the different rate-distortion (R-D) characteristics of videos and the coordination among distributed edge servers. Then, by the optimal caching placement of representations for multiple videos, we maximize the aggregate average video distortion reduction of all users while minimizing the additional cost of representation downloading from the base station, subject not only to the storage capacity constraints of the edge servers, but also to the transmission and initial startup delay constraints of the users. We formulate the proposed optimization problem as an integer linear program to provide the performance upper bound, and as a submodular maximization problem with a set of knapsack constraints to develop a practically feasible cost benefit greedy algorithm. The proposed algorithm has polynomial computational complexity and a theoretical lower bound on its performance. Simulation results further show that the proposed algorithm is able to achieve a near-optimal performance with very low time complexity. Therefore, the proposed optimization framework reveals the caching performance upper bound for general adaptive video streaming systems, while the proposed algorithm provides some design guidelines for the edge servers to select the cached representations in practice based on both the video popularity and content information.

In this paper we consider the time-varying characteristics of practical wireless networks and propose a joint dynamic rate allocation and transmission scheduling optimization scheme for scalable video multirate multicast based on opportunistic routing (OR) and network coding. With OR the decision of optimal routes for scalable video coding layered streaming is integrated into the joint optimization formulation. The network throughput is also increased by taking advantage of the broadcast nature of the wireless shared medium and by network coding operations in intermediate nodes. To maximize the overall video reception quality among all destinations the proposed scheme can jointly optimize the video reception rate the associated routes to different destinations and the time fraction scheduling of transmitter sets that are concurrently transmitting in the shared wireless medium. By using dual decomposition and primal-dual update approach we develop a cross-layer algorithm in a fully distributed manner. Simulation results demonstrate significant network multicast throughput improvement and adaptation to dynamic network changes relative to existing optimization schemes.

Real-time video services are usually delay sensitive and have strict constraints on the transmission reliability, which poses challenges to live video streaming over multi-hop wireless networks, since the unpredictable packet losses and network congestions caused by time-varying wireless channels greatly degrade the received video quality. To address this, in this paper, we propose a reinforcement learning (RL)-based opportunistic routing (OR) scheme for wireless video streaming with high-reliability and low-delay requirements. It can exploit the broadcast nature of the wireless shared medium and path diversity through OR to improve the transmission reliability, and find the low-delay paths between the source-destination pair dynamically for video packets through the RL module embedded in each relay node. Specifically, we design for the OR a new path-cost metric called the expected anypath delay (EAD), to estimate the end-to-end delay of a packet between the current relay node and the destination. The EAD is dynamically measured and updated over time, thereby reflecting the changes of link quality and the congestion level at the relay node. Moreover, we utilize the ACK message to piggyback the EAD of each relay node to its previous-hop node. Based on the local communication of the EADs from the neighbors, each node in the network can iteratively and independently run the RL module to update its own EAD value. Then, the next-hop forwarder node on a low delay route can be determined by assigning higher relay priority to the candidate forwarder nodes with lower EADs in OR. Simulation results show that the proposed RLOR algorithm can achieve a proper tradeoff between the transmission reliability and latency, so as to support the low-delay transmission of wireless video streams with high received video quality.

This study addresses the joint pricing and service selection problem between two primary operators (POs) and multiple secondary users (SUs). The SUs are divided into two priority classes in terms of their delay‐sensitive demands. A M/G/1 queueing model of four queues is established, in which each PO serves two queues to support SUs' heterogeneous sensitivity to price and delay. A unique Wardrop equilibrium point of the proposed service selection strategy is developed, at which not only the overall cost perceived by SUs at two POs are equal; however, the cost perceived by SUs of the same PO in different class are equal. Moreover, the authors propose two pricing strategies for maximising the revenue of POs. The first one is that both POs set their prices simultaneously, the second one formulates the price competition as a two‐stage Stackelberg game, and then the authors show the existence of a unique Nash equilibrium. The performance of the proposed mechanism is finally verified by numerical results.

Dictionary learning has emerged as a promising alternative to the conventional hybrid coding framework. However, the rigid structure of sequential training and prediction degrades its performance in scalable video coding. This paper proposes a progressive dictionary learning framework with hierarchical predictive structure for scalable video coding, especially in low bitrate region. For pyramidal layers, sparse representation based on spatio-temporal dictionary is adopted to improve the coding efficiency of enhancement layers with a guarantee of reconstruction performance. The overcomplete dictionary is trained to adaptively capture local structures along motion trajectories as well as exploit the correlations between the neighboring layers of resolutions. Furthermore, progressive dictionary learning is developed to enable the scalability in temporal domain and restrict the error propagation in a closed-loop predictor. Under the hierarchical predictive structure, online learning is leveraged to guarantee the training and prediction performance with an improved convergence rate. To accommodate with the state-of-the-art scalable extension of H.264/AVC and latest High Efficiency Video Coding (HEVC), standardized codec cores are utilized to encode the base and enhancement layers. Experimental results show that the proposed method outperforms the latest scalable extension of HEVC and HEVC simulcast over extensive test sequences with various resolutions.

To enable robust multiview video streaming over current wireless broadcast infrastructures, this paper proposes a joint source and channel coding optimization framework for video streaming based upon multiview video coding technique. It utilizes random linear coding (RLC) with ladder-shaped global coding coefficient matrix (GCM) for unequal erasure protection. An accurate expression of the decoding probability for RLC with ladder-shaped GCM is derived. A fast two-layer local search algorithm is developed to solve the source and channel coding rate allocation problem, where the source and channel rate of each layer can be adjusted independently for the overall video quality. Simulation results demonstrate significant video quality improvement to existing rate allocation schemes.

Caching at edge servers can smooth the temporal traffic variability and reduce the service load of base stations in wireless streaming. However, the assignment of the cached content for possibly multiple versions of different video sequences is still a challenging question in the context of adaptive streaming. In this paper, we propose a wireless video caching placement optimization problem for dynamic adaptive video streaming that properly takes into account the different rate-distortion (R-D) characteristics of the video sequences. Our objective is to minimize the expected video distortion by the optimal caching of compressed video sequences, such that the clients can effectively access video content storage and bandwidth constraints. We prove that our optimization problem is a submodular maximization problem subject to a knapsack constraint. A cost benefit greedy algorithm is developed to obtain an approximate solution with polynomial time complexity and theoretical approximation guarantees. Simulation results demonstrate significant video distortion reduction relative to different baseline caching placement schemes.

In this paper, we propose a representation selection optimization problem for complexity constrained adaptive video streaming that properly takes into account the different complexity-rate-distortion (C-R-D) characteristics of the videos when implementing rate control for desired representations. Our objective is to maximize the expected video distortion reduction of users, subject not only to encoding rate constraints, but also to complexity constraints. We prove that our optimization problem is a submodular maximization problem with two knapsack constraints. A weighted rate and complexity cost benefit greedy algorithm is then developed to obtain an approximate solution with polynomial time complexity and good approximation performance in simulations.

Classical dictionary learning methods for video coding suffer from high computational complexity and interfered coding efficiency by disregarding its underlying distribution. This paper proposes a spatio-temporal online dictionary learning (STOL) algorithm to speed up the convergence rate of dictionary learning with a guarantee of approximation error. The proposed algorithm incorporates stochastic gradient descents to form a dictionary of pairs of 3D low-frequency and high-frequency spatio-temporal volumes. In each iteration of the learning process, it randomly selects one sample volume and updates the atoms of dictionary by minimizing the expected cost, rather than optimizes empirical cost over the complete training data, such as batch learning methods, e.g., K-SVD. Since the selected volumes are supposed to be independent identically distributed samples from the underlying distribution, decomposition coefficients attained from the trained dictionary are desirable for sparse representation. Theoretically, it is proved that the proposed STOL could achieve better approximation for sparse representation than K-SVD and maintain both structured sparsity and hierarchical sparsity. It is shown to outperform batch gradient descent methods (K-SVD) in the sense of convergence speed and computational complexity, and its upper bound for prediction error is asymptotically equal to the training error. With lower computational complexity, extensive experiments validate that the STOL-based coding scheme achieves performance improvements than H.264/AVC or High Efficiency Video Coding as well as existing super-resolution-based methods in rate-distortion performance and visual quality.

In the past few decades, we have witnessed the success of bag-of-features (BoF) models in scene classification, object detection, and image segmentation. Whereas it is also well acknowledged that the limitation of BoF-based methods lies in the low-level feature encoding and coarse feature pooling. This paper proposes a novel scene classification method, which leverages several semantic codebooks learned in a multitask fashion for robust feature encoding, and designs a context-aware image representation for efficient feature pooling. Apart from conventional universal codebook learning approaches, the proposed method encodes each class of local features with a unique semantic codebook, which captures the distinct distribution of different semantic classes more effectively. Instead of learning each semantic codebook separately, we learn a compact global codebook, of which each semantic codebook is a sparse subset, with a two-stage iterative multitask learning algorithm. While minimizing the clustering divergence, the semantic codeword assignment is solved by submodular optimization simultaneously. Built upon the global and semantic codebooks, a context-aware image representation is further developed to encode both global and semantic features in image representation via contextual quantization, semantic response computation, and semantic pooling. Extensive experiments have been conducted to validate the effectiveness of the proposed method on various public benchmarks with several popular local features.

This paper proposes a multiview video scheduling scheme through asymmetric Nash bargaining for a tradeoff between user satisfaction and view fairness. The interactions of view streams queueing at the scheduler are modeled as a bargaining problem, where packets of different views share the transmission opportunities via negotiation. A utility function based upon the viewing angle is defined to indicate whether a view is desired by a user, then a network utility maximization problem is solved by the Nash bargaining solution (NBS). The impacts of the bargaining powers on both network performance and view fairness are analytically investigated. Simulation results are provided to compare the performances of three scheduling schemes including fair scheduling, symmetric bargaining, and proportional scheduling, which can be achieved by adjusting the bargaining powers.

Cooperative cognitive radio (CR) improves quality of service (e.g. rate) of primary users by cooperative transmission from secondary uses. However, as owners of the spectrum, primary users' traffic demands are relatively easy to be satisfied. The rationale is that they would be more interested in the benefits of other format (e.g. revenue), instead of the enhanced rate. In this paper, we propose a new cooperative CR framework, where primary users assist transmissions of secondary users. In exchange for this concession, primary users receive the payments of secondary users for the spectrum and cooperative transmit power being used in cooperation. An auction-theoretic model with multiple auctioneers, multiple bidders and multiple commodities is developed for a joint spectrum and cooperative power allocation. Finally, we mathematically prove the convergence of the proposed auction, and verify the performance of the proposed scheme by numerical results.

In the existing cooperative cognitive radio (CR), qualities of service (e.g., rate or outage probability) of primary users can be improved by cooperative transmission from secondary users. However, as owners of the spectrum, primary users' traffic demands are relatively easy to satisfy. The rationale is that they would be more interested in the benefits of other formats (e.g., revenue), rather than the enhanced rate. In this paper, we propose a new cooperative CR framework, where primary users assist in the transmissions of secondary users. In exchange for this concession, primary users receive payments from secondary users for the spectrum and cooperative transmit power being used in cooperation. An auction-theoretic model with multiple auctioneers, multiple bidders, and multiple commodities is developed for a joint spectrum and cooperative power allocation. Since the spectrum and power are two heterogeneous but correlated commodities, their properties are specifically considered in auction strategy design. Finally, we mathematically prove the convergence of the proposed auction game, and show with numerical results, that the proposed auction is beneficial to both primary and secondary users.

In this paper, we study the problem of correlated data gathering in wireless sensor networks. An opportunistic routing protocol joint with opportunistic network coding is proposed for correlated data gathering. In order to reduce the total transmission, we introduce the expected number of coded transmission (ECTX) as metric for the routing selection. Moreover, we also define the coding gain for the network nodes to count the coding opportunistic, with which the throughput of the wireless sensor network is further increased. Numerical simulations are conducted for performance analysis with the proposed routing scheme.

The overall performance of a cooperative relaying system largely depends on power allocation schemes. In this paper, we address the power allocation problem in a network-coded multiuser two-way relaying network, where multiple pair users communicate with their partners via a common relay node, and compete for the transmit power of the relay. An auction-based power allocation scheme is proposed, in which two users in each pair bid as a single player for a maximum utility of the whole pair, and share the total pair payment in proportion to the amount of power they obtained. The convergence of the proposed auction game (i.e., the convergence to a unique Nash Equilibrium) is theoretically proved by using a standard function. Moreover, the outage behavior is systematically analyzed and a closed form of the pair outage probability is derived. Finally, the performance of the proposed scheme is verified by simulation results.

This paper proposes a distributed robust optimization scheme to jointly optimize overall video quality and traffic performance for scalable video multirate multicast over practical wireless networks. In order to guarantee layered utility maximization, the initial nominal joint source and network optimization is defined, where each scalable layer is tailored in an incremental order and finds jointly optimal multicast paths and associated rates with network coding. To enhance the robustness of the nominal convex optimization formulation with nonlinear constraints, we reserve partial bandwidth for backup paths disjoint from the primal paths. It considers the path-overlapping allocation of backup paths for different receivers to take advantage of network coding, and takes into account the robust multipath rate-control and bandwidth reservation problem for scalable video multicast streaming when possible link failures of primary paths exist. Specifically, an uncertainty set of the wireless medium capacity is introduced to represent the uncertain and time-varying property of parameters related to the wireless channel. The targeted uncertainty in the robust optimization problem is studied in a form of protection functions with nonlinear constraints, to analyze the tradeoff between robustness and distributedness. Using the dual decomposition and primal-dual update approach, we develop a fully decentralized algorithm with regard to communication overhead. Through extensive experimental results under critical performance factors, the proposed algorithm could converge to the optimal steady-state more quickly, and adapt the dynamic network changes in an optimal tradeoff between optimization performance and robustness than existing optimization schemes.

In this paper, we address the problem of access to global information from any single point in wireless sensor networks. To this end, we propose a distributed in-network data acquisition approach, on the basis of compressive sensing, in which sparse random projections and randomized gossiping are jointly designed. In the context of unreliable distributed wireless settings, a simple random gossip algorithm is adopted. It allows the sensor to forward the linear combined data (readings) to randomly chosen neighbors. Also, each transmission path of the combined data is mapped to one row of the projection matrix, forming low-cost sparse random projections. Moreover, a theoretical model is developed that exactly characterizes the relationship of the number of sparse projections, the degree of the sparsity, and the error probability. Finally, simulation results show that, compared with the conventional approach, the proposed algorithm can achieve better reconstruction quality with less communication overhead.

In this paper, we consider the power allocation problem in a network coded multiuser two-way relaying network, where multiple pairs of users exchange information with the help of a relay node. An auction-based power allocation scheme is proposed to solve the power competition among multiple user pairs. To maximize the utility of the whole pair, each two users in the pair bid as a single player, with the total pair payment shared by the pair users in proportion to the amount of power they obtained. The convergence of the proposed auction game is theoretically proved by using a standard function. Moreover, the outage behavior is systematically analyzed and a closed form of the system outage probability is derived. Finally, the performance of the proposed scheme is verified by the simulation results.

This paper studies the problem of correlated data gathering in wireless sensor networks. For a maximum network utility, a efficient data capture and transmission framework is proposed for correlated sources, where localized S-W source coding, network coding based flow control and opportunistic routing are jointly optimized. To increase the throughput and guarantee the decodability simultaneously, a dynamic network coding strategy is proposed, with which the intermediate node can easily decide whether to make a combination among the incoming flows. Also, an opportunistic routing approach is presented, which adopts a new metric (minimum congestion price) for forwarding node selection and results in a maximum utility benefit for the sensor nodes. Through the Lagrange dual and gradient approach, a fully distributed algorithm is represented. And the convergence and performance are validated by the numerical results.

Thanks to recent advances in hierarchical video coding and overlay network, layered overlay multicast has emerged as an important solution for video streaming over heterogeneous network. In this paper, we formulate it into a joint network flow control and performance optimization problem where adaptive layer rates are determined through a greedy algorithm close to optimal. The overlay network is constrained by a practical two-level hierarchical overlay model where the bandwidth sharing is imposed on underlying edge-bottleneck links. To maintain less computational complexity and avoidance of global information, the M-layer maximization problem is transformed into multiple one-layer minimization subproblems. In turn, a fully distributed algorithm is developed by decomposing the one-layer minimization problem into both inequality constrained transportation subproblem and the shortest path subproblem by using Lagrangian duality. We demonstrate that the distributed algorithm converges quickly and attains effective performance in both static and dynamic networks.

This paper aims to optimize the overall video quality and traffic performance for multi-rate video multicast over hybrid wired/wireless networks. In order to perform layered utility maximization over tiered networks, we propose a joint source-network flow optimization scheme where individual layers of the scalable video stream are imposed on their optimal multicast paths and associated rates for the highest sustainable layered video quality with minimum costs. It sufficiently guarantees that each destination node accesses progressive layered stream in an incremental order, considers network coding across overlapping paths to destination nodes for decent multicast capacity, and addresses the link contention problem during wireless transmission. We formulate the problem into convex programming with the objective to minimize the total rate-distortion variations between layers. Using primal decomposition and the primal-dual approach, we develop a decentralized algorithm with two levels of optimization. The numerical and packet-level results compare extensive performance under different control conditions over coded and non-coded hybrid networks. It demonstrates that the proposed algorithm could actually achieve the max-flow throughput and provide better video quality with optimal layered access for heterogeneous receivers.

This paper studies a joint coding/routing optimization between network lifetime and video distortion by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding [distributed video coding and network coding (NC)] from both combinatorial optimization and information theory could make significant progress toward the performance limit and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps the total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Also, a generalized power consumption model for distributed video sources is developed, in which the coding complexity of Key frames and Wyner-Ziv frames is measured by translating specific coding behavior into energy consumption. On the basis of the distributed multiview video coding and NC-based multipath routing, the balance problem between lifetime (costs) and distortion (capacity) is modeled as an optimization formulation with a fully distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved by the gradient algorithm. The low-level optimization problem is further decomposed into a secondary master dual problem with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. The implementation of the distributed algorithm is discussed with regard to the communication overhead and dynamic network change. Simulation results validate the convergence and performance of the proposed algorithm.

In this paper, we study performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Content priority of video coding layers is considered during the flow routing to determine the optimal multicast paths and associated data rates for each layer. Our optimization scheme attempts to find content distribution meshes with minimum path costs for each video coding layer while satisfying the inter-layer dependency during scalable video coding. Based on primal decomposition and primal-dual analysis, we develop a decentralized algorithm with two optimization levels to solve the performance optimization problem. We also prove the stability and convergence of the proposed iterative algorithm using Lyapunov theories. Extensive experimental results demonstrate that the proposed algorithm not only achieves the max-flow throughput using network coding, but also provides better video quality with balanced layered access for heterogeneous receivers.

This paper aims to optimize joint overall video quality and traffic performance of multirate multicast of scalable video streaming accessing hybrid wired/wireless paths. In order to guarantee layered utility maximization over tiered wired/wireless coded networks, we propose a joint source and network flow optimization scheme where each scalable layer is tailored in an incremental order and finds jointly optimal multicast paths and associated rates. It not only achieves the highest sustainable layered video quality, but also takes into account both path-overlapping allocation for different receivers which takes advantage of network coding and the wireless link contention in a shared transmission medium. The convex programming is imposed on the total variation minimization of layered rate-distortion. Using primal decomposition and the primal-dual approach, we develop a decentralized algorithm with two levels of optimization. Experimental results demonstrate that the proposed algorithm not only provides better video quality with optimal layered access for heterogeneous receivers, but benefits the asymptotic capacity of a hierarchical "hybrid" wired/wireless architecture in a scalable sense.

This paper studies the power consumption performance and resource allocation optimization in wireless video sensor networks. Network coding based multipath routing, network flow control and video encoding bit rate are jointly optimized, aiming to maximize the network lifetime at a given power budget and video quality requirement. Importantly, to concretely measure the network coding power utilized on error recovery, a generalized power consumption model for network coding is first developed in this paper. Through the Lagrange dual and subgradient approach, a fully decentralized algorithm is proposed to solve the target convex optimization problem. Numerical results validate the convergence and performance of the proposed algorithm.

In this paper, we address the joint optimization problem of rate allocation and distortion control for scalable video coding (SVC) multicast networks. Firstly, we construct video distribution meshes by coupling network coding and multipath routing with multi-rate control to seek optimal routing paths and associated transmission rates. Secondly, a particular minimum bandwidth consumption scheme is presented for each video layer to reduce video distortion, where the content priority of the base layer and the minimum bandwidth consumption at higher layers are taken into account. Finally, a convex mathematic model combining these considerations is proposed. Through decomposition and dual approach, the target convex optimization problem is solved by a fully decentralized algorithm which is decomposed into a two-level optimization procedure. Simulation results validate the convergence behavior and performance of the proposed algorithm.

In this paper, we study the optimal bandwidth allocation for scalable video coding (SVC) streaming in multiple overlays. We model the whole bandwidth request and distribution process as a set of decentralized auction games between the competing peers. For the upstream peer, a bandwidth allocation mechanism is introduced to maximize the aggregate revenue. For the downstream peer, a dynamic bidding strategy is proposed. It achieves maximum utility and efficient resource usage by collaborating with a content-aware layer dropping/adding strategy. Also, the convergence of the proposed auction games is theoretically proved. Experimental results show that the auction strategies can adapt to dynamic join of competing peers and video layers.

This paper studies a joint coding/routing optimization between network lifetime and rate-distortion, by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding (distributed source coding and network coding) from both combinatorial optimization and information theory could make significant progress towards the performance limit of information networks and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps a total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Based on the localized Slepian-Wolf coding and network coding-based multipath routing, the balance problem between distortion (capacity) and lifetime (costs) is modeled as an optimization formulation with a distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved with the gradient algorithm. The low-level optimization problem is decomposed into a secondary master dual problem (encoding, energy, and congestion prices update) with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. Numerical results validate the convergence and performance of the proposed algorithm.

This paper addresses the performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Importantly, contextual priors of scalable video layers are imposed on the flow routing optimization problem, seeking to guarantee the transmission cost for each layer in an incremental order and find jointly optimal multicast paths and associated rates. Through a primal decomposition and the primal-dual approach, a decentralized algorithm with two-level optimization update is developed to solve the target convex optimization problem. Numerical and simulation results validate the convergence and network performance of the proposed algorithm.

In this paper, we seek the optimal overlay multicast performance for scalable video streaming. To adapt to layered multicast with multiple paths, a new metric layer stretch is introduced, measuring video layer's dissemination latency along the overlay. Multi-path routing, network coding based layer subscription and congest control are jointly optimized, formulating a linear programming to minimize the average layer stretch over all receivers. It not only solves the issue of inter-layer dependency of scalable video coding, but allows each receiver to determine the required video quality. Moreover, a heuristic algorithm that can be implemented in a distributed manner is provided to approximate the optimal solution. Simulation results validate the network performance of the proposed scheme.