Recent advancements in weakly-supervised video anomaly detection have achieved remarkable performance by applying the multiple instance learning paradigm based on multimodal foundation models such as CLIP to highlight anomalous instances and classify categories. However, their objectives may tend to detect the most salient response segments, while neglecting to mine diverse normal patterns separated from anomalies, and are prone to category confusion due to similar appearance, leading to unsatisfactory fine-grained classification results. Therefore, we propose a novel Disentangled Semantic Alignment Network (DSANet) to explicitly separate abnormal and normal features from coarse-grained and fine-grained aspects, enhancing the distinguishability. Specifically, at the coarse-grained level, we introduce a self-guided normality modeling branch that reconstructs input video features under the guidance of learned normal prototypes, encouraging the model to exploit normality cues inherent in the video, thereby improving the temporal separation of normal patterns and anomalous events. At the fine-grained level, we present a decoupled contrastive semantic alignment mechanism, which first temporally decomposes each video into event-centric and background-centric components using frame-level anomaly scores and then applies visual-language contrastive learning to enhance class-discriminative representations. Comprehensive experiments on two standard benchmarks, namely XD-Violence and UCF-Crime, demonstrate that DSANet outperforms existing state-of-the-art methods.

Text-prompted image segmentation enables fine-grained visual understanding and is critical for applications such as human-computer interaction and robotics. However, existing supervised fine-tuning methods typically ignore explicit chain-of-thought (CoT) reasoning at test time, which limits their ability to generalize to unseen prompts and domains. To address this issue, we introduce LENS, a scalable reinforcement-learning framework that jointly optimizes the reasoning process and segmentation in an end-to-end manner. We propose unified reinforcement-learning rewards that span sentence-, box-, and segment-level cues, encouraging the model to generate informative CoT rationales while refining mask quality. Using a publicly available 3-billion-parameter vision–language model, i.e., Qwen2.5-VL-3B-Instruct, LENS achieves an average cIoU of 81.2% on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks, outperforming the strong fine-tuned method, i.e., GLaMM, by up to 5.6%. These results demonstrate that RL-driven CoT reasoning significantly enhances text-prompted segmentation and offers a practical path toward more generalizable Segment Anything models (SAM).

This paper proposes a novel 3D neural network for RGB-D image compression, which treats depth as a temporal dimension of RGB and leverages 3D convolutions for unified cross-modal modeling. Specifically, the network employs a parallel Hybrid CNN and Gated SwinT-Mamba (HCGSM) module to jointly capture multi-scale local details and long-range global dependencies. Furthermore, it introduces a channel-wise autoregressive entropy model enhanced with a lightweight Feature Enhanced Attention (FEAtten) module to improve latent representation coding efficiency and preserve structural fidelity.

This paper proposes an image compression model based on high-low frequency decomposition and structural priors, as illustrated in Fig. 1. Specifically, we first employ Discrete Wavelet Transform (DWT) to decompose the image into high- and low-frequency components. Simultaneously, a Structural Prior Feature Extraction (SPFE) module is adopted to extract inherent structural information from medical images, which is incorporated into both the encoder and the entropy model at multiple scales as prior knowledge. Additionally, we design a Grayscale Feature Extraction (GFE) module by performing directional pooling and feature fusion on the image to further capture the gradually varying intensity characteristics commonly present in medical images.

Compound Facial Expressions consist of multiple basic expressions and present increased complexity in recognition tasks. Enhancing the representation of primary and secondary expressions is critical for improving the classification performance. However, when features are enhanced indiscriminately, irrelevant information may also be amplified, leading to the ‘Indiscriminate Enhancement Trap' and robustness degradation. To address these issues, we propose a novel Bi-Enhancement based Disentangled Dual-Attention (BED2A) Framework. The bi-enhancement strategy is devised to achieve signal complementarity between channel and spatial features. By transferring semantic information from channel attention to spatial attention in the latent space, the proposed method enhances the ability to localize salient regions. Subsequently, primary and secondary expression features are disentangled across tri-branch feature enhancement architecture, thereby enabling more effective dual-attention based classification. Specifically, the Differential Cross-Consistency mechanism is introduced to disentangle inter-expression features and enhance the precision of expression representations. Dual-Attention mechanism leverages attention to primary and secondary expressions to optimize the classification center and improve inter class separability. Experimental results demonstrate that BED2A effectively optimizes feature enhancement to improve robustness, achieving state-of-the-art performance.

The Geometric Partitioning Mode (GPM) in Versatile Video Coding (VVC) significantly enhances inter prediction efficiency by employing non-rectangular partitions. However, the Motion Vectors (MVs) for these irregular sub-partitions still follow the derivation of rectangular blocks, which often leads to weakly correlated MV candidates and motion mismatch. To address this issue, we propose a unified Geometric Centroid-based Correlation Model (GCCM) to quantify the correlation between GPM sub-partitions and their neighboring regions. By jointly considering geometric centrality and boundary connectivity, GCCM provides an effective metric to guide motion derivation for irregular sub-partitions. Based on GCCM, we redesign the Temporal MV Prediction (TMVP) derivation at the sub-partition level using the most correlated collocated position, and further perform correlation-guided Template Matching (TM) for MV refinement. Experimental results on VTM-12.0 demonstrate that the proposed method achieves 1.94% and 1.14% bit rate savings on Low-delay B and Random Access configurations, respectively.

Federated learning (FL) has emerged as a promising paradigm for collaborative medical image segmentation across institutions while preserving data privacy. Despite great efforts in addressing cross-client annotation heterogeneity FL, the prevalent annotation completeness heterogeneity in clinical practice due to varying diagnostic priorities has been completely overlooked, hindering the deployment of FL. In this paper, we formulate such a challenge and propose FedHAC for incompleteness-robust medical image segmentation. FedHAC consists of three modules, i.e., Global Class Prototype Alignment (GCPA), Annotation Completeness-Aware Aggregation (ACAA), and GMM-driven Progressive Correction (GPC). Specifically, GCPA constructs a noise-resilient warm-up model through proximal-term regularization and prototype alignment. ACAA estimates client-wise annotation completeness and dynamically prioritizes high-quality clients. GPC groups clients into "noisy" and "clean" via GMM for progressive annotation correction to minimize error propagation. Extensive comparison experiments and ablation studies on public datasets demonstrate the superiority of FedHAC over state-of-the-art methods under various levels of annotation incompleteness.

Despite the promising potential of multi-modal learning in medical image segmentation, real-world applications often encounter modal incompleteness sourced from diverse domains and institutions, sparking significant discussions on incomplete multi-modal learning. Existing approaches either train a unified model for all or develop individual models for specific multi-modal combinations to ensure model fairness and robustness during inference. However, the assumption of complete multi-modal data for training is unrealistic and infeasible in clinical practice. In this paper, we thoroughly formulate such a challenging setting and propose hierarchical gradient alignment (HGA) to address uni- and multi-modal imbalance. Specifically, gradient direction is aligned through sequential meta learning for multi-modal combinations and multi-level self-distillation for uni-modals within each combination. Gradient magnitude is aligned based on relative preference estimation to balance the dominance of each modal during training. Extensive experiments on five public benchmarks (BraTS2018, BraTS2020, BraTS2023, MyoPS2020, and MSSEG2016) demonstrate that HGA consistently outperforms state-of-the-art incomplete and imbalanced multi-modal learning methods, as well as representative multi-task learning optimization techniques. More importantly, HGA is validated to work as plug-and-play modules for consistent performance improvement across different backbones. Code is available at https://github.com/Jun-Jie-Shi/HGA.

In recent years, semantic segmentation has flourished in various applications. However, the high computational cost remains a significant challenge that hinders its further adoption. The filter pruning method for structured network slimming offers a direct and effective solution for the reduction of segmentation networks. Nevertheless, we argue that most existing pruning methods, originally designed for image classification, overlook the fact that segmentation is a location-sensitive task, which consequently leads to their suboptimal performance when applied to segmentation networks. To address this issue, this paper proposes a novel approach, denoted as Spatial-aware Information Redundancy Filter Pruning (SIRFP), which aims to reduce feature redundancy between channels. First, we formulate the pruning process as a maximum edge weight clique problem (MEWCP) in graph theory, thereby minimizing the redundancy among the remaining features after pruning. Within this framework, we introduce a spatial-aware redundancy metric based on feature maps, thus endowing the pruning process with location sensitivity to better adapt to pruning segmentation networks. Additionally, based on the MEWCP, we propose a low computational complexity greedy strategy to solve this NP-hard problem, making it feasible and efficient for structured pruning. To validate the effectiveness of our method, we conducted extensive comparative experiments on various challenging datasets. The results demonstrate the superior performance of SIRFP for semantic segmentation tasks.

How can we enable models to comprehend video anomalies occurring over varying temporal scales and contexts? Traditional Video Anomaly Understanding (VAU) methods focus on frame-level anomaly prediction, often missing the interpretability of complex and diverse real-world anomalies. Recent multimodal approaches leverage visual and textual data but lack hierarchical annotations that capture both short-term and long-term anomalies. To address this challenge, we introduce HIVAU-70k, a large-scale benchmark for hierarchical video anomaly understanding across any granularity. We develop a semi-automated annotation engine that efficiently scales high-quality annotations by combining manual video segmentation with recursive free-text annotation using large language models (LLMs). This results in over 70,000 multi-granular annotations organized at clip-level, event-level, and video-level segments. For efficient anomaly detection in long videos, we propose the Anomaly-focused Temporal Sampler (ATS). ATS integrates an anomaly scorer with a density-aware sampler to adaptively select frames based on anomaly scores, ensuring that the multimodal LLM concentrates on anomaly-rich regions, which significantly enhances both efficiency and accuracy. Extensive experiments demonstrate that our hierarchical instruction data markedly improves anomaly comprehension. The integrated ATS and visual-language model outperform traditional methods in processing long videos. Our benchmark and model are publicly available at https://github.com/pipixin321/HolmesVAU.

This paper aims to reduce the complexity of coding 360-degree videos using VVC. Since 360-degree video undergoes horizontal stretching and exhibits flat, horizontal texture characteristics in the bipolar region after projection to the ERP format, we propose a novel fast intra mode decision (FIMD) algorithm for 360-degree video, which is based on our previously proposed lightweight neural network [1] (LNN).

With the growing demand for machine analysis of images and videos, there is a strong need for new codecs that support both human and machine vision. However, existing scalable codecs suffer from redundancy between the enhancement and base layers, while scalable codecs for human-machine hybrids struggle to provide sufficiently high subjective image quality at low bit rates. To address these two issues, we propose a method to reduce redundancy by constraining the redundant information between the enhancement and base layer bit streams. We present a decoder based on a conditional diffusion model that significantly improves subjective image quality by enhancing the fidelity of human perception, especially at low bit rates. Our method enforces orthogonality during training and employs a channel-based entropy model to improve latent variable distribution prediction. Experimental results show that our method achieves up to 81.96% gain in bit rate and subjective image quality comparable to current state-of-the-art techniques in machine vision tasks.

Deep learning-based brain tumor segmentation on multi-sequence magnetic resonance imaging (MRI) has gained widespread attention due to its great potential in supporting brain disease diagnosis. Although, compared to single-sequence images, more information is available from multi-sequence MR images, noise and artifacts on any given MR sequence can result in significant performance degradations. As in clinical routine, it is not always possible to maintain high imaging quality across all MR sequences (e.g., foreign bodies, ventricular drainage, shunts, involuntary patient motion, etc.), ensuring robustness of brain tumor segmentation from multi-sequence MR images is of great importance in clinical practice, but rarely explored. Accordingly, in this paper, we propose a robust brain tumor segmentation framework to mitigate the performance degradation caused by noise and artifacts on multi-sequence MR images. Specifically, based on semantic affinity, we propose a unique cross-sequence semantic interaction module (CSSI) to exploit inter-sequence correlations and extract noise-resilient features. In addition, we incorporate a batch-level covariance mechanism to suppress the redundant background information and improve the semantic enhancement effect of the CSSI module. In order to further improve segmentation performance, we also incorporate a sequence-level variance regularization mechanism to exploit sequence-specific features. To validate the robustness of ROXSI, brain tumor segmentation performance was evaluated under the existence of four common artifacts, at five different perturbation levels. We further performed a blinded qualitative clinical evaluation with two experienced neuro-radiologists, evaluating results from ROXSI and other popular CNN and Transformer-based segmentation models. Experimental results on two benchmark datasets demonstrate the superior robustness of ROXSI over other state-of-the-art segmentation methods.

Segment anything model (SAM), a foundation model with superior versatility and generalization across diverse segmentation tasks, has attracted widespread attention in medical imaging. However, it has been proved that SAM would encounter severe performance degradation due to the lack of medical knowledge in training and local feature encoding. Though several SAM-based models have been proposed for tuning SAM in medical imaging, they still suffer from insufficient feature extraction and highly rely on high-quality prompts. In this paper, we propose a powerful foundation model SAMCT allowing labor-free prompts and train it on a collected large CT dataset consisting of 1.1M CT images and 5M masks from public datasets. Specifically, based on SAM, SAMCT is further equipped with a U-shaped CNN image encoder, a cross-branch interaction module, and a task-indicator prompt encoder. The U-shaped CNN image encoder works in parallel with the ViT image encoder in SAM to supplement local features. Cross-branch interaction enhances the feature expression capability of the CNN image encoder and the ViT image encoder by exchanging global perception and local features from one to the other. The task-indicator prompt encoder is a plug-and-play component to effortlessly encode task-related indicators into prompt embeddings. In this way, SAMCT can work in an automatic manner in addition to the semi-automatic interactive strategy in SAM. Extensive experiments demonstrate the superiority of SAMCT against the state-of-the-art task-specific and SAM-based medical foundation models on various tasks. The code, data, and model checkpoints are available at https://github.com/xianlin7/SAMCT.

Despite the great potential in capturing long-range dependency, one rarely-explored underlying issue of transformer in medical image segmentation is attention collapse, making it often degenerate into a bypass module in CNN-Transformer hybrid architectures. This is due to the high computational complexity of vision transformers requiring extensive training data while well-annotated medical image data is relatively limited, resulting in poor convergence. In this paper, we propose a plug-n-play transformer block with dynamic token merging, named DTMFormer, to avoid building long-range dependency on redundant and duplicated tokens and thus pursue better convergence. Specifically, DTMFormer consists of an attention-guided token merging (ATM) module to adaptively cluster tokens into fewer semantic tokens based on feature and dependency similarity and a light token reconstruction module to fuse ordinary and semantic tokens. In this way, as self-attention in ATM is calculated based on fewer tokens, DTMFormer is of lower complexity and more friendly to converge. Extensive experiments on publicly-available datasets demonstrate the effectiveness of DTMFormer working as a plug-n-play module for simultaneous complexity reduction and performance improvement. We believe it will inspire future work on rethinking transformers in medical image segmentation. Code: https://github.com/iam-nacl/DTMFormer.

Federated learning (FL) has emerged as a promising paradigm for training segmentation models on decentralized medical data, owing to its privacy-preserving property. However, existing research overlooks the prevalent annotation noise encountered in real-world medical datasets, which limits the performance ceilings of FL. In this paper, we, for the first time, identify and tackle this problem. For problem formulation, we propose a contour evolution for modeling non-independent and identically distributed (Non-IID) noise across pixels within each client and then extend it to the case of multi-source data to form a heterogeneous noise model (i.e., Non-IID annotation noise across clients). For robust learning from annotations with such two-level Non-IID noise, we emphasize the importance of data quality in model aggregation, allowing high-quality clients to have a greater impact on FL. To achieve this, we propose Federated learning with Annotation quAlity-aware AggregatIon, named FedA3I, by introducing a quality factor based on client-wise noise estimation. Specifically, noise estimation at each client is accomplished through the Gaussian mixture model and then incorporated into model aggregation in a layer-wise manner to up-weight high-quality clients. Extensive experiments on two real-world medical image segmentation datasets demonstrate the superior performance of FedA3I against the state-of-the-art approaches in dealing with cross-client annotation noise. The code is available at https://github.com/wnn2000/FedAAAI.

Neural Radiance Fields (NeRF) have recently achieved impressive results in synthesizing novel views. However, with the few-shot setup, the NeRF is prone to overfitting to the supervision views, which causes artifacts when rendering novel views. To this end, we design a human-perception-guided meta-training framework to enhance the novel views without needing daunting annotation efforts. Specifically, we first analyze that the few-shot NeRF suffers from quality fluctuation, distant view degradation, and geometrical inconsistency. Then, we design a human perception guider to evaluate and select views for enhancement. Further, a human perception score modulated hyper-restoration module is designed to handle different degradation. Finally, we adopt meta-learning to fit enhanced views to avoid any impairment to the supervision views. Besides, we can directly embed any NeRF model into the framework for training and enhancement. Experimental results on two widely used benchmark datasets, LLFF and DTU, demonstrate our superiority to the state-of-the-art methods.

In the realm of deep learning-driven computer vision, Video Coding for Machines (VCM) prioritizes machine vision optimization over human visual quality. This unique emphasis has resulted in the subdivision of VCM into two tracks, with one specifically targeting Feature Coding for Machines (FCM). While recent advancements in video feature compression show improvements, current FCM methods face challenges in eliminating spatial redundancy within intrafeatures and fully leveraging reference features for temporal redundancy removal. To address these issues, we propose a multi-frame fusion based low-resolution feature compression (MFLFC) method for object tracking. We introduce the concept of hierarchical coding for compressing video features, employing MFLFC to handle both P-features and B-features. Specifically, the proposed model incorporates a low-resolution layer to reduce spatial redundancy and a fusion layer to address temporal redundancy. Additionally, we have introduced a three-fold training strategy to improve training efficiency and stability, contributing to achieving superior performance. Experimental results verify that the proposed MFLFC significantly outperforms previous approaches by up to 82.21% BD-rate reduction.

2D heatmap-based human pose estimation has exhibited remarkable performance. However, constrained by quantization error, heatmap-based methods heavily rely on high-resolution heatmaps and intricate post-processing to enhance detection accuracy, thereby incurring substantial computational costs. To pursue more effective keypoint representation, we propose a novel scheme named Offset-based Disentangled Representation (ODR). ODR conducts coordinate classification and offset prediction simultaneously using 1D vectors and aggregates their outputs to precisely pinpoint keypoint positions, thus freeing from dependence on high-resolution heatmaps. To eliminate the impact of long-range offsets, we propose a scale-aware eraser that generates noticeable intervals based on the relative scale of different keypoints, directing the regression task to focus on short-range offsets. By doing so, upsampling layers are no longer necessary, enabling a more concise and effective architecture for human pose estimation. Extensive experiments conducted over COCO and MPII datasets validate the superiority of ODR over counterparts based on 2D or 1D heatmaps. Our source codes are available at the link.

3D-High Efficiency Video Coding (3D-HEVC), as an extension of HEVC in the realm of three-dimensional video, has brought significant coding performance improvements. However, traditional 3D video coding has faced many challenges such as compression distortion in texture and depth videos, as well as non-occlusion issues in Depth Image Based Rendering (DIBR) synthesis, which directly affected the visual quality of synthesized views. A Two-Stream Pyramid Attention Residual Network (TPARN) is proposed to achieve the quality enhancement of synthesized views. First of all, the Global Residual Attention (GRA) module and the Local Pyramid Attention (LPA) module are designed to extract global context information and intricate local texture details, which achieve a comprehensive scene understanding and preserve essential details across different scales. In addition, the Pyramid Attention Module (PAM) and skip connections are utilized to extract multiscale features, promoting seamless interaction among features. Experimental results demonstrate that the proposed method effectively reduces distortion caused by view synthesis, outperforming the latest methods in terms of performance.

To achieve efficient compression for both human vision and machine perception, scalable coding methods have been proposed in recent years. However, existing methods do not fully eliminate the redundancy between features corresponding to different tasks, resulting in suboptimal coding performance. In this paper, we propose a frequency-aware hierarchical image compression framework designed for humans and machines. Specifically, we investigate task relationships from a frequency perspective, utilizing only HF information for machine vision tasks and leveraging both HF and LF features for image reconstruction. Besides, the residual block embedded octave convolution module is designed to enhance the information interaction between HF features and LF features. Additionally, a dual-frequency channel-wise entropy model is applied to reasonably exploit the correlation between different tasks, thereby improving multi-task performance. The experiments show that the proposed method offers -69.3%∼-75.3% coding gains on machine vision tasks compared to the relevant benchmarks, and -19.1% gains over state-of-the-art scalable image codec in terms of image reconstruction quality.

Mobile-edge computing can help enable low-latency and accurate video analytics. However, it is difficult to make efficient utilization of limited edge resources because of the diverse requirements of video queries. In this article, we investigate edge coordination for resource-efficient video query processing, in order to accommodate real-time queries on end cameras, edge nodes, or the cloud, with accuracy guarantee. This problem is challenging because: 1) video queries are with unpredictable arrivals and different resource demands; 2) the decision space of both query scheduling and resource allocation varies over time; and 3) it is critical to maintain long-term accurate analytics for all arrived queries. This problem boils down to making scheduling and resource allocation decisions, which is formulated as a mixed-integer nonlinear programming with a long-term accuracy constraint. Observing that both the scheduling and resource allocation of each query have the Markovian property, the Markov decision process and Lyapunov optimization are adopted to decompose the problem into sequential subproblems. An adaptive reinforcement learning-based approach relying on edge coordination is proposed. Extensive experimental results show that our proposal outperforms other benchmarks on latency and accuracy at a higher level of resource utilization efficiency in real-world data sets.

Brain tumor segmentation is a fundamental task and existing approaches usually rely on multi-modality magnetic resonance imaging (MRI) images for accurate segmentation. However, the common problem of missing/incomplete modalities in clinical practice would severely degrade their segmentation performance, and existing fusion strategies for incomplete multi-modality brain tumor segmentation are far from ideal. In this work, we propose a novel framework named M$^{2}$FTrans to explore and fuse cross-modality features through modality-masked fusion transformers under various incomplete multi-modality settings. Considering vanilla self-attention is sensitive to missing tokens/inputs, both learnable fusion tokens and masked self-attention are introduced to stably build long-range dependency across modalities while being more flexible to learn from incomplete modalities. In addition, to avoid being biased toward certain dominant modalities, modality-specific features are further re-weighted through spatial weight attention and channel-wise fusion transformers for feature redundancy reduction and modality re-balancing. In this way, the fusion strategy in M$^{2}$FTrans is more robust to missing modalities. Experimental results on the widely-used BraTS2018, BraTS2020, and BraTS2021 datasets demonstrate the effectiveness of M$^{2}$FTrans, outperforming the state-of-the-art approaches with large margins under various incomplete modalities for brain tumor segmentation.

Multi-organ segmentation is a fundamental task and existing approaches usually rely on large-scale fully-labeled images for training. However, data privacy and incomplete/partial labels make those approaches struggle in practice. Federated learning is an emerging tool to address data privacy but federated learning with partial labels is under-explored. In this work, we explore generating full supervision by building and aggregating inter-organ dependency based on partial labels and propose a single-encoder-multi-decoder framework named FedIOD. To simulate the annotation process where each organ is labeled by referring to other closely-related organs, a transformer module is introduced and the learned self-attention matrices modeling pairwise inter-organ dependency are used to build pseudo full labels. By using those pseudo-full labels for regularization in each client, the shared encoder is trained to extract rich and complete organ-related features rather than being biased toward certain organs. Then, each decoder in FedIOD projects the shared organ-related features into a specific space trained by the corresponding partial labels. Experimental results based on five widely-used datasets, including LiTS, KiTS, MSD, BCTV, and ACDC, demonstrate the effectiveness of FedIOD, outperforming the state-of-the-art approaches under in-federation evaluation and achieving the second-best performance under out-of-federation evaluation for multi-organ segmentation from partial labels.

To implement a metaverse exhibition interaction system, the instability problem of high-quality avatar facial reenactment must be considered. How to void identity limitations and eliminate artifacts are key challenges for avatar reenactment. It also lacks the support of the application system it is implemented in. We propose a metaverse system architecture oriented to emotional interaction. And we propose a novel method for avatar expression reenactment named Overall-Local Feature Warping Fusion Model based on Optical-Flow field prediction. We solve the identity limitation by overall optical-flow estimation and local optical-flow estimation and eliminate artifacts by illumination consistency. We compare with the mainstream optical flow face reenactment methods and outperform them in identity similarity, structural similarity, and facial action unit recognition ratio. We experimentally compared our method improves by an average improvement of 3.79%. And we also implement our method in the metaverse exhibition system. Although we satisfy most of the interaction scenarios, our method is still insufficient in some side-face cases.

This paper studies the joint computing offloading and resource allocation problem in an MEC network. We formulate the problem as an optimization problem with the objective to maximize the network throughput while satisfying the delay requirements of as many service requests as possible. Meanwhile, we propose a deep-Q-network (DQN) based joint computing offloading and resource allocation (D-CORAL) algorithm to solve the formulated problem. The proposed D-CORAL algorithm attempts to jointly optimize edge node selection, and spectrum resource and computing resource allocation for each service request by learning online to better adapt to a dynamic environment. Simulation results show the proposed algorithm can achieve larger network throughput than two benchmark algorithms.

The extended computing power and ubiquity of smartphones offer tremendous opportunities for mobile crowdsourcing-assisted traffic information collection. Due to the dynamic and unsupervised nature of mobile sensing nodes, data quality is a major concern, and reputation systems are often employed to mitigate the impact of unreliable data sources. In this paper, we propose using federated learning with iterative filtering in traffic information crowdsourcing to discover the truth, calculate the reputation of all mobile sensing nodes, and recover the truth about traffic conditions. Our simulation results show that the proposed approach is robust to collusion attacks, provides better accuracy, and converges to the truth quickly.

Federated noisy label learning (FNLL) is emerging as a promising tool for privacy-preserving multi-source decentralized learning. Existing research, relying on the assumption of class-balanced global data, might be incapable to model complicated label noise, especially in medical scenarios. In this paper, we first formulate a new and more realistic federated label noise problem where global data is class-imbalanced and label noise is heterogeneous, and then propose a two-stage framework named FedNoRo for noise-robust federated learning. Specifically, in the first stage of FedNoRo, per-class loss indicators followed by Gaussian Mixture Model are deployed for noisy client identification. In the second stage, knowledge distillation and a distance-aware aggregation function are jointly adopted for noise-robust federated model updating. Experimental results on the widely-used ICH and ISIC2019 datasets demonstrate the superiority of FedNoRo against the state-of-the-art FNLL methods for addressing class imbalance and label noise heterogeneity in real-world FL scenarios.

Intelligent applications such as metaverse built on IoT technology are usually inseparable from the collection of surveillance video data. Yet, how to effectively compress indoor surveillance video has always been a huge challenge for data transmission. In this paper, we propose an image-feature parallel compression framework for indoor surveillance video. In other words, the video is divided into images and features which are compressed separately. Specifically, according to the Mean-shift algorithm, the background image is generated, encoded and shared among frames. Based on a series of detection and matching algorithms, a human feature (HF) extractor is designed to extract a frontage body image, identity features, boundary features and structural features which are used for the generation of foreground body images. At the same time, body masks are generated in order to segment body areas from body images. Finally, with the guidance of body mask and boundary features, the body area covers the corresponding area of generated background image to reconstruct the video, and its quality is improved by a quality enhancement network. Experimental results show that the proposed compression scheme achieves coding gain up to 66.91% and 51.84% averagely compared with HEVC and VVC respectively.

Autonomous driving vehicles, leveraging on-board cameras for real-time surrounding environment sensing, have boosted the development of safe and efficient transportation. However, limited on-board computation resources pose a challenge in obtaining low-latency video analytics, as performing multiple analytics tasks simultaneously with processing massive redundancy in spatial, temporal, and semantic dimensions related to video analytics is of daunting computational complexity, resulting in long analytics latency. In this paper, we propose CoVA, a collaborative architecture based on multi-hop V2V communications, to achieve low-latency and accurate video analytics. Specifically, CoVA first builds the same lane-based vehicle fleets and formulates a multi-objective optimization problem, aiming to minimize spatial redundancy and latency with reliable communications. Then, a variable-threshold video frame filtering algorithm is designed to dynamically handle unpredictable surrounding environment changes by adjusting the similarity threshold for video frames captured by vehicles, which minimizes temporal redundancy while guaranteeing analytics accuracy. Furthermore, considering the semantic redundancy, CoVA reconstructs the original formulated problem using collaborative vehicle sensing and processing across lanes. Finally, joint redundancy minimization and efficient resource allocation optimization is proposed for properly allocating computation and bandwidth resources. Experimental results demonstrate the superiority of CoVA against other video analytics approaches, significantly reducing up to 70% in latency while only sacrificing less tolerated accuracy.

Recently, efficient image coding for machines method is widely required under machine vision task-oriented coding scenarios. To achieve higher task accuracy under a certain bitrate constraint, existing solutions take advantage of cutting-edge learned image compression methods and cascade the compression and specific task model straightly to carry out the joint optimization, which lacks adaptability and flexibility to handling machine vision saliency and non-saliency regions, thus obtaining inferior performance. In this paper, we propose a learned image compression model with a saliency-driven rate allocation strategy for machine vision. Specifically, an auxiliary network is introduced to provide reference latent first which contains full image reconstruction required information. And then, a machine vision saliency detector is applied to generate the adjustment factor to achieve rate allocation in the latent domain. In addition, a spatial dynamic convolution module is designed to facilitate the compressor adaptively processing the different spatial areas. Finally, the multi-distortion constraint is utilized to achieve an end-to-end optimization. Experimental results show that the proposed method achieves -77.48% coding gains over VVC using Mask R-CNN as the back-end task network on the Cityscapes dataset. Moreover, the reconstructed image maintains considerable visual fidelity, making it possible both for human and machine consumption.

Abstract In the newest generation of video coding standard, Versatile Video Coding (VVC), a new technique called Quad Tree with nested Multi‐type Tree (QTMT) structure is introduced. QTMT significantly improves the coding efficiency, but the improvement in compression performance comes at the cost of drastically increased complexity. This paper proposes a fast intra partition algorithm using Lightweight Neural Network (LNN) to skip QTMT partition steps which are unlikely to be chosen as the best split modes. Specifically, five LNNs (for five QTMT split modes) are trained, using features that consider the characteristic of 360‐degree videos. The experimental results demonstrate that the proposed QTMT‐LNN can reduce the encoding time from 52.28% to 72.17% on average, with coding efficiency losses ranging from 0.71% to 1.78%. Compared to other fast algorithms for intra coding unit (CU) partition, the method outperforms the related works in terms of Bjontegaard delta bit‐rate (BDBR) and encoding time reduction.

Bottom-up human pose estimation decouples computational complexity from the number of people but requires additional operations to match the detected keypoints to each human instance. Existing approaches treat all keypoints equally while ignoring the relationships among keypoints, which in turn limit the performance ceilings. In this work, we propose a hierarchical associative encoding and decoding framework for bottom-up human pose estimation by introducing additional prior knowledge. Specifically, in addition to keypoint-level and instance-level associations, we further divide keypoints into groups and explore group-level associations. This way, prior knowledge is incorporated to determine the keypoint groups for better associative encoding. To deal with complex poses, we introduce a focal pulling loss to focus more on the hard-to-associate keypoints. Moreover, instead of using a pre-defined order for keypoint grouping, we propose a progressive associative decoding method to dynamically determine the order of keypoints for grouping, which helps reduce isolated keypoints. Experimental results on the MS-COCO, CrowdPose and MPII datasets show superior performance of our proposed associative encoding and decoding algorithms. More importantly, we prove, through validation, that hierarchical associative encoding and decoding can be used as a plug-n-play module for performance improvement regardless of backbone architecture. Our source code and pretrained models are available athttps://github.com/ducongju/HAE.

To improve the transportation efficiency of advanced manufacturing, cameras have been extensively deployed to enhance the on-road perception of autonomous ground vehicles in smart industrial parks. Considering the informative yet substantial data volume of contents generated by those cameras, we employ vehicle-to-everything links to deliver the captured video frames to neighboring vehicles, road-side units, or base stations, in order to respond to vehicle-control-related video queries. To help vehicles obtain low-latency and high-accuracy on-road information for autonomous driving, an optimization problem is formulated, taking into account the impact of vehicle mobility and diverse resource demands of different video queries. Then, a two-stage algorithm is proposed to determine the frame rate of video cameras, as well as the destination of the associated video frames based on matching theory. Extensive simulation results show that this approach can improve the accuracy by up to 18% and reduce the response delay by 13.2%.

Weakly supervised learning, releasing deep learning from highly labor-intensive pixel-wise annotations, has gained great attention, especially for medical image segmentation. With only image-level labels, pixel-wise segmentation/localization usually is achieved based on class activation maps (CAMs) containing the most discriminative regions. One common consequence of CAM-based approaches is incomplete foreground segmentation, i.e. under-segmentation/false negatives. Meanwhile, suffering from relatively limited medical imaging data, class-irrelevant tissues can hardly be suppressed during classification, resulting in incorrect background identification, i.e. over-segmentation/false positives. The above two issues are determined by the loose-constraint nature of image-level labels penalizing on the entire image space, and thus how to develop pixel-wise constraints based on image-level labels is the key for performance improvement which is under-explored. In this paper, based on unsupervised clustering, we propose a new paradigm called cluster-re-supervision to evaluate the contribution of each pixel in CAMs to final classification and thus generate pixel-wise supervision (i.e., clustering maps) for CAMs refinement on both over- and under-segmentation reduction. Furthermore, based on self-supervised learning, an inter-modality image reconstruction module, together with random masking, is designed to complement local information in feature learning which helps stabilize clustering. Experimental results on two popular public datasets demonstrate the superior performance of the proposed weakly-supervised framework for medical image segmentation. More importantly, cluster-re-supervision is independent of specific tasks and highly extendable to other applications.

OBJECTIVE: Transformers, born to remedy the inadequate receptive fields of CNNs, have drawn explosive attention recently. However, the daunting computational complexity of global representation learning, together with rigid window partitioning, hinders their deployment in medical image segmentation. This work aims to address the above two issues in transformers for better medical image segmentation. METHODS: We propose a boundary-aware lightweight transformer (BATFormer) that can build cross-scale global interaction with lower computational complexity and generate windows flexibly under the guidance of entropy. Specifically, to fully explore the benefits of transformers in long-range dependency establishment, a cross-scale global transformer (CGT) module is introduced to jointly utilize multiple small-scale feature maps for richer global features with lower computational complexity. Given the importance of shape modeling in medical image segmentation, a boundary-aware local transformer (BLT) module is constructed. Different from rigid window partitioning in vanilla transformers which would produce boundary distortion, BLT adopts an adaptive window partitioning scheme under the guidance of entropy for both computational complexity reduction and shape preservation. RESULTS: BATFormer achieves the best performance in Dice of 92.84 %, 91.97 %, 90.26 %, and 96.30 % for the average, right ventricle, myocardium, and left ventricle respectively on the ACDC dataset and the best performance in Dice, IoU, and ACC of 90.76 %, 84.64 %, and 96.76 % respectively on the ISIC 2018 dataset. More importantly, BATFormer requires the least amount of model parameters and the lowest computational complexity compared to the state-of-the-art approaches. CONCLUSION AND SIGNIFICANCE: Our results demonstrate the necessity of developing customized transformers for efficient and better medical image segmentation. We believe the design of BATFormer is inspiring and extendable to other applications/frameworks.

Vision transformers have recently set off a new wave in the field of medical image analysis due to their remarkable performance on various computer vision tasks. However, recent hybrid-/transformer-based approaches mainly focus on the benefits of transformers in capturing long-range dependency while ignoring the issues of their daunting computational complexity, high training costs, and redundant dependency. In this paper, we propose to employ adaptive pruning to transformers for medical image segmentation and propose a lightweight and effective hybrid network APFormer. To our best knowledge, this is the first work on transformer pruning for medical image analysis tasks. The key features of APFormer are self-regularized self-attention (SSA) to improve the convergence of dependency establishment, Gaussian-prior relative position embedding (GRPE) to foster the learning of position information, and adaptive pruning to eliminate redundant computations and perception information. Specifically, SSA and GRPE consider the well-converged dependency distribution and the Gaussian heatmap distribution separately as the prior knowledge of self-attention and position embedding to ease the training of transformers and lay a solid foundation for the following pruning operation. Then, adaptive transformer pruning, both query-wise and dependency-wise, is performed by adjusting the gate control parameters for both complexity reduction and performance improvement. Extensive experiments on two widely-used datasets demonstrate the prominent segmentation performance of APFormer against the state-of-the-art methods with much fewer parameters and lower GFLOPs. More importantly, we prove, through ablation studies, that adaptive pruning can work as a plug-n-play module for performance improvement on other hybrid-/transformer-based methods. Code is available at https://github.com/xianlin7/APFormer.

The just noticeable difference (JND) reveals the key characteristic of visual perception, which has been widely used in many perception-based image and video applications. Nevertheless, the modulatory mechanism of the human visual system (HVS) has not been fully exploited in JND threshold estimation, which results in the existing JND models not being accurate enough. In this article, by analyzing the feedforward and feedback modulatory behaviors in the HVS, an enhanced foveated JND (FJND) estimation model is proposed considering modulatory effects and masking effects in visual perception. The contributions of this article are mainly twofold. On the one hand, by analyzing the modulatory behaviors in the HVS, the modulatory mechanism is incorporated into JND estimation and a hierarchical modulation-based JND estimation framework is proposed for the first time. On the other hand, according to the response characteristics of visual neurons, modulatory effects on visual sensitivity are formulated as several modulatory factors to modulate the estimated JND threshold properly. Compared with existing models, the proposed model is developed in view of not only the masking effects but also the modulatory effects, which makes our model more consistent with the HVS. For different complex input images, experimental results show that the proposed FJND model tolerates more distortion at the same perceptual quality in comparison with other existing models.

In the past decade, with the deepening of the aging of the population and the strengthening of the health consciousness of the whole society, the Internet healthcare service has grown to be an inevitable trend of current society. We propose a collaborative adaptive architecture named Trusted Healthcare Smart Brain (THSB) for cross-blockchain intelligent collaboration of multi-institution healthcare services. THSB is an interdisciplinary system with Healthcare Internet of Things (H-IoT), blockchain, Artificial Intelligence, Cloud Computing, Big Data, and Internet. The participants of THSB include patients, rehabilitation institutions, medical service institutions, healthcare content service institutions, medical regulatory institutions, scientific research institutions, and government institutions. In addition, we propose a medical resources service balance method to maximize the utilization of medical service resources to solve the contradiction between random medical events and the normal distribution of medical resources.

The 3D extension of High Efficiency Video Coding (3D-HEVC) achieves excellent per-formance for 3D video coding while possessing significant computational complexity. To accelerate the time-consuming coding process of the depth map, a fast algorithm via XG-Boost is proposed in this paper. Specifically, a total of 14 specialized XGBoost models are used for different block sizes and viewpoint types to achieve early coding unit partition de-termination (ECP) and early prediction unit mode selection (EPM) to avoid executing the exhaustive traversal coding process. To promote the prediction accuracy of XGBoost mod-els, multi-domain correlations, including spatiotemporal, inter-view, and inter-component correlations are utilized and plenty of features are selected for model training. Evaluated on HTM-16.0 under random access configuration, the proposed ECP strategy can obtain 51.2% total encoding time saving with a 0.18% BDBR increase and the ECP+EPM can overall achieve 60.8% total encoding time saving with a 0.59% BDBR increase. The source code of our method is available at https://github.com/Joeyrr/_EPM.git.

This paper studies the joint spectrum and computing resource allocation problem in a mobile edge computing (MEC) network, where mobile users can offload computing tasks to an edge node for processing. We formulate the joint resource allocation problem as an optimization problem with an objective to maximize the system throughput while satisfying the service delay requirements of as many service requests as possible. A novel DQN-based resource allocation algorithm is proposed to solve the formulated problem. The proposed algorithm consists of a resource pre-allocation process and a resource allocation process. The former performs joint spectrum and computing resource allocation for each service request based on a deep-Q-network (DQN) model, which attempts to make an optimal decision on resource allocation for each service request by learning online to better adapt to dynamic traffic arrivals and diverse service requirements. The latter performs resource allocation for each service request according to a resource pre-allocation decision and service requirements. Simulation results show that the proposed DQN-based resource allocation algorithm outperforms three benchmark algorithms in terms of system throughput.

As 360-degree videos are with high data volume, it is a great challenge to deliver the content and provide a high quality of experience (QoE) for users. In this paper, we investigate the tile-level rate allocation problem with the purpose of optimizing users’ QoE. Specifically, we find the nonlinearity between video quality and bitrate through extensive experiments. Thus, a QoE metric is defined to better measure the perceptual quality. Considering the sequential decision nature of video streaming, we formulate the rate decision problem as an Markov Decision Process. Then we propose a deep reinforcement learning based rate adaptive streaming approach to solve this problem. However, the solution space is large as a 360-degree video is spatially partitioned into multiple tiles. In order to address the problem of combinatorial explosion, we propose a tile classification method based on the predicted viewpoint. Experimental results based on real-world traces show that our algorithm can improve the overall QoE by 16% − 22% compared to existing algorithms.

Hematoma expansion prediction is critical for the prognostic outcomes of patients with life-threatening intracerebral hemorrhage, which is time-consuming and replies on experts’ experience. Though deep learning based approaches have greatly reduced labor costs in hematoma expansion prediction, there still exist two challenges: the high computational complexity brought by 3D-CNN-based models, and the poor performance based on single-modality medical imaging data. In this paper, we propose a bidirectional-guided semi-3D network, named BGSNet, to utilize multi-modality data for accurate hematoma expansion prediction with less computational complexity. Specifically, a hematoma attentive semi-3D (HAS) module is constructed to extract high-level 3D visual features with fewer parameters in 2D-CNN level. To explore the potential cross-modality associations, we propose a bidirectional-guided (BG) strategy for joint training on medical imaging data and clinical records simultaneously. Experimental results demonstrate the superior performance of the proposed framework, achieving an accuracy of 0.817, a recall of 0.679, and a specificity of 0.864 respectively. More importantly, the proposed framework is more light-weight compared to 3D networks, making it more feasible in clinical applications.

Facial action unit (AU) detection is a hot topic in computer vision, but it remains challenging due to individual characteristics. Facial action features are a vital informative factor to explain facial anatomical variations but are often entangled with other facial attribution information leading to representation inconsistency within one category. We propose a novel Contrastive Disentangled Representation Autoencoder (CDAE) to learn discriminative identity-invariant representation for AU detection by factorizing face images into temporally varying action parts and stationary facial attributions components. Facial image space is mapped onto the facial action subspace and action-independent identity subspace to disentangle facial action information from identity information. In addition, we design a contrastive learning scheme to obtain a semantic-aware AU manifold by mapping the facial action features onto the continuous space of the latent variables, thus minimizing the misalignment between subjects and reducing the dimension of the facial action features. Experiment results show that CDAE outperforms or is comparable to previous AU detection methods on the challenging BP4D and DISFA benchmarks, demonstrating that the learned facial action representation is discriminative for AU detection.

The emotion of human beings tends to be complex in real conditions, generating compound expressions in human faces. Compound expression recognition is an important challenge for the assessment of human complex emotion. The recognition system based on six basic expressions cannot meet the demand of compound expressions recognition. The recognition performance of models learned from the basic expressions is poor due to the small number of compound expression datasets with highly accurate labels and insufficient sample diversity. Making full use of domains outside of compound expressions in small sample datasets will help promote diversity. We propose the Multi-Domain Fusion Generative Adversarial Network (MDFGAN), which innovatively fuses the face domain, compound expression domain and basic expression domain to obtain rich expression generation capability and high accuracy recognition. Pairing the face domain and the contour-unrelated compound expression domain in the generator will expand the sample diversity. The contour-related compound expression domain and the basic expression domain will jointly improve the expression recognition accuracy of the discriminator. Finally, we conducted comprehensive experiments on CFEE-26, CFEE-7 and CK+. In the experiments, the results of MDFGAN improved 6.79% on UF1 and 8.5% on UAR.

In versatile video coding (VVC), superior coding performance is achieved with incorporating many advanced coding tools. In this paper, a frame-level coding distortion model is proposed for VVC video coders for the first time. In comparison with the transform coefficient distribution (TCD) of High Effective Video Coding (HEVC), the TCD of VVC has a sharper peak. According to this observation, the TCDs of I, B and P frames are modeled by the probability density function (PDF) of generalized Gaussian distribution (GGD) with three fixed shape parameters. The GGD-based distortion model is then derived with a sliding window-based strategy, i.e., the frame-level coding distortion is formulated as the function of the distribution parameter of frame-level TCD and the quantization step. The experimental results show that the proposed model achieves accurate results of distortion estimation for VVC coders.

Abstract Multi‐person pose estimation is a challenging vision task that can be seriously affected by keypoint scale variation. Existing heatmap‐based approaches are devoted to reducing the effect by optimizing backbone architecture or loss functions, but the problem of an inaccurate heatmap representation with different keypoint scales still exists. A scale‐sensitive heatmap algorithm is presented to generate reasonable spatial and contextual features for the network to predict more precise coordinates, by systematically considering the standard deviation, truncated radius, and shape of Gaussian kernels. Specifically, the scale‐sensitive heatmap algorithm contains three parts: inter‐person heatmap, limited‐area heatmap, and shape‐aware heatmap. The inter‐person heatmap allocates different standard deviations for each human instance proportionally calculated by the keypoint‐based method, the limited‐area heatmap defines the truncated radius to limit the influence area of Gaussian kernels, and the shape‐aware heatmap modifies the Gaussian kernels generated by some ellipse‐shaped joints. Our scale‐sensitive heatmap algorithm outperforms the baseline by a considerable margin on the COCO and CrowdPose benchmark datasets. The code and pretrained models are available at https://github.com/ducongju/Scale‐sensitive‐Heatmap .

Abstract Screen content video (SCV) consists primarily of text areas, computer graphics and other computer‐generated content and possesses unique perceptual characteristics. To compress SCVs more effectively with less reduction in subjective quality, perceptual characteristics of SCVs are analyzed and a perceptual redundancy (PR) model for SCV compression is proposed, including spatial PR (SPR), temporal PR (TPR) and foveated PR (FPR) model. In SPR modeling, the SCV is divided into sharp edge (SE) areas and non‐SE areas, then SPR is estimated separately. In TPR modeling, both inter‐frame luminance adaptation effect and motion masking effect are taken into account. In FPR modeling, each frame of SCV is classified into abrupt frames, relative motion frames or static frames. Then fixation points of different kinds of frames are predicted using different methods, and FPR is modeled considering foveated masking effect and visual attention. Finally, the perceptual redundancy of SCV is estimated based on the product of SPR, TPR and FPR. It is experimentally demonstrated that compared to the state‐of‐the‐art models, the authors' model could obtain more accurate estimates of PR. Moreover, the model is incorporated into SCV compression with an adaptive perceptual quantizer. An average of 7.42% bits could be saved with less decline in subjective quality.

Video camera plays a growing important role in advancing industrial control towards a higher level of automation. Thus, video analytics become highly demanded, especially for low-latency and high-accuracy analytic results. Yet, the data volume produced by camera clusters is prohibitively high. In this article, we proposeRespire, a system that can remove redundant frames for reducing the cost of transmission and processing based on edge computing nodes, while maintaining useful frames for high analytic accuracy. Specifically,Respireincorporates a new way for characterizing the spatial–temporal redundancy between frames. Then,Respireprioritizes the uploading of frames for redundancy reduction. As the search space of the entire collected frames is exponential for the set of frames containing the maximal information, we jointly consider offline and online pruning of frames and propose a heuristic algorithm to reduce the search space. Extensive real-world dataset-based experiments demonstrate that the proposed system can significantly reduce communication and computation costs, while providing sufficient information for guaranteed video analytic accuracy.

Cerebral ventricles are one of the prominent structures in the brain, segmenting which can provide rich information for brain-related disease diagnosis. Unfortunately, cerebral ventricle segmentation in complex clinical cases, such as in the coexistence with other lesions/hemorrhages, remains unexplored. In this paper, we, for the first time, focus on cerebral ventricle segmentation with the presence of intra-ventricular hemorrhages (IVH). To overcome the occlusions formed by IVH, we propose a symmetry-aware deep learning approach inspired by contrastive self-supervised learning. Specifically, for each slice, we jointly employ the raw slice and the horizontally flipped slice as inputs and penalize the consistency loss between the corresponding segmentation maps in addition to their segmentation losses. In this way, the symmetry of cerebral ventricles is enforced to eliminate the occlusions brought by IVH. Extensive experimental results show that the proposed symmetry-aware deep learning approach achieves consistent performance improvements for ventricle segmentation in both normal (i.e. without IVH) and challenging cases (i.e. with IVH). Through evaluation of multiple backbone networks, we demonstrate the architecture-independence of the proposed approach for performance improvements. Moreover, we re-design an end-to-end version of symmetry-aware deep learning, making it more extendable to other approaches for brain-related analysis.

Perihematomal edema (PHE) volume, surrounding spontaneous intracerebral hemorrhage (SICH), is an important biomarker for the presence of SICH-associated diseases. However, due to irregular shapes and extremely low contrast of PHE on CT images, manually annotating PHE in pixel-wise is time-consuming and labour intensive even for experienced experts, which makes it almost infeasible to deploy current supervised deep learning approaches for automated PHE segmentation. How to develop annotation-efficient deep learning to achieve accurate PHE segmentation is an open problem. In this paper, we, for the first time, propose a cross-task supervised framework by introducing slice-level PHE labels and pixel-wise SICH annotations, which are more accessible in clinical scenarios compared to pixel-wise PHE annotations. Specifically, we first train a multi-level classifier based on slice-level PHE labels to produce high-quality class activation maps (CAMs) as pseudo PHE annotations. Then, we train a deep learning model to produce accurate PHE segmentation by iteratively refining the pseudo annotations via an uncertainty-aware corrective training strategy for noise removal and a distance-aware loss for background compression. Experimental results demonstrate that, the proposed framework achieves a comparative performance with the fully supervised methods on PHE segmentation, and largely improves the baseline performance where only pseudo PHE labels are used for training. We believe the findings from this study of using cross-task supervision for annotation-efficient deep learning can be applied to other medical imaging applications.

Bi-prediction is a fundamental module of inter prediction in the blocked-based hybrid video coding framework. Block-based motion estimation(ME) and motion compensation(MC) with simple models are adopted in bi-prediction process. Unfortunately, this MEMC-based scheme can't guarantee the prediction performance when it comes to video with complicated motions. In this paper, a novel inter prediction scheme based on deep frame prediction network (DFP-net) is proposed to enhance bi-prediction accuracy especially in complicate scenes. Specifically, the proposed DFP-net is composed of multi-scale motion alignment, fusion of temporal and spatial correlation and frame synthesis module. The DFP-net can precisely extract and fuse motion features in various scales and completely exploit temporal and spatial correlation to generate the prediction frame in a data-driven manner. Moreover, the DFP-net is integrated into VTM-6.2 to provide an additional prediction frame for biprediction. Since the prediction generated by DFP-net is more similar with to-be-coded frame in the sense of temporal distance and texture, it can be added to reference list to improve the diversity of references. In this manner, the proposed bi-prediction scheme has surpassed VTM-6.2 on average 1.8% BD-rate saving.

In the latest video coding standard, Versatile Video Coding (H.266/VVC), a new quadtree with nested multi-type tree (QTMTT) coding block structure is proposed. QTMTT significantly improves coding performance, but more complex block partitioning structure brings greater computational burden. To solve this problem, a fast intra block partition pattern pruning algorithm is proposed using gray level co-occurrence matrix (GLCM) to calculate texture direction information of coding units, terminating the horizontal or vertical split of the binary tree and the ternary tree in advance. Experimental results show that the proposed algorithm achieves up to 53.57% encoding time saving on average with negligible quality loss under all-intra conguration.

Traditional congestion control algorithms rely on various model-based methods to improve the end-to-end (E2E) performance of packet transmission. The resulting decisions quickly become less effective amid the dynamics of network conditions. In order to perform congestion control adaptively, reinforcement learning (RL) can be adopted to continuously learn the optimal strategy from the network environment. Oftentimes, the reward of such a learning problem is a weighted sum of multiple E2E performance metrics, such as throughput, delay, and fairness. Unfortunately, those weights can be only manually tuned based on extensive experiments. To address this issue, in this paper, we design a constrained RL algorithm for congestion control named CRL-CC to adaptively tune those weights, with the objective of effectively improving the overall E2E packet transmission performance. In particular, the multi-objective optimization problem is firstly formulated as a constrained optimization problem. Then, the Lagrangian relaxation method is leveraged to transform the constrained optimization problem into a single-objective optimization problem, which is solved by designing a multi-objective reward function with Lagrangian multipliers. Extensive experiments based on OpenAI-Gym show that the proposed CRL-CC algorithm can achieve higher overall performance in various network conditions. In particular, the CRL-CC algorithm outperforms the benchmark algorithm on Pantheon by 21.7%, 27.4%, and 5.3% in throughput, delay, and fairness, respectively.

In recent years, 360ovideos have gained higher and higher popularity. Nonetheless, compared to two dimensional videos, the large-scale data volume renders it a bottleneck to deliver 360ocontent with constrained bandwidth resources. In this paper, we investigate user viewing behavior when they explore in immersive environment and propose a novel 360ojust-noticeable-difference (JND) model to characterize user's tolerance to visual distortion. In order to maximize user's quality of experience (QoE), we present a scheme, named JND-Based Streaming (JBS), to jointly optimize 360o video coding and streaming over mobile devices. Specifically, tiled 360ovideos are firstly encoded with the proposed JND model to reduce video file size. Then, a quality-driven streaming approach is designed to instruct tile-level bitrate allocation, considering subjective sensation. Thanks to the video file size reduction, tiles can be delivered with higher quality, which provides users with improved QoE. Experimental results based on real-world network traces demonstrate that, on average, JBS outperforms its counterparts by 12% and 57% in terms of perceived quality.

Low-latency and accuracy-guaranteed video analytics is essential to many delay-sensitive camera-based applications. Analyzing video frames on edge nodes in proximity can effectively reduce the response delay compared with cloud-based solutions. However, the computation and bandwidth resources on an edge node are always limited. In this paper, we design a joint video query scheduling and resource allocation problem based on an edge coordinated architecture, in order to properly accommodate real-time video queries on end cameras, the edge nodes, or the cloud. This problem is challenging in that 1) the arrivals of video queries with different resource requirements are unknown in advance and 2) the design space (of both query scheduling and resource allocation) to provision video queries varies over time. Taking the two-fold uncertainty into consideration, we formulate the query provision problem as a mix integer non-linear program which is NP-hard and not solved directly. To deal with the NP-hardness and the absence of future information, the problem is re-formulated as a Markov decision process, which can leverage historical query information to make decisions about scheduling and resource allocation. The transformed problem calls for an online solution that can efficiently adapt to the dynamic design space. Hence, we propose an edge-coordinated reinforcement learning algorithm to continuously learn from the environment, and make decisions for query scheduling and resource allocation to achieve low latency and accurate video analytics. Extensive simulation results demonstrate the advantages of the proposed algorithm in latency and accuracy.

High Efficiency Video Coding - Screen Content Coding (HEVC-SCC) follows the traditional angular intra prediction technique in HEVC. However, the Planar mode and the DC mode are somewhat repetitive for screen content video with features such as no senor noise. Hence, this paper proposes a new intra prediction mode called linear regression (LR) mode, which combines the Planar mode and the DC mode into one mode. The LR mode improves the prediction accuracy of intra prediction for fading regions in screen content video. Besides, by optimizing the most probable mode (MPM) construction, the hit rate of the best mode in the MPM list is improved. The experimental results show that the proposed method can achieve 0.57% BD-BR reduction compared with HM$16.20+\text{SCM} 8.8$, while the coding time remains largely the same.

To reduce compression artifacts in video coding, prior knowledge from the codec is often utilized by deep learning based enhancement methods. However, most existing algorithms only take limited kinds of prior knowledge into consideration and directly feed it into neural networks, resulting in finite information obtained by the enhancement module. In this paper, a distortion-based neural network is proposed to better take advantage of features from the codec and further facilitate the quality of decoded videos. Firstly, a variety of prior knowledge is unitedly exploited to estimate the compression distortion. Secondly, an estimation module is also designed for the information obtained from the codec to improve the precision of estimation. With the accurately estimated distortion level, the enhancement module in the proposed method could reduce the corresponding artifacts by flexibly controlling the filtering strength. Moreover, the proposed method is integrated into VTM-7.1, achieving on average 5.92% BD-rate saving for Y component under All Intra configuration.

Abstract To alleviate the computation burden of the depth intra coding in 3D‐HEVC, a complexity reduction scheme based on texture feature and spatio‐temporal correlation is proposed. Firstly, a maximum splitting depth layer decision algorithm is proposed to reduce unnecessary splitting depth layer of the coding tree unit utilising the information of the previous encoded I frame in the same view. Secondly, a new texture complexity model is built by pixel‐based statistical method combined with edge detection. Based on the proposed model, the coding unit block is divided into the smooth block, texture or edge block. On the coding unit level, an early termination of coding unit splitting algorithm for smooth blocks is proposed to filter out unnecessary coding blocks. Thirdly, on the predicting unit level, a fast candidate mode decision algorithm considering predicting unit's types and spatial correlation is proposed to decide the candidate mode list directly. Experimental results describe that the proposed algorithm reduces 53.8% depth intra coding time on average, with 0.43% BD‐rate loss on synthesised views.

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

Conventional intra prediction usually utilizes the top and left reconstructed reference samples of the current block to generate prediction pixels. However, with the distance between reference samples and the predicting pixel increasing, the correlation of them becomes weaker. The loss of the bottom-right corner of the current block is bigger than that of the top-left corner. To improve the situation above mentioned, a novel secondary intra prediction scheme is proposed for video coding in this paper. In this scheme, the block will be predicted twice and the first prediction is the same as the conventional intra prediction. After the first prediction, the boundary samples of the reconstructed block can be applied as the reference samples for the second prediction. In our second prediction, the current block is predicted with a designed distance-based bi-directional prediction method. The optimal prediction is determined through the rate-distortion optimization (RDO) strategy. Experimental results show that the proposed algorithm achieves about 3.4% bitrate savings on average comparing to HEVC reference software HM16.9.

The just noticeable distortion (JND) has been widely applied in perceptual image/video compression. Yet, the existing JND estimation models are not accurate enough, which results in the degradation of perceptual quality. In this paper, we propose a JND compensation based perceptual video coding (PVC) scheme to compress videos with better perceptual quality. Specifically, a block-level JND estimation model is proposed at first, which leads to this model can be employed in variable block-sizes based video coding directly. Then, considering the quantization distortion (QD), a QD&JND-based residue filter is proposed to suppress the residues in the encoder. To improve the quality of the reconstructed video, a QD&JND-based compensation scheme is developed in the decoder. Experimental results show that the QD&JND-based filter achieves bit rate saving significantly, and the QD&JND compensation scheme improves the perceptual quality of the compressed videos effectively.

With the rapid development of Internet, short videos draw more and more attentions nowadays. Due to the small scale of short videos, image-level coding scheme can be applied to improve compression efficiency. In this paper, we propose a densely connected unit based loop filter for short video coding in H.266/VVC, as Fig. 1 shows. In the proposed loop filter, the densely connected units are specially designed to extract feature maps, and fully decompose videos. By densely connection between layers, the designed units can reuse feature maps, and reduce the redundancy of features. Meanwhile, the proposed loop filter has the light weight, which can recover short videos at image level fast. Moreover, the proposed loop filtering network is embedded into VVC to replace the original sample adaptive offset (SAO) and adaptive loop filter (ALF), which saves bits cost and lowers the complexity of coding. The experimental results show that the proposed loop filter, compared with H.266/VVC, can achieve better performance in both AI and RA mode.

Convolutional neural network has witnessed remarkable progress in compressed video quality enhancement in high efficiency video coding (HEVC) standard. But most existing methods focus on single frame quality enhancement where copious temporal and spatial information is neglected. In this paper, we propose a spatial-temporal fusion convolutional neural network (STEF-CNN) to employ spatial and temporal information to improve the performance of in-loop filter in HEVC. Specifically, the STEF-CNN adopts a pre-denoising network which in advance processes the compressed videos frame by frame. The pre-denoising operation alleviates the impact of noise and blocking artifacts. Then the denoised frames are sent to temporal-spatial fusion module which picks out valuable temporal and spatial information. The fused frames are eventually fed to quality enhancement network which is based on residual learning and dense network. The STEF-CNN is capable of capturing abundant information from consecutive neighboring frames. Extensive experimental results demonstrate the effectiveness of the proposed method. The STEF-CNN achieves 11.53% BD-BR reduction in all-intra (AI) configuration and 10.20% BD-BR reduction in random-access (RA) configuration.

In order to achieve highly compact representation for videos, we propose an adaptive quantization based perceptual video coding (PVC) scheme in this paper. Because human only perceive the limited discrete-scale quality levels, the perceptual quantization is transformed into the problem of how to determine the maximum quantization parameter (Qp) under the same perceptual quality level. So, the relationship between perceptual quality level and quantization parameter is analyzed with the statistical way in this paper. The frame-level Qp value for each quality level is determined based on the maximum probability criterion. Then, the just noticeable distortion is estimated to guide the Qp adjustment in the coding unit level (CU-level). In summary, the perceptual quantization is achieved in both the frame level and the CU level according to characteristics of human visual system (HVS). Experimental results show that the proposed PVC scheme achieves substantial bitrate reduction with better subjective and objective quality in comparison with other PVC schemes.

To alleviate the ever-increasing data demands, edge caching plays a crucial role in improving the performance of system, especially in data-intensive applications. Previous works mainly focus the caching policy over reliable channels. For unreliable channel scenarios, the system performance is jointly affected by the user preference and the channel reliability, whereas both the user preference and the reliability are unknown commonly. A high retrieval cost may be incurred on unreliable channels even when the requested content is in the nearby cache. To solve the issues mentioned above, we jointly optimize the service scheduling policy and the content caching policy in this paper. We propose a maximal reward priority (MRP) policy to serve user requests, and a collaborative multi-agent actor critic (CMA-AC) policy to update the local cache. Simulation results show that the proposed MRP policy outperforms the shortest distance priority (SDP) policy [4]. And the proposed CMA-AC policy obtains a better performance compared with a distributed multi-agent deep Q-network (DMA-DQN) policy, especially when the number of contents and the capacity of local cache are large. Furthermore, the proposed CMA-AC policy is robust.

Cooperative caching between multiple small base stations (SBSs) plays a critical role for easing the traffic congestion of the backhaul link. Previous works assume that cooperative caching policies can achieve good performance when each user in the region is mainly served by its nearest SBS, which may not be the case in small cell networks with heterogeneous channel qualities. In this paper, we study cooperative caching problem in a small cell network with heterogeneous channel qualities when content popularity profile is unknown. These two features impose new challenges in optimizing content placement in multiple SBSs. To address this problem, we first propose a bayes-based learning algorithm that learn the popularity profile by sampling from a Beta distribution at each time period. Based on the estimated popularity profile, we then optimize the content placement at each time period by caching contents with higher popularity/size ratio in SBSs with better channel qualities. Numerical results show that the proposed algorithms outperforms three baselines in terms of average transmission delay and cache hit rate.

In visual perception, the neural response is evoked by the central stimulus and inhibited by the surrounding stimuli. These two stimuli can be regarded as the positive and negative incentives for perception. In this letter, the aftereffect of two incentives is incorporated into the just noticeable distortion (JND) estimation to avoid the threshold overestimation issue. In neuroscience, how an incentive affects an observer is measured by the surprise of this incentive which is estimated by the residual self-information according to the effective coding theory. Considering the competition biases, we define the product of the residual self-information and the biases, named as perceptual surprise, to measure the positive effect. As for the negative effect, we measure it with the average surprise (i.e., local Shannon entropy). Besides, visual signals are processed hierarchically in the HVS. To simplify the JND modeling process, the HVS is divided into two levels and the perception is simply divided into two stages. In each stage, the threshold is estimated individually and the total JND threshold is obtained by non-linear superposition of two stage thresholds. Experimental results show that the proposed JND model outperforms the existing pixel-based JND models and effectively avoid the threshold overestimation issue.

To develop smart city and intelligent manufacturing, video cameras are being increasingly deployed. In order to achieve fast and accurate response to live video queries (e.g., license plate recording and object tracking), the real-time high-volume video streams should be delivered and analyzed efficiently. In this article, we introduce an end-edge-cloud coordination framework for low-latency and accurate live video analytics. Considering the locality of video queries, edge platform is designated as the system coordinator. It accepts live video queries and configures the related end cameras to generate video frames that meet quality requirements. By taking into account the latency constraint, edge computing resources are subtly distributed to process the live video frames from different sources such that the analytic accuracy of the accepted video queries can be maximized. Since the amount of required edge computing resource and video quality to accurately address different video queries are unknown in advance, we propose an online video quality and computing resource configuration algorithm to gradually learn the optimal configuration strategy. Extensive simulation results show that as compared to other benchmarks, the proposed configuration algorithm can effectively improve the analytic accuracy, while providing low-latency response.

Occlusions will reduce the performance of systems in many computer vision applications with discontinuous surfaces of 3D scenes. We explore a signal-processing framework of occlusions based on the light ray visibility to improve the rendering quality of views. An occlusion field (OCF) theory is derived by calculating the relationship between the occluded light rays and the nonoccluded light rays to quantify the occlusion degree (OCD). The OCF framework can describe the various in-scene information captured by the changes in the camera configuration (i.e., position and direction) through a quantitative description of the occlusion information. From a spectral analysis of the OCF, we mathematically derive analytical functions to determine the changing relationship between the scene and the camera configuration. A reconstruction filter can be designed to achieve interference cancellation and compensate for the missing information caused by the occlusions. Our measurements of different occlusions using this OCF framework included both synthetic and actual scenes. The experimental results show that the proposed OCF framework can improves the rendering quality of views and outperforms other known occlusion quantization schemes in a complex scene.

The complexity of 3D-HEVC is fairly high due to quad tree structure and traversal searching in depth intra coding. In order to reduce complexity caused by coding unit (CU) size decision in rate distortion optimization (RDO) process, a fast algorithm based on adaptive QP convolutional neural network (AQ-CNN) structure is proposed in this paper. For each size of CU, the proposed structure automatically extracts deep feature information to terminate CU partition early. Specially, the AQ-CNN structure is suitable for different QPs because the QP has a great influence on CU partition and is connected into the CNN structure appropriately. Benefiting from the accurate prediction of CU partition label, the proposed algorithm reduces coding complexity sharply. Experimental results show that the proposed algorithm reduces the depth coding time by 69.4% with negligible BD-rate increase, and outperforms other recent algorithms in 3D-HEVC.

Multi-view texture video coding is very important, we propose a bit allocation method based on view layer and a bitrate decision method for P-frame of the dependent view (DV). First of all, considering that the distortion in the base view (BV) is directly transmitted to the DV by inter-view skip mode, the RD model of the DV is improved based on the inter view dependency. In this paper, a precise power model is derived based on our joint RD model to represent the target bitrates relationship between the BV and the DV. Then, since the P frame in the DV (P-DV) is mainly predicted from the corresponding I frame in the BV (I-BV) by inter-view prediction, the constant proportional relationship between the P-DV and the I-BV is discovered in this paper. Based on this discovery, a novel linear model is built to assign the target bitrates of the P-DV. Extensive experimental results exhibit that the proposed scheme provides a better RD performance than the state-of-the-art algorithms.

In video encoder, hard-decision quantization (HDQ) is well-suited for parallel processing, but suffers from non-negligible coding performance degradation compared with soft-decision quantization (SDQ). In this paper, by fully simulating the behavior of SDQ, a coefficient-adaptive offset model constructed by the deep learning approach is proposed to adjust the output of HDQ. Experiment results show that the proposed algorithm achieves promising RD performance and well-suited for hardware encoder implementation design.

Mobile edge computing extends computing and storage resources to the proximity of mobile users, facilitating a number of innovative mobile applications. Particularly, video streaming is the most prevailing one that consumes substantial edge resources. In this paper, we investigate the multi-dimensional resource allocation for video service provisioning, with the objective of ensuring satisfied streaming experience at high resource utilization. Considering the diversified and constantly changing user preferences on the quality of video contents, the edge resource allocation process is modeled as a long-term utility maximization problem. To address this problem, we propose an online learning algorithm that actively estimates user preferences according to regression analysis on user feedback. This algorithm requires no training phase, and hence is adaptive to dynamic user interests and available edge resources. Both theoretical analysis and numerical results demonstrate that the performance of the proposed algorithm asymptotically approaches the hindsight optimal resource allocation strategy.

Conventional intra prediction applies the nearest reference line to predict the current block and improves the performance by adding more angular prediction modes. However, the performance of prediction near the bottom-right corner of the block suffers from non-negligible degradation due to the reference samples used for prediction locate at the above and left sides of the current block. Therefore, in this paper, to improve this situation, multi-weights intra prediction is proposed. Considering the coding complexity, the optimal two reference lines are picked out at first based on minimum distortion criterion. Following, the best weight combination is selected out from candidate weight combinations and is applied to predict the current block. Experimental results show that the proposed algorithm achieves 2.19% bitrate saving on average compared with the conventional intra prediction.

Image stitching technology has been widely used in immersive applications, such as 3D modeling, VR and AR. The quality of stitching results is crucial. At present, the objective quality assessment methods of stitched images are mainly based on the availability of ground truth (i.e., Full-Reference). However, in most cases, ground truth is unavailable. In this paper, a no-reference quality assessment metric specifically designed for stitched images is proposed. We first find out the corresponding parts of source images in the stitched image. Then, the isolated points and the outer points generated by spherical projection are eliminated. After that, we take advantage of the bounding rectangle of stitching seams to locate the position of overlapping regions in the stitched image. Finally, the assessment of overlapping regions is taken as the final scoring result. Extensive experiments have shown that our scores are consistent with human vision. Even for the nuances that cannot be distinguished by human eyes, our proposed metric is also effective.

The guidance of color images greatly improves the restoration accuracy of the depth map. However, due to the incomplete texture structure consistency between the color image and the depth map, and the limitations of$L^{2}$based model, static guidance tends to cause texture copy artifacts and blurring depth discontinuities, which seriously deteriorates the quality of result. To tackle those problems, we propose a dynamic guidance model which can adaptively adjustment itself based on local smooth characteristics and continuously optimize in iteration. The proposed method can significantly alleviate the negative impact of the incomplete consistency and makes full use of the guidance information to restore fine texture at the same time. Moreover, a novel edge correction mechanism is designed to filter out incorrect depth information around boundaries, ensuring the correction of edges. In experiments, visual comparison proves that our method can alleviate texture copy artifacts and blurring depth discontinuities effectively. Quantitative results show that the proposed method restores high quality depth map with the lowest mean absolute error.

The High Efficiency Video Coding (HEVC) standard adopts inter prediction to eliminate temporal correlation between the successive frames. However, a large amount of bits need to be explicitly signaled in the bitstream to specify the motion information. In this paper, we propose an extended skip strategy to alleviate bit consumption for motion data during the inter prediction process. Specifically, before the current frame is encoded, an additional picture generated by a deep convolutional neural network (CNN) is introduced to inter prediction. Since the additional reference picture is more similar with the current frame, most blocks of this frame can be skipped in the coding process. Consequently, to further improve the compression, an extended skip strategy is designed, i.e., the current frame can be skipped in multi-levels, including frame-level and coding tree unit level (CTU-level). Moreover, the skip-level of the current frame is decided in the sense of rate-distortion optimization (RDO). The proposed algorithm is implemented on the HM-16.6 software and an average of 4.4% BD-rate gain has been achieved in the experiments, which indicates the superiority of the proposed method.

Recently, deep learning technology has made significant progresses in high efficiency video coding (HEVC), especially in in-loop filter. In this paper, we propose a dense inception attention network (DIA_Net) to delve into image information and model capacity. The DIA_Net contains multiple inception blocks which have different size kernels so as to dig out various scales information. Meanwhile, attention mechanism including spatial attention and channel attention is utilized to fully exploit feature information. Further we adopt a dense residual structure to deepen the network. We attach DIA_Net to the end of in-loop filter part in HEVC as a post-processor and apply it to luma components. The experimental results demonstrate the proposed DIA_Net has remarkable improvement over the standard HEVC. With all-intra(AI) and random access(RA) configurations, It achieves 8.2% bd-rate reduction in AI configuration and 5.6% bd-rate reduction in RA configuration.

The HEVC standard has shown great effect in video coding, however, the block-level compression scheme introduces block artifacts into reconstructed videos. Meanwhile, the quantization also brings ringing effect and blur to the compressed sequence. In order to solve this problem, we propose a simplified inception unit based convolutional neural network to reduce the compression distortion in HEVC. In the proposed method, a simplified inception unit is designed to analyze and extract image features. By stacking multiple inception units, the internal and border information of each frame can be fully extracted and the images are able to achieve higher quality. Meanwhile, the proposed network is incorporated into the HEVC reference software, which can be directly used in coding process and improve coding performance. The experimental results demonstrate that the proposed method, compared with HEVC, reduces 5.5% BDrate at average in `Al' mode. In `RA' mode, the proposed method still achieves 6.02% BD-rate reduction at most.

Traditional loss-based congestion control protocols interpret packet loss as network congestion. Recently, Google proposed BBR, which is a congestion-based congestion control protocol. It employs delivery rate as the knob for congestion control, which achieves higher throughput and lower latency. Interestingly, BBR is found to have a preference for longer round-trip time (RTT) flows, which enjoy higher bandwidth ratio compared to flows with shorter RTT. To address this fairness issue, we proposed Adaptive-BBR, which creatively uses adaptive pacing gain to adjust the sending rate. The objective is that, via the proposed fine-grained adaptive mechanism, flows with different RTTs share similar portion of bottleneck bandwidth. Simulation results show that Adaptive-BBR can improve fairness by at least 47.8 %, and reduce average queuing delay by up to 93.3%, compared with that of BBR.

We propose a novel context-aware task allocation framework for mobile crowdsensing in the scenario of edge computing to enable the crowdsensing platform effectively and real-timely handle large-scale crowdsensing tasks in smart city. The task allocation performs in both cloud computing layer and edge computing layer. It aims to combine the merits of cloud and edge computing, i.e., diminishing communication latency while guaranteeing overall scheduling. The cloud layer evaluates the participants' task-oriented reputation based on the participants' background information, task context, and historical feedbacks (i.e., rewards) and sends the edge layer the most promising subset of participants. Then the edge layer communicates with the participants for the real-time information and makes optimization based on the task requirement (e.g., maximizing the sensing coverage under the constraint of the task budget). In the cloud layer, we propose a privacy-preserving and contextual online learning algorithm to manage the participants' reputation. The algorithm can adapt the decision-making strategy based on previous performances of participants. In the edge layer, plenty of existing centralized task allocation strategies can be directly applied to optimize based on the participants' real-time information. Theoretical analysis shows that our proposal achieves sublinear regret and differential privacy for both requesters and participants. Experiments results validate that our proposed algorithm supports increasing big dataset while striking a balance between the privacy-preserving level and the prediction accuracy.

By modeling an action as the evolution of postures, Posture-based recognition algorithms possess interpretative patterns. However, there are two limitations resulted from individual diversity. First, the same action can be performed by different organs. Such actions are denoted as ambiguous actions. Second, the postures of the same action can be badly influenced by personal characteristics (e.g., personal habits and the height). In order to tackle the problems above, we propose a hierarchical posture representation (HPR). A posture is composed of several skeletal points. Each skeletal point can perform a unique operation and provide a certain amount of information. Thus, an action can be recognized based on the relationships and the information contribution levels of all skeletal points. In HPR, each skeletal point is first represented by its interaction features with other skeletal points. The interaction features are independent of the type of posture. Then, the information contribution level of each skeletal point is estimated based on its interaction features. An end-to-end adaptive action recognition network (AARN) is proposed to accomplish the three processes (estimating the information contribution level, mining the relationships among skeletal points, and recognizing actions). The proposed algorithm shows promising accuracy on two databases. Our work does not only solve the problem above but also introduce a novel action modeling method.

Mobile edge caching aims to enable content delivery within the radio access network, which effectively alleviates the backhaul burden and reduces response time. To fully exploit edge storage resources, the most popular contents should be identified and cached. Observing that user demands on certain contents vary greatly at different locations, this paper devises location-customized caching schemes to maximize the total content hit rate. Specifically, a linear model is used to estimate the future content hit rate. For the case with zero-mean noise, a ridge regression-based online algorithm with positive perturbation is proposed. Regret analysis indicates that the hit rate achieved by the proposed algorithm asymptotically approaches that of the optimal caching strategy in the long run. When the noise structure is unknown, an$H_{\infty }$filter-based online algorithm is devised by taking a prescribed threshold as input, which guarantees prediction accuracy even under the worst-case noise process. Both online algorithms require no training phases and, hence, are robust to the time-varying user demands. The estimation errors of both algorithms are numerically analyzed. Moreover, extensive experiments using real-world datasets are conducted to validate the applicability of the proposed algorithms. It is demonstrated that those algorithms can be applied to scenarios with different noise features, and are able to make adaptive caching decisions, achieving a content hit rate that is comparable to that via the hindsight optimal strategy.

3D video coding extension of High Efficiency Video Coding (3D-HEVC) adopts many high complexity approaches to improve the coding performance of the depth video, which leads to heavy computation. To alleviate the computation burden, a fast intra coding scheme for the depth map coding is proposed based on texture feature and spatial correlation. The coding unit (CU) sizes and intra prediction modes are selected differently due to different depth features. The CU block is divided into smooth block, texture block and edge block by using gray-level co-occurrence matrix (GLCM) and sobel operator. Firstly, for CU level, an early termination strategy of CU splitting for smooth CU is proposed to filter out unnecessary coding blocks. Then, for PU level, a fast candidate mode decision method (FCMDM) is proposed to reduce the redundant candidate modes based on PU's types and the prediction mode of neighbor PUs. Finally, a fast wedgelet pattern determination method based on K-Means is explored to reduce the complexity of depth modelling mode 1 (DMM1). Experimental results show that the proposed algorithm achieves an average time reduction of 40.71% for depth intra coding, with negligible drop in encoding quality. The proposed algorithm achieves better the time saving and coding performance compared with the state-of-the-art method.

The aftereffect of visual attention (VA) and foveated masking (FM) are two important HVS characteristics not fully exploited in conventional foveated JND (FJND) models. This paper presents a fixation point estimation method to build adaptive foveated weighting model, by which an improved FJND profile is developed with VA and FM effects considered. The proposed FJND profile overcomes two limitations of the conventional FJND profiles. On one hand, conventional FJND profiles can't get accurate positions of fixation points. This paper proposes a fixation prediction algorithm based on the distribution of cones to identify fixation points and regions. On the other hand, conventional FJND profiles did not incorporate the VA effect. By fully exploiting the VA and FM effects, this work proposes an adaptive foveated weighting model, which is developed as a function of the fixation intensity and retinal eccentricity. Here, the fixation intensity is estimated from the saliency map which is modelled by the Gaussian Mixed Model. With the proposed weight model, a new FJND profile is developed. Experimental results show that the proposed profile tolerates more distortion at the same perceptual image quality compared with other JND profiles.

In this paper, we propose a hybrid scheme to measure depth values for a stereo structured light system with only a single shot. We design a dual-frequency monochromatic pattern, based on which depth values are computed in three steps. Firstly, phases and coarse depth maps are computed by following the idea of Fourier transform profilometry, where a novel phase unwrapping method is proposed. Afterward, errors in coarse depth maps are detected according to cross-view geometry consistency. Finally, spatial stereo matching is conducted to refine the detected errors. Experiments demonstrate that the proposed scheme can generate accurate stereo depth maps with only one shot, which can be used in a range of real-time applications.

Gesture recognition plays an important role in human computer interaction, but the accuracy is unsatisfactory in complex gestures with slight discrimination. In this paper, a framework facing to recognize complex and similar gestures is presented. In the framework, a parallel connection structure of convolutional neural network (CNN) is designed to extract deep features of complex and similar gestures from RGBD images. Then, a novel feature fusion method is proposed to achieve multi-feature fusion and dimension reduction simultaneously. According to experimental results on American Sign Language (ASL) dataset, the proposed framework reaches 99.042% recognition rate and outperforms current state-of-the-art methods.

Human activity recognition (HAR) is widely applied to many industrial applications. In the context of Industry 4.0, driven by the same demand of machines' self-organizing ability, HAR can also be adopted in elderly healthcare. However, HAR should be adaptive to the application scenarios in elderly healthcare. In this paper, we propose a nonintrusive activity recognition method that can be applied to long-term and unobtrusive monitoring for elderlies. The method is robust to obstruction and nontarget object interference. Skeleton sequence is estimated from RGB images. Based on two activity continuity metrics, an interframe matching algorithm is proposed to filter nontarget objects. In order to make full use of spatial-temporal information, we propose a novel activity encoding method based on the interframe joints distances. A convolutional neural network is used to learn the distinguishing features automatically. A specific data augmentation method is designed to avoid the overfitting problem on small-scale datasets. The experiments are performed on two public activity datasets and a newly released noisy activity dataset (NAD). The NAD contains obstruction, nontarget object interference. The experimental results show that the proposed method achieves the state-of-the-art performance while only using one ordinary camera. The proposed method is robust to a realistic environment.

Rendering quality optimization theory is one of the most basic and fascinating components of image-based rendering (IBR). The rendering quality of virtual views is related to the information in the source images. Capturing the image information depends on the geometric configuration of the camera (GCC) and, in particular, the positions and shooting directions of the cameras. Therefore, the rendering quality of virtual views can be improved by optimizing the GCC. This paper investigates the relationship between the GCC and the geometric configurations of the virtual view (GCVV). The influence of the GCC on the rendering quality is also analyzed. Based on the relationships among the GCC, GCVV, and rendering quality, a mathematical model of the noncoverage field (NCF) is proposed to quantify the rendering quality. The performance of the NCF is also analyzed using a set of GCVVs and GCCs. Furthermore, an optimization algorithm based on the NCF that simultaneously optimizes the position and direction of the GCC is presented. The proposed technique can be applied to obtain the optimal rendering quality of IBR for the linear case of camera positions and virtual views. Finally, experimental results are presented and compared with the theoretical results.

Mobile edge caching enables content delivery directly within the radio access network, which effectively alleviates the backhaul burden and reduces round-trip latency. To fully exploit the edge resources, the most popular contents should be identified and cached. Observing that content popularity varies greatly at different locations, to maximize local hit rate, this paper proposes an online learning algorithm that dynamically predicts content hit rate, and makes location-differentiated caching decisions. Specifically, a linear model is used to estimate the future hit rate. Considering the variations in user demand, a perturbation is added to the estimation to account for uncertainty. The proposed learning algorithm requires no training phase, and hence is adaptive to the time-varying content popularity profile. Theoretical analysis indicates that the proposed algorithm asymptotically approaches the optimal policy in the long term. Extensive simulations based on real world traces show that, the proposed algorithm achieves higher hit rate and better adaptiveness to content popularity fluctuation, compared with other schemes.

Scene surface exists some occlusions by mutual occlusions or self-occlusions, and then this phenomenon will serious influences 3D vision technique. We present a signal-processing framework to study occlusions of scene. Our method combines discontinuities and establish a mathematical model of occlusion phenomenon. The influences of occlusion on a scene are derived using Fourier theory. In this manner, spectral support of scene occlusion is analyzed in frequency field. Predictions on the frequency content can then be used to control the sampling and rendering in a scene. This extends previous work that the sampling and rendering to be analyzed and estimated for 3D technique. Extensive experimental evaluation demonstrates that our occlusion analysis method significantly outperform competing algorithms.

Depth videos affect the visual quality of virtual view greatly, while conventional encoders are not adapted to encode depth videos. To solve this problem, we propose, in the paper, a fault-tolerance based joint block-level bit allocation scheme for depth video coding. The scheme classifies depth blocks into two classes by the fault-tolerance, and constructs virtual view perceptual quality based rate-distortion (R-D) model for each class. Based on the model, a joint block-level bit allocation scheme is proposed to obtain the optimized quantization parameter (QP) to encode each class of blocks. Further, an adaptive RDO algorithm is designed to determine λMODE, in order to achieve better visual quality of virtual views. Experimental results demonstrate that, compared with 3D-HEVC, the proposed method improves the visual quality of virtual views, especially at preserving details and boundaries of objects.

Color guided depth up-sampling always suffers from texture copy artifacts and depth discontinuities blurring. This phenomenon is due to the depth discontinuity and color image edges at the corresponding location are not always consistent. In this paper, based on the analysis of the above two problems, we proposed an improved Markov Random Filed (MRF) model, which can reduce negative influence from pixels at the inconsistent location more effectively. Furthermore, the improved MRF model can also reduce the negative influence of noises from depth map. To determine pixels of inconsistent regions, a new concept of pixel confidence is proposed. Pixel confidence indicates the probability that pixel is at the inconsistent regions, which is embedded into the improved MRF model. The proposed method is tested on both the simulated and real datasets. The proposed method can better suppress texture copy artifact and preserve sharp depth discontinuities. Experimental results show that the proposed method also has lower mean absolutely error(MAE) than other methods in most cases.

We consider a control problem in wireless multi-hop networks in which source-destination pairs desire to secure communication. Specifically, a control algorithm is proposed based on the stochastic network optimization to maximize a global utility function, subject to end-to-end secrecy transmission and network stability. To achieve secure communication, we firstly exploit an independent randomization encoding strategy to guarantee the multi-hops secrecy transmission. Then, the control algorithm is decomposed into flow control, routing and resource allocation. Based on the control algorithm, each node makes decisions on the arrival confidential data as well as the users and links. The numerical analysis illustrates that the proposed algorithm can achieve a utility result, arbitrarily close to optimal value.

In this paper, we study worker selection in spatial crowdsourcing, which is the recruitment of human workers in a specific location to collect geographical data. To achieve better performance, spatial crowdsourcing task relies on both worker's effort and skill. Therefore, to maximize the long-term platform utility, we exploit fog platform as a service to identify valuable workers through learning their performance information. Worker's historical performance data are recorded at local fog server, based on which valuable workers are identified and selected to perform the tasks. During worker selection, we aim at balancing the exploration and exploitation, and propose an online algorithm that promotes workers who are not fully explored. With budget constraint, the proposed algorithm is able to maximize the long-term platform utility. Theoretical analysis indicates that the proposed learning algorithm achieves asymptotically diminishing regret. Finally, extensive simulations on real-world dataset are conducted, which demonstrate the advantage of our algorithm over other methods.

Recently, performance analysis for large-scale content-centric mobile ad hoc networks (MANETs) has received intense attention. In content-centric MANETs, content delivery consists of two operations, i.e., content placement and content retrieval, which may involve different network costs. However, the existing performance studies in content-centric MANETs mainly focus on content retrieval, and hence may not reflect the impact of content placement. In this paper, we investigate the asymptotic throughput and delay performance by considering the two operations of possibly different network costs. In particular, we introduce a general weighted sum delay cost of content placement and content retrieval as the delay performance metric. We consider an arbitrary content popularity distribution and study two mobility models in different time scales, i.e., fast and slow mobility. For each mobility model, we characterize the impacts of the network parameters on the network performance. By optimizing the content placement and retrieval for contents of different popularities, we design a general near-optimal scheme, the parameters of which reflect the delay weights of the two phases. We show that the network performance improves as the number of cached replicas increases until the number reaches a threshold. Finally, we show that our results are general and can incorporate some existing results as special cases.

In this paper, a new method is proposed to generate image-assisted geometry simplification for estimating minimum sampling rate of image-based rendering (IBR). If some geometry information (e.g., the shape of object surface and proxy geometry) on a scene is known, we can decompose the scene geometry into a collection of simpler structures on a block-by-block basis. Our framework predicts the characteristics of simpler structure such as irregular object. Predictions on the frequency content can then be used to control sampling rates for IBR. The new method allows the sampling of the IBR to be analyzed and estimated for the non-uniform sampling. Furthermore, the minimum sampling rate of the IBR necessary for alias-free rendering will be reduced as the number of simpler structures increases.

Compressed sensing (CS) provides a new method to encode depth videos, which utilizes the sparsity of depth maps to improve the coding efficiency. In this paper, we design a novel CS-based depth video codec. The codec adaptively decomposes blocks from 64 χ 64 to 8 χ 8 sub-blocks via wavelet transforming. The decomposition divides smooth regions and complex boundaries by frequencies, and minimizes the local entropy of sub-blocks. Moreover, a measurement rate allocation algorithm is also proposed, which utilizes the rate-distortion optimization (RDO) to allocate the measurement rate for each block. The experimental results demonstrate that, compared with H.264/AVC and H.265/HEVC, the proposed codec improves the quality of virtual views by 1-2 dB and 0.2-0.5 dB PSNR respectively. Meanwhile, the codec reduces coding complexity greatly.

The quality of depth is crucial in all depth-based applications. Unfortunately, the error-free ground truth is often unattainable for depth. Therefore, no-reference quality assessment is very much desired. This paper presents a novel depth quality assessment scheme that is completely different from conventional approaches. In particular, this scheme focuses on depth edge misalignment errors in texture-plus-depth (T + D) images and develops a robust method to detect them. Based on the detected misalignments, a no-reference metric is calculated to evaluate the quality of depth maps. In the proposed scheme, misalignments are detected by matching texture and depth edges through three constraints: 1) spatial similarity; 2) edge orientation similarity; and 3) segment length similarity. Furthermore, the matching is performed on edge segments instead of individual pixels, which enables robust edge matching. Experimental results demonstrate that the proposed scheme can detect misalignment errors accurately. The proposed no-reference depth quality metric is highly consistent with the full-reference metric, and is also well-correlated with the quality of synthesized virtual views. Moreover, the proposed scheme can also use the detected edge misalignments to facilitate depth enhancement in various practical texture-plus-depth-based applications.

In the applications based on depth, multiple TOF cameras are often required to capture the same scene. But if multiple cameras operate simultaneously on the same frequency, they interfere with each other. The multi-camera interference causes a lot of errors in depth measurement. The depth quality is severely reduced, which limits the application of TOF cameras and needs to be resolved. In this paper, a multi-camera interference model is presented. The interference signal for multiple frames is proved to be an ergodic and wide-sense stationary stochastic process. The least square estimation of noninterference signal is proposed to remove the multi-camera interference. The results of experiments prove the approach can recover the depth and amplitude information of multiple TOF cameras from the severe interference.

In this paper, a large-area depth recovery method for RGB-D camera is proposed. Considering that pixels along edges between different regions usually share similar depth values, we first select reliable pixels along edges of large-area depth missing regions and project them into the world coordinate system. Then, by examining the distribution of these pixels, we apply a weighted least squares method to approximate the surface function. With the help of the surface function, missing depth values can be recovered correctly. To the best of our knowledge, this is the first recovery method focusing on large areas of missing depth information. Qualitative evaluation demonstrates the effectiveness of the proposed method.

This paper presents a texture-free depth enhancement method for large-area depth recovery. The proposed algorithm identifies a large-area depth missing region, and iteratively segments its contour by setting different initial pixels in each iteration. Coordinate transformation is used to analyze the distribution of each contour segment. By examining distributions of all contour segments, statistical histogram analysis is applied in our approach to select contour pixels. Then, selected pixels are projected into the world coordinate system, and multiple linear regression is utilized for surface function approximation. Missing depth values of a large-area depth missing region can be recovered with guidance of the approximated surface function. Quantitative and qualitative evaluations over state-of-the-art depth enhancement methods demonstrate the effectiveness and superiority of our method. Being texture-free, the proposed method has the flexibility of being merged into traditional depth enhancement methods.

With the advent of smart portable devices and location-based applications, user's mobility pattern is found to be highly dependent on varying locations. In this paper, we analyze asymptotic throughput-delay performance of mobile ad hoc networks (MANETs) under a location popularity based scenario, where users are more likely to visit popular locations. This work provides a complementary perspective compared with previous studies on fundamental scaling laws for MANETs, mostly assuming that nodes move uniformly in the network. Specifically, we consider a cell-partitioned network model with cells of known popularity, which follows a Zipf's law distribution with popularity exponent α. We first conduct the analysis under traditional store-carry-forward paradigm, and find that location heterogeneity affects the network performance negatively, which is due to the waste of potential transmission opportunities in popular cells. Motivated by this observation, we further propose a novel store-carry-accelerate-forward paradigm to enhance the network communication, exploiting these potential transmissions. Theoretical results demonstrate that our proposed scheme outperforms all delay-capacity results obtained in conventional scheme for any α. In particular, when α = 1, it can achieve a constant capacity with an average delay of Θ(√n) (except for a polylogarithmic factor), while the delay is Θ(n) in conventional scheme. And by letting α = 0, our results can cover Neely's scaling laws. Moreover, we show that the delay-capacity tradeoff ratio satisfies ≥Θ(√n), revealing that exploiting location popularity can effectively improve the performance in MANETs.

In the asymptotic analysis of large scale mobile networks, most previous works assume that nodes move in all the cells identically. We put forward this line of research by considering location popularity, which is verified by recent experimental studies. Nodes tend to visit some popular locations and go to other locations less frequently. We first analyze its multicast capacity and delay under the traditional 2-hop store-carry and forward paradigm. With different location popularity distributions, network capacity and delay will vary according to the distribution exponent. As the popularity becomes more diverse, less concurrent transmissions are tolerated in the network, which brings down the performance. Observing that transmission opportunity is not fully utilized in popular cells for 2-hop paradigm, we put forward a 3-hop scheme, which is called store-carry-accelerate-forward scheme. In this 3-hop scheme, each packet is first sent to an initial relay, who carries the packets into the popular cells and then broadcasts to multiple nodes to accelerate the delivery. By so doing, we showed that the 3-hop scheme outperforms the 2-hop scheme for all popularity distributions. Furthermore, we study the delay-capacity tradeoffs for multicast under both schemes and the buffer needed for stability requirement. Our results reveals the joint impact of multicast and location heterogeneity on the design of transmission schemes, and may shed new insights for future studies.

In 3D video (3DV) and free-viewpoint video (FVV), it is vitally important to detect the errors and assess depth quality. However, since ground-truth depth maps are often unattainable, assessing depth quality without reference becomes an imperative task for many applications. This research considers the texture-plus-depth format in 3DV and FVV, and focuses on the misalignment error at depth discontinuities. A matching algorithm between depth and texture edges is proposed to determine corresponding matching pairs and to identify serious mismatches. The matching procedure is based on both spatial distances and direction similarities between texture and depth edges, and the algorithm is performed between edge segments, instead of edge pixels, in order to improve the robustness of matching. Furthermore, an adaptive algorithm is designed to divide depth edges into segments with different lengths based on the curvature of the edges. After the matching is completed, misalignments between matching pairs are used to generate a no-reference assessment metric. Experimental results demonstrate that the proposed matching scheme is able to achieve accurate matches between depth and texture edges. More importantly, it has been shown that the correlations between the proposed metric and the widely accepted full-reference metric are greater than 0.9, making this no-reference depth quality assessment scheme suitable for contemporary 3DV and FVV applications.

In this paper, we investigate the achievable throughput capacity of three-dimensional (3D) inhomogeneous wireless ad hoc networks. In particular, we consider nodes placed according to Shot Noise Cox process, which allows to model the clustering behavior usually recognized in large-scale systems. For this class of networks, we partition the transmission into two parts: the intra-cluster transmission and the inter-cluster transmission. For the intra-cluster transmission, we firstly construct a spheroidal percolation model in which the highway system exists. Different from regular 2D percolation model, the highways in our model are in the radial direction and around the cluster. Based on our model, we propose a novel routing strategy with five phases and derive the per-node rate for the intra-cluster transmission. As for the inter-cluster transmission, we build some "information pipes'' to connect the clusters and then get the inter-cluster transmission rate. Finally, we take the transmission of the whole networks into account and obtain the lower bound of the throughput capacity of 3D inhomogeneous networks.

Video delivery in television white space (TVWS) is a challenging task due to the variation of the spectrum availability in TVWS and the stringent bandwidth requirements of video streaming. Most existing frameworks employ an opportunistic spectrum access style which result in bandwidth inefficiency since only one spectrum hole in TVWS can be used. In this paper, we propose a bandwidth efficient TVWS video delivery framework. First, a multisector structure is employed to improve the bandwidth efficiency. Second, a novel spectrum sensing scheme and an efficient spectrum selection algorithm are proposed to obtain the optimal spectrum holes based on this structure. Third, a joint optimization approach is proposed to create an optimal map between the sectors and the spectrum, and to adapt the video rate commensurate with dynamic channel conditions. The simulation results demonstrate that the proposed framework can obtain significantly higher bandwidth efficiency and superior video performance than traditional approaches.

Previous works on multicast capacity mainly focus on deriving asymptotic order results in large-scale wireless networks, which can explore the general scaling laws of throughput capacity but cannot predict the exact achievable throughput. In this paper, we investigate the non-asymptotic capacity of multihop wireless networks for multicast applications wherein for each source node, k nodes are randomly selected as receivers. Since multicast routing has a dynamic nature, it is challenging for the exact performance analysis. To tackle the problem, we propose an explicit analytical model which describes multicast transmissions, considers networks of arbitrary size, takes data burst into account, and also covers the notion of time scales for transient analysis. By developing a practical multicast scheme, stochastic network calculus is employed for the exact analysis. With the analytical model, we derive lower and upper bounds on multicast capacity, which are non-asymptotic functions of the above variables, and also recover the scaling laws from an asymptotic point of view. Simulations further verify the accuracy of the analytical bounds.

In static hybrid wireless networks (SHWNs), it requires a large deployment cost in order to achieve a throughput of Θ(1)1. Motivated by this shortcoming, mobile hybrid wireless networks (MHWNs) are proposed in the most recent work, which can boost throughput to Θ(1) at a smaller cost under the same cell routing policy. However, such an improvement comes at the expense of larger delay. In this paper, we aim at reducing the delay while maintaining the optimal throughput in MHWNs. Hence, a novel scheme with K length routing policy is proposed, which combines multi-hop relay, mobile relay and infrastructure relay for the first time. Our main results demonstrate that a significant gain in delay is realized by the proposed scheme compared with the existing work of MHWNs while guaranteeing a throughput of Θ(1).

The plenoptic sampling theory studies the sampling rate of the 3D scene, and the spatial sampling rate can be determined by the spectrum analysis of the plenoptic function. In this paper we focus on the spectrum analysis of the non-Lambertian reflection. We establish a new scene model of non-Lambertian reflection by which the lightfield is resolved as an integration of the texture information, the illumination and the Bidirectional Reflectance Distribution Function (BRDF). Our analysis shows that the spectrum is expanded in the non-Lambertian reflection and the expanding range is controlled by the material property. More accurately, in the Phong model, the expanded range is controlled by a Gaussian function which the variance is Phong exponent. Our conclusion can be used to get a more accurate sampling rate when the object property is approximately known.

Microsoft Kinect cameras provide a fast and convenient way to acquire depth information. However, the interference problem of multiple Kinect cameras dramatically degrades the depth quality. In this paper, we study the interference problem and propose an interference cancelation approach based on the statistical properties of depth maps. The gradient values are investigated and propagated from the interference-free region to the interfered region. The gradient values are obtained based on the statistic gradient features of the depth maps and the optimal solution of depth values is derived with a least error criterion. Experiment results demonstrate that our proposed method can eliminate interference efficiently, and lead to better qualities of depth maps and rendered virtual views.

Mobile 3D video communication, especially with 2D and 3D compatible, is a new paradigm for both video communication and 3D video processing. Current techniques face challenges in mobile devices when bundled constraints such as computation resource and compatibility should be considered. In this work, we present a wireless 2D and 3D switchable video communication to handle the previous challenges, and name it as Stereotime. The methods of Zig-Zag fast object segmentation, depth cues detection and merging, and texture-adaptive view generation are used for 3D scene reconstruction. We show the functionalities and compatibilities on 3D mobile devices in WiFi network environment.

Depth image upsampling is an important issue in three-dimensional (3D) applications. However, edge blurring artifacts are still challenging problems in depth image upsampling, resulting in jagged artifacts in synthesized views which produce unpleasant visual perception. In this paper, an edge-preserving single depth image interpolation (ESDI) method is proposed. Specifically, local planar hypothesis (LPH) assuming that depth in natural scene are clustered as local planar planes is first explored. Then finite candidates generation (FCG) is proposed to generate limited discrete values satisfied with LPH to interpolated pixels. At last, the optimal combination of candidates is formulated as an energy minimization problem with a constraint in gradient domain, solved by iterated conditional modes (ICM) algorithm. Experiments demonstrate that ESDI achieves high resolution (HR) depth image with clear and sharp edges, and produces synthesized views with desirable quality.

In this paper, we investigate the capacity of ad hoc wireless networks with directional antennas and multi-packet reception (MPR), which is referred to as DMPR wireless networks hereinafter. MPR has been proven to be an effective technology to improve the capacity by increasing the number of concurrent transmission, however the capacity of MPR networks is constrained by interference. The directional antennas is a straightforward way to mitigate interference and increase the spatial reuse. To further improve the capacity, we firstly introduce the directional antennas into MPR networks, and then conduct the theoretical analysis on the capacity of DMPR wireless networks. Under both arbitrary and random networks, we derive the matching theoretical upper bound and constructive lower bound. The final results demonstrate that a higher capacity can be obtained by such networks.

Video delivery in TV White Space (TVWS) is a challenging task due to the variation of the spectrum availability in TVWS and the stringent bandwidth requirements of video streaming. Most existing frameworks employ an opportunistic spectrum access style which result in bandwidth inefficiency since only one spectrum hole in white space can be used. In this paper, we propose a bandwidth efficient white space video delivery framework. In our framework: firstly, multi-sector structure is employed to improve bandwidth efficiency; secondly, a novel spectrum sensing scheme and an efficient spectrum selection scheme are proposed to obtain optimal spectrum holes based on this structure; thirdly, a optimal resources scheduling scheme is proposed to further improve the system performance. The simulation results demonstrate that the proposed framework can obtain significantly higher bandwidth efficiency and superior video performance than traditional frameworks.

Preserving the edges in depth compression is important for improving the synthesized view quality, this paper presents a novel edge-preserving depth up-sampling method for down/up sampling-based depth coding using both the texture and depth information. We take into account the edge similarity between depth maps and their corresponding texture images as well as the structural similarity among depth maps to build a weight model. Based on the weight model, the optimal MMSE up-sampling coefficients are estimated from the local covariance coefficients of the down-sampled depth map. Experimental results show that our proposed interpolation method for down/up sampling-based depth coding improves both the coding efficiency and synthesized view quality.

Depth edge preservation is important for improving synthesized view quality in depth map coding. A joint texture/depth edge-directed up-sampling algorithm for depth map coding is proposed in this paper. The depth up-sampling algorithm takes into account the edge similarity between depth map and corresponding texture image, and structural similarity between low resolution depth map and high resolution depth map. Based on the these characters, the optimal MMSE up sampling coefficients are estimated from the local covariance coefficients of down-sampled depth map and corresponding texture image. Experimental results show that the proposed up-sampling algorithm for depth map coding improves both depth map coding efficiency and synthesized view quality.

In this letter, we propose a cross-view down/up-sampling (CDU) method for the framework of reduced resolution multiview depth video coding, which exploits cross-view information to assist the up-sampling at the decoder. In the down-sampling procedure of CDU, the odd-even interlaced extraction is employed to preserve more confident information of the original depth video with reduced resolution. In the decoder, the cross-view information is exploited for up-sampling the reconstructed depth video. An iterative interpolation process is proposed to eliminate the effect of compression distortion on this up-sampling. Experimental results demonstrate the gains of up to 3.88 dB for the proposed algorithm and better quality of synthesized views.

View synthesis technology generates virtual views for display and high quality virtual view is of significant importance to free-viewpoint TV (FTV) and three dimensional video(3DV). This paper proposes a novel virtual view synthesis method in multi-view system based on multi-view plus depth. Firstly, we project two reference depth maps to the intermediate virtual view and rectify the two candidate virtual depth maps. Secondly, we project reference texture to the virtual view with the rectified depth maps and blend the candidate virtual views. Finally, we in paint the remaining holes according to the adjacent depth and color samples in background regions. Experiment results demonstrate that the proposed method works well and generates high quality intermediate virtual views.