Dynamic LiDAR point cloud compression (LPCC) is crucial for the efficient transmission and storage of large-scale three-dimensional data in applications such as autonomous driving. However, many existing methods, which primarily focus on compressing geometric or motion information, face a fundamental limitation: they treat all points as equally important. This approach neglects the semantic priorities of a scene, resulting in inefficient bit allocation and particularly compromising the reconstruction quality of safety-critical regions, such as pedestrians and vehicles, which are vital to downstream perception tasks. To address these limitations, we propose R²D-LPCC, a relevance-ranking framework for region adaptive LPCC that prioritizes fidelity in semantically important regions. Central to our approach is the Adaptive Relevance Learning (ARL) module, which integrates semantic context with uncertainty to evaluate regional significance and guide compression. We also introduce a Multi-scale Region-Adaptive Transform (MRAT) module to enhance semantic feature modeling and preserve fine-grained details in key areas. Additionally, we develop an adaptive multi-modal motion estimation module to improve motion prediction in complex three-dimensional environments. Extensive experiments conducted on the SemanticKITTI benchmark demonstrate that R²D-LPCC significantly surpasses ten recent state-of-the-art methods, achieving a 45.48% BD-rate gain over the previous leading method, Unicorn, and a 98.58% gain over the GPCC standard, while ensuring superior reconstruction quality in semantically important regions.

Existing learned point cloud compression frameworks face two major limitations: (1) they focus almost exclusively on spatial redundancy and (2) rely on architectures built around local-global transformers or global Mamba blocks. Transformers incur quadratic complexity, while global Mamba lacks the granularity to capture structured correlations across multiple dimensions. We propose OctMamba, the first unified framework to jointly exploit spatial, channel, and topological redundancies, dimensions previously overlooked in point cloud geometry compression. Our approach introduces a new architectural principle: embedding Mamba modules within specialized subcomponents rather than applying them globally, challenging existing design paradigms. OctMamba combines two modules: Spatial-Channel Coupled Grouping Mamba (SCCGM) for spatial-channel fusion and Local Graph CNN-Mamba (LGCM) for topological encoding. This design enables efficient long-range modeling with linear complexity, delivering a smaller model and faster decoding while outperforming transformer-based and global Mamba baselines. On SemanticKITTI, OctMamba reduces bitrate by 60.2% over GPCC (D1 PSNR) and achieves state-of-the-art performance across LiDAR and dynamic human point cloud benchmarks with practical speed and scalability. By introducing multi-dimensional redundancy modeling, OctMamba has the potential to influence future research on efficient point cloud compression. The code is available at https://github.com/ZjgsVMC/OctMamba .

The storage and transmission of large-scale meteorological data have become a significant bottleneck hindering the in-depth development of meteorological research. This study proposes a novel meteorological data compression framework characterized by innovative thinking. It embeds the Mamba structure during the feature extraction stage and integrates a spatio-temporal channel attention mechanism to construct an entropy coding mechanism that facilitates the sharing of high-dimensional meteorological data channel information. The framework achieves a compression rate of 300x and, more importantly, preserves the essential spatio-temporal features of meteorological data. Validation on the ERA5 dataset demonstrates that the proposed method surpasses existing techniques in terms of both data compression efficiency and reconstruction accuracy. The accuracy of meteorological analyses based on compressed data is comparable to that of the original data, offering a highly practical solution to the challenges of storing meteorological big data. Our source code is publicly available online at https://github.com/cike0cop/MambaComp .

We propose a novel unsupervised cross-modal homography estimation learning framework, named Split Supervised Homography estimation Network (SSHNet). SSHNet reformulates the unsupervised cross-modal homography estimation into two supervised sub-problems, each addressed by its specialized network: a homography estimation network and a modality transfer network. To realize stable training, we introduce an effective split optimization strategy to train each network separately within its respective sub-problem. We also formulate an extra homography feature space supervision to enhance feature consistency, further boosting the estimation accuracy. Moreover, we employ a simple yet effective distillation training technique to reduce model parameters and improve cross-domain generalization ability while maintaining comparable performance. The training stability of SSHNet enables its cooperation with various homography estimation architectures. Experiments reveal that the SSHNet using IHN as homography estimation network, namely SSHNet-IHN, outperforms previous unsupervised approaches by a significant margin. Even compared to supervised approaches MHN and LocalTrans, SSHNet-IHN achieves 47.4% and 85.8% mean average corner errors (MACEs) reduction on the challenging OPT-SAR dataset. Source code is available at https://github.com/Junchen-Yu/SSHNet.

The increasing data volume and the demand for real-time transmission highlight the necessity for efficient compression of dynamic point cloud data. Existing methods primarily focus on reducing inter-frame redundancy by calculating per-point motion information, overlooking the computational and storage costs involved. In this paper, we propose a novel motion proxy-based dynamic point cloud compression framework to enhance the efficiency and accuracy of motion information utilization. Specifically, we introduce a feature proxy module to adaptively locate proxy points, which represent the overall motion through the motion of proxy points. Additionally, a motion enhancement module is employed to refine motion details and prevent local information loss caused by dense motion trajectories. Extensive experiments demonstrate the superiority of our approach. Compared with baseline methods, it achieves an average BD-rate improvement of 12.07% (D1) and 11.90% (D2).

Deep learning methods have recently shown significant promise in compressing the geometric features of point clouds. However, challenges arise when consecutive point clouds contain holes, resulting in incomplete information that complicates motion estimation. To our knowledge, most existing dynamic point cloud compression methods have largely overlooked this critical issue. Moreover, these methods typically employ a multi-scale single-pass approach for motion estimation, performing only one estimation at each scale. This limits accuracy and adversely impacts compression performance. To address these challenges, we propose a dynamic point cloud compression model called M2BR-DPCC (Multi-Modal Multi-Scale Bidirectional Recursion for Dynamic Point Cloud Compression). Our method introduces two key innovations. First, we integrate both point cloud and image data as inputs, leveraging a multi-modal feature representation completion (MFRepC) approach to align information across modalities. This addresses the issue of missing data in point clouds by using complementary information from images. Second, we implement a multi-scale bidirectional recursive (MSBR) motion estimation method. This module iteratively refines motion flows in both forward and backward directions, progressively enhancing point cloud features and improving motion estimation accuracy. Experimental results on widely used datasets, including MVUB and 8iVFB, demonstrate the effectiveness of our approach. Compared to existing methods, M2BR-DPCC achieves superior performance, with an average BD-rate improvement of 95.23% over V-PCC, 12.92% over D-DPCC, and 16.16% over patchDPCC. These results underscore the potential of leveraging multi-modal data and bidirectional refinement for dynamic point cloud compression.

Previous deep image registration methods that employ single homography, multi-grid homography, or thin-plate spline often struggle with real scenes containing depth disparities due to their inherent limitations. To address this, we propose an Exponential-Decay Free-Form Deformation Network (EDFFDNet), which employs free-form deformation with an exponential-decay basis function. This design achieves higher efficiency and performs well in scenes with depth disparities, benefiting from its inherent locality. We also introduce an Adaptive Sparse Motion Aggregator (ASMA), which replaces the MLP motion aggregator used in previous methods. By transforming dense interactions into sparse ones, ASMA reduces parameters and improves accuracy. Additionally, we propose a progressive correlation refinement strategy that leverages global-local correlation patterns for coarse-to-fine motion estimation, further enhancing efficiency and accuracy. Experiments demonstrate that EDFFDNet reduces parameters, memory, and total runtime by 70.5%, 32.6%, and 33.7%, respectively, while achieving a 0.5 dB PSNR gain over the state-of-the-art method. With an additional local refinement stage,EDFFDNet-2 further improves PSNR by 1.06 dB while maintaining lower computational costs. Our method also demonstrates strong generalization ability across datasets, outperforming previous deep learning methods.

Federated learning, as a collaborative model training method that protects data privacy, has become a prominent research focus in machine learning. Currently, many heterogeneous federated learning algorithms have been proposed to address the challenges posed by data and model heterogeneity among clients. However, existing heterogeneous federated learning algorithms typically do not take the specific structural characteristics of models into account. Instead, they tend to rely on simple, generic models for federated learning training, which may lead to insufficient feature extraction capabilities for certain client models, making it difficult to adapt to various data distributions. Moreover, these algorithms often replace the client models with the global model directly, which undermines the generalization ability of the client models and leads to a lack of personalization, preventing the client models from achieving optimal performance. To tackle these issues, we propose a novel heterogeneous federated learning algorithm, FedSA, which combines a structural-aware federated learning optimization algorithm with a client prediction header aggregation algorithm based on precise encoded features. While keeping the parameters of the participating training models unchanged, it enhances the feature extraction capability of the local client and improves the personalization of the client model. Experimental results show that the proposed method achieves significant improvements in accuracy compared to existing methods across multiple datasets and heterogeneous model scenarios.

We propose Dual-Branch Network (DBNet), a novel deepfake detection framework that addresses key limitations of existing works by jointly modeling 3D-temporal and fine-grained texture representations. Specifically, we aim to investigate how to (1) capture dynamic properties and spatial details in a unified model and (2) identify subtle inconsistencies beyond localized artifacts through temporally consistent modeling. To this end, DBNet extracts 3D landmarks from videos to construct temporal sequences for an RNN branch, while a Vision Transformer analyzes local patches. A Temporal Consistency-aware Loss is introduced to explicitly supervise the RNN. Additionally, a 3D generative model augments training data. Extensive experiments demonstrate our method achieves state-of-the-art performance on benchmarks, and ablation studies validate its effectiveness in generalizing to unseen data under various manipulations and compression.

We propose a normalizing flow based on the wavelet framework for super-resolution (SR) called WDFSR. It learns the conditional distribution mapping between low-resolution images in the RGB domain and high-resolution images in the wavelet domain to simultaneously generate high-resolution images of different styles. To address the issue of some flow-based models being sensitive to datasets, which results in training fluctuations that reduce the mapping ability of the model and weaken generalization, we designed a method that combines a T-distribution and QR decomposition layer. Our method alleviates this problem while maintaining the ability of the model to map different distributions and produce higher-quality images. Good contextual conditional features can promote model training and enhance the distribution mapping capabilities for conditional distribution mapping. Therefore, we propose a Refinement layer combined with an attention mechanism to refine and fuse the extracted condition features to improve image quality. Extensive experiments on several SR datasets demonstrate that WDFSR outperforms most general CNN- and flow-based models in terms of PSNR value and perception quality. We also demonstrated that our framework works well for other low-level vision tasks, such as low-light enhancement. The pretrained models and source code with guidance for reference are available at https://github.com/Lisbegin/WDFSR.

ABSTRACT This study focuses on transforming real‐world scenery into Chinese landscape painting masterpieces through style transfer. Traditional methods using convolutional neural networks (CNNs) and generative adversarial networks (GANs) often yield inconsistent patterns and artifacts. The rise of diffusion models (DMs) presents new opportunities for realistic image generation, but their inherent noise characteristics make it challenging to synthesize pure white or black images. Consequently, existing DM‐based methods struggle to capture the unique style and color information of Chinese landscape paintings. To overcome these limitations, we propose CLPFusion, a novel framework that leverages pre‐trained diffusion models for artistic style transfer. A key innovation is the Bidirectional State Space Models‐CrossAttention (BiSSM‐CA) module, which efficiently learns and retains the distinct styles of Chinese landscape paintings. Additionally, we introduce two latent space feature adjustment methods, Latent‐AdaIN and Latent‐WCT, to enhance style modulation during inference. Experiments demonstrate that CLPFusion produces more realistic and artistic Chinese landscape paintings than existing approaches, showcasing its effectiveness and uniqueness in the field.

ABSTRACT Speech‐driven talking face video generation has attracted growing interest in recent research. While person‐specific approaches yield high‐fidelity results, they require extensive training data from each individual speaker. In contrast, general‐purpose methods often struggle with accurate lip synchronization, identity preservation, and natural facial movements. To address these limitations, we propose a novel architecture that combines an alignment model with a rendering model. The rendering model synthesizes identity‐consistent lip movements by leveraging facial landmarks derived from speech, a partially occluded target face, multi‐reference lip features, and the input audio. Concurrently, the alignment model estimates optical flow using the occluded face and a static reference image, enabling precise alignment of facial poses and lip shapes. This collaborative design enhances the rendering process, resulting in more realistic and identity‐preserving outputs. Extensive experiments demonstrate that our method significantly improves lip synchronization and identity retention, establishing a new benchmark in talking face video generation.

Thermal infrared (TIR) images suffer from low contrast due to the atmospheric thermal radiation effect, especially under extreme conditions like low temperature. TIR image enhancement aims to improve image contrast, but previous enhancement approaches usually produce enhanced results with two limitations: spatial inconsistency and detail blurring. To deal with the limitations, we propose a novel TIR image enhancement method, named TEFormer, to preserve spatial consistency and restore fine-grained details during image enhancement. To preserve spatial consistency, we devise the global enhancement module (GEM) to enhance the low-resolution representation. The GEM performs long-range interactions across spatial dimensions and channel dimensions to condition the enhancement curve fitting. To keep details clear, we design the local enhancement module (LEM) as the decoding unit. The LEM injects additional detail structures into the enhanced low-resolution representation for high-resolution reconstruction. Besides, we further apply histogram-based supervision to facilitate learning in intensity distribution of clear images. Extensive experimental results on three challenging benchmarks demonstrate that the proposed method outperforms other state-of-the-art approaches.

Along with the rapid technological and commercial innovation on e-commerce platforms, an increasing number of frauds cause great harm to these platforms. Many frauds are conducted by organized groups of fraudsters for higher efficiency and lower costs, also known as group-based frauds. Despite the high concealment and strong destructiveness of group-based fraud, no existing research can thoroughly exploit the information within the transaction networks of e-commerce platforms for group-based fraud detection. In this work, we analyze and summarize the characteristics of group-based frauds. Based on this, we propose a novel end-to-end semi-supervised Group-based Fraud Detection Network (GFDN) to support such fraud detection in real-world applications. In addition, we introduce a module namedTemporal Group Dynamics Analyzer(TGDA) that strengthens the ability to analyze temporal information on group fraudulent activity. Based on this, we built an enhanced model named TGFDN. Experimental results on large-scale e-commerce datasets from Taobao and Bitcoin trading datasets show our proposed model's superior effectiveness and efficiency for group-based fraud detection on bipartite graphs.

Abstract Scene cartoonization aims to convert photos into stylized cartoons. While generative adversarial networks (GANs) can generate high‐quality images, previous methods focus on individual images or single styles, ignoring relationships between datasets. We propose a novel multi‐style scene cartoonization GAN that leverages multiple cartoon datasets jointly. Our main technical contribution is a multi‐branch style encoder that disentangles representations to model styles as distributions over entire datasets rather than images. Combined with a multi‐task discriminator and perceptual losses optimizing across collections, our model achieves state‐of‐the‐art diverse stylization while preserving semantics. Experiments demonstrate that by learning from inter‐dataset relationships, our method translates photos into cartoon images with improved realism and abstraction fidelity compared to prior arts, without iterative re‐training for new styles.

Color composition (or color theme) is a key factor to determine how well a piece of art work or graphical design is perceived by humans. Despite a few color harmony models have been proposed, their results are often less satisfactory since they mostly neglect the variations of aesthetic cognition among individuals and treat the influence of all ratings equally as if they were all rated by the same anonymous user. To overcome this issue, in this article we propose a new color theme evaluation model by combining a back propagation neural network and a kernel probabilistic model to infer both the color theme rating and the user aesthetic preference. Our experiment results show that our model can predict more accurate and personalized color theme ratings than state of the art methods. Our work is also the first-of-its-kind effort to quantitatively evaluate the correlation between user aesthetic preferences and color harmonies of five-color themes, and study such a relation for users with different aesthetic cognition.

Reconstructinga 3D face from a single image is a crucial task in numerous multimedia applications. Face images with ground-truth 3D face shapes are scarce, so unsupervised deep learning methods, which rely primarily on the free supervision signal derived from the visual disparity between the input image and the rendered counterpart of the predicted 3D face, have proven superior for reconstructing 3D faces. However, it is challenging for such techniques to decouple the dynamic 3D face properties such as pose or expression from a single 2D image, especially when similar local visual appearance changes can be caused by both pose and expression motion, resulting in imprecise 3D face reconstruction. In this article, a novel cycle-consistency in dynamic 3D face characteristics is introduced as a free supervisory signal for learning accurate 3D face shapes from unlabeled facial images. The main idea of cycle-consistency is to explicitly inject the head pose or facial expression variation between video frames into a face image, and then to extract and reverse the injected variation in order to reconstruct the face image to its original state. In our model, a CNN network with multiple branches is proposed to disentangle 3D face properties like identity, expression, pose, and texture from 2D facial images, one branch for each 3D face property. During training, our model learns to completely decouple the dynamic 3D face properties (pose and expression) to be useful for performing cycle-consistent face reconstruction. Extensive experiments demonstrate the superiority of our approach. On the challenging AFLW2000-3D, MICC Florence, and NoW datasets, our method outperforms or is on par with the state of the art.

Deep metric learning is crucial for finding an embedding function that can generalize to training and testing data, including unknown test classes. However, limited training samples restrict the model’s generalization to downstream tasks. While adding new training samples is a promising solution, determining their labels remains a significant challenge. Here, we introduce Hybrid Species Embedding (HSE), which employs mixed sample data augmentations to generate hybrid species and provide additional training signals. We demonstrate that HSE outperforms multiple state-of-the-art methods in improving the metric Recall@K on the CUB-200 , CAR-196 and SOP datasets, thus offering a novel solution to deep metric learning’s limitations.

Point cloud saliency detection is an important technique that support downstream tasks in 3D graphics and vision, like 3D model simplification, compression, reconstruction and viewpoint selection. Existing approaches often rely on hand-crafted features and are only applicable to specific datasets. In this paper, we propose a novel weakly supervised classification network, called C2SPoint, which directly performs saliency detection on the point clouds. Unlike previous methods that require per-point saliency annotations, C2SPoint only requires category labels of the point clouds during training. The network consists of two branches: a Classification branch and a Saliency branch. The former branch is composed of two Adaptive Set Abstraction layers for feature extraction and a Saliency Transform layer for learning saliency knowledge from the classification network. The latter branch introduces a multi-scale point-cluster similarity matrix for propagating the cluster saliency to each point within it, resulting in the prediction of point-level saliency. Experimental results demonstrate the effectiveness of our method in point cloud saliency detection, with improvements of 2% in both AUC and NSS compared to state-of-the-art methods.

Abstract We present a novel approach to differentiable rendering for participating media, addressing the challenge of computing scene parameter derivatives. While existing methods focus on derivative computation within volumetric path tracing, they fail to significantly improve computational performance due to the expensive computation of multiply‐scattered light. To overcome this limitation, we propose a differential diffusion theory inspired by the classical diffusion equation. Our theory enables real‐time computation of arbitrary derivatives such as optical absorption, scattering coefficients, and anisotropic parameters of phase functions. By solving derivatives through the differential form of the diffusion equation, our approach achieves remarkable speed gains compared to Monte Carlo methods. This marks the first differentiable rendering framework to compute scene parameter derivatives based on diffusion approximation. Additionally, we derive the discrete form of diffusion equation derivatives, facilitating efficient numerical solutions. Our experimental results using synthetic and realistic images demonstrate the accurate and efficient estimation of arbitrary scene parameter derivatives. Our work represents a significant advancement in differentiable rendering for participating media, offering a practical and efficient solution to compute derivatives while addressing the limitations of existing approaches.

Judging how an image is visually appealing is a complicated and subjective task. This highly motivates having a machine learning model to automatically evaluate image aesthetic by matching the aesthetics of general public. Although deep learning methods have been successfully learning good visual features from images, correctly assessing image aesthetic quality is still challenging for deep learning. To tackle this, we propose a novel multi-view convolutional neural network to assess image aesthetic by analyzing image color composition and space formation (IAACS). Specifically, from different views of an image, including its key color components with their contributions, the image space formation and the image itself, our network extracts their corresponding features through our proposed feature extraction module (FET) and the ImageNet weight-based classification model. By fusing the extracted features, our network produces an accurate prediction score distribution of image aesthetic. Experiment results have shown that we have achieved a superior performance.

Label distribution learning is a gradually emerging field of learning that can address label ambiguity. In a sense, label distribution learning includes common single-label learning and multi-label learning with broader applicability. For each instance, label distribution learning uses the label distribution to reflect the individual label importance. Previous studies have shown that label distribution learning is an effective method to address label ambiguity. In order to strengthen the learning performance and make the prediction of the distribution more accurate, this paper proposes a label distribution learning algorithm called SRPLDL. We propose a metric called the Square Root Pearson, and then we build our learning model on the basis of this metric. Comparative experiments on 11 real label distribution datasets prove that the SRPLDL algorithm can solve the label distribution learning problem and outperforms other algorithms.

The elastic network models (ENMs)are known as representative coarse-grained models to capture essential dynamics of proteins. Due to simple designs of the force constants as a decay with spatial distances of residue pairs in many previous studies, there is still much room for the improvement of ENMs. In this article, we directly computed the force constants with the inverse covariance estimation using a ridge-type operater for the precision matrix estimation (ROPE)on a large-scale set of NMR ensembles. Distance-dependent statistical analyses on the force constants were further comprehensively performed in terms of several paired types of sequence and structural information, including secondary structure, relative solvent accessibility, sequence distance and terminal. Various distinguished distributions of the mean force constants highlight the structural and sequential characteristics coupled with the inter-residue cooperativity beyond the spatial distances. We finally integrated these structural and sequential characteristics to build novel ENM variations using the particle swarm optimization for the parameter estimation. The considerable improvements on the correlation coefficient of the mean-square fluctuation and the mode overlap were achieved by the proposed variations when compared with traditional ENMs. This study opens a novel way to develop more accurate elastic network models for protein dynamics.

In vehicle retrieval, the vehicle patch should first be localized to remove the irrelevant background information. Moreover, the negative samples are much more prevalent than the positive samples, and the information from the negative samples is not fully exploited in the triple loss. What we need is a way to incorporate global knowledge and structure information to address these two issues. Therefore, we introduce a local-global context network for landmark alignment to update the predicted results by using the semantic information and the local compatibility and propose a structure-aware quadruple loss to use multiple and diverse negative samples in retrieval. Experiments on the VehicleID and the ENJOYOR vehicle retrieval datasets demonstrate that our approach obtains accuracy comparable to state-of-the-art approaches in vehicle retrieval.

Multi-frame human pose estimation in complicated situations is challenging. Although state-of-the-art human joints detectors have demonstrated remarkable results for static images, their performances come short when we apply these models to video sequences. Prevalent shortcomings include the failure to handle motion blur, video defocus, or pose occlusions, arising from the inability in capturing the temporal dependency among video frames. On the other hand, directly employing conventional recurrent neural networks incurs empirical difficulties in modeling spatial contexts, especially for dealing with pose occlusions. In this paper, we propose a novel multi-frame human pose estimation framework, leveraging abundant temporal cues between video frames to facilitate keypoint detection. Three modular components are designed in our framework. A Pose Temporal Merger encodes keypoint spatiotemporal context to generate effective searching scopes while a Pose Residual Fusion module computes weighted pose residuals in dual directions. These are then processed via our Pose Correction Network for efficient refining of pose estimations. Our method ranks No.1 in the Multi-frame Person Pose Estimation Challenge on the large-scale benchmark datasets PoseTrack2017 and PoseTrack2018. We have released our code, hoping to inspire future research.

DIBR (depth-image-based rendering) synthesized videos has become increasingly popular recently in supporting 3D-TV, Free-viewpoint video (FVV) and interactive graphic applications. Due to the synthesis process, it contains composite distortions, i.e., DCT-like distortions plus geometric distortions. It has led to new challenges for subjective and objective video quality assessment (VQA) community, where such composite distortions have not been taken into consideration. In this work, we construct a new database composed of DIBR synthesized videos, the reference view video is compressed by H.264 and HEVC encoders, which is subsequently synthesized to the virtual view with different DIBR algorithms. We also propose a novel no-reference (NR) synthesized video quality metric with multi-modal feature pooling and evaluate it against representative full reference (FR) and NR objective VQA models on the proposed database. The database and the proposed metric will be made available to the public.

Abstract Color structure of a home scene image closely relates to the material properties of its local regions. Existing color migration methods typically fail to fully infer the correlation between the coloring of local home scene regions, leading to a local blur problem. In this paper, we propose a color migration framework for home scene images. It picks the coloring from a template image and transforms such coloring to a home scene image through a simple interaction. Our framework comprises three main parts. First, we carry out an interactive segmentation to divide an image into local regions and extract their corresponding colors. Second, we generate a matching color table by sampling the template image according to the color structure of the original home scene image. Finally, we transform colors from the matching color table to the target home scene image with the boundary transition maintained. Experimental results show that our method can effectively transform the coloring of a scene matching with the color composition of a given natural or interior scenery.

Abstract To effectively control particle‐based cloud evolution without imposing strict position constraints, we propose a novel method integrating a control force field and a phase transition control into the position‐based fluids (PBF) framework. To produce realistic cloud simulation, we incorporate both fluid dynamics and thermodynamics to govern cloud particle movement. The fluid dynamics is simulated through our novel driving and damping force terms. As these terms are only formulated based on cloud particle density and position, they simplify the inputs and make our method free from artificial positional constraints. The thermodynamics is implemented by our phase transition control, which can effectively simulate cloud evolution between discrepant initial and target shapes, producing plausible results. Uniquely, our method can also support target shape change during cloud simulation. Experiment results have demonstrated our method surpasses existing methods.

We propose an automatic indoor furniture layout scheme based on functional area division and furniture filling. According to the function, we suppose each kind of furniture may be laid out in one or several functional areas, for example, a sofa may be located in the meeting area of a living room and a bed may be located in the sleeping area of a bedroom, etc.. Our automatic layout method divides an empty room region into several functional areas by using conditional generative adversarial networks (CGAN). We expound the learning process of the algorithm in the process of functional areas division, including the objective function construct and training process. Moreover, in order to fill furniture into a specific functional area, a learning-based furniture filling algorithm is proposed by training a fully connected network model for different kinds of functional area. Experiments show our automatic furniture layout method has its advantages in performance and effect compared with the existing methods.

Blind image quality metrics have achieved significant improvement on traditional 2D image dataset, yet still being insufficient for evaluating synthesized images generated from depth-image-based rendering. The geometric distortions in synthesized image are non-uniform, which is challenging for feature representation and pooling. To address this, we propose an end-to-end deep blind synthesized image quality metric SIQA-CFP. We particularly design a contextual multilevel feature pooling module to encode low- and high-level features, which are extracted by a deep pre-trained ResNet. Experimental results on IRCCyN/IVC DIBR dataset show that our method outperforms state-of-the-art synthesized image quality metrics. Our method also achieves competitive performance on traditional 2D image datasets like LIVE Challenge and TID2013.

Abstract Harmonious color combinations can stimulate positive user emotional responses. However, a widely open research question is: how can we establish a robust and accurate color harmony measure for the public and professional designers to identify the harmony level of a color theme or color set. Building upon the key discovery that color pairs play an important role in harmony estimation, in this paper we present a novel color‐pair based estimation model to accurately measure the color harmony. It first takes a two‐layer maximum likelihood estimation (MLE) based method to compute an initial prediction of color harmony by statistically modeling the pair‐wise color preferences from existing datasets. Then, the initial scores are refined through a back‐propagation neural network (BPNN) with a variety of color features extracted in different color spaces, so that an accurate harmony estimation can be obtained at the end. Our extensive experiments, including performance comparisons of harmony estimation applications, show the advantages of our method in comparison with the state of the art methods.

Depth-image-based rendering (DIBR) is widely used in 3DTV, free-viewpoint video, and interactive 3D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2D image quality metrics and state-of-the-art DIBR-related metrics.

Recent research on dense captioning based on the recurrent neural network and the convolutional neural network has made a great progress. However, mapping from an image feature space to a description space is a nonlinear and multimodel task, which makes it difficult for the current methods to get accurate results. In this paper, we put forward a novel approach for dense captioning based on hourglass-structured residual learning. Discriminant feature maps are obtained by incorporating dense connected networks and residual learning in our model. Finally, the performance of the approach on the Visual Genome V1.0 dataset and the region labelled MS-COCO (Microsoft Common Objects in Context) dataset are demonstrated. The experimental results have shown that our approach outperforms most current methods.

Abstract Dynamic mesh sequence (DMS) is a simple and accurate representation for precisely recording a 3D animation sequence. Despite its simplicity, this representation is typically large in data size, making storage and transmission expensive. This paper presents a novel framework that allows effective DMS compression and progressive streaming by eliminating spatial and temporal redundancy. To explore temporal redundancy, we propose a temporal frame‐clustering algorithm to organize DMS frames by their motion trajectory changes, eliminating intracluster redundancy by principal component analysis dimensionality reduction. To eliminate spatial redundancy, we propose an algorithm to transform the coordinates of mesh vertex trajectory into a decorrelated trajectory space, generating a new spatially nonredundant trajectory representation. We finally apply a spectral graph wavelet transform with color set partitioning embedded block encoding to turn the resultant DMS into a multiresolution representation to support progressive streaming. Experiment results show that our method outperforms several existing methods in terms of storage requirement and reconstruction quality.

With the increasing demand in using 3D mesh data over networks, supporting effective compression and efficient transmission of meshes has caught lots of attention in recent years. This article introduces a novel compression method for 3D mesh animation sequences, supporting user-defined and progressive transmissions over networks. Our motion-aware approach starts with clustering animation frames based on their motion similarities, dividing a mesh animation sequence into fragments of varying lengths. This is done by a novel temporal clustering algorithm, which measures motion similarity based on the curvature and torsion of a space curve formed by corresponding vertices along a series of animation frames. We further segment each cluster based on mesh vertex coherence, representing topological proximity within an object under certain motion. To produce a compact representation, we perform intra-cluster compression based on Graph Fourier Transform (GFT) and Set Partitioning In Hierarchical Trees (SPIHT) coding. Optimized compression results can be achieved by applying GFT due to the proximity in vertex position and motion. We adapt SPIHT to support progressive transmission and design a mechanism to transmit mesh animation sequences with user-defined quality. Experimental results show that our method can obtain a high compression ratio while maintaining a low reconstruction error.

Abstract Deformation method based on moving least squares (MLS) allows the user to manipulate 2D characters using either sets of points or line segments in real time. However, the traditional MLS deformation spreads the deformation of the controls with respect to the spatial distance, but oblivious to the shape topology, which would possibly lead to distortion. In this paper, we present a topology‐aware MLS deformation approach for 2D characters. First, a Laplace equation is solved to obtain a set of weights, which are called harmonic weights. Then, the MLS deformation is performed by using the harmonic weights as the deformation influence of the user‐specified controls. Finally, the possible distortion in the traditional MLS deformation can be effectively avoided, as the harmonic weights spread the deformation of the controls in a localized and topology‐aware way. In addition, a simple but effective area‐preserving variant of MLS deformation is proposed, which is suitable for the editing of incompressible objects. Copyright © 2015 John Wiley & Sons, Ltd.