MOTIVATION: Structure-based drug design (SBDD) aims to generate ligand molecules that tightly bind to specific protein targets, a critical step in drug discovery. Diffusion models have shown promise for this task, yet existing methods struggle to effectively incorporate protein-ligand interaction priors during generation. Most approaches rely on protein-specific structural priors that remain fixed throughout generation, limiting molecular diversity and failing to capture the dynamic interplay between protein pockets and ligand atoms, which is essential for achieving high binding affinity. RESULTS: We propose DPDiff, a disentangled prior-conditioned diffusion model for protein-specific 3D molecular generation. DPDiff introduces two complementary interaction prior networks that capture geometry-based spatial interactions and sequence-based interactions robust to structural noise. During generation, the model dynamically extracts interaction priors using intermediate diffusion predictions and adaptively fuses them via a time-dependent adapter. A disentangled denoising network balances prior guidance with generative flexibility. Experiments on the CrossDocked2020 dataset demonstrate that DPDiff generates molecules with more realistic 3D structures and state-of-the-art binding affinities, achieving an average Vina Dock score of -8.58 and a high affinity ratio of 69.4%, outperforming existing methods while maintaining favorable drug-likeness and synthetic accessibility. AVAILABILITY AND IMPLEMENTATION: The source code of DPDiff is available at https://github.com/ZerinHwang03/DPDiff.

Although diffusion models have demonstrated promising potential in 3D human pose estimation, existing approaches commonly adopt fixed noise variance scheduling strategies. This design overlooks the skeletal consistency constraints that should be maintained between 2D observations and 3D predictions during iterative denoising. Consequently, the generated 3D poses often exhibit geometric inconsistencies with the input 2D poses, thereby degrading the overall reconstruction quality. To address this limitation, we propose SkyPose: Skeleton Consistency Guided Diffusion Model for 3D Human Pose Estimation. The proposed method enforces skeletal consistency by reprojecting the previously generated 3D pose onto the 2D plane, decomposing it into skeletal direction and length, and comparing it with the input 2D skeletal structure to construct a dynamic skeletal consistency metric. This metric adaptively adjusts the noise variance σt during DDIM iterations, explicitly preserving skeletal structural coherence throughout the denoising process and enhancing the geometric fidelity of the generated 3D poses. Furthermore, we redesign the denoising network architecture to integrate skeletal structure features with joint coordinate features at the local joint, body-part, and full-body levels, thereby enabling joint–skeleton collaborative denoising under multi-granularity geometric constraints. Extensive experiments on the Human3.6M and MPI-INF-3DHP datasets demonstrate that our approach achieves superior accuracy compared with existing methods. The code is open-sourced at here.

3D object detection plays an important role in intelligent systems perceiving the world. Although many studies have been conducted to address this task, the detection accuracy is still limited by the network's learning capability. Therefore, we propose LVMF3D, a Large Vision Model (LVM) boosted multimodal fusion indoor 3D object detection framework, consisting of two branches. The pre-trained LVM is used as the RGB branch to better extract the image texture feature. The point branch is used to encode the spatial geometric feature. Furthermore, Point Fusion Module (PFM) and Multi-Scale Attention Fusion Module (MS-AFM) are specially designed in the 2D and 3D spaces, respectively, to realize more comprehensive and effective information fusion between the two branches. We conduct experiments on the indoor 3D object detection dataset SUN RGB-D and achieve state-of-the-art results compared to other 3D object detection methods.

Recently, two-stage 3D human pose estimation using monocular cameras has gained significant attention. However, the inherent uncertainty in the upscaling process from 2D to 3D often compromises the accuracy of deterministic methods. To address this, we propose a novel diffusion-based refinement framework (DRPose) which models the uncertainty during the upscaling process by introducing stochastic noise to the initially predicted 3D poses. This approach facilitates the generation of more realistic predictions through iterative refinement with multiple noise samples, ultimately producing multi-hypothesis predictions that better align with ground truth. Our framework incorporates two key components: a Graph Convolution Transformer module (SGCT), which integrates scaling and displacement adjustments based on conditional information with a joint temporal-spatial feature separation mechanism, and a Pose Refinement Module (PRM), which balances the initial and refined poses. This design allows DRPose to effectively refine pose estimation for both individual frames and sequential data. Furthermore, our framework establishes new benchmarks for performance in bothframe2frameandseq2framescenarios. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the Human3.6M and MPI-INF-3DHP datasets. Notably, when applied to the current state-of-the-art single-frame 3D pose extractor, our multi-hypothesis optimization achieves an 18.8% reduction in Mean Per Joint Position Error (MPJPE) and a 16.9% reduction in Procrustes MPJPE (P-MPJPE). Code is available at https://github.com/KHB1698/DRPose.

Existing 3D occupancy networks demand significant hardware resources, hindering the deployment of resource-limited devices. Binarized Neural Networks (BNNs) offer a potential solution by substantially reducing computational and memory requirements. However, their performance decrease notably compared to full-precision networks. In addition, it is challenging to enhance the performance of the binarized model by increasing the number of binarized convolutional layers, which limits its practicability for 3D occupancy prediction. In this paper, we reconsider the components in binarized convolutional layers, and structures, for 3D occupancy prediction task. Two original insights into binarized convolution are presented, substantiated with theoretical proofs: (a) 1×1 binarized convolution introduces minimal binarization errors as the network deepens, and (b) binarized convolution is inferior to full-precision convolution in capturing cross-channel feature importance. Building on the above insights, we propose a novel binarized deep convolution (BDC) unit that significantly enhances performance, even when the number of binarized convolutional layers increases to meet the requirements of 3D occupancy networks. Specifically, in the BDC unit, additional binarized convolutional kernels are constrained to 1×1 to minimize the effects of binarization errors. Further, we propose a per-channel refinement branch to reweight the output via first-order approximation. Then, we partition the 3D occupancy networks into four distinct convolutional modules, employing BDC units to explore the effects of binarizing each of these modules. The proposed BDC unit minimizes binarization errors and improves perceptual capability, meeting the stringent requirements for accuracy and computational efficiency in 3D occupancy prediction. Extensive quantitative and qualitative experiments demonstrate that the proposed BDC unit achieves state-of-the-art performance in 3D occupancy prediction and 3D object detection tasks, while significantly reducing parameters and computational costs. This highlights the potential of the BDC unit as an efficient fundamental component in binarized 3D occupancy networks. Code for our paper will be released on “https://github.com/zzk785089755/BDC”.

Reconstructing 3D poses from 2D poses lacking depth information is particularly challenging due to the complexity and diversity of human motion. The key is to effectively model the spatial constraints between joints to leverage their inherent dependencies. Thus, we propose a novel model, called Double-chain Graph Convolution Transformer (DC-GCT), to constrain the pose through a double-chain design consisting of local-to-global and global-to-local chains to obtain a complex representation more suitable for the current human pose. Specifically, we combine the advantages of GCN and Transformer and design a Local Constraint Module (LCM) based on GCN and a Global Constraint Module (GCM) based on self-attention mechanism as well as a Feature Interaction Module (FIM). The proposed method fully captures the multi-level dependencies between human body joints to optimize the modeling capability of the model. Moreover, we propose a method to use temporal information into the single-frame model by guiding the video sequence embedding through the joint embedding of the target frame, with negligible increase in computational cost. Experimental results demonstrate that DC-GCT achieves state-of-the-art performance on two challenging datasets (Human3.6 M and MPI-INF-3DHP). Notably, our model achieves state-of-the-art performance on all action categories in the Human3.6 M dataset using detected 2D poses from CPN, and our code is available at:https://github.com/KHB1698/DC-GCT.

Implicit neural representations (INRs) have revolutionized arbitrary-scale super-resolution (ASSR) by modeling images as continuous functions. Most existing INR-based ASSR networks first extract features from the given low-resolution image using an encoder, and then render the super-resolved result via a multi-layer perceptron decoder. Although these approaches have shown promising results, their performance is constrained by the limited representation ability of discrete latent codes in the encoded features. In this paper, we propose a novel ASSR method named GaussianSR that overcomes this limitation through 2D Gaussian Splatting (2DGS). Unlike traditional methods that treat pixels as discrete points, GaussianSR represents each pixel as a continuous Gaussian field. The encoded features are simultaneously refined and upsampled by rendering the mutually stacked Gaussian fields. As a result, long-range dependencies are established to enhance representation ability. In addition, a classifier is developed to dynamically assign Gaussian kernels to all pixels to further improve flexibility. All components of GaussianSR (i.e. encoder, classifier, Gaussian kernels, and decoder) are jointly learned end-to-end. Experiments demonstrate that GaussianSR achieves superior ASSR performance with fewer parameters than existing methods while enjoying interpretable and content-aware feature aggregations.

Gaussian Splatting has emerged as a prominent 3D representation in novel view synthesis, but it still suffers from appearance variations, which are caused by various factors, such as modern camera ISPs, different time of day, weather conditions, and local light changes. These variations can lead to floaters and color distortions in the rendered images/videos. Recent appearance modeling approaches in Gaussian Splatting are either tightly coupled with the rendering process, hindering real-time rendering, or they only account for mild global variations, performing poorly in scenes with local light changes. In this paper, we propose DAVIGS, a method that decouples appearance variations in a plug-and-play and efficient manner. By transforming the rendering results at the image level instead of the Gaussian level, our approach can model appearance variations with minimal optimization time and memory overhead. Furthermore, our method gathers appearance-related information in 3D space to transform the rendered images, thus building 3D consistency across views implicitly. We validate our method on several appearance-variant scenes, and demonstrate that it achieves state-of-the-art rendering quality with minimal training time and memory usage, without compromising rendering speeds. Additionally, it provides performance improvements for different Gaussian Splatting baselines in a plug-and-play manner.

Modeling 3D open-vocabulary language fields is challenging yet highly anticipated. Despite great progress, existing approaches heavily rely on a large number of training views to construct language-embedded 3D scenes, which is unfortunately impractical in real-world scenarios. This paper introduces SOVGaussian, the first method for few-shot novel view open-vocabulary language querying. We introduce a depth-constrained neural language field to mitigate the geometry degradation caused by overfitting training views. Rather than straightforwardly using dense depth maps for loosely accurate supervision, Language-Aware Depth Distillation (LAD) based on open-vocabulary object masks is proposed, ensuring intra-object geometric accuracy within the language field. To further refine the language-geometry consistency of the language field, we propose a novel Language-Guided Outlier Pruning (LOP) strategy, which identifies floating 3D Gaussian primitives overfitting training views based on their language-grouped densities. Our comprehensive experiments demonstrate that SOVGaussian is able to reconstruct a superior scene representation from few-shot images, outperforming existing state-of-the-art methods and achieving significantly better performance on novel view language querying and synthesis.

Text-based person retrieval (TPR) has gained significant attention as a fine-grained and challenging task that closely aligns with practical applications. Tailoring CLIP to person domain is now a emerging research topic due to the abundant knowledge of vision-language pretraining, but challenges still remain during fine-tuning: (i) Previous full-model fine-tuning in TPR is computationally expensive and prone to overfitting.(ii) Existing parameter-efficient transfer learning (PETL) for TPR lacks of fine-grained feature extraction. To address these issues, we propose Domain-Aware Mixture-of-Adapters (DM-Adapter), which unifies Mixture-of-Experts (MOE) and PETL to enhance fine-grained feature representations while maintaining efficiency. Specifically, Sparse Mixture-of-Adapters is designed in parallel to MLP layers in both vision and language branches, where different experts specialize in distinct aspects of person knowledge to handle features more finely. To promote the router to exploit domain information effectively and alleviate the routing imbalance, Domain-Aware Router is then developed by building a novel gating function and injecting learnable domain-aware prompts. Extensive experiments show that our DM-Adapter achieves state-of-the-art performance, outperforming previous methods by a significant margin.

Current state-of-the-art (SOTA) methods in 3D Human Pose Estimation (HPE) are primarily based on Transformers. However, existing Transformer-based 3D HPE backbones often encounter a trade-off between accuracy and computational efficiency. To resolve the above dilemma, in this work, we leverage recent advances in state space models and utilize Mamba for high-quality and efficient long-range modeling. Nonetheless, Mamba still faces challenges in precisely exploiting local dependencies between joints. To address these issues, we propose a new attention-free hybrid spatiotemporal architecture named Hybrid Mamba-GCN (Pose Magic). This architecture introduces local enhancement with GCN by capturing relationships between neighboring joints, thus producing new representations to complement Mamba's outputs. By adaptively fusing representations from Mamba and GCN, Pose Magic demonstrates superior capability in learning the underlying 3D structure. To meet the requirements of real-time inference, we also provide a fully causal version. Extensive experiments show that Pose Magic achieves new SOTA results (0.9 mm drop) while saving 74.1% FLOPs. In addition, Pose Magic exhibits optimal motion consistency and the ability to generalize to unseen sequence lengths.

Perspective projection has been extensively utilized in monocular 3D object detection methods. It introduces geometric priors from 2D bounding boxes and 3D object dimensions to reduce the uncertainty of depth estimation. However, due to errors originating from the object’s visual surface, the bounding box height often fails to represent the actual central height, which undermines the effectiveness of geometric depth. Direct prediction for the projected height unavoidably results in a loss of 2D priors, while multi-depth prediction with complex branches does not fully leverage geometric depth. This paper presents a Transformer-based monocular 3D object detection method called MonoDGP, which adopts perspective-invariant geometry errors to modify the projection formula. We also try to systematically discuss and explain the mechanisms and efficacy behind geometry errors, which serve as a simple but effective alternative to multi-depth prediction. Additionally, MonoDGP decouples the depth-guided decoder and constructs a 2D decoder only dependent on visual features, providing 2D priors and initializing object queries without the disturbance of 3D detection. To further optimize and finetune input tokens of the transformer decoder, we also introduce a Region Segmentation Head (RSH) that generates enhanced features and segment embeddings. Our monocular method demonstrates state-of-the-art performance on the KITTI benchmark without extra data. Code is available at https://github.com/PuFanqi23/MonoDGP.

Face restoration is a critical task in computer vision, aiming to restore high-quality facial images from degraded inputs. In existing diffusion models, identity information is not well preserved when confronted with severely degradation. To address this challenge, we propose a Local Patch-Based Identity-Preserving Diffusion (LPIP-Diff) framework. Our local patch-based strategy leverages the interrelationships between neighboring patches to model highly structured facial context, which facilitates the restoration of fine-grained details and the preservation of identity-related features. We also introduce a fusion degradation estimation method that makes each overlapping area restored multiple times by adjacent patches, effectively restoring local details. The experimental results of LPIP-Diff on three publicly available datasets, including one severely degraded dataset, consistently demonstrate its superiority over the state-of-the-art methods in terms of both quantitative and qualitative evaluations, strikes a good balance between realism and fidelity, and enhances robustness against degradation.

Holistic 3D scene understanding from a single image is challenging due to the information loss in 2D-to-3D reconstruction. Existing approaches either explore object properties independently or overlook their levels of correlation, leading to inaccurate estimations in complex scenes. To this end, we propose Correlative3D, a novel inter-object correlation-aware method for 3D scene understanding. Our method integrates a Scene Graph Attention Network to implicitly enhance features through a correlation-driven weighting strategy, selectively prioritizing relationships among objects. In addition, an auxiliary task pertaining to the relative arrangement of objects is formulated to impose explicit constraints. Furthermore, we introduce a novel 3DCIoU loss that sensitively responds to geometric variations in 3D bounding boxes. Extensive experiments demonstrate that our method produces more coherent scene layouts compared to existing methods.

Recent great advances in video generation models have demonstrated their potential to produce high-quality videos, bringing challenges to effective evaluation. Unlike human evaluation, existing automated evaluation metrics lack high-level semantic understanding and reasoning capabilities for video, thus making them infeasible and unexplainable. To fill this gap, we curate **GRADEO-Instruct**, a multi-dimensional T2V evaluation instruction tuning dataset, including 3.3k videos from over 10 existing video generation models and multi-step reasoning assessments converted by 16k human annotations. We then introduce **GRADEO**, one of the first specifically designed video evaluation models, which **grades** AI-generated **videos** for explainable scores and assessments through multi-step reasoning. Experiments show that our method aligns better with human evaluations than existing methods. Furthermore, our benchmarking reveals that current video generation models struggle to produce content that aligns with human reasoning and complex real-world scenarios. The models, datasets, and codes will be released soon.

The integration of 3D Gaussians has introduced a novel scene representation in Simultaneous Localization and Mapping (SLAM), characterized by explicit representation and differentiable rendering capabilities that enhance scene reconstruction and understanding. However, most current SLAM systems only exploit the basic representational capacity of 3D Gaussians, neglecting their potential to offer richer information and facilitate higher-dimensional scene comprehension. Furthermore, these systems often struggle with reconstruction when encountering rapid camera movements or depth missing. Drawing inspiration from 3D language field, which explores the intrinsic relationships among scene objects, we propose SAPSLAM, a dense SLAM system that combines high-fidelity reconstruction and advanced semantic understanding. Our approach leverages pre-trained visual models to extract semantic features, which are then fused, dimensionally reduced, and encoded into the 3D Gaussian model for optimization and rendering. The integration of these features improves the systems semantic comprehension and scene representation, ultimately enabling the creation of high-precision 3D semantic maps. Additionally, we introduce a semantic-guided Gaussian densification and pruning strategy, which uses semantic consistency to prioritize attention on poorly reconstructed areas, greatly improving performance in complex scenarios. SAP-SLAM achieves competitive results on both real-world and synthetic datasets, demonstrating superior capabilities in semantic understanding and reconstruction.

Robotic grasping, serving as the cornerstone of robot manipulation, is fundamental for embodied intelligence. Manipulation in challenging scenarios demands grasp detection algorithms with higher efficiency and generalizability. However, for general 6-Dof grasp detection, most data-driven methods directly extract scene-level features to generate grasp prediction, relying on a relatively heavy scene-level feature encoder and a significant amount of data with dense grasp labels for model training. In this letter, we propose a novel data-efficient 6-Dof grasp detection framework in cluttered scenes, named Region-Centric Grasp Detection (RCGD), consisting of an Iterative Search Module (ISM) and a Region Grasp Model (RGM). Concretely, ISM aims to retrieve potential region centers and aggregate multiple regions in a coarse-to-fine way. Then, RGM extracts aligned grasp-related embeddings and predicts grasps within these local regions. Benefiting from the region-centric paradigm and the training-free location strategy, RCGD significantly outperforms previous methods and shows minimal performance loss with even a very small portion of training data or labels. Furthermore, real-world robotic experiments in two distinct settings highlight the effectiveness of our method with a 95% success rate.

Visual object navigation, requiring agents to locate target objects in novel environments through egocentric visual observation, remains a critical challenge in Embodied AI. We propose FEG-VON, a training-free framework that constructs and maintains a Frontier Embedding Graph for efficient Visual Object Navigation. The graph initializes frontier embeddings using Vision Language Models (VLMs), where visual observations are encoded into spatially anchored semantic embeddings through cross-modal alignment with target text descriptors. We then update the graph by aggregating spatio-temporal semantic relations across frontiers, enabling online adaptation to new targets via similarity scoring without remapping. The evaluation results in public benchmarks demonstrate the superior performance of FEG-VON in both single- and multi-object navigation tasks compared with state-of-the-art methods. Crucially, FEG-VON eliminates dependency on task-specific training for exploration and advances the feasibility of zero-shot navigation in open-world environments.

The objective of video frame interpolation (VFI) methods is to enhance video fluency and visual quality by generating intermediate frames between consecutive original frames based on the source video. Recently, diffusion-based VFI methods have made promising progresses, with generated results performing well in perceptual quality. However, these methods have not fully explored how to effectively leverage external motion priors to enhance the model's ability to estimate motion information between adjacent frames, which is crucial for VFI models to avoid generating blurry results due to the motion ambiguity. In this paper, we propose an Enhanced Motion-Aware latent Diffusion model ( EMADiff ) for video frame interpolation. Specifically, we integrate motion priors into the decoder of vector-quantized enhanced motion-aware GAN to guide the information propagation during RGB interpolated frame reconstruction. Furthermore, we propose enhanced motion-aware noising and de-noising procedures. By reducing the discrepancy in attention to motion priors between the forward and reverse processes, our EMADiff effectively utilizes motion priors, alleviates motion ambiguity, and generates realistic content. Comprehensive experiments on benchmark datasets show EMADiff achieves state-of-the-art performance, surpassing existing approaches and producing visually plausible and content-clear results.

Long-tailed out-of-distribution learning aims to reduce performance bias in long-tailed in-distribution (ID) data while rejecting out-of-distribution (OOD) samples, which are often mistaken for under-represented tail classes. To achieve OOD detection, existing methods incorporate an outlier exposure (OE) term into the long-tailed recognition (LTR) loss. However, as we prove in this paper, the OE term induces a gradient conflict with the ID objectives, especially for tail classes, thereby contradicting the core motivation of LTR. To avoid the ID-OOD dilemma, we propose Dynamic Ambiguity-aware Recalibration for Logits (DARL), an ambiguity-guided long-tailed OOD learning approach, grounded on two theoretical insights. First, we show that the mixed ID data can mitigate the conflict in OE training and exhibits higher intrinsic ambiguity than the original ID data, thus able to serve as a surrogate for real OOD data. Second, we introduce an ambiguity-aware logit adjustment that can dynamically calibrate the class margins using energy-based ambiguity metrics, effectively reducing early-stage bias while avoiding late-stage overfitting. Extensive experiments show that DARL achieves the overall state-of-the-art performance of long-tailed OOD learning. Moreover, compared with the OE methods, DARL trains solely on the ID data, which can reduce the data requirements by 80%. The code is available in https://github.com/XuanZhang-A/DARL.

Low-light (LOL) conditions constantly restrict the performance of Internet of Things (IoT) image sensors, thereby impacting image quality and the precision of visual data analysis. The emerging edge intelligence is crucial for LOL image enhancement in improving image quality and data support reliability for IoT systems, which in turn fosters the intelligence and automation progress of the IoT. The enhancement of LOL images necessitates the restoration of both contextual information and spatial details, maintaining the semantic content of the original image and the point-to-point correspondence between inputs and outputs. However, existing methods predominantly concentrate on one aspect, either contextual information or spatial details, making it difficult to simultaneously balance both. To overcome this challenge, we introduce a novel two-branch network, the context-space balance network (CSBNet), and tailored for LOL image enhancement. It comprises a contextual information recovery network (CIRNet), which adeptly extracts contextual information from multiscale LOL images, and a spatial information recovery network (SIRNet), which is designed to preserve spatial details at the original resolution. We also implement a context-space feature fusion (CSFF) module to seamlessly integrate contextual information with spatial details. Qualitative and quantitative experimental results demonstrate that our CSBNet can better handle various kinds of degradations in lowlight images compared with state-of-the-art solutions on the benchmark LOL dataset. The source code of CSBNet is available athttps://github.com/Loong161/CSBNet.

The Diffusion Model (DM) has emerged as the SOTA approach for image synthesis. However, the existing DM cannot perform well on some image-to-image translation (I2I) tasks. Different from image synthesis, some I2I tasks, such as super-resolution, require generating results in accordance with GT images. Traditional DMs for image synthesis require extensive iterations and large denoising models to estimate entire images, which gives their strong generative ability but also leads to artifacts and inefficiency for I2I. To tackle this challenge, we propose a simple, efficient, and powerful DM framework for I2I, called DiffI2I. Specifically, DiffI2I comprises three key components: a compact I2I prior extraction network (CPEN), a dynamic I2I transformer (DI2Iformer), and a denoising network. We train DiffI2I in two stages: pretraining and DM training. For pretraining, GT and input images are fed into CPEN to capture a compact I2I prior representation (IPR) guiding DI2Iformer. In the second stage, the DM is trained to only use the input images to estimate the same IRP as CPEN. Compared to traditional DMs, the compact IPR enables DiffI2I to obtain more accurate outcomes and employ a lighter denoising network and fewer iterations. Through extensive experiments on various I2I tasks, we demonstrate that DiffI2I achieves SOTA performance while significantly reducing computational burdens.

The discovery of new pharmaceutical drugs remains an essential yet costly, time-consuming task in biomedical research. Recent developments in machine learning and bioinformatics have significantly accelerated this process. State-of-the-art approaches employ neural networks to optimize latent space vectors representing target molecules. In such processes, the optimized parameters correspond to specific chemical properties whose exact mathematical formulations are unknown; where only limited discrete measurements are available and difficult to approximate. We propose a novel loss function that effectively captures chemical property variations by enforcing molecular ordering based on their property values. Our experiments demonstrate that this "ordering loss" outperforms conventional mean squared error (MSE) as an optimization objective for property functions. The ordering loss enables surrogate functions to mirror the variation patterns of black-box property functions, establishing a new state-of-the-art framework for molecule discovery optimization. It exhibits superior performance under data scarcity constraints and shows promising potential for broader black-box function optimization applications.

Group decision making (GDM) problems with interval multiplicative preference relations (IMPRs) contain uncertain preference information from the decision makers (DMs). To help the DMs make decision quickly and accurately, this article proposes a novel framework to GDM problems with IMPRs by combining the ELECTRE III method with stochastic simulation. First, the stochastic multiplicative preference relations are derived by stochastic simulation, then the individual stochastic priority vector and the group stochastic priority matrix are defined. Second, the group stochastic net credibility degree is presented to evaluate all alternatives. Third, several optimal group measurements are proposed, including the group stochastic rank acceptability degree, the optimal preference rank, the group stochastic central weight vector, and the group stochastic optimal confidence factor. Moreover, a novel framework for GDM problems with IMPRs is put forward by using the stochastic simulation with the ELECTRE III method. The proposed approach retains the initial opinions of DMs and reduces the influence of DMs’ subjectivity, besides considers the ranking possibility of all alternatives from an expected perspective. Finally, a medicine decision example of Wilson disease is analyzed, comparison analysis and sensitivity analysis are conducted to verify the validity and feasibility of the proposed approach.

Text-based Person Retrieval (TPR) as a multi-modal task, which aims to retrieve the target person from a pool of candidate images given a text description, has recently garnered considerable attention due to the progress of contrastive visual-language pre-trained model. Prior works leverage pre-trained CLIP to extract person visual and textual features and fully fine-tune the entire network, which have shown notable performance improvements compared to uni-modal pre-training models. However, full-tuning a large model is prone to overfitting and hinders the generalization ability. In this paper, we propose a novelUnifiedParameter-Efficient Transfer Learning (PETL) method for Text-basedPersonRetrieval (UP-Person) to thoroughly transfer the multi-modal knowledge from CLIP. Specifically, UP-Person simultaneously integrates three lightweight PETL components including Prefix, LoRA and Adapter, where Prefix and LoRA are devised together to mine local information with task-specific information prompts, and Adapter is designed to adjust global feature representations. Additionally, two vanilla submodules are optimized to adapt to the unified architecture of TPR. For one thing, S-Prefix is proposed to boost attention of prefix and enhance the gradient propagation of prefix tokens, which improves the flexibility and performance of the vanilla prefix. For another thing, L-Adapter is designed in parallel with layer normalization to adjust the overall distribution, which can resolve conflicts caused by overlap and interaction among multiple submodules. Extensive experimental results demonstrate that our UP-Person achieves state-of-the-art results across various person retrieval datasets, including CUHK-PEDES, ICFG-PEDES and RSTPReid while merely fine-tuning 4.7% parameters. Code is available at https://github.com/Liu-Yating/UP-Person.

Image restoration is a critical task in low-level computer vision, aiming to restore high-quality images from degraded inputs. Various models, such as convolutional neural networks (CNNs), generative adversarial networks (GANs), transformers, and diffusion models (DMs), have been employed to address this problem with significant impact. However, CNNs have limitations in capturing long-range dependencies. DMs require large prior models and computationally intensive denoising steps. Transformers have powerful modeling capabilities but face challenges due to quadratic complexity with input image size. To tackle these challenges, we propose VmambaIR, one of the first works to introduce State Space Models (SSMs) with linear complexity into comprehensive image restoration tasks. Specifically, we utilize a Unet architecture to stack our proposed Omni Selective Scan (OSS) blocks, consisting of an OSS module and an Efficient Feed-Forward Network (EFFN). Our proposed omni selective scan mechanism overcomes the unidirectional modeling limitation of SSMs by efficiently modeling image information flows in all six directions to better exploit surrounding restoration information. Furthermore, we conducted a comprehensive evaluation of our VmambaIR across multiple image restoration tasks, including image deraining, single image super-resolution, and real-world image super-resolution. Extensive experimental results demonstrate that our proposed VmambaIR achieves state-of-the-art (SOTA) performance with much fewer computational resources and parameters. Our research highlights the potential of state space models as promising alternatives to the transformer and CNN architectures in serving as foundational frameworks for next-generation low-level visual tasks.

Due to rich textures and ease of acquisition, finger-based biometric features have gained significant attention for personal authentication in recent years. However, the majority of current finger-based authentication techniques predominantly rely on features extracted solely from a single modality, such as fingerprint or finger vein. Additionally, most current authentication methods utilize contact-based capture and identification, which poses a risk of bacterial or viral infection. To overcome these limitations, we advocate for the adoption of touchless multimodal finger features, providing a hygienic and robust authentication solution. Specifically, we design a device which can capture touchless finger vein and fingerprint images from four fingers, creating the THU-FVFP dataset. To the best of our knowledge, the THU-FVFP dataset is the first publicly available dataset that includes touchless finger vein and fingerprint data from four fingers. Subsequently, we introduce the Attention-based Cross-domain Fusion Network (ACFNet), which can leverage both intra and inter-features of finger vein and fingerprint data. To achieve this, we develop an Intra Multi-Level Feature Fusion Module (IMLFFM) for merging features from different layers within a single modality and an Inter Multi-Modal Feature Fusion Module (IMMFFM) for achieving optimal fusion of diverse features. We extensively evaluate the model on the THU-FVFP database, proving its outstanding performance with an equal error rate of 0.07%. The THU-FVFP dataset is available athttps://github.com/oneline-wsq/THU-FVFP-Dataset.

Generative diffusion prior captured from the off-the-shelf denoising diffusion generative model has recently attained significant interest. However, several attempts have been made to adopt diffusion models to noisy inverse problems either fail to achieve satisfactory results or require a few thousand iterations to achieve high-quality reconstructions. In this work, we propose a diffusion-based image restoration with error contraction and error correction (DiffECC) method. Two strategies are introduced to contract the restoration error in the posterior sampling process. First, we combine existing CNN-based approaches with diffusion models to ensure data consistency from the beginning. Second, to amplify the error contraction effects of the noise, a restart sampling algorithm is designed. In the error correction strategy, the estimation-correction idea is proposed on both the data term and the prior term. Solving them iteratively within the diffusion sampling framework leads to superior image generation results. Experimental results for image restoration tasks such as super-resolution (SR), Gaussian deblurring, and motion deblurring demonstrate that our approach can reconstruct high-quality images compared with state-of-the-art sampling-based diffusion models.

Structure-based drug design (SBDD) aims to generate 3D ligand molecules that bind to specific protein targets. Existing 3D deep generative models including diffusion models have shown great promise for SBDD. However, it is complex to capture the essential protein-ligand interactions exactly in 3D space for molecular generation. To address this problem, we propose a novel framework, namely Binding-Adaptive Diffusion Models (BindDM). In BindDM, we adaptively extract subcomplex, the essential part of binding sites responsible for protein-ligand interactions. Then the selected protein-ligand subcomplex is processed with SE(3)-equivariant neural networks, and transmitted back to each atom of the complex for augmenting the target-aware 3D molecule diffusion generation with binding interaction information. We iterate this hierarchical complex-subcomplex process with cross-hierarchy interaction node for adequately fusing global binding context between the complex and its corresponding subcomplex. Empirical studies on the CrossDocked2020 dataset show BindDM can generate molecules with more realistic 3D structures and higher binding affinities towards the protein targets, with up to -5.92 Avg. Vina Score, while maintaining proper molecular properties. Our code is available at https://github.com/YangLing0818/BindDM

Event Stream Super-Resolution (ESR) aims to address the challenge of insufficient spatial resolution in event streams, which holds great significance for the application of event cameras in complex scenarios. Previous works for ESR often process positive and negative events in a mixed paradigm. This paradigm limits their ability to effectively model the unique characteristics of each event and mutually refine each other by considering their correlations. In this paper, we propose a bilateral event mining and complementary network (BMCNet) to fully leverage the potential of each event and capture the shared information to complement each other simultaneously. Specifically, we resort to a two-stream network to accomplish comprehensive mining of each type of events individually. To facilitate the exchange of information between two streams, we propose a bilateral information exchange (BIE) module. This module is layer-wisely embedded between two streams, enabling the effective propagation of hierarchical global information while alleviating the impact of invalid information brought by inherent characteristics of events. The experimental results demonstrate that our approach outperforms the previous state-of-the-art methods in ESR, achieving performance improvements of over 11% on both real and synthetic datasets. Moreover, our method significantly enhances the performance of event-based downstream tasks such as object recognition and video reconstruction. Our code is available at https://github.com/Lqm26/BMCNet-ESR.

Existing NeRF-based methods for large scene reconstruction often have limitations in visual quality and rendering speed. While the recent 3D Gaussian Splatting works well on small-scale and object-centric scenes, scaling it up to large scenes poses challenges due to limited video memory, long optimization time, and noticeable appearance variations. To address these challenges, we present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. We propose a progressive partitioning strategy to divide a large scene into multiple cells, where the training cameras and point cloud are properly distributed with an airspace-aware visibility criterion. These cells are merged into a complete scene after parallel optimization. We also introduce decoupled appearance modeling into the optimization process to reduce appearance variations in the rendered images. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets, enabling fast optimization and high-fidelity real-time rendering. Project page: https://vastgaussian.github.io.

Real-time multi-person pose estimation presents signif-icant challenges in balancing speed and precision. While two-stage top-down methods slow down as the number of people in the image increases, existing one-stage meth-ods often fail to simultaneously deliver high accuracy and real-time performance. This paper introduces RTMO, a one-stage pose estimation framework that seamlessly inte-grates coordinate classification by representing keypoints using dual I-D heatmaps within the YOLO architecture, achieving accuracy comparable to top-down methods while maintaining high speed. We propose a dynamic coordi-nate classifier and a tailored loss function for heatmap learning, specifically designed to address the incompati-bilities between coordinate classification and dense pre-diction models. RTMO outperforms state-of-the-art one-stage pose estimators, achieving 1.1% higher AP on COCO while operating about 9 times faster with the same back-bone. Our largest model, RTMO-1, attains 74.8% AP on COCO va12017 and 141 FPS on a single V100 GPU, demonstrating its efficiency and accuracy. The code and models are available at https://github.com/open-mmlab/mmpose/tree/main/projects/rtmo.

In recent years, although Transformer has been introduced into stereo image super-resolution and accomplished great advances, the long-range complementary information in stereo images hasn’t been fully utilized. In view of beneficial non-local prior knowledge in both intra-view and cross-view, we propose an efficient Non-Local Stereo Interaction Transformer (NLSIT) to exploit long-range complementary prior. NLSIT mainly consists of non-local channel interaction block (NLCIB) and non-local spatial interaction block (NLSIB). NLCIB extracts channel correlations across the two views by a channel interaction attention mechanism with linear complexity and NLSIB is devised to capture non-local spatial dependencies with locality sensitive hashing (LSH) enforcing sparsity and relevancy of the attention range. Extensive experiments demonstrate that our NLSIT outperforms most SOTA methods on several popular stereo image datasets with much fewer parameters, showing the effectiveness of the proposed framework.

Ultrasound imaging is crucial for evaluating organ morphology and function, yet depth adjustment can degrade image quality and field-of-view, presenting a depth-dependent dilemma. Traditional interpolation-based zoom-in techniques often sacrifice detail and introduce artifacts. Motivated by the potential of arbitrary-scale super-resolution to naturally address these inherent challenges, we present the Residual Dense Swin Transformer Network (RDSTN), designed to capture the non-local characteristics and long-range dependencies intrinsic to ultrasound images. It comprises a linear embedding module for feature enhancement, an encoder with shifted-window attention for modeling non-locality, and an MLP decoder for continuous detail reconstruction. This strategy streamlines balancing image quality and field-of-view, which offers superior textures over traditional methods. Experimentally, RDSTN outperforms existing approaches while requiring fewer parameters. In conclusion, RDSTN shows promising potential for ultrasound image enhancement by overcoming the limitations of conventional interpolation-based methods and achieving depth-independent imaging.

Recent years, generative adversarial networks (GANs) have gained significant prominence in single image super-resolution (SISR) tasks. This can mainly be attributed to their exceptional ability to generate intricate details. However, the instability and lack of realism in the details generated by GANs have been challenges. Existing methods mainly concentrate on improving the generator and designing complex loss functions, often overlooking the important role of discrimination. To this end, we propose our discrimination enhanced GAN (DEGAN) by improving the discriminator and simplify the discrimination task. We introduce an efficient wide activation UNet to enhance the discriminator, enabling a more comprehensive and nuanced analysis of the input image. Additionally, we introduce a texture aware mask that provides more precise guidance and alleviates the difficulty of discrimination. Our DEGAN is simple yet effective. Quantitative and visual comparisons with state-of-the-art methods on benchmark datasets demonstrate the superiority of our method.

Previous probabilistic models for 3D Human Pose Estimation (3DHPE) aimed to enhance pose accuracy by generating multiple hypotheses. However, most of the hypotheses generated deviate substantially from the true pose. Compared to deterministic models, the excessive uncertainty in probabilistic models leads to weaker performance in single-hypothesis prediction. To address these two challenges, we propose a diffusion-based refinement framework called DRPose, which refines the output of deterministic models by reverse diffusion and achieves more suitable multi-hypothesis prediction for the current pose benchmark by multi-step refinement with multiple noises. To this end, we propose a Scalable Graph Convolution Transformer (SGCT) and a Pose Refinement Module (PRM) for denoising and refining. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets demonstrate that our method achieves state-of-the-art performance on both single and multi-hypothesis 3DHPE. Code is available at https://github.com/KHB1698/DRPose.

Text-based Person Retrieval (TPR) aims to retrieve the target person images given a textual query. The primary challenge lies in bridging the substantial gap between vision and language modalities, especially when dealing with limited large-scale datasets. In this paper, we introduce a CLIP-based Synergistic Knowledge Transfer (CSKT) approach for TPR. Specifically, to explore the CLIP’s knowledge on input side, we first propose a Bidirectional Prompts Transferring (BPT) module constructed by text-to-image and image-to-text bidirectional prompts and coupling projections. Secondly, Dual Adapters Transferring (DAT) is designed to transfer knowledge on output side of Multi-Head Self-Attention (MHA) in vision and language. This synergistic two-way collaborative mechanism promotes the early-stage feature fusion and efficiently exploits the existing knowledge of CLIP. CSKT outperforms the state-of-the-art approaches across three benchmark datasets when the training parameters merely account for 7.4% of the entire model, demonstrating its remarkable efficiency, effectiveness and generalization.

Gaze estimation is an important task in understanding human visual attention. Despite the performance gain brought by recent algorithm development, the task remains challenging due to two-eye appearance asymmetry resulting from head pose variation and nonuniform illumination. In this paper, we propose a novel architecture, Agent-guided Gaze Estimation Network (AGE-Net), to make full and efficient use of two-eye features. By exploring the appearance asymmetry and the consequent feature space asymmetry, we devise a main branch and two agent regression tasks. The main branch extracts related features of the left and right eyes from low-level semantics. Meanwhile, the agent regression tasks extract asymmetric features of the left and right eyes from high-level semantics, so as to guide the main branch to learn more about the eye feature space. Experiments show that our method achieves state-of-the-art gaze estimation task performance on both MPIIGaze and EyeDiap datasets.

Generating 3D ligand molecules that bind to specific protein targets via diffusion models has shown great promise for structure-based drug design. The key idea is to disrupt molecules into noise through a fixed forward process and learn its reverse process to generate molecules from noise in a denoising way. However, existing diffusion models primarily focus on incorporating protein-ligand interaction information solely in the reverse process, and neglect the interactions in the forward process. The inconsistency between forward and reverse processes may impair the binding affinity of generated molecules towards target protein. In this paper, we propose a novel Interaction Prior-guided Diffusion model (IPDiff) for the protein-specific 3D molecular generation by introducing geometric protein-ligand interactions into both diffusion and sampling process. Specifically, we begin by pretraining a protein-ligand interaction prior network (IPNet) by utilizing the binding affinity signals as supervision. Subsequently, we leverage the pretrained prior network to (1) integrate interactions between the target protein and the molecular ligand into the forward process for adapting the molecule diffusion trajectories (prior-shifting), and (2) enhance the binding-aware molecule sampling process (prior-conditioning). Empirical studies on CrossDocked2020 dataset show IPDiff can generate molecules with more realistic 3D structures and state-of-the-art binding affinities towards the protein targets, with up to -6.42 Avg. Vina Score, while maintaining proper molecular properties. https://github.com/YangLing0818/IPDiff

We present a novel framework named PortraitNeRF to generate high-fidelity talking portrait videos for performing faithful identity-preserving reenactment of source videos. This is a challenging task because the generated results should be natural and match the speaker’s head movement, expression, eye blinks and speech audio. To acquire sufficient guidance from source video, the proposed PortraitNeRF exploits not only speech audio but also detailed motion information derived from visual data, including facial expressions, head pose and head position information. By adopting only a single neural radiance field, PortraitNeRF is able to generate complete and coordinated portrait video without bells and whistles. The completeness is ensured by the single-NeRF structure, and the superior head-torso coordination ability comes from using head pose and position information as its conditional input. Moreover, a simple yet effective mouth region emphasis strategy that fits well with the NeRF mechanism helps improving the accuracy of mouth shape. Experimental results and ablation studies demonstrate the superiority and effectiveness of PortraitNeRF.

Video-based 3D human pose and shape estimation plays a crucial role in enhancing human understanding. However, existing methods typically employ a unified model for both pose and shape parameter estimation, neglecting the inherent uncertainty introduced by factors such as blurring and occlusion during pose estimation. This oversight can lead to suboptimal solutions, especially in challenging scenarios. To tackle this issue, we propose a Coarse-to-Fine Diffusion-based Refinement Network (DR-Net). The initial regressor undergoes pre-training on large datasets to comprehend human motion dynamics. In the refinement framework, a diffusion-based refinement regressor is introduced, utilizing reverse denoising to incrementally refine pose parameters. In order to capture the kinematics of human motion and model parameter-feature relationships, we design the GCN-ATT module as a denoiser within the diffusion-based regressor. Extensive experiments demonstrate its superiority over state-of-the-art methods on benchmark datasets Human3.6M [1] and 3DPW [2].

Generating ligand molecules that bind to specific protein targets via generative models holds substantial promise for advancing structure-based drug design. Existing methods generate molecules from scratch without reference or template ligands, which poses challenges in model optimization and may yield suboptimal outcomes. To address this problem, we propose an innovative interaction-based retrieval-augmented diffusion model named IRDiff to facilitate target-aware molecule generation. IRDiff leverages a curated set of ligand references, i.e., those with desired properties such as high binding affinity, to steer the diffusion model towards synthesizing ligands that satisfy design criteria. Specifically, we utilize a protein-molecule interaction network (PMINet), which is pretrained with binding affinity signals to: (i) retrieve target-aware ligand molecules with high binding affinity to serve as references, and (ii) incorporate essential protein-ligand binding structures for steering molecular diffusion generation with two effective augmentation mechanisms, i.e., retrieval augmentation and self augmentation. Empirical studies on CrossDocked2020 dataset show IRDiff can generate molecules with more realistic 3D structures and achieve state-of-the-art binding affinities towards the protein targets, while maintaining proper molecular properties. The codes and models are available at https://github.com/YangLing0818/IRDiff

Conventional state representations in reinforcement learning often omit critical task-related details, presenting a significant challenge for value networks in establishing accurate mappings from states to task rewards. Traditional methods typically depend on extensive sample learning to enrich state representations with task-specific information, which leads to low sample efficiency and high time costs. Recently, surging knowledgeable large language models (LLM) have provided promising substitutes for prior injection with minimal human intervention. Motivated by this, we propose LLM-Empowered State Representation (LESR), a novel approach that utilizes LLM to autonomously generate task-related state representation codes which help to enhance the continuity of network mappings and facilitate efficient training. Experimental results demonstrate LESR exhibits high sample efficiency and outperforms state-of-the-art baselines by an average of **29%** in accumulated reward in Mujoco tasks and **30%** in success rates in Gym-Robotics tasks. Codes of LESR are accessible at https://github.com/thu-rllab/LESR.

With the advancement of AIGC, video frame interpolation (VFI) has become a crucial component in existing video generation frameworks, attracting widespread research interest. For the VFI task, the motion estimation between neighboring frames plays a crucial role in avoiding motion ambiguity. However, existing VFI methods always struggle to accurately predict the motion information between consecutive frames, and this imprecise estimation leads to blurred and visually incoherent interpolated frames. In this paper, we propose a novel diffusion framework, Motion-Aware latent Diffusion models (MADiff), which is specifically designed for the VFI task. By incorporating motion priors between the conditional neighboring frames with the target interpolated frame predicted throughout the diffusion sampling procedure, MADiff progressively refines the intermediate outcomes, culminating in generating both visually smooth and realistic results. Extensive experiments conducted on benchmark datasets demonstrate that our method achieves state-of-the-art performance significantly outperforming existing approaches, especially under challenging scenarios involving dynamic textures with complex motion.

Recent research on Diffusion Models and Transformers has brought significant advancements to 3D Human Pose Estimation (HPE). Nonetheless, existing methods often fail to concurrently address the issues of accuracy and generalization. In this paper, we propose a Geometry-guided Dif fusion Model with Masked Transformer (Masked Gifformer) for robust multi-view 3D HPE. Within the framework of the diffusion model, a hierarchical multi-view trans-former-based denoiser is exploited to fit the 3D pose distribution by systematically integrating joint and view information. To address the long-standing problem of poor generalization, we introduce a fully random mask mechanism without any additional learnable modules or parameters. Furthermore, we incorporate geometric guidance into the diffusion model to enhance the accuracy of the model. This is achieved by optimizing the sampling process to minimize reprojection errors through modeling a conditional guidance distribution. Extensive experiments on two benchmarks demonstrate that Masked Gifformer effectively achieves a trade-off between accuracy and generalization. Specifically, our method outperforms other probabilistic methods by > 40% and achieves comparable results with state-of-the-art deterministic methods. In addition, our method exhibits robustness to varying camera numbers, spatial arrangements, and datasets.

Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.

Due to the limitations of capture devices and scenarios, egocentric videos frequently have low visual quality, mainly caused by high compression and severe motion blur. With the increasing application of egocentric videos, there is an urgent need to enhance the quality of these videos through super-resolution. However, existing Video Super-Resolution (VSR) works, focusing on third-person view videos, are actually unsuitable for handling blurring artifacts caused by rapid ego-motion and object motion in egocentric videos. To this end, we propose EgoVSR, a VSR framework specifically designed for egocentric videos. We explicitly tackle motion blurs in egocentric videos using a Dual Branch Deblur Network (DB2Net) in the VSR framework. Meanwhile, a blurring mask is introduced to guide the DB2Net learning, and can be used to localize blurred areas in video frames. We also design a MaskNet to predict the mask, as well as a mask loss to optimize the mask estimation. Additionally, an online motion blur synthesis model for common VSR training data is proposed to simulate motion blurs as in egocentric videos. In order to validate the effectiveness of our proposed method, we introduce an EgoVSR dataset containing a large amount of fast-motion egocentric video sequences. Extensive experiments demonstrate that our EgoVSR model can efficiently super-resolve low-quality egocentric videos and outperform strong comparison baselines. Our code, pre-trained models and data can be found athttps://github.com/chiyich/EGOVSR/.

Finger vein authentication, recognized for its high security and specificity, has become a focal point in biometric research. Traditional methods predominantly concentrate on vein feature extraction for discriminative modeling, with a limited exploration of generative approaches. Suffering from verification failure, existing methods often fail to obtain authentic vein patterns by segmentation. To fill this gap, we introduce DiffVein, a unified diffusion model-based framework which simultaneously addresses vein segmentation and authentication tasks. DiffVein is composed of two dedicated branches: one for segmentation and the other for denoising. For better feature interaction between these two branches, we introduce two specialized modules to improve their collective performance. The first, a mask condition module, incorporates the semantic information of vein patterns from the segmentation branch into the denoising process. Additionally, we also propose a Semantic Difference Transformer (SD-Former), which employs Fourier-space self-attention and cross-attention modules to extract category embedding before feeding it to the segmentation task. In this way, our framework allows for a dynamic interplay between diffusion and segmentation embeddings, thus vein segmentation and authentication tasks can inform and enhance each other in the joint training. To further optimize our model, we introduce a Fourier-space Structural Similarity (FSSIM) loss function, which is tailored to improve the denoising network’s learning efficacy. Extensive experiments on the USM and THU-MVFV3V datasets substantiates DiffVein’s superior performance, setting new benchmarks in both vein segmentation and authentication tasks.

Finger vein recognition is an emerging biometric technology with high security and various application scenarios. Most finger vein recognition methods are based on a single view. However, the inherent problems in single-view finger vein recognition, such as limited feature, sensitivity to finger translation and rotation, and the ambiguity issue in 2D projections, hinder the improvement of the system performance. To address these problems and enhance finger vein verification performance, we employ multi-view finger vein images that are capable of providing a more comprehensive feature of 3D finger vein. Specifically, we design a novel low-cost full-view finger vein imaging device that enables full-view capture of finger veins with only a single camera and establish a multi-view finger vein dataset, named THU-MVFV. In addition, we propose a Multi-view Finger Vein Feature Encoding and Selection Network (MFV-FESNet), which is based on an improved Transformer encoder that can learn the dependencies between different views. By fusing the extracted global context feature and local dominant feature, the network can generate a feature descriptor with high discrimination. Extensive experiments are conducted on THU-MVFV and demonstrate the superior performance of the proposed model. The THU-MVFV dataset will be publicly available athttps://github.com/Finger-Vein-Dataset/THU-MVFV.

Graph generation is a fundamental task in machine learning with broad impacts on numerous real-world applications such as biomedical discovery and social science. Most recently, generative models, especially diffusion models (DMs), have shown great promise in synthesizing realistic graphs. However, existing DMs methods typically conduct diffusion processes directly in complex graph space (i.e., node feature, adjacency matrix, or both), resulting in high modeling complexity and poor multimodal distribution coverage. In this paper, we propose Graphusion, a novel and unified latent-based graph generative framework to address the problems. Specifically, Graphusion is composed of a variational graph autoencoder mapping raw graphs with high-dimensional discrete space to low-dimensional topology-injected latent space, and latent DMs running there, producing a smoother, faster, and more expressive graph generation procedure. Thanks to the latest space modeling, we further develop principled latent self-guidance to sufficiently cover the whole semantical distribution of the unlabeled graph set. Experiments show that our Graphusion framework can consistently outperform previous graph generation baselines on both generic and molecular graph datasets, demonstrating the generality and extensibility along with further analytical justifications.

Transformers have been used for 3D human pose estimation with excellent performance; however, most transformers focus on encoding the global spatio-temporal correlation of all joints in the human body and there are few studies on the local Spatio-temporal correlation of each joint in the human body. In this article, we propose a Global and Local Spatio-Temporal Encoder (GLSTE) to model the Spatio-temporal correlation. Specifically, a Global Spatial Encoder (GSE) and a Global Temporal Encoder (GTE) are constructed to capture the global spatial information of all joints in a single frame and the global temporal information of all frames, respectively. A Local Spatio-Temporal Encoder (LSTE) is constructed to capture the spatial and temporal information of each joint in the local N frames. Furthermore, we propose a parallel attention module with weight sharing to better incorporate spatial and temporal information into each node simultaneously. Extensive experiments show that GLSTE outperforms state-of-the-art methods with fewer parameters and less computational overhead on two challenging datasets: Human3.6 M and MPI-INF-3DHP. Especially in the evaluation of Human3.6 M dataset, the results of our method with 27 frames as input are better than the vast majority of recent SOTA methods with 81 and 243 frames as input, which indicates that the model can learn more useful information with smaller inputs.

Multi-focus image fusion is a technique to fuse the images focused on different depth ranges to generate an all-in-focus image. Existing deep learning approaches to multi-focus image fusion can be categorized as end-to-end methods and decision map based methods. End-to-end methods can generate natural fusion near the focus-defocus boundaries (FDB), but the output is often inconsistent with the input in the areas far from the boundaries (FFB). On the contrary, decision map based methods can preserve original images in the FFB areas, but often generate artifacts near the FDB. In this paper, we propose a dual-branch network for multi-focus image fusion (DB-MFIF) to exploit the best of both worlds, achieving better results in both FDB and FFB areas, i.e. with naturally sharper FDB areas and more consistent FFB areas with the inputs. In our DB-MFIF, an end-to-end branch and a decision map based branch are proposed to mutually assist each other. In addition, to this end, two map-based loss functions are also proposed. Experiments show that our method surpasses existing algorithms on multiple datasets, both qualitatively and quantitatively, and achieves the state-of-the-art performance. The code and model is available on GitHub:https://github.com/Zancelot/DB-MFIF.

The target of space-time video super-resolution (STVSR) is to increase the spatial-temporal resolution of low-resolution (LR) and low-frame-rate (LFR) videos. Recent approaches based on deep learning have made significant improvements, but most of them only use two adjacent frames, that is, short-term features, to synthesize the missing frame embedding, which cannot fully explore the information flow of consecutive input LR frames. In addition, existing STVSR models hardly exploit the temporal contexts explicitly to assist high-resolution (HR) frame reconstruction. To address these issues, in this article, we propose a deformable attention network called STDAN for STVSR. First, we devise a long short-term feature interpolation (LSTFI) module that is capable of excavating abundant content from more neighboring input frames for the interpolation process through a bidirectional recurrent neural network (RNN) structure. Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction. Experimental results on several datasets demonstrate that our approach outperforms state-of-the-art STVSR methods. The code is available at https://github.com/littlewhitesea/STDAN.

The high computational costs of video super-resolution (VSR) models hinder their deployment on resource-limited devices, e.g., smartphones and drones. Existing VSR models contain considerable redundant filters, which drag down the inference efficiency. To prune these unimportant filters, we develop a structured pruning scheme called Structured Sparsity Learning (SSL) according to the properties of VSR. In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks. Specifically, we develop a Residual Sparsity Connection (RSC) scheme for residual blocks of recurrent networks to liberate pruning restrictions and preserve the restoration information. For upsampling networks, we design a pixel-shuffle pruning scheme to guarantee the accuracy of feature channel-space conversion. In addition, we observe that pruning error would be amplified as the hidden states propagate along with recurrent networks. To alleviate the issue, we design Temporal Finetuning (TF). Extensive experiments show that SSL can significantly outperform recent methods quantitatively and qualitatively. The code is available at https://github.com/Zj-BinXia/SSL.

Newton's rings experiment is a fundamental experiment. The rings counting method has been used to reveal the phenomenon of equal-thickness interference in physics for over 100 years. This paper proposes two perspectives on observing Newton's rings. From the perspective of signal processing, fractional Fourier transform are introduced into the Newton's rings experiment to reveal the mathematical nature of the fringe. From the perspective of data analysis, the deep neural network is trained so that it can intelligently analyze Newton's rings. High precision measurement of physical parameters such as the radius of curvature of a lens can be completed directly without counting rings. These two new perspectives provide good extensions to university physics experiments, which can be followed by additional theoretical and experimental sessions to further understand Newton's rings. The new session can be added to the current physics, signal processing and artificial intelligence courses. A brand-new course is designed to help students understand recent Newton's rings processing methods and know the pros and cons of them. which broaden their horizons and stimulate their creative thinking.

Human motion modeling is essential for video-based 3D human pose and shape estimation. Most existing methods model human motion by learning a deterministic mapping from the input videos to the human body parameters, while the uncertainties such as occlusions and depth ambiguities are ignored. To address this problem, we propose a probabilistic model based on conditional normalizing flows called FlowPose to learn the distribution of feasible 3D human motion. This model allows access to the most likely 3D human poses given a video input, which means that more accurate and temporally coherent human poses can be obtained. Additionally, a contrastive training strategy is utilized to maximize the mutual information between video features and their 3D human poses, resulting in an improvement on feature extraction of the conditional flow model. Experimental results on two benchmarks 3DPW and Human3.6M demonstrate that our method outperforms the state-of-the-art video-based methods.

Transformer-based methods have shown impressive potential in many low-level vision tasks but are rarely used for low-light image enhancement (LLIE). Direct use of Transformer in LLIE will bring unnatural visual effects. This phenomenon encourages us to attempt to learn from the theory of Retinex. After trial and analysis, we finally propose Retiformer. Retiformer decomposes images into reflectance and illumination attention maps by Retinex Window Self-Attention (R-WSA). It will replace element-wise multiplication with the attention mechanism. By the R-WSA, we respectively apply a Decom-Retiformer block and an Enhance-Retiformer block at the head and tail of a Transformer-based backbone. They can decompose and align the reflection and illumination components just like RetinexNet. With this pipeline, Retiformer combines the advantages of Transformer and Retinex theory and achieves state-of-the-art performance of Retinex-based methods.

Although image quality assessment (IQA) methods have achieved remarkable success in the past decades, full-reference IQA is limited to reference images, while no-reference IQA has relatively poor performance. To boost the performance of IQA models in the no-reference scenario, a new class of IQA methods using content-variant high-quality images as references have emerged. However, the existing approaches do not take advantage of the content information of the content-variant reference (CVR) images, resulting in the insufficient use of high-quality reference information and the unsatisfactory robustness of the algorithm performance. To effectively utilize CVR images and make the algorithm more robust, we propose a CVR IQA scheme based on similar patch matching. For each image patch to be evaluated, the patch with the most similar content is first searched in the CVR image as the reference patch. Since the two patches are more similar, more useful reference information can be extracted. A similarity calculation module based on cross-attention is designed to find content-similar patches. Extensive experimental results show that the proposed algorithm has good performance and robustness.

Few-shot semantic segmentation aims to learn new knowledge rapidly with very few annotated data to segment novel classes. Recent methods follow a metric learning framework with prototypes for foreground representation [1]. However, representing support images by one or more prototypes may face problems caused by inadequate representation for segmentation, noise in complex scenes, and close semantic relation to background features. We propose a Spatial Correlation Fusion Network(SCFNet) for few-shot segmentation to address the issues. Firstly, to better capture fine-grained features, we design a Spatial Correlation Fusion module to address the loss of spatial information in support images, thus improving the performance of Few-shot segmentation. Secondly, a Prototype Contrastive Transformation(PCT) module is proposed to learn a transformation matrix for the prototype, which is capable of alleviating close semantic information and noise by adopting transformation loss. Experiments on PASCAL-5i[2] and COCO-20i[3] validate the effectiveness of our network for few-shot semantic segmentation and show our approach achieves state-of-the-art results.

Replaying previous training data is the most effective approach for Class-Incremental Learning (CIL), with its performance bounded by data availability. Therefore, many recent studies consider the Data-Free Class-Incremental Learning (DFCIL) problem that requires no previous data. However, the existing methods do not consider synthesising data of heterogeneity, thus limiting models’ generalizability. Such homogenous images further hinder the knowledge distillation process when regularising only the deeper layers close to the output, resulting in catastrophic forgetting. To address these issues, we present CDHD: a contrastive dreamer for hint distillation. Our approach starts with training a generator for data synthesis. A model inversion technique is introduced to obtain a generator capable of producing heterogeneous images from the classifier by imposing the ContRastive Loss. Moreover, to better transfer the previous knowledge to the current model, we force the teacher network to provide more general knowledge to its students by enforcing the Hint Loss in shallower layers rather than only in deeper ones. We validate the performance of CDHD on CIFAR-100 for various tasks and compare it against the SOTA baseline for DFCIL, demonstrating our superiorities and thus constituting a new benchmark.

Recently, iterative optimization-based approaches have gained tremendous progress in the field of stereo matching. However, it still remains a challenge to accurately estimate disparity for occlusion and textureless regions. To address this challenge, we present the Global Matching-Optimization Stereo Network (GMOStereo), which contains three components: Conv-Trans Feature Extraction Module (C-TFEM), Global Matching Module (GMM), and scene-aware disparity optimization. Before iterative optimization, attention-based GMM builds stable interdependence across distinct views. The C-TFEM, which extracts features through a two-branch network based of convolution blocks and transformer blocks, is designed to obtain global representations of features while preserving fine-grained information. The scene self-similarity adopted in disparity optimization provides supplement for matching information. Finally, a Matching-Optimization loss is designed to guide the training by imposing a direct constraint on the correlation volume. Evaluation demonstrates that GMOStereo achieves superior cross-dataset generalization performance and outperforms typical methods in the foreground and challenging regions on KITTI-2015 benchmarks.

Multi-expert ensemble models for long-tailed learning typically either learn diverse generalists from the whole dataset or aggregate specialists on different subsets. However, the former is insufficient for tail classes due to the high imbalance factor of the entire dataset, while the latter may bring ambiguity in predicting unseen classes. To address these issues, we propose a novel Local and Global Logit Adjustments (LGLA) method that learns experts with full data covering all classes and enlarges the discrepancy among them by elaborated logit adjustments. LGLA consists of two core components: a Class-aware Logit Adjustment (CLA) strategy and an Adaptive Angular Weighted (AAW) loss. The CLA strategy trains multiple experts which excel at each subset using the Local Logit Adjustment (LLA). It also trains one expert specializing in an inversely long-tailed distribution through Global Logit Adjustment (GLA). Moreover, the AAW loss adopts adaptive hard sample mining with respect to different experts to further improve accuracy. Extensive experiments on popular long-tailed benchmarks manifest the superiority of LGLA over the SOTA methods.

Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speechvisual synchronization. Code: https://github.com/CVMILab/Speech2Lip.

Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network. However, different from image synthesis, image restoration (IR) has a strong constraint to generate results in accordance with ground-truth. Thus, for IR, traditional DMs running massive iterations on a large model to estimate whole images or feature maps is inefficient. To address this issue, we propose an efficient DM for IR (DiffIR), which consists of a compact IR prior extraction network (CPEN), dynamic IR transformer (DIRformer), and denoising network. Specifically, DiffIR has two training stages: pretraining and training DM. In pretraining, we input ground-truth images into CPENS1to capture a compact IR prior representation (IPR) to guide DIRformer. In the second stage, we train the DM to directly estimate the same IRP as pretrained CPENS1only using LQ images. We observe that since the IPR is only a compact vector, DiffIR can use fewer iterations than traditional DM to obtain accurate estimations and generate more stable and realistic results. Since the iterations are few, our DiffIR can adopt a joint optimization of CPENS2, DIRformer, and denoising network, which can further reduce the estimation error influence. We conduct extensive experiments on several IR tasks and achieve SOTA performance while consuming less computational costs. Code is available at https://github.com/Zj-BinXia/DiffIR.

Although softmax loss and its variants have achieved great success in face verification, the performance is still subject to the data imbalance and early saturation problems. In this paper, we define hard samples as minority class samples and early saturation samples, in order to address both issues, we propose a new loss function termed Hard-Samples based Margin (HSM) loss. Inspired by the class-variant margin normalized softmax loss, we add larger margin on minority classes, the proposed real-class margin overcomes the negative influence from the data imbalance via making the optimization more balanced, while by expanding the margin of early saturated samples, the proposed pseudo-class margin keeps the samples away from the saturation region. Comprehensive experiments show that our HSM loss consistently surpasses the state-of-the-art loss functions on four popular face verification benchmarks.

The development of loss functions over the past few years has brought great success to face recognition. Most algorithms focus on improving the intra-class compactness of face features but ignore the inter-class separability. In this paper, we propose a method named EvenFace, which introduces a regularization variance item and a mean term of inter-class separability to further promote the even distribution of class centers on the hypersphere, thereby increasing the inter-class distance. In order to evaluate the inter-class separability, a new index is proposed to better reflect the distribution of class centers and guide the classification. By penalizing the angle between each identity and its surrounding neighbors, the resulting uniform distribution of identities enables full exploitation of the feature space, leading to discriminative face representations. Our proposed loss function can effectively boost the performance of softmax loss variants. Quantitative comparisons with other state-of-the-art methods on several benchmarks demonstrate the superiority of EvenFace.

Super-resolution (SR) techniques designed for real-world applications commonly encounter two primary challenges: generalization performance and restoration accuracy. We demonstrate that when methods are trained using complex, large-range degradations to enhance generalization, a decline in accuracy is inevitable. However, since the degradation in a certain real-world applications typically exhibits a limited variation range, it becomes feasible to strike a trade-off between generalization performance and testing accuracy within this scope. In this work, we introduce a novel approach to craft training degradation distributions using a small set of reference images. Our strategy is founded upon the binned representation of the degradation space and the Frechet distance between degradation distributions. Our results indicate that the proposed technique significantly improves the performance of test images while preserving generalization capabilities in real-world applications.

With the development of deep learning, image quality assessment (IQA) methods have made significant progress, but full-reference (FR) methods are limited by the reference image, and the performance of no-reference (NR) methods is relatively poor. Therefore, some researchers have introduced a new scheme, external-reference (ER) IQA, which uses an arbitrary high-quality image as the reference image (external reference image). The key to ER-IQA is how to extract useful reference information from external reference images and use it effectively. Since the content of the external reference image is independent of the distorted image, we think that the content information of the external reference image is harmful to the algorithm. Therefore, a content-constrain loss is designed for training the network to suppress the content information of external reference images. To utilize the external reference information more effectively, we design an attention-based adaptive feature fusion (AAFF) module. Experimental results demonstrate the effectiveness of the designed loss and feature fusion module.

Image quality assessment is usually achieved by pooling local quality scores. However, commonly used pooling strategies, based on simple sample statistics, are not always sensitive to distortions. In this short communication, we propose a novel perspective of pooling: reliable pooling through statistical hypothesis testing, which enables effective detection of subtle changes of population parameters when the underlying distribution of local quality scores is affected by distortions. To illustrate the significance of this novel perspective, we design a new pooling strategy utilising simple one-sided one-sample t-test. The experiments on benchmark databases show the reliability of hypothesis testing-based pooling, compared with state-of-the-art pooling strategies.

Although single image super-resolution (SR) has achieved great success, super-resolving the real-world Ultra-High-Resolution (UHR) image remains a challenging issue. Confronted with UHR images, most existing SR methods resort to patch-splitting so that the interconnections among the cropped patches are not attracted reasonable attention during the training and inference procedure. Rather than considering global image degradation levels and types, previous methods only focus on local degradation and unavoidably lead to inter-patch inconsistency, like blocking artifacts in the UHR image. To address this issue, we propose a real-world super-resolution framework to integrate the restoration of different patches through a Global Degradation Supervision Super-Resolution (GDSSR) method. Specifically, a lightweight Global Degradation Extractor is used for extracting global degradation features, which can facilitate restoring better local patches independently and enforce inter-patch consistency. Additionally, a joint training method of local and global patches is proposed to exercise global supervision during the training process, which enhances the degradation estimation and restores more natural results. Experiments show that our GDSSR method achieves superior restoration performance on real-world and UHR image SR datasets.

Finger vein recognition has received more attention in recent years due to its high security and promising development potential. However, extracting complete vein patterns and obtaining features from the original images suffer from the low contrast of finger vein images, which dramatically restrains the performance of finger vein recognition algorithms. Inspired by this motivation, we propose an explicit and implicit feature fusion Network (EIFNet) for finger vein verification. It can extract more comprehensive and discriminative features by complementarily fusing the features extracted from binary vein masks and gray original images. We design a feature fusion module (FFM) acting as a bridge between mask feature extraction module (MFEM) and contextual feature extraction module (CFEM) to achieve the optimal fusion of features. To obtain more accurate vein masks, we develop a novel finger vein pattern extraction method and provide the first finger vein segmentation dataset THUFVS. We solve the difficulty of building finger vein segmentation datasets in a simple but effective way, and develop a complete process encompassing dataset creation, data augmentation refinement and network design, which refers to the Mask Generation Module (MGM), for the deep learning based finger vein pattern extraction method. Experimental results demonstrate the superior verification performance of EIFNet on three widely used datasets compared with other existing methods.

Transparent and reflective objects are omnipresent in our daily life, but their unique visual and optical characteristics are notoriously challenging even for state-of-the-art deep networks of semantic segmentation. To alleviate this challenge, we construct a new large-scale real-world RGB-D dataset called TROSD, which is more comprehensive than existing datasets for transparent and reflective object segmentation. Our TROSD dataset contains 11,060 RGB-D images with three semantic classes in terms of transparent objects, reflective objects, and others, covering a variety of daily scenes. Together with the dataset, we also introduce a novel network (TROSNet) as a high-standard baseline to assist other researchers to develop and benchmark their algorithms of transparent and reflective object segmentation. Moreover, extensive experiments also clearly show that the proposed TROSD dataset has an excellent capacity to facilitate the development of semantic segmentation algorithms with strong generalizability.

Blind image super-resolution (blind SR) aims to generate high-resolution (HR) images from low-resolution (LR) input images with unknown degradations. To enhance the performance of SR, the majority of blind SR methods introduce an explicit degradation estimator, which helps the SR model adjust to unknown degradation scenarios. Unfortunately, it is impractical to provide concrete labels for the multiple combinations of degradations (e.g., blurring, noise, or JPEG compression) to guide the training of the degradation estimator. Moreover, the special designs for certain degradations hinder the models from being generalized for dealing with other degradations. Thus, it is imperative to devise an implicit degradation estimator that can extract discriminative degradation representations for all types of degradations without requiring the supervision of degradation ground-truth. To this end, we propose a Meta-Learning based Region Degradation Aware SR Network (MRDA), including Meta-Learning Network (MLN), Degradation Extraction Network (DEN), and Region Degradation Aware SR Network (RDAN). To handle the lack of ground-truth degradation, we use the MLN to rapidly adapt to the specific complex degradation after several iterations and extract implicit degradation information. Subsequently, a teacher network MRDAT is designed to further utilize the degradation information extracted by MLN for SR. However, MLN requires iterating on paired LR and HR images, which is unavailable in the inference phase. Therefore, we adopt knowledge distillation (KD) to make the student network learn to directly extract the same implicit degradation representation (IDR) as the teacher from LR images. Furthermore, we introduce an RDAN module that is capable of discerning regional degradations, allowing IDR to adaptively influence various texture patterns. Extensive experiments under classic and real-world degradation settings show that MRDA achieves SOTA performance and can generalize to various degradation processes.

Polarization imaging is sensitive to sub-wavelength microstructures of various cancer tissues, providing abundant optical characteristics and microstructure information of complex pathological specimens. However, how to reasonably utilize polarization information to strengthen pathological diagnosis ability remains a challenging issue. In order to take full advantage of pathological image information and polarization features of samples, we propose a dual polarization modality fusion network (DPMFNet), which consists of a multi-stream CNN structure and a switched attention fusion module for complementarily aggregating the features from different modality images. Our proposed switched attention mechanism could obtain the joint feature embeddings by switching the attention map of different modality images to improve their semantic relatedness. By including a dual-polarization contrastive training scheme, our method can synthesize and align the interaction and representation of two polarization features. Experimental evaluations on three cancer datasets show the superiority of our method in assisting pathological diagnosis, especially in small datasets and low imaging resolution cases. Grad-CAM visualizes the important regions of the pathological images and the polarization images, indicating that the two modalities play different roles and allow us to give insightful corresponding explanations and analysis on cancer diagnosis conducted by the DPMFNet. This technique has potential to facilitate the performance of pathological aided diagnosis and broaden the current digital pathology boundary based on pathological image features.

Numerous CNN-based algorithms have been proposed to reconstruct high-quality face images. However, the inability of convolution operation to model long-distance relationships limits the performance of the CNN-based methods. Moreover, in the high-resolution (HR) image reconstruction stage, with the well decoded feature representations, more efficient architecture design can be explored to synthesize pixel-level image details. In this work, we propose a spatial attention-guided CNN-Transformer aggregation network (SCTANet) for face image super-resolution (FSR) tasks. The core component in the deep feature extraction stage is the Hybrid Attention Aggregation (HAA) block. The HAA block has two parallel paths, one for the Residual Spatial Attention (RSA) block, the other for the Multi-scale Patch embedding and Spatial-attention Masked Transformer (MPSMT) block. The HAA block combines the strengths of CNN and transformer to effectively exploit both local and global information. For the reconstruction stage, we propose to use the Sub-pixel MLP-based Upsampling (SMU) module instead of the conventional CNN architecture. The SMU module promotes the reconstruction of pixel-level image details and reduces computational complexity. Extensive experiments on both synthetic and real-world face datasets demonstrate the superiority of our proposed SCTANet over state-of-the-art methods.

Compression artifacts removal methods based on convolutional neural networks have attracted great attention. However, most existing methods require a specific trained model for a specific compression quality factor (QF), which inevitably leads to resource-consuming. Unfortunately, the QF is unknown in most practical applications, so it is intractable to choose a suitable model. In this work, we experimentally analyze the relationship between compression index estimation and compression artifacts removal. Based on the connection between them, we couple compression index estimation with compression artifacts removal into a unified network. A network named CRESNet is proposed, working for a wide range of QFs by integrating channel regulation with an exit strategy. Specifically, CRESNet adopts a multi-stage progressive structure with an exit strategy embedded to automatically select the optimal exit stage according to the estimated compression index reflecting the difficulty of the input sample. Benefiting from the exit strategy, CRESNet removes artifacts from slightly compressed images through a simple process while doing an elaborate process for severely compressed images. Furthermore, a compression-information-guided channel regulation (CICR) mechanism is developed to adaptively regulate feature maps based on the estimated compression index. CRESNet achieves a more elegant trade-off between artifacts removal and detail preservation in a resource-efficient manner. Experiments demonstrate that CRESNet achieves state-of-the-art performance.

Despite the great progress in 3D human pose estimation from videos, it is still an open problem to take full advantage of a redundant 2D pose sequence to learn representative representations for generating one 3D pose. To this end, we propose an improved Transformer-based architecture, called Strided Transformer, which simply and effectively lifts a long sequence of 2D joint locations to a single 3D pose. Specifically, a Vanilla Transformer Encoder (VTE) is adopted to model long-range dependencies of 2D pose sequences. To reduce the redundancy of the sequence, fully-connected layers in the feed-forward network of VTE are replaced with strided convolutions to progressively shrink the sequence length and aggregate information from local contexts. The modified VTE is termed as Strided Transformer Encoder (STE), which is built upon the outputs of VTE. STE not only effectively aggregates long-range information to a single-vector representation in a hierarchical global and local fashion, but also significantly reduces the computation cost. Furthermore, a full-to-single supervision scheme is designed at both full sequence and single target frame scales applied to the outputs of VTE and STE, respectively. This scheme imposes extra temporal smoothness constraints in conjunction with the single target frame supervision and hence helps produce smoother and more accurate 3D poses. The proposed Strided Transformer is evaluated on two challenging benchmark datasets, Human3.6 M and HumanEva-I, and achieves state-of-the-art results with fewer parameters. Code and models are available athttps://github.com/Vegetebird/StridedTransformer-Pose3D.

One-stage space-time video super-resolution (STVSR) aims to directly reconstruct high-resolution (HR) and high frame rate (HFR) video from its low-resolution (LR) and low frame rate (LFR) counterpart. Due to the wide application, one-stage STVSR has drawn much attention recently. However, existing one-stage methods suffer from ineffective exploration of the auxiliary information from adjacent time steps that may be useful to STVSR at the current time step. To address this issue, we propose a novel Bidirectional Recurrent Space-Time Upsampling network called Bi-RSTU for one-stage STVSR to utilize auxiliary information at various time steps. Specifically, an efficient channel attention feature interpolation (ECAFI) module is devised to synthesize the intermediate frame’s LR feature by exploiting its two neighboring LR video frame features. Subsequently, we fuse the information from the previous time step into these intermediate and neighboring features. Finally, second-order attention spindle (SOAS) blocks are stacked to form the feature reconstruction module that learns a mapping from LR fused feature space to HR feature space. Experimental results on public datasets demonstrate that our Bi-RSTU shows competitive performance compared with current two-stage and one-stage state-of-the-art STVSR methods.

Non-Local Attention (NLA) brings significant improvement for Single Image Super-Resolution (SISR) by leveraging intrinsic feature correlation in natural images. However, NLA gives noisy information large weights and consumes quadratic computation resources with respect to the input size, limiting its performance and application. In this paper, we propose a novel Efficient Non-Local Contrastive Attention (ENLCA) to perform long-range visual modeling and leverage more relevant non-local features. Specifically, ENLCA consists of two parts, Efficient Non-Local Attention (ENLA) and Sparse Aggregation. ENLA adopts the kernel method to approximate exponential function and obtains linear computation complexity. For Sparse Aggregation, we multiply inputs by an amplification factor to focus on informative features, yet the variance of approximation increases exponentially. Therefore, contrastive learning is applied to further separate relevant and irrelevant features. To demonstrate the effectiveness of ENLCA, we build an architecture called Efficient Non-Local Contrastive Network (ENLCN) by adding a few of our modules in a simple backbone. Extensive experimental results show that ENLCN reaches superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.

Reference-based super-resolution (RefSR) has made significant progress in producing realistic textures using an external reference (Ref) image. However, existing RefSR methods obtain high-quality correspondence matchings consuming quadratic computation resources with respect to the input size, limiting its application. Moreover, these approaches usually suffer from scale misalignments between the low-resolution (LR) image and Ref image. In this paper, we propose an Accelerated Multi-Scale Aggregation network (AMSA) for Reference-based Super-Resolution, including Coarse-to-Fine Embedded PatchMatch (CFE-PatchMatch) and Multi-Scale Dynamic Aggregation (MSDA) module. To improve matching efficiency, we design a novel Embedded PatchMacth scheme with random samples propagation, which involves end-to-end training with asymptotic linear computational cost to the input size. To further reduce computational cost and speed up convergence, we apply the coarse-to-fine strategy on Embedded PatchMacth constituting CFE-PatchMatch. To fully leverage reference information across multiple scales and enhance robustness to scale misalignment, we develop the MSDA module consisting of Dynamic Aggregation and Multi-Scale Aggregation. The Dynamic Aggregation corrects minor scale misalignment by dynamically aggregating features, and the Multi-Scale Aggregation brings robustness to large scale misalignment by fusing multi-scale information. Experimental results show that the proposed AMSA achieves superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.

Pose-invariant face recognition is a practically useful but challenging task. This paper introduces a novel method to learn pose-invariant feature representation without normalizing profile faces to frontal ones or learning disentangled features. We first design a novel strategy to learn pose-invariant feature embeddings by distilling the angular knowledge of frontal faces extracted by teacher network to student network, which enables the handling of faces with large pose variations. In this way, the features of faces across variant poses can cluster compactly for the same person to create a pose-invariant face representation. Secondly, we propose a Pose-Adaptive Angular Distillation loss to mitigate the negative effect of uneven distribution of face poses in the training dataset to pay more attention to the samples with large pose variations. Extensive experiments on two challenging benchmarks (IJB-A and CFP-FP) show that our approach consistently outperforms the existing methods.

Drug-target interaction(DTI) prediction is one of the most important topics in drug design and drug development, and deep learning approaches have achieved state-of-the-art performance in this field. However, the current methods are difficult to successfully combine the local and global features of drug molecules and protein sequences, while ignoring the modeling of complicated interaction mechanisms, which leads to a certain limitation of prediction performance. To overcome this barrier, we propose an end-to-end method based on Convolutional Neural Network (CNN) and Transformer to predict DTI problems, named ConformerDTI. The CNN and Transformer branches extract features from the simplified molecular input line entry system (SMILES) string of drugs and the amino acid sequence of proteins, respectively. The local and global features are coupled by the mutual transfer of the two branches through cross attention. Decoupling of local and global features in parallel leverages CNN’s power in extracting local features as well as the efficiency of Transformer at global processing. I n addition, ConformerDTI exploits the convolutional interaction network to model the interaction mechanism, both drugs and targets are convoluted by dynamic filters generated based on each other. Experimental results demonstrate that our model has better prediction performance than the most advanced deep learning methods on three different datasets. Furthermore, this performance improvement was validated by ablation experiments.

Image harmonization aims to achieve visual consistency in composite images by adapting a foreground to make it compatible with a background. However, existing methods always only use the real image as the positive sample to guide the training, and at most introduce the corresponding composite image as a single negative sample for an auxiliary constraint, which leads to limited distortion knowledge, and further causes a too large solution space, making the generated harmonized image distorted. Besides, none of them jointly constrain from the foreground selfstyle and foreground-background style consistency, which exacerbates this problem. Moreover, recent region-aware adaptive instance normalization achieves great success but only considers the global background feature distribution, making the aligned foreground feature distribution biased. To address these issues, we propose a self-consistent style contrastive learning scheme (SCS-Co). By dynamically generating multiple negative samples, our SCS-Co can learn more distortion knowledge and well regularize the generated harmonized image in the style representation space from two aspects of the foreground self-style and foreground-background style consistency, leading to a more photorealistic visual result. In addition, we propose a background-attentional adaptive instance normalization (BAIN) to achieve an attention-weighted background feature distribution according to the foreground-background feature similarity. Experiments demonstrate the superiority of our method over other state-of-the-art methods in both quantitative comparison and visual analysis.

Video frame interpolation (VFI) aims to synthesize an intermediate frame between two consecutive original frames. Most existing methods simply linearly combine the warped frames, leading to a loss of image texture. Since moving objects usually have similarities in consecutive frames, we propose a similarity-aware video frame interpolation method (SAIN) that searches patches with similar texture in the embedding space from input frames to extract features and capture image details. To gather the frame details and restore image texture, SAIN incorporates an implicit neural representation learning from similar patches to enrich image details and refine outputs in frame synthesis networks. Experiments demonstrate that SAIN preserves image texture and enhances interpolated image quality significantly.

With the increasing availability of portable cameras and smart phones, directly estimating PM2.5based on digital photography shows advantages in efficiency and economic costs. In this paper, a novel Two-stream Non-uniform Concentration Reasoning Network (TNCR-Net) is proposed for single image PM2.5concentration estimation. Motivated by locally non-uniform particle pollution concentration distribution in images, we adopt patch-based scheme and adaptive weighted average mechanism to obtain patch-wise concentration and relative weight based on spatially varying perceptual relevance of local particle pollution concentration. Then aggregate patch-wise concentrations according to relative weights. To learn more effective feature from particular pollution image, we use a two-stream network structure with the dark channel map as the input of one stream. Besides, we employ attention-based feature fusion method to flexibly aggregate the feature maps of the two streams. Experiments on real-world dataset indicate that our TNCR-Net outperforms other state-of-the-art methods with fewer parameters.

Full-reference image quality assessment aims to predict the perceptual quality of a distorted image based on its similarity to the pristine reference. In this paper, we propose a robust image similarity metric by fully exploring the representation power of deep learning-based features. A convolutional neu-ral network (CNN) is adopted to extract deep features from multiple scales. We show that such CNN features that con-tain multi -scale visual information are comprehensive and ro-bust enough for quality assessment. We further propose a quality-oriented feature regression (QOFR) module based on the multi-layer perceptron architecture. The QOFR module can efficiently integrate hierarchy CNN features and generate the final quality score. Extensive experiments on the bench-mark datasets demonstrate that our method achieves state-of-the-art performance with outstanding robustness and general-ization ability.

Existing depth completion methods with standard CNN and fixed guidance information often produce invalid value diffusion and mismatch between the guidance and the depth. To address this issue, we propose a SADG-Net for depth completion. Specifically, a sparse adaptive module is designed to infer the initial dense depth map and its confidence, as well as the affinity between pixels. Then we develop a dynamic guidance spatial propagation network to refine the initial depth map and dynamically update the guidance information with the inferred depth. In contrast to previous algorithms, our method effectively optimizes the processing for sparse depth and significantly alleviates the error accumulation issues in spatial propagation. Extensive experiments demonstrate that our model improves upon the state-of-the-art performance on NYUv2 and BIDCD datasets.

The main focus of most existing face hallucination methods is to generate visually pleasing results. However, in many applications, the final goal is to identify the person in the low-resolution (LR) image. In this paper, we propose a texture and identity integration network (TIIN) to effectively incorporate identity information into face hallucination tasks. TIIN consists of an identity-preserving denormalization module (IDM) and an equalized texture enhance module (ETEM). The IDM exploits the identity prior and the ETEM improves image quality through histogram equalization. To extract identity information effectively, we propose a resolution-robust identity knowledge distillation network (RIKDN). RIKDN is specifically designed for LR face recognition and can be of independent interest. It employs two teacher-student streams. One stream narrows the performance gap between high-resolution (HR) and LR images. The other distills correlation information from the HR-HR teacher stream to guide learning in the LR-HR student stream. We conduct extensive experiments on multiple datasets to demonstrate the effectiveness of our methods.

With the popularity of image sensors in various mobile devices, image blurring caused by hand shaking or out of focus becomes ubiquitous, which deteriorates image quality and poses challenges for vision tasks, including object detection, image classification and image segmentation. Designing an efficient blur detection algorithm which can automatically detect and locate blurred regions becomes necessary. In this letter, we design an end-to-end convolution neural network called heterogeneous attention nested U-shaped network (HANUN) for blur detection. We introduce pyramid pooling into encoders to enhance the feature extraction at different scales and reduce the gradual information loss. Inspired by the nested network design, small U-shaped networks are embedded into our decoders to increase the network depth and promote feature fusion with different receptive field scales. In addition, we incorporate a channel attention mechanism in the proposed network to highlight the informative features for detecting the blurry regions. Experimental results show that HANUN outperforms other state-of-the-art algorithms for blur detection tasks on public datasets and real-world images.

Self-supervised representation learning for videos has been very attractive recently because these methods exploit the information inherently obtained from the video itself instead of annotated labels that is quite time-consuming. However, existing methods ignore the importance of global observation while performing spatio-temporal transformation perception, which highly limits the expression capabilities of the video representation. This paper proposes a novel pretext task that combines the temporal information perception of the video with the motion amplitude perception of moving objects to learn the spatio-temporal representation of the video. Specifically, given a video clip containing several video segments, each video segment is sampled by different sampling rates and the order of video segments is disrupted. Then, the network is used to regress the sampling rate of each video segment and classify the order of input video segments. In the pre-training stage, the network can learn rich spatio-temporal semantic information where content-related contrastive learning is introduced to make the learned video representation more discriminative. To alleviate the appearance dependency caused by contrastive learning, we design a novel and robust vector similarity measurement approach, which can take feature alignment into consideration. Moreover, a view synthesis framework is proposed to further improve the performance of contrastive learning by automatically generating reasonable transformed views. We conduct benchmark experiments with several 3D backbone networks on two datasets. The results show that our proposed method outperforms the existing state-of-the-art methods across the three backbones on two downstream tasks of human action recognition and video retrieval.

The finger vein trait has attracted widespread attention for personal authentication in recent years. However, most finger vein verification methods are performed on the single perspective, captured by a monocular near-infrared camera fixed at one side of the finger. Consequently, the contents of a single perspective have few details of the spatial network structure of the finger vein and show noticeable differences even if the posture of the same finger is slightly different. Both of them impact the verification performance. Hence, finger vein images captured from different viewpoints are considered in this work. We first design a low-cost multi-perspective based dorsal finger vein imaging device for data collection. A deep neural network named Hierarchical Content-Aware Network (HCAN) is then proposed to extract the discriminative hierarchical features of the finger vein. Specifically, HCAN is compound of a Global Stem Network (GSN) and a Local Perception Module (LPM). GSN aims to extract the latent global 3D feature from all perspectives through a recurrent neural network. It enables the model to retain the details in previous hidden states by incorporating a memory weighting strategy. LPM is designed to perceive each perspective from the aspect of image entropy. Guided by the entropy loss, LPM captures the prominent local feature and improves the discriminability and robustness of the hierarchical feature. The experimental results on the newly collected THU-MFV database demonstrate the superiority of the proposed method in comparison with other multi-perspective and single-perspective based methods.

In recent years, face recognition has made a remarkable breakthrough due to the emergence of deep learning. However, compared with frontal face recognition, plenty of deep face recognition models still suffer serious performance degradation when handling profile faces. To address this issue, we propose a novel Frontal-Centers Guided Loss (FCGFace) to obtain highly discriminative features for face recognition. Most existing discriminative feature learning approaches project features from the same class into a separated latent subspace. These methods only model the distribution at the identity-level but ignore the latent relationship between frontal and profile viewpoints. Different from these methods, FCGFace takes viewpoints into consideration by modeling the distribution at both the identity-level and the viewpoint-level. At the identity-level, a softmax-based loss is employed for a relatively rough classification. At the viewpoint-level, centers of frontal face features are defined to guide the optimization conducted in a more refined way. Specifically, our FCGFace is capable of adaptively adjusting the distribution of profile face features and narrowing the gap between them and frontal face features during different training stages to form compact identity clusters. Extensive experimental results on popular benchmarks, including cross-pose datasets (CFP-FP, CPLFW, VGGFace2-FP, and Multi-PIE) and non-cross-pose datasets (YTF, LFW, AgeDB-30, CALFW, IJB-B, IJB-C, and RFW), have demonstrated the superiority of our FCGFace over the SOTA competitors.

With the help of convolutional neural networks (CNNs), deep learning-based methods have achieved remarkable performance in face super-resolution (FSR) task. Despite their success, most of the existing methods neglect non-local correlations of face images, leaving much room for improvement. In this paper, we introduce a novel end-to-end trainable attention-driven graph neural network (AD-GNN) for more discriminative feature extraction and feature relation modeling. This is achieved by two major components. The first component is a cross-scale dynamic graph (CDG) block. The CDG block considers cross-scale relationships of patches in distant areas and employs two dynamic graphs to construct enhanced features. The second component is a series of channel attention and spatial dynamic graph (CASDG) blocks. A CASDG block has a channel-wise attention unit and a spatial-aware dynamic graph (SDG) unit. The SDG unit extracts informative features by exploring spatial non-local self-similarity information of the patches using dynamic graph convolution. Using these two components, facial details can be effectively reconstructed with the help of information supplemented by similar but spatially remote patches and structural information of faces. Extensive experiments on two public benchmarks demonstrate the superiority of AD-GNN over the state-of-the-art FSR methods.

Brain stroke lesion segmentation is of great importance for stroke rehabilitation neuroimaging analysis. Due to the large variance of stroke lesion shapes and similarities of tissue intensity distribution, it remains a challenging task. To help detect abnormalities, the anatomical symmetries of brain magnetic resonance (MR) images have been widely used as visual cues for clinical practices. However, most methods for brain images segmentation do not fully utilize structural symmetry information. This paper presents a novel mirror difference aware network (MDAN) for stroke lesion segmentation. The network uses an encoder-decoder architecture, aiming at holistically exploiting the symmetries of image features. Specifically, a differential feature augmentation (DFA) module is developed in the encoding path to highlight the semantically pathological asymmetries of features in abnormalities. In the DFA module, a Siamese contrastive supervised loss is designed to enhance discriminative features, and a mirror position-based difference augmentation (MDA) module is used to further magnify the discrepancy. Moreover, mirror feature fusion (MFF) modules are applied to efficiently fuse and transfer the information both of the original input and the horizontally flipped features to the decoding path. Extensive experiments on the Anatomical Tracings of Lesions After Stroke (ATLAS) dataset show the proposed MDAN outperforms the state-of-the-art methods.

Ophthalmologists have used fundus images to screen and diagnose eye diseases. However, different equipments and ophthalmologists pose large variations to the quality of fundus images. Low-quality (LQ) degraded fundus images easily lead to uncertainty in clinical screening and generally increase the risk of misdiagnosis. Thus, real fundus image restoration is worth studying. Unfortunately, real clinical benchmark has not been explored for this task so far. In this paper, we investigate the real clinical fundus image restoration problem. Firstly, We establish a clinical dataset, Real Fundus (RF), including 120 low- and high-quality (HQ) image pairs. Then we propose a novel Transformer-based Generative Adversarial Network (RFormer) to restore the real degradation of clinical fundus images. The key component in our network is the Window-based Self-Attention Block (WSAB) which captures non-local self-similarity and long-range dependencies. To produce more visually pleasant results, a Transformer-based discriminator is introduced. Extensive experiments on our clinical benchmark show that the proposed RFormer significantly outperforms the state-of-the-art (SOTA) methods. In addition, experiments of downstream tasks such as vessel segmentation and optic disc/cup detection demonstrate that our proposed RFormer benefits clinical fundus image analysis and applications.

Object detection has made enormous progress and has been widely used in many applications. However, it performs poorly when only limited training data is available for novel classes that the model has never seen before. Most existing approaches solve few-shot detection tasks implicitly without directly modeling the detectors for novel classes. In this article, we propose GenDet, a new meta-learning-based framework that can effectively generate object detectors for novel classes from few shots and, thus, conducts few-shot detection tasks explicitly. The detector generator is trained by numerous few-shot detection tasks sampled from base classes each with sufficient samples, and thus, it is expected to generalize well on novel classes. An adaptive pooling module is further introduced to suppress distracting samples and aggregate the detectors generated from multiple shots. Moreover, we propose to train a reference detector for each base class in the conventional way, with which to guide the training of the detector generator. The reference detectors and the detector generator can be trained simultaneously. Finally, the generated detectors of different classes are encouraged to be orthogonal to each other for better generalization. The proposed approach is extensively evaluated on the ImageNet, VOC, and COCO data sets under various few-shot detection settings, and it achieves new state-of-the-art results.

Probabilistic meta-learning methods recently have achieved impressive success in few-shot image classification. However, they introduce a huge number of random variables for neural network weights and thus severe computational and inferential challenges. In this paper, we propose a novel probabilistic meta-learning method called amortized Bayesian prototype meta-learning. In contrast to previous methods, we introduce only a small number of random variables for latent class prototypes rather than a huge number for network weights; we learn to learn the posterior distributions of these latent prototypes in an amortized inference way with no need for an extra amortization network, such that we can easily approximate their posteriors conditional on few labeled samples, whenever at meta-training or meta-testing stage. The proposed method can be trained end-to-end without any pre-training. Compared with other probabilistic meta-learning methods, our proposed approach is more interpretable with much less random variables, while still be able to achieve competitive performance for few-shot image classification problems on various benchmark datasets. Its excellent robustness and predictive uncertainty are also demonstrated through ablation studies.

The depth maps captured by RGB-D cameras usually are of low resolution, entailing recent efforts to develop depth super-resolution (DSR) methods. However, several problems remain in existing DSR methods. First, conventional DSR methods often suffer from unexpected artifacts. Secondly, high-resolution (HR) RGB features and low-resolution (LR) depth features are often fused in shallow layers only. Thirdly, only the last layer of features is used for reconstruction. To address the above problems, we propose Coupled U-Net (CU-Net), a new color image guided DSR method built on two U-Net branches for HR color images and LR depth maps, respectively. The CU-Net embeds a dual skip connection structure to leverage the feature interaction of the two branches, and a multi-scale fusion to fuse the deeper and multi-scale features of two branch decoders for more effective feature reconstruction. Moreover, a channel attention module is proposed to eliminate artifacts. Extensive experiments show that the proposed CU-Net outperforms state-of-the-art methods.

Network compression has been widely studied since it is able to reduce the memory and computation cost during inference. However, previous methods seldom deal with complicated structures like residual connections, group/depth-wise convolution and feature pyramid network, where channels of multiple layers are coupled and need to be pruned simultaneously. In this paper, we present a general channel pruning approach that can be applied to various complicated structures. Particularly, we propose a layer grouping algorithm to find coupled channels automatically. Then we derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Moreover, we find that inference speedup on GPUs is more correlated with the reduction of memory rather than FLOPs, and thus we employ the memory reduction of each channel to normalize the importance. Our method can be used to prune any structures including those with coupled channels. We conduct extensive experiments on various backbones, including the classic ResNet and ResNeXt, mobile-friendly MobileNetV2, and the NAS-based RegNet, both on image classification and object detection which is under-explored. Experimental results validate that our method can effectively prune sophisticated networks, boosting inference speed without sacrificing accuracy.

Face detection is an upstream task of facial image analysis. In many real-world scenarios, we need to detect small, occluded or dense faces that are hard to detect, but hard face detection is a challenging task in particular considering the balance between accuracy and inference speed for real-world applications. This paper proposes an Hourglass Face Detector (HFD) for hard face by developing a deep one-stage fully-convolutional hourglass network, which achieves an excellent balance between accuracy and inference speed. To this end, the HFD firstly shrinks a feature map by a series of stridden convolutional layers rather than pooling layers, so that useful subtle information is preserved better. Secondly, it exploits context information by merging fine-grained shallow feature maps with deep ones full of semantic information, making a better fusion of detailed information and semantic information to achieve a better detection of small faces. Moreover, the HFD exploits prior and multiscale information from the training data to enhance its scale-invariance and adaptability of anchor scales. Compared with the SSH and S3FD methods, the HFD can achieve a better performance in average precision on detecting hard faces as well as a quicker inference. Experiments on the WIDER FACE and FDDB datasets demonstrate the superior performance of our proposed method.

Although face recognition has been widely applied in many areas, pose-robust face recognition is still a challenging topic due to the large pose variations in real scenes. In this paper, we propose to learn the pose-robust face representation by normalizing the profile face in feature level directly and jointly considering both intra-class compactness and inter-class separability. Our approach minimizes the angular distance between the profile face and the positive frontal anchor. And it maximizes the angular distance between the profile face and the negative frontal anchor simultaneously. Furthermore, we modify the Triplet loss and derive the Triplet Angular loss to guarantee the intra-class compactness and the inter-class separability in angular space. In this way, the faces under varying poses can cluster compactly to create a pose-robust feature representation. Extensive experiments on two challenging benchmarks (CFP-FP and IJB-A) illustrate that our approach achieves a competitive performance in the field of pose-robust face recognition.

This paper proposes a novel approach for Sketch-Based Image Retrieval (SBIR), for which the key is to bridge the gap between sketches and photos in terms of the data representation. Inspired by channel-wise attention explored in recent years, we present a Domain-Aware Squeeze-and-Excitation (DASE) network, which seamlessly incorporates the prior knowledge of sample sketch or photo into SE module and make the SE module capable of emphasizing appropriate channels according to domain signal. Accordingly, the proposed network can switch its mode to achieve a better domain feature with lower intra-class discrepancy. Moreover, while previous works simply focus on minimizing intra-class distance and maximizing inter-class distance, we introduce a loss function, named Multiplicative Euclidean Margin Softmax (MEMS), which introduces multiplicative Euclidean margin into feature space and ensure that the maximum intra-class distance is smaller than the minimum inter-class distance. This facilitates learning a highly discriminative feature space and ensures a more accurate image retrieval result. Extensive experiments are conducted on two widely used SBIR benchmark datasets. Our approach achieves better results on both datasets, surpassing the state-of-the-art methods by a large margin.

Elastic weight consolidation (EWC) has been successfully applied for general incremental learning to overcome the catastrophic forgetting issue. It adaptively constrains each parameter of the new model not to deviate much from its counterpart in the old model during fine-tuning on new class data sets, according to its importance weight for old tasks. However, the previous study demonstrates that it still suffers from catastrophic forgetting when directly used in object detection. In this article, we show EWC is effective for incremental object detection if with critical adaptations. First, we conduct controlled experiments to identify two core issues why EWC fails if trivially applied to incremental detection: 1) the absence of old class annotations in new class images makes EWC misclassify objects of old classes in these images as background and 2) the quadratic regularization loss in EWC easily leads to gradient explosion when balancing old and new classes. Then, based on the abovementioned findings, we propose the corresponding solutions to tackle these issues: 1) utilize pseudobounding box annotations of old classes on new data sets to compensate for the absence of old class annotations and 2) adopt a novel Huber regularization instead of the original quadratic loss to prevent from unstable training. Finally, we propose a general EWC-based incremental object detection framework and implement it under both Fast R-CNN and Faster R-CNN, showing its flexibility and versatility. In terms of either the final performance or the performance drop with respect to the upper bound of joint training on all seen classes, evaluations on the PASCAL VOC and COCO data sets show that our method achieves a new state of the art.

In deep face recognition, the commonly used softmax loss and its newly proposed variations are not yet sufficiently effective to handle the class imbalance and softmax saturation issues during the training process while extracting discriminative features. In this brief, to address both issues, we propose a class-variant margin (CVM) normalized softmax loss, by introducing a true-class margin and a false-class margin into the cosine space of the angle between the feature vector and the class-weight vector. The true-class margin alleviates the class imbalance problem, and the false-class margin postpones the early individual saturation of softmax. With negligible computational complexity increment during training, the new loss function is easy to implement in the common deep learning frameworks. Comprehensive experiments on the LFW, YTF, and MegaFace protocols demonstrate the effectiveness of the proposed CVM loss function.

Data clustering is one of the most fundamental techniques in exploratory data analysis. It is widely used for determining the underlying data structure, classifying natural data and compressing data in engineering, business management, social statistics, computer science, and medicine. Under the assumption that clusters are high density regions in the feature space separated by relatively low density neighbors, a novel approach is proposed for modeling any high dimensional clustering problem as a one-dimensional analysis of the probability distribution. First, a special path between two vertexes, namely eigenpath, is defined in this paper to represent their close connection. Second, we propose the connectedness index based on the eigenpath for quantitatively describing the connection between two vertexes. Third, the connectedness index is applied to the candidates of cluster centers and measures the connection between different candidates. Then an indicative curve can be drawn with the knowledge of connectedness index. This approach not only provides effective indicative curve for unknown data sets but also facilitates eliminating the curse of dimensionality partly as well as correctly recognizes arbitrary cluster forms and automatically excludes outliers. Extensive experiments showed the effectiveness and efficiency of the proposed approach.

Recent years have witnessed great success of applying deep convolutional neural networks (CNNs) to single image super-resolution (SISR). However, most of these algorithms focus on increasing modeling capability through developing deeper and wider networks, improving the performance but at a cost of huge computation. Targeting at a better trade-off between efficiency and effectiveness, we propose ESASN, an efficient second-order attention spindle network for lightweight SISR. ESASN is built upon efficient second-order attention spindle (ESAS) blocks, each of which contains two well-designed new modules, efficient multi-scale (EMS) module and second-order attention (SOA) module. EMS reduces a considerable number of parameters while retaining the multi-scale structure to explore rich features. SOA further rescales the multi-scale feature maps, capturing the inter-dependencies among channels pixel-wisely with little additional cost. Both qualitative and quantitative experimental results demonstrate that the combination of EMS and SOA works out favorably for SISR, lifting the performance with fewer parameters. Code is available at https://github.com/yiyunchen/ESASN.

Recently, deep convolutional neural network (CNN) have been widely used in image restoration and obtained great success. However, most of existing methods are limited to local receptive field and equal treatment of different types of information. Besides, existing methods always use a multi-supervised method to aggregate different feature maps, which can not effectively aggregate hierarchical feature information. To address these issues, we propose an attention cube network (A-CubeNet) for image restoration for more powerful feature expression and feature correlation learning. Specifically, we design a novel attention mechanism from three dimensions, namely spatial dimension, channel-wise dimension and hierarchical dimension. The adaptive spatial attention branch (ASAB) and the adaptive channel attention branch (ACAB) constitute the adaptive dual attention module (ADAM), which can capture the long-range spatial and channel-wise contextual information to expand the receptive field and distinguish different types of information for more effective feature representations. Furthermore, the adaptive hierarchical attention module (AHAM) can capture the long-range hierarchical contextual information to flexibly aggregate different feature maps by weights depending on the global context. The ADAM and AHAM cooperate to form an 'attention in attention' structure, which means AHAM's inputs are enhanced by ASAB and ACAB. Experiments demonstrate the superiority of our method over state-of-the-art image restoration methods in both quantitative comparison and visual analysis.

In this paper, we develop a concise but efficient network architecture called linear compressing based skipconnecting network (LCSCNet) for image super-resolution. Compared with two representative network architectures with skip connections, ResNet and DenseNet, a linear compressing layer is designed in LCSCNet for skip connection, which connects former feature maps and distinguishes them from newly-explored feature maps. In this way, the proposed LCSCNet enjoys the merits of the distinguish feature treatment of DenseNet and the parametereconomic form of ResNet. Moreover, to better exploit hierarchical information from both low and high levels of various receptive fields in deep models, inspired by gate units in LSTM, we also propose an adaptive element-wise fusion strategy with multisupervised training. Experimental results in comparison with state-of-the-art algorithms validate the effectiveness of LCSCNet.

In this paper, we propose a new loss function, termed the equalized margin (EqM) loss, which is designed to make both intra-class scopes and inter-class margins similar over all classes, such that all the classes can be evenly distributed on the hypersphere of the feature space. The EqM loss controls both the lower limit of intra-class similarity by exploiting hard-sample mining and the upper limit of inter-class similarity by assuring equalized margins. Therefore, using the EqM loss, we can not only obtain more discriminative features, but also overcome the negative impacts from the data imbalance on the inter-class margins. We also observe that the EqM loss is stable with the variation of the scale in normalized Softmax. Furthermore, by conducting extensive experiments on LFW, YTF, CFP, MegaFace and IJB-B, we are able to verify the effectiveness and superiority of the EqM loss, compared with other state-of-the-art loss functions for face recognition.

Deep learning methods have witnessed the great progress in image restoration with specific metrics (e.g., PSNR, SSIM). However, the perceptual quality of the restored image is relatively subjective, and it is necessary for users to control the reconstruction result according to personal preferences or image characteristics, which cannot be done using existing deterministic networks. This motivates us to exquisitely design a unified interactive framework for general image restoration tasks. Under this framework, users can control continuous transition of different objectives, e.g., the perception-distortion trade-off of image super-resolution, the trade-off between noise reduction and detail preservation. We achieve this goal by controlling the latent features of the designed network. To be specific, our proposed framework, named Controllable Feature Space Network (CFSNet), is entangled by two branches based on different objectives. Our framework can adaptively learn the coupling coefficients of different layers and channels, which provides finer control of the restored image quality. Experiments on several typical image restoration tasks fully validate the effective benefits of the proposed method. Code is available at https://github.com/qibao77/CFSNet.

Although state-of-the-art methods for human pose estimation have achieved superior results on the single image, their performance on videos usually deteriorates dramatically due to motion blur and occlusion. Since there is close temporal correlation among video frames, exploiting the contextual information properly can be helpful to tackle the problem. In this paper, we present a Temporal Feature Enhancing Network (TFEN) for video human pose estimation. It boosts the per-frame features by utilizing motion information in terms of optical flow and conducting temporal feature encoding by the convolution gated recurrent units (convGRU). It is an end-to-end learning framework and can extend any image based algorithm to video pose estimation. The experimental results validate the effectiveness of the proposed approach on two large-scale video pose estimation benchmarks.

Single image super-resolution (SISR) has witnessed great progress as convolutional neural network (CNN) gets deeper and wider. However, enormous parameters hinder its application to real world problems. In this letter, We propose a lightweight feature fusion network (LFFN) that can fully explore multi-scale contextual information and greatly reduce network parameters while maximizing SISR results. LFFN is built on spindle blocks and a softmax feature fusion module (SFFM). Specifically, a spindle block is composed of a dimension extension unit, a feature exploration unit. and a feature refinement unit. The dimension extension layer expands low dimension to high dimension and implicitly learns the feature maps which are suitable for the next unit. The feature exploration unit performs linear and nonlinear feature exploration aimed at different feature maps. The feature refinement layer is used to fuse and refine features. SFFM fuses the features from different modules in a self-adaptive learning manner with softmax function, making full use of hierarchical information with a small amount of parameter cost. Both qualitative and quantitative experiments on benchmark datasets show that LFFN achieves favorable performance against state-of-the-art methods with similar parameters.

In this paper, a novel method that utilizes feature-level fusion of finger vein (FV) and finger dorsal texture (FDT) images is proposed for human identification. Motivated by Weber's law, we present α-trimmed Weber representation (α-TWR) to enhance the foreground lines (FLs), i.e., vessels underneath skin and line-like texture on skin. The proposed α-TWR is robust to illumination variation, as validated by a basic reflective and transmitted imaging model. Cross section asymmetrical coding (CSAC) is performed to extract features for each pixel. The coding value contains discriminative information on the orientation and internal point location of the FLs. The CSAC values of FV and FDT in each point are abreast in terms of binary representation. Local density weighted matching is developed to obtain the matching score between two feature maps. We experimentally show that the proposed method outperforms other unimodal and multimodal identification methods in terms of equal-error-rate.

In finger vein verification, the most important and challenging part is to robustly extract finger vein patterns from low-contrast infrared finger images with limited a priori knowledge. Although recent convolutional neural network (CNN)-based methods for finger vein verification have shown powerful capacity for feature representation and promising perspective in this area, they still have two critical issues to address. First, these CNN-based methods unexceptionally utilize fully connected layers, which restrict the size of finger vein images to process and increase the processing time. Second, the capacity of CNN for feature representation generally suffers from the low quality of finger vein ground-truth pattern maps for training, particularly due to outliers and vessel breaks. To address these issues, in this paper, we propose a novel approach termed FV-GAN to finger vein extraction and verification, based on generative adversarial network (GAN), as the first attempt in this area. Unlike the CNN-based methods, FV-GAN learns from the joint distribution of finger vein images and pattern maps rather than the direct mapping between them, with the aim at achieving stronger robustness against outliers and vessel breaks. Moreover, FV-GAN adopts fully convolutional networks as the basic architecture and discards fully connected layers, which relaxes the constraint on the input image size and reduces the computational expenditure for feature extraction. Furthermore, we design an adversarial training strategy and propose a hybrid loss function for FV-GAN. The experimental results on two public databases show significant improvement by FV-GAN in finger vein verification in terms of both verification accuracy and equal error rate.

Single image super-resolution (SISR) is a notoriously challenging ill-posed problem that aims to obtain a high-resolution output from one of its low-resolution versions. Recently, powerful deep learning algorithms have been applied to SISR and have achieved state-of-the-art performance. In this survey, we review representative deep learning-based SISR methods and group them into two categories according to their contributions to two essential aspects of SISR: The exploration of efficient neural network architectures for SISR and the development of effective optimization objectives for deep SISR learning. For each category, a baseline is first established, and several critical limitations of the baseline are summarized. Then, representative works on overcoming these limitations are presented based on their original content, as well as our critical exposition and analyses, and relevant comparisons are conducted from a variety of perspectives. Finally, we conclude this review with some current challenges and future trends in SISR that leverage deep learning algorithms.

This paper presents a new full-reference method to assess the quality of contrast changed images. In this method, we employ a linear model to describe the relationship between local patches of reference images and contrast changed images. With parameters of this model, three quality measures considering contrast comparison, structure variation, and luminance change are defined. Among them, the first measure produces larger quality scores for higher contrast, which is different from traditional forms of quality measures used in most existing full-reference methods. Experiments on four benchmark databases show that the proposed method is superior to state-of-the-art methods in assessing the quality of contrast changed images.

The key point of face recognition is creating a discriminative feature representation to ensure intraclass compactness and interclass separability. Softmax loss is widely used in deep learning networks, but it is indirect for face verification. Center loss is effective to improve intraclass compactness, while interclass distances are ignored. In this letter, we propose a novel loss function, termed margin loss, to enlarge distances of interclass and reduce intraclass variations simultaneously. Margin loss aims to focus on samples hard to classify by a distance margin. Different from Softmax loss, margin loss is based on Euclidean distances that can directly measure face similarity. Experiments on different datasets have demonstrated the effectiveness of our method.

Face images captured by surveillance videos usually have limited resolution. Due to resolution mismatch, it is hard to match high-resolution (HR) faces with low-resolution (LR) faces directly. Recently, multidimensional scaling (MDS) has been employed to solve the problem. In this letter, we proposed a more discriminative MDS method to learn a mapping matrix, which projects the HR images and LR images to a common subspace. Our method is discriminative since both interclass distances and intraclass distances are taken into consideration. We add an interclass constraint to enlarge the distances of different subjects in the subspace to ensure discriminability. Besides, we consider not only the relationship of HR-LR images, but also the relationship of HR-HR images and LR-LR images in order to preserve local consistency. Experimental results on FERET, Multi-PIE, and SCface databases demonstrate the effectiveness of our proposed approach.

This paper presents an adaptive anchor-point selection method for single image super-resolution (SR), which is based upon internal example-based SR model via locality constrained anchored neighborhood regression. The anchor points are fixed in anchored SR methods and are not flexible and customized for different input low-resolution (LR) images. To overcome this defect, we adaptively select anchor points via constructing customized training set for different input LR images, which can be realized by an internal example-based SR method. We introduce a locality-constrained anchored neighborhood regression to learn the relationship between LR space and high-resolution (HR) space. Extensive experimental results demonstrate that the performance of proposed method is competitive with several state-of-the-art SR methods.

While recently published face alignment algorithms mainly focused on occlusion, low image quality, and complex head poses, subtle variances of facial components were often overlooked. In this correspondence paper, we propose a new approach called cascaded elastically progressive model aiming for pixel-wise landmark localization. First of all, elastically progressive model (EPM) is designed to synthesize the prior knowledge of face shape and appearance of test image. More specifically, a novel framework referred to as inherent linear structure (ILS) is explored for capturing the characteristics of the shape, which is more plastic and flexible than extensively used principle component analysis-based modeling. A locally linear support vector machine (LL-SVM) is used as local expert for searching candidate feature points. In order to optimally integrate ILS with localization results of LL-SVM, we introduce Kalman filter (KF) to dynamically estimate the true shape in the sense of least mean square error. Two schemes are utilized based on our modeling of KF. First, we embedded heuristic line-like search strategy into the framework to guarantee and accelerate the convergence. Second, Kalman gain is manipulated adaptively in accordance with the confidence of the localizers so that poorly localized points are more subject to global constraint than well localized ones. To further improve robustness to initializations, two EPMs are cascaded, in which primary EPM detects the global structure and secondary EPM captures the details. Validation experiments are conducted on in-the-wild LFPW and HELEN databases. Our method shows advantages for accurate landmark localization compared with prevailing methods.

In this paper, we present a new learning-based single-image super-resolution (SR) approach, inspired by existing sparse representation-based methods. As a promising image modeling theory, sparse representation has been effectively applied to solve the image SR problem, usually with the use of pretrained coupled or semi-coupled dictionaries. In our proposed method, we train independent dictionaries for high-resolution (HR) and low-resolution (LR) image patches to endow them more flexibility of expression. We use local subdictionaries to adaptively code image patches, which can characterize image local structures better and ensure the sparsity property of the image. Furthermore, we use kernel regression to relate HR and LR coding coefficients to capture and map the intrinsic nonlinear relationship between them. Such mapping is of central importance in the image SR problem, because high-order statistics play a significant role in the reconstruction of the detail structure of an HR image. The proposed model is generic for image SR in terms of two categories of blurring kernel. Experimental results show that our method can effectively reconstruct image details and outperform state-of-the-art algorithms in both quantitative and visual comparisons.

Saliency detection has been a significant problem in computer vision and helpful to object detection. In this paper, we propose a new computational saliency detection model under the Bayesian framework. First, central bias and the reweighting of the salient regions in the convex hull are applied to guide the prior map. Then, multi-scale for superpixels is proposed to detect objects with various scales. At last, the Bayes formula is adopted to obtain the final saliency map. Experimental results on a standard database show that the proposed model outperforms state-of-the-art methods.

The detection and localization of abnormal activities are considered in this work. An efficient approach called oriented G-MM(OGMM) is proposed. The approach uses optical flow as low-level feature and quantizes the orientation of optical flow into 8 sections. In training stage, the approach will learn a GMM model at each orientation section and each position. In testing stage, the proposed approach estimates the probability of whether a position is abnormal using likelihood method. The proposed approach is a local method and can detect and locate anomaly. What's more, in the proposed approach, the same process is done to each position with little interaction between different positions. This makes the approach suit for parallel computing and can deal with large-scale tasks in Big Data times. The experiments verify that the proposed approach is efficient and effective.

In this paper, we present a novel self-learning single image super-resolution (SR) method, which restores a high-resolution (HR) image from self-examples extracted from the low-resolution (LR) input image itself without relying on extra external training images. In the proposed method, we directly use sampled image patches as the anchor points, and then learn multiple linear mapping functions based on anchored neighborhood regression to transform LR space into HR space. Moreover, we utilize the flipped and rotated versions of the self-examples to expand the internal patch space. Experimental comparison on standard benchmarks with state-of-the-art methods validates the effectiveness of the proposed approach.

In this paper, we propose Two-stage Patch-based Sparse Multi-value Descriptor (TPSMD), a generalization of Sparse Linear Regression Binary method. The TPSMD makes two contributions. First, the multi-value strategy introduces user-specified parameters to improve the binarization, which makes our method more discriminant and less sensitive to noise. The multi-value strategy is a comprise between the simplification and discrimination. Second, the two-stage patch-based strategy contains two independent patch-segmentations for the face image. In the first stage, according to the Multi-value strategy we obtain the discriminative local descriptor based on small patches. In the second stage, we calculate weights for larger patches, and the discriminative face regions, such as eyes and month, are strengthened by the weights. The Two-stage strategy considers local similarity in the first stage and global differences in the second one. Extensive experiments on Extended Yale B and FERET show that our method outperforms state-of-the-art methods.

In this paper, we present a unified frame based on collaborative representation (CR) for single-image super-resolution (SR), which learns low-resolution (LR) and high-resolution (HR) dictionaries independently in the training stage and adopts a consistent coding scheme (CCS) to guarantee the prediction accuracy of HR coding coefficients during SR reconstruction. The independent LR and HR dictionaries are learned based on CR with l2-norm regularization, which can well describe the corresponding LR and HR patch space, respectively. Furthermore, a mapping function is learned to map LR coding coefficients onto the corresponding HR coding coefficients. Propagation filtering can achieve smoothing over an image while preserving image context like edges or textural regions. Moreover, to preserve the edge structures of a super-resolved image and suppress artifacts, a propagation filtering-based constraint and image nonlocal self-similarity regularization are introduced into the SR reconstruction framework. Experimental comparison with state-of-the-art single image SR algorithms validates the effectiveness of proposed approach.

In this letter, we propose a novel approach for single-image super-resolution (SR). Our method is based on the idea of learning a dictionary which can capture the high-order statistics of high-resolution (HR) images. It is of central importance in image SR application, since the high-order statistics play a significant role in the reconstruction of HR image structure. Kernel principal component analysis (KPCA) is adopted to learn such a dictionary. A compact solution is adopted to reduce the time complexity of learning and testing for KPCA. Meanwhile, kernel ridge regression is employed to connect the input low-resolution (LR) image patches with the HR coding coefficients. Experimental results show that the proposed method is effective and efficient in comparison with state-of-art algorithms.

In this paper, we present a sparse coding (SC) inspired method to reconstruct a high-resolution (HR) image from one single low-resolution (LR) image. Instead of restricting the coding coefficients of LR and HR image patches to be equal or linearly mapped, we introduce kernel regression to nonlinearly relate the coding coefficients of LR patches and those of corresponding HR ones in an implicit fashion. Meanwhile, principal component analysis (PCA) is employed to train independent dictionaries which can well express image geometrical structure and ensure image sparse property. Experimental results show that the proposed method can effectively reconstruct image details and outperforms state-of-the-art algorithms in both quantitative and visual comparisons.

This paper studies offline text-independent writer identification of Chinese handwriting. The Bag of Features method is adopted for Chinese writer identification and performs much better than previous state-of-the-art methods. The feature adopted is scale invariant transform feature (SIFT) descriptor for it can extract local directional information from Chinese characters. Instead of Hard Voting, we use two newly devised coding strategies: Improved Fisher Kernels and Locality-constrained Linear Coding, to encode each SIFT descriptor. To make these coding strategies suitable to this new application area, absolute average pooling function is utilized. At last the K-nearest-neighbor classifier is used to identify the author of a handwriting image. Experimental results are conducted on a newly collected dataset of Chinese handwriting, CASIA Offline DB 2.1. Experimental results show our approach not only outperforms previous state-of-the-art methods, but also the traditional Bag of Word method using Hard Voting.

In this paper, we present a new approach to single image super-resolution (SR). The basic idea is to learn a dictionary which can capture the high-order statistics of high-resolution (HR) images. This is of central importance in image SR application, since the high-order statistics play a significant role in the reconstruction of HR image structure. Kernel principal component analysis (KPCA) is used to learn such a dictionary. To reduce the time complexity of learning and testing for KPCA, a sparse solution is adopted. Meanwhile, kernel ridge regression is employed to relate the input low-resolution (LR) image patches and the HR coding coefficients. Experimental results show that the proposed method can effectively reconstruct image details and outperform state-of-the-art algorithms in both quantitative and visual comparisons.

A new method for air quality evaluation using only visible image analysis is introduced in this paper. Based on the fact that suspended particles in air are visible, we attempted to use visible images to develop an appropriate measure which can be closely related to the density of suspended particles (namely the values of PM2.5 and PM10). Furthermore, using this measure, we can evaluate the values of PM2.5 and PM10 via digital image processing. Combined with water droplets, suspended particles in air can form fog or haze. Based on the monochrome atmospheric scattering model, which has been widely used to describe the formation of a haze image, we propose a measure, normalized first-order absolute sum of high-frequency spectrum (NFAS) and attempt to investigate its relationship with the values of PM2.5 and PM10. The experimental results showed the proposed measure is closely related to PM2.5 and has a relation with PM10.

In this paper, we present a new method to reconstruct a high-resolution (HR) face image from a low-resolution (LR) observation. Inspired by position-patch based face hallucination approach, we design position-based dictionaries to code image patches, and recovery HR patch using the coding coefficients as reconstruction weights. In order to capture nonlinear similarity of face features, we implicitly map the data into a high dimensional feature space. By applying kernel principal analysis (KPCA) on the mapped data in the high dimensional feature space, we can obtain reconstruction coefficients in a reduced subspace. Experimental results show that the proposed method can effectively reconstruct details of face images and outperform state-of-the-art algorithms in both quantitative and visual comparisons.

In this paper, we consider the problem of reconstructing a 3D model from a set of pictures taken from calibrated and arbitrarily placed cameras. Our method is based on existing space carving algorithm which considers photo hull as the final result. Our goal is to solve the visibility problem during carving the visual hull. A new concept Discrete Viewing Edge (DVE) is proposed to represent the visual hull instead of a 3D array. DVE is based on voxels and is simple but effective. With models represented by DVEs, we present a surface extraction algorithm and a carve procedure, during which the visibility of a voxel can be determined rapidly and easily. Our way of determining the visibility of a voxel is global, i.e., we take all possible cameras to which this voxel is visible into account. We apply our method to a set of synthetic pictures and provide arbitrary views of target model which are different from existing cameras.

In this paper, we propose an effective water droplets segmentation algorithm based on HSV color space and watershed method. Water droplets segmentation is the key issue to design an automatic hydrophobicity classification algorithm, and the challenge is two-fold: highlight spots on water droplets and the transparency of water. By decomposing the color images into HSV color space, water droplets are easy to be separated in the saturation channel, and watershed method is incorporated to reduce the side-effect of highlight spots. Experimental results on real images demonstrate the advantage of our proposed method.

In this paper, we present a multimodal personal identification system using finger vein and finger dorsal images with their fusion applied at the feature level. A scheme which combines the registration of image pairs with the region-of-interest (ROI) segmentation, is explored on simultaneously captured finger ventral vein and finger dorsal images. We developed a “Comparative Competitive Coding” (C2Code) fusion scheme. It is capable of discarding undesired information in unimodal feature extraction stage. And only discriminative information can be preserved. Furthermore, the C2Code contains new feature of junction points from the finger vein and finger dorsal image pairs. Experimentally, we establish a dataset of finger vein and finger dorsal images. Comparing the performance of proposed fusion scheme with unimodal methods, higher identification accuracy and lower Equal-Error-Rate (EER) are achieved.

In this paper, a coarse-to-fine framework is proposed to register accurately the local regions of interest (ROIs) of images with independent perspective motions by estimating their deformation parameters. A coarse registration approach based on control points (CPs) is presented to obtain the initial perspective parameters. This approach exploits two constraints to solve the problem with a very limited number of CPs. One is named the point-point-line topology constraint, and the other is named the color and intensity distribution of segment constraint. Both of the constraints describe the consistency between the reference and sensed images. To obtain a finer registration, we have converted the perspective deformation into affine deformations in local image patches so that affine refinements can be used readily. Then, the local affine parameters that have been refined are utilized to recover precise perspective parameters of a ROI. Moreover, the location and dimension selections of local image patches are discussed by mathematical demonstrations to avoid the aperture effect. Experiments on simulated data and real-world sequences demonstrate the accuracy and the robustness of the proposed method. The experimental results of image super-resolution are also provided, which show a possible practical application of our method.

In this paper, we propose a new interpolation-based method of image super-resolution reconstruction. The idea is using multisurface fitting to take full advantage of spatial structure information. Each site of low-resolution pixels is fitted with one surface, and the final estimation is made by fusing the multisampling values on these surfaces in the maximum a posteriori fashion. With this method, the reconstructed high-resolution images preserve image details effectively without any hypothesis on image prior. Furthermore, we extend our method to a more general noise model. Experimental results on the simulated and real-world data show the superiority of the proposed method in both quantitative and visual comparisons.

Weber's law suggests that for a stimulus, the ratio between the smallest perceptual change and the background is a constant, which implies stimuli are perceived not in absolute terms but in relative terms. Inspired from this, we exploit and analyze a novel illumination insensitive representation of face images under varying illuminations via a ratio image, called “Weber-face,” where a ratio between local intensity variation and the background is computed. Experimental results on both CMU-PIE and Yale B face databases show that Weber-face performs better than the existing representative approaches.

Personal authentication has attracted great attention due to its large potential of security application, and many researches have shown that fusion of features or decisions obtained from various single-modal biometrics verification systems can enhance the overall performance of system. In this paper, we proposed a novel multimodal biometric approach fusing finger vein pattern with finger-dorsa texture. Firstly, Finger Vein image and finger-dorsa image from the same finger are captured simultaneously, and a method is designed to segment Regions Of Interest(ROI) of vein image and dorsal image. Secondly, two strategies are designed to extract finger vein pattern and finger-dorsa texture respectively. Vein extraction strategy consists of four steps: local thresholding, modified line tracking, thorough probability map creating and directional neighbor analysis. Gray normalization is performed on finger-dorsa image to extract main finger-dorsa texture. Thirdly, the binarized vein pattern and normalized dorsal texture are fused into one feature image. Finally, a block-based texture feature is proposed for personal authentication. Experimental results showed that the proposed fusion method outperforms any one of finger-dorsa and finger vein methods.

We present a video object extraction algorithm combining image partition with motion detection. It consists of three stages: spatial image partition (SIP), temporal motion detection (TMD) and spatiotemporal projection (STP). Firstly, we partition the current video frame into a series of image regions by a modified watershed transform with double open-closing reconstruction and non-linear pixel classification, and the corresponding region boundaries are achieved. In the next stage, global motion estimation and compensation (GME & GMC) are performed, then motion detection based on Gaussianity test is applied to pixels on region boundaries, not to all pixels belonging to regions. In final stage, motion mask is projected to the current frame to segmenting video objects. Preliminary simulation results demonstrate the performance of the proposed algorithm.