EDBT 2026 Demo / reviewers in the wild / expert
Donghao Luo 0001
dblp:202/1915
· DBLP profile ↗
27ranked-venue papers
2as first author
22since 2021 · last 2026
0009-0003-9937-7181ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Graphics, computer vision, multimedia, augmented reality and games · 25 · 2 first-author · 20 since 2021Artificial intelligence and machine learning · 23 · 2 first-author · 18 since 2021Applied, interdisciplinary, general and emerging computing · 1 · 1 since 2021
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2026 | SwiftVideo: A Unified Framework for Few-Step Video Generation Through Trajectory-Distribution AlignmentabstractDiffusion-based or flow-based models have achieved significant progress in video synthesis but require multiple iterative sampling steps, which incurs substantial computational overhead. While many distillation methods that are solely based on trajectory-preserving or distribution-matching have been developed to accelerate video generation models, these approaches often suffer from performance breakdown or increased artifacts in few-step settings. To address these limitations, we propose SwiftVideo, a unified and stable distillation framework that combines the advantages of trajectory-preserving and distribution-matching strategies. Our approach introduces continuous-time consistency distillation to ensure precise preservation of ODE trajectories. Subsequently, We propose a dual-perspective alignment encompassing distribution alignment between synthetic and real data along with trajectory alignment across different inference steps. Our method maintains high-quality video generation while substantially reducing the number of inference steps. Quantitative evaluations on the OpenVid-1M benchmark demonstrate that our method significantly outperforms existing approaches in few-step video generation. Yanxiao Sun, Jiafu Wu, Yun Cao 0002, Chengming Xu 0001, Yabiao Wang, Weijian Cao, Donghao Luo 0001, Chengjie Wang 0001, Yanwei Fu 0001 |
AAAI | 7 |
| 2025 | Unveil Inversion and Invariance in Flow Transformer for Versatile Image EditingabstractLeveraging the large generative prior of the flow transformer for tuning-free image editing requires authentic inversion to project the image into the model’s domain and a flexible invariance control mechanism to preserve non-target contents. However, the prevailing diffusion inversion performs deficiently in flow-based models, and the invariance control cannot reconcile diverse rigid and non-rigid editing tasks. To address these, we systematically analyze the inversion and invariance control based on the flow transformer. Specifically, we unveil that the Euler inversion shares a similar structure to DDIM yet is more susceptible to the approximation error. Thus, we propose a two-stage inversion to first refine the velocity estimation and then compensate for the leftover error, which pivots closely to the model prior and benefits editing. Meanwhile, we propose the invariance control that manipulates the text features within the adaptive layer normalization, connecting the changes in the text prompt to image semantics. This mechanism can simultaneously preserve the non-target contents while allowing rigid and non-rigid manipulation, enabling a wide range of editing types such as visual text, quantity, facial expression, etc. Experiments on versatile scenarios validate that our framework achieves flexible and accurate editing, unlocking the potential of the flow transformer for versatile image editing. Project Page is here. Pengcheng Xu 0008, Boyuan Jiang, Xiaobin Hu, Donghao Luo 0001, Qingdong He, Jiangning Zhang, Chengjie Wang 0001, Yunsheng Wu, Charles Ling 0001, Boyu Wang 0004 |
CVPR | 4 |
| 2025 | Sonic: Shifting Focus to Global Audio Perception in Portrait AnimationabstractThe study of talking face generation mainly explores the intricacies of synchronizing facial movements and crafting visually appealing, temporally-coherent animations. However, due to the limited exploration of global audio perception, current approaches predominantly employ auxiliary visual and spatial knowledge to stabilize the movements, which often results in the deterioration of the naturalness and temporal inconsistencies. Considering the essence of audio-driven animation, the audio signal serves as the ideal and unique priors to adjust facial expressions and lip movements, without resorting to interference of any visual signals. Based on this motivation, we propose a novel paradigm, dubbed as Sonic, to shift focus on the exploration of global audio perception. To effectively leverage global audio knowledge, we disentangle it into intra-and inter-clip audio perception and collaborate with both aspects to enhance overall perception. For the intra-clip audio perception, 1). Context-enhanced audio learning, in which long-range intra-clip temporal audio knowledge is extracted to provide facial expression and lip motion priors implicitly expressed as the tone and speed of speech. 2). Motion-decoupled controller, in which the motion of the head and expression movement are disentangled and independently controlled by intra-audio clips. Most importantly, for inter-clip audio perception, as a bridge to connect the intra-clips to achieve the global perception, Time-aware position shift fusion, in which the global inter-clip audio information is considered and fused for long-audio inference via through consecutively time-aware shifted windows. Extensive experiments demonstrate that the novel audio-driven paradigm outperform existing SOTA methodologies in terms of video quality, temporally consistency, lip synchronization precision, and motion diversity. Xiaozhong Ji, Xiaobin Hu, Chuming Lin, Qingdong He, Jiangning Zhang, Donghao Luo 0001, Qin Lin 0003, Qinglin Lu, Chengjie Wang 0001 |
CVPR | 8 |
| 2025 | CustAny: Customizing Anything from A Single ExampleabstractRecent advances in diffusion-based text-to-image models have simplified creating high-fidelity images, but preserving the identity (ID) of specific elements, like a personal dog, is still challenging. Object customization, using reference images and textual descriptions, is key to addressing this issue. Current object customization methods are either object-specific, requiring extensive fine-tuning, or object-agnostic, offering zero-shot customization but limited to specialized domains. The primary issue of promoting zero-shot object customization from specific domains to the general domain is to establish a large-scale general ID dataset for model pre-training, which is time-consuming and labor-intensive. In this paper, we propose a novel pipeline to construct a large dataset of general objects and build the Multi-Category ID-Consistent (MC-IDC) dataset, featuring 315k text-image samples across 10k categories. With the help of MC-IDC, we introduce Customizing Anything (CustAny), a zero-shot framework that maintains ID fidelity and supports flexible text editing for general objects. CustAny features three key components: a general ID extraction module, a dual-level ID injection module, and an ID-aware decoupling module, allowing it to customize any object from a single reference image and text prompt. Experiments demonstrate that CustAny outperforms existing methods in both general object customization and specialized domains like human customization and virtual try-on. Our contributions include a large-scale dataset, the CustAny framework and novel ID processing to advance this field. The official project page is in https://lingjiekong-fdu.github.io. Lingjie Kong, Chengming Xu 0001, Xiaobin Hu, Wenhui Han, Jinlong Peng, Donghao Luo 0001, Mengtian Li 0002, Jiangning Zhang, Chengjie Wang 0001, Yanwei Fu 0001 |
CVPR | 7 |
| 2025 | DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision RecognitionabstractRecently, Vision Graph Neural Network (ViG) has gained considerable attention in computer vision. Despite its groundbreaking innovation, Vision Graph Neural Network encounters key issues including the quadratic computational complexity caused by its K-Nearest Neighbor (KNN) graph construction and the limitation of pairwise relations of normal graphs. To address the aforementioned challenges, we propose a novel vision architecture, termed Dilated Vision HyperGraph Neural Network (DVHGNN), which is designed to leverage multi-scale hypergraph to efficiently capture high-order correlations among objects. Specifically, the proposed method tailors Clustering and Dilated HyperGraph Construction (DHGC) to adaptively capture multi-scale dependencies among the data samples. Furthermore, a dynamic hypergraph convolution mechanism is proposed to facilitate adaptive feature exchange and fusion at the hypergraph level. Extensive qualitative and quantitative evaluations of the benchmark image datasets demonstrate that the proposed DVHGNN significantly outperforms the state-of-the-art vision backbones. For instance, our DVHGNN-S achieves an impressive top-1 accuracy of 83.1% on ImageNet-1K, surpassing ViG-S by +1.0↑ and ViHGNN-S by +0.6↑. Caoshuo Li, Tanzhe Li, Xiaobin Hu, Donghao Luo 0001, Taisong Jin |
CVPR | 4 |
| 2025 | VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors EmbeddingabstractAlthough diffusion-based image virtual try-on has made considerable progress, emerging approaches still struggle to effectively address the issue of hand occlusion (i.e., clothing regions occluded by the hand part), leading to a notable degradation of the try-on performance. To tackle this issue widely existing in real-world scenarios, we propose VTON-HandFit, leveraging the power of hand priors to reconstruct the appearance and structure for hand occlusion cases. Firstly, we tailor a Hand-Pose Aggregation Net using the ControlNet-based structure explicitly and adaptively encoding the global hand and pose priors. Besides, to fully exploit the hand-related structure and appearance information, we propose Hand-feature Disentanglement Embedding module to disentangle the hand priors into the hand structure-parametric and visual-appearance features, and customize a masked cross attention for further decoupled feature embedding. Lastly, we customize a hand-canny constraint loss to better learn the structure edge knowledge from the hand template of model image. VTON-HandFit outperforms the baselines in qualitative and quantitative evaluations on the public dataset and our self-collected hand-occlusion Handfit-3K dataset particularly for the arbitrary hand pose occlusion cases in real-world scenarios. Our project page is at: https://vton-handfit.github.io. Yujie Liang, Xiaobin Hu, Boyuan Jiang, Donghao Luo 0001, Chengming Xu 0001, Wenhui Han, Taisong Jin, Chengjie Wang 0001, Rongrong Ji |
CVPR | 4 |
| 2025 | OracleFusion: Assisting the Decipherment of Oracle Bone Script with Structurally Constrained Semantic TypographyabstractAs one of the earliest ancient languages, Oracle Bone Script (OBS) encapsulates the cultural records and intellectual expressions of ancient civilizations. Despite the discovery of approximately 4,500 OBS characters, only about 1,600 have been deciphered. The remaining undeciphered ones, with their complex structure and abstract imagery, pose significant challenges for interpretation. To address these challenges, this paper proposes a novel two-stage semantic typography framework, named OracleFusion. In the first stage, this approach leverages the Multimodal Large Language Model (MLLM) with enhanced Spatial Awareness Reasoning (SAR) to analyze the glyph structure of the OBS character and perform visual localization of key components. In the second stage, we introduce Oracle Structural Vector Fusion (OSVF), incorporating glyph structure constraints and glyph maintenance constraints to ensure the accurate generation of semantically enriched vector fonts. This approach preserves the objective integrity of the glyph structure, offering visually enhanced representations that assist experts in deciphering OBS. Extensive qualitative and quantitative experiments demonstrate that OracleFusion outperforms state-of-the-art baseline models in terms of semantics, visual appeal, and glyph maintenance, significantly enhancing both readability and aesthetic quality. Furthermore, OracleFusion provides expert-like insights on unseen oracle characters, making it a valuable tool for advancing the decipherment of OBS. Caoshuo Li, Zengmao Ding, Xiaobin Hu, Bang Li, Donghao Luo 0001, AndyPian Wu, Chengjie Wang 0001, Taisong Jin, SevenShu, Yunsheng Wu, Yongge Liu, Rongrong Ji |
ICCV | 5 |
| 2025 | CrossVTON: Mimicking the Logic Reasoning on Cross-Category Virtual Try-On Guided by Tri-Zone PriorsabstractDespite remarkable progress in image-based virtual try-on systems, generating realistic and robust fitting images for cross-category virtual try-on remains a challenging task. The primary difficulty arises from the absence of human-like reasoning, which involves addressing size mismatches between garments and models while recognizing and leveraging the distinct functionalities of various regions within the model images. To address this issue, we draw inspiration from human cognitive processes and disentangle the complex reasoning required for cross-category try-on into a structured framework. This framework systematically decomposes the model image into three distinct regions: try-on, reconstruction, and imagination zones. Each zone plays a specific role in accommodating the garment and facilitating realistic synthesis. To endow the model with robust reasoning capabilities for cross-category scenarios, we propose an iterative data constructor. This constructor encompasses diverse scenarios, including intra-category try-on, any-to-dress transformations (replacing any garment category with a dress), and dress-to-any transformations (replacing a dress with another garment category). Utilizing the generated dataset, we introduce a tri-zone priors generator that intelligently predicts the try-on, reconstruction, and imagination zones by analyzing how the input garment is expected to align with the model image. Guided by these tri-zone priors, our proposed method, CrossVTON, achieves state-of-the-art performance, surpassing existing baselines in both qualitative and quantitative evaluations. Notably, it demonstrates superior capability in handling cross-category virtual try-on, meeting the complex demands of real-world applications. Donghao Luo 0001, Yujie Liang, Xiaobin Hu, Boyuan Jiang, Chengming Xu 0001, Taisong Jin, Chengjie Wang 0001, Yanwei Fu 0001 |
IJCAI | 1 |
| 2024 | PortraitBooth: A Versatile Portrait Model for Fast Identity-Preserved PersonalizationabstractRecent advancements in personalized image generation using diffusion models have been noteworthy. However, existing methods suffer from inefficiencies due to the requirement for subject-specific fine-tuning. This computationally intensive process hinders efficient deployment, limiting practical usability. Moreover, these methods often grapple with identity distortion and limited expression diversity. In light of these challenges, we propose Portrait-Booth, an innovative approach designed for high efficiency, robust identity preservation, and expression-editable text-to-image generation, without the need for fine-tuning. Por-traitBooth leverages subject embeddingsfrom aface recognition model for personalized image generation without fine-tuning. It eliminates computational overhead and mitigates identity distortion. The introduced dynamic identity preservation strategy further ensures close resemblance to the original image identity. Moreover, PortraitBooth incorporates emotion-aware cross-attention control for diverse facial expressions in generated images, supporting text-driven expression editing. Its scalability enables efficient and high-quality image creation, including multi-subject generation. Extensive results demonstrate superior performance over other state-of-the-art methods in both single and multiple image generation scenarios. Our project page is at https://portraitbooth.github.io. Boyuan Jiang, Ying Tai, Donghao Luo 0001, Jiangning Zhang, Wei Lin 0004, Taisong Jin, Chengjie Wang 0001, Rongrong Ji |
CVPR | 5 |
| 2024 | DiffuMatting: Synthesizing Arbitrary Objects with Matting-Level Annotation
Xiaobin Hu, Donghao Luo 0001, Xiaozhong Ji, Jinlong Peng, Zhengkai Jiang 0001, Jiangning Zhang, Taisong Jin, Chengjie Wang 0001, Rongrong Ji |
ECCV (68) | 3 |
| 2024 | DF40: Toward Next-Generation Deepfake DetectionabstractWe propose a new comprehensive benchmark to revolutionize the current deepfake detection field to the next generation. Predominantly, existing works identify top-notch detection algorithms and models by adhering to the common practice: training detectors on one specific dataset (e.g., FF++) and testing them on other prevalent deepfake datasets. This protocol is often regarded as a "golden compass" for navigating SoTA detectors. But can these stand-out "winners" be truly applied to tackle the myriad of realistic and diverse deepfakes lurking in the real world? If not, what underlying factors contribute to this gap? In this work, we found the dataset (both train and test) can be the "primary culprit" due to the following: (1) forgery diversity: Deepfake techniques are commonly referred to as both face forgery (face-swapping and face-reenactment) and entire image synthesis (AIGC, especially face). Most existing datasets only contain partial types of them, with limited forgery methods implemented (e.g., 2 swapping and 2 reenactment methods in FF++); (2) forgery realism: The dominated training dataset, FF++, contains out-of-date forgery techniques from the past four years. "Honing skills" on these forgeries makes it difficult to guarantee effective detection generalization toward nowadays' SoTA deepfakes; (3) evaluation protocol: Most detection works perform evaluations on one type, e.g., face-swapping types only, which hinders the development of universal deepfake detectors.To address this dilemma, we construct a highly diverse and large-scale deepfake detection dataset called DF40, which comprises 40 distinct deepfake techniques (10 times larger than FF++). We then conduct comprehensive evaluations using 4 standard evaluation protocols and 8 representative detection methods, resulting in over 2,000 evaluations. Through these evaluations, we provide an extensive analysis from various perspectives, leading to 7 new insightful findings contributing to the field. We also open up 4 valuable yet previously underexplored research questions to inspire future works. We release our dataset, code, and pre-trained weights at https://github.com/YZY-stack/DF40. Zhiyuan Yan 0002, Taiping Yao, Shen Chen 0004, Yandan Zhao, Xinghe Fu, Donghao Luo 0001, Chengjie Wang 0001, Shouhong Ding, Yunsheng Wu, Li Yuan 0007 |
NeurIPS | 7 |
| 2024 | Efficiently Exploiting Spatially Variant Knowledge for Video DeblurringabstractVideo deblurring is a challenging task as the blur is often spatially variant. Existing methods mainly engage in building the spatial-temporal correspondence among the frames. As one of the widely-used frameworks, the long-range temporal propagation usually suffers from the expensive computation cost and error accumulation caused by the numerous connections among temporal frames. Meanwhile, the exploration of spatial-variant information from the neighbor frames is often ignored in video deblurring. To tackle these issues, we tailor an efficient short-range multi-scale framework slimming the long-range propagation and exploiting the most relevant neighbor temporal knowledge. For capturing spatial knowledge, we further propose a spatial feature extractor, named the spatially variant adaptive block, to adaptively generate the location-wise kernel to cater to the spatially variant character of blur. For efficient temporal exploitation, a simple inter-frame shift as a motion compensation is developed to avoid expensive long temporal relevance modeling. Both quantitative and qualitative evaluation results on benchmark datasets demonstrate that the proposed algorithm performs favorably against state-of-the-art methods. Qian Xu 0014, Xiaobin Hu, Donghao Luo 0001, Ying Tai, Chengjie Wang 0001, Yuntao Qian |
IEEE Trans. Circuits Syst. Video Technol. | 3 |
| 2023 | High-Resolution Iterative Feedback Network for Camouflaged Object DetectionabstractSpotting camouflaged objects that are visually assimilated into the background is tricky for both object detection algorithms and humans who are usually confused or cheated by the perfectly intrinsic similarities between the foreground objects and the background surroundings. To tackle this challenge, we aim to extract the high-resolution texture details to avoid the detail degradation that causes blurred vision in edges and boundaries. We introduce a novel HitNet to refine the low-resolution representations by high-resolution features in an iterative feedback manner, essentially a global loop-based connection among the multi-scale resolutions. To design better feedback feature flow and avoid the feature corruption caused by recurrent path, an iterative feedback strategy is proposed to impose more constraints on each feedback connection. Extensive experiments on four challenging datasets demonstrate that our HitNet breaks the performance bottleneck and achieves significant improvements compared with 29 state-of-the-art methods. In addition, to address the data scarcity in camouflaged scenarios, we provide an application example to convert the salient objects to camouflaged objects, thereby generating more camouflaged training samples from the diverse salient object datasets. Code will be made publicly available. Xiaobin Hu, Xuebin Qin, Hang Dai, Wenqi Ren, Donghao Luo 0001, Ying Tai, Ling Shao 0001 |
AAAI | 6 |
| 2023 | High-Resolution GAN Inversion for Degraded Images in Large Diverse DatasetsabstractThe last decades are marked by massive and diverse image data, which shows increasingly high resolution and quality. However, some images we obtained may be corrupted, affecting the perception and the application of downstream tasks. A generic method for generating a high-quality image from the degraded one is in demand. In this paper, we present a novel GAN inversion framework that utilizes the powerful generative ability of StyleGAN-XL for this problem. To ease the inversion challenge with StyleGAN-XL, Clustering \& Regularize Inversion (CRI) is proposed. Specifically, the latent space is firstly divided into finer-grained sub-spaces by clustering. Instead of initializing the inversion with the average latent vector, we approximate a centroid latent vector from the clusters, which generates an image close to the input image. Then, an offset with a regularization term is introduced to keep the inverted latent vector within a certain range. We validate our CRI scheme on multiple restoration tasks (i.e., inpainting, colorization, and super-resolution) of complex natural images, and show preferable quantitative and qualitative results. We further demonstrate our technique is robust in terms of data and different GAN models. To our best knowledge, we are the first to adopt StyleGAN-XL for generating high-quality natural images from diverse degraded inputs. Code is available at https://github.com/Booooooooooo/CRI. Yanbo Wang 0003, Chuming Lin, Donghao Luo 0001, Ying Tai, Zhizhong Zhang 0001, Yuan Xie 0006 |
AAAI | 3 |
| 2023 | Dynamic Frame Interpolation in Wavelet DomainabstractVideo frame interpolation is an important low-level vision task, which can increase frame rate for more fluent visual experience. Existing methods have achieved great success by employing advanced motion models and synthesis networks. However, the spatial redundancy when synthesizing the target frame has not been fully explored, that can result in lots of inefficient computation. On the other hand, the computation compression degree in frame interpolation is highly dependent on both texture distribution and scene motion, which demands to understand the spatial-temporal information of each input frame pair for a better compression degree selection. In this work, we propose a novel two-stage frame interpolation framework termed WaveletVFI to address above problems. It first estimates intermediate optical flow with a lightweight motion perception network, and then a wavelet synthesis network uses flow aligned context features to predict multi-scale wavelet coefficients with sparse convolution for efficient target frame reconstruction, where the sparse valid masks that control computation in each scale are determined by a crucial threshold ratio. Instead of setting a fixed value like previous methods, we find that embedding a classifier in the motion perception network to learn a dynamic threshold for each sample can achieve more computation reduction with almost no loss of accuracy. On the common high resolution and animation frame interpolation benchmarks, proposed WaveletVFI can reduce computation up to 40% while maintaining similar accuracy, making it perform more efficiently against other state-of-the-arts. Lingtong Kong, Boyuan Jiang, Donghao Luo 0001, Wenqing Chu, Ying Tai, Chengjie Wang 0001, Jie Yang 0002 |
IEEE Trans. Image Process. | 3 |
| 2022 | IFRNet: Intermediate Feature Refine Network for Efficient Frame InterpolationabstractPrevailing video frame interpolation algorithms, that generate the intermediate frames from consecutive inputs, typically rely on complex model architectures with heavy parameters or large delay, hindering them from diverse real-time applications. In this work, we devise an efficient encoder-decoder based network, termed IFRNet, for fast in-termediate frame synthesizing. It first extracts pyramid features from given inputs, and then refines the bilateral in-termediate flow fields together with a powerful intermedi-ate feature until generating the desired output. The gradu-ally refined intermediate feature can not only facilitate in-termediate flow estimation, but also compensate for con-textual details, making IFRNet do not need additional syn-thesis or refinement module. To fully release its potential, we further propose a novel task-oriented optical flow dis-tillation loss to focus on learning the useful teacher knowl-edge towards frame synthesizing. Meanwhile, a new ge-ometry consistency regularization term is imposed on the gradually refined intermediate features to keep better structure layout. Experiments on various benchmarks demon-strate the excellent performance and fast inference speed of proposed approaches. Code is available at https://github.com/ltkong218/IFRNet. Lingtong Kong, Boyuan Jiang, Donghao Luo 0001, Wenqing Chu, Ying Tai, Chengjie Wang 0001, Jie Yang 0002 |
CVPR | 3 |
| 2022 | ColorFormer: Image Colorization via Color Memory Assisted Hybrid-Attention Transformer
Xiaozhong Ji, Boyuan Jiang, Donghao Luo 0001, Guangpin Tao, Wenqing Chu, Chengjie Wang 0001, Ying Tai |
ECCV (16) | 3 |
| 2022 | AutoGAN-Synthesizer: Neural Architecture Search for Cross-Modality MRI Synthesis
Xiaobin Hu, Ruolin Shen, Donghao Luo 0001, Ying Tai, Chengjie Wang 0001, Bjoern Menze |
MICCAI (6) | 3 |
| 2022 | Joint Learning Content and Degradation Aware Feature for Blind Super-ResolutionabstractTo achieve promising results on blind image super-resolution (SR),some attempts leveraged the low resolution (LR) images to predict the kernel and improve the SR performance. However, these Supervised Kernel Prediction (SKP) methods are impractical due to the unavailable real-world blur kernels. Although some Unsupervised Degradation Prediction (UDP) methods are proposed to bypass this problem, the inconsistency between degradation embedding and SR feature is still challenging. By exploring the correlations between degradation embedding and SR feature, we observe that jointly learning the content and degradation aware feature is optimal. Based on this observation, a Content and Degradation aware SR Network dubbed CDSR is proposed. Specifically, CDSR contains three newly-established modules: (1) a Lightweight Patch-based Encoder (LPE) is applied to jointly extract content and degradation features; (2) a Domain Query Attention based module (DQA) is employed to adaptively reduce the inconsistency; (3) a Codebook-based Space Compress module (CSC) that can suppress the redundant information. Extensive experiments on several benchmarks demonstrate that the proposed CDSR outperforms the existing UDP models and achieves competitive performance on PSNR and SSIM even compared with the state-of-the-art SKP methods. Chuming Lin, Donghao Luo 0001, Yong Liu 0032, Ying Tai, Chengjie Wang 0001, Mingang Chen |
ACM Multimedia | 3 |
| 2021 | Learning Comprehensive Motion Representation for Action RecognitionabstractFor action recognition learning, 2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing inter-frame connections while still suffering the limited temporal receptive field or high latency. Moreover, the feature enhancement is often only performed by channel or space dimension in action recognition. To address these issues, we first devise a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. The channel gates generated by CME incorporate the information from all the other frames in the video. We further propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps. The intuition is that the change of background is typically slower than the motion area. Both CME and SME have clear physical meaning in capturing action clues. By integrating the two modules into the off-the-shelf 2D network, we finally obtain a Comprehensive Motion Representation (CMR) learning method for action recognition, which achieves competitive performance on Something-Something V1 & V2 and Kinetics-400. On the temporal reasoning datasets Something-Something V1 and V2, our method outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames as input, respectively. Mingyu Wu 0002, Boyuan Jiang, Donghao Luo 0001, Junchi Yan, Yabiao Wang, Ying Tai, Chengjie Wang 0001, Feiyue Huang, Xiaokang Yang 0001 |
AAAI | 3 |
| 2021 | Learning Salient Boundary Feature for Anchor-free Temporal Action LocalizationabstractTemporal action localization is an important yet challenging task in video understanding. Typically, such a task aims at inferring both the action category and localization of the start and end frame for each action instance in a long, untrimmed video. While most current models achieve good results by using pre-defined anchors and numerous actionness, such methods could be bothered with both large number of outputs and heavy tuning of locations and sizes corresponding to different anchors. Instead, anchor-free methods is lighter, getting rid of redundant hyper-parameters, but gains few attention. In this paper, we propose the first purely anchor-free temporal localization method, which is both efficient and effective. Our model includes (i) an end-to-end trainable basic predictor, (ii) a saliency-based refinement module to gather more valuable boundary features for each proposal with a novel boundary pooling, and (iii) several consistency constraints to make sure our model can find the accurate boundary given arbitrary proposals. Extensive experiments show that our method beats all anchor-based and actionness-guided methods with a remarkable margin on THUMOS14, achieving state-of-the-art results, and comparable ones on ActivityNet v1.3. Code is available at https://github.com/TencentYoutuResearch/ActionDetection-AFSD. Chuming Lin, Chengming Xu 0001, Donghao Luo 0001, Yabiao Wang, Ying Tai, Chengjie Wang 0001, Feiyue Huang, Yanwei Fu 0001 |
CVPR | 3 |
| 2021 | Spectrum-to-Kernel Translation for Accurate Blind Image Super-ResolutionabstractDeep-learning based Super-Resolution (SR) methods have exhibited promising performance under non-blind setting where blur kernel is known; however, blur kernels of Low-Resolution (LR) images in different practical applications are usually unknown. It may lead to a significant performance drop when degradation process of training images deviates from that of real images. In this paper, we propose a novel blind SR framework to super-resolve LR images degraded by arbitrary blur kernel with accurate kernel estimation in frequency domain. To our best knowledge, this is the first deep learning method which conducts blur kernel estimation in frequency domain. Specifically, we first demonstrate that feature representation in frequency domain is more conducive for blur kernel reconstruction than in spatial domain. Next, we present a Spectrum-to-Kernel (S$2$K) network to estimate general blur kernels in diverse forms. We use a conditional GAN (CGAN) combined with SR-oriented optimization target to learn the end-to-end translation from degraded images' spectra to unknown kernels. Extensive experiments on both synthetic and real-world images demonstrate that our proposed method sufficiently reduces blur kernel estimation error, thus enables the off-the-shelf non-blind SR methods to work under blind setting effectively, and achieves superior performance over state-of-the-art blind SR methods, averagely by 1.39dB, 0.48dB (Gaussian kernels) and 6.15dB, 4.57dB (motion kernels) for scales $2\times$ and $4\times$ respectively. Guangpin Tao, Xiaozhong Ji, Wenzhuo Wang, Shuo Chen 0003, Chuming Lin, Yun Cao 0002, Tong Lu 0002, Donghao Luo 0001, Ying Tai |
NeurIPS | 8 |
| 2020 | Fast Learning of Temporal Action Proposal via Dense Boundary GeneratorabstractGenerating temporal action proposals remains a very challenging problem, where the main issue lies in predicting precise temporal proposal boundaries and reliable action confidence in long and untrimmed real-world videos. In this paper, we propose an efficient and unified framework to generate temporal action proposals named Dense Boundary Generator (DBG), which draws inspiration from boundary-sensitive methods and implements boundary classification and action completeness regression for densely distributed proposals. In particular, the DBG consists of two modules: Temporal boundary classification (TBC) and Action-aware completeness regression (ACR). The TBC aims to provide two temporal boundary confidence maps by low-level two-stream features, while the ACR is designed to generate an action completeness score map by high-level action-aware features. Moreover, we introduce a dual stream BaseNet (DSB) to encode RGB and optical flow information, which helps to capture discriminative boundary and actionness features. Extensive experiments on popular benchmarks ActivityNet-1.3 and THUMOS14 demonstrate the superiority of DBG over the state-of-the-art proposal generator (e.g., MGG and BMN). Chuming Lin, Jian Li 0062, Yabiao Wang, Ying Tai, Donghao Luo 0001, Zhipeng Cui, Chengjie Wang 0001, Feiyue Huang, Rongrong Ji |
AAAI | 5 |
| 2020 | TEINet: Towards an Efficient Architecture for Video RecognitionabstractEfficiency is an important issue in designing video architectures for action recognition. 3D CNNs have witnessed remarkable progress in action recognition from videos. However, compared with their 2D counterparts, 3D convolutions often introduce a large amount of parameters and cause high computational cost. To relieve this problem, we propose an efficient temporal module, termed as Temporal Enhancement-and-Interaction (TEI Module), which could be plugged into the existing 2D CNNs (denoted by TEINet). The TEI module presents a different paradigm to learn temporal features by decoupling the modeling of channel correlation and temporal interaction. First, it contains a Motion Enhanced Module (MEM) which is to enhance the motion-related features while suppress irrelevant information (e.g., background). Then, it introduces a Temporal Interaction Module (TIM) which supplements the temporal contextual information in a channel-wise manner. This two-stage modeling scheme is not only able to capture temporal structure flexibly and effectively, but also efficient for model inference. We conduct extensive experiments to verify the effectiveness of TEINet on several benchmarks (e.g., Something-Something V1&V2, Kinetics, UCF101 and HMDB51). Our proposed TEINet can achieve a good recognition accuracy on these datasets but still preserve a high efficiency. Zhaoyang Liu 0001, Donghao Luo 0001, Yabiao Wang, Limin Wang 0002, Ying Tai, Chengjie Wang 0001, Feiyue Huang, Tong Lu 0002 |
AAAI | 2 |
| 2020 | Learning by Analogy: Reliable Supervision From Transformations for Unsupervised Optical Flow EstimationabstractUnsupervised learning of optical flow, which leverages the supervision from view synthesis, has emerged as a promising alternative to supervised methods. However, the objective of unsupervised learning is likely to be unreliable in challenging scenes. In this work, we present a framework to use more reliable supervision from transformations. It simply twists the general unsupervised learning pipeline by running another forward pass with transformed data from augmentation, along with using transformed predictions of original data as the self-supervision signal. Besides, we further introduce a lightweight network with multiple frames by a highly-shared flow decoder. Our method consistently gets a leap of performance on several benchmarks with the best accuracy among deep unsupervised methods. Also, our method achieves competitive results to recent fully supervised methods while with much fewer parameters. Liang Liu 0007, Jiangning Zhang, Ruifei He, Yong Liu 0007, Yabiao Wang, Ying Tai, Donghao Luo 0001, Chengjie Wang 0001, Feiyue Huang |
CVPR | 7 |
| 2020 | Temporal Distinct Representation Learning for Action Recognition
Junwu Weng, Donghao Luo 0001, Yabiao Wang, Ying Tai, Chengjie Wang 0001, Feiyue Huang, Xudong Jiang 0001, Junsong Yuan 0001 |
ECCV (7) | 2 |
| 2017 | Image Matching via Loopy RNNabstractMost existing matching algorithms are one-off algorithms, i.e., they usually measure the distance between the two image feature representation vectors for only one time. In contrast, human's vision system achieves this task, i.e., image matching, by recursively looking at specific/related parts of both images and then making the final judgement. Towards this end, we propose a novel loopy recurrent neural network (Loopy RNN), which is capable of aggregating relationship information of two input images in a progressive/iterative manner and outputting the consolidated matching score in the final iteration. A Loopy RNN features two uniqueness. First, built on conventional long short-term memory (LSTM) nodes, it links the output gate of the tail node to the input gate of the head node, thus it brings up symmetry property required for matching. Second, a monotonous loss designed for the proposed network guarantees increasing confidence during the recursive matching process. Extensive experiments on several image matching benchmarks demonstrate the great potential of the proposed method. Donghao Luo 0001, Bingbing Ni, Yichao Yan, Xiaokang Yang 0001 |
IJCAI | 1 |