Relightable portrait animation aims to animate a static reference portrait to match the head movements of a driving video while adapting to user-specified or reference lighting conditions. Existing portrait animation methods fail to achieve relightable portraits because they do not separate and manipulate intrinsic (identity and appearance) and extrinsic (pose and lighting) features. In this paper, we present a Lighting Controllable Video Diffusion model (LCVD) for high-fidelity, relightable portrait animation. We address this limitation by distinguishing these feature types through dedicated subspaces within the feature space of a pre-trained image-to-video diffusion model. Specifically, for each frame, we use the 3D mesh of the reference portrait, the pose of the driving frame as well as the specified lighting to render an image called shading hint. While the reference image represents the intrinsic attributes, the shading hint encodes the extrinsic attributes. In the training phase, we employ a reference adapter to map the reference into an intrinsic feature subspace and a shading adapter to map the shading hints into an extrinsic feature subspace. By merging features from these subspaces, the model achieves nuanced control over lighting and pose in generated animations. Extensive evaluations show that LCVD outperforms state-of-the-art methods in lighting realism, image quality, and video consistency, setting a new benchmark in relightable portrait animation. Our project is available at https://github.com/MingtaoGuo/Relightable-Portrait-Animation

Retrieving target pedestrians from cross-modal images captured by infrared and visible cameras is critical in 24-hour intelligent surveillance. The primary challenge lies in narrowing the modality gap between the visible and infrared modalities. In view of this, existing research tends to extract modality-shared features to bridge the modality gap. However, the extraction process and effectiveness of the shared features are often insufficiently justified. In contrast, we observe that a certain portion of semantics remains invariant across visible and infrared modalities. These invariant semantics provide the basis for extracting modality-shared features. Based on this criterion, we propose a novel method named Low-light Invariant Representation Learning (IRL), which aims to construct an invariant space shared between visible and infrared modalities. Specifically, we introduce a Modality Invariant Extractor, which divides invariance into modality invariance and scale invariance, and extracts the invariant features from different scales and dimensions respectively. Furthermore, a Low-light Representation Enhancement module is designed, which reuses the invariant features and shallow modality features through paired enhancement units and compensation units to highlight cross-modality shared features. Extensive experiments on SYSU-MM01, RegDB, and LLCM benchmarks demonstrate the effectiveness of our method. Code is available https://github.com/Mapzzone/IRL.

Virtual human avatars are essential for applications such as gaming, augmented reality, and virtual production. However, existing methods struggle to achieve high fidelity reconstruction from monocular input while keeping hardware costs low. Many approaches rely on the SMPL body prior and apply vertex offsets to represent clothed avatars. Unfortunately, excessive offsets often cause misalignment and blurred contours, particularly around clothing wrinkles, silhouette boundaries, and facial regions. To address these limitations, we propose a dual branch framework for human avatar reconstruction from monocular video. A lightweight Vertex Align Net (VAN) predicts per-vertex normal direction offsets on the SMPL mesh to achieve coarse geometric alignment and guide Gaussian-based human avatar modeling. In parallel, we construct a high resolution facial Gaussian branch based on FLAME estimated parameters, with facial regions localized via pretrained detectors. The facial and body renderings are fused using a semantic mask to enhance facial clarity and ensure globally consistent avatar appearance. Experiments demonstrate that our method surpasses state of the art approaches in modeling animatable human avatars with fine grained fidelity.

Keypoint detection and matching have garnered significant attention, yet remain challenging in low-light environments. Most current studies follow anenhance-then-detectpipeline, which consists of independent enhancers and detectors. While the enhancer focuses on improving the visual quality of low-light images to satisfy human perception standards, the detector prioritizes detection accuracy for machine vision tasks. The unaligned optimization objectives of the enhancer and detector overlook the gap between human and machine vision and lead to sub-optimal performance in low-light keypoint detection. To tackle this problem, a jointenhance-and-detectpipeline is proposed to unify the optimization objectives of enhancement and detection by regarding the improvement of keypoint detection accuracy with enhanced features under machine vision standards. Specifically, we propose a low-light keypoint detection network named DeRFeat, which learns a degradation-equivariant representation between normal and dark domains using AutoEncoding transformation and domain descriptor similarity constraints to indirectly enhance the features from the encoder in the training stage. Then, DeRFeat guides the shared encoder to obtain the degradation-equivariant representations from dark images in the inference stage. With the dark degradation predictions, the encoder is capable of generating equivariant representations between normal and dark domains. The proposed domain descriptor similarity module further aids the encoder in mitigating the impact of dark degradation factors, enabling local descriptors to acquire undisturbed representations. Moreover, a coarse-to-fine point selection strategy is proposed to provide reliable prior keypoints for a globally optimal descriptor construction. Experimental results on four benchmark datasets demonstrate that the proposed method significantly outperforms state-of-the-art methods under varying low-light conditions.

Inserting an object into a background scene has wide applications in image editing and mixed reality. However, existing methods still struggle to seamlessly adapt the object to the background while maintaining its individual characteristics. In this article, we propose to fine-tune a pre-trained diffusion-based insertion model such that it learns to establish a unique correspondence between a few weights and the target object, given as input few-shot images of an object. A novel individualized feature extraction (IFE) module is designed to extract the individual detail features from few-shot object images. Then, the individual features of the target object, together with the semantic features of the target object and the background context features extracted by the pre-trained image encoders are injected into the cross-attention modules of the latent diffusion model, enabling it to learn the correlation information of the target object and the background scene through the attention mechanism. The weights obtained by fine-tuning implicitly serve as an alternative representation of the target object, with which the object can be easily inserted into any background images. Extensive comparative experiments validate the superiority of the proposed method to the state-of-the-art insertion methods in maintaining the individual details of the inserted object and adapting it to background scenes, including allowing the interaction between the inserted object and the background scene, correctly handling their occlusion relationship, maintaining the consistency of their viewpoints and poses.

It is well-recognized that correctly detecting shadow boundaries is much more difficult than shadow interior pixels in real-world situations due to the rapid intensity and color transitions near shadow boundaries and the complex background textures. This paper presents a novel boundary-aware shadow detection network to enhance the overall detection performance, especially on challenging shadow boundaries. To enable the network aware of shadow boundaries and balance the supervision between image pixels on the boundary and inside of shadows, the shadow mask is decoupled into a body mask and boundary mask. The feature correction module is designed after feature interaction to eliminate various shadow inferences further. Experimental results show that the proposed network outperforms the state-of-the-art shadow detection methods on four shadow detection benchmark datasets.

Driver action recognition aims to identify different driving actions of drivers, which is of great significance for traffic safety monitoring. However, it is still challenging for the existing methods to identify similar actions which are common situations in driving scenarios. The fundamental reason is that they ignore the global features of videos and fail to extract the most discriminative features. To address this issue, we propose a video-based driver action recognition network via deep learning of spatial-temporal and motion features, which includes two important modules: the global spatial-temporal modeling (GSTM) module and the motion-spatial-temporal joint attention (MSTJA) module. GSTM effectively extracts the global spatial-temporal features of driving actions by expanding the equivalent receptive field of the spatial-temporal dimension through spatial-temporal separable convolution and hierarchical residual connection structures. MSTJA forces the network to focus on the most discriminative areas by jointly exciting the motion patterns of subtle variations and significant spatial-temporal features through the extracted dual path motion features and spatial-temporal features. Experiments demonstrate that the proposed network achieves state-of-the-art classification accuracy on two benchmark driver action datasets Drive&Act and SFD3.

Shadow removal is a challenging computer vision and multimedia task that aims to restore image content in shadow regions. The state-of-the-art shadow removal methods introduce artifacts near shadow boundaries or inconsistencies between shadow and nonshadow areas, which can be easily noticed by the human eye at first glance. In this paper, we design a boundary-aware shadow removal network (BA-ShadowNet) that improves shadow removal accuracy by increasing the removal performance at shadow boundaries. In contrast with previously developed methods, which usually consider shadow boundary optimization to be a postprocessing technique, our method performs shadow removal and shadow boundary optimization simultaneously. For this purpose, the proposed BA-ShadowNet is designed as a multiscale encoder-decoder structure, where the decoder consists of a shadow removal branch and a shadow optimization branch. An interaction module is then introduced to fuse and exchange the features of the two branches. This module facilitates the removal branch in perceiving the locations and colors of shadow boundaries. Additionally, it optimizes the boundary branch according to the image context extracted from the removal branch. A three-term loss function is further developed to supervise the shadow removal results and to address the issue of imbalanced supervision between shadow boundary pixels and pixels inside shadows. Extensive experiments conducted on the ISTD+ and SRD datasets demonstrate that the proposed BA-ShadowNet greatly outperforms the state-of-the-art methods with respect to shadow removal.

Abstract In this paper, we propose a novel second‐order explicit midpoint method to address the issue of energy loss and vorticity dissipation in Eulerian fluid simulation. The basic idea is to explicitly compute the pressure gradient at the middle time of each time step and apply it to the velocity field after advection. Theoretically, our solver can achieve higher accuracy than the first‐order solvers at similar computational cost. On the other hand, our method is twice and even faster than the implicit second‐order solvers at the cost of a small loss of accuracy. We have carried out a large number of 2D, 3D and numerical experiments to verify the effectiveness and availability of our algorithm.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this article proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average$F_{measure}$on a self-collected database. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR applications.

Today's mainstream shadow detection methods are manually designed via a case-by-case approach. Accordingly, these methods may only be able to detect shadows for specific scenes. Given the complex and diverse shadow scenes in reality, none of the existing methods can provide a one-size-fits-all solution with satisfactory performance. To address this problem, this paper introduces a new concept, named shadow detection confidence, which can be used to evaluate the effect of any shadow detection method for any given scene. The best detection effect for a scene is achieved by combining prediction results by multiple methods. To measure the shadow detection confidence characteristics of an image, a novel relative confidence map prediction network (RCMPNet) is proposed. Experimental results show that the proposed method outperforms multiple state-of-the-art shadow detection methods on four shadow detection benchmark datasets.

Illumination normalization for face recognition is very important when a face is captured under harsh lighting conditions. Instead of designing hand-crafted features, in this paper we formulate face illumination normalization as an image-to-image translation task. A great challenge of face normalization is that human facial structures are particularly sensitive to image structure distortion, which frequently occurs in traditional image-to-image translation tasks. Unfortunately, sometimes even slight facial structure distortions may prohibit human eyes and machine face recognition methods from identifying face identities. To address this issue, a novel GAN- based network architecture called the asymmetric joint generative adversarial network (AJGAN) is developed to normalize face images under arbitrary illumination conditions, without known face geometry and albedo information. In addition, an illumination normalization GAN $G_1$ and an asymmetric relighting GAN $G_2$ that maps a frontal-illuminated image to images with various lighting conditions are incorporated in AJGAN to maintain personalized facial structures. To avoid image blurring caused by the under-constrained relighting mapping, we introduce a scheme of one-hot lighting labels into $G_2$ and enforce label classification loss. Furthermore, the number of training images starting from a very limited number of labels is dynamically extended by the combination of different lighting labels. Qualitative and quantitative experiments on three databases validate that AJGAN significantly outperforms the state-of-the-art methods.

We propose an automatic framework to recover the illumination of indoor scenes based on a single RGB-D image. Unlike previous works, our method can recover spatially varying illumination without using any lighting capturing devices or HDR information. The recovered illumination can produce realistic rendering results. To model the geometry of the visible and invisible parts of scenes corresponding to the input RGB-D image, we assume that all objects shown in the image are located in a box with six faces and build a planar-based geometry model based on the input depth map. We then present a confidence-scoring based strategy to separate the light sources from the highlight areas. The positions of light sources both in and out of the camera's view are calculated based on the classification result and the recovered geometry model. Finally, an iterative procedure is proposed to calculate the colors of light sources and the materials in the scene. In addition, a data-driven method is used to set constraints on the light source intensities. Using the estimated light sources and geometry model, environment maps at different points in the scene are generated that can model the spatial variance of illumination. The experimental results demonstrate the validity and flexibility of our approach.

We propose a new path planning approach for crowd evacuation in buildings. Based on the crowd distribution at current moment, our approach is capable of predicting congestion at intersections in the near future so that a more reasonable path can be achieved than previous method. We also introduce a stage-wise path planning mechanism to adjust routes to respond to the dynamic changes of crowd distribution. Experimental results on several evacuation scenarios indicate that our method can reduce congestion comparing with previous method so that more reasonable evacuation routes can be achieved and the evacuation time is shortened.

In this paper, we present a data mining based algorithm to automatically locate performance bottlenecks at algorithm level for a complex rendering system. The basic idea is to treat the bottleneck identification problem as a variable importance analysis problem from a large volume of performance data which is generated by collecting the time costs under different combinations of algorithm level parameters. Based on the performance data set, random forest is adopted to conduct the variable importance ranking task. We also note an important fact that there might no performance bottleneck exists in the scope of the whole rendering system, but it is likely that bottlenecks could be found under some specific conditions. Thus we propose a bottleneck analysis tree to split the parameter space into many subspaces in which performance bottlenecks can be identified.

Abstract We propose a novel approach to simulate the illumination of augmented outdoor scene based on a legacy photograph. Unlike previous works which only take surface radiosity or lighting related prior information as the basis of illumination estimation, our method integrates both of these two items. By adopting spherical harmonics, we deduce a linear model with only six illumination parameters. The illumination of an outdoor scene is finally calculated by solving a linear least square problem with the color constraint of the sunlight and the skylight. A high quality environment map is then set up, leading to realistic rendering results. We also explore the problem of shadow casting between real and virtual objects without knowing the geometry of objects which cast shadows. An efficient method is proposed to project complex shadows (such as tree's shadows) on the ground of the real scene to the surface of the virtual object with texture mapping. Finally, we present an unified scheme for image composition of a real outdoor scene with virtual objects ensuring their illumination consistency and shadow consistency. Experiments demonstrate the effectiveness and flexibility of our method.

In augmented reality, one of key tasks to achieve a convincing visual appearance consistency between virtual objects and video scenes is to have a coherent illumination along the whole sequence. As outdoor illumination is largely dependent on the weather, the lighting condition may change from frame to frame. In this paper, we propose a full image-based approach for online tracking of outdoor illumination variations from videos captured with moving cameras. Our key idea is to estimate the relative intensities of sunlight and skylight via a sparse set of planar feature-points extracted from each frame. To address the inevitable feature misalignments, a set of constraints are introduced to select the most reliable ones. Exploiting the spatial and temporal coherence of illumination, the relative intensities of sunlight and skylight are finally estimated by using an optimization process. We validate our technique on a set of real-life videos and show that the results with our estimations are visually coherent along the video sequences.

In augmented reality, consistent illumination plays an important role when integrating a virtual object into a video of real scene. In this paper, we propose a novel image based framework to estimate-on-line the dynamic ally changing illumination parameters of outdoor video sequences captured by a fixed camera. Unlike previous approaches which either request to know the scene geometry or involve huge storage to preserve time dependent basis images or statistic parameters, our approach requires very simple interaction at the initialization stage by a few brushes to select areas with specified surface normal, which are used to calculate the sunlight parameters. An optimization procedure is also applied, ensuring the robustness and precision of our estimation. Experimental results demonstrate the effectiveness and flexibility of the proposed approach.

Abstract Illumination consistency plays an important role in realistic rendering of virtual characters which are integrated into a live video of real scene. This paper proposes a novel method for estimating the illumination conditions of outdoor videos captured by a fixed viewpoint. We first derive an analytical model which relates the statistics of an image to the lighting parameters of the scene adhering to the basic illumination model. Exploiting this model, we then develop a framework to estimate the lighting conditions of live videos. In order to apply the above approach to scenes containing dynamic objects such as intrusive pedestrians and swaying trees, we enforce two constraints, namely spatial and temporal illumination coherence, to refine the solution. Our approach requires no geometric information of the scenes and is sufficient for real‐time performance. Experiments show that with the lighting parameters recovered by our method, virtual characters can be seamlessly integrated into the live video. Copyright © 2010 John Wiley & Sons, Ltd.