Video Diffusion Models (VDMs) have demonstrated remarkable capabilities in synthesizing realistic videos by learning from large-scale data. Although vanilla Low-Rank Adaptation (LoRA) can learn specific spatial or temporal movement to driven VDMs with constrained data, achieving precise control over both camera trajectories and object motion remains challenging due to the unstable fusion and non-linear scalability. To address these issues, we propose LiON-LoRA, a novel framework that rethinks LoRA fusion through three core principles: Linear scalability, Orthogonality, and Norm consistency. First, we analyze the orthogonality of LoRA features in shallow VDM layers, enabling decoupled low-level controllability. Second, norm consistency is enforced across layers to stabilize fusion during complex camera motion combinations. Third, a controllable token is integrated into the diffusion transformer (DiT) to linearly adjust motion amplitudes for both cameras and objects with a modified self-attention mechanism to ensure decoupled control. Additionally, we extend LiON-LoRA to temporal generation by leveraging static-camera videos, unifying spatial and temporal controllability. Experiments demonstrate that LiON-LoRA outperforms state-of-the-art methods in trajectory control accuracy and motion strength adjustment, achieving superior generalization with minimal training data. Project Page: https://fuchengsu.github.io/lionlora.github.io/

Synthesizing 3D content from single image has great potential in many real-world applications. To deal with the inherent ambiguity of single image, existing methods usually leverage pre-trained 2D diffusion models for computational intensive per-instance optimization. Although having been able to create 3D assets in a feed-forward manner, the efficacy of recent advances in 3D foundation models is still limited due to neglecting geometric cues from images. To address this issue, we propose an efficient 3D foundation model named LPM to synthesize 3D content from an image. Like the masked modeling in the 2D image domain, the key of our approach is to learn 3D representations from incomplete visible shapes. By taking advantage of a synthesis-by-analysis paradigm, we establish an efficient pipeline to first estimate the visible portions and then generate the complete 3D representations. Based on the principle that an image is the projection of 3D model, we initially estimate partial 3D voxel features from single image, which are further projected onto orthogonal planes to form an incomplete yet efficient triplane representation. Subsequently, an autoencoder is employed to model a complete triplane representation based on the incomplete parts. We train our model on massive data with over 250 million parameters to enhance its generalization capability. The experimental results show that LPM can generate high-fidelity 3D objects from an image within 0.1 seconds, which is more effective than the existing feed-forward approaches. Our implementation and pre-trained models will be made publicly available.

Deep learning-based multi-view stereo has emerged as a powerful paradigm for reconstructing the complete geometrically-detailed objects from multi-views. Most of the existing approaches only estimate the pixel-wise depth value by minimizing the gap between the predicted point and the intersection of ray and surface, which usually ignore the surface topology. It is essential to the textureless regions and surface boundary that cannot be properly reconstructed. To address this issue, we suggest to take advantage of point-to-surface distance so that the model is able to perceive a wider range of surfaces. To this end, we predict the distance volume from cost volume to estimate the signed distance of points around the surface. Our proposed RA-MVSNet is patch-awared, since the perception range is enhanced by associating hypothetical planes with a patch of surface. Therefore, it could increase the completion of textureless regions and reduce the outliers at the boundary. Moreover, the mesh topologies with fine details can be generated by the introduced distance volume. Comparing to the conventional deep learning-based multi-view stereo methods, our proposed RA-MVSNet approach obtains more complete reconstruction results by taking advantage of signed distance supervision. The experiments on both the DTU and Tanks & Temples datasets demonstrate that our proposed approach achieves the state-of-the-art results.

Although having achieved the promising results on shape and color recovery through self-supervision, the multi-layer perceptrons-based methods usually suffer from heavy computational cost on learning the deep implicit surface representation. Since rendering each pixel requires a forward network inference, it is very computationally intensive to synthesize a whole image. To tackle these challenges, we propose an effective coarse-to-fine approach to recover the textured mesh from multi-views in this paper. Specifically, a differentiable Poisson Solver is employed to represent the object’s shape, which is able to produce topology-agnostic and watertight surfaces. To account for depth information, we optimize the shape geometry by minimizing the differences between the rendered mesh and the predicted depth from multi-view stereo. In contrast to the implicit neural representation on shape and color, we introduce a physically-based inverse rendering scheme to jointly estimate the environment lighting and object’s reflectance, which is able to render the high resolution image at real-time. The texture of reconstructed mesh is interpolated from a learnable dense texture grid. We have conducted the extensive experiments on several multi-view stereo datasets, whose promising results demonstrate the efficacy of our proposed approach. The code is available athttps://github.com/l1346792580123/diff.