In real-world underwater scene reconstruction, there are two inherent challenges: Firstly, failing to account for the impact of water on imaging often results in blurry artifacts in the reconstruction; Secondly, data in real-world scenarios is typically captured from sparse viewpoints, which leads to an under-constrained problem for scene reconstruction. Existing NeRF-based reconstruction methods are limited by high training costs, slow inference speeds, and their failure to account for underwater optical conditions, as well as the sparsity of input data. In contrast, 3D Gaussian Splatting (3DGS) provides a more efficient alternative for reconstructing scenes in the wild. However, it still faces inherent challenges in underwater scene. To this end, we introduce PhD-GS, which integrates physics-based constraints and depth information into the 3DGS framework, thereby enhancing the optimization process for underwater scenes. Specifically, our physics-based constraints incorporate underwater imaging formation into the splatting process, while the depth priors provide geometric regularization to guide the optimization of 3DGS. We validate the proposed method on four real underwater scenes from the SeaThru-NeRF dataset with sparse input views. Experiments demonstrate that our approach achieves state-of-the-art results, with significant improvements, especially compared to existing underwater reconstruction methods.

Unsupervised non-rigid point cloud shape correspondence underpins a multitude of 3D vision tasks, yet itself is non-trivial given the exponential complexity stemming from inter-point degree-of-freedom, i.e., pose transformations. Based on the assumption of local rigidity, one solution for reducing complexity is to decompose the overall shape into independent local regions using Local Reference Frames (LRFs) that are equivariant to SE(3) transformations. However, the focus solely on local structure neglects global geometric contexts, resulting in less distinctive LRFs that lack crucial semantic information necessary for effective matching. Furthermore, such complexity introduces out-of-distribution geometric contexts during inference, thus complicating generalization. To this end, we introduce 1) EquiShape, a novel structure tailored to learn pair-wise LRFs with global structural cues for both spatial and semantic consistency, and 2) LRF-Refine, an optimization strategy generally applicable to LRF-based methods, aimed at addressing the generalization challenges. Specifically, for EquiShape, we employ cross-talk within separate equivariant graph neural networks (Cross-GVP) to build long-range dependencies to compensate for the lack of semantic information in local structure modeling, deducing pair-wise independent SE(3)-equivariant LRF vectors for each point. For LRF-Refine, the optimization adjusts LRFs within specific contexts and knowledge, enhancing the geometric and semantic generalizability of point features. Our overall framework surpasses the state-of-the-art methods by a large margin on three benchmarks. Codes are available at https://github.com/2019EPWL/EquiShape.

Learning policies for multi-entity systems in 3D environments is far more complicated against single-entity scenarios, due to the exponential expansion of the global state space as the number of entities increases. One potential solution of alleviating the exponential complexity is dividing the global space into independent local views that are invariant to transformations including translations and rotations. To this end, this paper proposes Subequivariant Hierarchical Neural Networks (SHNN) to facilitate multi-entity policy learning. In particular, SHNN first dynamically decouples the global space into local entity-level graphs via task assignment. Second, it leverages subequivariant message passing over the local entity-level graphs to devise local reference frames, remarkably compressing the representation redundancy, particularly in gravity-affected environments. Furthermore, to overcome the limitations of existing benchmarks in capturing the subtleties of multi-entity systems under the Euclidean symmetry, we propose the Multi-entity Benchmark (MEBEN), a new suite of environments tailored for exploring a wide range of multi-entity reinforcement learning. Extensive experiments demonstrate significant advancements of SHNN on the proposed benchmarks compared to existing methods. Comprehensive ablations are conducted to verify the indispensability of task assignment and subequivariance.

Learning a shared policy that guides the locomotion of different agents is of core interest in Reinforcement Learning (RL), which leads to the study of morphology-agnostic RL. However, existing benchmarks are highly restrictive in the choice of starting point and target point, constraining the movement of the agents within 2D space. In this work, we propose a novel setup for morphology-agnostic RL, dubbed Subequivariant Graph RL in 3D environments (3D-SGRL). Specifically, we first introduce a new set of more practical yet challenging benchmarks in 3D space that allows the agent to have full Degree-of-Freedoms to explore in arbitrary directions starting from arbitrary configurations. Moreover, to optimize the policy over the enlarged state-action space, we propose to inject geometric symmetry, i.e., subequivariance, into the modeling of the policy and Q-function such that the policy can generalize to all directions, improving exploration efficiency. This goal is achieved by a novel SubEquivariant Transformer (SET) that permits expressive message exchange. Finally, we evaluate the proposed method on the proposed benchmarks, where our method consistently and significantly outperforms existing approaches on single-task, multi-task, and zero-shot generalization scenarios. Extensive ablations are also conducted to verify our design.

Predicting the future trajectory of multi-agent in dynamic scenarios is a crucial part of autonomous driving. Despite fruitful progress, existing methods still suffer from immatu-rity in tackling social interaction modeling, which is essential for trajectory prediction. A key reason is that the previous works only consider the interaction of a single category of agents while ignoring the influence between different types of agents. This paper contends a novel framework LG-LSTM to model local and global social interactions between different types of agents by scenario graph and attention fusion. It is mainly composed of two functional components: 1. LSTM-based interaction architecture consists of LSTM-based en-coding function, graph-aware encoding function, attention-fusion encoding function, and LSTM-based decoding function. 2. Multi-task training, which conducts trajectory pre-diction and multi-agent classification. Extensive experiments on two public autonomous driving benchmarks verify the ef-ficacy of the proposed techniques and achieve superior per-formance against state-of-the-art approaches.

Unsupervised image-to-image translation is a central task in computer vision. Current translation frameworks will abandon the discriminator once the training process is completed. This paper contends a novel role of the discriminator by reusing it for encoding the images of the target domain. The proposed architecture, termed as NICE-GAN, exhibits two advantageous patterns over previous approaches: First, it is more compact since no independent encoding component is required; Second, this plug-in encoder is directly trained by the adversary loss, making it more informative and trained more effectively if a multi-scale discriminator is applied. The main issue in NICE-GAN is the coupling of translation with discrimination along the encoder, which could incur training inconsistency when we play the min-max game via GAN. To tackle this issue, we develop a decoupled training strategy by which the encoder is only trained when maximizing the adversary loss while keeping frozen otherwise. Extensive experiments on four popular benchmarks demonstrate the superior performance of NICE-GAN over state-of-the-art methods in terms of FID, KID, and also human preference. Comprehensive ablation studies are also carried out to isolate the validity of each proposed component. Our codes are available at https://github.com/alpc91/NICE-GAN-pytorch.