Sim-to-real gap has long posed a significant challenge for robot learning in simulation, preventing the deployment of learned models in the real world. Previous work has primarily focused on domain randomization and system identification to mitigate this gap. However, these methods are often limited by the realism of the simulator and graphics engines renderings. In this work, we propose Vid2Sim, a novel framework that effectively bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline with 3D scene reconstruction and simulation. Given a monocular video as input, Vid2Sim can generate photo-realistic and physically interactable 3D simulation environments to enable the reinforcement learning of visual navigation agents in complex urban environments. Extensive experiments demonstrate that Vid2Sim significantly improves the performance of urban navigation in the digital twins and real world by 31.2% and 68.3% in success rate compared with agents trained with prior simulation methods.

Account alignment plays a significant role in various domains such as identity recognition, security monitoring, and public opinion analysis. However, existing account alignment methods largely rely on knowledge graphs, which is very costly. More importantly, existing methods do not support conditionconstrained account alignment, thereby limiting practical applicability. In this paper, we propose CondAlign, an account alignment method under conditional constraints. The method first employs a large language model (LLM)-based conditional semantic extraction technique to identify key semantic features that meet the specified constraints; then, a filtering mechanism is proposed to generate a set of highly relevant candidate accounts; finally, a large language model (LLM)-based conditional constraint alignment determination is used to perform semantic verification candidate accounts. Experimental results on multiple real-world account datasets verify that CondAlign significantly outperforms existing baseline methods.

Autonomous driving simulation system plays a crucial role in enhancing self-driving data and simulating complex and rare traffic scenarios, ensuring navigation safety. However, traditional simulation systems, which often heavily rely on manual modeling and 2D image editing, struggled with scaling to extensive scenes and generating realistic simulation data. In this study, we present S-NeRF++, an innovative autonomous driving simulation system based on neural reconstruction. Trained on widely-used self-driving datasets, such as nuScenes and Waymo, S-NeRF++ can generate a large number of realistic street scenes and foreground objects with high rendering quality as well as offering considerable flexibility in manipulation and simulation. Specifically, S-NeRF++ is an enhanced neural radiance field for synthesizing large-scale scenes and moving vehicles, with improved scene parameterization and camera pose learning. The system effectively utilizes noisy and sparse LiDAR data to refine training and address depth outliers, ensuring high-quality reconstruction and novel-view rendering. It also provides a diverse foreground asset bank by reconstructing and generating different foreground vehicles to support comprehensive scenario creation. Moreover, we have developed an advanced foreground-background fusion pipeline that skillfully integrates illumination and shadow effects, further enhancing the realism of our simulations. With the high-quality simulated data provided by our S-NeRF++, we found the perception methods enjoy performance boosts on several autonomous driving downstream tasks, further demonstrating our proposed simulator's effectiveness.

This paper reveals that large language models (LLMs), despite being trained solely on text data, are surprisingly}strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy -- employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks} encompassing pure 2D or 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks. We additionally propose the information filtering hypothesis to explain the effectiveness of pre-trained LLMs in visual encoding -- the pre-trained LLM transformer blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature activation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that our work inspires new perspectives on utilizing LLMs and deepening our understanding of their underlying mechanisms.

While bird’s-eye-view (BEV) perception models can be helpful in building high-definition maps (HD maps) with less human labor, their results are often unreliable and demonstrate noticeable inconsistencies in the predicted HD maps from different viewpoints. This is because BEV perception is typically set up in an "onboard" manner, which restricts the computation and prevents algorithms from simultaneously reasoning multiple views. This paper overcomes these limitations and advocates a more practical "offboard" HD map generation setup that removes the computation constraints, based on the fact that HD maps are commonly reusable infrastructures built offline in data centers. To this end, we propose a novel offboard pipeline called MV-Map that capitalizes multi-view consistency and can handle an arbitrary number of frames with the key design of a "region-centric" framework. In MV-Map, the target HD maps are created by aggregating all the frames of onboard predictions, weighted by the confidence scores assigned by an "uncertainty network." To further enhance multi-view consistency, we augment the uncertainty network with the global 3D structure optimized by a voxelized neural radiance field (Voxel-NeRF). Extensive experiments on nuScenes show that our MV-Map significantly improves the quality of HD maps, further highlighting the importance of offboard methods for HD map generation. Our code and model are available at https://github.com/ZiYang-xie/MV-Map.

Lifting tasks have been identified to be highly associated with work-related low back pain. Posture prediction can be used for simulating workers’ posture of lifting tasks and thus facilitate the prevention of low back pain (LBP). This study adopts two generative models, conditional variational encoder and conditional generative adversarial network, to predict lifting postures. A regular feed-forward neural network (FNN) developed upon previous studies is also investigated for comparison purposes. Ground-truth lifting posture data collected by a motion capture system is used for training and testing the models. The models are trained with datasets of different size and loss functions, and the results are compared. The conditional variational autoencoder and the regular FNN achieved comparable top performance in lifting posture prediction in terms of accuracy and posture validity. Both generative models are able to partially capture the variability of constrained postures. Overall, the results prove that using a generative model is able to predict postures with reasonable accuracy and validity (RMSE of coordinates = 0.049 m; RMSE of joint angles = 19.58$^\circ$). The predicted postures can support biomechanical analysis and ergonomics assessment of a lifting task to reduce the risk of low back injuries.