Abstract A long‐standing goal in scene understanding is to obtain interpretable and editable representations that can be directly constructed from a raw monocular RGB‐D video, without requiring specialized hardware setup or priors. The problem is significantly more challenging in the presence of multiple moving and/or deforming objects. Traditional methods have approached the setup with a mix of simplifications, scene priors, pretrained templates, or known deformation models. The advent of neural representations, especially neural implicit representations and radiance fields, opens the possibility of end‐to‐end optimization to collectively capture geometry, appearance, and object motion. However, current approaches produce global scene encoding, assume multiview capture with limited or no motion in the scenes, and do not facilitate easy manipulation beyond novel view synthesis. In this work, we introduce a factored neural scene representation that can directly be learned from a monocular RGB‐D video to produce object‐level neural presentations with an explicit encoding of object movement(e.g., rigid trajectory)and/or deformations(e.g., nonrigid movement). We evaluate ours against a set of neural approaches on both synthetic and real data to demonstrate that the representation is efficient, interpretable, and editable(e.g., change object trajectory).Code and data are available at: http://geometry.cs.ucl.ac.uk/projects/2023/factorednerf/ .

Multi-object tracking from RGB-D video sequences is a challenging problem due to the combination of changing viewpoints, motion, and occlusions over time. We observe that having the complete geometry of objects aids in their tracking, and thus propose to jointly infer the complete geometry of objects as well as track them, for rigidly moving objects over time. Our key insight is that inferring the complete geometry of the objects significantly helps in tracking. By hallucinating unseen regions of objects, we can obtain additional correspondences between the same instance, thus providing robust tracking even under strong change of appearance. From a sequence of RGB-D frames, we detect objects in each frame and learn to predict their complete object geometry as well as a dense correspondence mapping into a canonical space. This allows us to derive 6DoF poses for the objects in each frame, along with their correspondence between frames, providing robust object tracking across the RGB-D sequence. Experiments on both synthetic and real-world RGB-D data demonstrate that we achieve state-of-the-art performance on dynamic object tracking. Furthermore, we show that our object completion significantly helps tracking, providing an improvement of 6.5% in mean MOTA.

Abstract Although surface reconstruction from depth data has made significant advances in the recent years, handling changing environments remains a major challenge. This is unsatisfactory, as humans regularly move objects in their environments. Existing solutions focus on a restricted set of objects (e.g., those detected by semantic classifiers) possibly with template meshes, assume static camera, or mark objects touched by humans as moving. We remove these assumptions by introducing RigidFusion. Our core idea is a novel asynchronous moving‐object detection method, combined with a modified volumetric fusion. This is achieved by a model‐to‐frame TSDF decomposition leveraging free‐space carving of tracked depth values of the current frame with respect to the background model during run‐time. As output, we produce separate volumetric reconstructions for the background and each moving object in the scene, along with its trajectory over time. Our method does not rely on the object priors (e.g., semantic labels or pre‐scanned meshes) and is insensitive to the motion residuals between objects and the camera. In comparison to state‐of‐the‐art methods (e.g., Co‐Fusion, MaskFusion), we handle significantly more challenging reconstruction scenarios involving moving camera and improve moving‐object detection (26% on the miss‐detection ratio), tracking (27% on MOTA), and reconstruction (3% on the reconstruction F1) on the synthetic dataset. Please refer the supplementary and the project website for the video demonstration ( geometry.cs.ucl.ac.uk/projects/2021/rigidfusion ).

Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, Keh-Yih Su. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

A goal-oriented meaning-based statistical framework is presented in this paper to solve the math word problem that requires multiple arithmetic operations with understanding, reasoning and explanation. It first analyzes and transforms sentences into their meaning-based logical forms, which represent the associated context of each quantity with role-tags (e.g., nsubj, verb, etc.). Logic forms with role-tags provide a flexible and simple way to specify the physical meaning of a quantity. Afterwards, the main-goal of the problem is decomposed recursively into its associated sub-goals. For each given sub-goal, the associated operator and operands are selected with statistical models. Lastly, it performs inference on logic expressions to get the answer and explains how the answer is obtained in a human comprehensible way. This process thus resembles the human cognitive understanding of the problem and produces a more meaningful problem solving interpretation.

Abstract RGBD images with high quality annotations, both in the form of geometric (i.e., segmentation) and structural (i.e., how do the segments mutually relate in 3D) information, provide valuable priors for a diverse range of applications in scene understanding and image manipulation. While it is now simple to acquire RGBD images, annotating them, automatically or manually, remains challenging. We present SmartAnnotator, an interactive system to facilitate annotating raw RGBD images. The system performs the tedious tasks of grouping pixels, creating potential abstracted cuboids, inferring object interactions in 3D, and generates an ordered list of hypotheses. The user simply has to flip through the suggestions for segment labels, finalize a selection, and the system updates the remaining hypotheses. As annotations are finalized, the process becomes simpler with fewer ambiguities to resolve. Moreover, as more scenes are annotated, the system makes better suggestions based on the structural and geometric priors learned from previous annotation sessions. We test the system on a large number of indoor scenes across different users and experimental settings, validate the results on existing benchmark datasets, and report significant improvements over low‐level annotation alternatives. (Code and benchmark datasets are publicly available on the project page.)

Current medium access control (MAC) protocols for wireless sensor networks (WSNs) do not support prioritized channel access and often suffer from varying degrees of starvation and fairness problems. That is, with a little bad luck, a sensor node might have to wait arbitrarily long to send its frame, and thus results in the invisibility of some sensed area. In this paper we introduce a simple, but yet well performing starvation free backoff scheme based on sensor MAC (S-MAC) protocol for WSNs. The proposed scheme allows a sensor node to adjust its backoff window size according to its re-transmission times. An analytical analysis is carried out to study the saturation throughput of proposed scheme under backlogged conditions. In addition to the theoretical analysis, simulations were also conducted to analyze the proposed scheme and the results show that our scheme can effectively alleviate the starvation and fairness problem and achieve better performance in high density WSNs.