Understanding scene contexts is crucial for machines to perform tasks and adapt prior knowledge in unseen or noisy 3D environments. As data-driven learning is intractable to comprehensively encapsulate diverse ranges of layouts and open spaces, we propose teaching machines to identify relational commonalities in 3D spaces. Instead of focusing on point-wise or object-wise representations, we introduce 3D scene analogies, which are smooth maps between 3D scene regions that align spatial relationships. Unlike well-studied single instance-level maps, these scene-level maps smoothly link large scene regions, potentially enabling unique applications in trajectory transfer in AR/VR, long demonstration transfer for imitation learning, and context-aware object rearrangement. To find 3D scene analogies, we propose neural contextual scene maps, which extract descriptor fields summarizing semantic and geometric contexts, and holistically align them in a coarse-to-fine manner for map estimation. This approach reduces reliance on individual feature points, making it robust to input noise or shape variations. Experiments demonstrate the effectiveness of our approach in identifying scene analogies and transferring trajectories or object placements in diverse indoor scenes, indicating its potential for robotics and AR/VR applications. Project page including the code is available through this link: https://82magnolia.github.io/3d_scene_analogies/.

Unlike computed tomography (CT) and magnetic resonance imaging (MRI) in which the image quality (IQ) is controlled by predefined acquisition setups, the IQ of ultrasound (US) is heavily dependent upon operators. In particular, an operator often adjusts the system parameters in a real-time manner based on his/her preference. Unfortunately, there are many cases where such real-time control of IQ is difficult, especially in the intensive care unit (ICU) or operating room (OR), since the operator should simultaneously treat patients in sterile status and adjust the system parameters. To address this, inspired by the recent success of Switchable CycleGAN using Adaptive Instance Normalization (AdaIN) layers, here we propose a novel speech-assisted Switchable CycleGAN architecture that can be controlled by operator’s verbal commands. Specifically, we employ a Speech Recognition Module (SRM) to generate AdaIN codes for Switchable CycleGAN. In particular, our SRM is based on the pre-trained model so that it can be less effected by the operator’s voice and robustly generate AdaIN codes. Various experimental results confirm that the proposed method can successfully control IQ of US by speech.

The absolute depth values of surrounding environments provide crucial cues for various assistive technologies, such as localization, navigation, and 3D structure estimation. We propose that accurate depth estimated from panoramic images can serve as a powerful and light-weight input for a wide range of downstream tasks requiring 3D information. While panoramic images can easily capture the surrounding context from commodity devices, the estimated depth shares the limitations of conventional image-based depth estimation; the performance deteriorates under large domain shifts and the absolute values are still ambiguous to infer from 2D observations. By taking advantage of the holistic view, we mitigate such effects in a self-supervised way and fine-tune the network with geometric consistency during the test phase. Specifically, we construct a 3D point cloud from the current depth prediction and project the point cloud at various viewpoints or apply stretches on the current input image to generate synthetic panoramas. Then we minimize the discrepancy of the 3D structure estimated from synthetic images without collecting additional data. We empirically evaluate our method in robot navigation and map-free localization where our method shows large performance enhancements. Our calibration method can therefore widen the applicability under various external conditions, serving as a key component for practical panorama-based machine vision systems.

Patient scans from MRI often suffer from noise, which hampers the diagnostic capability of such images. As a method to mitigate such artifacts, denoising is largely studied both within the medical imaging community and beyond the community as a general subject. However, recent deep neural network-based approaches mostly rely on the minimum mean squared error (MMSE) estimates, which tend to produce a blurred output. Moreover, such models suffer when deployed in real-world situations: out-of-distribution data, and complex noise distributions that deviate from the usual parametric noise models. In this work, we propose a new denoising method based on score-based reverse diffusion sampling, which overcomes all the aforementioned drawbacks. Our network, trained only with coronal knee scans, excels even on out-of-distribution in vivo liver MRI data, contaminated with a complex mixture of noise. Even more, we propose a method to enhance the resolution of the denoised image with the same network. With extensive experiments, we show that our method establishes state-of-the-art performance while having desirable properties which prior MMSE denoisers did not have: flexibly choosing the extent of denoising, and quantifying uncertainty.

We propose a light-weight, self-supervised adaptation for a visual navigation agent to generalize to unseen environment. Given an embodied agent trained in a noiseless environment, our objective is to transfer the agent to a noisy environment where actuation and odometry sensor noise is present. Our method encourages the agent to maximize the consistency between the global maps generated at different time steps in a round-trip trajectory. The proposed task is completely self-supervised, not requiring any supervision from ground-truth pose data or explicit noise model. In addition, optimization of the task objective is extremely light-weight, as training terminates within a few minutes on a commodity GPU. Our experiments show that the proposed task helps the agent to successfully transfer to new, noisy environments. The transferred agent exhibits improved localization and mapping accuracy, further leading to enhanced performance in downstream visual navigation tasks. Moreover, we demonstrate test-time adaptation with our self-supervised task to show its potential applicability in real-world deployment.

Tagging animals with sensors is a powerful approach to acquire critical information about the behavioural ecology of free-living animals, which ultimately can provide data to inform best practice in conservation efforts. Sensor tags for such deployments need long lifetimes and incorporate multiple sensors, especially location because space use can contextualize behavior. The tag size needs to be minimal so as not to affect the activities of the animal. Aquatic animals in particular present challenges due to lack of wireless communication and water-proofing. Taking these points into consideration we have designed Aquamote: an ultra-low power, tiny sensor tag (20 x 29 mm2) which integrates accelerometer, gyroscope, magnetometer, depth sensor, GPS and BLE. Our poster will showcase the performance of AquaMote and highlight our design decisions to reduce its size and power consumption.