We propose a novel method for estimating head orientation from a low-resolution far-infrared (LFIR) image sequence captured by a 32 × 32 FIR sensor array. Estimating head orientation from LFIR images is challenging due to their inherent noise and limited resolution. Manually an-notating head orientation is also difficult, and no suitable dataset currently exists. We propose the LFIR2Head model, which leverages a multimodal-temporal framework for robust estimation. We also introduce the LFIR2Head dataset, which was created using an automatic annotation system. Extensive experiments demonstrate the accuracy of head orientation estimation with the proposed dataset. We con-firmed that head orientation can be estimated accurately from an LFIR image sequence.

The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods. Our implementation and dataset are publicly available at https://github.com/Howardkhh/FROSS.

Sea-surface temperature (SST) images obtained by satellites contain noise and missing SSTs due to cloud covers. We propose a method for reconstructing denoised cloud-free SST images via deep-learning-based image inpainting. For denoizing, we use data-assimilation images to train a reconstruction network by considering the physical correctness of SSTs. For reconstruction stability, we introduceanomaly inpainting network, which does not directly complete missing SSTs but estimates the difference between the unobserved SSTs and the average SSTs. SSTs do not fluctuate much over a few days; thus, we can use recent average SSTs as a rough estimation of SSTs and can assume that the SST difference will be within a specific range. We conducted experiments to evaluate our method with satellite SST images andin situSST data. The results indicate that our method with anomaly inpainting network qualitatively and quantitatively outperformed conventional SST image inpainting methods.

We propose a learning-based multi-view stereo (MVS) method in scattering media, such as fog or smoke, with a novel cost volume, called the dehazing cost volume. Images captured in scattering media are degraded due to light scattering and attenuation caused by suspended particles. This degradation depends on scene depth; thus, it is difficult for traditional MVS methods to evaluate photometric consistency because the depth is unknown before three-dimensional (3D) reconstruction. The dehazing cost volume can solve this chicken-and-egg problem of depth estimation and image restoration by computing the scattering effect using swept planes in the cost volume. We also propose a method of estimating scattering parameters, such as airlight, and a scattering coefficient, which are required for our dehazing cost volume. The output depth of a network with our dehazing cost volume can be regarded as a function of these parameters; thus, they are geometrically optimized with a sparse 3D point cloud obtained at a structure-from-motion step. Experimental results on synthesized hazy images indicate the effectiveness of our dehazing cost volume against the ordinary cost volume regarding scattering media. We also demonstrated the applicability of our dehazing cost volume to real foggy scenes.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.