This study proposes a method for forecasting sea surface temperature one day in advance using a deep learning approach. Unlike conventional machine learning-based forecasting methods, our proposed method is novel in its approach for predicting deviations from simulation results using real-time satellite observations. The method involves predicting an optical flow, representing the movement of water masses derived from the simulation results. This predicted flow is then used to deform the satellite observation data, which serves as the input for the deep learning model. Experimental results show that our method effectively improves prediction performance in the area where warm and cold currents meet.

We propose a machine learning-based method for sea surface temperature (SST) estimation and cloud detection from satellite observations. To deal with estimation errors due to thin cloud cover, our approach uses observations from spectral bands not directly related to SST prediction as explanatory variables and constructs an ensemble model that takes into account the likelihood of cloudiness in each observation. This approach allows for implicit estimation of cloud conditions, which improves the performance of SST regression. Experimental results show that our method outperforms existing methods.

Abstract We propose deep depth from focal stack (DDFS), which takes a focal stack as input of a neural network for estimating scene depth. Defocus blur is a useful cue for depth estimation. However, the size of the blur depends on not only scene depth but also camera settings such as focus distance, focal length, and f-number. Current learning-based methods without any defocus models cannot estimate a correct depth map if camera settings are different at training and test times. Our method takes a plane sweep volume as input for the constraint between scene depth, defocus images, and camera settings, and this intermediate representation enables depth estimation with different camera settings at training and test times. This camera-setting invariance can enhance the applicability of DDFS. The experimental results also indicate that our method is robust against a synthetic-to-real domain gap.

Sea-surface temperature (SST) images obtained by satellites contain noise and missing SSTs due to cloud covers. We propose a method for reconstructing denoised cloud-free SST images via deep-learning-based image inpainting. For denoizing, we use data-assimilation images to train a reconstruction network by considering the physical correctness of SSTs. For reconstruction stability, we introduceanomaly inpainting network, which does not directly complete missing SSTs but estimates the difference between the unobserved SSTs and the average SSTs. SSTs do not fluctuate much over a few days; thus, we can use recent average SSTs as a rough estimation of SSTs and can assume that the SST difference will be within a specific range. We conducted experiments to evaluate our method with satellite SST images andin situSST data. The results indicate that our method with anomaly inpainting network qualitatively and quantitatively outperformed conventional SST image inpainting methods.

We propose a learning-based multi-view stereo (MVS) method in scattering media, such as fog or smoke, with a novel cost volume, called the dehazing cost volume. Images captured in scattering media are degraded due to light scattering and attenuation caused by suspended particles. This degradation depends on scene depth; thus, it is difficult for traditional MVS methods to evaluate photometric consistency because the depth is unknown before three-dimensional (3D) reconstruction. The dehazing cost volume can solve this chicken-and-egg problem of depth estimation and image restoration by computing the scattering effect using swept planes in the cost volume. We also propose a method of estimating scattering parameters, such as airlight, and a scattering coefficient, which are required for our dehazing cost volume. The output depth of a network with our dehazing cost volume can be regarded as a function of these parameters; thus, they are geometrically optimized with a sparse 3D point cloud obtained at a structure-from-motion step. Experimental results on synthesized hazy images indicate the effectiveness of our dehazing cost volume against the ordinary cost volume regarding scattering media. We also demonstrated the applicability of our dehazing cost volume to real foggy scenes.

Images captured in participating media such as murky water, fog, or smoke are degraded by scattered light. Thus, the use of traditional three-dimensional (3D) reconstruction techniques in such environments is difficult. In this paper, we propose a photometric stereo method for participating media. The proposed method differs from prvious studies with respect to modeling shape-dependent forward scatter. In the proposed model, forward scatter is described as an analytical form using lookup tables and is represented by spatially-variant kernels. We also propose an approximation of a large-scale dense matrix as a sparse matrix, which enables the removal of forward scatter. We discuss the approximation in the proposed method using synthesized data. Then, experiments with real data demonstrate that the proposed method improves 3D reconstruction in participating media.

Images captured in participating media such as murky water, fog, or smoke are degraded by scattered light. Thus, the use of traditional three-dimensional (3D) reconstruction techniques in such environments is difficult. In this paper, we propose a photometric stereo method for participating media. The proposed method differs from previous studies with respect to modeling shape-dependent forward scatter. In the proposed model, forward scatter is described as an analytical form using lookup tables and is represented by spatially-variant kernels. We also propose an approximation of a large-scale dense matrix as a sparse matrix, which enables the removal of forward scatter. Experiments with real and synthesized data demonstrate that the proposed method improves 3D reconstruction in participating media.

Sea surface temperature(SST) satellite images are often partially occluded by clouds. Image inpainting is one approach to restore the occluded region. Considering the sparseness of SST images, they can be restored via learning-based inpainting. However, state-of-the-art learning-based inpainting methods using deep neural networks require large amount of non-occluded images as a training set. Since most SST images contain occluded regions, it is hard to collect sufficient non-occluded images. In this paper, we propose a novel method that uses occluded images as training images hence we can enlarge the amount of available training images from a certain SST image set. This is realized by comprising a novel reconstruction loss and adversarial loss. Experimental results confirm the effectiveness of our method.

Sea surface temperature (SST) images taken from satellites are partially occluded by clouds. In this paper, we propose an inpainting approach for restoration of the partially occluded images. Assuming the sparseness of the SST images, we employ a learning based inpainting for filling the occluded parts. Images taken in the past several days is another clue for filling the occluded parts. These images are regarded as time series data and a video inpainting method is also available. We employ PCA-based inpainting as a learning-based approach and optical-flow-based inpainting as video inpainting, and combine the two restored images according to the expected their restoration error. Experimental results with real satellite images show the effectiveness of our method.

In this paper, we present a regression for predicting 3D rigid transformations from real-valued vectors. We use a unit dual quaternion to represent the transformation. The regression is formulated as blending unit dual quaternions. To formulate it in a closed form, we introduce an approximation based on error metrics according to geometric algebra. Finally, we take an articulated motion and an elastic deformation as examples to present the descriptive power of our method in modeling the motion and the deformation.

Object tracking from camera images is a key technique in the field of computer vision. Because the most representative application of this technique is security or surveillance, objects such as pedestrians or vehicles in public spaces are considered as tracking targets. However, it is also useful to track objects such as daily items found in indoor living spaces, including offices, private rooms, and other locations, when they go missing or need to be located. However, it is not known in advance which part of the living space can move as a single object to be tracked. Moreover, because objects in a living space do not move by themselves but are moved by human hands, the bodies of objects in motion are usually occluded by the hands for a relatively long period of time, and drastic changes in the position and appearance of the objects may occur after a long-term occlusion. In this article, we propose an approach to finding the correspondence between objects observed before and after their occlusion for object tracking purposes by detecting the grasp and release of such objects by the same hand. Segmentation of the body of each object is also realized through the detection of its grasp and release.

Conventional studies on human behavior recognition have mainly focused on individual actions, including facial expressions and postures. However, most human behavior involves face-to-face interactions with other humans or objects, such as PCs. The previous work on recognizing face-to-face interaction focused on recognition of each group of humans engaged in conversation in an open space, which is considered a free space without obstacles. However, face-to-face conversation in our daily activities also occurs in a space such as an office with various obstacles, which could interfere with the interaction. To cope with this interference, the arrangement of obstacles and humans in the space should be considered. Moreover, in an office space, humans interact not only with other humans but also with certain interactive objects, such as PCs. This study aims to detect the target of interaction of each person in a space with obstacles by considering the 3D positions and orientations of humans, interactive objects, and other obstacles. The 3D arrangement of humans and obstacles is obtained by observing the space with an RGB-D camera.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

In this paper, we adopt the coded aperture technique to the alignment process for industrial machinery. As a special setting for assembly and inspection, such machinery uses an illumination of narrow wavelength range for imaging with less aberration. This leads to significant influence of light diffraction on the image restoration. Although the diffraction was treated as negligible in most previous studies of coded aperture since they were carried out for natural images, the aperture patterns of them do not achieve enough accuracy for the alignment. We optimize the aperture pattern by performing a simulation of light diffraction, and we experimentally show that it achieves better performance in the alignment.

Most vision-based 3D acquisition methods including both passive and active methods have a limitation in that cameras must be able to observe the surface to be measured. If this is not possible, that is to say, if the surface is occluded, most of the methods cannot acquire the surface shape. In this paper, we present a method that can acquire the 3D points on the occluded surfaces. The main idea is to observe the scattering of reflected light in a participating medium instead of observing only the reflected light. We place the target in a participating medium, specifically a liquid tank filled with a participating medium, and irradiate a laser beam on it. Even if the reflecting point is occluded, our method can acquire the 3D position of occluded reflecting points from the scattering. Experimental results showed that our method can measure 3D reflecting points on several objects even if the objects are completely occluded.

This article discusses the problem of recognizing groups of people in conversation with each other in an open space. Previous work on this problem takes an approach based on the knowledge that people in the same conversation group often makes a circular formation called an F-formation, by referring to the work in social psychology. Since the F-formation describes spatial and orientational characteristics of the lower bodies of people, their positions and orientations need to be obtained for employing the approach. However, it is difficult to observe lower bodies by cameras, especially for people in a circular formation, due to occlusions between their bodies. We propose to recognize conversation groups while estimating the positions and orientations of the lower bodies of the people only from the information of their head positions and facial orientations observable from a single camera, by considering that the lower bodies of the people are arranged in a circular pattern if they are in the same conversation group.

Most shape and reflectance acquisition methods use the light reflected from objects' surfaces. When the reflection cannot be observed, e.g., when the object's surface is black matte or highly specular, its shape and reflectance are difficult to acquire. In this paper, we propose a method that can measure shape and reflectance with another approach. Our method involves the use of the scattering of reflected light in a participating media instead of using only the light reflection. We place the target in a participating medium and focus a laser beam on it. Cameras can observe the scattering of reflected light toward all reflection angles even if they can only observe light reflected toward the cameras. Experimental results showed that our method can acquire the shape and reflectance of various surfaces.

In this paper, we propose a novel method of skeleton estimation for the purpose of constructing and manipulating individualized hand models via a data glove. To reconstruct actual hand accurately, we derive motion constraints in fully 6-DOF at the joints without assuming either a center of rotation or a joint axis. The constraints are derived by regression analysis on the configuration of the finger segments with respect to the sensor data. In order to acquire pairs of the configuration and the sensor data, we introduce graspable reference objects for reproducing postures with and without the data glove. To achieve accurate regression on fewer samples, we introduce a regression model according to a detailed investigation using a fused motion capture system, which enabled us to perform optical motion capture and sensor data collection simultaneously. The effectiveness of the method is demonstrated through practical applications involving grasped reference objects.

This paper discusses virtual object manipulation using a human hand in an AR environment. For this purpose, we need to measure the configuration, which includes the position, pose and posture, of the human hand. However, if we employ a data glove for the measurement, the glove appears in the video images synthesized by AR. If we otherwise employ a vision-based approach, the obtained configuration often includes larger errors due to occlusions among fingers. Moreover, the configuration of the human hand in anyway often includes some cognitive errors in manipulating the virtual object, which is only visible in the synthesized videos. To cope with these problems, we propose to replace the human hand with a data glove with a virtual hand in the video and adjust its configuration so that it properly grasps the virtual objects without errors.

This paper presents an artifact-free super resolution texture mapping from multiple-view images. The multiple-view images are upscaled with a learning-based super resolution technique and are mapped onto a 3D mesh model. However, mapping multiple-view images onto a 3D model is not an easy task, because artifacts may appear when different upscaled images are mapped onto neighboring meshes. We define a cost function that becomes large when artifacts appear on neighboring meshes, and our method seeks the image-and mesh assignment that minimizes the cost function. Experimental results with real images demonstrate the effectiveness of our method.

Photometric stereo is a method of recovering surface normals (needle map) from images. The surface integral of surface normals is used to reconstruct a depth map; however, the depth edges, which are discontinuous boundaries of the depth map, pose a problem for photometric stereo. When the surface of objects includes depth edges, the reconstructed depth map may contain errors. To solve this problem, we detect depth edges using shadows and compute a relative depth between two distant points using the widths of the corresponding shadows. We define an error function and reconstruct the depth map by minimizing the error function. Experimental results with synthetic and with real image data demonstrate the effectiveness of our approach.

3D shapes are reconstructed from silhouettes obtained by multiple cameras with the volume intersection method. In recent work, methods of integrating silhouettes in time sequences have been proposed. The number of silhouettes can be increased by integrating silhouettes in multiple frames. The silhouettes of a rigid object in multiple frames are integrated with its rigid motion. This motion is often estimated with 3D feature points extracted from silhouettes. When the estimated motion has large error, shapes are reconstructed with missing parts. This error is given by the incomplete extraction of 3D feature points, which is caused by additional and missing regions of extracted silhouettes. We cannot prevent silhouettes from being extracted with the additional and missing regions in real environments. Here, we propose an intelligent method of integrating incomplete silhouettes where outcrop points, which are 3D feature points for estimating motion, play an important role. The reconstructed shape can be evaluated referring to how many outcrop points have been included in the reconstructed shape of another frame. Although the evaluation does not represent the accuracy of estimated motion directly, it does guarantee that outstanding parts will be preserved in the reconstructed shape. Silhouettes in multiple frames can be integrated with fewer missing and additional parts based on this evaluation.

In this article, we discuss 3D shape reconstruction of an object in a rigid motion with the volume intersection method. When the object moves rigidly, the cameras change their relative positions to the object at every moment. To estimate the motion correctly, we propose new feature points called outcrop points on the reconstructed 3D shape. These points are guaranteed to be located on the real surface of the object. If the rigid motion of the object can be correctly estimated, cameras at different moments serve as the cameras in different positions virtually. With these cameras in time sequences, we can increase accuracy of the reconstructed 3D shape without increasing the number of cameras. Based on this idea, we reconstruct an accurate shape of the object in motion from images obtained by limited number of cameras. As the result, we can acquire an accurate shape from images in time sequences

Recognizing structure of the human body is important for modeling human motion. The human body is usually represented as an articulate model, which consists of the rigid parts and the joint points between them. The structure of the human body is specified by the joint points. We propose a method for estimating the location of joint points from successive volume data. Our joint point estimation method consists of three steps. In the first step, rigid parts are extracted from two successive volume data under the constraint of the rigid transformation. In the second step, the joint points are estimated based on the rigid parts. As the last step, false joint points are eliminated by using more successive data. Applying the method to the simulated data, the locations of joint points of the human are correctly estimated.