Satellite imagery with multiple spectral bands plays a crucial role in the analysis and prediction of atmospheric phenomena. Tracking tropical cyclones, i.e typhoons, are easily monitored by infrared bands with predefined typhoon life stages. In the life cycle of such typhoons, they typically exhibit formation of a few clouds to fully developed spiral clouds with a clear eye. The life stage is classified by analyzing the maximum sustained winds at a distance from the center of the eye. The lowest pressure is also used to characterize the magnitude of typhoons. Moreover, apparent typhoon shapes can be roughly classified into different life stages according to experts. However, this limits other applications due to auto-operational tasks. Therefore, this paper proposes a Deep Learning based satellite image classification for identifying the life stage of tropical cyclones, namely, DeepTrCy. Due to the limited training on past typhoon images, applying data augmentation by introducing a fluid-like noise, i.e., Perlin noise, is firstly proposed. Using different DL models by introducing different augmentation techniques like Perlin noise levels, and different input data like optical flow, results are compared to that of CNN and transformer architectures. The proposed DeepTrCy has demonstrated high accuracy on identifying tropical cyclone stages from the several experiments.

Weather observations are usually derived from measurements and monitoring by sensors, radars, satellites, and human observation. Remote cameras are also used; however, meteorological experts typically report changes in the sky, clouds, and weather based on these images every hour or longer. Moreover, it is challenging to install additional cameras in more locations due to the lack of such experts. Because of the complexity of cloud textures, state-of-the-art (SOTA) Deep Learning (DL) models have been applied to limited cloud types, mainly in clear weather conditions. In addition, sky, cloud, and weather recognition from camera images typically implement multiple DL models. However, SOTA DL models have not considered the impact of weather changes, i.e., rain. This paper proposes PanopticBlue, an integrated transformer-based camera-sky-cloud-weather image recognition and classification model under adversarial weather conditions. Scene object recognition is also used to reduce cloud-type classification errors by DL. Experimental results show the superiority of the proposed DeepTex for many cloud types, i.e., unique cloud recognition, twilight cloud recognition, and Cb type cloud, and weather changes, i.e., rain, lightning, and fog, in enhanced accuracy and robustness over SOTA DL models. Moreover, vision language is used to enrich auto-briefing in weather conditions.

This paper presents the integrated Deep Learning models for camera image semantic segmentation of road conditions with dry, wet, and snow types along with the rejection of adversarial images. For drivers, road conditions are the most important visible factor for safe driving. Also, traffic control through remote cameras is required for sudden changes like dry to snow as well. However, adversarial images like lens reflection, strong light, and fog can impede normal driving and monitoring. Mostly, these are unpredictable to avoid. Using only simple pre- and post-image processing is limiting in alleviating or excluding such adversarial images. Most Deep Learning models have shown their performances in no or less adversarial images, i.e., fine weather in the daytime. Therefore, two Deep Learning models, DeepReject and DeepRoad, have been proposed to overcome such previous issues even under many adversarial events, i.e., strong lights and low light. Using various road conditions from dry to snow, experimental results have proven that the proposed models/methods outperform the previous single Deep Learning models in terms of stability, robustness, and accuracy. The proposed DeepReject and DeepRoad can contribute to warning drivers and supporting road maintainers for safety. Moreover, DeepReject is helpful whenever small image datasets for training many Deep Learning models are available.

This paper presents a conversion method of cloud to precipitation images based on an improved Generative Adversarial Network (GAN) using multiple satellite and radar images. Since heavy rainfall events have been yearly increasing everywhere on the earth, precipitation radar images on lands become more important to use and predict, where much denser data is observed than on-the-ground sensor data. However, the coverage of such radar sites is very limited in small regions like land and/or near the sea. On the other hand, satellite images, i.e., Himawari-8, are available globally, but no direct precipitation images, i.e., rain clouds, can be obtained. GAN is a good selection for image translation, but it is known that high edges and textures can be lost. This paper proposes ‘sat2rain’, a two-step algorithm with a new constraint of the loss function. First, multiple satellite band and topography images are input to GAN, where block-wised images from overall images are used to cover over 2500 km x 2500 km. Second, enhanced GAN-based training between satellite images and radar images is conducted. Experimental results show the effectiveness of the proposed sat2rain mesh-wise method over the previous point-wise Random Forest method in terms of high edge and texture.

This paper proposes a small and blurry foreground object detection method from highly similar textured background images based on Deep Learning (DL). In satellite imagery, volcanic smoke can be observed as ash clouds that present highly similar shape and texture to surrounding normal clouds. Moreover, very small size of such ash clouds can further impede the detectability when applying previous image processing-based methods. Therefore, a special care for discriminating foreground and background images has been desired. For this issue, this paper presents a multiple, i.e., three-step, DL based important object detection method for satellite imagery, i.e., specially mixed infrared channels of the Himawari-8, with volcanic smoke/ash clouds. Moreover, DL, i.e. Fully Convolutional Network, generalization performances with blurry object contours have been enhanced by multiple loss functions and re-selection of pooling layer. Experimental results on world-wide volcanos have demonstrated that the proposed multiple DL models have outperformed a single DL based detection methods, i.e., edge and contour detection.

This paper proposes an image-based prediction method that can physically predict near-future spatio-temporal image changes, i.e., nowcasting, using fluid-like image sequences from precipitation image sequences. Since motion estimation is one of the most influential factors for very short, i.e., 30 mins or 1 hr, to short term, i.e., 3 hrs or longer, mix of local and global motion estimation methods, i.e., multiple optical flow models, has been proposed to apply. A physically motivated advection equation is used to predict the next time image sequence from the past time image sequence, where previous image degradation issue of losing heavy rainfall regions has been improved by the Cubic Interpolation Profile. Navier-Stokes equation also conducts to provide a time-varying and physical future motion field. A proposed switching prediction model integrates different prediction models in order to spatio-temporally adapt different radar image property during a few hours or longer durations. Using heavy rainfall events, i.e., squall lines and typhoons, it has been proven that the proposed nowcasting method has outperformed the world best nowcasting method, i.e., Rainymotion of open source, in terms of accuracy of precipitation amount.

Videos are spatio-temporally rich in static to dynamic objects/scenes, sparse to dense, and periodic to non-periodic motions. Particularly, the dynamic texture (DT) exhibits complex appearance and motion changes that remain a challenge to deal with. This paper presents an energy optimization method for feature extraction and recognition in videos. For noise and background jitter, the Tikhonov regularization with eigen-vector and Frenet-Serret formula-based energy constraints is also proposed. The different periodicity of DT can be adapted by the time-varying number of learning temporal frames. The optimal duration of an image sequence is determined from the temporal property of its eigen-values. Unlike the state-of-the-art recognition methods, i.e., sparse coding and slow feature analysis, the proposed method can capture the physical property of objects and scenes: velocity, acceleration, and orientation. Also, the static and dynamic image regions can be locally classified. Owing to these spatio-temporal features, stability, robustness, and accuracy of feature extraction and recognition are enhanced. Using DT videos, the superiority of the proposed method compared to the state-of-the-art recognition methods is experimentally shown.

Video extrapolation/prediction methods are often used to synthesize new videos from images. For fluid-like images and dynamic textures as well as moving rigid objects, most state-of-the-art video extrapolation methods use non-physics-based models that learn orthogonal bases from a number of images but at high computation cost. Unfortunately, data truncation can cause image degradation, i.e., blur, artifact, and insufficient motion changes. To extrapolate videos that more strictly follow physical rules, this paper proposes a physics-based method that needs only a few images and is truncation-free. We utilize physics-based equations with image intensity and velocity: optical flow, Navier-Stokes, continuity, and advection equations. These allow us to use partial difference equations to deal with the local image feature changes. Image degradation during extrapolation is minimized by updating model parameters, where a novel time-varying energy balancer model that uses energy based image features, i.e., texture, velocity, and edge. Moreover, the advection equation is discretized by high-order constrained interpolation profile for lower quantization error than can be achieved by the previous finite difference method in long-term videos. Experiments show that the proposed energy based video extrapolation method outperforms the state-of-the-art video extrapolation methods in terms of image quality and computation cost.

Motion estimation, i.e., optical flow, of fluid-like and dynamic texture (DT) images/videos is an important challenge, particularly for understanding outdoor scene changes created by objects and/or natural phenomena. Most optical flow models use smoothness-based constraints using terms such as fluidity from the fluid dynamics framework, with constraints typically being incompressibility and low Reynolds numbers (Re ). Such constraints are assumed to impede the clear capture of locally abrupt image intensity and motion changes, i.e., discontinuities and/or high Re over time. This paper exploits novel physics-based optical flow models/constraints for both smooth and discontinuous changes using a wave generation theory that imposes no constraint on Re or compressibility of an image sequence. Iterated two-step optimization between local and global optimization is also used: first, an objective function with varying multiple sine/cosine bases with new local image properties, i.e., orientation and frequency, and with a novel transformed dispersion relationship equation are used. Second, the statistical property of image features is used to globally optimize model parameters. Experiments on synthetic and real DT image sequences with smooth and discontinuous motions demonstrate that the proposed locally and globally varying models outperform the previous optical flow models.

This paper presents an extended Markov chain Monte Carlo (MCMC) method for tracking and an extended hidden Markov model (HMM) method for learning/recognizing multiple moving objects in videos with jittering backgrounds. A graphical user interface (GUI) with enhanced usability is also proposed. Previous MCMC and HMM-based methods are known to suffer performance impairments, degraded tracking and recognition accuracy, and higher computation costs when challenged with appearance and trajectory changes such as occlusion, interaction, and varying numbers of moving objects. This paper proposes a cost reduction method for the MCMC approach by taking moves, i.e., birth and death, out of the iteration loop of the Markov chain when different moving objects interact. For stable and robust tracking, an ellipse model with stochastic model parameters is used. Moreover, our HMM method integrates several different modules in order to cope with multiple discontinuous trajectories. The GUI proposed herein offers an auto-allocation module of symbols from images and a hand-drawing module for efficient trajectory learning and for interest trajectory addition. Experiments demonstrate the advantages of our method and GUI in tracking, learning, and recognizing spatiotemporal smooth and discontinuous trajectories.

This paper proposes an image-based prediction method that can physically predict near-future spatio-temporal image changes using fluid-like image sequences, i.e., dynamic texture, from different image sources such as ground-based radar imagers, satellite sensors, and lightning detectors. Previous alternatives, i.e., tracking radar echo by correlation or thunderstorm identification, tracking, analysis, and nowcasting, employ pattern matching or linear extrapolation of the centroid of an image object to predict the next time image with many tuning model parameters. However, such methods fail to handle the high degree of motion and deformation of fluid-like images, i.e., vortex. To remedy this issue, this paper presents a spatio-temporal prediction method based on a computer vision framework; it employs a physics-based model with time-variant optical flow. Initial local motions from image sequences are estimated by the extended optical flow method, where a locally optimal weighting parameter and a statistically robust function are applied to Horn and Schunck's model. The next time image sequence from the past image sequence is physically predicted by the extended advection equation for image intensities and the Navier–Stokes equation with a continuity equation for varying optical flow over time. For different source images, our method offers no prior knowledge of size, shape, texture, and motion of moving objects. Experiments demonstrate that the proposed prediction method outperforms a previous prediction method with respect to prediction accuracy.

This paper proposes a new semitransparency-based optical-flow model with a point trajectory (PT) model for particle-like video. Previous optical-flow models have used ranging from image brightness constancy to image brightness change models as constraints. However, two important issues remain unsolved. The first is how to track/match a semitransparent object with a very large displacement between frames. Such moving objects with different shapes and sizes in an outdoor scene move against a complicated background. Second, due to semitransparency, the image intensity between frames can also violate a previous image brightness-based optical-flow model. Thus, we propose a two-step optimization for the optical-flow estimation model for a moving semitransparent object, i.e., particle. In the first step, a rough optical flow between particles is estimated by a new alpha constancy constraint that is based on an image generation model of semitransparency. In the second step, the optical flow of a particle with a continuous trajectory in a definite temporal interval based on a PT model can be refined. Many experiments using various falling-snow and foggy scenes with multiple moving vehicles show the significant improvement of the optical flow compared with a previous optical-flow model.

This paper contributes a novel technique for reducing the interference of non-Gaussian texture noise from images. Since the inherent properties of texture noise are very different from those of Gaussian white noise, the basic assumption of conventional image denoising techniques is invalid. Here we present a spatial adaptive filtering scheme to remove non-Gaussian texture noise from textile images based on local and non-local similarities. In order to exploit the high correlations among pixels, pixels with uniform texture local regions are estimated differently from those pixels located near edges, that is, for points located in local uniform texture regions, Gaussian weighted averaging of their neighbors can achieve the adaptive effect of the human visual system, whereas for edge points, to find pixels with similar local statistics both in the vicinity and far away can produce a sufficient set of pixels for reasonable averaging. This filtering strategy is applied to textile images corrupted by texture noise and the performance is demonstrated to outperform current state-of-art image denoising techniques.

Edge detection or interesting point detection is one of the most fundamental methods in CV and image processing. Most previous methods have devoted to detect spatial image features. For videos with natural phenomena, not only the spatial features but also the temporal features are important to analyze and to classify a dynamic scene. In this paper, a spatio-temporal (ST) derivative tensor based on multiple frames is proposed. The spatio-temporal information containing in the multiple frames enable us to estimate the magnitude and orientation of the dynamic edges. With the estimated magnitude and orientation of dynamic edges, we can classify the dynamic scene into different regions with distinctive motion activities. The present experimental results demonstrate the method's ability to classify both rigid motions, and non-rigid motions as well when compared with some state-of-the-art techniques.

This paper presents a motion estimation method for semitransparent objects with a long-range displacement between frames, i.e., falling snow in video. Previous optical flow based methods have been treated with non-transparent, rigid, and fluid-like moving objects in a short-range displacement. However, they fail to match between frames when moving objects are transparent/homogenoeous color in a long-range displacement. To meet with such objects' properties, a two-step algorithm is proposed from rough to refined motion estimation via an energy minimization. First, rough motion of every snow particles is extracted from video using a novel “time filter” method in order to obtain/update a quasi-stationary background in every 30 fps. Second, using such rough optical flow from the first step, the long-range snowflakes' trajectories are estimated and refined by propagation, linking, pruning, and optimization. Experimental results using real falling snow videos show that the proposed method is more effective than a previous optical flow method. Our proposed method is useful for the analysis of natural environment changes.

Natural phenomena such as smoke, waterfall, and stream show both texture and motion changes over time. The underlying physical properties bring complicated 3D variations which have attracted our interest. Velocity is one of important physical properties, which affects fluidity in small or large scale. Thus, this paper presents a 3D photodynamic tool (PDT) using a single fluid-like 2D image with 3D velocity primitives and physical models. Our tool offers only a few manipulations to generate a long term animation from a single image in a minute. Experimental results show that our 3D PDT can animate a realistic texture change from a single 2D photo even with occlusion in a scene.

This paper presents a fluid flow estimation method for ocean/river waves, clouds, and smoke based on the physical properties of waves. Most of the previous optical flow methods based on fluid dynamics/mechanics estimate a smooth flow using a continuity equation and/or div-curl velocity constraint. However, abrupt or inhomogeneous image motion changes in such fluid-like images are not estimated well. In this paper, we assume that many fluid-like motion changes are due to wave phenomena that lead to a brightness change. Thus, a wave generation equation is applied with a two-step optimization. A novel constraint based on the velocity-frequency relationship equation and a wave statistical property are used. The results of experiments on synthetic and real image sequences show the validity of our method.

This paper proposes a new animation editing method for creating animations from a single texture image. A standard warping method can not handle a physical motion effect because of a manual control. The Navier-Stokes equation based method is known to be able to animate a fluid-like animation. However, the difficulties of defining initial- and boundary-conditions with many control parameters remain unsolved for a stable and desired animation. And it offers a heavy computation. In this paper, an advection equation that is a physical-based and compact equation is used. A unique interface is also introduced with a gallery of many velocity vectors: convergence, rotation and vibration. Our method is versatile because that the combination of such velocity patterns can animate a quasi-linear motion over time by solving a set of the linear equations. We have carried out several experiments using some real images. The results of animations show that the PDT is effective, efficient, and easy to use.

The paper presents a moving vehicle velocity estimator for interpreting scenes under bad weather conditions. The estimator is based on the use of a non-linear robust velocity estimator with a contrast and a brightness variation model. This model is robust against falling-snow patterns that exhibit properties of infinite shapes and size. In this model, no prior knowledge of white color is applied because both snowfalls and vehicles may have the same color. Instead, the method is to minimize the objective function of the model with four variables: horizontal and vertical velocity, brightness, and contrast. Two velocity components of moving vehicles are accurately detected because the falling snow patterns are captured in the brightness and the contrast images. A simple scene interpretation is examined with a modified clustering algorithm. To verify the effectiveness of the method, the recognition rate is compared with that of a conventional velocity detection method.

This paper describes a prediction method, the Dynamics Texture (DT) method, of three different radar echo image sequences based on partial difference equations (PDEs) and an extended optical flow over time. Three image patterns are observed from precipitation-, lightning-, and satellite-radar sites. Such patterns exhibit development, disappearing, and fusion processes with local discontinuities. The DT method can calculate future dynamic pattern changes over time from past sequences of these image sequences by making use of a changeable texture pattern model and the extended optical flow method Five basic spatio-temporal patterns with corresponding PDEs are responsible for a wide range of the patterns. For discontinuities and noise due to abrupt local changes in shape and intensity, a nonlinear robust function with brightness and contrast change model is also proposed. Using the heterogenous patterns, experimental results show a high prediction accuracy for the DT method over time and hence demonstrate the usefulness for disaster prevention in short-term forcasting or nowcasting.