Most supervised image segmentation methods require delicate and time-consuming pixel-level labeling of building or objects, especially for small objects. In this paper, we present a weakly supervised segmentation network for aerial/satellite images, separately considering small and large objects. First, we propose a simple point labeling method for small objects, while large objects are fully labeled. Then, we present a segmentation network trained with a small object mask to separate small and large objects in the loss function. During training, we employ a memory bank to cope with the limited number of point labels. Experiments results with three public datasets demonstrate the feasibility of our approach.

We propose a novel imputation method for highly missing data. Though most existing imputation methods focus on moderate missing rate, imputation for high missing rate over 80% is still important but challenging. As we expect that multiple imputation is indispensable for high missing rate, we propose a generative adversarial multiple imputation network (GAMIN) based on generative adversarial network (GAN) for multiple imputation. Compared with similar imputation methods adopting GAN, our method has three novel contributions: 1)We propose a novel imputation architecture which generates candidates of imputation. 2)We present a confidence prediction method to perform reliable multiple imputation. 3)We realize them with GAMIN and train it using novel loss functions based on the confidence. We synthesized highly missing datasets using MNIST and CelebA to perform various experiments. The results show that our method outperforms baseline methods at high missing rate from 80% to 95%.

In this paper, we propose an efficient method for reliably detecting road lanes based on spatiotemporal images. In an aligned spatiotemporal image generated by accumulating the pixels on a scanline along the time axis and aligning consecutive scanlines, the trajectory of the lane points appears smooth and forms a straight line. The aligned spatiotemporal image is binarized, and two dominant parallel straight lines resulting from the temporal consistency of lane width on a given scanline are detected using a Hough transform, reducing alignment errors. The left and right lane points are then detected near the intersections of the straight lines and the current scanline. Our spatiotemporal domain approach is more robust missing or occluded lanes than existing frame-based approaches. Furthermore, the experimental results show not only computation times reduced to as little as one-third but also a slightly improved rate of detection.

This paper presents an efficient method for adjusting the 3D depth of an object including as much as a whole scene in stereo 3D images by utilizing a virtual fronto-parallel planar projection in the 3D space perceived by a viewer. The proposed method just needs to establish object correspondence instead of the accurate estimation of the disparity field or point correspondence. We simulate the depth adjustment of a 3D point perceived by a viewer through a corresponding pair of points on the stereo 3D images by moving the virtual fronto-parallel plane on which the left and right points are projected. We show that the resulting transformation of image coordinates of the points can be simply expressed by three values of a scale factor and two translations that depend on one parameter for the depth adjustment. The experimental results demonstrate the feasibility of the proposed approach that yields less visual fatigue and smaller 3D shape distortion than the conventional parallax adjustment method. The overall procedure can be efficiently applied to each frame of a stereo video without causing any artifact.

This paper presents an adaptive parallax control method based on 3D scene change detection for stereoscopic video sequences. For each pair of left and right frames, the maximum disparity corresponding to the closest object is estimated by using a simple block matching algorithm. The 3D scene change is detected if there exists an abrupt change in the values of either the maximum disparity or color histogram between successive pair of frames. Then, if the maximum disparity is excessive at a pair of frames where the scene change has occurred, the horizontal parallax is controlled by reducing the excessive maximum disparity that could induce the visual discomfort. The experimental results demonstrate the feasibility of the proposed approach.

An efficient frame loss concealment algorithm for stereoscopic video based on the inter-view similarity of motion vectors and intensity differences is proposed in this work. Suppose that a frame at time t in the right view is lost during the transmission. To conceal its loss, we use the information in the previous frame at time t - 1 in the right view and the frames at t - 1 and t in the left view. More specifically, we first estimate the disparity vector field of the previous frame to find the matching pixels in the left view, and determine the motion vectors and the intensity differences of those matching pixels. By projecting those motion vectors and intensity differences onto the right view, we recover the lost frame. Simulation results demonstrate that the proposed algorithm provides significantly better performance than conventional algorithms.

For efficient image retrieval, most existing methods that adopt a tree-structured hierarchy exploit a subtree pruning scheme, based on the triangle inequality of a distance metric, during the tree traversal. In this paper, we propose a refined pruning scheme for the tree-structured hierarchy by introducing a novel notion of the node distance to subcluster (NDS) and validate it in both theoretical and experimental aspects. Experimental results demonstrate that the proposed method outperforms the existing ones by 10%-25% in terms of retrieval efficiency at the negligible increase of storage for a retrieval system

The H.264/AVC video coding standard adopted a new intra prediction technique that enables high coding efficiency for intra coded frames. However, it is known to produce more flickers in intra coded frames when the modes are selected based on conventional rate distortion optimized mode decision methods. In this paper, we propose to use a modified distortion measure for rate distortion optimized mode decision to reduce flicker in intra coded frames of H.264/AVC. Experimental results show that our proposed method can reduce significant amount of flicker maintaining similar PSNR and bitrate.

An efficient image resizing algorithm in the compressed domain with mixed field/frame-mode macroblocks is proposed. A 16/spl times/16 field/frame-mode macroblock is converted into an 8/spl times/8 reduced block in the discrete cosine transform (DCT) domain using a modified inverse DCT (IDGT) kernel which performs stronger lowpass filtering than the simple truncation in the DCT domain. Experimental results show that the proposed algorithm provides the resized image quality similar to an existing method with significantly lower computational complexity.

We propose a novel scheme for generating transcoded (e.g. scaled and cropped) image to fit the size of the respective client display when an image is transmitted to a variety of client devices with different display sizes. The scheme has two key components: (i) perceptual hint for each image block, and (ii) an image transcoding algorithm. For a given semantically important block in an image, the perceptual hint provides the information on the minimum allowable spatial resolution reduction. The image transcoding algorithm that is basically a content adaptation process selects the best image representation to meet the client capabilities while delivering the largest content value. The content value is defined as a quantitative measure of the information on importance and spatial resolution for the transcoded version of an image. Experimental results demonstrate the feasibility of the proposed scheme with a variety of client devices having different display sizes.

Extracting a small number of key frames that can abstract the content of video is very important for efficient browsing and retrieval in video databases. In this paper, the key frame extraction problem is considered from a set-theoretic point of view, and systematic algorithms are derived to find a compact set of key frames that can represent a video segment for a given degree of fidelity. The proposed extraction algorithms can be hierarchically applied to obtain a tree-structured key frame hierarchy that is a multilevel abstract of the video. The key frame hierarchy enables an efficient content-based retrieval by using the depth-first search scheme with pruning., Intensive experiments on a variety of video sequences are presented to demonstrate the improved performance of the proposed algorithms over the existing approaches.

This paper proposes a method for detecting obstacles on a runway by controlling their expected flow disparities. The runway is modeled as a planar surface. By approximating the runway by a planar surface, the initial model flow field (MFF) corresponding to an obstacle-free runway is described by the data from on-board sensors (OBS). The initial residual flow field (RFF) is obtained after warping (or stabilizing) the image using the initial MFF. The error variance of the initial MFF is estimated. The initial RFF and the error variance are first used to identify the pixels corresponding to the obstacle-free runway and then to noniteratively estimate the MFF and RFF. Obstacles are detected by comparing the expected residual flow disparities with the RFF. Expected temporal and spatial residual disparities are obtained from the use of the OBS. This allows us to control the residual disparities by increasing the temporal baseline and/or by utilizing the spatial baseline if distant objects cannot be detected for a given temporal baseline. Experimental results for two real flight image sequences are presented.

We present an efficient video parsing method using shot boundary detection and camera operation analysis technique. In the shot boundary detection, the local color information and an adaptive time window is used. The local spatio-temporal images and multilayer perceptron are used for analyzing the camera operations. In order to verify the performance of the proposed video parsing method, experiments with video database have been carried out. Experimental results demonstrate the efficiency of the video parsing technique.

This paper proposes a method for detecting obstacles on a runway by controlling their expected disparities. By approximating the runway by a planar surface, the initial model flow field (MFF) corresponding to an obstacle-free runway is described by the data from on-board sensors (OBS). The error variance of the initial MFF is computed and used to estimate the MFF. Obstacles are detected by comparing the expected residual flow disparities with the residual flow field (RFF) estimated after warping (or stabilizing) an image using the MFF. Expected temporal and spatial disparities are obtained from the use of the OBS. This allows us to control the residual disparities by increasing the temporal baseline and/or by utilizing the spatial baseline if distant objects cannot be detected for a given temporal baseline. Experimental results for two real flight image sequences are presented.

This paper proposes a method for detecting obstacles on a runway from monocular image sequences. The surface of the runway is modelled as a plane and the model flow field corresponding to the runway is described by 8 coefficients. The set of 8 coefficients describing the initial model flow field is given by the data from inertial navigation system (INS). The uncertainties of this initial model flow field are estimated and used to obtain the accurate model flow field. The residual flow field after warping images by the accurate model flow field is computed by solving overdetermined gradient-based optical flow equations using a singular value decomposition (SVD). This SVD gives us a new insight of the uncertainties inherent in the optical flow computation. Those pixels with large residual flow vectors are considered as obstacles. Experimental results for two real image sequences are presented.

We present an approach for estimating and segmenting the displacement field (DF) between two frames. Our method is based on local affine (first-order) approximation of the displacement field which is derived under the assumption of locally rigid motion. Each distinct motion is represented in the image plane by a distinct set of values of affine coefficients. All sets of values supported by the feature locations in two frames are identified by exhaustive coarse-to-fine search. The integrated use of multiple features (points, regions and lines) increases the probability of finding well-supported sets. The sets of coefficients thus obtained are used to describe the DF. Two experimental results with real images are presented to demonstrate the feasibility of our approach.>

This paper is concerned with three-dimensional (3D) analysis, and analysis-guided syntheses, of images showing 3-D motion of an observer relative to a scene. There are two objectives of the paper. First, it presents an approach to recovering 3D motion and structure parameters from multiple cues present in a monocular image sequence, such as point features, optical flow, regions, lines, texture gradient, and vanishing line. Second, it introduces the notion that the cues that contribute the most to 3-D interpretation are also the ones that would yield the most realistic synthesis, thus suggesting an approach to analysis guided 3-D representation. For concreteness, the paper focuses on flight image sequences of a planar, textured surface. The integration of information in these diverse cues is carried out using optimization. For reliable estimation, a sequential batch method is used to compute motion and structure. Synthesis is done by using (i) image attributes extracted from the image sequence, and (ii) simple, artificial image attributes which are not present in the original images. For display, real and/or artificial attributes are shown as a monocular or a binocular sequence. Performance evaluation is done through experiments with one synthetic sequence, and two real image sequences digitized from a commercially available video tape and a laserdisc. The attribute based representation of these sequences compressed their sizes by 502 and 367. The visualization sequence appears very similar to the original sequence in informal, monocular as well as stereo viewing on a workstation monitor.>

An algorithm is presented which estimates 10 parameters for motion and structure of a rigid planar patch given point correspondences in a monocular image sequence under perspective projection. The rotational velocity is assumed constant and the rotation center arbitrary. The algorithm mainly consists of two steps. First, the 3-D space of ( omega /sub x/, omega /sub y/, omega /sub z/) is searched exhaustively and for each ( omega /sub x/, omega /sub y/, omega /sub z/) all the other parameters with the value of an objective function are computed linearly. some of the ( omega /sub x/, omega /sub y/, omega /sub z/) and the corresponding structure values are used as the initial guesses in the second step. The objective function is iteratively minimized with respect to five variables for rotation and structure. The solution corresponding to the global minimum is used to obtain least squares estimates of the remaining unknowns, for translation and rotation center. It was experimentally found that the objective function converges well so that the 3-D space need not be densely searched. Results are presented for three image sequences, two simulated and one real.>