In this paper, we propose a Sequencelet Corpus Model (SCM) that is quite versatile for human activity recognition in video. A 'sequencelet', a sequence of actional words, is automatically learned using a data mining approach without any prior knowledge about activity structure and describes a representative partial structure of an activity. We model an activity as a combination of sequencelets in the 'sequencelet corpus' that is a set of learned sequencelets. The SCM does not depend on the type of low-level feature as well as can perform various activity recognition tasks. We evaluate our model on the UT-Interaction dataset and the ActivityNet dataset for activity classification from both fully- and partially-observed video, untrimmed video classification and measuring semantic similarity between activities. The SCM achieved state-of-theart results in activity classification even though the activity is partially observed and promising results in untrimmed video classification, Moreover, by constructing the hierarchy of activities from the similarity computed using the SCM, we demonstrate that the SCM can be used a metric for measuring semantic similarity between activities; these results indicate that the SCM has potential to be an all-around activityrecognition model.

In multi-class indoor semantic segmentation using RGB-D data, it has been shown that incorporating depth feature into RGB feature is helpful to improve segmentation accuracy. However, previous studies have not fully exploited the potentials of multi-modal feature fusion, e.g., simply concatenating RGB and depth features or averaging RGB and depth score maps. To learn the optimal fusion of multimodal features, this paper presents a novel network that extends the core idea of residual learning to RGB-D semantic segmentation. Our network effectively captures multilevel RGB-D CNN features by including multi-modal feature fusion blocks and multi-level feature refinement blocks. Feature fusion blocks learn residual RGB and depth features and their combinations to fully exploit the complementary characteristics of RGB and depth data. Feature refinement blocks learn the combination of fused features from multiple levels to enable high-resolution prediction. Our network can efficiently train discriminative multi-level features from each modality end-to-end by taking full advantage of skip-connections. Our comprehensive experiments demonstrate that the proposed architecture achieves the state-of-the-art accuracy on two challenging RGB-D indoor datasets, NYUDv2 and SUN RGB-D.

We propose Direct Multi-scale Dual-stream network (DMDnet) for pedestrian detection. DMDnet takes a full-size image as input and detects pedestrians of various sizes directly without extracting proposals or using resampling. To improve detection accuracy we adopt dual-stream architecture that combines two types of features that are branched off from two different layers. The lower layer observes the proper sizes of receptive fields depending on the scales of pedestrians; the upper layer contains contextual information. The whole network is trained end-to-end. In experiments DMDnet yields state-of-the-art detection accuracy on the Caltech pedestrian benchmark with new annotation, and has quite fast detection speed.

We propose an automatic framework for quality assessment of a photograph as well as analysis of its aesthetic attributes. In contrast to the previous methods that rely on manually designed features to account for photo aesthetics, our method automatically extracts such features using a pretrained deep convolutional neural network (DCNN). To make the DCNN-extracted features more suited to our target tasks of photo quality assessment and aesthetic attribute analysis, we propose a novel feature encoding scheme, which supports vector machines-driven sparse restricted Boltzmann machines, which enhances sparseness of features and discrimination between target classes. Experimental results show that our method outperforms the current state-of-the-art methods in automatic photo quality assessment, and gives aesthetic attribute ratings that can be used for photo editing. We demonstrate that our feature encoding scheme can also be applied to general object classification task to achieve performance gains.

In this paper, we propose a new halftoning scheme that preserves the structure and tone similarities of images while maintaining the simplicity of Floyd-Steinberg error diffusion. Our algorithm is based on the Floyd-Steinberg error diffusion algorithm, but the threshold modulation part is modified to improve the over-blurring issue of the Floyd-Steinberg error diffusion algorithm. By adding some structural information on images obtained using the Laplacian operator to the quantizer thresholds, the structural details in the textured region can be preserved. The visual artifacts of the original error diffusion that is usually visible in the uniform region is greatly reduced by adding noise to the thresholds. This is especially true for the low contrast region because most existing error diffusion algorithms cannot preserve structural details but our algorithm preserves them clearly using threshold modulation. Our algorithm has been evaluated using various types of images including some with the low contrast region and assessed numerically using the MSSIM measure with other existing state-of-art halftoning algorithms. The results show that our method performs better than existing approaches both in the textured region and in the uniform region with the faster computation speed.

This paper presents a novel motion compensation approach for stabilizing video sequences captured by CMOS image sensors. Although a number of papers have been dedicated to the compensation of shaky camera motions, most of them have focused on CCD image sensors. By relating the homography model to the characteristics of CMOS video sequences, we estimated the dominant motion in image sequences degraded by random shake, simultaneously estimating the distortion parameters of CMOS image sensors. Also, we considered the real-time applications with a fast linear-programming approach for the estimation of camera motion.

This paper presents a practical method that estimates illumination distribution from shadows where the shadows are assumed to be cast on a textured, Lambertian surface. Previous methods usually require that the reflectance property of the surface be constant or uniform, or need an additional image to cancel out the effects of varying albedo of the textured surface. We deal with an estimation problem for which surface albedo information is not available. In this case, the estimation problem corresponds to an underdetermined one. We show that combination of regularization by correlation and some user-specified information can be a practical method for solving the problem. In addition, as an optimization tool for solving the problem, we develop a constrained nonnegative quadratic programming (NNQP) technique into which not only regularization but also user-specified information are easily incorporated. We test and validate our method on both synthetic and real images and present some experimental results.

In this paper, we propose a novel method for the simultaneous localization and mapping (SLAM) problem with two cameras. A single camera based approach suffers from a lack of information for feature initialization and the instability of covariance of the 3D camera location and feature position. To solve this problem, we use two cameras which move independently, unlike the stereo camera. We derive new formulations for the extended Kalman filter and map management of two cameras. We also present a method for the new features initialization and feature matching with two cameras. In our method, the covariance of camera and feature location converges more rapidly. This characteristic enables a reduction of the computational complexity by fixing the feature position whose covariance converges. Experimental results prove that our approach estimates the 3D camera location and feature position more accurately and the covariance of camera and feature location converges more rapidly when compared with the single camera case.

With a monocular view, the nonrigid recovery of 3D motion and time-varying shapes of a deforming object may be impossible without any prior information, because ambiguous, multiple solutions exist for motion and shapes which produce the same projection image. In this paper, as a preceding step to the nonrigid recovery of a deforming object, we develop an approach for estimating the approximate average shape and motion of the object. This reasonably solves the ambiguity problem in nonrigid recovery. By investigating the internal structures of nonrigid objects, we introduce a novel concept, called DoN (Degree of Nonrigidity). Based on this, we propose an iterative certainty reweighted factorization method. In addition, we refine and improve the method by reformulating it in a robust manner to cope with outliers existing in the tracked features. Finally, we present some experimental results on both synthetic data and a real video sequence.

This paper presents a global approach to mosaic blending, reconstructing original scene radiances up to scale, and simultaneously estimating intrinsic camera parameters. By incorporating a geometric constraint of rotating and zooming cameras and a constraint of non-linear mapping functions into robust optimization problems, we ascertained optimal blending parameters for multiple images.

A number of papers have been dedicated to the estimation of the projective transform between a pair of frames. Some of them have focused on the error accumulation problem caused by the concatenation of these transforms. In this paper, a new method that uses sequential graph system is proposed to minimize the error accumulation. And, we propose a robust matching method using sequential shortest path search algorithm and uncertainty measure. Our method is fast and robust. Also, because our system is sequential and uses limited system memory, our method is useful for real time applications.

This paper presents a technique for the detection of roads in a spaceborne synthetic aperture radar (SAR) image using a genetic algorithm (GA). Roads in a spaceborne SAR image can be modeled as curvilinear structures that possess width. Curve segments, which represent the candidate positions for roads, are extracted from the image using a curvilinear structure detector, and the roads are accurately detected by grouping those curve segments. For this purpose, the authors designed a grouping method based on a GA, which is a global optimization method. They combined perceptual grouping factors with it and tried to reduce its overall computational cost by introducing a concept of region growing. In this process, a selected initial seed is grown into a finally grouped segment by the iterated GA process, which considers segments only in a search region. To detect roads more accurately, postprocessing, including noisy curve segment removal, is performed after grouping. The authors applied their method to ERS-1 SAR and SIR-C/X-SAR images that have a resolution of about 30 m. The experimental results show that our method can accurately detect road networks as well as single-track roads and is much faster than a globally applied GA approach.

This paper deals with the problem of estimating structure and motion from long continuous image sequences, applying the expectation maximization algorithm based on an extended Kalman smoother to impose time-continuity of the motion parameters. By repeatedly estimating the state transition matrix of the dynamic equation and the parameters of noise processes in dynamic and measurement equations, this optimization gives maximum likelihood estimates of the motion and structure parameters. Practically, this research is essential for dealing with a long video-rate image sequence with partially unknown system equation and noise. The algorithm is implemented and tested for a real image sequence.

In this paper we introduce a new tool, called a pseudo-distance map (PDM), for extracting skeletons from grayscale images without region segmentation or edge detection. Given an edge-strength function (ESF) of a gray-scale image, the PDM is computed from the ESF using the partial differential equations we propose. The PDM can be thought of as a relaxed version of a Euclidean distance map. Therefore, its ridges correspond to the skeleton of the original gray-scale image and it provides information on the approximate width of skeletonized structures. Since the PDM is directly computed from the ESF without thresholding it, the skeletonization result is generally robust and less noisy. We tested our method using a variety of synthetic and real images. The experimental results show that our method works well on such images.

In this paper, we propose a method for generating realistic shadows of virtual objects inserted into a real video sequence. Our aim is to improve and extend the work of Sato, Sato, and Ikeuchi, (1999), which is based on a static camera, to the case of a video sequence. This extension consists of several procedures: calibration of a moving video camera and a graphic camera, removing false shadows occurring due to a shortcoming of the static camera approach for the estimation of an illumination distribution, and so on. The calibration of the moving camera is solved by camera self-calibration and, with it, we designed a flexible graphic world coordinate system embedding technique called "match move". We also show that the shortcoming of the previous static camera approach is overcome by using information from video sequence. Finally we present the experimental results of a real video sequence.

This paper presents an edge-enhancing super-resolution algorithm using anisotropic diffusion technique. Because we solve the super-resolution problem by incorporating anisotropic diffusion, our technique does more than merely reconstruct a high-resolution image from several overlapping noisy low-resolution images and preserve them. In addition to reducing image noise during the restoration process, our method also enhances edges. We apply this technique to a video stream which can be aligned by 3/spl times/3 projective transformations.

This paper presents a technique for the detection of roads in a spaceborne SAR image using a genetic algorithm. Roads in a spaceborne SAR image can be modelled as curvilinear structures with some thickness. Curve segments, which represent the candidate positions of roads, are extracted from the image using a curvilinear structure detector, and roads are detected accurately by grouping those curve segments. For this purpose, we designed a grouping method based on a genetic algorithm (GA), which is a global optimization method. We combined perceptual grouping factors with it, and tried to reduce its overall computational cost by introducing a concept of region growing. In this process, a selected initial seed is grown into a finally grouped segment by the iterated GA process which considers segments only in a search region. To detect roads more accurately, postprocessing, including noisy curve segment removal, is performed after grouping. We applied our method to ERS-1 SAR images that have a resolution of about 30 meters. The experimental results show that our method can detect roads accurately and is much faster than a globally applied GA approach.

We propose a practical self-calibration method of rotating and zooming cameras. The problem of previous methods occurs when the camera motion is almost all zooming with very little rotation, which we call the degenerate configuration. In that case, the solutions become unstable and rotation angles cannot be calculated. When we adopt a pan-tilt camera (without z-axis rotation) and simplify intrinsic camera parameters, we can overcome the degenerate configuration and we can get a closed form solution. As pan-tilt cameras can be assumed for most stationary cameras (i.e. without translation) and not seem to affect the self-calibration, our method provides a simple, practical solution to the self-calibration problem. In addition, we introduce a nonlinear algorithm that adjusts not only camera parameters but also inter-image homography so that more accurate image registration is made possible. Simulations and experiments with real images are presented.

In this paper we propose an accurate and robust image mosaicing method of soccer video using line tracking and self-calibration. The mosaicing of soccer video is not easy because their playing fields are low textured and moving players are included in the fields. Our approach is to track line features on the playing fields. The line features are detected and tracked using self-calibration technique for rotating and zooming camera. In order to track line features efficiently, we propose a new line tracking algorithm called the camera parameter guided line tracking, which works even when the camera motion undergoes sudden changes. Since we need not know any model for scenes beforehand, the proposed algorithm can be easily extended to other video sources as well as other sports videos. Experimental results show the accuracy and robustness of the algorithm.

This paper deals with video-based augmented reality and proposes an algorithm for augmenting a real video sequence with views of graphics objects without metric calibration of the video camera by representing the motion of the video camera in projective space. A virtual camera, by which views of graphics objects are generated, is attached to a real camera by specifying image locations of the world coordinate system of the virtual world. The virtual camera is decomposed into calibration and motion components in order to make full use of graphics tools. The projective motion of the real camera recovered from image matches has the function of transferring the virtual camera and makes the virtual camera move according to the motion of the real camera. The virtual camera also follows the change of the internal parameters of the real camera. This paper shows the theoretical and experimental results of our application of nonmetric vision to augmented reality.

This paper deals with the uniqueness of the self-calibration of a rotating and zooming camera theoretically. We assume that the principal point and the aspect ratio are fixed but the focal length changes as the camera moves. In this case, at least one inter-image homography is required in order to compute the internal calibration parameters as well as the rotation. We analyze the effects of the deviation of the principal point on the estimation of the focal length and the rotation. The more the camera changes its zoom, the larger the effects are, and the larger the rotation angle is, the smaller the effects are. Thus, we may take the image center as the principal point in practical applications. Experiments using real images are given.

This paper presents an automatic and map-based road detection algorithm for spaceborne SAR images, aiming at subpixel accuracy on average. We propose a two-step algorithm which is composed of the coarse and the fine matching. In the coarse matching step, we find roughly road positions by using water areas or the generalized Hough transform. The fine matching step detects roads accurately by applying the active contour models (snake) to the novel field called the potential field. In the fine matching step, we first extract ridges or ravines of curvilinear structures in SAR images, then construct the potential field by smoothing extracted structures. The proposed method utilizes digital maps to extract information of the road model in the coarse matching and to initialize positions and shapes of snakes in the fine matching. Experimental results show the performance of detecting roads with the average error less than one pixel.

In this paper, we propose a method for locating 3D position of a soccer ball from monocular image sequence of soccer games. Toward this goal, we adopted ground-model-to-image transformation together with physics-based approach, that a ball follows the parabolic trajectory in the air. By using the transformation the heights of a ball can be easily calculated using simple triangular geometric relations given the start and the end position of the ball on the ground. Here the heights of a ball are determined in terms of a player's height. Even if the end position of a ball is not given on the ground due to kicking or heading of a falling ball before it touches the ground, the most probable trajectory can be determined by searching based on the physical fact that the ball follows a parabolic trajectory in the air. We have tested and experimented with a real image sequence the results of which seem promising.

In this paper video streams are augmented by novel views of an object generated by an image-based rendering method. Reference views of the object and video images are captured by general uncalibrated perspective cameras. Our algorithm can be divided into three parts: 1) recovering structure information for reference views and video images; 2) interactive embedding of the model object into video images; and 3) rendering views of the object according to the motion of the video camera. Multiple model views are used to render views as the video camera changes its viewpoint. Generated views of the model are overlayed on the corresponding video images. Experimental results for real video and model images are given. Finally, a discussion on the limitations of the proposed method and subjects for future research are provided.

This paper describes a method for calibrating a stereo-camera using its pure translational motion. The left and right cameras translate in several directions simultaneously. For each camera and each translation, an image sequence is obtained, feature points being extracted and tracked. An epipole is estimated for each image sequence and calibration is carried out using the relation between known translations and estimated epipoles. After finding calibration and rotation matrices of each camera, 3D reconstruction of feature points is performed. Reconstructed points are used to find translational relation between both cameras. The method was tested with real image data, and experimental results show that it is applicable in practice.

This paper reconstructs structure of lines sequentially in projective space. Grassmann (Plucker) coordinates are used in order to represent the structure of 3D lines. Since intrinsic camera parameters are not known and are changed during motion, the reconstruction is only up to a projectivity. Line correspondences are obtained through tracking 2D line segments in a sequence of images. Initial camera matrices are computed using trilinear constraints and Grassmann coordinates are used to represent recovered 3D lines. In the sequence, the Grassmann coordinates are utilized to obtain camera matrices linearly, which are then refined through nonlinear optimization with a constraint among three views. After camera matrix is computed the Grassmann coordinates are updated through IEKF (iterated extended Kalman filter). Performance of the algorithm is shown by experimental results.

In this paper we present a two-stage classifier for the recognition of freight train ID numbers under outdoor environment. This kind of recognition is different from ordinary OCR in that images of numerals are usually corrupted by "noises" caused by dirty materials, varying outdoor illumination, and so on. We focused our attention on developing a robust classifier against such noises. The two-stage classifier is constructed by cascade connection of a main classifier and an auxiliary classifier. Since ID numbers are printed with relatively small size and shape variations, we adopted a template-matching technique in the main classifier. Three top-scored candidates from the main classifier are given to the auxiliary classifier which consists of 10 expert neural networks. These neural nets exploit local characteristics of each numeral. We collected several hundreds of sample images to test our algorithm, and some promising results were obtained.