In this paper, we discuss input/output architectures for convolutional neural network (CNN)-based cross-view gait recognition. For this purpose, we consider two aspects: verification versus identification and the tradeoff between spatial displacements caused by subject difference and view difference. More specifically, we use the Siamese network with a pair of inputs and contrastive loss for verification and a triplet network with a triplet of inputs and triplet ranking loss for identification. The aforementioned CNN architectures are insensitive to spatial displacement, because the difference between a matching pair is calculated at the last layer after passing through the convolution and max pooling layers; hence, they are expected to work relatively well under large view differences. By contrast, because it is better to use the spatial displacement to its best advantage because of the subject difference under small view differences, we also use CNN architectures where the difference between a matching pair is calculated at the input level to make them more sensitive to spatial displacement. We conducted experiments for cross-view gait recognition and confirmed that the proposed architectures outperformed the state-of-the-art benchmarks in accordance with their suitable situations of verification/identification tasks and view differences.

Segmenting reddish lesions in capsule endoscopy (CE) images is an initial step for further computer-assisted applications such as image enhancement, abnormal measurement/tracking, and so on. In this paper, we propose an automatic segmentation method that is successful even with CE image including unclear reddish lesions. To obtain this, the proposed method seeks good features to discriminate the reddish lesions from normal tissues. For implementations, we first extract only meaningful regions in a CE image through a pre-segmentation step. The proposed features then are extracted for the meaningful regions in stead of the whole image. We approaches segmentation task through considering a statistical operator for the extracted features, that is local mean image. Candidates of the abnormal regions are located in the local mean image with assistants of a diffusion process. Evaluations in the experiments confirm effectiveness of the proposed method with both qualitative and quantitative measurement.

The interpretive skills of medical doctors and medical technologists who examine video capsule endoscopy (VCE) in clinical practice are usually improved through hands-on courses. Such courses require that a large volume of cases be undertaken as part of the training, and they thus consume a considerable amount of the trainees' time. This paper describes an e-learning system that reduces the training time in addition to enhancing the quality of the educational process with regard to reading VCE. To achieve this goal, we focused on organizing training courses in order to appropriate for the laborious conditions that exist when reading VCE. The designed courses help the trainees acquire knowledge of abnormal regions and become familiar with reading VCE before taking examinations under conditions similar to those actual clinical practice. The proposed training modality was developed as an e-learning application on the World Wide Web. Thus, it can be easily extended to a wide range of trainees. In the experiments, 20 participants completed the self-learning training procedures in approximately 3 hours. The proposed system is much faster than conventional hands-on courses, which require a minimum of 8 hours. Furthermore, the trainees' learning performances in the final examinations confirmed that the proposed system is particularly effective for inexperienced examining doctors.

This paper presents an efficient method for automatically segmenting the digestive organs in a Video Capsule Endoscopy (VCE) sequence. The method is based on unique characteristics of color tones of the digestive organs. We first introduce a color model of the gastrointestinal (GI) tract containing the color components of GI wall and non-wall regions. Based on the wall regions extracted from images, the distribution along the time dimension for each color component is exploited to learn the dominant colors that are candidates for discriminating digestive organs. The strongest candidates are then combined to construct a representative signal to detect the boundary of two adjacent regions. The results of experiments are comparable with previous works, but computation cost is more efficient.

High dynamic range images (HDRIs) are needed for capturing scenes that include drastic lighting changes. This paper presents a method to improve the dynamic range of a camera by using a reflective liquid crystal. The system consists of a camera and a reflective liquid crystal placed in front of the camera. By controlling the attenuation rate of the liquid crystal, the scene radiance for each pixel is adaptively controlled. After the control, the original scene radiance is derived from the attenuation rate of the liquid crystal and the radiance obtained by the camera. A prototype system has been developed and tested for a scene that includes drastic lighting changes. The radiance of each pixel was independently controlled and the HDRIs were obtained by calculating the original scene radiance from these results.

This paper describes a method of mirror localization to calibrate a catadioptric imaging system. Even though the calibration of a catadioptric system includes the estimation of various parameters, in this paper we focus on the localization of the mirror. Since some previously proposed methods assume a single view point system, they have strong restrictions on the position and shape of the mirror. We propose a method that uses parallel lights to simplify the geometry of projection for estimating the position of the mirror, thereby not restricting the position or shape of the mirror. Further, we omit the translation process between the camera and calibration objects from the parameters to be estimated by observing some parallel lights from a different direction. We obtain the constraints on the projection and compute the error between the model of the mirror and the measurements. The position of the mirror is estimated by minimizing the error. We also test our method by simulation and real experiments, and finally we evaluate the accuracy of our method.

You can create catadioptric omnidirectional stereovision using several mirrors with a single camera. These systems have interesting advantages, for instance in the case of mobile robot navigation and environment reconstruction. Our paper aims at estimating the” quality” of such stereovision system. What happens when the number of mirrors increases? Is it better to increase the base-line or to increase the number of mirrors? We propose some criteria and a methodology to compare different significant categories (seven): three already existing systems and four new designs that we propose. We also study and propose a global comparison between the best configurations.

This paper describes a new method to automatically calibrate lens distortion of wide-angle lenses. We project structured-light patterns using a flat display to generate a map between the display and the image coordinate systems. This approach has two advantages. First, it is easier to take correspondences of image and marker (display) coordinates around the edge of a camera image than using a usual marker, e.g. a checker board. Second, since we can easily construct a dense map, a simple linear interpolation is enough to create an undistorted image. Our method is not restricted by the distortion parameters because it directly generates the map. We have evaluated the accuracy of our method and the error becomes smaller than results by parameter fitting.

This paper describes a novel system for detecting objects close to our sensor. For real time detection and portability, we have developed a small sensor with compound spherical mirrors. Since an object is projected onto each mirror, our method computes the range by a catadioptric stereo method. Our method creates a lookup table of corresponding points for an infinite range. If an object is close enough to the sensor, the projected points of the object are different from these corresponding points. Thus, our method can detect near objects by taking the differences in intensity of the corresponding points between the images in the mirrors. We show the experimental setup of our sensor and the result for detecting near objects.

Given a set of synchronized video sequences of a dynamic scene taken by different cameras, we address the problem of creating a virtual video of the scene from a novel viewpoint. A key aspect of our algorithm is a method for recursively propagating dense and physically accurate correspondences between the two video sources. By exploiting temporal continuity and suitably constraining the correspondences, we provide an efficient framework for synthesizing realistic virtual video. The stability of the propagation algorithm is analyzed, and experimental results are presented.

We present a new framework for generating personalized video digests from detailed event metadata. In the new approach high level semantic features (e.g., number of offensive events) are extracted from an existing metadata signal using time windows (e.g., features within 16 sec. intervals). Personalized video digests are generated using a supervised learning algorithm which takes as input examples of important/unimportant events. Window-based features are extracted from the metadata and used to train the system and build a classifier that, given metadata for a new video, classifies segments into important and unimportant, according to a specific user, to generate personalized video digests. Our experimental results using soccer video suggest that extracting high level semantic information from existing metadata can be used effectively (80% precision and 85% recall using cross validation) in generating personalized video digests.

This paper presents a framework for providing video digests that are personalized by profiles of individual users. Video contents have meta-data described manually from a set of predefined keywords that have temporal duration. Content profiles are prepared by a provider, which are vectors of the importance value of keywords, and only one should be selected by a user. In addition, a user profile is collected by the user, which has the same components. The importance scores of an image sequence along the time axis can be calculated from a combination of these profiles. Finally, the video clips can be collected as the video digest from the whole contents, which have higher importance scores than a threshold transformed from the length of the user requirement.

We demonstrate that in some situations, perceptual quality can be maintained using an approach based on synthesizing "virtual" images of a scene that match frames from a source video clip. We use this algorithm for interpolation of video frames in the time domain, using a small amount of information to construct an approximation of the original video. Our algorithm is well-suited for the limitations in bandwidth and complexity characteristic of wireless multimedia channels. Since the approach is based on estimating functions of the underlying camera motion parameters, it can capture relationships between image correspondences that extend across many (perhaps hundreds) of video frames. Each interpolated image can be rendered using only a few tens of bytes of side information, and the rendering process itself has low computational requirements. We present experimental results to demonstrate that for certain types of video, our algorithm can give significant perceptual improvement over MPEG-4 coded video at the same low bit rate.

We propose video summarization using reinforcement learning. The importance score of each frame in a video is calculated from the user's actions in handling similar previous frames; if such frames were watched rather than skipped, a high score is assigned. To calculate the score, instead of using raw feature vectors extracted from images, we use feature vectors projected on eigenspace: as a result, we can deal with the features comprehensively. We also give an algorithm that uses the reinforcement learning method to create a personalized video summary. The summarization algorithm is applied to a soccer video to confirm its effectiveness.

The estimation of the parameters of a projective transformation that relates the coordinates of two image planes is a standard problem that arises in image and video mosaicking, virtual video, and problems in computer vision. This problem is often posed as a least squares minimization problem based on a finite set of noisy point samples of the underlying transformation. While in some special cases this problem can be solved using a linear approximation, in general, it results in an 8-dimensional nonquadratic minimization problem that is solved numerically using an 'off-the-shelf' procedure such as the Levenberg-Marquardt algorithm. We show that the general least squares problem for estimating a projective transformation can be analytically reduced to a 2-dimensional nonquadratic minimization problem. Moreover, we provide both analytical and experimental evidence that the minimization of this function is computationally attractive. We propose a particular algorithm that is a combination of a projection and an approximate Gauss-Newton scheme, and experimentally verify that this algorithm efficiently solves the least squares problem.

This paper is concerned with the efficient temporal propagation of correspondences between frames of two video sequences, an integral component of many video processing tasks. The main contribution is a framework for the recursive propagation of these correspondences. The propagation consists of a time update step and a measurement update step. The time update depends only on the dynamics of the rotating source cameras, while the measurement update can be tailored to any member of a general class of image correspondence algorithms. Using these results, the correspondence between points of each frame pair can be propagated and updated in a fraction of the time required to estimate correspondences anew at every frame. We discuss an application of the recursive correspondence propagation framework to the creation of virtual video. Previous virtual view algorithms have been used to generate synthetic video of a static scene, in which objects seem frozen in time. In contrast, the algorithms described here allow the creation of "true" virtual video, in the sense that the synthetic video evolves dynamically along with the scene. While virtual video is our motivating application, the recursive correspondence propagation framework applies to any two-camera video application in which correspondence is difficult and prohibitively time-consuming to estimate by processing frame pairs independently.

A visual tracking system is presented in which a combination of techniques is used to obtain motion features of objects from a video sequence. Further processing of the motion features gives the spatio-temporal trajectories of the objects, that can be used as cues for annotation. The system solves problems found in tracking objects in unconstrained environments, such as in sports games, where there are multiple objects in motion, the camera performs pan, tilt and zoom movements, and there are objects other than the players in the background. Coarse segmentation is performed with multi-class statistical color models, constructed from samples of the representative colors of each team. Motion vectors are computed to find region correspondence between consecutive frames. Background elements are eliminated by using camera motion parameters, and other false matches are detected by analyzing motion pattern consistency. Finally, objects are registered by placing windows centered in each tracked region. An experimental realization was used to test the system for tracking players in a soccer game, but it could had also been used for generating annotation cues of videos from other sports as well.

The paper presents an approach for region based video mosaicing, treating moving objects separately from the background, and with improved ghost-like noise elimination. The mosaic images show the moving objects superimposed over a stationary background. Conventional technologies can reduce the ghost-like noise that occurs from moving objects by using temporal median filtering, but its efficiency depends on the ratio between the speeds of the camera and the moving object. Our technology eliminates these noises more efficiently by using segmented images of a spatio-temporal video sequence. Segmentation is performed using a novel technique that uses different configurations of quad-trees for the initial separation in the split-and-merge process. The segmented images are also used to display tracked moving objects on the panoramic image.

In this paper we present a method for representing and retrieving video sequences based on the domain-specific behavior of objects present in the scenes. The representation includes three parts: (a) Action description, representing the action performed by a single object. (b) Interaction description, which describes interactions between multiple objects and is mapped directly to the event semantics in the content domain. (c) Event Structure, which provides a set of spatial and temporal relationship functions, along with a syntax to define the necessary conditions that a particular interaction should meet. Retrieval is performed by processing Event Structures, interpreting object relationships and selecting the relevant combinations of Action descriptions which match the conditions defined in the Event Structures. We describe an implementation of this system for retrieving scenes of soccer plays from among several soccer video sequences.

Proposes a new approach in which a 3D scene is segmented into free areas and object surfaces using: (1) sparse data which are obtained from trinocular stereo; and (2) occluded edges which can not be seen from at least one of the three cameras. Trinocular stereo is employed not only because it is more reliable for finding correspondences than binocular stereo, but also because it clearly classifies edge segments into 3D edge segments which can be seen from all three cameras and ones which can not be seen from at least one of the three cameras. The approach proposed consists of three steps: (1) to obtain free areas by using 3D edge segments which can be seen from all three cameras; (2) the free areas can be expanded by using edges occluded from at least one of the three cameras; and (3) to determine object surfaces. It is necessary to determine whether the 3D edge segments are occluding boundaries by using occluded edges which can not be seen from one of the cameras. One side of an occluding boundary can be taken to be an object surface. Therefore, object surfaces can be inferred using the occluded edges in trinocular stereo.>