Abstract Automatic detection and counting of vehicles in a video is a challenging task and has become a key application area of traffic monitoring and management. In this paper, an efficient real-time approach for the detection and counting of moving vehicles is presented based on YOLOv2 and features point motion analysis. The work is based on synchronous vehicle features detection and tracking to achieve accurate counting results. The proposed strategy works in two phases; the first one is vehicle detection and the second is the counting of moving vehicles. Different convolutional neural networks including pixel by pixel classification networks and regression networks are investigated to improve the detection and counting decisions. For initial object detection, we have utilized state-of-the-art faster deep learning object detection algorithm YOLOv2 before refining them using K-means clustering and KLT tracker. Then an efficient approach is introduced using temporal information of the detection and tracking feature points between the framesets to assign each vehicle label with their corresponding trajectories and truly counted it. Experimental results on twelve challenging videos have shown that the proposed scheme generally outperforms state-of-the-art strategies. Moreover, the proposed approach using YOLOv2 increases the average time performance for the twelve tested sequences by 93.4% and 98.9% from 1.24 frames per second achieved using Faster Region-based Convolutional Neural Network (F R-CNN ) and 0.19 frames per second achieved using the background subtraction based CNN approach (BS-CNN ), respectively to 18.7 frames per second.

We introduce a method of recovering the shape of a smooth dielectric object using diffuse polarization images taken with different directional light sources. We present two constraints on shading and polarization and use both in a single optimization scheme. This integration is motivated by photometric stereo and polarization-based methods having complementary abilities. Polarization gives strong cues for the surface orientation and refractive index, which are independent of the light direction. However, employing polarization leads to ambiguities in selecting between two ambiguous choices of the surface orientation, in the relationship between the refractive index and zenith angle (observing angle). Moreover, polarization-based methods for surface points with small zenith angles perform poorly owing to the weak polarization. In contrast, the photometric stereo method with multiple light sources disambiguates the surface normals and gives a strong relationship between surface normals and light directions. However, the method has limited performance for large zenith angles and refractive index estimation and faces strong ambiguity when light directions are unknown. Taking the advantages of these methods, our proposed method recovers surface normals for small and large zenith angles, light directions, and refractive indexes of the object. The proposed method is positively evaluated in simulations and real-world experiments.

Recovering the 3D shape of a person from its 2D appearance is ill-posed due to ambiguities. Nevertheless, with the help of convolutional neural networks (CNN) and prior knowledge on the 3D human body, it is possible to overcome such ambiguities to recover detailed 3D shapes of human bodies from single images. Current solutions, however, fail to reconstruct all the details of a person wearing loose clothes. This is because of either (a) huge memory requirement that cannot be maintained even on modern GPUs or (b) the compact 3D representation that cannot encode all the details. In this paper, we propose the tetrahedral outer shell volumetric truncated signed distance function (TetraTSDF) model for the human body, and its corresponding part connection network (PCN) for 3D human body shape regression. Our proposed model is compact, dense, accurate, and yet well suited for CNN-based regression task. Our proposed PCN allows us to learn the distribution of the TSDF in the tetrahedral volume from a single image in an end-to-end manner. Results show that our proposed method allows to reconstruct detailed shapes of humans wearing loose clothes from single RGB images.

Recently, deep neural networks have demonstrated excellent performance in foreground segmentation tasks such as moving object detection and change detection tasks. Various types of neural networks have been proposed, however, the previous works mainly discuss the accuracy. Analytics of the neural networks is important to utilize them effectively and improve their performance. In this paper, we investigate a foreground segmentation network and background subtraction network. In our analysis, we discuss differences of behaviors of the two networks in specific scenes and feature distributions in each layer of a background subtraction network to investigate feature learning. In addition, we provide suggestions about the comparison with these networks.

Calibration of relative poses between cameras is a challenging problem, known as extrinsic calibration, for nonoverlapping cameras that do not share the field of view. We propose a method for calibrating non-overlapping in-vehicle cameras placed at front, back, left and right positions by using visual SLAM(vSLAM). Our proposal is to calibrate the cameras during the motion of 90-degree backing-up parking on the fly, without using any dedicated calibration equipment. With this motion, the adjacent cameras are able to have the close field of view at different moments. The relative poses can be computed if the maps computed with vSLAM on each camera are merged by using the common structures. Therefore, we propose an efficient calibration framework with this feature. The proposed method is divided into three steps: map reconstruction with vSLAM on each camera, map merging for all the cameras, and extrinsic calibration. Especially, we propose to separately utilize the frames for vSLAM and the ones for the calibration so that the accuracy of vSLAM can be maximized for the calibration. In the evaluation, the calibration was performed in a practical environment to investigate the performance in comparison with the ground truth acquired by using a calibration equipment.

Our objective is to achieve a city where everyone can move safely and comfortably by developing and implementing ICT-based mobile support system at the actual transport hub. Our system uses cameras installed at the transport hub to detect people who have difficulty moving, and notifies this information to the transportation staff in real time to help them move more smoothly. This system makes it possible to aggregate and provide information on places that is useful for COVID-19 measures, such as measuring the congestion of places and the social distances of the people who gather there.

Recently, background subtraction based on deep convolutional neural networks has demonstrated excellent performance in change detection tasks. However, most of the reported approaches require pixel-level label images for training the networks. To reduce the cost of rendering pixel-level annotation data, weakly supervised learning approaches using frame-level labels have been proposed. These labels indicate if a target class is present. Frame-level supervised learning is challenging because we cannot use location information for training the networks. Therefore, some constraints are introduced for guiding foreground locations. Previous works exploit prior information on foreground sizes and shapes. In this work, we propose two constraints for weakly supervised background subtraction networks. Our constraints use binary mask images generated by simple background subtraction. Unlike previous works, our approach does not require prior information on foreground sizes and shapes. Moreover, our constraints are more suitable for change detection tasks. We also present an experiment verifying that our constraints can improve foreground detection accuracy compared to other methods, which do not include them.

This paper presents a 3D positioning system based on one-handed thumb interactions for simple 3D annotation placement with a smart-phone. To place an annotation at a target point in the real environment, the 3D coordinate of the point is computed by interactively selecting the corresponding points in multiple views by users while performing SLAM. Generally, it is difficult for users to precisely select an intended pixel on the touchscreen. Therefore, we propose to compute the 3D coordinate from multiple observations with a robust estimator to have the tolerance to the inaccurate user inputs. In addition, we developed three pixel selection methods based on one-handed thumb interactions. A pixel is selected at the thumb position at a live view in FingAR, the position of a reticle marker at a live view in SnipAR, or that of a movable reticle marker at a freezed view in FreezAR. In the preliminary evaluation, we investigated the 3D positioning accuracy of each method.

This paper presents SegmentedFusion, a method possessing the capability of reconstructing non-rigid 3D models of a human body by using a single depth camera with skeleton information. Our method estimates a dense volumetric 6D motion field that warps the integrated model into the live frame by segmenting a human body into different parts and building a canonical space for each part. The key feature of this work is that a deformed and connected canonical volume for each part is created, and it is used to integrate data. The dense volumetric warp field of one volume is represented efficiently by blending a few rigid transformations. Overall, SegmentedFusion is able to scan a non-rigidly deformed human surface as well as to estimate the dense motion field by using a consumer-grade depth camera. The experimental results demonstrate that SegmentedFusion is robust against fast inter-frame motion and topological changes. Since our method does not require prior assumption, SegmentedFusion can be applied to a wide range of human motions.

This paper presents a method for localizing primitive shapes in a dense point cloud computed by the RGB-D SLAM system. To stably generate a shape map containing only primitive shapes, the primitive shape is incrementally modeled by fusing the shapes estimated at previous frames in the SLAM, so that an accurate shape can be finally generated. Specifically, the history of the fusing process is used to avoid the influence of error accumulation in the SLAM. The point cloud of the shape is then updated by fusing the points in all the previous frames into a single point cloud. In the experimental results, we show that metric primitive modeling in texture-less and unprepared environments can be achieved online.

We propose a texture synthesis method to enhance the trackability of a target planar object by embedding natural features into the object in the object design process. To transform an input object into an easy-to-track object in the design process, we extend an inpainting method for naturally embedding the features into the texture. First, a feature-less region in an input object is extracted based on feature distribution based segmentation. Then, the region is filled by using an inpainting method with a feature-rich region searched in an object database. By using context based region search, the inpainted region can be consistent in terms of the object context while improving the feature distribution.

Finding high-level semantic information from a point cloud is a challenging task, and it can be used in various applications. For instance, it is useful to compactly represent the scene structure and efficiently understand the scene context. This task is even more challenging when using a hand-held monocular visual SLAM system that outputs a noisy sparse point cloud. In order to tackle this issue, we propose an incremental primitive modeling method using both geometric and statistical analyses for such point cloud. The main idea is to select only reliably-modeled shapes by analyzing the geometric relationship between the point cloud and the estimated shapes. Besides that, a statistical evaluation is incorporated to filter wrongly-detected primitives in a noisy point cloud. As a result of this processing, our approach largely improved precision when compared with state of the art methods. We also show the impact of segmenting and representing a scene using primitives instead of a point cloud.

Recently, deep neural networks (DNNs) have demonstrated excellent performance for change detection. The DNN-based background subtraction automatically discovers background features from datasets and outperforms traditional background modeling based on handcraft features and/or subtraction strategies. Most researchers mainly discuss the accuracy of foreground detection and do not analyze how and why the DNN works well for change detection tasks. It is necessary to understand what the DNN learns as background features in order to discuss the potential of the DNN in background subtraction. In this paper, we focus on the filters in the first convolution layer and the activations of neurons in the last fully connected layer to understand the behavior of the DNN. From the experiment, we found that 1) the first layer performs the role of background subtraction using several filters, and 2) the last layer categorizes some background changes into a group without supervised signals. These findings suggest the possibility of a new background modeling strategy based on data-driven extracted features.

We present a fast and accurate method for reconstructing surfaces of revolution (SoR) on 3D data and its application to structural modeling of a cluttered scene in real-time. To estimate a SoR axis, we derive an approximately linear cost function for fast convergence. Also, we design a framework for reconstructing SoR on dense SLAM. In the experiment results, we show our method is accurate, robust to noise and runs in real-time.

We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our framework is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is tracked using a blendshape representation while the fine geometric details are captured using a Bump image mapped over the template mesh. We propose an efficient algorithm to grow and refine the 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion tracking and 3D head modeling results on a wide range of subjects with various head poses and facial expressions. Our proposed method offers interesting possibilities for animation production and 3D video telecommunications.

Several recent studies in compressive video sensing have realized scene capture beyond the fundamental trade-off limit between spatial resolution and temporal resolution using random space-time sampling. However, most of these studies showed results for higher frame rate video that were produced by simulation experiments or using an optically simulated random sampling camera, because there are currently no commercially available image sensors with random exposure or sampling capabilities. We fabricated a prototype complementary metal oxide semiconductor (CMOS) image sensor with quasi pixel-wise exposure timing that can realize nonuniform space-time sampling. The prototype sensor can reset exposures independently by columns and fix these amount of exposure by rows for each 8×8 pixel block. This CMOS sensor is not fully controllable via the pixels, and has line-dependent controls, but it offers flexibility when compared with regular CMOS or charge-coupled device sensors with global or rolling shutters. We propose a method to realize pseudo-random sampling for high-speed video acquisition that uses the flexibility of the CMOS sensor. We reconstruct the high-speed video sequence from the images produced by pseudo-random sampling using an over-complete dictionary. The proposed method also removes the rolling shutter effect from the reconstructed video.

Background modeling and subtraction are essential to video surveillance applications. There are two main issues related to background modeling: how to initialize the background model, and how to update the model based on observations. In this paper, we consider the first issue with the aim of generating a clear background image that does not contain foreground objects or noise. We used a bidirectional analysis and consensus voting strategy to achieve this goal. We demonstrated the effectiveness of our technique using open access datasets.

The photometric stereo method enables estimation of surface normals from images that have been captured using different but known lighting directions. The classical photometric stereo method requires at least three images to determine the normals in a given scene. However, this method cannot be applied to dynamic scenes because it is assumed that the scene remains static while the required images are captured. In this work, we present a dynamic photometric stereo method for estimation of the surface normals in a dynamic scene. We use a multi-tap complementary metal-oxide-semiconductor (CMOS) image sensor to capture the input images required for the proposed photometric stereo method. This image sensor can divide the electrons from the photodiode from a single pixel into the different taps of the exposures and can thus capture multiple images under different lighting conditions with almost identical timing. We implemented a camera lighting system and created a software application to enable estimation of the normal map in real time. We also evaluated the accuracy of the estimated surface normals and demonstrated that our proposed method can estimate the surface normals of dynamic scenes.

As smartwatches are becoming more widely used in society, gesture recognition, as an important aspect of interaction with smartwatches, is attracting attention. An accelerometer that is incorporated in a device is often used to recognize gestures. However, a gesture is often detected falsely when a similar pattern of action occurs in daily life. In this paper, we present a novel method of designing a new gesture that reduces false detection. We refer to such a gesture as a low-false-positive (LFP) gesture. The proposed method enables a gesture design system to suggest LFP motion gestures automatically. The user of the system can design LFP gestures more easily and quickly than what has been possible in previous work. Our method combines primitive gestures to create an LFP gesture. The combination of primitive gestures is recognized quickly and accurately by a random forest algorithm using our method. We experimentally demonstrate the good recognition performance of our method for a designed gesture with a high recognition rate and without false detection.

We present a fast registration framework with estimating surface normals from depth images. The key component in the framework is to utilize adjacent pixels and compute the normal at each pixel on a depth image by following three steps. First, image gradients on a depth image are computed with a 2D differential filtering. Next, two 3D gradient vectors are computed from horizontal and vertical depth image gradients. Finally, the normal vector is obtained from the cross product of the 3D gradient vectors. Since horizontal and vertical adjacent pixels at each pixel are considered composing a local 3D plane, the 3D gradient vectors are equivalent to tangent vectors of the plane. Compared with existing normal estimation based on fitting a plane to a point cloud, our depth image gradients based normal estimation is extremely faster because it needs only a few mathematical operations. We apply it to normal space sampling based 3D registration and validate the effectiveness of our registration framework by evaluating its accuracy and computational cost with a public dataset.

In this paper, we present a visualization tool for person re-identification when tracking objects across non-overlapping cameras. Tracking objects across non-overlapping cameras is challenging because the observations from different cameras are widely separated in both time and space. Hence, these systems need a large amount of labeled training data. Commonly, this training data is constructed manually at significant human cost. We support this process efficiently by visualizing the correspondences of objects across multiple cameras. Our tool facilitates the construction of a database for person re-identification with ease. Moreover, the accuracy of person re-identification can be increased using the generated database because the amount of training data is increased. In the experiments, we apply the proposed tool to real world situations to verify the validity of the proposed system.

Existing background model based change detection methods have difficulty in distinguishing between foreground and background changes when both changes are caused by the same factors. We explore the possibility of using a light field camera to resolve the problem of existing single-view camera-based approaches. We present a new change detection strategy that processes light rays captured by the light field camera. The light rays are used for three purposes: 1) generating an active surveillance field (ASF) to determine in-focus and out-focus areas, 2) evaluating focusness to determine whether the light rays come from the ASF, and 3) creating and updating light-ray background models to capture temporal changes in light rays. To investigate the effectiveness of the proposed approach, we evaluated several video sequences captured by a light field camera. Experimental results show that our change detection scheme can robustly handle challenging situations that cannot be resolved by existing single-view approaches.

We introduce a method to recover the shape of a smooth dielectric object from polarization images taken with a light source from different directions. We present two constraints on shading and polarization and use both in a single optimization scheme. This integration is motivated by the fact that photometric stereo and polarization-based methods have complementary abilities. The polarization-based method can give strong cues for the surface orientation and refractive index, which are independent of the light direction. However, it has ambiguities in selecting between two ambiguous choices of the surface orientation, in the relationship between refractive index and zenith angle (observing angle), and limited performance for surface points with small zenith angles, where the polarization effect is weak. In contrast, photometric stereo method with multiple light sources can disambiguate the surface orientation and give a strong relationship between the surface normals and light directions. However, it has limited performance for large zenith angles, refractive index estimation, and faces the ambiguity in case the light direction is unknown. Taking their advantages, our proposed method can recover the surface normals for both small and large zenith angles, the light directions, and the refractive indexes of the object. The proposed method is successfully evaluated by simulation and real-world experiments.

The segmentation of transparent objects can be very useful in computer vision applications. However, because they borrow texture from their background and have a similar appearance to their surroundings, transparent objects are not handled well by regular image segmentation methods. We propose a method that overcomes these problems using the consistency and distortion properties of a light-field image. Graph-cut optimization is applied for the pixel labeling problem. The light-field linearity is used to estimate the likelihood of a pixel belonging to the transparent object or Lambertian background, and the occlusion detector is used to find the occlusion boundary. We acquire a light field dataset for the transparent object, and use this dataset to evaluate our method. The results demonstrate that the proposed method successfully segments transparent objects from the background.

We propose a strategy of background subtraction for an image sequence captured by a moving camera. To adapt for camera motion, it is necessary to estimate the relation between consecutive frames in background subtraction. However, simple background subtraction using the relation between consecutive frames results in many false detections. We use re-projection error to handle this problem. The re-projection error has a low value in a background region. According to re-projection error, our method searches neighboring background models and tunes a threshold value for detection in order to reduce false detections. We evaluated the accuracy of detection of our method in experiments. Our method provided better detection than a method that does not search neighboring background models. Our method thus reduced the number of false detections.

In this paper, we investigate the problem of modeling images and associated text for cross-modal retrieval tasks such as text-to-image search and image-to-text search. To make the data from image and text modalities comparable, previous cross-modal retrieval methods directly learn two projection matrices to map the raw features of the two modalities into a common subspace, in which cross-modal data matching can be performed. However, the different feature representations and correlation structures of different modalities inhibit these methods from efficiently modeling the relationships across modalities through a common subspace. To handle the diversities of different modalities, we first leverage the coupled dictionary learning method to generate homogeneous sparse representations for different modalities by associating and jointly updating their dictionaries. We then use a coupled feature mapping scheme to project the derived sparse representations from different modalities into a common subspace in which cross-modal retrieval can be performed. Experiments on a variety of cross-modal retrieval tasks demonstrate that the proposed method outperforms the state-of-the-art approaches.

Nowadays massive amount of images and texts has been emerging on the Internet, arousing the demand of effective cross-modal retrieval such as text-to-image search and image-to-text search. To eliminate the heterogeneity between the modalities of images and texts, the existing subspace learning methods try to learn a common latent subspace under which cross-modal matching can be performed. However, these methods usually require fully paired samples (images with corresponding texts) and also ignore the class label information along with the paired samples. This may inhibit these methods from learning an effective subspace since the correlations between two modalities are implicitly incorporated. Indeed, the class label information can reduce the semantic gap between different modalities and explicitly guide the subspace learning procedure. In addition, the large quantities of unpaired samples (images or texts) may provide useful side information to enrich the representations from learned subspace. Thus, in this paper we propose a novel model for cross-modal retrieval problem. It consists of 1) a semi-supervised coupled dictionary learning step to generate homogeneously sparse representations for different modalities based on both paired and unpaired samples; 2) a coupled feature mapping step to project the sparse representations of different modalities into a common subspace defined by class label information to perform cross-modal matching. Experiments on a large scale web image dataset MIRFlickr-1M with both fully paired and unpaired settings show the effectiveness of the proposed model on the cross-modal retrieval task.

User-provided image tags are usually incomplete or noisy to describe the visual content of corresponding images. In this paper, we consider defective tagging which covers both incomplete and noisy situations, and address the problem of tag completion where tag assignments of training images are defective. While previous studies on tag completion usually assign equal penalty to empirical loss when processing each missing or noisy tag for each image, we show that this may be suboptimal as the relatedness of each tag to each image varies due to the defective setting. Thus, we introduce an image-tag re-weighting scheme to re-weight the penalty term of each tag to each image considering both image similarities and tag associations, and formulate a unified re-weighted empirical loss function. Experimental evaluations show that embedding proposed re-weighted empirical loss function in state-of-the-art tag completion algorithms achieves significant improvement in dealing with defective tag assignments.

Privacy protection in the surveillance video data has received great attention. Although tremendous works have been proposed to provide effective privacy protection techniques, most of the algorithms are based on post-processing that deletes, obscures or encrypts the privacy information after privacy-included raw data are recorded. Consequently, they are vulnerable to raw data leak out, which may lead to unauthorized use. Therefore, it is imperative to develop a new privacy protection scheme which is capable of excluding any privacy information at the video recording phase. In this paper, we propose an anonymous camera aiming to protect the privacy of individuals at the video capturing phase by optical masking technique. It effectively reduces the risk of raw data leakage because no privacy information will be recorded by the camera. We implemented a prototype camera, which consists of an infrared camera, a RGB camera and a liquid crystal on silicon (LCoS) device. We introduce optical design and performance of the anonymous camera, the masking algorithm as well as the calibration methodology. Experimental results demonstrate that our prototype anonymous camera can perform accurate real time masking of the face for privacy protection.

The self-organizing map (SOM) is a traditional neural network algorithm used to achieve feature extraction, clustering, visualization and data exploration. However, it is known that the computational cost of the traditional SOM, used to search for the winner neuron, is expensive especially in case of treating high-dimensional data. In this paper, we propose a novel hierarchical SOM search algorithm which significantly reduces the expensive computational cost associated with traditional SOM. It is shown here that the computational cost of the proposed approach, compared to traditional SOM, to search for the winner neuron is reduced into O(D1 + D2 + ⋯ + DN) instead of O(D1 × D2 × ⋯ × DN), where Dj is the number of neurons through a dimension dj of the feature map. At the same time, the new algorithm maintains all merits and qualities of the traditional SOM. Experimental results show that the proposed algorithm is a good alternate to traditional SOM, especially, in high-dimensional feature space problems.

Current object-recognition algorithms use local features, such as scale-invariant feature transform (SIFT) and speeded-up robust features (SURF), for visually learning to recognize objects. These approaches though cannot apply to transparent objects made of glass or plastic, as such objects take on the visual features of background objects, and the appearance of such objects dramatically varies with changes in scene background. Indeed, in transmitting light, transparent objects have the unique characteristic of distorting the background by refraction. In this paper, we use a single-shot light field image as an input and model the distortion of the light field caused by the refractive property of a transparent object. We propose a new feature, called the light field distortion (LFD) feature, for identifying a transparent object. The proposal incorporates this LFD feature into the bag-of-features approach for recognizing transparent objects. We evaluated its performance in laboratory and real settings.

Background modeling and subtraction is an essential task in video surveillance applications. Most traditional studies use information observed in past frames to create and update a background model. To adapt to background changes, the background model has been enhanced by introducing various forms of information including spatial consistency and temporal tendency. In this paper, we propose a new framework that leverages information from a future period. Our proposed approach realizes a low-cost and highly accurate background model. The proposed framework is called bidirectional background modeling, and performs background subtraction based on bidirectional analysis, i.e., analysis from past to present and analysis from future to present. Although a result will be output with some delay because information is taken from a future period, our proposed approach improves the accuracy by about 30% if only a 33-millisecond of delay is acceptable. Furthermore, the memory cost can be reduced by about 65% relative to typical background modeling.

For the task of image annotation, traditional probabilistic topic models based on Latent Dirichlet Allocation (LDA) [1], assume that an image is a mixture of latent topics. An inevitable limitation of LDA is the inability to model topic correlation since topic proportions of an image are generated independently. Motivated by Correlated Topic Model (CTM) [2] which derives from natural language processing to model topic correlation of a document, we extend the popular LDA based models (corrLDA [3], sLDA-bin [4], trmmLDA [5]) to CTM based models (corrCTM, sCTM-bin, trmmCTM). We present a comprehensive comparison between CTM based and LDA based models on three benchmark datasets, illustrating the superior annotation performance of proposed CTM based models, by means of propagating topic correlation among image features and annotation words.

Gabor-based face representation has achieved enormous success in face recognition. However, one drawback of Gabor-based face representation is the huge amount of data that must be stored. Due to the nonlinear structure of the data obtained from Gabor response, classical linear projection methods like principal component analysis fail to learn the distribution of the data. A nonlinear projection method based on a set of self-organizing maps is employed to capture this nonlinearity and to represent face in a new reduced feature space. The Multiple Self-Organized Gabor Features (MSOGF) algorithm is used to represent the input image using all winner indices from each SOM map. A new local matching algorithm based on the similarity between local features is also proposed to classify unlabeled data. Experimental results on FERET database prove that the proposed method is robust to expression variations.

Background modeling has been widely researched to detect moving objects from image sequences. It is necessary to adapt the background model various changes of illumination condition. A hybrid type of background model which consists of more than one background model has been used for object detection since it is very robust for illumination changes. In this paper, we also propose a new hybrid type of background model named "hybrid spatial-temporal background model". Our model consists of two different kinds of background models. One is pixel-level background model which is robust for long-term illumination changes. The other is spatial-temporal background model which is robust for short-term illumination changes. Our experimental results demonstrate superiority of our method to some related works.

We propose an approach to recognize time-series gesture partems wilh Hierarchical Self-Organizing Map(HSOM). Dlle of the key issue of the time-series ponem recognition is to absorb the time Variant appropriately and to make clusters which indude the same gesture class. In our approach. we arrange the SOM hierarchically. In each layer of the SOM the time-series pattenrs divided into some periods; postures, gesture elements and gestures. They are learned in each layer of HSOM. For example, postures and learned in the first layer, gesture elements are learned in the second layer and so on. Using the sparse code in the bottom layer, the SOM can perform time invariant recognition of the gesture elemems and gestures.

Hypercolumn model (HCM) is a neural network model previously proposed to solve image recognition problem. In this paper, we propose an improved version of HCM network and demonstrate its ability to solve face recognition problem. HCM network is a hierarchical model based on self-organizing map (SOM) that closely follows the organization of visual cortex and builds an increasingly complex and invariant feature representation. This invariance achieved by alternating between feature extraction and feature integration operation. To improve the recognition rate of HCM, we propose a variable dimension for each map in the feature extraction layer. The number of neurons in each map-side is decided automatically from training data. We demonstrate the performance of the approach using ORL face database.

We propose an approach to improve the accuracy of estimating feature points of human body on a vision-based motion capture system (MCS) by using the Variable-density Self-Organizing Map (VDSOM). The VDSOM is a kind of Self-Organizing Map (SOM) and has an ability to learn training samples incrementally. We let VDSOM learn 3-D feature points of human body when the MCS succeeded in estimating them correctly. On the other hand, one or more 3-D feature point could not be estimated correctly, we use the VDSOM for the other purpose. The SOM including VDSOM has an ability to recall a part of weight vector which have learned in the learning process. We use this ability to recall correct patterns and complement such incorrect feature points by replacing such incorrect feature points with them.

Non-parametric representation of pixel intensity distribution is quite effective to construct proper background model and to detect foreground objects accurately. However, from the viewpoint of practical application, the computation cost of the distribution estimation should be reduced. In this paper, we present fast estimation of the probability density function (PDF) of pixel value using Parzen density estimation and foreground object detection based on the estimated PDF. Here, the PDF is computed by partially updating the PDF estimated at the previous frame, and it greatly reduces the computation cost of the PDF estimation. Thus, the background model adapts quickly to changes in the scene and, therefore, foreground objects can be robustly detected. Several experiments show the effectiveness of our approach.

In this paper, we present a virtual object manipulation tool that can directly control virtual objects using physical objects. Physical objects can be recognized by vision based sensing. Basic manipulations using physical objects are translating, rotating, scaling and replacing with a virtual object. Those manipulations are performed by handling an object, a block. In addition to this, complicated manipulations can be realized by handling two objects with both hands. We have implemented a prototype system and assessed the effectiveness of the system by experimentally comparing it with general modeling tool. The results suggest that our object manipulation system can provide a more effective means of handling than operations in general modeling tool.

We propose a method for create a background model in non-stationary scenes. Each pixel has a dynamic Gaussian mixture model. Our approach can automatically change the number of Gaussians in each pixel. The number of Gaussians increases when pixel values often change because of Illumination change, object moving and so on. On the other hand, when pixel values are constant in a while, some Gaussians are eliminated or integrated. This process helps reduce computational time. We conducted experiments to investigate the effectiveness of our approach.

The Self Organizing Map, or Kohonen SOM, is one of the most widely used neural network paradigm based on unsupervised competitive learning. However, the search algorithm introduced by Kohonen is slow when the size of the map is large. This slowness is caused by seeking about the best matching neuron among "all" the map neurons which tuned to "each" input sample. In this paper, we present a new strategy capable to accelerate the SOM's competition algorithm. Instead of Kohonen SOM strategy, the new approach concerns "only" with the neurons which are aligned along the low-order principal components of the feature space and neglects the rest of neurons. The idea is based on the fact that most of the data variance lie on the low-order principal components of the manifold which often contain the most important features of the data [1] [6]. The new SOM can works effectively as a feature extractor for all kinds of manifolds even in the curved ones. Two data sets are utilized to illustrate how the proposed algorithm reduces the computation efforts (or time) of SOM effectively. For N-dimensions feature space, it is shown here that the computation effort to get the best matching units is reduced to O(D1+D2+...+DN) instead of O(D1× D2× ... × DN), where Diis the number of neurons through the dimension i. Also, under same experimental conditions, our method computation time is less than that of fast DCT by sixth times.

In this paper, we describe a tangible interface for hands-on learning using physical-tangible objects (pattern blocks). A computer recognizes the states of the user-handled objects in real time and it gives the users advice to execute learning tasks if necessary. In our approach, a pattern block is employed as a primitive piece. It helps the users to do direct manipulation in real environments, while displayed events in virtual environments help them to support the learning tasks. Concretely, we have developed the interaction system can support hands-on math activities

In this paper we propose a new appearance based system which consists of two stages: visual speech feature extraction and classification, followed by recognition of the extracted feature, thereby the result is a complete lip-reading system. This lip-reading system employs our Hyper Column Model (HCM) approach to extract and classify the visual features and uses the Hidden Markov Model (HMM) for recognition. This paper addresses mainly the first stage; i.e. feature extraction and classification. We investigate the HCM performance to achieve feature extraction and classification and then compare the performance when replacing HCM with Fast Discrete Cosine Transform (FDCT). Unlike FDCT, HCM could extract the entire features without any loss. Also the experiments have shown that HCM is generally better than FDCT and provides a good distribution of the phonemes in the feature space for recognition purposes. For fair comparison, two databases are exploited with three different sets of resolution for each database. One of these two databases is designed to include shifted and scaled objects. Experiments reveal that HCM is capable of recovering and dealing with such image restrictions whereas the effectiveness of FDCT drops drastically especially for new subjects.

A fundamental task in the pattern recognition field is to find a suitable representation for a feature. We present a new visual speech feature representation approach that combines hypercolumn model (HCM) with HMM to perform a complete lip-reading system. We use HCM to extract visual speech features from the input image. The extracted features are modeled by Gaussian distributions using HMM. The proposed lip-reading system can work under varying lip positions and sizes. All images were captured in a natural environment without using special lighting or lip markers. Experimental results are shown to compare favourably with the results of two reported systems, SOM and DCT based systems. HCM provides better performance than both of these systems.

We have developed an automatic motion generation method for human-like characters. In previous works, we have already developed vision-based human motion capture technique and avatar motion control method toward realization of seamless interaction between users and virtual world. In this paper, we extend the avatar motion control method into augmented motion generation method for the support of animating manipulation tasks.

We propose a concept of real-time human proxy (RHP) for avatar-based interaction systems, which virtualizes a human in the real world in real-time and which lets the virtualized human behave as if he/she was present at a distant place. For evaluating the RHP, we apply it to VEIDL (virtual environment for immersive distributed learning), which is a virtual classroom system. The experimental results shows us that the RHP is useful for avatar-based interaction.

We present human figure control software using vision-based human motion capturing. We have already developed core techniques for vision-based human motion tracking and motion synthesis. We have provided the first version software as an open source kit. In this work, we have developed the second version software with modified motion synthesis. Our software consists of the following software: the vision software (includes 2D blob tracking modules and stereo vision module) and the motion synthesis software (includes skeletal structure reconstruction from a limit number of 3D positions, i.e., physically simulated joint position estimator). This software can be employed for many real-virtual applications such as avatar based video conferencing, virtual human control, and 3D game consoles which need the user postures as input. Our software requires the following hardware: more than two firewire cameras for vision processing and more than one PC for vision processing, motion synthesis, and rendering. In our real-virtual system, this software is also employed for virtual object manipulation interface with augmented avatar representation.

This paper describes an avatar motion control by body postures. Our goal is to do seamless mapping of human motion in the real world into virtual environments. We hope that the idea of direct human motion sensing will be used on future interfaces. With the aim of making computing systems suited for users, we have developed a computer vision based avatar motion control. The human motion sensing is based on skin-color blob tracking. Our method can generate realistic avatar motion from the sensing data. We address our framework to use virtual scene context as a priori knowledge. We assume that virtual objects in virtual environments can afford avatar's action, that is, the virtual environments provide action information for the avatar. Avatar's motion is controlled, based on simulating the idea of affordance extended into the virtual environments.

This paper presents real-time human motion analysis for human-machine interface. In general, man-machine 'smart' interface requires real-time human motion capturing systems without special devices or markers. Although vision-based human motion capturing systems do not use such special devices and markers, they are essentially unstable and can only acquire partial information because of self-occlusion. When we analyze full-body motion, the problem becomes more severer. Therefore, we have to introduce a robust pose estimation strategy to deal with relatively poor results of image analysis. To solve this problem, we have developed a method to estimate full-body human postures, where an initial estimation is acquired by real-time inverse kinematics and, based on the estimation, more accurate estimation is searched for referring to the processed image. The key point is that our system can estimate full-body human postures from limited perceptual cues such as positions of a head, hands and feet, which can be stably acquired by silhouette contour analysis.

A sign language recognition system is required to use information from both global features, such as hand movement and location, and local features, such as hand shape and orientation. We present an adequate local feature recognizer for a sign language recognition system. Our basic approach is to represent the hand images extracted from sign-language images as symbols which correspond to clusters by a clustering technique. The clusters are created from a training set of extracted hand images so that a similar appearance can be classified into the same cluster on an eigenspace. The experimental results indicate that our system can recognize a sign language word even in two-handed and hand-to-hand contact cases.

This paper presents a real-time human figure motion control method using color blob tracking, and human motion synthesis based on real-time inverse kinematics. Our purpose is to do seamless mapping of human motion in the real world into virtual environments. In general, virtual environment applications such as man-machine 'smart' interaction require real-time human full-body motion capturing systems without special devices or markers. However, since such vision-based human motion capturing systems are essentially unstable and can only acquire partial information because of self-occlusion, we have to introduce a robust pose estimation strategy, or an appropriate human motion synthesis based on motion filtering. In this paper, we have demonstrated a real-time and online real-virtual interaction system which realizes human full-body motion capturing and synthesis.

Presents a shape and pose estimation method for 3D multi-part objects, the purpose of which is to easily map objects from the real world into virtual environments. In general, complex 3D multi-part objects cause unwanted self-occlusion and non-rigid motion. To deal with the problem, we employ multiple viewpoint time-varying sequences, since there is enough information to estimate the parameters in the sensory data. In our framework, to minimize the error between the selected image feature points and the estimated model parameters, we employ a model fitting procedure which can adaptively select corresponding pairs. We have demonstrated that our system works well both for single part objects and for multiple-part objects using real image data.

In this paper, we propose a strategy to improve the performance of image reconstruction using a selective attention mechanism in a multi-layered neural network. The selective attention mechanism enables us to use top-down information as high-level and global constraints. The traditional algorithms using regularization techniques are quite sensitive to values of parameters, and it is quite difficult to select their appropriate values, because the algorithms use low-level and local constraints. Our strategy uses high-level and global constraints, and modifies the values of parameters locally and automatically.

Latency, caused by remote memory access and remote procedure call, is one of the most serious problems in massively parallel computers. In order to eliminate the processors' idle time caused by these latencies, processors must perform fast context switching among fine-grain concurrent processes. In this paper, we propose a processor architecture, called Datarol-II, that promotes efficient fine-grain multi-thread execution by performing fast context switching among fine-grain concurrent processes. In the Datarol-II processor, an implicit register load/store mechanism is embedded in the execution pipeline in order to reduce memory access overhead caused by context switching. In order to reduce local memory access latency, a two-level hierarchical memory system and a load control mechanism are also introduced. We describe the Datarol-II processor architecture, and show its evaluation results.>

Article A dataflow language with object-based extension and its implementation on a commercially available parallel machine Share on Authors: Shigeru Kusakabe Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, Japan Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, JapanView Profile , Taku Nagai Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, Japan Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, JapanView Profile , Yoshihiro Yamashita Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, Japan Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, JapanView Profile , Rin-ichiro Taniguchi Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, Japan Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, JapanView Profile , Makoto Amamiya Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, Japan Dept.of Information Systems, Kyushu University, Kasuga Fukuoka 816, JapanView Profile Authors Info & Claims ICS '95: Proceedings of the 9th international conference on SupercomputingJuly 1995 Pages 308–317https://doi.org/10.1145/224538.224575Online:03 July 1995Publication History 8citation202DownloadsMetricsTotal Citations8Total Downloads202Last 12 Months0Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In this paper, we describe a method to construct automatically, from a series of images of objects, a model of an object class. The model is described by a set of 2 dimensional features of segmented regions and the relations among them. To make models of objects, we introduce the concept of the segmentation tree-which represents image segmentation at various levels of abstraction-and also a model generation method based on a series of segmentation trees. Using this segmentation tree, we can cope with the problem of diversity of segmentation patterns and can easily make stable object-models.>

An autonomous multiprocessor (AMP) for image processing and computer vision is presented. It provides high-performance image processing. It has many processor elements (PEs) connected to each other through a communication network. Each PE is asynchronously executed by a data flow control mechanism. It is designed on the basis of a circular pipeline architecture and has a mechanism to work like multiple logical PEs. Each pixel of an image is mapped onto one logical PE, and its data is stored in an operand memory register of a logical PE. Image data is stored distributively in the PEs of the system, and required pixel values are transferred to other PEs by inter-PE communication.>

A novel image thresholding method called minimal complexity measure thresholding (MCMT) is presented. It is based on the idea that image thresholding should produce the simplest picture possible, which in turn is based on the idea that a simple pattern is easier for humans to perceive. A hierarchical MCMT (HMCMT) algorithm is also shown; it categorizes portions of each local image area as 'significant' and 'less significant'. The image thresholding operation can be performed by a set of logical operations on the image bit-planes. This operation is called the logical convolution on the bit-planes (LCBP). The DFE (depth-first picture expression) is useful for LCBP operation, because the coded bit-planes are directly processed to produce the threshold pattern in the DFE.>

The depth-first picture expression (DF-expression) is a compact quadtree expression providing a high data compression capability. It is primarily a hierarchically structured data representation for binary pictures. The authors first review the basic ideas of the DF-expression, and its potential data-compression competence. Then the practical problem is studied of how much reduction can be achieved data by converting the expression to a binary coding. It is also shown that there are two directions in the DF-expression strategy for the gray-image coding. One is the bit-plane based, and the other is the three-dimensional (three-valued) DF-expressions. For most of the natural images, the bit-plane based strategy is more effective than the three-dimensional one. The reason for that is discussed.>