It is important to understand the expiration date. However, it is challenging for machines to understand it. Most previous methods recognize expiration dates in limited conditions. To address this problem, a generalized framework for detecting and understanding expiration dates has been proposed. This framework handles challenging cases and distinguishes 13 different date formats. Unlike previous methods, a neural network-based date parser is adopted in the framework to understand the meaning of an expiration date by identifying the day, month, and year. The experimental results demonstrate the proposed framework achieves 97.74% recognition accuracy for expiration dates in various formats and challenging cases. Since there is no publicly available dataset of expiration dates, a novel dataset collection named ExpDate was created and opened.

We present a rapid and fully automatic 3D avatar creation system that can produce personalized 3D avatars within two minutes using a single depth camera and a motorized turntable. The created 3D avatar is able to make all the details of facial expressions and whole body motions including fingers. To our best knowledge, it is the first completely automatic system that can generate realistic 3D avatars in the common 3D file format, which is ready for the direct use in virtual reality applications or various services.

We present a depth-based fingertip recognition method for interactive projectors. We use a depth camera attached to a projector, so it is possible to change the relative pose between the projector and the projection surface without manual recalibration. For detection and classification of fingertips, we propose using cascaded random forests boosted by our 3-D pose-normalized pixel-difference features. The ensemble probabilities from the cascaded random forests are used to define a score function of a subset of detected fingertips. By finding the subset maximizing the score function, the fingertips in the subset are correctly classified, and the remaining incorrectly detected fingertips are rejected. Experiments show that the proposed method outperforms conventional random forest and convolutional neural network classifiers. In addition, our developed applications show the advantage of the proposed method in assigning different roles to different fingers.

Real world objects can be characterized effectively by their shape, color and material types. Material recognition of an arbitrary object at a distance is an important task for the improvement of object recognition, scene understanding, realistic rendering and various virtual and augmented reality applications. Researchers have tried to recognize material types based on color features, however material type of an object is not completely correlated with its visual appearance. In this paper, we propose a simple but effective surface roughness estimation method using single time-of-flight (ToF) camera. A set of features extracted from the estimated roughness together with conventional color features are used for material type recognition. Experimental results on our material data set with 122 subjects show promising material type recognition results.

In this paper, we present a unified approach for Expectation Maximization (EM) based motion segmentation, estimation and analysis from dense point cloud data. When identifying an underlying motion, literature mainly focuses on three related topics: motion segmentation, estimation and analysis. These topics are, however, mostly considered separately while integrated approaches are rare. Our approach specifically focuses on analyzing articulated motion from dense point cloud data by simultaneously solving for three topics using an integrated approach. No prior knowledge, such as background regions, number of segments and correspondence, is required since two iterations in this algorithm allow us to seamlessly accomplish integration of the three tasks. The first iteration of the algorithm is performed between segmentation and estimation, followed by the second iteration between motion estimation and analysis. For the first iteration, we propose EM based subspace clustering algorithm. For the second iteration, we simply fuse the motion analysis method from [1] into an iterative motion estimation algorithm. As a result, we can extract label, correspondence and motion of moving objects simultaneously from dense point cloud sequence. In experiment, we validate the performance of the proposed method on both synthetic and real world data.

Reconstructing 3D models given a single-view 2D information is inherently an ill-posed problem and requires additional information such as shape prior or user input.We introduce a method to generate multiple 3D models of a particular category given corresponding photographs when the topological information is known. While there is a wide range of shapes for an object of a particular category, the basic topology usually remains constant.In consequence, the topological prior needs to be provided only once for each category and can be easily acquired by consulting an existing database of 3D models or by user input. The input of topological description is only connectivity information between parts; this is in contrast to previous approaches that have required users to interactively mark individual parts. Given the silhouette of an object and the topology, our system automatically finds a skeleton and generates a textured 3D model by jointly fitting multiple parts. The proposed method, therefore, opens the possibility of generating a large number of 3D models by consulting a massive number of photographs. We demonstrate examples of the topological prior and reconstructed 3D models using photos.

We present a Generalized Deformable Spatial Pyramid (GDSP) matching algorithm for calculating the dense correspondence between a pair of images with large appearance variations. The main challenges of the problem generally originate in appearance dissimilarities and geometric variations between images. To address these challenges, we improve the existing Deformable Spatial Pyramid (DSP) [10] model by generalizing the search space and devising the spatial smoothness. The former is leveraged by rotations and scales, and the latter simultaneously considers dependencies between high-dimensional labels through the pyramid structure. Our spatial regularization in the high-dimensional space enables our model to effectively preserve the meaningful geometry of objects in the input images while allowing for a wide range of geometry variations such as perspective transform and non-rigid deformation. The experimental results on public datasets and challenging scenarios show that our method outperforms the state-of-the-art methods both qualitatively and quantitatively.

We present RGB-D Fusion, a framework which robustly tracks and reconstructs dense textured surfaces of scenes and objects by integrating both color and depth images streamed from a RGB-D sensor into a global colored volume in real-time. To handle failure of the ICP-based tracking approach, KinectFusion, due to the lack of sufficient geometric information, we propose a novel approach which registers the input RGB-D image with the colored volume by pho- tometric tracking and geometric alignment. We demonstrate the strengths of the proposed approach compared with the ICP-based approach and show superior performance of our algorithm with real-world data. I. INTRODUCTION Real-time photo-realistic 3D reconstruction of real world scenes and objects plays an important role for a variety of applications such as robotics, augmented reality, human computer interaction, and entertainment. One of the main requirements in the robot interaction with the environment is the 3D reconstruction and representation of real world environments. With the invention of the low-cost Microsoft Kinect sensor as an representative of a consumer-grade range sensing device, the high-resolution range images aligned with pho- tometric information become relatively easily available. The Kinect sensor opens up new possibilities to build a low- cost 3D modeling system with well-studied range mapping technologies.

This demonstration shows a novel approach for Webizing mobile augmented reality, which uses HTML as its content structure, and its real-life use cases. Insight is a mobile AR Web browser that executes HTML5-based AR applications. By extending physical objects and places with a uniform resource identifier (URI), we could build objects of interest for mobile AR application as document object model (DOM) elements and control their behavior and user interactions through DOM events in standard HTML documents. A new CSS media type is defined to augment virtual objects to the physical objects. In this model, we introduce the concept of PLACE, which is the model of a physical space in which the user can be located. With this approach, mobile AR applications can be seamlessly developed as common HTML documents under the current Web eco-system. The advantages of the webized mobile AR, which is able to utilize all kind of Web resources without reworking, and its high productivity due to the seamless development of AR contents in HTML documents are shown with real-life use cases in various domains, such as shopping, entertainment, education, and manufacturing.

Information presentation techniques are important in augmented reality applications as they are in the traditional desktop user interface (WIMP) and web user interface [6]. This paper introduces view management for a web-based augmented reality mobile platform. We use a webized mobile AR browser called Insight that provides separation of the application logic, including the tracking engine and AR content, so that the view management logic and contents are easy to reuse. In addition, the view management is able to accommodate in-situ context of an AR application.

We present a new framework that reconstructs a 3D surface by incorporating a color image into sparse depth points. Assuming that the image intensity change is highly correlated with the scene geometry, we first generate a planar mesh on the image using color variation, and then, we unproject the triangulated image to the world space by integrating the sparse depth points. A quadric error metric-based mesh simplification method is employed for effective image triangulation, and a non-linear optimization is formulated to estimate the length of the projected ray from the camera center to each vertex of the triangulated image by minimizing the errors between the reconstructed surface and the depth points. Our approach can achieve an accurate 3D surface with smooth planar regions and sharp edges on the object boundaries, and it also outperforms other surface reconstruction methods (which use only depth points) in terms of the accuracy.

This paper presents a content structure to build mobile AR applications in HTML5 to achieve a clean separation of mobile AR contents from their application logic to scale like the web. By extending POIs (Point of Interest) to objects and places with Uniform Resource Identifier (URI), we could build objects of interest for mobile AR application as DOM (Document Object Model) elements and control their behavior and user interactions through DOM events. Using our content structure, a mobile AR applications can be developed as normal HTML documents seamlessly under current web eco-system.

There are growing needs for the 3D environment. The 3D devices, such like 3D television, have already started to be sold in markets to the general public. In other words, it means that 3D environment is becoming accustomed to everything. In the same manner, the ways for the 3D manipulation are also getting more important, and there are lots of ways for the 3D manipulation on large screen, which has been used so far, such like 6DOF tracking devices. The problem, however, is that they require high cost, and appropriate place with good environment should be prepared. In this paper, another easier and cheaper way for the 3D manipulation is introduced. The method which is introduced is a way using a mobile device instead of the expensive trackers. Using the method that this paper shows, it is expected that this method is accustomed to everybody and can be used easily.

We are currently developing a new tour guide service on mobile phone using augmented reality. Among the components for the service, in-situ annotation is one of the most important components for capturing of lots of personal experiences in the form of digital multimedia during his/her trip. Our in-situ annotation system provides several pieces of functionality that include recording MART (Mobile Augmented Reality based Tour) media format, transmitting the recorded media and retrieving the related experiences of others. MART media data should be easy for users to store, interchange and manage. Additionally, it can integrate various types of media, such as video, audio, GPS data, motion, 2D/3D graphics and annotated texts. For this, we define a new format for MART media, called MART-MAF, which can compose MPEG standards and non-MPEG standards for various kinds of media.

In this paper, we present a tele-meeting system which uses an augmented robot agent as the representation of the remote participant. In this system, we augment a 3D volume video of the remote participant over the on-site robot. The robot agent in this system represents a remote user with camera, microphone, and mobility. Using this robot agent, the remote user and the local user can show their appearances and interact with tangible interfaces. This robot agent system can be applied to various tele-meeting applications such as tele-marketing and tele-tutoring. For example, we implemented a tele-marketing system to show the feasibility of the suggested system.

Recent advanced interface technologies allow the user to interact with different spaces such as Virtual Reality (VR), Augmented Reality (AR) and Ubiquitous Computing (UC) spaces. Previously, human computer interaction (HCI) issues in VR, AR and UC have been largely carried out in separate communities. Here, we combine these three interaction spaces into a single interaction space, called Tangible Space. We propose the VARU framework which is designed for rapid prototyping of a tangible space application. It is designed to provide extensibility, flexibility and scalability. Depending on the available resources, the user could interact with either the virtual, physical or mixed environment. By having the VR, AR and UC spaces in a single platform, it gives us the possibility to explore different types of collaboration across the different spaces. As a result, we present our prototype application which is built using the VARU framework.

This paper presents the robotic security service with interaction between human and robot. First, this paper addresses the concept of virtual URS. Second, this paper proposes an algorithm for modeling of virtual URS for interacting with human and robot. Third, the system architecture for interaction service is described considering the platform of PC, PDA and Mobile phone. Then, this paper introduces a case of interaction service on WIPI-based mobile phone.

The UPnP (universal plug and play) defines an architecture for pervasive peer-to-peer network connectivity of intelligent appliances. It shares the service oriented architecture with emerging Web service technology, and has many advantages for future robot middleware such as automatic discovery of services and accommodation of dynamic distributed computing environment. However, the UPnP needs some additional features for being used as a robot middleware. This paper discusses them, and presents some requirements when developing a UPnP SDK for robot middleware as well. This paper also presents an experimental result of applying the UPnP to robot components

This paper presents a flying display system using an autonomous blimp (small indoor airship) with a visual tracking system and an image projection system. The real-time visual tracking system tracks the blimp while it flies along a given spatial path to follow a wall. The image projection system projects still images or a video stream, whose rectangular shape is pre-compensated to look natural and flat using an image warping algorithm, on the surface of the blimp. The blimp is designed to have holonomic dynamics and it can maintain a stable pose and position in the presence of bounded air flow disturbances during the wall following motion. The real-time visual tracking system tracks the blimp and calculates its position in three dimensional space. Finally, we verify the capability of the autonomous blimp, the real-time visual tracking and the image projection system by experiments in the public exhibition environment. We also verify that the system is useful for transferring information and advertising in a crowded public area such as an exhibition hall or a department store

This paper presents an approach to utilize UPnP as a middleware for robots. It describes the advantages of UPnP by comparing UPnP with TAO CORBA that was used in a few robot development projects. In order to consider real situation, we select a sample robot architecture, and examine the possibility of UPnP as a robot middleware with the robot architecture. This paper shows how UPnP architecture can be applied to building a robot in the view of software architecture, message mapping, realtime, network selection, performance, memory footprint, and deployment issues.

In this paper, we propose a new method of removing glasses from a human frontal facial image. We first detect the regions occluded by the glasses and generate a natural looking facial image without glasses by recursive error compensation using PCA reconstruction. The resulting image has no trace of the glasses frame or of the reflection and shade caused by the glasses. The experimental results show that the proposed method provides an effective solution to the problem of glasses occlusion and we believe that this method can also be used to enhance the performance of face recognition systems.

This paper concerns extracting 3D motion information from a 3D input device in real time focused to enabling effective human-computer. interaction. In particular, we develop a novel algorithm for extracting 6 degrees-of-freedom motion information from a 3D input device by employing an epipolar geometry of stereo camera, color, motion, and structure information, free from requiring the aid of camera calibration object. To extract 3D motion, we first determine the epipolar geometry of stereo camera by computing the perspective projection matrix and perspective distortion matrix. We then incorporate the proposed “Motion Adaptive Weighted Unmatched Pixel Count” algorithm performing color transformation, unmatched pixel counting, discrete Kalman filtering, and principal component analysis. The extracted 3D motion information can be applied to controlling virtual objects or aiding the navigation device that controls the viewpoint of a user in virtual reality setting. Since the stereo vision-based 3D input device is wireless, it provides users with a means for more natural and efficient interface, thus effectively realizing a feeling of immersion.

Recently we have built the largest Virtual Reality (VR) theatre in the world for the Kyongju World Culture EXPO 2000. Unlike single user VR systems, the VR theatre is characterized by a single shared screen and controlled by a kind of tightly coupled user inputs from several hundreds of people in the audience. The large computer-generated stereo images by the huge cylindrical screen provide the immersive feeling augmenting the physical audience space with of 3D virtual space. In addition to the visual immersion, the theatre provides 3D audio, vibration and olfactory display as well as keypads for the audience in their seats interactively controlling the virtual environment. This paper introduces the issues raised and addressed during the design of making such a versatile VR theatre, production and presentation of the virtual heritage at Kyongju, one thousand years ago.

We present a 3D tracking method of multi-objects by color and stereo. The results are applied to navigation and manipulation in a virtual environment. We choose a human face and hands as tracking targets for HCI. To extract the area of face and hands in a complex background, we transform an input color image using the GSCD (generalized skin color distribution). Based on the transformed image, we detect the area of face and hands by WUPC. Furthermore, the face and hands candidate region are estimated by a Kalman filter in the following frame. Then we can extract the depth information about the only corresponding region, respectively, by stereo matching. Finally, we can navigate and manipulate objects based on the extracted 3D information of face and hands naturally in the virtual environment without any additional device.

Recently we have built a VR(Virtual Reality) theater in Kyongju, Korea. It combines the advantages of VR and IMAX theater. The VR theater can be characterized by a single shared screen and by multiple inputs from several hundreds of people. In this case, multi-user interaction is different from that of networked VR systems and must be reconsidered. This paper defines the multi-user interaction in such a VR theater as Audience Interaction, and discusses key issues for the implementation of the Audience Interaction. This paper also presents a real implementation example in the Kyongju VR theater.

This paper presents an automatic FDP (Facial Definition Parameters) and FAP (Facial Animation Parameters) generation method from an image sequence that captures a frontal face. The proposed method is based on facial feature tracking without markers on a face. We present an efficient method to extract 2D facial features and to generate the FDP by applying 2D features to a generic face model. We also propose a template matching based FAP generation method. The advantage of this approach is that it can be easily applied to single camera MPEG-4 SNHC encoding systems.

This paper introduces web-based 3D media information system. We first address two promising 3D modeling techniques, i.e., image-based 3D modeling and laser scanning based 3D modeling. Especially, we present two approaches of the image-based 3D modeling. One is an off-line approach using multiview images which is captured with single camera and a robot arm. The another one is an on-line approach that extends a commercial triclops camera system. We also utilize a 3D modeling scheme based on a laser scanner and a 3D reverse modeler. Using our 3D modeling environments, we construct several kinds of 3D models. We also implement web-based 3D media information management and retrieval system using XML data server, which provides services of 3D models. Our web-based 3D media information system has a goal of services of various types of 3D models and contents through WWW, which is currently focused on the development and management of 3D models of Korea cultural heritage.

A new method of constructing 3D maps based on the relative magnification and blurring of a pair of images is presented, where the images are taken at two camera positions of a small displacement. The method, referred to as depth from magnification and blurring, aims at generating a precise 3D map of local scenes or objects to be manipulated by a robot arm with a hand-eye camera. The method uses a single standard camera with telecentric lens, and assumes neither active illumination nor active control of camera parameters. The proposed depth extraction algorithm is simple in computation. Fusing the two disparate sources of depth information, magnification and blurring, the proposed method provides more accurate and robust depth estimation. Experimental results demonstrate the effectiveness of the proposed method.

A new method of extracting depth from blurring and magnification of objects or local scene is presented. Assuming no active illumination, the images are taken at two camera positions of a small displacement, using a single standard camera with telecentric lens. Thus, the depth extraction method is simple in structure and efficient in computation. Fusing the two disparate sources of depth information, magnification and blurring, the proposed method provides more accurate and robust depth estimation. This paper describes the process of various experimentations performed to validate this concept and describes the present work that has been done in that field. The experimental result shows less than 1% error for an optimal depth range. The ultimate aim of this concept would be the construction of dense 3D maps of objects and real time continuous estimation of depth.