Diminished Reality (DR) has been considered as a sub-technology of Mixed and Augmented Reality (AR/MR). While AR/MR means technologies that add and/or overlay visual information onto images of real scene for providing users to enhance their visual experiences with the added/overlaid information, DR aims the similar enhanced visual experiences by deleting visual information from the images of real scene. Adding and deleting visual information might be considered as same technical issues, but they are actually totally different. In DR, visual information that is hidden by the deleted object should be recovered for filling into the deleted area. This recovery of the hidden area is not required for general adding/overlaying based AR/MR, but should be one of the typical issues for achieving DR. Camera pose estimation and tracking is a typical issue in AR/MR, but the condition of the scene and required performance for DR are not always the same as AR/MR. For example, the object to be diminished/removed should be detected and tracked while the camera is freely moving for DR. In this tutorial, challenging technical issues for DR are addressed, such as recovery of hidden area, detecting and tracking the object to be removed/diminished, tracking camera poses, illumination matching and re-lighting, etc. In addition to those technical issues for DR, a survey of applications of DR, expected futures with DR, and human factors of DR are also presented.

Most 3-D modeling software are difficult for beginners to learn. The operations are often complicated, and the user is required to have prior mathematical knowledge. Therefore, we developed a simple modeling system using ToolDevice to simplify such operations. ToolDevice consists of a set of interaction devices, which use metaphors of real-life hand tools to help users recognize each device's unique functions. Using TweezersDevice, KnifeDevice, and HammerDevice, we developed a mixed reality (MR) 3-D modeling system that imitates real-life woodworking. In the system, TweezersDevice is used to pick up and move objects, while KnifeDevice and HammerDevice are, respectively, used to cut and join virtual objects represented as wood materials. In this study, we describe the motivation for developing the system, the available interactions, and the procedures for creating 3-D models in the system. We also present the results of a user study in which we compare user performance in our system and a common 3-D modeling software. Finally, we discuss the contributions and limitations of this study and future work.

When humans sense the hardness of real objects, their perception is known to be influenced by not only tactile information but also visual information. In a mixed-reality (MR) environment, the appearance of touchable objects can be changed by superimposing a computer-generated image onto them (MR visual stimulation). In our previous study, we first superimposed computer-generated images (CGI) on real objects, and then, applied extreme deformation to the superimposed images while the participants pressed the real objects with their fingers. The results showed that humans sense hardness differently while receiving MR visual stimulation; we named this psychophysical effect the dent-softness illusion. As the next step, in this study, we verify whether humans perceive hardness differently when pressing a real object with and without superimposed animation and whether the dent-softness illusion occurs in different parts of the body such as the finger, palm, and arm.

We focus on a challenging trial of a Look-Change mixed reality (MR) space. Look refers to perceived ambience of an image created by illumination and color tone. Regarding the Look-Change, we have been working on a relighting technique which allows MR space to have additional virtual illumination. Furthermore, we demonstrate a photorealistic approach to create photorealistic virtual objects with free-form polarized reflectometry which requires no specialized hardware and is portable, thus making it easy apply to MR.

Mixed reality (MR) is a technology that merges real and virtual worlds in real time. In MR environments, visual appearance of a real object can be changed by superimposing a virtual object on it. In this study, we focus on the center-of-gravity (COG) and verify the influence of MR visual stimulation on the COG in MR environments. This paper describes the systematic experiments performed to study the influence. The results obtained are interesting: (1) the presence of COG can be changed by MR visual stimulation; (2) although COG differs in vision and force, the presence of COG can be represented by MR visual stimulation under certain conditions; (3) COG perception can also be changed by varying the mass of the real object. We named this psychophysical influence the “Shape-COG Illusion.”

Today's technology enables users to manipulate complex, multi-part 3D virtual objects such as industrial products, structures designed by CAD, and models of the human body in large 3D space. In general modeling software, parts of such a complex 3D virtual objects are grouped and manipulated together, but not individually, for efficient operation. Therefore, a separating operation is necessary when the user wants to observe or manipulate only a part of a complex object. We have developed a system that realizes gesture-based separation and observation of a group of parts within complex virtual objects in 3D space. One of the practical applications of our system is in training, such as learning the structure of the human body or industrial products. In our system, users can separate a group of parts freely by pulling and cutting a virtual rubber band. The widths of these virtual rubber bands are relative to the connection strength between each part. This allows the user to easily understand relationships between parts. Additionally, this feedback provides a comfortable operational familiarity.

PreViz is a computer graphics movie that represents desired scenes in preproduction and is useful for sharing the director's imagination among crews. Therefore, film industry looks upon PreViz as one of the most important processes in filmmaking as the processes become complicated [The Previs Society 2011]. Stereoscopic 3D (S3D) movie production not only changed ways of expressiveness but also changed ways of production and tools. Consequently, it requires more careful planning and PreViz is still important. This paper describes a prototype of S3D MR-PreViz system for HD S3D PreViz shooting using mixed reality (MR) technologies in order to facilitate PreViz making in S3D. This system is designed as an extension of MR-PreViz for classic filmmaking [Ichikari et al. 2010].

In a mixed-reality (MR) environment, the appearance of touchable objects can be changed by superimposing a computer-generated image (CGI) onto them (MR visual stimulation). At the same time, when humans sense the hardness of real objects, it is known that their perception is influenced not only by tactile information but also by visual information. In this paper, we studied the psychophysical influence on the sense of hardness by using a real object that has a CGI superimposed on it. In this experiment, we deform in an extreme way the CGI animation on the real object, while the subject pushes the real object using his/her finger. The results of the experiments found that human subjects sensed different hardnesses by emphasizing the dent deformation of the CGI animation.

In this paper, we present various novel tangible devices suitable for interactions in a mixed reality (MR) environment. They are aimed at making the best use of the features of MR, which allows users to touch or handle both virtual and physical objects. Furthermore, we consider usability and intuitiveness as important characteristics of the interface, and thus designed our devices to imitate traditional tools and help users understand their use.

Many digital painting systems have been proposed and their quality is improving. In these systems, graphics tablets are widely used as input devices. However, because of its rigid nib and indirect manipulation, the operational feeling of a graphics tablet is different from that of real paint brush. We solved this problem by developing the MR-based Artistic Interactive (MAI) Painting Brush, which imitates a real paint brush, and constructed a mixed reality (MR) painting system that enables direct painting on physical objects in the real world.

This is a proposal to the ISMAR community from Japanese AR/MR researchers for the future progress of AR/MR technology. We hope to expand our activities over the ISMAR community and call for international participants who would take part in a number of voluntary works. At the same time, this paper presents a current view of the outcomes of these activities may have be in due course. We will focus here on the various tracking methods, one of the most active themes at the annual ISMAR symposiums. The main goal of our activities is to build a framework to comprehensively evaluate a variety of existing and future tracking methods. Strictly speaking, our targets should include all the geometric registration methods that merge the real and virtual world seamlessly. They can also be termed as real-time 3D matchmove. After initial registration (calibration) of two spaces is achieved, either object tracking or camera tracking is required when the subject or camera moves. In addition to methods suited to static registration, there exist also methods that focus solely on improving the performance of tracking, without calibration. In this paper, when we use the terms "registration and tracking" or simply "tracking" we will be referring to the general definition of tracking as given above.

This paper describes the constructing method of consistent action scene data of CG charactors. The proposed method realizes to construct multiple characters' action scenes by merging one actor's action sequences with keeping time and space consistencies between multiple individually recorded or manually-designed action sequences semi-automatically. This technique is planned as a part of "Mixed Reality Pre-visualizing As Pre-Production Tool in Film-making; MR-PreViz" which can streamline the filmmaking process by overlaying CG action data on the real background image in real time.

This paper describes the influence of visual stimulation on the tactual sense in a mixed-reality environment; i.e., how a tactual impression of a real object is affected by seeing a superimposed image of a different type of material. If the behavior and the extent of such an influence, a sort of illusion, are investigated in detail, the objects composed of a limited variety of materials can be perceived differently. This would be useful in the field of digital engineering. Therefore, we performed various experiments systematically.

There have been many implementations of virtual reality, using audio and visual senses. However, implementations of mixed reality (MR) have thus far only dealt with the visual sense. We have developed an MR system that merges real and virtual worlds in both the audio and visual senses, wherein the geometric consistency of the audio sense was fully coordinated with the visual sense. We tried two approaches for merging real and virtual worlds in the audio sense, using open-air and closed-air headphones.

This paper describes a novel image-based geometric registration method using visually elegant and robust semi-fiducial markers in mixed reality (MR). Most traditional visual markers stand out against the background, i.e., they can be recognized and identified easily. Here, we try to construct new visual markers for MR registration, which achieve a good balance between visual elegance and robustness. As the first step, we propose "two-tone colored markers". These markers have the following characteristics: 1) color similar to that of the background object, 2) placed at the corners of real objects in an inconspicuous manner. This paper describes the registration method using two- tone colored markers and a few experiments.

In this paper, we introduce the outline of "The MR-PreViz Project" performed in Japan. In the pre-production process of filmmaking, Pre Viz, pre-visualizing the desired scene by CGI, is used as a new technique. As its advanced approach, we propose MR-PreViz that utilized mixed reality technology in current PreViz. MR-PreViz makes it possible to merge the real background and the computer-generated humans and creatures in open set or at outdoor location. The user can consider the camerawork and camera blocking efficiently by using MR-PreViz. This paper introduces the outline of the MR-PreViz project, the design of hardware configuration, camera-work authoring method and the results of prototyping.

This paper presents a mixed reality (MR) system with tangible interface as well as visual fusion in an MR space. The sense of touch is given by physical objects on which computer generated imagery is accurately registered and superimposed. The proposed approach is especially useful in industrial design where digital mockups and physical mockups are thoroughly utilized.

This paper describes new vision-based registration methods utilizing not only cameras on a user's head-mounted display but also a bird's-eye view camera that observes the user from an objective viewpoint. Two new methods, the line constraint method (LCM) and global error minimization method (GEM), are proposed. The former method reduces the number of unknown parameters concerning the user's viewpoint by restricting it to be on the line of sight from the bird's-eye view. The other method minimizes the sum of errors, which is the sum of the distance between the fiducials on the view and the calculated positions of them based on the current viewing parameters, for both the user's view and the bird's-eye view. The methods proposed here reduce the number of points that should be observed from the user's viewpoint for registration, thus improving the stability. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments in comparison with methods that use only a user's view camera or a bird's-eye view camera.

This paper describes a platform package, called "MR Platform," which we have been implementing for research and development of augmented reality technology and applications. This package includes a parallax-less stereo video see-through HMD and a software development kit (SDK) for a Linux PC environment. The SDK is composed of a C++ class library for making runtime MR applications and related utilities such as a camera calibration tool. By using the SDK, the following functions are available: capturing video, handling a six degree-of-freedom (DOF) sensor, image processing such as color detection, estimating head position and orientation, displaying the real world image, and calibrating sensor placement and camera parameters of two cameras mounted on the HMD.

This paper presents a method to generate dynamic shading of image-based objects without geometric models. Since conventional rendering techniques cannot be used to render the shading of this type of objects, image processing operation, such as interpolation or combination of image data, is performed. However, the variation of the shading of objects is not linear. Therefore, complicated processing is required to generate the desired shading, and they may not be compatible with real-time systems. Then, to perform the variation of the shading of image-based objects in a real-time system, we adopted a very simple method. From a set of image-based data with several different light conditions we select the image-based data with the light condition that is the closest to that of the rendering time, and use it to reconstruct the image. However, as it is necessary to save a large number of image data in this method, the volume of data is huge. Then we developed a method to reduce the data volume while maintaining real-time rendering. In this paper, we present not only our method, but also its application in the real-time system CyberMirage.

ARGUS is a prototype system which achieves awareness before starting face-to-face communication by utilizing multiple far-end controllable cameras on a broadband network. A user desiring awareness information can change camera directions and angular fields of view at will in order to get the desired image. We call this active control of cameras to aquire awareness information “active awareness.” Camera locations and viewing fields can be quickly grasped by camera icons on the Map Viewer, which shows the actual office layout. ARGUS has a merit to provide both wide area views and close, detailed views via the control of remote cameras. However, the privacy of people may be violated, for example, through extreme magnificaiton. Therefore, it is very important to carefully consider the camera placement and usage. We have modeled office enviroments and have introduced two camera types: private cameras and public cameras. The aim of the former type is to catch a personal view whereas the aim of the latter type is to monitor common spaces. To protect privacy, ARGUS has several levels of access restrictions for each of these camera types. In this video, we describe the policy and implementation of ARGUS from the viewpoints of user interface and privacy.

An expert system for algorithm design for industrial machine vision is presented. Its knowledge base includes human experts' knowledge about image processing techniques, and it is capable of solving given vision problems. As a problem domain, the authors have chosen vision algorithms for a parts feeder, which determines the attitude of mechanical parts on a conveyor belt and rejects parts with inappropriate attitudes. The expert system for a parts feeder is designed to consist of three components: a feature selection expert, an image processing expert, and a decision tree generator. The knowledge necessary for the design of a vision algorithm for determining the attitudes of parts is discussed. A framework for representing the knowledge needed in order to find solutions to pattern classification problems is established.>

An image processing workstation called the vision and image engineering workstation, or VIEW-Station, which as developed in order to merge the state-of-the-art software environment of a UNIX workstation and the computing power of the fast image processor, is described. The software system of the VIEW-Station is constructed hierarchically in order to allow device independence in porting to various hardware configurations and to achieve flexibility in developing application systems. An image computing language called V-Sugar, which provides specific data types and a functional notation, allowing image processing algorithms to be expressed simply, is described. The image memory management mechanism, V-server, and the window system extensions, VIEW-Windows, are described. They are integrated to provide an independent basis to hardware configurations.>

Textural features corresponding to human visual perception are very useful for optimum feature selection and texture analyzer design. We approximated in computational form six basic textural features, namely, coarseness, contrast, directionality, line-likeness, regularity, and roughness. In comparison with psychological measurements for human subjects, the computational measures gave good correspondences in rank correlation of 16 typical texture patterns. Similarity measurements using these features were attempted. The discrepancies between human vision and computerized techniques that we encountered in this study indicate fundamental problems in digital analysis of textures. Some of them could be overcome by analyzing their causes and using more sophisticated techniques.