This paper explores the impact of an augmented reality (AR) virtual dog, designed to mimic user behavior, on the exercise experience in both solo and group settings. Focusing on the virtual pet's role as a companion during physical activity, we conducted a human-subject experiment comparing three conditions: a mimicking virtual dog, a randomly behaving virtual dog, and no virtual dog. Participants exercised either solo or in groups, specifically in pairs, allowing for a detailed analysis of how the behavior and physical presence of the virtual dog influenced users' exercise experience and social connections. The findings demonstrate that the mimicking virtual dog significantly enhanced the exercise experience, especially in solo settings, by fostering a stronger sense of companionship. In group exercises, the virtual dog acted as a social facilitator, improving group cohesion and interaction. This research highlights the potential of behavior-mimicking virtual pets to enhance both individual and group exercise experiences and offers valuable insights for developing AR-based fitness applications.

The proliferation of personalized video content consumption, amplified by advances in virtual reality (VR) and augmented reality (AR), has introduced new paradigms in media engagement.This paper presents the development and evaluation of “Watch Buddy,” an expressive virtual agent designed to enhance the video-watching experience within an AR environment. The research assesses the impact of the agent’s presence and expressiveness on user satisfaction, technology acceptance, social presence, and emotional intimacy. A comparative user study with 30 participants reveals that the expressive agent significantly improves the viewing experience, as well as the perceived social connection between users and the agent, compared with an inexpressive agent and no agents at all. This paper contributes to VR/AR and human-agent interaction by demonstrating how virtual companions can influence media consumption and foster social connections, particularly for isolated demographics like the elderly or those living alone.

Immersive virtual reality (VR) simulations become more and more popular for basic nursing skills training in a realistic VR environment. However, there is still a gap of research that specifically focuses on pediatric nursing interventions where nurse trainees should be able to recognize and deal with the patient parent’s emotional responses appropriately. In this paper, we propose a novel nursing intervention analysis framework that evaluates the user’s nursing performance, by analyzing not only their momentary multimodal (verbal and nonverbal) behaviors, but also accumulated intervention behaviors that capture the overall nursing context. Based on the proposed framework, we developed an immersive VR-based nursing education system with an emotionally responsive virtual parent, and designed a realistic pediatric nursing intervention scenario in collaboration with a subject-matter expert (a nursing faculty in our university). An expert-evaluation study with professional nurses was conducted to assess the potential of the developed system—in particular, the effects of the emotionally responsive virtual parent— as an effective education tool, by comparing with an emotionally static/neutral virtual parent. Several factors of learning experience, e.g., immersion, realism, learning efficacy, and usefulness, were examined through a subjective questionnaire, and the results support the effectiveness of our system in clinical practice or educational settings. We discuss the findings and implications of our research with the nurse participants’ qualitative feedback, while also addressing possible limitations and future research directions.

This paper presents a framework that enables a user to manipulate his/her face shape three-dimensionally in video. Existing face manipulation applications and methods have some limitations: single photo, manipulation on image domain, or limited deformation. In the proposed framework, face is tracked from video by using landmark tracking and fitting 3d morphable face model to image, and the face model is further deformed according to the user input with mesh deformation method and rendered with texture from the frame image onto the camera preview. Therefore, unlike conventional applications and researches for face manipulation, the proposed framework allows the user to perform free-form 3d deformation of face in video and to view the deformed face at various viewpoints.

In conventional imaging, the geometry of image formation is described by the pinhole camera model, and it allows multiview reconstruction based on the perspective projection. If a light ray passes through different media, however, the pin-hole camera model becomes invalid due to refraction, causing changes in direction. In this paper, we present a method for 3D trajectory reconstruction under refraction at a cylindrical surface. In the proposed method, the refractive geometry is represented by a geometric mapping from image points to refracted rays, instead of physically modeling it. 3D points inside the cylindrical surface are estimated by computing intersection points between segments of corresponding refracted rays in multiview. In our experiments, we demonstrate that 3D trajectories of multiple moving objects inside a transparent cylinder, which is filled with water, are successfully recovered using multiple cameras.

This paper presents a method for interactive deformation of a real object. Our method uses a predefined 3D model of a target object for tracking and deformation. A camera pose relative to the target object is estimated using 3D model-based tracking. Object region in camera image is obtained by projecting the 3D model onto image plane with the estimated camera pose, and a texture map is extracted from the object region and mapped to the 3D model. Then a texture-mapped model is rendered based on a mesh deformed by user via Laplacian operation. Experimental results demonstrate that our method provides user interactions with 3D real objects on real scenes, not augmented virtual contents.

Edge-based tracking is a fast and plausible approach for textureless 3D object tracking, but its robustness is still very challenging in highly cluttered backgrounds due to numerous local minima. To overcome this problem, we propose a novel method for fast and robust textureless 3D object tracking in highly cluttered backgrounds. The proposed method is based on optimal local searching of 3D-2D correspondences between a known 3D object model and 2D scene edges in an image with heavy background clutter. In our searching scheme, searching regions are partitioned into three levels (interior, contour, and exterior) with respect to the previous object region, and confident searching directions are determined by evaluating candidates of correspondences on their region levels; thus, the correspondences are searched among likely candidates in only the confident directions instead of searching through all candidates. To ensure the confident searching direction, we also adopt the region appearance, which is efficiently modeled on a newly defined local space (called a searching bundle). Experimental results and performance evaluations demonstrate that our method fully supports fast and robust textureless 3D object tracking even in highly cluttered backgrounds.

This paper proposes a new mobile mosaic system using the spherical surface for the panoramic image synthesis. The proposed system consists of image acquisition interface, exposure compensation, local image alignment, spherical projection, and blending. The image acquisition interface guides the users to capture images by displaying a wire frame on the spherical surface. The camera pose and direction are estimated by Gyro sensor and accelerometer. We correct the different brightness levels of images using local means and variances in the overlapped regions. Then, the images are locally realigned since the sensor information has much noise. We implement the successive template matching on the spherical surface, which reduces the sensor errors and misalignment in the 3-D rotational motion. After getting the 3-D rotation information of images, the images are rotated in the virtual 3-D space and projected on the spherical surface. We derive the spherical projection using the radius of virtual mosaic sphere and focal length. Finally, the overlapped multiple images are blended only at the boundaries on the spherical surface. We have implemented the system in the usual tablet PC and mobile phone. According to the various experiments, the proposed spherical mosaic system composes the full environment around the user in real-time.

We propose new features and a redundancy-free feature selection scheme for discriminating targets from clutter in high-resolution synthetic aperture radar imagery. These were found to be very useful for discriminating targets from clutter and well combined with various classical discriminative features. If a feature set with a larger dimension is used, its redundancy may increase. In order to reduce this redundancy, a rank-based feature selection scheme is devised. Our discriminating features and feature selection algorithm are evaluated using the moving and stationary target acquisition and recognition (MSTAR) data set.

Hand shape is a natural and human-friendly interface for human-computer interaction. This paper proposes a real- time and 2D vision-based hand shape recognition method. The method is robust to hand pose changes because the hand pose is neutralized after recognizing a hand pose using distance transform, principal component analysis (PCA), and histogram analysis. Also, the context-based recognition method using shape decomposition can effectively recognize tiny changes of fingers. The method worked at 44.8 fps and had a recognition rate of 83% on average in the experiment with 800 images including 5 hand shapes and 16 hand poses.

This paper presents 3-D visual tracking on a mobile phone, which is one of the next promising challenges of visual tracking for mobile augmented reality applications. Our approach is based on model-based tracking using a 3-D model of an object on a target scene. A camera pose is estimated by iteratively updating infinitesimal motions of a camera, explicitly represented by 3-D rigid-body transformations. The 3-D visual tracking is implemented on recent Android smartphones. Its feasibility is shown by experimental results and discussed along with its performance. To our knowledge, this is the first work that demonstrates the 3-D model-based tracking on a mobile phone.

This paper proposes an augmented reality interface that provides natural hand-based interaction with virtual objects on mobile phones. Assume that one holds a mobile phone in a hand and sees the other hand through mobile phone's camera. Then, a virtual object is rendered on his/her palm and reacts to hand and finger movements. Since the proposed interface does not require any additional sensors or markers, one freely interacts with the virtual object anytime and anywhere. The proposed interface worked at 5 fps on a mobile phone (Galaxy S2 having a dual-core processor).

Human, especially human-of-interest, detection and tracking is a necessary part for immersive and realistic human-computer interaction. Thus, we propose an intelligent multi-vision-based human detection and tracking system that consists of a visible camera, an infrared camera, a dynamic infrared lighting system, and a cold mirror. The system segments a human region of interest and tracks his/her motion. Experimental results are given for verifying the usefulness of the proposed method.

To estimate camera pose, existing augmented reality (AR) techniques usually require fiducial markers with known geometry. However, placing a marker in the workspace of the user can be very visually intrusive. To overcome this limitation, a nonintrusive AR method using invisible markers that are created/drawn with an infrared (IR) ink is proposed and an IR marker tracking system is presented. This system includes additional algorithms to maintain reliable performance in a cluttered background. Then, the working conditions of IR markers are examined and compared with those of visual markers, and the optimal working conditions of IR markers are discussed. Next, the qualitative evaluation of an IR marker–based AR is presented through a variety of experiments and user evaluations. Finally, the potential applications of IR marker–based AR are explored.

Abstract Model‐based camera tracking is a technology that estimates a precise camera pose based on visual cues (e.g., feature points, edges) extracted from camera images given a 3D scene model and a rough camera pose. This paper proposes an automatic method for flexibly adjusting the confidence of visual cues in model‐based camera tracking. The adjustment is based on the conditions of the target object/scene and the reliability of the initial or previous camera pose. Under uncontrolled or less‐controlled working environments, the proposed object‐adaptive tracking method works flexibly at 20 frames per second on an ultra mobile personal computer (UMPC) with an average tracking error within 3 pixels when the camera image resolution is 320 by 240 pixels. This capability enabled the proposed method to be successfully applied to a mobile augmented reality (AR) guidance system for a museum. Copyright © 2009 John Wiley & Sons, Ltd.

In projection-based augmented reality (AR) alleviating visual distraction of patterns has been a great challenge. As a representative one, a method of embedding patterns and their complements (hereafter, we call the pairs complementary patterns) into AR images alternately has been proposed recently. This paper presents subjective evaluation results and their statistical analysis on the visual perceptibility of embedding complementary patterns in different ways in a standard hardware environment. Then, we explore the constraints for embedded complementary patterns to be less perceptible. As expected, high projector refresh rate and low pattern strength were the general conditions for a decrease in the perception of embedded complementary patterns. However, reducing pattern size and projecting complementary patterns with an interval were also among the factors affecting the results. Detailed constraints are given in the experimental results. Also, we present which constraint is more dominant for pattern perceptibility.

We propose a nonintrusive method for generating all-focused projection. Unlike traditional restoration algorithms that operate on a blurred image to recover the original, the nature of our problem requires that the correction be applied to the original image before blurring. To accomplish this, a pattern image is generally projected and a series of spatially varying point-spread-functions (PSFs) across the projection input image were estimated from its camera image. In this paper, the pattern image was imperceptibly projected and thus not visible to the observers. Experimental results demonstrate that the proposed method effectively removes the out-of-focus projection blur in dynamic environments where the projector or screen object moves.

Existing geometric and radiometric compensation methods for direct-projected augmented reality focus on static projection surfaces rather than dynamic surfaces (with varying geometry in time). We aim at providing an effective framework for projecting a sequence of augmented reality images onto dynamic surfaces without geometric and radiometric distortion. We present our design of a special pattern image for simultaneous geometric and radiometric compensation and evaluate two different techniques for embedding the pattern image into augmented reality images. The validity of the proposed method is examined through a variety of experiments with a mobile projector-camera system.

Many vision tasks such as scene segmentation, or the recognition of materials within a scene, become considerably easier when it is possible to measure the spectral reflectance of scene surfaces. In this paper, we present an efficient and robust approach for recovering spectral reflectance in a scene that combines the advantages of using multiple spectral sources and a multispectral camera. We have implemented a system based on this approach using a cluster of light sources with different spectra to illuminate the scene and a conventional RGB camera to acquire images. Rather than sequentially activating the sources, we have developed a novel technique to determine the optimal multiplexing sequence of spectral sources so as to minimize the number of acquired images. We use our recovered spectral measurements to recover the continuous spectral reflectance for each scene point by using a linear model for spectral reflectance. Our imaging system can produce multispectral videos of scenes at 30fps. We demonstrate the effectiveness of our system through extensive evaluation. As a demonstration, we present the results of applying data recovered by our system to material segmentation and spectral relighting.

This paper presents separation of specular and diffuse reflection components from an image pair. The proposed approach is based on the dichromatic reflectance model and Markov random field models. The proposed method estimates specular and diffuse components by minimizing observation color noise and prior potential of specular reflectance. The specular reflection component is modelled as an MRF and estimated in maximum a posteriori framework. This paper proposes likelihood term of specular component and the prior model of specular reflectance based on Phong's shading. Some experiments show that the proposed approach separates the specular and diffuse reflection components effectively. The separated specular components can be utilized in image-based lighting which renders a scene with virtual lighting sources.

In direct-projected augmented reality, use of projector makes it possible to utilize 3-D real and large objects as displays and frees from discomforts incidental to wearing a device such as HMD. However, the resulting augmentation usually has poor depth resolution due to projectors with low contrast. In direct-projected augmented reality, the radiometric compensation, which is originally employed to recover the color properties of the projector input image in direct-projected augmented reality, seems to be another cause of decreasing the final contrast of the augmentation. In this paper, a contrast enhancement method is proposed that combines a hue- and saturation-preserving histogram equalization with the radiometric compensation in direct-projected augmented reality. The method guarantees that the radiometrically compensated color is maintained while the brightness (or intensity) contrast is enhanced. Experimental results demonstrate the validity of our method

In projection-based augmented reality, specular reflection may distract the users. This work demonstrates that the specular reflection can be eliminated by redundantly illuminating the projection surface using multiple overlapping projectors mounted at different locations. Our initial system using two projectors is presented. The system automatically determines the pixels which include specular reflection in one projector, blank the light falling on the pixel and boost the other projector's output so that the added light is projected onto the pixel consistently.

This paper deals with disparity estimation based on Markov random field (MRF) models and color coherence. The disparity and line fields are explicitly modeled as MRFs, and are estimated by the stochastic diffusion. The potential functions are defined from the novel stochastic models between disparity and line fields. The color information is also utilized to model the textureless regions where the disparity estimation generally fails. And, the derived potential functions are minimized by the novel energy minimization method called stochastic diffusion. The stochastic diffusion diffuses the potential space using the probability distribution of neighboring fields, and searches for the optimal fields in the converged potential space. Some experiments show good performances of disparity estimation, which are compared with the other methods in a Webpage.

We introduce a novel tracking system based on invisible markers which are created/drawn with an IR fluorescent pen. The tracking system consists of one scene camera, one IR camera, and one half mirror. The two cameras are positioned in each side of half mirror so that their optical centers coincide with each other. We track the invisible markers using the IR camera and visualize AR in the view of the scene camera. Thus, it works as a robust marker-less tracking system. Experimental results are given to demonstrate the viability of the proposed system.

This paper proposes a Bayesian maximum a posteriori (MAP) method to estimate disparity field and to extract objects in the stereoscopic images. The disparity and segmentation fields are modelled as the Markov random fields (MRFs), and are estimated by a stochastic approach called stochastic diffusion. The stochastic diffusion is a new energy minimization method to search for the solution fields in the MAP estimation. A clustering method utilizes the disparity field and color information to classify the regions into foreground objects or background. The line field is also included to improve the detection of the object boundaries. According to the some experiments, the proposed approach shows good performances in the foreground objects extraction even in the complex background.

This paper proposes a stochastic approach to estimate the occlusion and disparity fields of stereoscopic images. The fields are estimated by the Bayesian maximum a posteriori (MAP) framework and Markov random field (MRF) models. The occlusion field model is based on the stochastic observation that the probability distribution in MAP estimator is relatively unstable and uniform at occluded regions. The occlusion is explicitly modelled as MRF, and is estimated in an energy optimization method called stochastic diffusion. The detected occluded region is compensated for the re-estimated disparity field in the same stochastic diffusion and the dynamic programming approach. Experimental results show good occlusion detection and disparity estimation. These results show the novel stochastic approach is suitable for occlusion detection and disparity estimation.

This paper talks about a hierarchical approach on stochastic diffusion in the MAP-based estimation. Stochastic diffusion has been proposed for an optimization method to minimize the potential function in the MAP-based estimation, and showed good performances in the simultaneous estimations of correspondence, line, and segmentation fields. This paper applies stochastic diffusion to the MAP-based estimation of disparity and line fields in the hierarchical scheme. The proposed hierarchical method combines two successive approximations of the disparity field and potential space at the same time. This hierarchical method propagates not only the geometric relation but also interactions of neighborhood fields. The experimental results show that the proposed hierarchical stochastic diffusion decreases the memory and computational burden and improves the estimation performances in the occluded or textureless regions.

The knowledge of wireless ATM cell headers in the uplink data bursts can be used to improve the performance of the concatenated FEC system. This paper proposes a protocol aided error control scheme for wireless ATM. The proposed scheme achieves the performance improvement of about 0.2-1.0 dB at a BER of 10/sup -3/ in the fading channels with respect to the conventional concatenated FEC.

In the usual object segmentation, the main procedure is first motion estimation and then motion clustering to segment a few regions. This paper proposes a new approach to perform the segmentation using MAP-based motion estimation and the cooperative line field model. In this approach, the motion clustering or region classification are not performed to the segmentation, but the line field generates the segmentation of objects. The cooperative line field is based on the gradients of color and the estimated correspondence field in the MAP-based estimator. The stochastic diffusion is an optimization method to search for the minimal potential in the MAP-based estimation. The estimated correspondence field is the depth information in the stereo image, and the final line field is the boundary to distinguish the objects from background or different objects. From the experiments, the estimation of the correspondence field is improved by the line field, and the cooperative line field shows good performances of the whole object segmentation and detailed contour extraction.

This paper proposes the dense correspondence field estimation with the stochastic diffusion based on maximum a posteriori (MAP) estimation. MAP-based correspondence field estimation including line field is derived with reflecting the joint probabilistic distribution function (PDF) of the neighborhoods in the Markov random field (MRF) models, and is applied to the stereoscopic images. The exploits of the joint PDF of the neighborhoods is the main difference from the previous MAP-based algorithms. The segmentation field is introduced in the low activity region, where the intensities are uniform, in order to improve the performance of the correspondence estimation. According to the experiments, the proposed algorithm had good estimation performance with fast convergence. Especially, the line field improved the estimation at the object boundaries, and the segmentation field did effectively in the low activity region.

We derive the general formula of Bayesian model for stereo matching algorithm and implement it with simplified probabilistic models. The probabilistic models use the independence property and similarity between the neighborhood disparities in the configuration. The formula is the generalization of the Bayesian model of stereo matching, and can be implemented in the different forms corresponding to the probabilistic models in the disparity neighborhood system or configuration. We propose a new probabilistic model in order to simplify the joint probability distribution of the disparities in the configuration. According to the experimental results, we can conclude that the derived formula generalizes the Bayesian model for the stereo matching algorithm, and the simplified probabilistic models are reasonable and approximate the pure joint probability distribution very well. Compared with the conventional method of the Bayesian model and the sum of squared difference (SSD) algorithm, furthermore, our proposed algorithm outperforms the others.

In the development of a method of estimating the global camera motion from encoded video, we propose a robust estimation based on an outlier rejection scheme. The extended Kalman filter and the Levenberg-Marquardt method are implemented for the estimation. Both implementations show promising results, but from a computational aspect, the LM method is more efficient than the extended Kalman filter in our implementation.

The concatenation of an equalizer and a Viterbi decoder is a powerful means for improving the performance of mobile radio. The best system performance is obtained when the Viterbi algorithm is used with soft decisions. The soft output equalizer enables the use of soft decision Viterbi decoding. This paper proposes new soft output adaptive equalization techniques based on the SOVA and the modified-MAP algorithm for /spl pi//4-DQPSK mobile radio over multipath fading channels. Simulation results were obtained by using a modified version of the common functional blocks which were used previously by Hamied and Stuber (1996). A complexity analysis is also presented.

We propose a method to generate an arbitrary view based on 5 reference views. We attempt to synthesize a new view of best attainable quality given limited information. The method is based on estimation of depth map and successive image warping: transformation of depth map, guessing depth map for uncovered area, and back-mapping using the filled depth map with elaborate check. Uncovered area due to view change is shown to be handled on the basis of observable viewpoint. It is confirmed that a variety of view of acceptable quality can be synthesized efficiently.

In this paper, we propose a video composition method based on a robust estimation of camera parameters from image sequence. The proposed method is based on an investigation that any camera operation containing a change of scale and a pure 3D rotation can be represented by a 2D geometric transformation. The transformation can explain all the video composition procedure consisting of locating, synchronizing, and operating images. The parameters in the image transformation are estimated from image sequence. The estimation scheme consists of first establishing correspondence and then estimating the parameters by fitting the correspondence data to the transformation model. A robust algorithm based on rejection of outliers is proposed for reliable parameter estimation. Experimental results support the validity of the estimation method.>

The authors describe a method for estimating camera parameters from image sequences for application to emerging model-based video coding systems. The parameters to be estimated include focal length, zoom, and 3-D rotation parameters. The method consists of first establishing a correspondence and then, estimating the parameters by fitting the correspondence data to a transformation model based on a perspective mapping model and a 3-D rotation and zoom operation model. They show by simulations and experiments that the proposed method successfully estimates the focal length from the image sequences, it explains very well the induced motion field of images undergoing camera operation (3-D rotation and zoom), and that it significantly outperforms conventional estimation methods, especially for wide-angled images. It is anticipated that the proposed method will be successfully applied to compensating for the motion field induced by camera operation in extracting a 3-D object model and a 3-D object motion, and to synchronizing the viewing direction and scale of an image, in model-based video coding technology.>