In this paper we present a comparative study of visual instructions in Immersive Virtual Reality (IVR), i.e., annotation (ANN) that employs 3D texts and objects for instructions and virtual tutor (TUT) that demonstrates a task with a 3D character. The comparison is based on three tasks, maze escape (ME), stretching exercise (SE), and crane manipulation (CM), defined by the types of a unit instruction. We conducted an automated evaluation of user's memory recall performances (recall time, accuracy, and error) by mapping a sequence of user's behaviors and events as a string. Results revealed that ANN group showed significantly more accurate performance (1.3 times) in ME and time performance (1.64 times) in SE than TUT group, while no statistical main difference was found in CM. Interestingly, although ANN showed statistically shorter execution time, the recalling time pattern of TUT group showed a steep convergence after initial trial. The results can be used in the field in terms of informing designers of IVR on what types of visual instruction are best for different task purpose.

Up to now, only limited research has been conducted on crossmodal retrieval of suitable music for a specified video or vice versa. Moreover, much of the existing research relies on metadata such as keywords, tags, or description that must be individually produced and attached posterior. This paper introduces a new content-based, cross-modal retrieval method for video and music that is implemented through deep neural networks. We train the network via inter-modal ranking loss such that videos and music with similar semantics end up close together in the embedding space. However, if only the inter-modal ranking constraint is used for embedding, modality-specific characteristics can be lost. To address this problem, we propose a novel soft intra-modal structure loss that leverages the relative distance relationship between intra-modal samples before embedding. We also introduce reasonable quantitative and qualitative experimental protocols to solve the lack of standard protocols for less-mature video-music related tasks. All the datasets and source code can be found in our online repository (https://github.com/csehong/VM-NET).

Real-world face recognition using a single sample per person (SSPP) is a challenging task. The problem is exacerbated if the conditions under which the gallery image and the probe set are captured are completely different. To address these issues from the perspective of domain adaptation, we introduce an SSPP domain adaptation network (SSPP-DAN). In the proposed approach, domain adaptation, feature extraction, and classification are performed jointly using a deep architecture with domain-adversarial training. However, the SSPP characteristic of one training sample per class is insufficient to train the deep architecture. To overcome this shortage, we generate synthetic images with varying poses using a 3D face model. Experimental evaluations using a realistic SSPP dataset show that deep domain adaptation and image synthesis complement each other and dramatically improve accuracy. Experiments on a benchmark dataset using the proposed approach show state-of-the-art performance.

In this paper we propose an adaptive Augmented Reality interface for hand gestures based on a probabilistic model. The proposed method provides an in-situ interface and the corresponding functionalities by recognizing a context of hand shape and gesture which requires the accurate recognition of static and dynamic hand states. We present an appearance-based hand feature representation that yields robustness against hand shape variations, and a feature extraction method based on the fingertip likelihood from a GMM model. Experimental results show that both context-sensitivity and accurate hand gesture recognition are achieved throughout the quantitative evaluation and its implementation as a three-in-one virtual interface.

In this paper, we propose a sorted consecutive local binary pattern (scLBP) for texture classification. Conventional methods encode only patterns whose spatial transitions are not more than two, whereas scLBP encodes patterns regardless of their spatial transition. Conventional methods do not encode patterns on account of rotation-invariant encoding; on the other hand, patterns with more than two spatial transitions have discriminative power. The proposed scLBP encodes all patterns with any number of spatial transitions while maintaining their rotation-invariant nature by sorting the consecutive patterns. In addition, we introduce dictionary learning of scLBP based on kd-tree which separates data with a space partitioning strategy. Since the elements of sorted consecutive patterns lie in different space, it can be generated to a discriminative code with kd-tree. Finally, we present a framework in which scLBPs and the kd-tree can be combined and utilized. The results of experimental evaluation on five texture data sets--Outex, CUReT, UIUC, UMD, and KTH-TIPS2-a--indicate that our proposed framework achieves the best classification rate on the CUReT, UMD, and KTH-TIPS2-a data sets compared with conventional methods. The results additionally indicate that only a marginal difference exists between the best classification rate of conventional methods and that of the proposed framework on the UIUC and Outex data sets.

Our paper introduces a novel approach for collision avoidance (CA) among social groups that uses the collective gaze-movement angle (CGMA) between groups as the primary approach for collision prediction. Rules are used for handling the CA steering such that it mimics how real human groups travel together in wide-open walk able areas. We also show how to apply reciprocal velocity obstacle (RVO) based CA to groups as an alternative, and we use this alternative as a baseline for comparison in our approach for group-to-group CA. In an immersive evaluation using a head-mounted display, a group of people determined that our CA approach was much more believable than the RVO based CA approach for groups.

Most prior crowd simulations do not have groups of people moving in a social manner. In our work, we use a two-level steering system based on two classes: group agent and pedestrian agent. By interpolating the current and desired slot positions of the group agent according to formation templates, dynamic social group formations can be achieved and can also adapt to the width of passageways by using our proposed ray casting technique. Based on this interpolation approach, we introduce slot-locking which keeps subgroups in a group shoulder-to-shoulder regardless of the current formation state assuming sufficient surrounding space exists. When this happens, hand holding between adjacent members may occur. Individual characteristics such as age and body shape impact the overall group speed. In an immersive evaluation using a head-mounted display (HMD), participants validated the realism of dynamic social group behavior as virtual pedestrians safely navigated via traffic light crosswalks and sidewalks with vehicular traffic encountered on the way.

ABSTRACT This paper proposes a collision avoidance behavior model for crowd simulation based on psychological findings of human behaviors such as gaze movement angle (GMA), side stepping, gait motion, and personal reaction bubble to have better results in crowd simulation. By calculating the GMA between agents, collision can be predicted and avoided without knowing the exact trajectories of the agents. The proposed model consists of four phases: (1) GMA‐based collision prediction for mid/long range by using speed‐variant information process space, (2) collision avoidance steering, (3) gait‐based locomotion generation, and (4) space keeping based on personal reaction bubble. The effectiveness of the proposed speed‐variant information process space was tested on various types of agent flows with different densities. The total loss of kinetic energy accumulated during an agent's movement and the ratio of the length of the path actually traveled to the length of the original path are used as key metrics to figure out the features between the different types of flows. Finally, examples of tuning the parameters with well‐known fundamental diagrams are presented. Copyright © 2013 John Wiley & Sons, Ltd.

Wireless visual sensor networks for surveillance can dissipate their limited resources to process irrelevant images maliciously injected by compromised nodes. In particular, since nodes compromised by tampering reveal their security keys to encrypt messages, traditional data authentication techniques cannot identify false data deceivably encrypted by such stolen keys. To challenge this problem, this paper first presents a surveillance-fitted network model where our false data sensitive protocol efficiently works. The protocol allows malicious messages to travel only one hop by authentically and semantically testing every packet and every image sent from wireless cameras vulnerable to tampering. We additionally suggest a dynamic key scheme which lets wireless sensors employ a different key in each communication for reasonable resource consumption, in order to reduce a possibility of key disclosure itself. Two lemmas and three comparison results verify how well our three approaches outperform their alternatives in resiliency to compromised nodes and memory, computation and communication overheads.

In this paper, a new mobile Augmented Reality (AR) framework which is scalable to the number of objects being augmented is proposed. The scalability is achieved by a visual word recognition module on the remote server and a mobile phone which detects, tracks, and augments target objects with the received information from the server. The server and the mobile phone are connected through a conventional Wi-Fi. In the experiment, it takes 0.2 seconds for the cold start of an AR service initiation on a 10k object database, which is fairly acceptable in a real-world AR application.

ABSTRACT An augmented reality book (AR book) is an application in which such multimedia elements as virtual 3D objects, movie clips, or sound clips are augmented to a conventional book using augmented reality technology. It can provide better understanding about the contents and visual impressions for users. For AR books, this paper presents a markerless tracking method, which recognizes and tracks a large number of pages in real‐time, even on PCs with low computation power. For fast recognition with respect to a large number of pages, we propose a generic randomized forest that is an extension of a randomized forest. In addition, we define the spatial locality of the subregions in an image to resolve the problem of a dropping recognition rate under a complex background. For tracking with minimal jittering, we also propose the adaptive keyframe‐based tracking method, which automatically updates the current frame as a keyframe when it describes the page better than the existing one. Copyright © 2011 John Wiley & Sons, Ltd.

This paper describes a visual odometry algorithm that deals with the nearly degenerated situation caused by a false motion vector generated by independently moving objects, repetitive patterns and wrong depth information that often arise in visual odometry for outdoor service robots. To filter out these false motion vectors, we use temporal and spatial motion vector filter. The temporal motion vector filter uses the previous motion models to filter out abruptly changed motion vectors, and the spatial motion vector filter uses the motion vector's length information and the motion vector's direction information. The direction information of the motion vectors generated by independently moving objects are different from the direction of the vector generated by camera movement in 3D space, and the length information of the motion vector caused by triangulation error is different from the correctly triangulated points. We uses voting scheme to determine primary motion vectors. This algorithm has been tested on a service robot that works in outdoor environment. By using our method, we can deal with independently moving objects and problem caused by repetitive patterns and triangulation errors.

In this paper we propose a sub-window scanning method and a face/non-face classifier using a facial color filter. The scanning method skips over sub-windows that do not contain possible faces based on a facial color membership function. The face/non-face classifier using facial color has low computational cost and can reject non-face sub-windows in an early stage of the cascade face detector. The proposed model reduces the overall computation time of face detection and eliminates false alarms.

There have been many multi‐item auction mechanisms. As it is not allowed for both sellers and buyers to submit bids and offers in nearly all existing multi‐item auction mechanisms, the impartial profit distribution for both sellers and buyers cannot be realized in those mechanisms. Although it is possible for both sellers and buyers to submit bids and offers in a combinatorial exchange, the impartial profit distribution for all individual participants cannot be realized due to its pricing mechanism that focuses only on enforcing budget balance. This paper proposes an iterative multi‐item unit‐demand and unit‐supply double‐auction mechanism with a new winner determination and pricing mechanism for distributing profit impartially from the viewpoints of individual participants regardless of whether they are buyers or sellers. It also analyzes the theoretical features of the proposed auction mechanism, including those related to its strategic behavior and efficiency.

In this paper, a novel method to extend the grayscale histogram equalization (GHE) for color images in a multi-dimension is proposed. Unlike most current techniques, the proposed method can generate a uniform histogram, thus minimizing the disparity between the histogram and uniform distribution. A histogram of any dimension is regarded as a mixture of isotropic Gaussians. This method is a natural extension of the GHE to a multi-dimension. An efficient algorithm for the histogram equalization is provided. The results show that this approach is valid, and a psycho-visual study on a target distribution will improve the practical use of the proposed method.

Abstract In this paper, we propose a new method that processes multiple synchronized video sequences and generates 3D rendering of dynamic objects in the video. It exploits an efficient image‐based reconstruction scheme that constructs and shades 3D models of objects from silhouette images by combining image‐based visual hull and view morphing. The proposed hybrid method improves the speed and the quality of the previous visual hull sampling methods. We designed and implemented a system based on this method which is relatively low cost and does not require any special hardware or specific environment. Copyright © 2003 John Wiley & Sons, Ltd.

Face motion is composed of rigid motion and non-rigid motion. The rigid motion occurs from movements of the human head and the non-rigid motion derives from human's facial expression. In this paper, we present a technique for estimating these rigid/non-rigid motions of the human face simultaneously. First, we test whether the face motion is rigid. If it is rigid motion, we estimate the translation and rotation parameters over the image sequences. Otherwise, the non-rigid motion parameters based on the spring-mass damper model are estimated using optical flow. We separate the rigid motion parameters from the non-rigid parameters for parameters decoupling, so that we can estimate the face motion more accurately and more efficiently. We describe the details of our methods and show their efficacy with experiments.

A navigation system for an autonomous mobile robot working in indoor environments is presented. The system takes advantage of the deliberative (plan-based) approach and the reactive (behavior-based) approach. In the reactive part of the system are local behaviors that are independent, action-generating entities. We also provide higher-level deliberative modules that make interactions manageable so the system can accomplish more meaningful tasks. The deliberative modules control the activation and deactivation of individual local behaviors based on current situations, representing the position of the robot and the path to the goal. The situation is determined by a mapping subsystem consisting of an enhanced topological map, a localization module and a planning module. The use of the explicit world model, especially in a topological manner, makes it possible to reliably localize the robot and plan an efficient path. The paper also provides a detailed description of a localization module based on dead reckoning, and a planning module that selects an efficient and reliable path.

This paper presents a unified approach for the image understanding problem based on the Markov random field models. In the proposed scheme, the image segmentation and interpretation processes cooperate in the simultaneous optimization process so that the erroneous segmentation and misinterpretation can be compensatedly recovered by continuous estimation of the unified energy function.

In this paper, we describe the design and implementation of a desktop conference system, called DeskShare, which provides a distributed and real-time conferencing facility for cooperative work. The main objective of our research is to bring users of remote systems together to collaborate in group work using readily available commodity audio and video technology, and currently used networks based on asynchronous communications, such as Ethernet and FDDI, with adaptability in network congestion. Since DeskShare is implemented as a group of cooperating processes with highly modular fashion based on on object-oriented design technique, we can easily accommodate further system configuration modifications, for example changing video and/or audio codec. Even if DeskShare was originally designed for multimedia desktop conferencing, it is possible to apply it to diverse areas, such as remote education, teleshopping, group design, and remote operation, which are necessary for various group activities.

We describe a tracker that provides real-time visual feedback using on-board low-cost processors. The proposed tracker is based on the two stage visual tracking method (TSVTM) which consists of a real-time kernel, image saver, database, and vision module. The real-time kernel based on the earliest-deadline-first scheduling policy provides the capability of processing tasks with time constraints within the deadline. Image saver takes the responsibility for keeping all the incoming images until they can be processed. The database keeps both the estimated and the predictive location, velocity, intensity, etc. of each region that makes up the target. The vision module consists of two modules: the first stage vision module (FSVM), and the second-stage vision module (SSVM). The FSVM processes the whole image to initially recognize targets using the sophisticated vision algorithms while the SSVM can easily find and track them using the focus-of-attention strategy based on Kalman filter since the SSVM knows the approximated location and useful features of the targets. Combining the above four mechanisms effectively, TSVTM can track targets every one-thirtieth of a second.