This paper presents an adaptive low-light image enhancement algorithm for video see-through (VST) augmented reality (AR). The proposed method constructs an illumination map that preserves structural details while suppressing background noise through edge-preserving optimization. Each segmented patch is modeled as an affine transformation of its corresponding input image patch. The resulting illumination map generates an adaptive exposure ratio map, which—combined with light-adaptive parameters—forms an adaptive brightness transform model without requiring multi-exposure inputs. Integrated with the camera response model, the method enhances image quality under varying illumination without additional hardware. Experimental results demonstrate improved brightness, contrast, and clarity across diverse lighting conditions. The algorithm is efficient and suitable for real-time deployment in VST AR systems.

In this paper, an image guided depth point extraction and depth map reconstruction approach is presented for generating high quality depth maps used for Space-Warp functions on XR devices. The approach can be divided into two parts. The first part is for adaptive depth point extraction to obtain sparse depth points which can efficiently represent the original depth map which runs on device side and the second part is for depth map recovery with the extracted depth points and Space-Warp with the recovered depth map which runs on HMD side. In adaptive depth point extraction, feature points, edges, object contours are extracted as a candidate point set which represents fast depth changes. A criterion is defined by depth map, color image, and spatial information to compute confidence of the candidate points. They are integrated together with the confidence map to build a depth point set. In depth map recovery, an image guided depth map reconstruction from the sparse depth points with depth propagation is presented. During depth map reconstruction, it keeps the sparse points unchanged and propagates the depths in the neighborhood points to the considered points guided by the color image and original depth map information. The reconstructed depth map is used for Space-Warp to create a new frame based on the current frame and a predicted head pose for the new frame. The algorithm is applied to XR systems and applications. The results demonstrate the advantages of the proposed approach. We compare results created by our algorithm and generally used approaches including bilinear interpolation and multilevel B-Spline interpolation. The comparison shows that our algorithm can create high-quality new frames with clear object boundaries and less artifacts.

Recently, general-purpose computing on graphics processing units (GPGPU) has been enabled on mobile devices thanks to the emerging heterogeneous programming models such as OpenCL. The capability of GPGPU on mobile devices opens a new era for mobile computing and can enable many computationally demanding computer vision algorithms on mobile devices. As a case study, this paper proposes to accelerate an exemplar-based inpainting algorithm for object removal on a mobile GPU using OpenCL. We discuss the methodology of exploring the parallelism in the algorithm as well as several optimization techniques. Experimental results demonstrate that our optimization strategies for mobile GPUs have significantly reduced the processing time and make computationally intensive computer vision algorithms feasible for a mobile device. To the best of the authors' knowledge, this work is the first published implementation of general-purpose computing using OpenCL on mobile GPUs.

User identification and access control have become a high demand feature on mobile devices because those devices are wildly used by employees in corporations and government agencies for business and store increasing amount of sensitive data. This paper describes SenGuard, a user identification framework that enables continuous and implicit user identification service for smartphone. Different from traditional active user authentication and access control, SenGuard leverages availability of multiple sensors on today's smartphones and passively use sensor inputs as sources of user authentication. It extracts sensor modality dependent user identification features from captured sensor data and performs user identification at background. SenGuard invokes active user authentication when there is a mounting evidence that the phone user has changed. In addition, SenGuard uses a novel virtualization based system architecture as a safeguard to prevent subversion of the background user identification mechanism by moving it into a privileged virtual domain. An initial prototype of SenGuard was created using four sensor modalities including, voice, location, multitouch, and locomotion. Preliminary empirical studies with a set of users indicate that those four modalities are suited as data sources for implicit mobile user identification.

We present a fast and high-quality image stitching approach for creating high-resolution and high-quality panoramic images on mobile devices. In the approach, a sequential panorama stitching procedure is created with color and luminance compensation for reducing differences between source images, fast image labeling for optimal seam finding, and gradient domain image blending for transition smoothing. The color and luminance compensation is performed in the gamma-corrected sRGB color space to avoid pixel saturation problems, optimal seams between adjacent source images are found by dynamic programming optimization for fast speed and less memory, and the source images are blended together by Poisson blending for a high blending quality. We compare results and performance with other approaches to demonstrate the advantages of our approach in mobile panorama applications. The main contribution of this work is studying how applying color and luminance compensation before Poisson blending improves the blending quality and processing speed of Poisson blending in long image sequences where colors and luminance vary between source images, and how it also improves the quality of image labeling. The integration of color correction, fast labeling, and Poisson blending into a sequential panorama stitching procedure allows creation of high-resolution panoramic images with large source images more quickly and using less memory. These are very important aspects for mobile devices that have limited computational power and memory resources.

We propose an effective image segmentation approach with a novel automatic boundary refinement procedure that requires little user interaction and makes the object cutout process more robust and convenient. It achieves these goals by the following three steps. First, merge over-segmented regions according to the maximal similarity rule using a few marking strokes as input. Second, detect possible erroneous low-contrast object boundaries by analyzing image content. Third, automatically refine those boundary regions using both local and global information. Experimental results are good even on very complex images.

This paper addresses the problem of color and luminance compensation for sequences of overlapping images where the source images have very different colors and luminance. We apply the method for panoramic image construction on mobile phones. A simple approach is proposed that minimizes both the color differences of neighboring images and the overall color correction over the whole sequence. We compare several combinations of gamma correction and linear adjustment over different color representations, and select the method with best results: use YCbCr and apply gamma correction for the luminance component and linear correction for the chrominance components

We provide an efficient technique of color and luminance compensation for sequences of overlapping images. It can be used in construction of high-resolution and high-quality panoramic images even when the input images have very different colors and luminance. The technique uses color matching in the overlapping areas of source image pairs to balance colors and luminance in the whole image sequence. It performs gamma correction for the luminance component and linear correction for the chrominance components of source images in the image sequence. Compared to existing approaches, our technique is simple and efficient, yet it can avoid color saturation problems during color correction and perform color and luminance compensation globally in the whole image sequence. We apply the technique to create high-quality panoramic images on mobile phones

Detection of meeting events is one of the most important tasks in multimodal analysis of planning meetings. Speaker detection is a key step for extraction of most meaningful meeting events. In this paper, we present an approach of speaker localization using combination of visual and audio information in multimodal meeting analysis. When talking, people make a speech accompanying mouth movements and hand gestures. By computing correlation of audio signals, mouth movements, and hand motion, we detect a talking person both spatially and temporally. Three kinds of features are extracted for speaker localization. Hand movements are expressed by hand motion efforts; audio features are expressed by computing 12 mel-frequency cepstral coefficients from audio signals, and mouth movements are expressed by normalized cross-correlation coefficients of mouth area between two successive frames. A time delay neural network is trained to learn the correlation relationships, which is then applied to perform speaker localization. Experiments and applications in planning meeting environments are provided.

This paper presents a gradient domain approach for eliminating ghosting artifacts caused by moving objects in panoramic images. In this approach, an eliminating ghosting operation is integrated into a panorama stitching process. For each source image, we pre-stitch it to the panoramic image and perform ghosting detection in the overlapping area. If a ghosting artifact is found, we locate the corresponding moving object in the source image, apply a gradient domain object removing and region filling operation to remove the moving object, and stitch the image again. In the gradient domain operation, a gradient vector field is created with gradients of the source image. The object region is emptied after the object is removed and filled with the best-fit patches found by the region filling process in the pixel color domain and the gradient domain. The updated gradient vector field is used for constructing a Poisson equation. A modified source image can be recovered from the gradient vector field by solving the Poisson equation with boundary conditions. When all source images are processed, the final panoramic image can be obtained and the ghosting artifacts are eliminated. The approach presents several advantages. Since the eliminating ghosting operation is performed in the gradient domain, it is very easy to integrate it with gradient domain image blending to produce high quality panoramic images. It can be applied to a general ghosting artifact case caused by moving objects. The approach is applied for producing mobile panoramic images and eliminating ghosting artifacts. It presents good performance in applications.

We present an image labeling approach for merging a set of aligned source images into a composite image by finding optimal seams in the overlapping areas of the source images quickly and using little memory. A minimal-cost path in the overlapping area of two images is found by dynamic programming and used as an optimal seam to label images. The overlapping images are cut along the seam and merged together. A sequential image stitching procedure is integrated with the fast image labeling for producing high-resolution and high-quality panoramic images using large source images under limited computational and memory resources. The approach presents several advantages: the use of dynamic programming optimization for finding the minimal-cost path over adjacent source images guarantees finding the optimal seam and allows images to be merged quickly; ghosting and blurring problems caused by moving objects and small registration errors can be avoided by the optimal seam finding process; the combination of the sequential image stitching procedure with the fast image labeling allows processing large source images for creating high-resolution panoramic images using little memory; the fast labeling process is easy to combine with intensive blending to produce high-quality panoramic images. The method is implemented in our mobile panorama system and runs with good performance on mobile devices.

Recent convergence of imaging sensors and general purpose processors on mobile phones creates an opportunity for a new class of augmented reality applications. Robust image feature extraction is a crucial enabler of this type of systems. In this article, we discuss an efficient mobile phone implementation of a state-of-the-art algorithm for computing robust image features called SURF. We implement several improvements to the basic algorithm that significantly improve its performance and reduce its memory footprint making the use of this algorithm on the mobile phone practical. Our prototype implementation has been applied to several practical applications such as image search, object recognition and augmented reality applications.

In this paper, we study the calibration of head pose estimation in stereo camera setting for meeting room video event analysis. Head pose information infers the direction of attention of the subjects in video, therefore is valuable for video event analysis/indexing, especially in meeting room scenario. We are developing a multi-modal meeting room data analyzing system for studying meeting room interaction dynamics, in which head pose estimation is one of the key components. As each subject in the meeting room can be observed by a pair of stereo cameras, we do 2D head tracking for the subject in each camera, and the 3D coordinate of the head can be obtained by triangulation. The 3D head pose is estimated in one of the camera coordinate system, we develop a procedure to accurately convert the estimated 3D pose in the camera coordinate system to that in the world coordinate system. In the experiment, visualization of the estimated head pose and location in world coordinate system verifies the soundness of our design. The estimated head pose and 3D location of the subjects in the meeting room allows further analysis of meeting room interaction dynamics, such as F-formation, floor-control, etc.

This paper presents an approach for registering aerial images taken at different time, viewpoints, or heights. Different from conventional image registration algorithms, our approach does not need image matching or correspondence. In this approach, we extract a number of corner features as the basis for registration and create a number of image patches with the corner points as centers on both reference and observed images. In order to let the corresponding patches cover same scene, we use a circle which the radius can be changed as the shape of the image patches. In this way, the image patches can handle the case in which there are rotation and scaling at the same time between reference and observed images. With the orientation differences of patches between these two images, we create an angle histogram with a voting procedure. The rotation angle between the two images can be determined by seeking the orientation difference that corresponds to the maximum peak in the histogram. Once we get the rotation angle, we seek back for the two corresponding patches which the value of orientation difference is the same as the rotation angle. The ratio of radii of these two patches is the value of the scaling. The proposed approach can handle the situation of large rotation and scaling between reference and observed images. It is applied to real aerial images and the results are very satisfying.

We present an adaptive skin color model for hand gesture tracking which is applied to cross-modal analysis of planning meetings. We build a skin color model with the Gaussian distribution and a skin color filter for each participant in meetings. By combining with the vector coherence mapping (VCM) algorithm, we track hand motion and obtain 3D trajectories. The hand gesture stream is extracted from hand motion trajectories. Different skin color models are created for different people to handle the differences of skin color. We update each model dynamically to adapt changes of environments. A parallel system has been implemented to track and extract hand motion trajectories. Examples of hand motion gesture tracking in meeting environments are provided. The applications of the adaptive skin color model can increase the speed of hand tracking

Gesture and speech are part of a single human language system. They are coexpressive and complementary channels in the act of speaking. Whereas speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, our work on oscillatory gestures and multimodal discourse is discussed. Our new techniques of analyzing hand gestures in the frequency domain are described. By tracking an individual's hands during a speech, hand motion trajectory signals are extracted from real video datasets. Our wavelet-based approach in gestural oscillation extraction is presented as frequency ridges in a frequency-time space. Wavelet ridges are extracted from responses of wavelet analysis. These wavelet ridges are employed to characterize frequency properties of hand motion trajectory signals. Hand motion oscillatory gestures can be extracted from these frequency properties. The potential of such computational cross-modal language analysis is motivated by performing a microanalysis of 2 video datasets. In the first dataset, a participant describes her living space to an interlocutor. In the second, a participant describes her action plan to an interlocutor. The ability of our algorithm to extract gestural oscillations is demonstrated, and the way that oscillatory gestures reveal portions of the discourse structure is shown.

In video based cross-model analysis of planning meeting, the meeting events are recorded by multiple cameras distributed in the entire meeting room. Subject's hand gestures, hand motion, head orientations, gaze targets, body poses are very important for the meeting event analysis. In order to register everything to the same global coordinate system, build 3D model, get 3D data from the video, we need to create a proper meeting room configuration and calibrate all cameras to obtain their intrinsic and extrinsic parameters. However, the calibration of multiple cameras distributed in the entire meeting room area is a challenging task because it is impossible to let all cameras in the meeting room see a reference object at the same time and wide field-of-view cameras suffer under radial distortion. In this paper, we propose a simple approach to create a good meeting room configuration and calibrate multiple cameras in the meeting room. The proposed approach includes several steps. First, we create stereo camera pairs according to the room configuration and the requirements of the targets, the participants of the meeting. Second, we apply Tsai's algorithm to calibrate each stereo camera pair and obtain the parameters in its own local coordinate system. Third, we use Vicon motion capture data to transfer all local coordinate systems of stereo camera pairs into a global coordinate system in the meeting room. We can obtain the positions, orientations, and parameters for all cameras in the same global coordinate system, so that we can register everything into this global coordinate system. Next, we do calibration error analysis for the current camera and meeting room configuration. We can obtain error distribution in the entire meeting room area. Finally, we improve the current camera and meeting room configuration according to the error distribution. By repeating these steps, we can obtain a good meeting room configuration and parameters of all cameras for this room configuration.

Gesture and speech are part of a single human language system. They are co-expressive and complementary channels in the act of speaking. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established contemporality of gesture and speech, we discuss our work on oscillatory gestures and speech. We present our wavelet-based approach in gestural oscillation extraction as geodesic ridges in frequency-time space. We motivate the potential of such computational cross-modal language analysis by performing a micro analysis of a video dataset in which a subject describes her living space. We demonstrate the ability of our algorithm to extract gestural oscillations and show how oscillatory gestures reveal portions of the discourse structure.

Gesture and speech are part of a single human language system. They are co-expressive and complementary channels in the act of speaking. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, we discuss our work on oscillatory gestures and speech. We present our wavelet-based approach in gestural oscillation extraction as geodesic ridges in frequency-time space. We motivate the potential of such computational cross-modal language analysis by performing a micro analysis of a video dataset in which a subject describes her living space. We demonstrate the ability of our algorithm to extract gestural oscillations and show how oscillatory gestures reveal portions of the discourse structure.

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.

Our approach is motivated by the conviction that gesture and speech are coexpressive of the underlying dynamic ideation that drives human communication. As such, transitions and cohesions is gestural behavior would inform us as to the discourse conceptualization. In this paper, we examine the role of motion symmetries of twohanded gestures in the structuring of speech. We employ a set of hand motion traces extracted from video and compute the correlation of these traces. The signs and magnitudes of the correlation coefÞcients computed in the cardinal directions of the subject’s torso (lateral and vertical in this work) characterize the symmetries. We employ a windowed computation approach that permits a balance between temporal resolution and robustness to noise. The resulting correlation proÞles are merged according to a temporal proximity rule. We apply this analysis to two conversational video sequences. A detailed analysis of the Þrst sequence reveals the persistence of gestural imagery between semantically-similar discourse pieces. A symmetry transition analysis is applied to the second dataset and compared against a manually generated discourse segmentation to demonstrate the potential of cross-modal discourse segmentation. 1.