Authoring site-specific outdoor augmented reality (AR) experiences requires a nuanced understanding of real-world context to create immersive and relevant content. Existing ex-situ authoring tools typically rely on static 3D models to represent spatial information. However, in our formative study (n=25), we identified key limitations of this approach: models are often outdated, incomplete, or insufficient for capturing critical factors such as safety considerations, user flow, and dynamic environmental changes. These issues necessitate frequent on-site visits and additional iterations, making the authoring process more time-consuming and resource-intensive. To mitigate these challenges, we introduce CoCreatAR, an asymmetric collaborative mixed reality authoring system that integrates the flexibility of ex-situ workflows with the immediate contextual awareness of in-situ authoring. We conducted an exploratory study (n=32) comparing CoCreatAR to an asynchronous workflow baseline, finding that it enhances engagement, creativity, and confidence in the authored output while also providing preliminary insights into its impact on task load. We conclude by discussing the implications of our findings for integrating real-world context into site-specific AR authoring systems.

As drone use has become more widespread, there is a critical need to ensure safety and security. A key element of this is robust and accurate drone detection and localization. While cameras and other optical sensors like LiDAR are commonly used for object detection, their performance degrades under adverse lighting and environmental conditions. Therefore, this has generated interest in finding more reliable alternatives, such as millimeter-wave (mmWave) radar. Recent research on mmWave radar object detection has predominantly focused on 2D detection of road users. Although these systems demonstrate excellent performance for 2D problems, they lack the sensing capability to measure elevation, which is essential for 3D drone detection. To address this gap, we propose CubeDN, a single-stage end-to-end radar object detection network specifically designed for flying drones. CubeDN overcomes challenges such as poor elevation resolution by utilizing a dual radar configuration and a novel deep learning pipeline. It simultaneously detects, localizes, and classifies drones of two sizes, achieving decimeter-level tracking accuracy at closer ranges with overall 95% average precision (AP) and 85% average recall (AR). Furthermore, CubeDN completes data processing and inference at 10Hz, making it highly suitable for practical applications.

Augmented reality displays are becoming more powerful and simultaneously more mobile. Although mobile AR is gaining popularity, it remains difficult to get an insight into users' cognitive load, despite its relevance for many mobile-based tasks. Usually, cognitive load is measured via subjective, task-disruptive self-reports such as NASA TLX. While biosensors such as galvanic skin response, heart rate variability, or pulse have been used to obtain more objective measures, these are highly susceptible to motion-induced noise. More robust techniques like EEG offer higher reliability but are impractical for mobile, real-world use. In this paper, we report on a non-contact multi-sensor approach to assess cognitive load in mobile AR. Our approach combines pupillometry, facial expression tracking, and thermal imaging for respiratory rate analysis. Within the frame of our study, we analysed the aptness of the methods, comparing load assessment for low and high cognitive load tasks under both stationary and mobile conditions. Using an XGBoost classifier, our model achieved 86.11% accuracy for binary cognitive load assessment (low vs. high cognitive load) and 84.24% accuracy for four-way classification (cognitive load$\times$mobility). Feature importance analysis revealed that robust predictors included gaze dynamics (e.g., fixation, pursuit, and saccade durations), pupil diameter metrics (such as FFT band power and variability measures), and facial and respiratory features (including brow lowering and nostril temperature quantiles) for assessing cognitive load in mobile AR.

Autonomous and semi-autonomous systems are using deep learning models to improve decision-making. However, deep classifiers can be overly confident in their incorrect predictions, a major issue especially in safety-critical domains. The present study introduces three foundational desiderata for developing real-world risk-aware classification systems. Expanding upon the previously proposed Evidential Deep Learning ( EDL ), we demonstrate the unity between these principles and EDL ’s operational attributes. We then augment EDL empowering autonomous agents to exercise discretion during structured decision-making when uncertainty and risks are inherent. We rigorously examine empirical scenarios to substantiate these theoretical innovations. In contrast to existing risk-aware classifiers, our proposed methodologies consistently exhibit superior performance, underscoring their transformative potential in risk-conscious classification strategies. • Evidential deep learning uses Dirichlet distributions to represent the predictive uncertainty of neural classifiers. • Pignistic probabilities can be used to model rational decision-making under uncertainty. • Risk awareness can be integrated into evidential classifiers using pignistic Dirichlet priors.

Change detection (CD) is critical in everyday tasks. While current algorithmic approaches for CD are improving, they remain imprecise, often requiring human intervention. Cognitive science research focuses on understanding CD mechanisms, especially through change blindness studies. However, these do not address the primary requirement in real-life CD - detecting changes as effectively as possible. Such a requirement is directly relevant to the visual comparison field - studying visualisation techniques to compare data and identify differences or changes effectively. Recent studies have used Virtual Reality (VR) to improve visual comparison by providing an immersive platform where users can interact with 3D data at a real-life scale, enhancing spatial reasoning. We believe VR could also improve CD performance accordingly. Particularly, VR offers stereoscopic depth perception over traditional displays, potentially enhancing the detection of spatial change. In this paper, we develop and analyse three 3D visual comparison techniques for CD in VR: Sliding Window, 3D Slider, and Switch Back. These techniques are evaluated under synthetic but realistic environments and frequently occurring Perceptual Challenges, including different Changed Object Size, Lighting Variation, and Scene Drift conditions. Experimental results reveal significant differences between the techniques in detection time measures and subjective user experience.

Mobile 3D Printing (M3DP), using printing-in-motion, is a powerful paradigm for automated construction. A mobile robot, equipped with its own power, materials and an arm-mounted extruder, simultaneously navigates and creates its environment. Such systems can be highly scalable, parallelizable and flexible. However, planning and controlling the motion of the arm and base at the same time is challenging and most deployments either avoid robot-base motion entirely or use human prescribed robot-base paths. In a previous paper, we developed a high-level planning algorithm to automate M3DP given a print task. The generated robot-base paths avoid collisions and maintain task reachability. In this paper, we extend this work to robot control. We develop and compare three different ways to integrate the long-duration planned path with a short horizon Model Predictive Controller. Experiments are carried out via a new M3DP system - Armstone. We evaluate and demonstrate our algorithm in a 250 m long multi-layer print which is about 5 times longer than any previous physical printing-in-motion system.

Micro-Doppler signatures contain considerable information about target dynamics. However, the radar sensing systems are easily affected by noisy surroundings, resulting in uninterpretable motion patterns on the micro-Doppler spectrogram. Meanwhile, radar returns often suffer from multipath, clutter and interference. These issues lead to difficulty in, for example motion feature extraction, activity classification using micro Doppler signatures ($\mu$-DS), etc. In this paper, we propose a latent feature-wise mapping strategy, called Feature Mapping Network (FMNet), to transform measured spectrograms so that they more closely resemble the output from a simulation under the same conditions. Based on measured spectrogram and the matched simulated data, our framework contains three parts: an Encoder which is used to extract latent representations/features, a Decoder outputs reconstructed spectrogram according to the latent features, and a Discriminator minimizes the distance of latent features of measured and simulated data. We demonstrate the FMNet with six activities data and two experimental scenarios, and final results show strong enhanced patterns and can keep actual motion information to the greatest extent. On the other hand, we also propose a novel idea which trains a classifier with only simulated data and predicts new measured samples after cleaning them up with the FMNet. From final classification results, we can see significant improvements.

Distributed virtual environments (DVEs) are challenging to create as the goals of consistency and responsiveness become contradictory under increasing latency. DVEs have been considered as both distributed transactional databases and force-reflection systems. Both are good approaches, but they do have drawbacks. Transactional systems do not support Level 3 (L3) collaboration: manipulating the same degree-of-freedom at the same time. Force-reflection requires a client-server architecture and stabilisation techniques. With Consensus Based Networking (CBN), we suggest DVEs be considered as a distributed data-fusion problem. Many simulations run in parallel and exchange their states, with remote states integrated with continous authority. Over time the exchanges average out local differences, performing a distribued-average of a consistent, shared state. CBN aims to build simulations that are highly responsive, but consistent enough for use cases such as the piano-movers problem. CBN's support for heterogeneous nodes can transparently couple different input methods, avoid the requirement of determinism, and provide more options for personal control over the shared experience. Our work is early, however we demonstrate many successes, including L3 collaboration in room-scale VR, 1000's of interacting objects, complex configurations such as stacking, and transparent coupling of haptic devices. These have been shown before, but each with a different technique; CBN supports them all within a single, unified system.

To capture the geometry of an object by an autonomous system, next best view (NBV) planning can be used to determine the path a robot will take. However, current NBV planning algorithms do not distinguish between objects that need to be mapped and everything else in the environment; leading to inefficient search strategies. In this paper we present a novel approach for NBV planning that accounts for the importance of objects in the environment to inform navigation. Using weighted entropy to encode object utilities computed via semantic segmentation, we evaluate our approach over a set of virtual Gazebo environments comparable to construction scales. Our results show that using semantic information reduces the time required to capture a target object by at least 40 percent.

In this paper, we explore the problem of task-consistent path planning for printing-in-motion via Mobile Manipulators (MM). MM offer a potentially unlimited planar workspace and flexibility for print operations. However, most existing methods have only mobility to relocate an arm which then prints while stationary. In this paper we present a new fully autonomous path planning approach for mobile material deposition. We use a modified version of Rapidly-exploring Random Tree Star (RRT*) algorithm, which is informed by a constrained Inverse Reachability Map (IRM) to ensure task consistency. Collision avoidance and end-effector reachability are respected in our approach. Our method also detects when a print path cannot be completed in a single execution. In this case it will decompose the path into several segments and reposition the base accordingly.

In this paper, we propose risk-calibrated evidential deep classifiers to reduce the costs associated with classification errors. We use two main approaches. The first is to develop methods to quantify the uncertainty of a classifier's predictions and reduce the likelihood of acting on erroneous predictions. The second is a novel way to train the classifier such that erroneous classifications are biased towards less risky categories. We combine these two approaches in a principled way. While doing this, we extend evidential deep learning with pignistic probabilities, which are used to quantify uncertainty of classification predictions and model rational decision making under uncertainty.We evaluate the performance of our approach on several image classification tasks. We demonstrate that our approach allows to (i) incorporate misclassification cost while training deep classifiers, (ii) accurately quantify the uncertainty of classification predictions, and (iii) simultaneously learn how to make classification decisions to minimize expected cost of classification errors.

In this paper, we propose a method to use semantic information to improve the use of map priors in a sparse, feature-based MonoSLAM system. To incorporate the priors, the features in the prior and SLAM maps must be associated with one another. Most existing systems build a map using SLAM and then align it with the prior map. However, this approach assumes that the local map is accurate, and the majority of the features within it can be constrained by the prior. We use the intuition that many prior maps are created to provide semantic information. Therefore, valid associations only exist if the features in the SLAM map arise from the same kind of semantic object as the prior map. Using this intuition, we extend ORB-SLAM2 using an open source pre-trained semantic segmentation network (DeepLabV3+) to incorporate prior information from Open Street Map building footprint data. We show that the amount of drift, before loop closing, is significantly smaller than that for original ORB-SLAM2. Furthermore, we show that when ORB-SLAM2 is used as a prior-aided visual odometry system, the tracking accuracy is equal to or better than the full ORB-SLAM2 system without the need for global mapping or loop closure.

Many human activities, such as manufacturing and assembly, are sequence-constrained procedural tasks (SPTs): they consist of a series of steps that must be executed in a specific spatial/temporal order. However, these tasks can be error prone - steps can be missed out, executed out-of-order, and repeated. The ability to automatically predict if a person is about to commit an error could greatly help in these cases. The prediction could be used, for example, to provide feedback to prevent mistakes or mitigate their effects. In this paper, we present a novel approach for real-time error prediction for multi-step sequence tasks which uses a minimum viable set of behavioural signals. We have three main contributions. The first we present an architecture for real-time error prediction based on task tracking and intent prediction. The second is to explore the effectiveness of using hand position and eye-gaze tracking for task tracking. We confirm that eye-gaze is more effective for intent prediction, hand tracking is more accurate for task tracking and that combining the two provides the best overall response. We show that using Hands and Gaze tracking data we can predict selection/placement errors with an F1 score of 97%, approximately 300ms before the error would occur. Finally, we discuss the application of this hand-gaze error detection architecture used in conjunction with head-mounted AR displays, to support industrial manual assembly.

Filmmakers of panoramic videos frequently struggle to guide attention to Regions of Interest (ROIs) due to consumers’ freedom to explore. Some researchers hypothesize that peripheral cues attract reflexive/involuntary attention whereas cues within central vision engage and direct voluntary attention. This mixed-methods study evaluated the effectiveness of using central arrows and peripheral flickers to guide and focus attention in panoramic videos. Twenty-five adults wore a head-mounted display with an eye tracker and were guided to 14 ROIs in two panoramic videos. No significant differences emerged in regard to the number of followed cues, the time taken to reach and observe ROIs, ROI-related memory and user engagement. However, participants’ gaze travelled a significantly greater distance toward ROIs within the first 500 ms after flicker-onsets compared to arrow-onsets. Nevertheless, most users preferred the arrow and perceived it as significantly more rewarding than the flicker. The findings imply that traditional attention paradigms are not entirely applicable to panoramic videos, as peripheral cues appear to engage both involuntary and voluntary attention. Theoretical and practical implications as well as limitations are discussed.

Automatically monitoring and quantifying stress-induced thermal dynamic information in real-world settings is an extremely important but challenging problem. In this paper, we explore whether we can use mobile thermal imaging to measure the rich physiological cues of mental stress that can be deduced from a person's nose temperature. To answer this question we build i) a framework for monitoring nasal thermal variable patterns continuously and ii) a novel set of thermal variability metrics to capture a richness of the dynamic information. We evaluated our approach in a series of studies including laboratory-based psychosocial stress-induction tasks and real-world factory settings. We demonstrate our approach has the potential for assessing stress responses beyond controlled laboratory settings.

Deep Neural Networks (DNNs) deliver state-of-the-art performance in many image recognition and understanding applications. However, despite their outstanding performance, these models are black-boxes and it is hard to understand how they make their decisions. Over the past few years, researchers have studied the problem of providing explanations of why DNNs predicted their results. However, existing techniques are either obtrusive, requiring changes in model training, or suffer from low output quality. In this paper, we present a novel method, NeuroMask, for generating an interpretable explanation of classification model results. When applied to image classification models, NeuroMask identifies the image parts that are most important to classifier results by applying a mask that hides/reveals different parts of the image, before feeding it back into the model. The mask values are tuned by minimizing a properly designed cost function that preserves the classification result and encourages producing an interpretable mask. Experiments using state-of-art Convolutional Neural Networks for image recognition on different datasets (CIFAR-10 and ImageNet) show that NeuroMask successfully localizes the parts of the input image which are most relevant to the DNN decision. By showing a visual quality comparison between NeuroMask explanations and those of other methods, we find NeuroMask to be both accurate and interpretable.

Overlaying a building with images from the past can be an engaging way to explore a historic site. However, little is known about what type of content functions well when using augmented reality (AR) in outdoor contexts. This research investigates how different types of AR content – such as text or image – can affect the flow experience as well as other user affective and behavioural responses. We ran an experimental study with 85 participants at a university campus, where three groups used different apps – AR app with overlaid textual information, AR app with overlaid both textual information and images and an app with audio guide – to explore the area's historic and cultural background. The results show that overlaying images in addition to the text was clearly the most successful way of attracting attention and providing stronger flow in comparison to the apps that overlaid only text or delivered audio. However, we also discovered that users occasionally interpreted the overlaid imagery to be cues for something else other than what they were designed to represent. Finally, we discuss how AR content can enlarge otherwise invisible details, depict historic elements and unveil interiors of buildings.

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognize 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.

Kalman filters are routinely used for many data fusion applications including navigation, tracking, and simultaneous localization and mapping problems. However, significant time and effort is frequently required to tune various Kalman filter model parameters, e.g. process noise covariance, pre-whitening filter models for non-white noise, etc. Conventional optimization techniques for tuning can get stuck in poor local minima and can be expensive to implement with real sensor data. To address these issues, a new “black box” Bayesian optimization strategy is developed for automatically tuning Kalman filters. In this approach, performance is characterized by one of two stochastic objective functions: normalized estimation error squared (NEES) when ground truth state models are available, or the normalized innovation error squared (NIS) when only sensor data is available. By intelligently sampling the parameter space to both learn and exploit a nonparametric Gaussian process surrogate function for the NEESINIS costs, Bayesian optimization can efficiently identify multiple local minima and provide uncertainty quantification on its results.

Recent advances in Machine Learning (ML) have profoundly changed many detection, classification, recognition and inference tasks. Given the complexity of the battlespace, ML has the potential to revolutionise how Coalition Situation Understanding is synthesised and revised. However, many issues must be overcome before its widespread adoption. In this paper we consider two - interpretability and adversarial attacks. Interpretability is needed because military decision-makers must be able to justify their decisions. Adversarial attacks arise because many ML algorithms are very sensitive to certain kinds of input perturbations. In this paper, we argue that these two issues are conceptually linked, and insights in one can provide insights in the other. We illustrate these ideas with relevant examples from the literature and our own experiments.

We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating handcrafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.

Building a complete 3D model of a scene, given only a single depth image, is underconstrained. To gain a full volumetric model, one needs either multiple views, or a single view together with a library of unambiguous 3D models that will fit the shape of each individual object in the scene. We hypothesize that objects of dissimilar semantic classes often share similar 3D shape components, enabling a limited dataset to model the shape of a wide range of objects, and hence estimate their hidden geometry. Exploring this hypothesis, we propose an algorithm that can complete the unobserved geometry of tabletop-sized objects, based on a supervised model trained on already available volumetric elements. Our model maps from a local observation in a single depth image to an estimate of the surface shape in the surrounding neighborhood. We validate our approach both qualitatively and quantitatively on a range of indoor object collections and challenging real scenes.

The articles in this special section were presented at the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

The IEEE International Symposium on Mixed and Augmented Reality (ISMAR) is the leading venue for publishing the latest Mixed and Augmented Reality research, applications, and technologies. This special section presents significantly extended versions of the five best papers from the IEEE ISMAR 2014 proceedings. Within the past few years, Augmented Reality (AR) has reached a critical mass in both research and commercial applications. It is now becoming truly feasible to use augmented reality to place graphics anywhere at any time. However, although the basic capabilities exist, many open research problems continue. This collection of papers considers underlying issues and technologies. IEEE ISMAR 2014 had 89 paper submissions; each paper was reviewed by at least four experts in the field. An international programcommittee of 15 ARexperts invited reviewers, led discussions, invited a rebuttal by the paper authors and prepared a consensus review. To select the final papers for publication, an online two-day PC meeting was held connecting three continents, where each paper was discussed. Thirty-five papers were accepted either as long or short publications, giving an overall acceptance rate of 40%. An independent Award Committee reviewed the highest- ranked submissions again to determine the awards for Best Paper and Honorable Mention. For this special section, the authors of the award papers were invited to submit an extended version of their conference papers, with a clear focus on additional content that expands the scientific contribution of the original conference paper. A standard TVCG reviewing cycle was initiated in which all papers were reviewed, feedback was provided, and papers were revised to suit. Out of all submitted papers, less than 6% appear in this TVCG Special Section.

Motivated by the risks and costs associated with outdoor experiments, this paper presents a new multi-platform quadrotor simulator. The simulator implements a novel second-order dynamic model for a quadrotor, produced through evolutionary programming, and explained by domain knowledge. The model captures the effects of mechanics, aerodynamics, wind and rotational stabilization control on the flight platform. In addition, the simulator implements military-grade models for wind and turbulence, as well as noise models for satellite navigation, barometric altitude and orientation. The usefulness of the simulator is shown qualitatively by a comparing how coloured and white position noise affect the performance of offline, range-only SLAM. The simulator is intended to be used for planning experiments, or for stress-testing application performance over a wide range of operating conditions.

Non-photorealistic rendering (NPR) has been shown as a powerful way to enhance both visual coherence and immersion in augmented reality (AR). However, it has only been evaluated in idealized pre-rendered scenarios with handheld AR devices. In this paper we investigate the use of NPR in an immersive, stereoscopic, wide field-of-view head-mounted video see-through AR display. This is a demanding scenario, which introduces many real-world effects including latency, tracking failures, optical artifacts and mismatches in lighting. We present the AR-Rift, a low-cost video see-through AR system using an Oculus Rift and consumer webcams. We investigate the themes of consistency and immersion as measures of psychophysical non-mediation. An experiment measures discernability and presence in three visual modes: conventional (unprocessed video and graphics), stylized (edge-enhancement) and virtualized (edge-enhancement and color extraction). The stylized mode results in chance-level discernability judgments, indicating successful integration of virtual content to form a visually coherent scene. Conventional and virutalized rendering bias judgments towards correct or incorrect respectively. Presence as it may apply to immersive AR, and which, measured both behaviorally and subjectively, is seen to be similarly high over all three conditions.

We present PanoInserts: a novel teleconferencing system that uses smartphone cameras to create a surround representation of meeting places. We take a static panoramic image of a location into which we insert live videos from smartphones. We use a combination of marker- and image-based tracking to position the video inserts within the panorama, and transmit this representation to a remote viewer. We conduct a user study comparing our system with fully-panoramic video and conventional webcam video conferencing for two spatial reasoning tasks. Results indicate that our system performs comparably with fully-panoramic video, and better than webcam video conferencing in tasks that require an accurate surrounding representation of the remote space. We discuss the representational properties and usability of varying video presentations, exploring how they are perceived and how they influence users when performing spatial reasoning tasks.

We introduce a method to discover objects from RGB-D image collections which does not require a user to specify the number of objects expected to be found. We propose a probabilistic formulation to find pairwise similarity between image segments, using a classifier trained on labelled pairs from the recently released RGB-D Object Dataset. We then use a correlation clustering solver to both find the optimal clustering of all the segments in the collection and to recover the number of clusters. Unlike traditional supervised learning methods, our training data need not be of the same class or category as the objects we expect to discover. We show that this parameter-free supervised clustering method has superior performance to traditional clustering methods.

We are delighted to welcome you to ISMAR 2013, the 12th symposium on Mixed and Augmented Reality! This year's symposium continues a long tradition of ISMAR meetings, a series that itself followed a related series of IWAR, ISMR, and ISAR meetings.

Within mobile mixed reality experiences, we would like to engage the user's head and hands for interaction. However, this requires the use of multiple tracking systems. These must be aligned, both as part of initial system setup and to counteract inter-tracking system drift that can accumulate over time. Traditional approaches to alignment use obtrusive procedures that introduce explicit constraints between the different tracking systems. These can be highly disruptive for the user's experience. In this paper, we propose another type of information which can be exploited to effect alignment: the behaviour of the user. The crucial insight is that user behaviours - such as selection through pointing - introduce implicit constraints between tracking systems. These constraints can be used as the user continually interacts with the system to infer alignment without the need for disruptive procedures. We call this concept behaviour-aware sensor fusion. We introduce two different interaction techniques-the redirected pointing technique and the yaw fix technique - to illustrate this concept. Pilot experiments show that behaviour-aware sensor fusion can increase ease of use and speed of interaction in exemplar mixed-reality interaction tasks.

In this paper, we consider the problem of developing a model of the behaviour of a Missing Person (MP) who is lost in the wilderness. Traditional models have treated the movement of an MP as a type of diffusion process, without regard for the MP's internal state or goals. However, this fails to include many important factors, including the effects of fatigue or the use of reorienting strategies, which are known to be important from empirical studies of actual lost person behaviour. To overcome these limitations, we develop a novel agent-based model of MP behaviour. This model incorporates the effects of the environment, perception and goals on the MP's movement and uses them to model the continuous switching of short and medium term goals by missing people. To validate the model, we compared the trajectories simulated by it with actual recordings of movements of participants in an unfamiliar wilderness environment. By comparing the predicted and actual trajectories, we show that the generated trajectories much more faithfully represent the actual movements of the participants than state-of-the-art diffusion models.

To travel effectively in an uncertain environment, a robot should use a path-planning algorithm which takes the impact of this uncertainty into account. In previous work, we developed the Path Distribution (PD) Planner which uses a Monte Carlo approach to sample the environment, generate the distribution of optimal paths, and plan a path using this distribution. We have shown that this approach outperforms other approaches to planning with uncertainty. However, Monte Carlo sampling can become extremely computationally expensive. In this paper, we develop two strategies to reduce the computational cost of the algorithm. The first, called Sampling in Planning Process (SiPP), performs lazy sampling within the planning algorithm itself. The second, which we call the Hierarchal PD Planner, performs dimensionality reduction by decomposing the environment into homogeneous regions. We show that these approaches can reduce computational costs by more than a factor of two with minimal loss of performance.

In this paper we present a general method for exploiting prior information to constrain the location of land marks in GraphSLAM. Prior information can be obtained for many environments in many different ways. However, this information can be incomplete, out-of-date, or presented in a different form than that used by the robot. Therefore, we argue that prior information is most naturally modelled as sets of potential constraints that act between landmarks. We present an extension of GraphSLAM that incorporates these constraints. We illustrate the results in an experiment with a 3D laser scanner and demonstrate a significant improvement in performance.

Recent work in the domain of classification of point clouds has shown that topic models can be suitable tools for inferring class groupings in an unsupervised manner. However, point clouds are frequently subject to non-negligible amounts of sensor noise. In this paper, we analyze the effect on classification accuracy of noise added to both an artificial data set and data collected from a Light Detection and Ranging (LiDAR) scanner, and show that topic models are less robust to ‘misspelled’ words than the more näive k-means classifier. Furthermore, standard spin images prove to be a more robust feature under noise than their derivative, ‘angular’ spin images. We additionally show that only a small subset of local features are required in order to give comparable classification accuracy to a full feature set.

Conventional complex valued signal processing algorithms assume rotation invariant (circular) signal distributions, and are thus suboptimal for real world processes which exhibit rotation dependent distributions (noncircular). In nonlinear sequential state space estimation, noncircularity can arise from the data, state transition model, and state and observation noises. We provide further insight by revisiting the augmented complex unscented Kalman filter (ACUKF) and illuminating its operation in such scenarios. The analysis establishes a relationship between the estimation error and the degree of second order noncircularity (improperness) in the system for the conventional complex unscented Kalman filter (CUKF), and is supported by simulations on both synthetic and real world proper and improper signals.

Multiple autonomous Unmanned Aerial Vehicles (UAVs) can be used to complement human teams. This paper presents the results of an exploratory study to investigate gesture/speech interfaces for interaction with robots in a situated manner and the development of three iterations of a prototype command set. A command set was compiled from observing users interacting with a simulated interface in a virtual reality environment. We discovered that users find this type of interface intuitive and their commands tend to naturally group into both `High-Level' and `Low-Level' instructions. However, as the robots moved further away, the loss of depth perception and direct feedback was inimical to the interaction. In a second experiment we found that using simple heads up display elements could mitigate these issues.

This paper is motivated by the real world problem of search and rescue by unmanned aerial vehicles (UAVs). We consider the problem of tracking a static target from a bird's-eye view camera mounted to the underside of a quadrotor UAV. We begin by proposing a target detection algorithm, which we then execute on a collection of video frames acquired from four different experiments. We show how the efficacy of the target detection algorithm changes as a function of altitude. We summarise this efficacy into a table which we denote the observation model. We then run the target detection algorithm on a sequence of video frames and use parameters from the observation model to update a recursive Bayesian estimator. The estimator keeps track of the probability that a target is currently in view of the camera, which we refer to more simply as target presence. Between each target detection event the UAV changes position and so the sensing region changes. Under certain assumptions regarding the movement of the UAV, the proportion of new information may be approximated to a value, which we then use to weight the prior in each iteration of the estimator. Through a series of experiments we show how the value of the prior for unseen regions, the altitude of the UAV and the camera sampling rate affect the accuracy of the estimator. Our results indicate that there is no single optimal sampling rate for all tested scenarios. We also show how the prior may be used as a mechanism for tuning the estimator according to whether a high false positive or high false negative probability is preferable.

We consider how prior information can be exploited to improve the quality of SLAM. Prior information, such as aerial imagery, can be readily obtained for many environments. However, this information is often collected at a different time, using different sensors, different representations and from different vantage points than those used by the robot undertaking SLAM. In this paper, we describe a general probabilistic framework to overcome these difficulties. Our framework models the environment as a random set of latent structures which are observed by a set of sensing systems. Each sensing system gives rise to a different kind of map and, by associating features from the same structure across the different maps, parameterised constraints between the sets of features can be constructed. These parameterised constraints make it possible to transfer information between map representations. We demonstrate the use of the framework in a simulated environment to illustrate how geometric features of different dimensions can be fused together.

This paper presents the common state filter (CSF), a novel and efficient suboptimal multiple hypothesis slam (MHSLAM) method for Kalman Filter-based SLAM algorithms. Conventional MHSLAM algorithms require the entire vehicle and map state to be copied for each hypothesis. The CSF, by contrast, maintains a single, common instance of the vast majority of the map and only copies the map portion that varies substantially across different hypotheses. We demonstrate the performance of the algorithm on the Victoria Park data set.

In this paper we consider ways to alleviate negative estimated depth for the inverse depth parameterisation of bearing-only SLAM. This problem, which can arise even if the beacons are far from the platform, can cause catastrophic failure of the filter.We consider three strategies to overcome this difficulty: applying inequality constraints, the use of truncated second order filters, and a reparameterisation using the negative logarithm of depth. We show that both a simple inequality method and the use of truncated second order filters are successful. However, the most robust performance is achieved using the negative log parameterisation.

Many augmented reality (AR) applications use marker-based vision tracking systems to recover camera pose by detecting one or more planar landmarks. However, most of these systems do not interactively quantify the accuracy of the pose they calculate. Instead, the accuracy of these systems is either ignored, assumed to be a fixed value, or determined using error tables (constructed in an off-line ground-truthed process) along with a run-time interpolation scheme. The validity of these approaches are questionable as errors are strongly dependent on the intrinsic and extrinsic camera parameters and scene geometry. In this paper we present an algorithm for predicting the statistics of marker tracker error in real-time. Based on the scaled spherical simplex unscented transform (SSSUT), the algorithm is applied to the augmented reality toolkit plus (ARToolKitPlus). The results are validated using precision off-line photogrammetric techniques.

In this paper we consider the use of Covariance Union (CU) with multi-hypothesis techniques (MHT) and Gaussian mixture models (GMMs) to generalize the conventional mean and covariance representation of information. More specifically, we address the representation of multi-modal information using multiple mean and covariance estimates. A significant challenge is to define a rigorous fusion algorithm that can bind the complexity of the filtering process. This requires a mechanism for subsuming subsets of modes into single modes so that the complexity of the representation satisfies a specified upper bound. We discuss how this can be accomplished using CU. The practical challenge is to develop efficient implementations of the CU algorithm. Because of the novelty of the CU algorithm, there are no existing real-time codes for use in real applications. In this paper we address this deficiency by considering a general-purpose implementation of the CU algorithm based on general nonlinear optimization techniques. Computational results are reported

This paper considers the problem of developing algorithms for the distributed fusion of Gaussian mixture models through the use of Chernoff information. We derive a first order approximation and show that, in a distributed tracking problem in which sensor nodes are equipped with only range-only or bearing-only sensors, it yields consistent estimates

In this paper we describe a software framework to enable heterogeneous, distributed data fusion of disparate information sources. The framework is agent-based and consists of three main elements. The first is a generalization of the target state to a container of arbitrary, uncertain attributes. The structure of this estimate can vary both across time and across different nodes in the same network. The second is the development of composable process and observation models. These make it possible to dynamically change the models at runtime to fit the current target state estimate

A significant problem encountered when building Augmented Reality (AR) systems is that all spatial knowledge about the world has uncertainty associated with it. This uncertainty manifests itself as registration errors between the graphics and the physical world, and ambiguity in user interaction. In this paper, we show how estimates of the registration error can be leveraged to support predictable selection in the presence of uncertain 3D knowledge. These ideas are demonstrated in osgAR, an extension to OpenSceneGraph with explicit support for uncertainty in the 3D transformations. The osgAR runtime propagates this uncertainty throughout the scene graph to compute robust estimates of the probable location of all entities in the system from the user's viewpoint, in real-time. We discuss the implementation of selection in osgAR, and the issues that must be addressed when creating interaction techniques in such a system.

An important problem for augmented reality is registration error. No system can be perfectly tracked, calibrated or modeled. As a result, the overlaid graphics are not aligned perfectly with objects in the physical world. This can be distracting, annoying or confusing. In this paper, we propose a method for mitigating the effects of registration errors that enables application developers to build dynamically adaptive AR displays. Our solution is implemented in a programming toolkit called OSGAR. Built upon OpenSceneGraph (OSG), OSGAR statistically characterizes registration errors, monitors those errors and, when a set of criteria are met, dynamically adapts the display to mitigate the effects of the errors. Because the architecture is based on a scene graph, it provides a simple, familiar and intuitive environment for application developers. We describe the components of OSGAR, discuss how several proposed methods for error registration can be implemented, and illustrate its use through a set of examples.

The extended Kalman filter (EKF) is probably the most widely used estimation algorithm for nonlinear systems. However, more than 35 years of experience in the estimation community has shown that is difficult to implement, difficult to tune, and only reliable for systems that are almost linear on the time scale of the updates. Many of these difficulties arise from its use of linearization. To overcome this limitation, the unscented transformation (UT) was developed as a method to propagate mean and covariance information through nonlinear transformations. It is more accurate, easier to implement, and uses the same order of calculations as linearization. This paper reviews the motivation, development, use, and implications of the UT.

The state of the art in unscented techniques for nonlinear estimation is surveyed. The process noise covariance matrix used on each filter is not the same as the process noise used to drive the motion of the true projectile in the simulation. All the Jacobian matrices for the extended Kalman filter (EKF) are calculated numerically using a central difference scheme and a step size of 10-3. It is observed that EKF yields an inconsistent estimate, while the unscented Kalman filter yields a consistent estimate.

This paper analyses the consequences of using Covariance Inflation Methods for Simultaneous Localisation and Map Building (SLAM). Covariance Inflation refers to the process of adding a positive semidefinite matrix to the system covariance matrix to improve the properties of a SLAM algorithm. Because this approach can be used to decorrelate the state estimates in the covariance matrix, it has the potential to greatly reduce both computational and storage costs. However, it also raises the risk that the covariance can increase without bound. This paper analyses the properties of covariance inflation algorithms to assess their impact on performance. We prove that, to prevent the steady-state covariance from being increased, the computational and storage costs must be linear in the number of beacons. Furthermore, if the steady-state covariance is to remain finite, the inflation method cannot impose structures on the filter which are continually broken down. These results are illustrated in a simple linear example.

Simultaneous localisation and map building (SLAM) is one of the most important and challenging areas of mobile robotics. Unfortunately, the optimal Kalman filter solution incurs computational costs that scale quadratically with the number of beacons, which is prohibitive for many real time and large scale applications. Consequently, there is a significant practical need for more efficient approaches. The challenge is to develop methods that are both efficient and mathematically rigorous. In this paper we show that the full SLAM problem can be decomposed into two distinct mathematical operations. One is the maintenance of global state information for both the vehicle and the beacons, and the other is the maintenance of relative state information. These operations are distinct because the former is an unobserservable estimation problem while the latter is not. We argue that solutions to these two problems can be applied as scaffolding for the development of a wide variety of specialized SLAM algorithms. As a practical demonstration of the power of the two operations when applied as a generic solution to the SLAM problem, we provide empirical results for a scenario requiring the real-time construction and maintenance of a map containing 1.1 million beacons.

To achieve accurate registration, the transformations which locate the tracking system components with respect to the environment must be known. These transformations relate the base of the tracking system to the virtual world and the tracking system's sensor to the graphics display. In this paper we present a unified, general calibration method for calculating these transformations. A user is asked to align the display with objects in the real world. Using this method, the sensor to display and tracker base to world transformations can be determined with as few as three measurements.

A useful function of augmented reality (AR) systems is their ability to visualize occluded infrastructure directly in a user's view of the environment. This is especially important for our application context, which utilizes mobile AR for navigation and other operations in an urban environment. A key problem in the AR field is how to best depict occluded objects in such a way that the viewer can correctly infer the depth relationships between different physical and virtual objects. Showing a single occluded object with no depth context presents an ambiguous picture to the user. But showing all occluded objects in the environments leads to the "Superman's X-ray vision" problem, in which the user sees too much information to make sense of the depth relationships of objects. Our efforts differ qualitatively from previous work in AR occlusion, because our application domain involves far-field occluded objects, which are tens of meters distant from the user. Previous work has focused on near-field occluded objects, which are within or just beyond arm's reach, and which use different perceptual cues. We designed and evaluated a number of sets of display attributes. We then conducted a user study to determine which representations best express occlusion relationships among far-field objects. We identify a drawing style and opacity settings that enable the user to accurately interpret three layers of occluded objects, even in the absence of perspective constraints.

We describe two engineering experiments designed to evaluate the effectiveness of Measurand's ShapeTape for wearable, mobile interaction. Our initial results suggest that the ShapeTape is not appropriate for interactions which require a high degree of accuracy. However, ShapeTape is capable of reproducing the qualitative motion the user is performing and thus could be used to support 3D gesture-based interaction.

The full power of mobile augmented and virtual reality systems is realized when these systems are connected to one another to immersive virtual environments, and to remote information servers. Connections are usually made through wireless networks. However, wireless networks cannot guarantee connectivity and their bandwidth can be highly constrained. The authors present a robust event-based data distribution mechanism for mobile augmented reality and virtual environments. It is based on replicated databases, pluggable networking protocols, and communication channels. We demonstrate the mechanism in the Battlefield Augmented Reality System (BARS) situation awareness system, which is composed of several mobile augmented reality systems, immersive and desktop-based virtual reality systems, a 2D map-based multi-modal system, handheld PCs, and other sources of information.

This paper examines the role played by vehicle models and their impact on the performance of sensor-based navigation systems for autonomous land vehicles. In a navigation system, information from internal and external vehicle sensors is combined to estimate the motion of the vehicle. However, while the issue of sensing and effects of sensor accuracy have been widely studied, there are few results or insights into the complementary role played by the vehicle model. This paper has two main contributions: a theoretical analysis of the role of the vehicle model in navigation system performance, and an empirical study of three models of increasing complexity, used in a navigation system for a conventional road vehicle. The theoretical analysis focuses on understanding the effect of estimation errors caused by approximations to the "true" vehicle model. It shows that while substantial performance improvements can be obtained from better vehicle modeling, there is, in general, no definitive "best" model for such complex nonlinear estimation problems. The empirical study shows that an appropriate choice of a higher order model can lead to significant improvements in the performance of the navigation system. However, the highest order model suffers from problems related to the observability of some of its parameters. We show how this problem can be overcome through the imposition of weak constraints.

All augmented reality (AR) systems must deal with registration errors. While most AR systems attempt to minimize registration errors through careful calibration, registration errors can never be completely eliminated in any realistic system. In this paper, we describe a robust and efficient statistical method for estimating registration errors. Our method generates probabilistic error estimates for points in the world, in either 3D world coordinates or 2D screen coordinates. We present a number of examples illustrating how registration error estimates can be used in AR interfaces, and describe a method for estimating registration errors of objects based on the expansion and contraction of their 2D convex hulls.

The paper analyzes the properties of the full covariance simultaneous localization and map building problem (SLAM). We prove that, even for the special case of a stationary vehicle (with no process noise) which uses a range-bearing sensor and has non-zero angular uncertainty, the full covariance SLAM algorithm always yields an inconsistent map. We also show, through simulations, that these conclusions appear to extend to a moving vehicle with process noise. However, these inconsistencies only become apparent after several hundred beacon updates.

This paper describes a sparse weight Kalman filter algorithm for simultaneous localisation and map building (SLAM). This algorithm trades optimality for a form of the weight equation which confers computational advantages. For a map of n beacons, the storage is O(n/sup 2/) and the computational costs are O(n). We show that, in a simulation, the method yields results which are similar to the optimal Kalman filter and the suboptimal update method proposed by Guivant et al. (2000).

This paper develops a simultaneous localisation and map building (SLAM) algorithm which utilises the split covariance intersection (SCI) update rule. This algorithm decomposes estimates into a correlated component (whose precise structure is unknown) and a lower bound on an independent component. For a map of n beacons, the storage is O(n) and the computational costs are constant irrespective of map size. In a simple simulation example we show that the SCI algorithm, through exploiting a lower bound on independent information, performs substantially better than the traditional covariance intersection SLAM algorithm.

This paper describes the software architecture of Dragon, a real-time situational awareness virtual environment for battlefield visualization. Dragon receives data from a number of different sources and creates a single, coherent, and consistent three-dimensional display. We describe the problem of Battlefield Visualization and the challenges it imposes. We discuss the Dragon architecture, the rational for its design, and its performance in an actual application. The battlefield VR system is also suitable for similar civilian domains such as large-scale disaster relief and hostage rescue.

We describe the efforts being carried out at the Naval Research Laboratory (NRL) towards VR scientific visualization. We are exploring scientific visualization in an immersive virtual environment: the NRL's CAVE/sup TM/-like device known as GROTTO (Graphical room for observation, Training and Tactical Orientation). We describe the AVS GROTTO viewer, a VR interface to the AVS visualization system. The AVS GROTTO viewer has been used by a number of scientists in current, ongoing research projects within NRL.

A nonlinear process model for the navigation of a high-speed conventional road vehicle is described. In simulations it is shown to significantly reduce the errors in estimating of vehicle position and orientation. The model also performs limited online estimation of certain critical tyre parameters such as mean radius and stiffness.