In musical collaboration, digital musical instruments often hinder effective communication and engagement by restricting visibility and limiting gestural and non-verbal interactions.These challenges reduce musicians' situational awareness and complicate cohesive performance.To address this, we developed a head-mounted augmented reality (AR) system to enhance collaborative musical experiences by visualising musicians' hand movements, eye gaze positions, and instrument interactions in real-time.We conducted a user study involving four pairs of musicians performing live music using different AR interface configurations.The results suggest that the AR system can enhance situational awareness and assist collaboration, as reflected in questionnaire responses.Interviews indicated that real-time visualisations of bodily movements and interactions helped participants better understand the collaborative process and anticipate their collaborators' actions.These findings point to the potential of AR-assisted visualisation to support creative collaboration by tailoring visual information to different needs.Future research could explore its application in broader contexts of real-time creative cooperation.

To enable robots to comprehend high-level human instructions and perform complex tasks, a key challenge lies in achieving comprehensive scene understanding: interpreting and interacting with the 3D environment in a meaningful way. This requires a smart map that fuses accurate geometric structure with rich, human-understandable semantics. To address this, we introduce the 3D Queryable Scene Representation (3D QSR), a novel framework built on multimedia data that unifies three complementary 3D representations: (1) 3D-consistent novel view rendering and segmentation from panoptic reconstruction, (2) precise geometry from 3D point clouds, and (3) structured, scalable organization via 3D scene graphs. Built on an object-centric design, the framework integrates with large vision-language models to enable semantic queryability by linking multimodal object embeddings, and supporting object-level retrieval of geometric, visual, and semantic information. The retrieved data are then loaded into a robotic task planner for downstream execution.

Gesture Elicitation Studies (GES) are a widely used empirical method to develop gesture vocabularies, interaction models and methods for gesture-based systems in different contexts. While GES show great promise to identify user-defined gestures, there are inherent problems with current methods used for GES. Especially during the ongoing pandemic, it has been nearly impossible to conduct in-person, in-lab GES, while ensuring the safety and well-being of the participants, and complying with social distancing regulations. Further, with prevailing experiment designs, increasing the number of participants is time consuming, while in-lab environments also limit ecological validity. This study explores an intuitive way of conducting self-guided GES using immersive Virtual Reality (VR), utilizing its capability to simulate various contexts to enhance ecological validity. We present a methodology and a tool set that use an immersive VR environment to conduct ecologically valid GES (as a use case) while requiring minimal involvement by the investigator. We evaluate our method using the case of a smart home environment and measure participant acceptance and discuss opportunities and challenges involved in this method. We believe that this study will help HCI research to move forward with participatory design research, even when lab experiments are difficult to conduct.

In augmented reality (AR), gesture/hand-based interactions are becoming more common place over tangible user interfaces (TUIs) and physical controllers. Major concerns regard portability and battery-life with TUIs and physical controllers in an AR environment. For a TUI or physical controller to be usable ”on the go” and in the field, they need to be compact such that they can be easily deployable. Subsequently, it is desirable to be low-energy or powerless, such that the device remains functional after long periods. In this paper, we present our initial design towards a powerless, passive controller that leverages the optical camera of a AR head-mounted display (such as the Hololens or Magic Leap) for tracking. We discuss related work, design motivations, high and low fidelity designs, opportunistic/adaptable input modalities, and use cases. We conclude with propositions for future work.

Indoor asset tracking is an essential task in many areas of industry such as shipping and warehousing. Widely-used asset tracking technologies typically require supporting infrastructure (signal transponders) to communicate with active tags on the assets. Continuous asset tracking in large indoor spaces like warehouses can be costly, and reliable reference points are needed to calculate asset positions accurately. In this paper, we describe an indoor asset tracking technique for augmented reality (AR) that combines the use of (passive) fiducial markers together with flexible spatial anchors. Our approach, called SABIAT, continuously tracks the approximate location of an asset using spatial anchors and, when needed, the precise location of that asset using fiducial markers. We have applied our SABIAT technique to build a demonstrator system, AR-IPS, to show how assets can be tracked and located inside of a large, multi-level building.

In this paper, we present a novel collaboration tool, OmniGlobeVR, which is an asymmetric system that supports communication and collaboration between a VR user (occupant) and multiple non-VR users (designers) across the virtual and physical platform. OmniGlobeVR allows designer(s) to explore the VR space from any point of view using two view modes: a 360° first-person mode and a third-person mode. In addition, a shared gaze awareness cue is provided to further enhance communication between the occupant and the designer(s). Finally, the system has a face window feature that allows designer(s) to share their facial expressions and upper body view with the occupant for exchanging and expressing information using nonverbal cues. We conducted a user study to evaluate the OmniGlobeVR, comparing three conditions: (1) first-person mode with the face window, (2) first-person mode with a solid window, and (3) third-person mode with the face window. We found that the first-person mode with the face window required significantly less mental effort, and provided better spatial presence, usability, and understanding of the partner's focus. We discuss the design implications of these results and directions for future research.

Remote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D virtual reconstruction of their surroundings. However, each of these techniques has benefits and drawbacks. In this paper we explore mixing 360 video and 3D reconstruction together for remote collaboration, by preserving benefits of both systems while reducing drawbacks of each. We developed a hybrid prototype and conducted user study to compare benefits and problems of using 360 or 3D alone to clarify the needs for mixing the two, and also to evaluate the prototype system. We found participants performed significantly better on collaborative search tasks in 360 and felt higher social presence, yet 3D also showed potential to complement. Participant feedback collected after trying our hybrid system provided directions for improvement.

We propose enhancing live 360 panorama-based Mixed Reality (MR) remote collaboration through supporting visual annotation cues. Prior work on live 360 panorama-based collaboration used MR visualization to overlay visual cues, such as view frames and virtual hands, yet they were not registered onto the shared physical workspace, hence had limitations in accuracy for pointing or marking objects. Our prototype system uses spatial mapping and tracking feature of an Augmented Reality head-mounted display to show visual annotation cues accurately registered onto the physical environment. We describe the design and implementation details of our prototype system, and discuss on how such feature could help improve MR remote collaboration.

Mixed Reality (MR) remote collaboration provides an enhanced immersive experience where a remote user can provide verbal and nonverbal assistance to a local user to increase the efficiency and performance of the collaboration. This is usually achieved by sharing the local user's environment through live 360 video or a 3D scene, and using visual cues to gesture or point at real objects allowing for better understanding and collaborative task performance. While most of prior work used one of the methods to capture the surrounding environment, there may be situations where users have to choose between using 360 panoramas or 3D scene reconstruction to collaborate, as each have unique benefits and limitations. In this paper we designed a prototype system that combines 360 panoramas into a 3D scene to introduce a novel way for users to interact and collaborate with each other. We evaluated the prototype through a user study which compared the usability and performance of our proposed approach to live 360 video collaborative system, and we found that participants enjoyed using different ways to access the local user's environment although it took them longer time to learn to use our system. We also collected subjective feedback for future improvements and provide directions for future research.

Digitisation of natural history collections not only preserves precious information about biological diversity, it also enables us to share, analyse, annotate and compare specimens to gain new insights. High-resolution, full-colour 3D capture of biological specimens yields color and geometry information complementary to other techniques (e.g., 2D capture, electron scanning and micro computed tomography). However 3D colour capture of small specimens is slow for reasons including specimen handling, the narrow depth of field of high magnification optics, and the large number of images required to resolve complex shapes of specimens. In this paper, we outline techniques to accelerate 3D image capture, including using a desktop robotic arm to automate the insect handling process; using a calibrated pan-tilt rig to avoid attaching calibration targets to specimens; using light field cameras to capture images at an extended depth of field in one shot; and using 3D Web and mixed reality tools to facilitate the annotation, distribution and visualisation of 3D digital models.

This paper presents our new Virtual Reality (VR) interactive visualization techniques to assist users querying large image sets. The VR system allows users to query a set of images on four different filters such as locations and keywords. The goal is to investigate if a VR platform is preferred over a non-VR platform for viewing and querying large image sets. We employed an HTC Vive and a traditional desktop screen to represent VR and non-VR platforms. We found users preferred the VR platform over the traditional desktop screen.

The Lamposcope is a low-cost mixed reality system that enables a shared experience of art or design communication and collaboration mediated by light. Two participants in different places share a single drawing, to which they can contribute using familiar materials such as pencils, pens and oil pastels.

It is our great pleasure to present the ISMAR Tutorials. We proudly host two tutorials that provide sharing of knowledge from seasoned researchers. Our tutorials cover open standards and development using HTML and instantAR. Through these exciting tutorials we hope to expand the minds of ISMAR 2013 attendees and help to foster the next generation of Mixed and Augmented Reality researchers, practitioners, and artists.

This paper presents a method for using object based manipulation and spatial augmented reality for the purpose of remote guidance. Previous remote guidance methods have typically not made use of any semantic information about the physical properties of the environment and require the helper and worker to provide context. Our new prototype system introduces a level of abstraction to the remote expert, allowing them to directly specify the object movements required of a local worker. We use 3D tracking to create a hidden virtual reality scene, mirroring the real world, with which the remote expert interacts while viewing a camera feed of the physical workspace. The intended manipulations are then rendered to the local worker using Spatial Augmented Reality (SAR). We report on the implementation of a functional prototype that demonstrates an instance of this approach. We anticipate that techniques such as the one we present will allow more efficient collaborative remote guidance in a range of physical tasks.

This sketch describes a collaborative virtual environment application involving haptic interaction over long Internet distances. We have developed algorithms to accommodate significant latency for certain applications, notably in the medical domain. The results have shown that we can manipulate simulated human body organs, as well as guide each other's 'hands' (and shake hands!) over 22,000 km.