Modularity, modifiability, reusability, and API usability are important software qualities that determine the maintainability of software architectures. Virtual, Augmented, and Mixed Reality (VR, AR, MR) systems, modern computer games, as well as interactive human-robot systems often include various dedicated input-, output-, and processing subsystems. These subsystems collectively maintain a real-time simulation of a coherent application state. The resulting interdependencies between individual state representations, mutual state access, overall synchronization, and flow of control implies a conceptual close coupling whereas software quality asks for a decoupling to develop maintainable solutions. This article presents five semantics-based software techniques that address this contradiction: Semantic grounding, code from semantics, grounded actions, semantic queries, and decoupling by semantics. These techniques are applied to extend the well-established entity-component-system (ECS) pattern to overcome some of this pattern's deficits with respect to the implied state access. A walk-through of central implementation aspects of a multimodal (speech and gesture) VR-interface is used to highlight the techniques' benefits. This use-case is chosen as a prototypical example of complex architectures with multiple interacting subsystems found in many VR, AR and MR architectures. Finally, implementation hints are given, lessons learned regarding maintainability pointed-out, and performance implications discussed.

Classical reminiscence therapy has been shown to effectively enhance the stability of memory and identity in people with dementia. Typically, reminiscence therapy uses biography artifacts like photos and personal items and objects. Today, many of these artifacts are from the digital realm providing new options to adapt or even improve the purely analog therapy. In this work we propose a method to enhance reminiscence therapy by computer simulated biographic associations. Our approach provides assistance for associative reasoning on affective stimuli and thus enables access to biographic content so that no deliberate search is required. We develop a recommender model for mapping mental states to biographic content based on similarity. The system dynamically adapts its state and the depicted digital artifacts to the responses of the user. It is a first step towards an immersive reminiscence therapy which will incorporate associated stimuli on multiple channels to increase effectiveness. A preliminary study showed encouraging results concerning the usability of the system.

This paper presents a mixed reality tabletop role-playing game with a novel combination of interaction styles and gameplay mechanics. Our contribution extends previous approaches by abandoning the traditional turn-based gameplay in favor of simultaneous real-time interaction. The increased cognitive and physical load during the simultaneous control of multiple game characters is counteracted by two features: First, certain game characters are equipped with AI-driven capabilities to become semi-autonomous virtual agents. Second, (groups of) these agents can be instructed by high-level commands via a multimodal - speech and gesture - interface.

We present four raycast-based techniques that determine the transformation between a depth camera's coordinate system and the coordinate system defined by a rectangular surface. In addition, the surface's dimensions are measured. In contrast to other approaches, these techniques limit additional hardware requirements to commonly available, low-cost artifacts and focus on simple non-laborious procedures. A preliminary study examining our Kinect v2-based proof of concept revealed promising first results. The utilized software is available as an open-source project.

Today's physics engines mainly simulate classical mechanics and rigid body dynamics, with some late advances also capable of simulating massive particle systems and some approximations of fluid dynamics. An accurate numerical simulation of complex nonmechanical processes in real time is beyond the state of the art in the respective fields. This paper illustrates an alternative approach to a purely numerical solution. It uses a semantic representation of physical properties and processes as well as a reasoning engine to model cause and effect between objects, based on their material properties. Classical collision detection is combined with semantic rules to model various physical processes, for example, in the areas of thermodynamics, electrodynamics, and fluid dynamics as well as chemical processes. Each process is broken down into fine-grained subprocesses capable of approximating continuous transitions with discretized state changes. Our system applies these high-level state descriptions to low-level value changes, which are directly mapped to a graphical representation of the scene. We demonstrate our framework's ability to support multiple complex, causally connected physical and chemical processes by simulating a Goldberg machine. Our performance benchmarks validate its scalability and potential application for entertainment or edutainment purposes.

Precise, accurate, fast, and low-latency motion tracking is a core requirement of real-time human-computer interfaces. The choices of tracking systems for a particular set of 3D interaction techniques are manifold. Hence, guidance in this task is greatly beneficial. In this paper, we propose to establish a set of canonical and simple game-based benchmarks for a potentially standardised comparison of tracking systems. The benchmarks focus on usability scores given reoccurring interaction tasks without requiring potentially missing, incomplete, or complex latency or accuracy raw measurements. Our first two benchmarks evaluate three tracking systems regarding motion-parallax and 3D object manipulation techniques. Our usability comparisons confirmed an expected advantage of low-latency/high-accuracy systems, while they also demonstrated that certain tracking systems perform better than suggested by previous measurements of their raw performances. This indicates that our approach provides an adequate replacement and improvement over the pure comparison of technical specifications. We believe our benchmarks could benefit the research community by facilitating a usability-based comparison of motion tracking systems.

This article evaluates the utility of three technical design approaches implemented during the development of a Realtime Interactive Systems (RIS) architecture focusing on the areas of Virtual and Augmented Reality (VR and AR), Robotics, and Human-Computer Interaction (HCI). The design decisions are (1) the choice of the Scala programming language, (2) the implementation of the actor computational model, and (3) the central incorporation of ontologies as a base for semantic modeling, required for several Artificial Intelligence (AI) methods. A white-box expert review is applied to a detailed use case illustrating an interactive and multimodal game scenario, which requires a number of complex functional features like speech and gesture processing and instruction mapping. The review matches the three design decisions against three comprehensive non-functional requirements from software engineering: Reusability, scalability, and extensibility. The qualitative evaluation is condensed to a semi-quantitative summary, pointing out the benefits of the chosen technical design.

We present SiXton 's Curse-a computer game-to illustrate the ben efits of a novel simulation platform. Simulator X [2] targets vir tual, augmented, and mixed reality applications as well as computer games. The game simulates a medieval village called SiXton that can be explored and experienced using gestures and speech for in put. SiXton's Curse utilizes multiple independent components for physical simulation, sound and graphics rendering, artificial intelli gence, as well as for multi-modal interaction (MMI). The compo nents are already an integral part of Simulator X's current version. Building on Hewitt's actor model, the Simulator X platform enables the developer to easily exploit the capabilities of modern hardware architectures. A state variable concept is implemented on top of the actor model to grant uniform and easy access to global states and values by using the internal mechanisms of the actor model. Communication via an asynchronous messaging in terface reduces component coupling. The scalability of the actor model provides a uniform concurrency paradigm on different levels of granularity as well as exchangeability of architectural elements and components.