The ‘keyword method’ is an effective technique for learning vocabulary of a foreign language. It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like in the learner’s native language. However, these memorable visual links remain implicit in the people’s mind and are not easy to remember for a large number of words. To enhance the memorisation and recall of the vocabulary, we developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals. These visuals represent additional stimuli during the memorisation process. To explore the effectiveness of this approach we first run a pilot study to investigate how difficult it is to externalise the descriptions of mental visualisations of memorable links, by asking participants to write them down. We used these descriptions as prompts for text-to-image generator (DALL-E 2) to convert them into images and asked participants to select their favourites. Next, we compared different text-to-image generators (DALL-E 2, Midjourney, Stable and Latent Diffusion) to evaluate the perceived quality of the generated images by each. Despite heterogeneous results, participants mostly preferred images generated by DALL-E 2, which was used also for the final study. In this study, we investigated whether providing such images enhances the retention of vocabulary being learned, compared to the keyword method alone. Our results indicate that people did not encounter difficulties describing their visualisations of memorable links and that providing corresponding images significantly increases memory retention.

We examined eye and head movements to gain insights into skill development in clinical settings. A total of 24 practitioners participated in simulated baby delivery training sessions. We calculated key metrics, including pupillary response rate, fixation duration, or angular velocity. Our findings indicate that eye and head tracking can effectively differentiate between trained and untrained practitioners, particularly during labor tasks. For example, head-related features achieved an F1 score of 0.85 and AUC of 0.86, whereas pupil-related features achieved F1 score of 0.77 and AUC of 0.85. The results lay the groundwork for computational models that support implicit skill assessment and training in clinical settings by using commodity eye-tracking glasses as a complementary device to more traditional evaluation methods such as subjective scores.

Augmented reality (AR) magic-lens (ML) displays, such as handheld devices, offer a convenient and accessible way to enrich our environment using virtual imagery. Several display technologies, including conventional monocular, less common stereoscopic, and varifocal displays, are currently being used. Vergence and accommodation effects on depth perception, as well as vergence-accommodation conflict, have been studied, where users interact only with the content on the display. However, little research exists on how vergence and accommodation influence user performance and cognitive-task load when users interact with the content on a display and its surroundings in a short timeframe. Examples of this are validating augmented instructions before making an incision and performing general hand-eye coordinated tasks such as grasping augmented objects. To improve interactions with future AR displays in such scenarios, we must improve our understanding of this influence. To this end, we conducted two fundamental visual-acuity user studies with 28 and 27 participants, while investigating eye vergence and accommodation distances on four ML displays. Our findings show that minimizing the accommodation difference between the display and its surroundings is crucial when the gaze between the display and its surroundings shifts rapidly. Minimizing the difference in vergence is more important when viewing the display and its surroundings as a single context without shifting the gaze. Interestingly, the vergence-accommodation conflict did not significantly affect the cognitive-task load nor play a pivotal role in the accuracy of interactions with AR ML content and its physical surroundings.

This article explores video-viewing behavior when videos are wrapped in interactive content in the case of iOtok, a 13-episodes web documentary series. The interaction and viewing data were collected over a period of one year, providing a dataset of more than 12,200 total video views by 6000 users. Standard metrics (video views, percentage viewed, number of sessions) show higher active participation for registered users compared to unregistered users. Results also indicate that serialization over multiple weeks is an effective strategy for audience building over a long period of time without negatively affecting video views. In viewing behavior analysis, we focused on three perspectives: (i) regularity (watching on a weekly basis or not), (ii) intensity (number of videos per session), and (iii) order of watching. We performed a perspective based and combined perspectives analysis involving manual coding techniques, rule-based, and k-means clustering algorithms to reveal different user profiles (intermittent, exemplary, detached, enthusiastic users, and nibblers) and highlight further viewing behavior differences (e.g., post-series users binge-watched more than concurrent users during first 13 weeks while the series was weekly released). We discuss how these results can be used to inform the design and promotion of future web documentaries.

Ubiquitous nature, versatility, and durability enabled paper to maintain its importance in the digital age. It is not surprising there have been numerous attempts to combine paper with the digital content. One way to do so is to place paper on a horizontal interactive display (e.g. tabletop or tablet). The paper thus becomes "the screen" on which the digital content is viewed, yet it also acts as a barrier that degrades the quality of the perceived image. This research tries to address this problem by proposing and evaluating a novel paper display concept called Dynamic pinhole paper. The concept is based on perforating the paper (to decrease its opacity) and moving digital content beneath the perforated area (to increase the resolution). To evaluate this novel concept, we fabricated the pinhole paper and implemented the software in order to run multiple user studies exploring the concept’s viability, optimal movement trajectory (amount, direction and velocity), and the effect of perforation on printing, writing and reading. The results show that the movement of digital content is a highly effective strategy for improving the resolution of the digital content through perforation where the optimal velocity is independent from trajectory direction (e.g. horizontal or circular) and amount of movement. Results also show the concept is viable on of the shelf hardware and that it is possible to write and print on perforated paper.

Despite the drive to digitise learning, paper still holds a prominent role within educational settings. While computational devices have several advantages over paper (e.g. changing and showing content based on user interaction and needs) their prolonged or incorrect usage can hinder educational achievements. In this paper, we combine the interactivity of computational devices with paper whilst reducing the usage of technology to the minimum. To this end, we developed and evaluated a novel back-print illumination paper display called LightMeUp where different information printed on the back side of the paper becomes visible when paper is placed on an interactive display and back-illuminated with a particular colour. To develop this novel display, we first built a display simulator that enables the simulation of various spectral characteristics of the elements used in the system (i.e. light sources such as tablet computers, paper types and printing inks). By using our simulator, we designed various use-case prototypes that demonstrate the capabilities and feasibility of the proposed system. With our simulator and use-cases presented, educators and educational content designers can easily design multi-stable interactive visuals by using readily available paper, printers and touch displays.

Humans have been developing and playing musical instruments for millennia. With technological advancements, instruments were becoming ever more sophisticated. In recent decades computer-supported innovations have also been introduced in hardware design, usability, and aesthetics. One of the most commonly digitally augmented instruments is the piano. Besides electronic keyboards, several prototypes augmenting pianos with different projections providing various levels of interactivity on and around the keyboard have been implemented in order to support piano players. However, it is still unclear whether these solutions support the learning process. In this paper, we present a systematic review of augmented piano prototypes focusing on instrument learning based on the four themes derived from interviews with piano experts to understand better the problems of teaching the piano. These themes are (i) synchronised movement and body posture, (ii) sight-reading, (iii) ensuring motivation, and (iv) encouraging improvisation. We found that prototypes are saturated on the synchronisation themes, and there are opportunities for sight-reading, motivation, and improvisation themes. We conclude by presenting recommendations on augmenting piano systems towards enriching the piano learning experience as well as on possible directions to expand knowledge in the area.

Gesture recognition with miniaturised radar sensors has received increasing attention as a novel interaction medium. The practical use of radar technology, however, often requires sensing through materials. Yet, it is still not well understood how the internal structure of materials impacts recognition performance. To tackle this challenge, we collected a large dataset of 14,090 radar recordings for 6 paradigmatic gesture classes sensed through a variety of everyday materials, performed by humans (6 materials) and a robot system (75 materials). Next, we developed a hybrid CNN+LSTM deep learning model and derived a robust indirect method to measure signal distortions, which we used to compile a comprehensive catalogue of materials for radar-based interaction. Among other findings, our experiments show that it is possible to estimate how different materials would affect gesture recognition performance of arbitrary classifiers by selecting just 3 reference materials. Our catalogue, software, models, data collection platform, and labeled datasets are publicly available.

Virtual Reality (VR) has the potential to support mobile knowledge workers by complementing traditional input devices with a large three-dimensional output space and spatial input. Previous research on supporting VR knowledge work explored domains such as text entry using physical keyboards and spreadsheet interaction using combined pen and touch input. Inspired by such work, this paper probes the VR design space for authoring presentations in mobile settings. We propose PoVRPoint-a set of tools coupling pen- and touch-based editing of presentations on mobile devices, such as tablets, with the interaction capabilities afforded by VR. We study the utility of extended display space to, for example, assist users in identifying target slides, supporting spatial manipulation of objects on a slide, creating animations, and facilitating arrangements of multiple, possibly occluded shapes or objects. Among other things, our results indicate that 1) the wide field of view afforded by VR results in significantly faster target slide identification times compared to a tablet-only interface for visually salient targets; and 2) the three-dimensional view in VR enables significantly faster object reordering in the presence of occlusion compared to two baseline interfaces. A user study further confirmed that the interaction techniques were found to be usable and enjoyable.

Virtual Reality (VR) provides new possibilities for modern knowledge work. However, the potential advantages of virtual work environments can only be used if it is feasible to work in them for an extended period of time. Until now, there are limited studies of long-term effects when working in VR. This paper addresses the need for understanding such long-term effects. Specifically, we report on a comparative study $i$, in which participants were working in VR for an entire week-for five days, eight hours each day-as well as in a baseline physical desktop environment. This study aims to quantify the effects of exchanging a desktop-based work environment with a VR-based environment. Hence, during this study, we do not present the participants with the best possible VR system but rather a setup delivering a comparable experience to working in the physical desktop environment. The study reveals that, as expected, VR results in significantly worse ratings across most measures. Among other results, we found concerning levels of simulator sickness, below average usability ratings and two participants dropped out on the first day using VR, due to migraine, nausea and anxiety. Nevertheless, there is some indication that participants gradually overcame negative first impressions and initial discomfort. Overall, this study helps lay the groundwork for subsequent research, by clearly highlighting current shortcomings and identifying opportunities for improving the experience of working in VR.

Learning vocabulary in a primary or secondary language is enhanced when we encounter words in context. This context can be afforded by the place or activity we are engaged with. Existing learning environments include formal learning, mnemonics, flashcards, use of a dictionary or thesaurus, all leading to practice with new words in context. In this work, we propose an enhancement to the language learning process by providing the user with words and learning tools in context, with VocabulARy. VocabulARy visually annotates objects in AR, in the user's surroundings, with the corresponding English (first language) and Japanese (second language) words to enhance the language learning process. In addition to the written and audio description of each word, we also present the user with a keyword and its visualisation to enhance memory retention. We evaluate our prototype by comparing it to an alternate AR system that does not show an additional visualisation of the keyword, and, also, we compare it to two non-AR systems on a tablet, one with and one without visualising the keyword. Our results indicate that AR outperforms the tablet system regarding immediate recall, mental effort and task-completion time. Additionally, the visualisation approach scored significantly higher than showing only the written keyword with respect to immediate and delayed recall and learning efficiency, mental effort and task-completion time.

Experiential learning (ExL) is the process of learning through experience or more specifically "learning through reflection on doing". In this paper, we propose a simulation of these experiences, in Augmented Reality (AR), addressing the problem of language learning. Such systems provide an excellent setting to support "adaptive guidance", in a digital form, within a real environment. Adaptive guidance allows the instructions and learning content to be customised for the individual learner, thus creating a unique learning experience. We developed an adaptive guidance AR system for language learning, we call Arigato (Augmented Reality Instructional ¯ Guidance & Tailored Omniverse), which offers immediate assistance, resources specific to the learner's needs, manipulation of these resources, and relevant feedback. Considering guidance, we employ this prototype to investigate the effect of the amount of guidance (fixed vs. adaptive-amount) and the type of guidance (fixed vs. adaptive-associations) on the engagement and consequently the learning outcomes of language learning in an AR environment. The results for the amount of guidance show that compared to the adaptive-amount, the fixed-amount of guidance group scored better in the immediate and delayed (after 7 days) recall tests. However, this group also invested a significantly higher mental effort to complete the task. The results for the type of guidance show that the adaptive-associations group outperforms the fixed-associations group in the immediate, delayed (after 7 days) recall tests, and learning efficiency. The adaptive-associations group also showed significantly lower mental effort and spent less time to complete the task.

Interactive web documentaries have the potential to engage people with important topics through innovative storytelling approaches. Understanding how users interact with web documentaries and to what extent they consume their content can help documentary filmmakers reach a broader or more engaged audience. In this article, we provide an in-depth and quantitative analysis of an interactive web documentary called iOtok. The 13 episodes of iOtok were launched on an online portal showcasing a variety of features (chat, souvenirs, animated 360 panoramas, etc.), on a weekly basis. Through the analysis of more than 20 000 sessions, newsletter statistics and one online questionnaire, we discuss the impact of serialization and interactivity on audience reception and user engagement. Data show that in the case of iOtok, serialization had a positive impact (large number of sessions over time, long time on site). However, and somehow unexpectedly, every click led to a considerable amount of users dropping off and quitting the web portal before reaching the core videos. This article also addresses other aspects of web documentaries, such as the efficiency of various media to reach and engage users over time and differences in behaviors between registered and unregistered users.

Digital-augmentation of print-media can provide contextually relevant audio, visual, or haptic content to supplement the static text and images. The design of such augmentation--its medium, quantity, frequency, content, and access technique--can have a significant impact on the reading experience. In the worst case, such as where children are learning to read, the print medium can become a proxy for accessing digital content only, and the textual content is avoided. In this work, we examine how augmented content can change the reader's behaviour with a comic book. We first report on the usage of a commercially available augmented comic for children, providing evidence that a third of all readers converted to simply viewing the digital media when printed content is duplicated. Second, we explore the design space for digital content augmentation in print media. Third, we report a user study with 136 children that examined the impact of both content length and presentation in a digitally-augmented comic book. From this, we report a series of design guidelines to assist designers and editors in the development of digitally-augmented print media.

Since ancient times travellers and tourists were carving or writing their names and messages on historic landmarks. This behaviour has prevailed to this day as tourists try to leave their marks at places they visit. Such behaviour, today often seen as vandalism, is particularly problematic since the society tries to preserve historic landmarks while graffiti often leave indelible markings. One solution to this problem is to allow tourists to write digital graffiti projected on historic landmarks and other public surfaces as an additional tourist offer. Projecting digital information on walls does not leave permanent marks while still allows authors to “physically” mark the place they visited. In this paper we frame our vision and highlight the approach we plan to pursue within the context of this topic.

The magic lens paradigm, a commonly used descriptor for handheld Augmented Reality (AR), presents the user with dual views: the augmented view (magic lens) that appears on the device, and the real view of the surroundings (what the user can see around the perimeter of the device). The augmented view is typically implemented by rendering the video captured by the rear-facing camera directly onto the device's screen. This results in dual perspectives - the real world being captured from the device's perspective rather than the user's perspective (what an observer would see looking through a transparent glass pane). These differences manifest themselves in misaligned and/or incorrectly scaled transparency resulting in the dual-view problem.

Typically handheld AR systems utilize a single back-facing camera and the screen in order to implement device transparency. This creates the dual-view problem a consequence of virtual transparency which does not match true transparency - what the user would see looking through a transparent glass pane. The dual-view problem affects usability of handheld AR systems and is commonly addressed though user-perspective rendering solutions. Whilst such approach produces promising results, the complexity of implementing user-perspective rendering and the fact it does not solve all sources that produce the dual-view problem, means it only ever addresses part of the problem. This paper seeks to create a more complete solution for the dual-view problem that will be applicable to readily available handheld-device. We pursue this goal by designing, implementing and evaluating a novel interaction paradigm we call `contact-view'. By utilizing the back and front-facing camera and the environment base-plane texture - predefined or incrementally created on the fly, we enable placing the device directly on top of the base-plane. As long as the position of the phone in relation to the base-plane is known, appropriate segment of the occluded base-plane can be rendered on the device screen, result of which is transparency in which dual-view problem is eliminated.

In Augmented Reality (AR) the real world is enhanced by superimposed digital information commonly visualized through augmented annotations. The visualized data comes from many different data sources. One increasingly important source of data is user generated content. Unfortunately, AR tools that support user generated content are not common hence the majority of augmented data within AR applications is not generated utilizing AR technology. In this paper we discuss the main reasons for this and evaluate how the contact-view paradigm could enhance annotation authoring process within the class of tabletop size AR workspaces. This evaluation is based on a prototype that allows musicians to annotate a music score manuscript utilizing freehand drawing on top of device screen. Experimentation showed the potential of contact-view paradigm as an annotation authoring method that performs well in single and collaborative multi-user situations.

In handheld Augmented Reality (AR) the magic-lens paradigm is typically implemented by rendering the video stream captured by the back-facing camera onto the device's screen. Unfortunately, such implementations show the real world from the device's perspective rather than the user's perspective. This dual-perspective results in misaligned and incorrectly scaled imagery, a predominate cause for the dual-view problem with potential to distort user's spatial perception. This paper presents a user study that analyzes users' expectations, spatial-perception, and their ability to deal with the dual-view problem, by comparing device-perspective and fixed Point-of-View (POV) user-perspective rendering. The results confirm the existence of the dual-view perceptual issue and that the majority of participants expect user-perspective rendering irrespective of their previous AR experience. Participants also demonstrated significantly better spatial perception and preference of the user-perspective view.

Handheld Augmented Reality (AR) is often presented using the magic-lens paradigm in which a magic-lens is a transparent interface. Such transparency is usually implemented by rendering camera captured video on the device's screen. The transparency quality is limited by the video stream quality which may be affected by: unfocused camera lens, poor lighting conditions and limited video stream resolution. All these factors may reduce the readability of the AR scene. To address quality of rendering and increase scene readability, this paper presents an enhanced virtual transparency solution where segments of the scene are replaced by high definition digital content. The proposed enhanced virtual transparency isdemonstrated through the design of a digital magnifying glass which has been implemented on of-the-shelf mobile phone.

Handheld Augmented Reality (AR) is often presented using the magic-lens paradigm where the handheld device is portrayed as if it was transparent. Such a virtual transparency is usually implemented using video captured by a single camera rendered on the device's screen. This removes binocular-disparity, which may undermine user's ability to correctly estimate depth when seeing the world through the magic-lens. To confirm such an assumption this paper presents a qualitative user study that compares a magic-lens implemented on a mobile phone and a transparent glass replica. Observational results and questionnaire analysis indicate that binocular-disparity may play a significant role in participants' depth perception. These promising results led to the subsequent implementation of a stereoscopic magic-lens prototype on a commercially available mobile device.

The majority of mobile devices nowadays harness location and orientation-sensing capabilities, permitting for sensor based Mixed Reality (MR) gaming experiences such as Augmented Reality (AR) games. Nonetheless, only few entertainment implementations of sensor based AR systems emerged, predominantly due to low precision of sensory information causing crude and jerky augmentation, significantly impacting meaningful augmentation that inevitably affects gameplay experience. In this paper, we present a novel pervasive mobile AR Location Based Game (LBG) named Time-wARpXplorer (TARX) where a seamless design approach is used in order to address the quality of augmentation. TARX comprises of a mobile client and an online authoring tool, created to encourage visitors and locals to explore the city of Lancaster by travelling back in time and space to discover their immediate and distant surroundings. Within the game, time travel is linked to present day through the implementation of Foursquare's checkin platform to further raise awareness of historic sites. This paper focuses on the design considerations implemented through an iterative design process and prototyping of the mobile client, of which a novel interface emerged.

One of the key challenges of markerless Augmented Reality (AR) systems, where no a priori information of the environment is available, is map and scale initialization. In such systems, the scale is unknown as it is impossible to determine the scale from a sequence of images alone. Implementing scale is vital for ensuring that augmented objects are contextually sensitive to the environment they are projected upon. In this paper we demonstrate a sensor and vision fusion approach for robust and user-friendly initialization of map and scale. The map is initialized, using inbuilt accelerometers, whilst scale is initialized by the camera auto-focusing capability. The later is possible by applying the Depth From Focus (DFF) method, which was, till now, limited to high precision camera systems. The demonstrated illustrates benefits of such a system, which is running on a commercially available mobile phone Nokia N900.