In automated UI design generation, a key challenge is the lack of support for iterative processes, as most systems focus solely on end-to-end output. This stems from limited capabilities in interpreting design intent and a lack of transparency for refining intermediate results. To better understand these challenges, we conducted a formative study that identified concrete and actionable requirements for supporting iterative design with Generative Tools. Guided by these findings, we propose PrototypeFlow, a human-centered system for automated UI generation that leverages multi-modal inputs and models. PrototypeFlow takes natural language descriptions and layout preferences as input to generate the high-fidelity UI design. At its core is a theme design module that clarifies implicit design intent through prompt enhancement and orchestrates sub-modules for component-level generation. Designers retain full control over inputs, intermediate results, and final prototypes, enabling flexible and targeted refinement by steering generation and directly editing outputs. Our experiments and user studies confirmed the effectiveness and usefulness of our proposed PrototypeFlow.

Figure 1: We review and categorize VMIs aimed at enhancing context awareness.Our key contribution is a Macro-Micro-Macro level (whole-detail-whole) system design framework, providing actionable references from a Data Modality-Driven perspective: (1) Macro-level contextual factors: considerations for context understanding (Section 3); (2) Micro-level system foundations: input data modality (visual + other modalities), data integration stages, multimodal data processing and evaluation strategies (Sections 4, 5); (3) Macro-level design synthesis: application domains, design considerations and key challenges (Sections 6, 7).

The rise of Large Language Models (LLMs) has streamlined frontend interface creation through tools like Vercel's v0, yet surfaced challenges in design quality (e.g., accessibility, and usability). Current solutions, often limited by their focus, generalisability, or data dependency, fall short in addressing these complexities. Moreover, none of them examine the quality of LLM-generated UI design. In this work, we introduce DesignRepair, a novel dual-stream design guideline-aware system to examine and repair the UI design quality issues from both code aspect and rendered page aspect. We utilised the mature and popular Material Design as our knowledge base to guide this process. Specifically, we first constructed a comprehensive knowledge base encoding Google's Material Design principles into low-level component knowledge base and high-level system design knowledge base. After that, DesignRepair employs a LLM for the extraction of key components and utilizes the Playwright tool for precise page analysis, aligning these with the established knowledge bases. Finally, we integrate Retrieval-Augmented Generation with state-of-the-art LLMs like GPT-4 to holistically refine and repair frontend code through a strategic divide and conquer approach. Our extensive evaluations validated the efficacy and utility of our approach, demonstrating significant enhancements in adherence to design guidelines, accessibility, and user experience metrics.

The ‘keyword method’ is an effective technique for learning vocabulary of a foreign language. It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like in the learner’s native language. However, these memorable visual links remain implicit in the people’s mind and are not easy to remember for a large number of words. To enhance the memorisation and recall of the vocabulary, we developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals. These visuals represent additional stimuli during the memorisation process. To explore the effectiveness of this approach we first run a pilot study to investigate how difficult it is to externalise the descriptions of mental visualisations of memorable links, by asking participants to write them down. We used these descriptions as prompts for text-to-image generator (DALL-E 2) to convert them into images and asked participants to select their favourites. Next, we compared different text-to-image generators (DALL-E 2, Midjourney, Stable and Latent Diffusion) to evaluate the perceived quality of the generated images by each. Despite heterogeneous results, participants mostly preferred images generated by DALL-E 2, which was used also for the final study. In this study, we investigated whether providing such images enhances the retention of vocabulary being learned, compared to the keyword method alone. Our results indicate that people did not encounter difficulties describing their visualisations of memorable links and that providing corresponding images significantly increases memory retention.

Plant sensing plays an important role in modern smart agriculture and the farming industry. Remote radio sensing allows for monitoring essential indicators of plant health, such as leaf water content (WC). While recent studies have shown the potential of using millimeter-wave (mmWave) radar for plant sensing, many overlook crucial factors, such as leaf structure and surface roughness, which can impact the accuracy of the measurements. In this article, we introduce Leafeon, which leverages mmWave radar to measure leaf WC noninvasively. Utilizing electronic beam steering, multiple leaf perspectives are sent to a custom deep neural network, which discerns unique reflection patterns from subtle antenna variations, ensuring accurate and robust leaf WC estimations. We implement a prototype of Leafeon using a Commercial Off-The-Shelf mmWave radar and evaluate its performance with a variety of different leaf types. Leafeon was trained in-lab using high-resolution destructive leaf measurements, achieving a mean absolute error (MAE) of leaf WC as low as 3.17% for the Avocado leaf, significantly outperforming the state-of-the-art approaches with an MAE reduction of up to 55.7%. Furthermore, we conducted experiments on live plants in both indoor and glasshouse experimental farm environments. Our results showed a strong correlation between predicted leaf WC levels and drought events.

In cinematic VR applications, haptic feedback can significantly enhance the sense of reality and immersion for users. The increasing availability of emerging haptic devices opens up possibilities for future cinematic VR applications that allow users to receive haptic feedback while they are watching videos. However, automatically rendering haptic cues from real-time video content, particularly from video motion, is a technically challenging task. In this article, we propose a novel framework called "Video2Haptics" that leverages the emerging bio-inspired event camera to capture event signals as a lightweight representation of video motion. We then propose efficient event-based visual processing methods to estimate force or intensity from video motion in the event domain, rather than the pixel domain. To demonstrate the application of Video2Haptics, we convert the estimated force or intensity to dynamic vibrotactile feedback on emerging haptic gloves, synchronized with the corresponding video motion. As a result, Video2Haptics allows users not only to view the video but also to perceive the video motion concurrently. Our experimental results show that the proposed event-based processing methods for force and intensity estimation are one to two orders of magnitude faster than conventional methods. Our user study results confirm that the proposed Video2Haptics framework can considerably enhance the users' video experience.

An omni-directional (360°) camera captures the entire viewing sphere surrounding its optical center. Such cameras are growing in use to create highly immersive content and viewing experiences. When such a camera is held by a user, the view includes the user’s hand grip, finger, body pose, face, and the surrounding environment, providing a complete understanding of the visual world and context around it. This capability opens up numerous possibilities for rich mobile input sensing. In OmniSense, we explore the broad input design space for mobile devices with a built-in omni-directional camera and broadly categorize them into three sensing pillars: i) near device ii) around device and iii) surrounding device. In addition we explore potential use cases and applications that leverage these sensing capabilities to solve user needs. Following this, we develop a working system to put these concepts into action, by leveraging these sensing capabilities to enable potential use cases and applications. We studied the system in a technical evaluation and a preliminary user study to gain initial feedback and insights. Collectively these techniques illustrate how a single, omni-purpose sensor on a mobile device affords many compelling ways to enable expressive input, while also affording a broad range of novel applications that improve user experience during mobile interaction.

This work focuses on an active topic in the HCI community, namely tutorial creation by demonstration. We present a novel tool named SmartRecorder that facilitates people, without video editing skills, creating video tutorials for smartphone interaction tasks. As automatic interaction trace extraction is a key component to tutorial generation, we seek to tackle the challenges of automatically extracting user interaction traces on smartphones from screencasts. Uniquely, with respect to prior research in this field, we combine computer vision techniques with IMU-based sensing algorithms, and the technical evaluation results show the importance of smartphone IMU data in improving system performance. With the extracted key information of each step, SmartRecorder generates instructional content initially and provides tutorial creators with a tutorial refinement editor designed based on a high recall (99.38%) of key steps to revise the initial instructional content. Finally, SmartRecorder generates video tutorials based on refined instructional content. The results of the user study demonstrate that SmartRecorder allows non-experts to create smartphone usage video tutorials with less time and higher satisfaction from recipients.

Everyday, billions of people use footwear for walking, running, or exercise. Of emerging interest are “smart footwear”, which help users track gait, count steps or even analyse performance. However, such nascent footwear lack fine-grain ground surface context awareness, which could allow them to adapt to the conditions and create usable functions and experiences. Hence, this research aims to recognize the walking surface using a radar sensor embedded in a shoe, enabling ground context-awareness. Using data collected from 23 participants from an in-the-wild setting, we developed several classification models. We show that our model can detect five common terrain types with an accuracy of 80.0% and further ten terrain types with an accuracy of 66.3%, while moving. Importantly, it can detect the gait motion types such as ‘walking’, ‘stepping up’, ‘stepping down’, ‘still’, with an accuracy of 90%. Finally, we present potential use cases and insights for future work based on such ground-aware smart shoes.

User engagement in Virtual Reality (VR) games is crucial for creating immersive and captivating gaming experiences that meet the expectations of players. However, understanding and measuring these levels in VR games presents a challenge for game designers, as current methods, such as self-reports, may be limited in capturing the full extent of user engagement. Additionally, approaches based on biological signals to measure engagement in VR games present complications and challenges, including signal complexity, interpretation difficulties, and ethical concerns. This study explores body movements, as a novel approach to measure user engagement in VR gaming. We employ E4, emteqPRO, and off-the-shelf IMUs to measure the body movements from diverse participants engaged in multiple VR games. Further, we examine the simultaneous occurrence of player motivation and physiological responses to explore potential associations with body movements. Our findings suggest that body movements hold promise as a reliable and objective indicator of user engagement, offering game designers valuable insights on generating more engaging and immersive experiences.

International audience

Learning vocabulary in a primary or secondary language is enhanced when we encounter words in context. This context can be afforded by the place or activity we are engaged with. Existing learning environments include formal learning, mnemonics, flashcards, use of a dictionary or thesaurus, all leading to practice with new words in context. In this work, we propose an enhancement to the language learning process by providing the user with words and learning tools in context, with VocabulARy. VocabulARy visually annotates objects in AR, in the user's surroundings, with the corresponding English (first language) and Japanese (second language) words to enhance the language learning process. In addition to the written and audio description of each word, we also present the user with a keyword and its visualisation to enhance memory retention. We evaluate our prototype by comparing it to an alternate AR system that does not show an additional visualisation of the keyword, and, also, we compare it to two non-AR systems on a tablet, one with and one without visualising the keyword. Our results indicate that AR outperforms the tablet system regarding immediate recall, mental effort and task-completion time. Additionally, the visualisation approach scored significantly higher than showing only the written keyword with respect to immediate and delayed recall and learning efficiency, mental effort and task-completion time.

Experiential learning (ExL) is the process of learning through experience or more specifically "learning through reflection on doing". In this paper, we propose a simulation of these experiences, in Augmented Reality (AR), addressing the problem of language learning. Such systems provide an excellent setting to support "adaptive guidance", in a digital form, within a real environment. Adaptive guidance allows the instructions and learning content to be customised for the individual learner, thus creating a unique learning experience. We developed an adaptive guidance AR system for language learning, we call Arigato (Augmented Reality Instructional ¯ Guidance & Tailored Omniverse), which offers immediate assistance, resources specific to the learner's needs, manipulation of these resources, and relevant feedback. Considering guidance, we employ this prototype to investigate the effect of the amount of guidance (fixed vs. adaptive-amount) and the type of guidance (fixed vs. adaptive-associations) on the engagement and consequently the learning outcomes of language learning in an AR environment. The results for the amount of guidance show that compared to the adaptive-amount, the fixed-amount of guidance group scored better in the immediate and delayed (after 7 days) recall tests. However, this group also invested a significantly higher mental effort to complete the task. The results for the type of guidance show that the adaptive-associations group outperforms the fixed-associations group in the immediate, delayed (after 7 days) recall tests, and learning efficiency. The adaptive-associations group also showed significantly lower mental effort and spent less time to complete the task.

The ability to understand, process and evaluate arguments made by others and ourselves is important in many personal and professional spheres, such as political debates. Analysis typically appears in written form, but a growing number of tools support analysis through diagram-based graphical representations. These UIs might support better argument analysis because arguments have non-linear structures that are difficult to convey through linear text. However, there is little empirical evidence on the advantages or mechanisms that might make graph UIs superior to traditional textual documents. We ran and analyzed a study with twenty participants who used text and graph editors to analyze political debates. Our findings demonstrate the tradeoffs between the two approaches and explain key mechanisms that support the analysis in both media.

The automatic recognition of how people use their hands and fingers in natural settings -- without instrumenting the fingers -- can be useful for many mobile computing applications. To achieve such an interface, we propose a vision-based 3D hand pose estimation framework using a wrist-worn camera. The main challenge is the oblique angle of the wrist-worn camera, which makes the fingers scarcely visible. To address this, a special network that observes deformations on the back of the hand is required. We introduce DorsalNet, a two-stream convolutional neural network to regress finger joint angles from spatio-temporal features of the dorsal hand region (the movement of bones, muscle, and tendons). This work is the first vision-based real-time 3D hand pose estimator using visual features from the dorsal hand region. Our system achieves a mean joint-angle error of 8.81 degree for user-specific models and 9.77 degree for a general model. Further evaluation shows that our system outperforms previous work with an average of 20% higher accuracy in recognizing dynamic gestures, and achieves a 75% accuracy of detecting 11 different grasp types. We also demonstrate 3 applications which employ our system as a control device, an input device, and a grasped object recognizer.

research-article Share on Special Issue on Highlights of ACM Intelligent User Interface (IUI) 2018 Authors: Mark Billinghurst School of ITMS, University of South Australia, Adelaide, South Australia, Australia School of ITMS, University of South Australia, Adelaide, South Australia, AustraliaView Profile , Margaret Burnett School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USAView Profile , Aaron Quigley School of Computer Science, University of St. Andrews, St. Andrews, Scotland, United Kingdom School of Computer Science, University of St. Andrews, St. Andrews, Scotland, United KingdomView Profile Authors Info & Claims ACM Transactions on Interactive Intelligent SystemsVolume 10Issue 1March 2020 Article No.: 1pp 1–3https://doi.org/10.1145/3357206Published:12 October 2019Publication History 0citation192DownloadsMetricsTotal Citations0Total Downloads192Last 12 Months27Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We propose RotoSwype, a technique for word-gesture typing using the orientation of a ring worn on the index finger. RotoSwype enables one-handed text-input without encumbering the hand with a device, a desirable quality in many scenarios, including virtual or augmented reality. The method is evaluated using two arm positions: with the hand raised up with the palm parallel to the ground; and with the hand resting at the side with the palm facing the body. A five-day study finds both hand positions achieved speeds of at least 14 words-per-minute (WPM) with uncorrected error rates near 1%, outperforming previous comparable techniques.

To better explore the incorporation of pointing and gesturing into ubiquitous computing, we introduce WRIST, an interaction and sensing technique that leverages the dexterity of human wrist motion. WRIST employs a sensor fusion approach which combines inertial measurement unit (IMU) data from a smartwatch and a smart ring. The relative orientation difference of the two devices is measured as the wrist rotation that is independent from arm rotation, which is also position and orientation invariant. Employing our test hardware, we demonstrate that WRIST affords and enables a number of novel yet simplistic interaction techniques, such as (i) macro-micro pointing without explicit mode switching and (ii) wrist gesture recognition when the hand is held in different orientations (e.g., raised or lowered). We report on two studies to evaluate the proposed techniques and we present a set of applications that demonstrate the benefits of WRIST. We conclude with a discussion of the limitations and highlight possible future pathways for research in pointing and gesturing with wearable devices.

We introduce a vision-based technique to recognize static hand poses and dynamic finger tapping gestures. Our approach employs a camera on the wrist, with a view of the opisthenar (back of the hand) area. We envisage such cameras being included in a wrist-worn device such as a smartwatch, fitness tracker or wristband. Indeed, selected off-the-shelf smartwatches now incorporate a built-in camera on the side for photography purposes. However, in this configuration, the fingers are occluded from the view of the camera. The oblique angle and placement of the camera make typical vision-based techniques difficult to adopt. Our alternative approach observes small movements and changes in the shape, tendons, skin and bones on the opisthenar area. We train deep neural networks to recognize both hand poses and dynamic finger tapping gestures. While this is a challenging configuration for sensing, we tested the recognition with a real-time user test and achieved a high recognition rate of 89.4% (static poses) and 67.5% (dynamic gestures). Our results further demonstrate that our approach can generalize across sessions and to new users. Namely, users can remove and replace the wrist-worn device while new users can employ a previously trained system, to a certain degree. We conclude by demonstrating three applications and suggest future avenues of work based on sensing the back of the hand.

Novices can learn sports in a variety of ways ranging from guidance from an instructor to watching video tutorials. In each case, subsequent and repeated self-directed practice sessions are an essential step. However, during such self-directed practice, constant guidance and feedback is absent. As a result, the novices do not know if they are making mistake or if there are any areas for improvement. In this position paper, we propose using wearable devices to augment such self-directed practice sessions by providing augmented guidance and feedback. In particular, a head-worn display can provide real-time guidance whilst wrist-worn devices can provide real-time tracking and monitoring of various states. We envision this approach being applied to various sports, and in particular this is suitable for sports that utilize precise hand motion such as snooker, billiards, golf, archery, cricket, tennis and table tennis.

Interface designs on both small and large displays can encourage people to alter their physical distance to the display. Mobile devices support this form of interaction naturally, as the user can move the device closer or further away as needed. The current generation of mobile devices can employ computer vision, depth sensing and other inference methods to determine the distance between the user and the display. Once this distance is known, a system can adapt the rendering of display content accordingly and enable proximity-aware mobile interfaces. The dominant method of exploiting proximity-aware interfaces is to remove or superimpose visual information. In this paper, we investigate change blindness in such interfaces. We present the results of two experiments. In our first experiment we show that a proximity-aware mobile interface results in significantly more change blindness errors than a non-moving interface. The absolute difference in error rates was 13.7%. In our second experiment we show that within a proximity-aware mobile interface, gradual changes induce significantly more change blindness errors than instant changes---confirming expected change blindness behavior. Based on our results we discuss the implications of either exploiting change blindness effects or mitigating them when designing mobile proximity-aware interfaces.

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

Psychology researchers employ the Experience Sampling Method (ESM) to capture thoughts and behaviours of participants within their everyday lives. Smartphone-based ESM apps are increasingly used in such research. However, the diversity of researchers' app requirements, coupled with cost and complexity of their implementation, has prompted end-user development (EUD) approaches. In addition, limited evaluation of such environments beyond lab-based usability studies precludes discovery of factors pertaining to real-world EUD adoption. We first describe the extension of Jeeves, our visual programming environment for ESM app creation, in which we implemented additional functional requirements, derived from a survey and analysis of previous work. We further describe interviews with psychology researchers to understand their practical considerations for employing this extended environment in their work practices. Results of our analysis are presented as factors pertaining to the adoption of EUD activities within and between communities of practice.

The popularity of mobile devices with large screens is making single-handed interaction difficult. We propose and evaluate a novel design point around a tilt-based text entry technique which supports single handed usage. Our technique is based on the gesture keyboard (shape writing). However, instead of drawing gestures with a finger or stylus, users articulate a gesture by tilting the device. This can be especially useful when the user's other hand is otherwise encumbered or unavailable. We show that novice users achieve an entry rate of 15 words-per-minute (wpm) after minimal practice. A pilot longitudinal study reveals that a single participant achieved an entry rate of 32 wpm after approximate 90 minutes of practice. Our data indicate that tilt-based gesture keyboard entry enables walk-up use and provides a suitable text entry rate for occasional use and can act as a promising alternative to single-handed typing in certain situations.

SpeCam is a lightweight surface color and material sensing approach for mobile devices which only uses the front-facing camera and the display as a multi-spectral light source. We leverage the natural use of mobile devices (placing it face-down) to detect the material underneath and therefore infer the location or placement of the device. SpeCam can then be used to support discreet micro-interactions to avoid the numerous distractions that users daily face with today's mobile devices. Our two-parts study shows that SpeCam can i) recognize colors in the HSB space with 10 degrees apart near the 3 dominant colors and 4 degrees otherwise and ii) 30 types of surface materials with 99% accuracy. These findings are further supported by a spectroscopy study. Finally, we suggest a series of applications based on simple mobile micro-interactions suitable for using the phone when placed face-down.

Today we can see an increasing number of object recognition systems of very different sizes, portability, embedability and form factors which are starting to become part of the ubiquitous, tangible, mobile and wearable computing ecosystems that we might make use of in our daily lives. These systems rely on a variety of technologies including computer vision, radar, acoustic sensing, tagging and smart objects.

Real-time identification and tracking of the joint positions of people can be achieved with off-the-shelf sensing technologies such as the Microsoft Kinect, or other camera-based systems with computer vision. However, tracking is constrained by the system’s field of view of people. When a person is occluded from the camera view, their position can no longer be followed. Out of Sight addresses the occlusion problem in depth-sensing tracking systems. Our new tracking infrastructure provides human skeleton joint positions during occlusion, by combining the field of view of multiple Kinects using geometric calibration and affine transformation. We verified the technique’s accuracy through a system evaluation consisting of 20 participants in stationary position and in motion, with two Kinects positioned parallel, $$45^{\circ }$$ , and $$90^{\circ }$$ apart. Results show that our skeleton matching is accurate to within 16.1 cm (s.d. = 5.8 cm), which is within a person’s personal space. In a realistic scenario study, groups of two people quickly occlude each other, and occlusion is resolved for $$85\%$$ of the participants. A RESTful API was developed to allow distributed access of occlusion-free skeleton joint positions. As a further contribution, we provide the system as open source.

We present the findings from a longitudinal study of iPad use in a Primary school classroom. While tablet devices have found their way into classroom environments, we still lack in-depth and long-term studies of how they integrate into everyday classroom activities. Our findings illustrate in-classroom tablet use and the broad range of learning activities in subjects such as maths, languages, social sciences, and even physical education. Our observations expand current models on teaching and learning supported by tablet technology. Our findings are child-centred, focusing on three different roles that tablets can play as part of learning activities: Friend, Functionary, and Facilitator. This new perspective on in-classroom tablet use can facilitate critical discussions around the integration and impact of these devices in the educational context, from a design and educational point of view.

The screen size of a smartwatch provides limited space to enable expressive multi-touch input, resulting in a markedly difficult and limited experience. We present WatchMI: Watch Movement Input that enhances touch interaction on a smartwatch to support continuous pressure touch, twist, pan gestures and their combinations. Our novel approach relies on software that analyzes, in real-time, the data from a built-in Inertial Measurement Unit (IMU) in order to determine with great accuracy and different levels of granularity the actions performed by the user, without requiring additional hardware or modification of the watch. We report the results of an evaluation with the system, and demonstrate that the three proposed input interfaces are accurate, noise-resistant, easy to use and can be deployed on a variety of smartwatches. We then showcase the potential of this work with seven different applications including, map navigation, an alarm clock, a music player, pan gesture recognition, text entry, file explorer and controlling remote devices or a game character.

In RadarCat we present a small, versatile radar-based system for material and object classification which enables new forms of everyday proximate interaction with digital devices. We demonstrate that we can train and classify different types of materials and objects which we can then recognize in real time. Based on established research designs, we report on the results of three studies, first with 26 materials (including complex composite objects), next with 16 transparent materials (with different thickness and varying dyes) and finally 10 body parts from 6 participants. Both leave one-out and 10-fold cross-validation demonstrate that our approach of classification of radar signals using random forest classifier is robust and accurate. We further demonstrate four working examples including a physical object dictionary, painting and photo editing application, body shortcuts and automatic refill based on RadarCat. We conclude with a discussion of our results, limitations and outline future directions.

Display devices on and around the body such as smartwatches, head-mounted displays or tablets enable users to interact on the go. However, diverging input and output fidelities of these devices can lead to interaction seams that can inhibit efficient mobile interaction, when users employ multiple devices at once. We present MultiFi, an interactive system that combines the strengths of multiple displays and overcomes the seams of mobile interaction with widgets distributed over multiple devices. A comparative user study indicates that combined head-mounted display and smartwatch interfaces can outperform interaction with single wearable devices.

The Experience Sampling Method (ESM) captures participants' thoughts and feelings in their everyday environments. Mobile and wearable technologies afford us opportunities to reach people using ESM in varying contexts. However, a lack of programming knowledge often hinders researchers in creating ESM applications. In practice, they rely on specialised tools for app creation. Our initial review of these tools indicates that most are expensive commercial services, and none utilise the full potential of sensors for creating context-aware applications. We present “Jeeves”, a visual language to facilitate ESM application creation. Inspired by successful visual languages in literature, our block-based notation enables researchers to visually construct ESM study specifications. We demonstrate its applicability by replicating existing ESM studies found in medical and psychology literature. Our preliminary study with 20 participants demonstrates that both non-programmers and programmers are able to successfully utilise Jeeves. We discuss future work in extending Jeeves with alternative mobile technologies.

Increasing screen real-estate allows for the development of applications where a single user can manage a large amount of data and related tasks through a distributed user interface. However, such users can easily become overloaded and become unaware of display changes as they alternate their attention towards different displays. We propose AwToolkit, a novel widget set for developers that supports users in maintaing awareness in multi-display systems. The AwToolkit widgets automatically determine which display a user is looking at and provide users with notifications with different levels of subtlety to make the user aware of any unattended display changes. The toolkit uses four notification levels (unnoticeable, subtle, intrusive and disruptive), ranging from an almost imperceptible visual change to a clear and visually saliant change. We describe AwToolkit's six widgets, which have been designed for C# developers, and the design of a user study with an application oriented towards healthcare environments. The evaluation results reveal a marked increase in user awareness in comparison to the same application implemented without AwToolkit.

Traditional behaviour change interventions are typically delivered with a fixed set of components, providing identical content to all participants in a trial. The disregard of personal differences often leads to weak effects and inconclusive results. Tools are required that let researchers identify effective components for specific users and contexts. This paper presents a system design incorporating user models and a visual programming language to allow end-users with varying technical expertise to develop tailored interventions using feedback from a series of visual and non-visual interfaces.

With the proliferation of large multi-faceted datasets, a critical question is how to design collaborative environments, in which this data can be analysed in an efficient and insightful manner. Exploiting people's movements and distance to the data display and to collaborators, proxemic interactions can potentially support such scenarios in a fluid and seamless way, supporting both tightly coupled collaboration as well as parallel explorations. In this paper we introduce the concept of collaborative proxemics: enabling groups of people to collaboratively use attention- and proximity-aware applications. To help designers create such applications we have developed SpiderEyes: a system and toolkit for designing attention- and proximity-aware collaborative interfaces for wall-sized displays. SpiderEyes is based on low-cost technology and allows accurate markerless attention-aware tracking of multiple people interacting in front of a display in real-time. We discuss how this toolkit can be applied to design attention- and proximity-aware collaborative scenarios around large wall-sized displays, and how the information visualisation pipeline can be extended to incorporate proxemic interactions.

This paper explores techniques for visualising display changes in multi-display environments. We present four subtle gaze-dependent techniques for visualising change on unattended displays called FreezeFrame, PixMap, WindowMap and Aura. To enable the techniques to be directly deployed to workstations, we also present a system that automatically identifies the user's eyes using computer vision and a set of web cameras mounted on the displays. An evaluation confirms this system can detect which display the user is attending to with high accuracy. We studied the efficacy of the visualisation techniques in a five-day case study with a working professional. This individual used our system eight hours per day for five consecutive days. The results of the study show that the participant found the system and the techniques useful, subtle, calm and non-intrusive. We conclude by discussing the challenges in evaluating intelligent subtle interaction techniques using traditional experimental paradigms.

An increasing number of interactive displays of very different sizes, portability, projectability and form factors are starting to become part of the display ecosystems that we make use of in our daily lives. Displays are shaped by human activity into an ecological arrangement and thus an ecology. Each combination or ecology of displays offer substantial promise for the creation of applications that effectively take advantage of the wide range of input, affordances, and output capability of these multi-display, multi-device and multi-user environments. Although the last few years have seen an increasing amount of research in this area, knowledge about this subject remains under explored, fragmented, and cuts across a set of related but heterogeneous issues. This workshop brings together researchers and practitioners interested in the challenges posed by infrastructure and design.

Attaching a large external display can help a mobile device user view more content at once. This paper reports on a study investigating how different configurations of input and output across displays affect performance, subjective workload and preferences in map, text and photo search tasks. Experimental results show that a hybrid configuration where visual output is distributed across displays is worst or equivalent to worst in all tasks. A mobile device-controlled large display configuration performs best in the map search task and equal to best in text and photo search tasks (tied with a mobile-only configuration). After conducting a detailed analysis of the performance differences across different UI configurations, we give recommendations for the design of distributed user interfaces.

In this paper we present a new bimanual markerless gesture interface for 3D full-body motion tracking sensors, such as the Kinect. Our interface uses a probabilistic algorithm to incrementally predict users' intended one-handed and twohanded gestures while they are still being articulated. It supports scale and translation invariant recognition of arbitrarily defined gesture templates in real-time. The interface supports two ways of gesturing commands in thin air to displays at a distance. First, users can use one-handed and two-handed gestures to directly issue commands. Second, users can use their non-dominant hand to modulate single-hand gestures. Our evaluation shows that the system recognizes one-handed and two-handed gestures with an accuracy of 92.7%--96.2%.

A key problem for social network analysis is the lack of ground-truth data upon which to validate an analysis. Consider for example community-finding algorithms. The ``communities'' identified by such algorithms are typically justified on the basis of their structural properties, rather than on their ability to recover communities which can be independently verified. A ground truth of actual community data isn't always available and at best only partial ground-truth community information is. However, this problem isn't unique to community-finding algorithms. In previous publications, we introduced an automated Social Network Assembly Pipeline we refer to as SNAP. This is intended for the large scale actor identification, tie interference and strength measurement of social networks from non-relational data sets. In this paper we describe a validation study of SNAP through an intensive user-study of a portion of the individuals in the network. Individuals are asked to validate the network relationships uncovered by SNAP and where misclassified relationships are found, the individuals are interviewed in order to determine the underlying cause of the misclassification. The findings provide feedback on the rules through which relationships are inferred. For instance, it becomes clear that an error in actor identification can result in a propagation of this error though the network relations leading to follow-on relationship misclassifications. Also, we observe how outliers lead to a propagation of error in the inferred network. The results help us validate and invalidate different hypotheses we have about SNAP and suggests domain specific rule-sets for SNAP.

A smartphone having touchscreen and short-range networking facilities makes efficient remote control for a large display. In this paper, we report on the results of a case study examining the user performance of Proximal Selection (PS) and Distal Selection (DS) of remote control widgets. DS uses a mobile pointer to zoom-in the region of interest and select the widgets on the large display. PS involves pointing at the large display to transfer the zoom-in view of the pointed region onto the mobile touchscreen and make selections thereafter. The experimental results indicate that PS outperforms DS in terms of speed and user satisfaction with physical effort involved especially in complex tasks requiring multiple widget selection. DS was found to be favorable for simple tasks as it has lower error rate and it does not require attention switch between the mobile and the large display.

Interactive displays are increasingly distributed in a broad spectrum of everyday life environments: They have very diverse form factors and portability characteristics, support a variety of interaction techniques, and can be used by a variable number of people. The coupling of multiple displays can thus create interactive "ecosystems" which mingle in the social context, and generate novel settings of communication, performance and ownership. The objective of this workshop is to focus on the range of research challenges and opportunities afforded by applications that rely on visual interfaces that can spread across multiple displays. Such displays are physically decoupled (i.e. connected to multiple computers) yet are visually coupled due to the interfaces and interactions they support. This can range from visual interfaces spread across multiple small private input displays (e.g. information exchange or game play) to small private displays coupled with larger public displays (e.g. public photo sharing).

Large interactive displays are an effective means to exchange contents with mobile devices for co-located collaboration in offices and schools. It is very important that the users are able to easily comprehend and learn the interaction techniques to pair their mobile devices with large displays. In this paper, we report on the results of an exploratory case study investigating the comprehension and understandability of the display labels for four pairing techniques i.e. pointing, touching, drawing and typing.

Software tools are used to compare multiple versions of a textual document to help a reader understand the evolution of that document over time. These tools generally support the comparison of only two versions of a document, requiring multiple comparisons to be made to derive a full history of the document across multiple versions. We present Deep Diffs, a novel visualisation technique that exposes the multiple layers of history of a document at once, directly in the text, highlighting areas that have changed over multiple successive versions, and drawing attention to passages that are new, potentially unpolished or contentious. These composite views facilitate the writing and editing process by assisting memory and encouraging the analysis of collaboratively-authored documents. We describe how this technique effectively supports common text editing tasks and heightens participants' understanding of the process in collaborative editing scenarios like wiki editing and paper writing.

Visualisations of dynamic networks are animated over time, reflecting changes in the underlying data structure. As viewers of these visualisations, it is up to us to accurately perceive and keep up with the constantly shifting view, mentally noting as visual elements are added, removed, changed and rearranged, sometimes at great pace. In a complex data set with a lot happening, this can put a strain on the observer's perceptions, with changes in layout and visual population disrupting their internalised mental model of the visualisation, making it difficult to understand what the changes represent. We present Showtime, a novel visualisation technique which dilates the flow of time so that observers have proportionally more time to understand each change based on the density of activity in the visualisation. This is paired with a novel timeline element which tracks the flow of time visually.

Hundreds of technical, special interest Internet weblogs are already generating thousands of niche articles worldwide, and many institutions are starting to create internal blogs for team collaboration. As this style of communication becomes more pervasive in the lives of employees and researchers, the difficulty of finding relevant information only grows with the number of authors and articles. To reduce the load, we propose using implicit group messaging (IGM) to automatically deliver relevant content to readers grouped by shared characteristics or interests. In this paper, we outline a context-aware application suited to special interest messaging and describe three alternative delivery models including our peer-to-peer (P2P) design called SPICE and a broker-based design. We investigate the advantages and disadvantages of each approach through detailed simulations driven by realistic data and actual national/global network topologies. We find that although a broker-based design is generally the most network efficient and lowest latency, a structured P2P system can offer exceptionally low and fair loading across peers and network links without relying on specialized broker nodes.

When designing interfaces and visualizations how does a human or automatic visual interface designer know how easy or hard it will be for viewers to see the interface? In this paper we present a perceptual usability measure of how easy or hard visual designs are to see when viewed over different distances. The measure predicts the relative perceivability of sub-parts of a visual design by using simulations of human visual acuity coupled with an information theoretic measure. We present results of the perceptual measure predicting the perceivability of optometrists eye charts, a webpage and a small network graph.

This paper investigates the use of a multi-representation of a same graph in order to interactively cluster it. Both a node-link and a matrix representation of the graph are provided. The clustering technique under consideration requires parameters to be set. The matrix representation of the graph is provided in order to visually decide on the value of those parameters. The approach has been implemented and is illustrated through an example.

BACKGROUND: Various measures of semantic similarity of terms in bio-ontologies such as the Gene Ontology (GO) have been used to compare gene products. Such measures of similarity have been used to annotate uncharacterized gene products and group gene products into functional groups. There are various ways to measure semantic similarity, either using the topological structure of the ontology, the instances (gene products) associated with terms or a mixture of both. We focus on an instance level definition of semantic similarity while using the information contained in the ontology, both in the graphical structure of the ontology and the semantics of relations between terms, to provide constraints on our instance level description.Semantic similarity of terms is extended to annotations by various approaches, either though aggregation operations such as min, max and average or through an extrapolative method. These approaches introduce assumptions about how semantic similarity of terms relates to the semantic similarity of annotations that do not necessarily reflect how terms relate to each other. RESULTS: We exploit the semantics of relations in the GO to construct an algorithm called SSA that provides the basis of a framework that naturally extends instance based methods of semantic similarity of terms, such as Resnik's measure, to describing annotations and not just terms. Our measure attempts to correctly interpret how terms combine via their relationships in the ontological hierarchy. SSA uses these relationships to identify the most specific common ancestors between terms. We outline the set of cases in which terms can combine and associate partial order constraints with each case that order the specificity of terms. These cases form the basis for the SSA algorithm. The set of associated constraints also provide a set of principles that any improvement on our method should seek to satisfy. CONCLUSION: We derive a measure of semantic similarity between annotations that exploits all available information without introducing assumptions about the nature of the ontology or data. We preserve the principles underlying instance based methods of semantic similarity of terms at the annotation level. As a result our measure better describes the information contained in annotations associated with gene products and as a result is better suited to characterizing and classifying gene products through their annotations.

Control systems must increasingly be designed to involve collections of hardware and software components, both of which may evolve over the lifetime of the system, and which are expected to provide self-managing, adaptive, autonomic behaviour. Understanding the behaviour such a system will exhibit under any specific conditions is a significant design challenge. We present a model derived from approaches to modelling dynamical systems in which the adaptive behaviour of an autonomic system may be described and analysed as a whole. We explain our ideas with reference to a hybrid hardware/software system, and argue that it generalises to other classes of autonomic systems.

We have recently begun to see hardware support for the tabletop user interface, offering a number of new ways for humans to interact with computers. Tabletops offer great potential for face-to-face social interaction; advances in touch technology and computer graphics provide natural ways to directly manipulate virtual objects, which we can display on the tabletop surface. Such an interface has the potential to benefit a wide range of the population and it is important that we design for usability and learnability with diverse groups of people.This paper describes the design of SharePic -- a multiuser, multi-touch, gestural, collaborative digital photograph sharing application for a tabletop -- and our evaluation with both young adult and elderly user groups. We describe the guidelines we have developed for the design of tabletop interfaces for a range of adult users, including elders, and the user interface we have built based on them. Novel aspects of the interface include a design strongly influenced by the metaphor of physical photographs placed on the table with interaction techniques designed to be easy to learn and easy to remember. In our evaluation, we gave users the final task of creating a digital postcard from a collage of photographs and performed a realistic think-aloud with pairs of novice participants learning together, from a tutorial script.

We propose a decoupled distribution paradigm for connecting large numbers of content publishers and consumers called implicit group messaging (IGM). Unlike traditional multicast or publish/subscribe messaging, IGM delivers content to "implicit groups" of consumers matching combinations of attributes specified by the publisher at the time of publication. The contributions of this paper include a characterisation of IGM encompassing the types of groups that must be supported, and an innovative distributed P2P implementation that we compare via simulation to a functionally equivalent client/server approach. Our results show the P2P model to efficiently support a range of implicit groups with acceptable delivery times and vastly less maximum peer and link stress than a client/server approach