This study aims to utilise mid-air hand-pose movements to implement various interactive controls, e.g. dial and slider controlling, through independent low-dimensional embeddings. Towards this, we develop a novel adjustable hand-pose space disentanglement approach for a learnable VAE-based high-to-low dimensional embedding model (HandSolo). It disentangles the latent embeddings intomultiple independent one- or two-dimensional embedding spaces, enabling independent control. HandSolo allows multi-dimensional settings and multi-DOF combinations, providing a new paradigm for flexible and extensible hand-pose interaction systems. Additionally, to exploit model potential and make user interaction comfortable, we propose a visual interaction evaluation strategy (VIEs) to help system designers understand model capability and user habits. Finally, we provide an example virtual interaction system that integrates various virtual interaction objects, showing how our innovations improve their interaction capabilities. Experimental user studies demonstrate the effectiveness of our embedding-disentanglement designs, including discovery experiment (n=4) for VIEs, inspiration experiment (n=4) for approach extensibility, and exploration experiment (n=8) for the virtual interaction system.

Active Inference is a closed-loop computational theoretical basis for understanding behaviour, based on agents with internal probabilistic generative models that encode their beliefs about how hidden states in their environment cause their sensations. We review Active Inference and how it could be applied to model the human–computer interaction loop. Active Inference provides a coherent framework for managing generative models of humans, their environments, sensors and interface components. It informs off-line design and supports real-time, online adaptation. It provides model-based explanations for behaviours observed in HCI, and new conceptual tools with the potential to measure important concepts such as agency and engagement in interaction. We discuss how Active Inference offers a new basis for a theory of interaction in HCI, tools for design of modern, complex sensor-based systems, and integration of artificial intelligence technologies, enabling it to cope with diversity in human users and contexts. We discuss the practical challenges in implementing such Active Inference-based systems.

We present a novel Hand-pose Embedding Interactive System (HpEIS) as a virtual sensor, which maps users’ flexible hand poses to a two-dimensional visual space using a Variational Autoencoder (VAE) trained on a variety of hand poses. HpEIS enables visually interpretable and guidable support for user explorations in multimedia collections, using only a camera as an external hand pose acquisition device. We identify general usability issues associated with system stability and smoothing requirements through pilot experiments with expert and inexperienced users. We then design stability and smoothing improvements, including hand-pose data augmentation, an anti-jitter regularisation term added to loss function, stabilising post-processing for movement turning points and smoothing post-processing based on One Euro Filters. In target selection experiments (n=12), we evaluate HpEIS by measures of task completion time and the final distance to target points, with and without the gesture guidance window condition. Experimental responses indicate that HpEIS provides users with a learnable, flexible, stable and smooth mid-air hand movement interaction experience.

We introduce the first continuous-time score-based generative model that leverages fractional diffusion processes for its underlying dynamics. Although diffusion models have excelled at capturing data distributions, they still suffer from various limitations such as slow convergence, mode-collapse on imbalanced data, and lack of diversity. These issues are partially linked to the use of light-tailed Brownian motion (BM) with independent increments. In this paper, we replace BM with an approximation of its non-Markovian counterpart, fractional Brownian motion (fBM), characterized by correlated increments and Hurst index $H \in (0,1)$, where $H=0.5$ recovers the classical BM. To ensure tractable inference and learning, we employ a recently popularized Markov approximation of fBM (MA-fBM) and derive its reverse-time model, resulting in *generative fractional diffusion models* (GFDM). We characterize the forward dynamics using a continuous reparameterization trick and propose *augmented score matching* to efficiently learn the score function, which is partly known in closed form, at minimal added cost. The ability to drive our diffusion model via MA-fBM offers flexibility and control. $H \leq 0.5$ enters the regime of *rough paths* whereas $H>0.5$ regularizes diffusion paths and invokes long-term memory. The Markov approximation allows added control by varying the number of Markov processes linearly combined to approximate fBM. Our evaluations on real image datasets demonstrate that GFDM achieves greater pixel-wise diversity and enhanced image quality, as indicated by a lower FID, offering a promising alternative to traditional diffusion models

In this work, we introduce Pixelsmith, a zero-shot text-to-image generative framework to sample images at higher resolutions with a single GPU. We are the first to show that it is possible to scale the output of a pre-trained diffusion model by a factor of 1000, opening the road to gigapixel image generation at no extra cost. Our cascading method uses the image generated at the lowest resolution as baseline to sample at higher resolutions. For the guidance, we introduce the Slider, a mechanism that fuses the overall structure contained in the first-generated image with enhanced fine details. At each inference step, we denoise patches rather than the entire latent space, minimizing memory demands so that a single GPU can handle the process, regardless of the image's resolution. Our experimental results show that this method not only achieves higher quality and diversity compared to existing techniques but also reduces sampling time and ablation artifacts.

Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce sim2vr, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that sim2vr can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.

Abstract We study the problem of predicting the next query to be recommended in interactive data exploratory analysis to guide users to correct content. Current query prediction approaches are based on sequence-to-sequence learning, exploiting past interaction data. However, due to the resource-hungry training process, such approaches fail to adapt to immediate user feedback. Immediate feedback is essential and considered as a signal of the user’s intent. We contribute with a novel query prediction ensemble mechanism, which adapts to immediate feedback relying on multi-armed bandits framework. Our mechanism, an extension to the popular Exp3 algorithm, augments Transformer-based language models for query predictions by combining predictions from experts, thus dynamically building a candidate set during exploration. Immediate feedback is leveraged to choose the appropriate prediction in a probabilistic fashion. We provide comprehensive large-scale experimental and comparative assessment using a popular online literature discovery service, which showcases that our mechanism (i) improves the per-round regret substantially against state-of-the-art Transformer-based models and (ii) shows the superiority of causal language modelling over masked language modelling for query recommendations.

For medical image semantic segmentation (MISS), Vision Transformers have emerged as strong alternatives to convolutional neural networks thanks to their inherent ability to capture long-range correlations. However, existing research uses off-the-shelf vision Transformer blocks based on linear projections and feature processing which lack spatial and local context to refine organ boundaries. Furthermore, Transformers do not generalize well on small medical imaging datasets and rely on large-scale pre-training due to limited inductive biases. To address these problems, we demonstrate the design of a compact and accurate Transformer network for MISS, CS-Unet, which introduces convolutions in a multi-stage design for hierarchically enhancing spatial and local modeling ability of Transformers. This is mainly achieved by our well-designed Convolutional Swin Transformer (CST) block which merges convolutions with Multi-Head Self-Attention and Feed-Forward Networks for providing inherent localized spatial context and inductive biases. Experiments demonstrate CS-Unet without pre-training out- performs other counterparts by large margins on multi-organ and cardiac datasets with fewer parameters and achieves state-of-the-art performance. Our code is available at Github1.

For widespread adoption, public security and surveillance systems must be accurate, portable, compact, and real-time, without impeding the privacy of the individuals being observed. Current systems broadly fall into two categories – image-based which are accurate, but lack privacy, and RF signal-based, which preserve privacy but lack portability, compactness and accuracy. Our paper proposes mmSense, an end-to-end portable miniaturised real-time system that can accurately detect the presence of concealed metallic objects on persons in a discrete, privacy-preserving modality. mm-Sense features millimeter wave radar technology, provided by Google’s Soli sensor for its data acquisition, and TransDope, our real-time neural network, capable of processing a single radar data frame in 19 ms. mmSense achieves high recognition rates on a diverse set of challenging scenes while running on standard laptop hardware, demonstrating a significant advancement towards creating portable, cost-effective real-time radar based surveillance systems.

This paper presents a new continuous interaction strategy with visual feedback of hand pose and mid-air gesture recognition and control for a smart music speaker, which utilizes only 2 video frames to recognize gestures. Frame-based hand pose features from MediaPipe Hands, containing 21 landmarks, are embedded into a 2 dimensional pose space by an autoencoder. The corresponding space for interaction with the music content is created by embedding high-dimensional music track profiles to a compatible two-dimensional embedding. A PointNet-based model is then applied to classify gestures which are used to control the device interaction or explore music spaces. By jointly optimising the autoencoder with the classifier, we manage to learn a more useful embedding space for discriminating gestures. We demonstrate the functionality of the system with experienced users selecting different musical moods by varying their hand pose.

We present DiffInfinite, a hierarchical diffusion model that generates arbitrarily large histological images while preserving long-range correlation structural information. Our approach first generates synthetic segmentation masks, subsequently used as conditions for the high-fidelity generative diffusion process. The proposed sampling method can be scaled up to any desired image size while only requiring small patches for fast training. Moreover, it can be parallelized more efficiently than previous large-content generation methods while avoiding tiling artifacts. The training leverages classifier-free guidance to augment a small, sparsely annotated dataset with unlabelled data. Our method alleviates unique challenges in histopathological imaging practice: large-scale information, costly manual annotation, and protective data handling. The biological plausibility of DiffInfinite data is evaluated in a survey by ten experienced pathologists as well as a downstream classification and segmentation task. Samples from the model score strongly on anti-copying metrics which is relevant for the protection of patient data.

We propose a novel transformer, capable of segmenting medical images of varying modalities. Challenges posed by the fine-grained nature of medical image analysis mean that the adaptation of the transformer for their analysis is still at nascent stages. The overwhelming success of the UNet lay in its ability to appreciate the fine-grained nature of the segmentation task, an ability which existing transformer based models do not currently posses. To address this shortcoming, we propose The Fully Convolutional Transformer (FCT), which builds on the proven ability of Convolutional Neural Networks to learn effective image representations, and combines them with the ability of Transformers to effectively capture long-term dependencies in its inputs. The FCT is the first fully convolutional Transformer model in medical imaging literature. It processes its input in two stages, where first, it learns to extract long range semantic dependencies from the input image, and then learns to capture hierarchical global attributes from the features. FCT is compact, accurate and robust. Our results show that it outperforms all existing transformer architectures by large margins across multiple medical image segmentation datasets of varying data modalities without the need for any pre-training. FCT outperforms its immediate competitor on the ACDC dataset by 1.3%, on the Synapse dataset by 4.4%, on the Spleen dataset by 1.2% and on ISIC 2017 dataset by 1.1% on the dice metric, with up to five times fewer parameters. On the ACDC Post-2017-MICCAI-Challenge online test set, our model sets a new state-of-the-art on unseen MRI test cases out-performing large ensemble models as well as nnUNet with considerably fewer parameters. Our code, environments and models will be available via GitHub†.

Leakage of data from publicly available Machine Learning (ML) models is an area of growing significance since commercial and government applications of ML can draw on multiple sources of data, potentially including users' and clients' sensitive data. We provide a comprehensive survey of contemporary advances on several fronts, covering involuntary data leakage which is natural to ML models, potential malicious leakage which is caused by privacy attacks, and currently available defence mechanisms. We focus on inference-time leakage, as the most likely scenario for publicly available models. We first discuss what leakage is in the context of different data, tasks, and model architectures. We then propose a taxonomy across involuntary and malicious leakage, followed by description of currently available defences, assessment metrics, and applications. We conclude with outstanding challenges and open questions, outlining some promising directions for future research.

We develop a new type of model for solving the task of inverting the transmission effects of multi-mode optical fibres through the construction of an $\mathrm{SO}^{+}(2,1)$-equivariant neural network. This model takes advantage of the of the azimuthal correlations known to exist in fibre speckle patterns and naturally accounts for the difference in spatial arrangement between input and speckle patterns. In addition, we use a second post-processing network to remove circular artifacts, fill gaps, and sharpen the images, which is required due to the nature of optical fibre transmission. This two stage approach allows for the inspection of the predicted images produced by the more robust physically motivated equivariant model, which could be useful in a safety-critical application, or by the output of both models, which produces high quality images. Further, this model can scale to previously unachievable resolutions of imaging with multi-mode optical fibres and is demonstrated on $256 \times 256$ pixel images. This is a result of improving the trainable parameter requirement from $\mathcal{O}(N^4)$ to $\mathcal{O}(m)$, where $N$ is pixel size and $m$ is number of fibre modes. Finally, this model generalises to new images, outside of the set of training data classes, better than previous models.

Forward biomechanical simulation in HCI holds great promise as a tool for evaluation, design, and engineering of user interfaces. Although reinforcement learning (RL) has been used to simulate biomechanics in interaction, prior work has relied on unrealistic assumptions about the control problem involved, which limits the plausibility of emerging policies. These assumptions include direct torque actuation as opposed to muscle-based control; direct, privileged access to the external environment, instead of imperfect sensory observations; and lack of interaction with physical input devices. In this paper, we present a new approach for learning muscle-actuated control policies based on perceptual feedback in interaction tasks with physical input devices. This allows modelling of more realistic interaction tasks with cognitively plausible visuomotor control. We show that our simulated user model successfully learns a variety of tasks representing different interaction methods, and that the model exhibits characteristic movement regularities observed in studies of pointing. We provide an open-source implementation which can be extended with further biomechanical models, perception models, and interactive environments.

We consider the query recommendation problem in closed loop interactive learning settings like online information gathering and exploratory analytics. The problem can be naturally modelled using the Multi-Armed Bandits (MAB) framework with countably many arms. The standard MAB algorithms for countably many arms begin with selecting a random set of candidate arms and then applying standard MAB algorithms, e.g., UCB, on this candidate set downstream. We show that such a selection strategy often results in higher cumulative regret and to this end, we propose a selection strategy based on the maximum utility of the arms. We show that in tasks like online information gathering, where sequential query recommendations are employed, the sequences of queries are correlated and the number of potentially optimal queries can be reduced to a manageable size by selecting queries with maximum utility with respect to the currently executing query. Our experimental results using a recent real online literature discovery service log file demonstrate that the proposed arm selection strategy improves the cumulative regret substantially with respect to the state-of-the-art baseline algorithms. Our data model and source code are available at \url{https://anonymous.4open.science/r/0e5ad6b7-ac02-4577-9212-c9d505d3dbdb/}

COVID-19 abatement strategies have risks and uncertainties which could lead to repeating waves of infection. We show-as proof of concept grounded on rigorous mathematical evidence-that periodic, high-frequency alternation of into, and out-of, lockdown effectively mitigates second-wave effects, while allowing continued, albeit reduced, economic activity. Periodicity confers (i) predictability, which is essential for economic sustainability, and (ii) robustness, since lockdown periods are not activated by uncertain measurements over short time scales. In turn-while not eliminating the virus-this fast switching policy is sustainable over time, and it mitigates the infection until a vaccine or treatment becomes available, while alleviating the social costs associated with long lockdowns. Typically, the policy might be in the form of 1-day of work followed by 6-days of lockdown every week (or perhaps 2 days working, 5 days off) and it can be modified at a slow-rate based on measurements filtered over longer time scales. Our results highlight the potential efficacy of high frequency switching interventions in post lockdown mitigation. All code is available on Github at https://github.com/V4p1d/FPSP_Covid19. A software tool has also been developed so that interested parties can explore the proof-of-concept system.

We present Intermittent Control (IC) models as a candidate framework for modelling human input movements in Human–Computer Interaction (HCI). IC differs from continuous control in that users are not assumed to use feedback to adjust their movements continuously, but only when the difference between the observed pointer position and predicted pointer positions becomes large. We use a parameter optimisation approach to identify the parameters of an intermittent controller from experimental data, where users performed one-dimensional mouse movements in a reciprocal pointing task. Compared to previous published work with continuous control models, based on the Kullback–Leibler divergence from the experimental observations, IC is better able to generatively reproduce the distinctive dynamical features and variability of the pointing task across participants and over repeated tasks. IC is compatible with current physiological and psychological theory and provides insight into the source of variability in HCI tasks.

Machine learning methods for computational imaging require uncertainty estimation to be reliable in real settings. While Bayesian models offer a computationally tractable way of recovering uncertainty, they need large data volumes to be trained, which in imaging applications implicates prohibitively expensive collections with specific imaging instruments. This paper introduces a novel framework to train variational inference for inverse problems exploiting in combination few experimentally collected data, domain expertise and existing image data sets. In such a way, Bayesian machine learning models can solve imaging inverse problems with minimal data collection efforts. Extensive simulated experiments show the advantages of the proposed framework. The approach is then applied to two real experimental optics settings: holographic image reconstruction and imaging through highly scattering media. In both settings, state of the art reconstructions are achieved with little collection of training data.

We use complex-weighted, deep networks to invert the effects of multimode optical fibre distortion of a coherent input image. We generated experimental data based on collections of optical fibre responses to greyscale input images generated with coherent light, by measuring only image amplitude (not amplitude and phase as is typical) at the output of \SI{1}{\metre} and \SI{10}{\metre} long, \SI{105}{\micro\metre} diameter multimode fibre. This data is made available as the {\it Optical fibre inverse problem} Benchmark collection. The experimental data is used to train complex-weighted models with a range of regularisation approaches. A {\it unitary regularisation} approach for complex-weighted networks is proposed which performs well in robustly inverting the fibre transmission matrix, which fits well with the physical theory. A key benefit of the unitary constraint is that it allows us to learn a forward unitary model and analytically invert it to solve the inverse problem. We demonstrate this approach, and show how it can improve performance by incorporating knowledge of the phase shift induced by the spatial light modulator.

The association of tabletop interaction with gesture typing presents interaction potential for situationally or physically impaired users. In this work, we use depth cameras to create touch surfaces on regular tabletops. We describe our prototype system and report on a supervised learning approach to fingertips touch classification. We follow with a gesture typing study that compares our system with a control tablet scenario and explore the influence of input size and aspect ratio of the virtual surface on the text input performance. We show that novice users perform with the same error rate at half the input rate with our system as compared to the control condition, that an input size between A5 and A4 present the best tradeoff between performance and user preference and that users' indirect tracking ability seems to be the overall performance limiting factor.

This article presents an empirical comparison of four models from manual control theory on their ability to model targeting behaviour by human users using a mouse: McRuer’s Crossover, Costello’s Surge, second-order lag (2OL), and the Bang-bang model. Such dynamic models are generative, estimating not only movement time, but also pointer position, velocity, and acceleration on a moment-to-moment basis. We describe an experimental framework for acquiring pointing actions and automatically fitting the parameters of mathematical models to the empirical data. We present the use of time-series, phase space, and Hooke plot visualisations of the experimental data, to gain insight into human pointing dynamics. We find that the identified control models can generate a range of dynamic behaviours that captures aspects of human pointing behaviour to varying degrees. Conditions with a low index of difficulty (ID) showed poorer fit because their unconstrained nature leads naturally to more behavioural variability. We report on characteristics of human surge behaviour (the initial, ballistic sub-movement) in pointing, as well as differences in a number of controller performance measures, including overshoot, settling time, peak time, and rise time . We describe trade-offs among the models. We conclude that control theory offers a promising complement to Fitts’ law based approaches in HCI, with models providing representations and predictions of human pointing dynamics, which can improve our understanding of pointing and inform design.

In everyday life, people use their mobile phones on-the-go with different walking speeds and with different touch input techniques. Unfortunately, much of the published research in mobile interaction does not quantify the influence of these variables. In this article, we analyze the influence of walking speed, gait pattern, and input techniques on commonly used performance parameters like error rate, accuracy, and tapping speed, and we compare the results to the static condition. We examine the influence of these factors on the machine learned offset model used to correct user input, and we make design recommendations. The results show that all performance parameters degraded when the subject started to move, for all input techniques. Index finger pointing techniques demonstrated overall better performance compared to thumb-pointing techniques. The influence of gait phase on tap event likelihood and accuracy was demonstrated for all input techniques and all walking speeds. Finally, it was shown that the offset model built on static data did not perform as well as models inferred from dynamic data, which indicates the speed-specific nature of the models. Also, models identified using specific input techniques did not perform well when tested in other conditions, demonstrating the limited validity of offset models to a particular input technique. The model was therefore calibrated using data recorded with the appropriate input technique, at 75% of preferred walking speed, which is the speed to which users spontaneously slow down when they use a mobile device and which presents a trade-off between accuracy and usability. This led to an increase in accuracy compared to models built on static data. The error rate was reduced between 0.05% and 5.3% for landscape-based methods and between 5.3% and 11.9% for portrait-based methods.

We identify usability challenges facing consumers adopting Virtual Reality (VR) head-mounted displays (HMDs) in a survey of 108 VR HMD users. Users reported significant issues in interacting with, and being aware of their real-world context when using a HMD. Building upon existing work on blending real and virtual environments, we performed three design studies to address these usability concerns. In a typing study, we show that augmenting VR with a view of reality significantly corrected the performance impairment of typing in VR. We then investigated how much reality should be incorporated and when, so as to preserve users' sense of presence in VR. For interaction with objects and peripherals, we found that selectively presenting reality as users engaged with it was optimal in terms of performance and users' sense of presence. Finally, we investigated how this selective, engagement-dependent approach could be applied in social environments, to support the user's awareness of the proximity and presence of others.

This paper contributes novel measures of user engagement in mobile music retrieval, linking these to work in music psychology, and illustrating resulting design guidelines in a demonstrator system. The large music collections available to users today can be overwhelming in mobile settings, they offer 'too-much-choice' to users, who often resort to shuffle-based playback. Work in music psychology has introduced the concept of music engagement -- listeners vary in their desired control over their music listening, and engagement varies with listening context. We develop a series of metrics to capture music listening behaviour from users' interaction logs. In a survey of 94 music listeners, we show significant correlations between music engagement from questionnaires and the presented quantitative metrics. We show how music retrieval can adapt to this engagement, developing a tablet-based demonstrator system, with an exploratory evaluation.

We demonstrate that front-of-screen targeting on mobile phones can be predicted from back-of-device grip manipulations. Using simple, low-resolution capacitive touch sensors placed around a standard phone, we outline a machine learning approach to modelling the grip modulation and inferring front-of-screen touch targets. We experimentally demonstrate that grip is a remarkably good predictor of touch, and we can predict touch position 200ms before contact with an accuracy of 18mm.

We describe the focused-casual continuum, a framework for describing interaction techniques according to the degree to which they allow users to adapt how much attention and effort they choose to invest in an interaction conditioned on their current situation. Casual interactions are particularly appropriate in scenarios where full engagement with devices is frowned upon socially, is unsafe, physically challenging or too mentally taxing. Novel sensing approaches which go beyond direct touch enable wider use of casual interactions, which will often be 'around device' interactions. We consider the degree to which previous commercial products and research prototypes can be considered as fitting the focused-casual framework, and describe the properties using control theoretic concepts. In an experimental study we observe that users naturally apply more precise and more highly engaged interaction techniques when faced with a more challenging task and use more relaxed gestures in easier tasks.

A novel interaction style is presented, allowing in-pocket music selection by tapping a song's rhythm on a device's touchscreen or body. We introduce the use of rhythmic queries for music retrieval, employing a trained generative model to improve query recognition. We identify rhythm as a fundamental feature of music which can be reproduced easily by listeners, making it an effective and simple interaction technique for retrieving music. We observe that users vary in which instruments they entrain with and our work is the first to model such variability. An experiment was performed, showing that after training the generative model, retrieval performance improved two-fold. All rhythmic queries returned a highly ranked result with the trained generative model, compared with 47% using existing methods. We conclude that generative models of subjective user queries can yield significant performance gains for music retrieval and enable novel interaction techniques such as rhythmic filtering.

We present a machine learning approach to train user-specific offset models, which map actual to intended touch locations to improve accuracy. We propose a flexible framework to adapt and apply models trained on touch data from one device and user to others. This paper presents a study of the first published experimental data from multiple devices per user, and indicates that models not only improve accuracy between repeated sessions for the same user, but across devices and users, too. Device-specific models outperform unadapted user-specific models from different devices. However, with both user- and device-specific data, we demonstrate that our approach allows to combine this information to adapt models to the targeted device resulting in significant improvement. On average, adapted models improved accuracy by over 8%. We show that models can be obtained from a small number of touches (≈60). We also apply models to predict input-styles and identify users.

We consider the reasons why we organise books in a physical environment and investigate whether situating interactions with a smartphone could improve the user experience of e-readers. Our prototype uses the Kinect depth sensor to detect the position of a user in relation to sections of a physical bookshelf. We also built a mobile application that allows users to browse and organise digital books by moving between each section. We present our initial observations of a user study that evaluated search and categorisation tasks with our prototype. Our findings motivate reasons to explore digital books in a physical environment and indicate issues to consider when designing situated interactions with e-readers.

We present a novel approach for quantifying the impact of uncertainty in manual control, based on information and control theories and utilizing the information-theoretic capacity of empowerment, a task-independent universal utility measure. Empowerment measures, for agent-environment systems with stochastic transitions, how much influence, which can be sensed by the agent sensors, an agent has on its environment. It enables combining different types of disturbances, arising in human-machine systems (i.e. noise, delays, errors, etc.), into one single measure. We expand empowerment to manual control, demonstrate its application in the field of HCI and evaluate it in a user study. Results showed that empowerment is strictly monotonic in relation to the means of standard performance metrics total time off-target, perceived uncertainty, perceived performance and frustration, which suggests its potential in making theoretical predictions of other measures. Loss of empowerment implicated interesting trends in activity levels, which open a new area for future work. Results suggest the potential empowerment has in providing better theoretical foundations for the science of HCI.

This paper presents a systematic and general technique for establishing a set of motions suitable for use with sensor systems, by drawing performable and measurable motions directly from users. It uses reinforcement which rewards originality to induce users to explore the space of motions they can perform. A decomposition of movements into motion primitives is constructed, among which a meaningful originality metric can be defined. Because the originality measure is defined in terms of the sensed input, the resulting space contains only movements which can both be performed and sensed. We show how this can be used to evaluate the relative performance of different joint user-sensor systems, providing objective analyses of gesture lexicons with regard to the technical limitations of sensors and humans. In particular, we show how the space of motions varies across the arm for a body-mounted inertial sensor.

A tablet interface for manipulating microscopic particles is augmented with vibrotactile and audio feedback. The feedback is generated using a novel real-time synthesis library based on approximations to physical processes, and is efficient enough to run on mobile devices, despite their limited computational power. The feedback design and usability testing was done with a realistic simulator on appropriate tasks, allowing users to control objects more rapidly, with fewer errors and applying more consistent forces. The feedback makes the interaction more tangible, giving the user more awareness of changes in the characteristics of the optical tweezers as the number of optical traps changes.

We present a flexible Machine Learning approach for learning user-specific touch input models to increase touch accuracy on mobile devices. The model is based on flexible, non-parametric Gaussian Process regression and is learned using recorded touch inputs. We demonstrate that significant touch accuracy improvements can be obtained when either raw sensor data is used as an input or when the device's reported touch location is used as an input, with the latter marginally outperforming the former. We show that learned offset functions are highly nonlinear and user-specific and that user-specific models outperform models trained on data pooled from several users. Crucially, significant performance improvements can be obtained with a small (≈200) number of training examples, easily obtained for a particular user through a calibration game or from keyboard entry data.

We describe an approach to improving the design and development of Brain-Computer Interface (BCI) applications by simulating the error-prone characteristics and subjective feel of electroencephalogram (EEG), motor-imagery based BCIs. BCIs have the potential to enhance the quality of life of people who are severely disabled, but it is often time-consuming to test and develop the systems. Simulation of BCI characteristics allows developers to rapidly test design options, and gain both subjective and quantitative insight into expected behaviour without using an EEG cap. A further motivation for the use of simulation is that 'impairing' a person without motor disabilities in a game with a disabled BCI user can create a level playing field and help carers empathise with BCI users. We demonstrate a use of the simulator in controlling a game of Brain Pong.

We present a finger-tracking system for touch-based interaction which can track 3D finger angle in addition to position, using low-resolution conventional capacitive sensors, therefore compensating for the inaccuracy due to pose variation in conventional touch systems. Probabilistic inference about the pose of the finger is carried out in real-time using a particle filter; this results in an efficient and robust pose estimator which also gives appropriate uncertainty estimates. We show empirically that tracking the full pose of the finger results in greater accuracy in pointing tasks with small targets than competitive techniques. Our model can detect and cope with different finger sizes and the use of either fingers or thumbs, bringing a significant potential for improvement in one-handed interaction with touch devices. In addition to the gain in accuracy we also give examples of how this technique could open up the space of novel interactions.

"In all the wonderful worlds that writing opens, the spoken word still resides and lives. Written texts all have to be related somehow, directly or indirectly, to the world of sound, the natural habitat of language, to yield their meanings."

The Microsoft Kinect sensor can be combined with a modern mobile phone to rapidly create digitally augmented environments. This can be used either directly as a form of ubiquitous computing environment or indirectly as framework for rapidly prototyping ubicomp environments that are otherwise implemented using conventional sensors. We describe an Android mobile application that supports rapid prototyping of spacial interaction by using 3D position data from the Kinect to simulate a proximity sensor. This allows a developer, or end user, to easily associate content or services on the device with surfaces or regions of a room. The accuracy of the hotspot marking was tested in an experiment where users selected points marked on a whiteboard using a mobile phone. The distribution of the sample points were analysed and showed that the bulk of the selections were within about 13cm of the target and the distributions were characteristically skewed depending on whether the user came to the target from the left or right. This range is sufficient for prototyping many common ubicomp scenarios based on proximity in a room. To illustrate this approach, we describe the design of a novel mobile application that associates a virtual book library with a region of a room, integrating the additional sensors and actuators of a smartphone with the position sensing of the Kinect. We highlight limitations of this approach and suggest areas for future work.

We describe a novel dynamic method for collaborative virtual environments designed for mobile devices and evaluated in a mobile context. Participants interacted in pairs remotely and through touch while walking in three different feedback conditions: 1) visual, 2) audio-tactile, 3) spatial audio-tactile. Results showed the visual baseline system provided higher shared awareness, efficiency and a strong learning effect. However, and although very challenging, the eyes-free systems still offered the ability to build joint awareness in remote collaborative environments, particularly the spatial audio one. These results help us better understand the potential of different feedback mechanisms in the design of future mobile collaborative environments.

We describe a particle filtering approach to inferring finger movements on capacitive sensing arrays. This technique allows the efficient combination of human movement models with accurate sensing models, and gives high-fidelity results with low-resolution sensor grids and tracks finger height. Our model provides uncertainty estimates, which can be linked to the interaction to provide appropriately smoothed responses as sensing perfomance degrades; system autonomy is increased as estimates of user behaviour become less certain. We demonstrate the particle filter approach with a map browser running with a very small sensor board, where finger position uncertainty is linked to autonomy handover.

We describe a virtual "tether" for mobile devices that allows groups to have quick, simple and privacy-preserving meetups. Our design provides cues which allow dynamic coordination of rendezvous without revealing users' positions. Using accelerometers and magnetometers, combined with GPS positioning and non-visual feedback, users can probe and sense a dynamic virtual object representing the nearest meeting point. The Social Gravity system makes social bonds tangible in a virtual world which is geographically grounded, using haptic feedback to help users rendezvous. We show dynamic navigation using this physical model-based system to be efficient and robust in significant field trials, even in the presence of low-quality positioning. The use of simulators to build models of mobile geolocated systems for pre-validation purposes is discussed, and results compared with those from our trials. Our results show interesting behaviours in the social coordination task, which lead to guidelines for geosocial interaction design. The Social Gravity system proved to be very successful in allowing groups to rendezvous efficiently and simply and can be implemented using only commercially available hardware.

The paper demonstrates the feasibility of using mobile phones for fitness and rehabilitation purposes by training them to recognise a user's hula-hooping movements. It also proposes several parameters which can be used as a measure of rhythmic movement quality. Experimental measurements were achieved with two test subjects performing two sets of steady hula-hooping. The paper compares algorithm performance with accelerometer, gyroscope and magnetometer sensor readings. Analysis of the recorded data indicated that magnetometers had some advantages over accelerometers for reliable phase extraction. Hilbert transforms were used to extract the phase information, and a Dynamic Rhythmic Primitive Model was identified for the hula-hooping movement. Together these tools allow the creation of hula-hooping performance metrics which can be used in wellness, rehabilitation or entertainment applications for mobile devices. We outline open technical challenges and possible future research directions.

In this article we describe a novel approach to pedestrian navigation using bearing-based haptic feedback. People are guided in the general direction of their destination via vibration, but additional exploratory navigation is stimulated by varying feedback based on the potential for taking alternative routes. We describe two mobile prototypes that were created to examine the possible benefits of the approach. The successful use of this exploratory navigation method is demonstrated in a realistic field trial, and we discuss the results and interesting participant behaviours that were recorded.

This paper introduces the First International Workshop on Mobile Social Signal Processing (SSP).The Workshop aims at bringing together the Mobile HCI and Social Signal Processing research communities.The former investigates approaches for effective interaction with mobile and wearable devices, while the latter focuses on modeling, analysis and synthesis of nonverbal behavior in human-human and humanmachine interactions.While dealing with similar problems, the two domains have different goals and methodologies.However, mutual exchange of expertise is likely to raise new research questions as well as to improve approaches in both domains.After providing a brief survey of Mobile HCI and SSP, the paper introduces general aspects of the workshop (including topics, keynote speakers and dissemination means).

Developing interfaces for mobile situations requires that devices are useable on the move. Here, we explore head tilting as an input technique to allow a user to interact with a mobile device 'hands free'. A Fitts' Law style evaluation is described where a user acquires targets, moving the cursor by head tilt. We explore d position and velocity control cursor mechanisms in both static and mobile situations to see which provided the best level of performance. Results show that participants could successfully acquire targets using head tilting. Position control was shown to be si gnificantly faster and more accurate in a static context, but exhi bited significantly poorer accuracy and longer target acquisition times when the user was on the move. We further demonstrate how analysis of user's gait shows consistent targeting biases at different stages in the gait cycle.

Stane is a hand-held interaction device controlled by tactile input: scratching or rubbing textured surfaces and tapping. The system has a range of sensors, including contact microphones, capacitive sensing and inertial sensing, and provides audio and vibrotactile feedback. The surface textures vary around the device, providing perceivably different textures to the user. We demonstrate that the vibration signals generated by stroking and scratching these surfaces can be reliably classified, and can be used as a very cheaply manufacturable way to control different aspects of interaction. The system is demonstrated as a control for a music player.

In this paper, we investigate wrist rotation as a hands-free method of interaction with a mobile device. To evaluate this technique, a Fitts' Law targeting study is described in four different postures: resting, seated, standing and walking. Results show correlations in movement time and the Index of Difficulty of the task and similarities in the targeting performance for the first three conditions, but show walking and targeting using this method was significantly more difficult.

This paper presents a dynamic system approach to the design of multimodal interactive systems. We use an example where we support human behavior in a browsing task, by adapting the dynamics of navigation using speed-dependent automatic zooming (SDAZ), allowing the user to switch smoothly among different modes of control. We show how the user’s intention is coupled to the browsing technique via the dynamic model, and how the sdaz method couples the document structure to audio samples using a model-based sonification. We demonstrate that this approach is well suited to mobile and wearable applications, and audio feedback provides valuable information, supporting intermittent interaction, i.e. allowing movement-based interaction techniques to continue while the user is simultaneously involved with real life tasks. Author Keywords Multimodality, speed-dependent automatic zooming, mode

We demonstrate the use of uncertain prediction in asystem for pedestrian navigation via audio with a combination ofGlobal Positioning System data, a music player, inertial sensing,magnetic bearing data and Monte Carlo sampling for a densityfollowing task, where a listener's music is modulated according tothe changing predictions of user position with respect to a targetdensity, in this case a trajectory or path. We show that this system enables eyes-free navigation around set trajectories or paths unfamiliar to the user and demonstrate that the system may be used effectively for varying trajectory width and context.

Shoogle is a novel, intuitive interface for sensing data withina mobile device, such as presence and properties of textmessages or remaining resources. It is based around activeexploration: devices are shaken, revealing the contents rattlingaround "inside". Vibrotactile display and realistic impactsonification create a compelling system. Inertial sensingis used for completely eyes-free, single-handed interactionthat is entirely natural. Prototypes are described runningboth on a PDA and on a mobile phone with a wireless sensorpack. Scenarios of use are explored where active sensing ismore appropriate than the dominant alert paradigm.

This paper presents a photo browsing system on mobile devices to browse and search photos efficiently by tilting action. It employs tilt dynamics and a multi-scale photo screen layout for enhancing the browsing and the search capability respectively. The implementation uses continuous inputs from an accelerometer, and a multimodal (visual, audio and vibrotactile) display coupled with the states of this model. The model is based on a simple physical model, with its characteristics shaped to enhance controllability. The multi-scale layout holds both local and global view for users to both control photos and look at the surrounding context in a single framework. The experiment on Samsung MITs PDA used seven novice users browsing from 100 photos. We compare a tilt-based interaction method with a button-based browser and an iPod wheel by a quantitative usability criteria and subjective experience. The proposed tilt dynamics improves the usability over conventional dynamics. The iPod wheel has mixed performance comparing worse on some metrics than button pushing or tilt interaction, despite its commercial popularity.

Conversation partners on mobile phones can align their walking gait without physical proximity or visual feedback. We investigate gait synchronization, measured by accelerometers while users converse via mobile phones. Hilbert transforms are used to infer gait phase angle, and techniques from synchronization theory are used to infer level of alignment. Experimental conditions include the use of vibrotactile feedback to make one conversation partner aware of the other's footsteps. Three modes of interaction are tested: reading a script, discussing a shared image and spontaneous conversation. The vibrotactile feedback loop on its own is sufficient to create synchronization, but there are complex interference effects when users converse spontaneously. Even without vibration crosstalk, synchronisation appeared for long periods in the spontaneous speech condition, indicating that users were aligning their walking behaviour from audible cues alone.

We present a mobile, GPS-based multimodal navigation system, equipped with inertial control that allows users to explore and navigate through an augmented physical space, incorporating and displaying the uncertainty resulting from inaccurate sensing and unknown user intentions. The system propagates uncertainty appropriately via Monte Carlo sampling and predicts at a user-controllable time horizon. Control of the Monte Carlo exploration is entirely tilt-based. The system output is displayed both visually and in audio. Audio is rendered via granular synthesis to accurately display the probability of the user reaching targets in the space. We also demonstrate the use of uncertain prediction in a trajectory following task, where a section of music is modulated according to the changing predictions of user position with respect to the target trajectory. We show that appropriate display of the full distribution of potential future users positions with respect to sites-of-interest can improve the quality of interaction over a simplistic interpretation of the sensed data.

The paper presents a robot cell for multimodal standing-up motion augmentation. The robot cell is aimed at augmenting the standing-up capabilities of impaired or paraplegic subjects. The setup incorporates the rehabilitation robot device, functional electrical stimulation system, measurement instrumentation and cognitive feedback system. For controlling the standing-up process a novel approach was developed integrating the voluntary activity of a person in the control scheme of the rehabilitation robot. The simulation results demonstrate the possibility of “patient-driven” robot-assisted standing-up training. Moreover, to extend the system capabilities, the audio cognitive feedback is aimed to guide the subject throughout rising. For the feedback generation a granular synthesis method is utilized displaying high-dimensional, dynamic data. The principle of operation and example sonification in standing-up are presented. In this manner, by integrating the cognitive feedback and “patient-driven” actuation systems, an effective motion augmentation system is proposed in which the motion coordination is under the voluntary control of the user.

Understanding how humans assist each other in haptic interaction teams could lead to improved robotic aids to solo human dextrous manipulation. Inspired by experiments reported in Reed et al. (2004), which suggested two-person haptically interacting teams could achieve a lower movement time (MT) than individuals for discrete aiming movements of specified accuracy, we report that two-person teams (dyads) can also achieve lower MT for cyclical, continuous aiming movements. We propose a model, called endpoint compromise, for how the intended endpoints of both subjects' motion combine during haptic interaction; it predicts a ratio of /spl radic/2 between slopes of MT fits for individuals and dyads. This slope ratio prediction is supported by our data.

We combine the functionality of a mobile Global Positioning System (GPS) with that of an MP3 player, implemented on a PocketPC, to produce a handheld system capable of guiding a user to their desired target location via continuously adapted music feedback. We illustrate how the approach to presentation of the audio display can benefit from insights from control theory, such as predictive 'browsing' elements to the display, and the appropriate representation of uncertainty or ambiguity in the display. The probabilistic interpretation of the navigation task can be generalised to other context-dependent mobile applications. This is the first example of a completely handheld location- aware music player. We discuss scenarios for use of such systems.

This work introduces the idea of haptic rendering using granular synthesis - an established technique for synthesising audio. It describes the technique along with potential application areas, and initial results from an implementation on a PHANToM force feedback device. Three main applications are considered. Firstly, rendering of probabilistic vector fields for presenting ambiguity and context information to the user. Secondly, the possibility of producing textured virtual objects using granular synthesis is discussed. Thirdly, we use the approach to display scatterplot data on haptic devices.

The inspiration for this study is the observation that swing dancing involves coordination of actions between two humans that can be accomplished by pure haptic signaling. This study implements a leader-follower dance to be executed between a human and a PHANToM haptic device. The data demonstrates that the participants' understanding of the motion as a random sequence of known moves informs their following, making this vocabulary-based interaction fundamentally different from closed loop pursuit tracking. This robot leader does not respond to the follower's movement other than to display error from a nominal path. This work is the first step in an investigation of the successful haptic coordination between dancers, which will inform a subsequent design of a truly interactive robot leader.

We consider the problem of multi-step ahead prediction in time series analysis using the non-parametric Gaussian process model. -step ahead forecasting of a discrete-time non-linear dynamic system can be per- formed by doing repeated one-step ahead predictions. For a state-space at time model of the form is based on the point estimates of the previous outputs. In this pa-   per, we show how, using an analytical Gaussian approximation, we can formally incorporate the uncertainty about intermediate regressor values, thus updating the uncertainty on the current prediction.

Gaussian processes provide an approach to nonparametric modelling which allows a straightforward combination of function and derivative observations in an empirical model. This is of particular importance in identification of nonlinear dynamic systems from experimental data. 1) It allows us to combine derivative information, and associated uncertainty with normal function observations into the learning and inference pro- cess. This derivative information can be in the form of priors specified by an expert or identified from perturbation data close to equilibrium. 2) It allows a seamless fusion of multiple local linear models in a consis- tent manner, inferring consistent models and ensuring that integrability constraints are met. 3) It improves dramatically the computational ef- ficiency of Gaussian process models for dynamic system identification, by summarising large quantities of near-equilibrium data by a handful of linearisations, reducing the training set size – traditionally a problem for Gaussian process models.

Dynamic Takagi-Sugeno fuzzy models are not always easy to interpret, in particular when they are identified from experimental data. It is shown that there exists a close relationship between dynamic Takagi-Sugeno fuzzy models and dynamic linearization when using affine local model structures, which suggests that a solution to the multiobjective identification problem exists. However, it is also shown that the affine local model structure is a highly sensitive parametrization when applied in transient operating regimes. Due to the multiobjective nature of the identification problem studied here, special considerations must be made during model structure selection, experiment design, and identification in order to meet both objectives. Some guidelines for experiment design are suggested and some robust nonlinear identification algorithms are studied. These include constrained and regularized identification and locally weighted identification. Their usefulness in the present context is illustrated by examples.

Normalisation of the basis function activations in a Radial Basis Function (RBF) network is a common way of achieving the partition of unity often desired for modelling applications. It results in the basis functions covering the whole of the input space to the same degree. However, normalisation of the basis functions can lead to other effects which are sometimes less desirable for modelling applications. This paper describes some side effects of normalisation which fundamentally alter properties of the basis functions, e.g. the shape is no longer uniform, maxima of basis functions can be shifted from their centres, and the basis functions are no longer guaranteed to decrease monotonically as distance from their centre increases--in many cases basis functions can 'reactivate', i.e. re-appear far from the basis function centre. This paper examines how these phenomena occur, discusses their relevance for non-linear function approximation and examines the effect of normalisation on the network condition number and weights.

We establish the functional equivalence of a generalized class of Gaussian radial basis function (RBFs) networks and the full Takagi-Sugeno model (1983) of fuzzy inference. This generalizes an existing result which applies to the standard Gaussian RBF network and a restricted form of the Takagi-Sugeno fuzzy system. The more general framework allows the removal of some of the restrictive conditions of the previous result.