“Serious games” are becoming extremely relevant to individuals who have specific needs, such as children with an autism spectrum condition (ASC). Often, individuals with an ASC have difficulties in interpreting verbal and nonverbal communication cues during social interactions. The ASC-Inclusion EU-FP7 funded project aims to provide children who have an ASC with a platform to learn emotion expression and recognition, through play in the virtual world. In particular, the ASC-Inclusion platform focuses on the expression of emotion via facial, vocal, and bodily gestures. The platform combines multiple analysis tools, using onboard microphone and webcam capabilities. The platform utilizes these capabilities via training games, text-based communication, animations, video, and audio clips. This paper introduces current findings and evaluations of the ASC-Inclusion platform and provides detailed description for the different modalities.

Driver fatigue is one of the main causes of road accidents. It is essential to develop a reliable driver drowsiness detection system which can alert drivers without disturbing them and is robust to environmental changes. This paper explores yawning behaviour as a sign of drowsiness in spontaneous expressions of drowsy drivers in simulated driving scenarios. We analyse a labelled dataset of videos of sleep-deprived versus alert drivers and demonstrate the correlation between hand-over-face touches, face occlusions and yawning. We propose that face touches can be used as a novel cue in automated drowsiness detection alongside yawning and eye behaviour. Moreover, we present an automatic approach to detect yawning based on extracting geometric and appearance features of both mouth and eye regions. Our approach successfully detects both hand-covered and uncovered yawns with an accuracy of 95%. Ultimately, our goal is to use these results in designing a hybrid drowsiness-detection system.

Abstract We present GazeDirector, a new approach for eye gaze redirection that uses model‐fitting. Our method first tracks the eyes by fitting a multi‐part eye region model to video frames using analysis‐by‐synthesis, thereby recovering eye region shape, texture, pose, and gaze simultaneously. It then redirects gaze by 1) warping the eyelids from the original image using a model‐derived flow field, and 2) rendering and compositing synthesized 3D eyeballs onto the output image in a photorealistic manner. GazeDirector allows us to change where people are looking without person‐specific training data, and with full articulation, i.e. we can precisely specify new gaze directions in 3D. Quantitatively, we evaluate both model‐fitting and gaze synthesis, with experiments for gaze estimation and redirection on the Columbia gaze dataset. Qualitatively, we compare GazeDirector against recent work on gaze redirection, showing better results especially for large redirection angles. Finally, we demonstrate gaze redirection on YouTube videos by introducing new 3D gaze targets and by manipulating visual behavior.

This paper presents a multimodal approach to in-vehicle classification of driver glances. Driver glance is a strong predictor of cognitive load and is a useful input to many applications in the automotive domain. Six descriptive glance regions are defined and a classifier is trained on video recordings of drivers from a single low-cost camera. Visual features such as head orientation, eye gaze and confidence ratings are extracted, then statistical methods are used to perform failure analysis and calibration on the visual features. Non-visual features such as steering wheel angle and indicator position are extracted from a RaceLogic VBOX system. The approach is evaluated on a dataset containing multiple 60 second samples from 14 participants recorded while driving in a natural environment. We compare our multimodal approach to separate unimodal approaches using both Support Vector Machine (SVM) and Random Forests (RF) classifiers. RF Mean Decrease in Gini Index is used to rank selected features which gives insight into the selected features and improves the classifier performance. We demonstrate that our multimodal approach yields significantly higher results than unimodal approaches. The final model achieves an average F1score of 70.5% across the six classes.

Assessing pain levels in animals is a crucial, but time-consuming process in maintaining their welfare. Facial expressions in sheep are an efficient and reliable indicator of pain levels. In this paper, we have extended techniques for recognising human facial expressions to encompass facial action units in sheep, which can then facilitate automatic estimation of pain levels. Our multi-level approach starts with detection of sheep faces, localisation of facial landmarks, normalisation and then extraction of facial features. These are described using Histogram of Oriented Gradients, and then classified using Support Vector Machines. Our experiments show an overall accuracy of 67% on sheep Action Units classification. We argue that with more data, our approach on automated pain level assessment can be generalised to other animals.

Current navigational systems rarely consider generic road landmarks in their navigation instructions, which can lead to mistakes, frustration, and distraction. However, automatic detection of road landmarks is difficult, as current approaches to object detection focus either on out-of-context objects which have special characteristics or on very specific domains. This work presents a future direction for a user-friendly navigational system based on state-of-the-art computer vision techniques that use deep learning for object detection. We propose an automatic hierarchical approach for detecting and classifying a set of static and dynamic road landmarks that would be useful in automatic navigational systems. We further demonstrate a set of optimisations that improve performance and accuracy of the basic system. We evaluate our approach on a natural, 'in-the-wild' dataset to determine how well it handles natural automotive input. Finally, we demonstrate a use-case for our system that extracts information about a vehicle's location and intention.

Learning-based methods for appearance-based gaze estimation achieve state-of-the-art performance in challenging real-world settings but require large amounts of labelled training data. Learning-by-synthesis was proposed as a promising solution to this problem but current methods are limited with respect to speed, appearance variability, and the head pose and gaze angle distribution they can synthesize. We present UnityEyes, a novel method to rapidly synthesize large amounts of variable eye region images as training data. Our method combines a novel generative 3D model of the human eye region with a real-time rendering framework. The model is based on high-resolution 3D face scans and uses real-time approximations for complex eyeball materials and structures as well as anatomically inspired procedural geometry methods for eyelid animation. We show that these synthesized images can be used to estimate gaze in difficult in-the-wild scenarios, even for extreme gaze angles or in cases in which the pupil is fully occluded. We also demonstrate competitive gaze estimation results on a benchmark in-the-wild dataset, despite only using a light-weight nearest-neighbor algorithm. We are making our UnityEyes synthesis framework available online for the benefit of the research community.

Tilting refers to losing control due to negative emotions, behaving erratically, and thereby suffering adverse consequences. The term tilt originated in poker playing communities, where it is often a consequence of so called bad beats, i.e. losing with an unlikely-to-lose poker hand. Often resulting in devastating monetary losses, tilting is ubiquitous and well known in poker, but rarely studied despite its significance. In this paper, we argue that tilting is a fertile topic for interdisciplinary research both for psychologists and computer scientists. Specifically, we propose to study the manifestation of tilting via facial emotion expressions, and motivate the development of an automatic tilt-detection system. The scientific understanding of the psychology of tilting can be increased via a computing approach, which has not been previously attempted. Automatic tilting detection will lead to a practical technology that reduces poker players' monetary losses and improves their well being through reduced tilting. We also argue that while tilting is best known as a poker phenomenon, it also exists in other contexts. Thus, the idea we suggest is a novel application of computer vision, affective computing and multimedia technologies in the real world, across many domains.

Over the past few years, there has been an increased interest in automatic facial behavior analysis and understanding. We present OpenFace - an open source tool intended for computer vision and machine learning researchers, affective computing community and people interested in building interactive applications based on facial behavior analysis. OpenFace is the first open source tool capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation. The computer vision algorithms which represent the core of OpenFace demonstrate state-of-the-art results in all of the above mentioned tasks. Furthermore, our tool is capable of real-time performance and is able to run from a simple webcam without any specialist hardware. Finally, OpenFace allows for easy integration with other applications and devices through a lightweight messaging system.

In this paper we present a method for localisation of facial landmarks on human and sheep. We introduce a new feature extraction scheme called triplet-interpolated feature used at each iteration of the cascaded shape regression framework. It is able to extract features from similar semantic location given an estimated shape, even when head pose variations are large and the facial landmarks are very sparsely distributed. Furthermore, we study the impact of training data imbalance on model performance and propose a training sample augmentation scheme that produces more initialisations for training samples from the minority. More specifically, the augmentation number for a training sample is made to be negatively correlated to the value of the fitted probability density function at the sample's position. We evaluate the proposed scheme on both human and sheep facial landmarks localisation. On the benchmark 300w human face dataset, we demonstrate the benefits of our proposed methods and show very competitive performance when comparing to other methods. On a newly created sheep face dataset, we get very good performance despite the fact that we only have a limited number of training samples and a set of sparse landmarks are annotated. Source code is available for academic use l.

One of the main factors that limit the accuracy of facial analysis systems is hand occlusion. As the face becomes occluded, facial features are lost, corrupted, or erroneously detected. Hand-over-face occlusions are considered not only very common but also very challenging to handle. However, there is empirical evidence that some of these hand-over-face gestures serve as cues for recognition of cognitive mental states. In this article, we present an analysis of automatic detection and classification of hand-over-face gestures. We detect hand-over-face occlusions and classify hand-over-face gesture descriptors in videos of natural expressions using multi-modal fusion of different state-of-the-art spatial and spatio-temporal features. We show experimentally that we can successfully detect face occlusions with an accuracy of 83%. We also demonstrate that we can classify gesture descriptors ( hand shape , hand action , and facial region occluded ) significantly better than a naïve baseline. Our detailed quantitative analysis sheds some light on the challenges of automatic classification of hand-over-face gestures in natural expressions.

Automatic analysis of affect from facial expressions has been extensively studied, but most work has considered only a small set of discrete emotions, typically Ekman's six basic emotions, or a small number of continuous measures, typically valence and arousal. We have developed a system that accommodates a much larger vocabulary of discrete emotions and links them with continuous measures that have been aggregated over a few seconds. The approach has a sound theoretical basis in multi-dimensional statistics, making it both principled and robust, while a graphical presentation makes it easy to understand.

Classifying complex categorical emotions has been a relatively unexplored area of affective computing. We present a classifier trained to recognize 18 complex emotion categories. A leave-one-out training approach was used on 181 acted videos from the EU-Emotion Stimulus Set. Performance scores for the 18-choice classification problem were AROC = 0.84, 2AFC = 0.84, F1 = 0.33, Accuracy = 0.47. On a simplified 6-choice classification problem, the classifier had an accuracy of 0.64 compared with the validated human accuracy of 0.74. The classifier has been integrated into an expression training interface which gives meaningful feedback to humans on their portrayal of complex emotions through face and head movements. This work has applications as an intervention for Autism Spectrum Conditions.

Imitation is an important aspect of emotion recognition. We present an expression training interface which evaluates the imitation of facial expressions and head movements. The system provides feedback on complex emotion expression, via an integrated emotion classifier which can recognize 18 complex emotions. Feedback is also provided for exact-expression imitation via dynamic time warping. Discrepancies in intensity and frequency of action units are communicated via simple graphs. This work has applications as a training tool for customer-facing professionals and people with Autism Spectrum Conditions.

Perception of emotion through facial expressions and head motion is of interest to both psychology and affective computing researchers. However, very little is known about the importance of each modality individually, as they are often treated together rather than separately. We present a study which isolates the effect of head motion from facial expression in the perception of complex emotions in videos. We demonstrate that head motions carry emotional information that is complementary rather than redundant to the emotion content in facial expressions. Finally, we show that emotional expressivity in head motion is not limited to nods and shakes and that additional gestures (such as head tilts, raises and general amount of motion) could be beneficial to automated recognition systems.

The recognition of continuous dimensional emotion remains a challenging task due to large variations in the expression of emotion, and the difficulty of modeling emotion as temporal processes. This work proposes the use of a Nonlinear AutoRegressive with eXogenous inputs recurrent neural network (NARX-RNN) to learn emotional patterns in a given a dataset. The application of particle swarm optimisation in training the NARX-RNN is considered and compared to a gradient descent algorithm. We show that the NARX-RNN outperforms other methods in its emotion recognition ability, and can be easily trained with both gradient-free and gradient-based optimization methods.

In this paper we propose a supervised initialization scheme for cascaded face alignment based on explicit head pose estimation. We first investigate the failure cases of most state of the art face alignment approaches and observe that these failures often share one common global property, i.e. the head pose variation is usually large. Inspired by this, we propose a deep convolutional network model for reliable and accurate head pose estimation. Instead of using a mean face shape, or randomly selected shapes for cascaded face alignment initialisation, we propose two schemes for generating initialisation: the first one relies on projecting a mean 3D face shape (represented by 3D facial landmarks) onto 2D image under the estimated head pose; the second one searches nearest neighbour shapes from the training set according to head pose distance. By doing so, the initialisation gets closer to the actual shape, which enhances the possibility of convergence and in turn improves the face alignment performance. We demonstrate the proposed method on the benchmark 300W dataset and show very competitive performance in both head pose estimation and face alignment.

Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model's controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

Digital Games for Empowerment and Inclusion have the potential to improve our society by preparing particular groups of people to meet social challenges in their everyday lives, and to do so in an enjoyable way through games. These games are developing rapidly to exploit new algorithms for computational intelligence supported by increasing availability of computing power to help analyze players' behavior, monitor their motivation and interest, and to adapt the progress of the games accordingly. The workshop on Intelligent Digital Games for Empowerment and Inclusion (IDGEI) explores the use of machine intelligence in serious digital games. In this context, we summarize the third international workshop on IDGEI held at the International Conference on Intelligent User Interfaces (IUI) 2015.

Emotional expression and recognition are important in social interactions between people. This is particularly evident in communications between a teacher and a pupil when facial expressions signal levels of understanding and enjoyment will change the teacher's explanation to a pupil, and effective e-learning systems must adapt in the same way if they are provide the social interactions that are necessary for effective pedagogy. Affective computing can equip computer systems with the ability to process social signals and respond accordingly. However, social signals are inherently ambiguous and confusion will result if the approach to processing them is too mechanistic. This paper presents and analyses empirical evidence for this ambiguity, and proposes a possible solution. The techniques are applicable in a wide variety of applications where continuous measures of performance are being assessed.

One of the main factors that limit the accuracy of facial analysis systems is hand occlusion. As the face becomes occluded, facial features are either lost, corrupted or erroneously detected. Hand-over-face occlusions are considered not only very common but also very challenging to handle. Moreover, there is empirical evidence that some of these hand-over-face gestures serve as cues for recognition of cognitive mental states. In this paper, we detect hand-over-face occlusions and classify hand-over-face gesture descriptors in videos of natural expressions using multi-modal fusion of different state-of-the-art spatial and spatio-temporal features. We show experimentally that we can successfully detect face occlusions with an accuracy of 83%. We also demonstrate that we can classify gesture descriptors (hand shape, hand action and facial region occluded) significantly higher than a naive baseline. To our knowledge, this work is the first attempt to automatically detect and classify hand-over-face gestures in natural expressions.

When people talk to each other, they express their feelings through facial expressions, tone of voice, body postures and gestures. They even do this when they are interacting with machines. These hidden signals are an important part of human communication, but most computer systems ignore them. Emotions need to be considered as an important mode of communication between people and interactive systems. Affective computing has enjoyed considerable success over the past 20 years, but many challenges remain.

Continuous emotion prediction in the arousal-valence space is now being used in various modalities: music, facial expressions, gestures, text, etc. In order to be able to compare the work of different research groups effectively, we believe it is necessary to set certain guidelines for how to conduct research-the choice of evaluation metrics of emotion recognition algorithms in particular. In this paper we focus on the field of musical emotion recognition and describe a study designed to discover people's instinctive preference among the most commonly used evaluation techniques. We gather strong evidence that root mean squared error or Kullback-Leibler divergence should be used for regression based approaches. The raw study data we collected is made publicly available.

Prolonged durations of rhythmic body gestures were proved to be correlated with different types of psychological disorders. To-date, there is no automatic descriptor that can robustly detect those behaviours. In this paper, we propose a cyclic gestures descriptor that can detect and localise rhythmic body movements by taking advantage of both colour and depth modalities. We show experimentally how our rhythmic descriptor can successfully localise the rhythmic gestures as: hands fidgeting, legs fidgeting or rocking, significantly higher than the majority vote classification baseline. Our experiments also demonstrate the importance of fusing both modalities, with a significant increase in performance when compared to individual modalities.

We present 3D Constrained Local Model (CLM-Z) for robust facial feature tracking under varying pose. Our approach integrates both depth and intensity information in a common framework. We show the benefit of our CLM-Z method in both accuracy and convergence rates over regular CLM formulation through experiments on publicly available datasets. Additionally, we demonstrate a way to combine a rigid head pose tracker with CLM-Z that benefits rigid head tracking. We show better performance than the current state-of-the-art approaches in head pose tracking with our extension of the generalised adaptive view-based appearance model (GAVAM).

Elderly and disabled people can be hugely benefited through the advancement of modern electronic devices, as those can help them to engage more fully with the world. However, existing design practices often isolate elderly or disabled users by considering them as users with special needs. This article presents a simulator that can reflect problems faced by elderly and disabled users while they use computer, television, and similar electronic devices. The simulator embodies both the internal state of an application and the perceptual, cognitive, and motor processes of its user. It can help interface designers to understand, visualize, and measure the effect of impairment on interaction with an interface. Initially a brief survey of different user modeling techniques is presented, and then the existing models are classified into different categories. In the context of existing modeling approaches the work on user modeling is presented for people with a wide range of abilities. A few applications of the simulator, which shows the predictions are accurate enough to make design choices and point out the implication and limitations of the work, are also discussed.

We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.

This paper presents a new classification algorithm for real-time inference of affect from nonverbal features of speech and applies it to assessing public speaking skills. The classifier identifies simultaneously occurring affective states by recognizing correlations between emotions and over 6,000 functional-feature combinations. Pairwise classifiers are constructed for nine classes from the Mind Reading emotion corpus, yielding an average cross-validation accuracy of 89 percent for the pairwise machines and 86 percent for the fused machine. The paper also shows a novel application of the classifier for assessing public speaking skills, achieving an average cross-validation accuracy of 81 percent and a leave-one-speaker-out classification accuracy of 61 percent. Optimizing support vector machine coefficients using grid parameter search is shown to improve the accuracy by up to 25 percent. The emotion classifier outperforms previous research on the same emotion corpus and is successfully applied to analyze public speaking skills.

Cooperative gestures are a key aspect of human-human pro-social interaction. Thus, it is reasonable to expect that endowing humanoid robots with the ability to use such gestures when interacting with humans would be useful. However, while people are used to responding to such gestures expressed by other humans, it is unclear how they might react to a robot making them. To explore this topic, we conducted a within-subjects, video based laboratory experiment, measuring time to cooperate with a humanoid robot making interactional gestures. We manipulated the gesture type (beckon, give, shake hands), the gesture style (smooth, abrupt), and the gesture orientation (front, side). We also employed two measures of individual differences: negative attitudes toward robots (NARS) and human gesture decoding ability (DANVA2-POS). Our results show that people cooperate with abrupt gestures more quickly than smooth ones and front-oriented gestures more quickly than those made to the side, people's speed at decoding robot gestures is correlated with their ability to decode human gestures, and negative attitudes toward robots is strongly correlated with a decreased ability in decoding human gestures.

Emotions have a functional relevance to learning and achievement. Not surprisingly then, affective diagnoses are an important aspect of expert human mentoring. Computer-based learning environments aim to model such social dynamics to make learning with computers more immersive, engaging and hence, more effective. This paper draws on the recent surge of interest in studying emotions in learning, highlights available techniques for measuring emotions and surveys recent efforts to automatically measure emotional experience in learning environments. Finally, a context-sensitive dataset is used to develop an automatic system for modeling six pertinent emotions. This paper attempts to bring together the motivation, methodological issues, and modeling approaches for affect inference in learning environments in order to contribute to an understanding of the problem and the current state-of-art.

We have developed a simulator to help with the design and evaluation of assistive interfaces. The simulator can predict possible interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different dis-abilities. In this paper, we have presented a study to evaluate the simulator by considering a representative application being used by able-bodied, visually impaired and mobility impaired people. The simulator predicted task completion times for all three groups with statistically significant accuracy. The simulator also predicted the effects of different interface designs on task completion time accurately.

We present a classification algorithm for inferring affective states (emotions, mental states, attitudes, and the like) from their nonverbal expressions in speech. It is based on the observations that affective states can occur simultaneously and different sets of vocal features, such as intonation and speech rate, distinguish between nonverbal expressions of different affective states. The input to the inference system was a large set of vocal features and metrics that were extracted from each utterance. The classification algorithm conducted independent pairwise comparisons between nine affective-state groups. The classifier used various subsets of metrics of the vocal features and various classification algorithms for different pairs of affective-state groups. Average classification accuracy of the 36 pairwise machines was 75 percent, using 10-fold cross validation. The comparison results were consolidated into a single ranked list of the nine affective-state groups. This list was the output of the system and represented the inferred combination of co-occurring affective states for the analyzed utterance. The inference accuracy of the combined machine was 83 percent. The system automatically characterized over 500 affective state concepts from the Mind Reading database. The inference of co-occurring affective states was validated by comparing the inferred combinations to the lexical definitions of the labels of the analyzed sentences. The distinguishing capabilities of the system were comparable to human performance.

There is growing interest in tabletop interfaces that enable remote collaboration by providing shared workspaces. This approach assumes that these remote tabletops afford the same beneficial work practices as co-located tabletop interfaces and traditional tables. This assumption has not been tested in practice. We explore two such work practices in remote tabletop collaboration: (a) coordination by territorial partitioning of space; and (b) transitioning between individual and group work within a shared task. We have evaluated co-located and remote tabletop collaboration. We found that remote collaborators did not coordinate territorially as co-located collaborators did. We found no differences between remote and co-located interfaces in their ability to afford individual and group work. However, certain interaction techniques impaired the ability to transition fluidly between these working styles. We discuss causes and the implications for the design and future study of these interfaces.

A long-standing question within the robotics community is about the degree of human-likeness robots ought to have when interacting with humans. We explore an unexamined aspect of this problem: how people empathize with robots along the anthropomorphic spectrum. We conducted an experiment that measured how people empathized with robots shown to be experiencing mistreatment by humans. Our results indicate that people empathize more strongly with more human-looking robots and less with mechanicallooking robots.

Fusion engines are fundamental components of multimodal inter-active systems, to interpret input streams whose meaning can vary according to the context, task, user and time. Other surveys have considered multimodal interactive systems; we focus more closely on the design, specification, construction and evaluation of fusion engines. We first introduce some terminology and set out the major challenges that fusion engines propose to solve. A history of past work in the field of fusion engines is then presented using the BRETAM model. These approaches to fusion are then classified. The classification considers the types of application, the fusion principles and the temporal aspects. Finally, the challenges for future work in the field of fusion engines are set out. These include software frameworks, quantitative evaluation, machine learning and adaptation.

In this paper we present a novel system for driver-vehicle interaction which combines speech recognition with facial-expression recognition to increase intention recognition accuracy in the presence of engine- and road-noise. Our system would allow drivers to interact with in-car devices such as satellite navigation and other telematic or control systems. We describe a pilot study and experiment in which we tested the system, and show that multimodal fusion of speech and facial expression recognition provides higher accuracy than either would do alone.

Computers offer valuable assistance to people with physical disabilities. However designing human-computer interfaces for these users is complicated. The range of abilities is more diverse than for able-bodied users, which makes analytical modelling harder. Practical user trials are also difficult and time consuming. We are developing a simulator to help with the evaluation of assistive interfaces. It can predict the likely interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different disabilities and for different levels of skill. In this paper we describe the different components of the simulator in detail and present a prototype of its implementation.

pb400 @ cam.ac.uk Computers offer valuable assistance to people with physical disabilities. However designing human-computer interfaces for these users is complicated. The range of abilities is more diverse than for able-bodied users, which makes analytical modelling harder. Practical user trials are also difficult and time consuming. We have developed a simulator to help with the evaluation of assistive interfaces. It can predict the likely interaction patterns when undertaking a task using a variety of input devices, and estimate the time to complete the task in the presence of different disabilities and for different levels of skill.

Visual material presented in lectures can be enhanced for students with disabilities by using high-resolution digital-still cameras. The Photonote system uses a digital-still camera to capture visual information, a digital-video camera to capture a lecturer and a second digital-video camera to capture a sign-language interpreter, if necessary. The visual information is enhanced using computer-vision algorithms and presented alongside the recorded video and audio to provide an accurate representation of a lecture which can be used by students with disabilities for review purposes. This paper presents the Photonote system and a user study evaluating its effectiveness at aiding students with disabilities in a lecture environment.

Displaying small text on large multiprojector tiled displays is challenging. Problems arise because text is badly affected by the image-warping techniques that these displays apply to rectify projector misalignment. As a consequence, there has been little progress with important large-display applications that require small text, such as collaborative tutoring or Web-browsing. In this paper we present a new warping technique designed to preserve crisp text, based on recent work by Hereld and Stevens. Our technique produces good results, free of artifacts, when used in today's multiprojector displays. We evaluate the legibility of our technique against conventional interpolation-based warping and find that users prefer our technique. We describe an efficient and reusable implementation, and show how the increased legibility has allowed us to investigate two new applications.

Visual content in lectures can be enhanced for use by students with visual disabilities by using high-resolution digital still cameras. This paper presents a system which uses two high-resolution cameras; one to capture multiple sources of visual content and another to monitor the head pose of up to 20 audience members. This capture technique eliminates the need for multiple cameras or intrusive and distracting instrumentation but introduced some new problems which were solved with an algorithm used to distinguish between two possible sources of visual interest.

Abstract There is no ideal programming language. Each is better suited to some tasks rather than others. This is true not only for complete programs, but also for different stages such as architectural design, detailed implementation and maintenance. The situation is even more acute in the case of end‐user programming languages, which cater for a much more varied user population. It would therefore be advantageous to allow the same program to be viewed, and edited, in a number of different languages. We have developed a system, Lingua Franca, that provides this facility for end‐user programming languages in the setting of ubiquitous computing in the home. Copyright © 2006 John Wiley & Sons, Ltd.

Work on remote collaboration has often focused on the person space created by a conventional videoconference where the participants see each other's faces, but we argue that a task space containing shared visual information is more important for most tasks. Trends in display technology mean that large visual task spaces can be created to maximize the shared context between collaborators. We have created a system called the Escritoire that presents users with a desk-sized projected display with bimanual input that allows documents and images to be arranged and modified by multiple remote collaborators. We describe the software architecture, the protocol that is used between the client and server programs, and the pen traces we have added to allow participants to gesture to each other in the large visual space to enrich their communication. Our user trials have shown that participants were able to use the system with a minimum of training, and found the traces useful in the collaborative setting. In future we will be connecting three or more sites together to explore the issues that arise with multi-party interaction on large shared desks. Copyright © 2005 John Wiley & Sons, Ltd.

In this paper we examine some of the issues involved in analyzing the expression of emotions, mental states and attitudes in speech. We investigate timing issues that may affect the structure of future automated inference systems. These timing issues include tracking transitions and changes over time within expressions, existence of thresholds between expressions, and the necessity of tracking time-frequency features within an utterance. We also suggest an outline for a system that includes these features.

One of the limitations in traditional instant messaging platforms is that they predominantly rely on text messages as the primary form of expression. This paper presents FAIM, an instant messaging application that analyzes a person's facial affect in real time and augments the dialogue with an emotive character representing them. Throughout the paper, we identify a number of design challenges that arise from integrating facial affect into instant messaging, and discuss how each of these issues is addressed in the design of FAIM. We also present a use case scenario of how FAIM works.

User interfaces using windows, keyboard and mouse have been in use for over 30 years, but only offer limited facilities to the user. Conventional displays are small, at least compared with a physical desk; conventional input devices restrict both manual expression and cognitive flexibility; remote collaboration is a poor shadow of sitting in the same room. We show how recent technological advances in large display devices and input devices can address these problems. The Escritoire is a desk-based interface using overlapping projectors to create a large display with a high resolution region in the centre for detailed work. Two pens provide bimanual input over the entire area, and an interface like physical paper addresses some of the affordances not provided by the conventional user interface. Multiple desks can be connected to allow remote collaboration. The system has been tested with single users and collaborating pairs.

"Point and click" interactions remain one of the key features of graphical user interfaces (GUIs). People with motion-impairments, however, can often have difficulty with accurate control of standard pointing devices. This paper discusses work that aims to reveal the nature of these difficulties through analyses that consider the cursor's path of movement. A range of potential cursor measures was applied, and a number of them were found to be significant in capturing the differences between able-bodied users and motion-impaired users, as well as the differences between a haptic force feedback condition and a control condition. cursor measures found in the literature, however, do not make up a comprehensive list, but provide a starting point for analysing cursor movements more completely. Six new cursor characteristics for motion-impaired users are introduced to capture aspects of cursor movement different from those already proposed.

It is known that many products, both software and hardware, are not accessible to large sections of the population. Designers instinctively design for able-bodied users and are either unaware of the needs of users with different capabilities, or do not know how to accommodate their needs into the design cycle. The aim of this paper is to present a methodological design approach for implementing inclusive interface design. This will begin with a discussion about what constitutes good inclusive design and suitable measures of success. A summary of the principal methods for designing for users with different capabilities is given along with a description of a model, the inclusive design cube, that displays how the different approaches are complementary and can provide complete population coverage. Two case studies shall be used to illustrate the use of the model. The first details the design of an interface for an interactive robot. The second looks at the design of an information point for use in a post office, and shall be used to highlight the use of both the design approach and the model.

The active guide book is a context-aware information management system that uses a combination of spatial and keyword indexing to retrieve data. The system has three principal components: A new document description language extends HTML to include facilities for tagging with spatial locations. Retrieval uses two separate indexes - a segment tree is used for spatial indexing and an inverted file is used for keyword indexing. A user interface allows queries involving keywords and location data to be expressed, and presents their results. The system has been evaluated with the implementation of an interactive guidebook. The test data was drawn from existing Web pages describing the City of Cambridge in England, which were augmented with spatial information. A GPS system is used to provide the default location information for retrieval, but can be overridden with explicit coordinates.

This paper considers the differences between users with motion-impairments and able-bodied users when they interact with computers and the implications for user models. Most interface design and usability assessment practices are based on explicit or implicit models of user behaviour. This paper studies the applicability of an existing interface design user model to motion-impaired users for the relatively straightforward task of button activation. A discussion of the empirical results is provided and the paper concludes that there are significant differences between the behaviour of motion-impaired users and the accepted modelling theory.

We present a local clocking mechanism based on a tunable delay line which calibrates itself from a low frequency global clock. After initial tuning, the local clock remains calibrated when environmental conditions change. Each module of a large system on a chip can use one of these clock generators running at the optimal frequency for the module. Communication between locally synchronous blocks is provided by a globally asynchronous interconnect. Reliable low latency communication between the asynchronous interconnect and a local clock domain is made possible by stretching the local clock if a metastable condition could be encountered. Stretching the clock just requires the rising clock edge to be prevented from entering the tuned delay line. Similarly, a sleep state can be entered by stopping the clock and wakeup is almost instantaneous. Fine grained sleeping is possible by sleeping whenever there is no work to be undertaken and waking up as soon as new data appears over the asynchronous interconnect.

For users with motion impairments, the standard keyboard and mouse arrangement for computer access often presents problems. Other approaches have to be adopted to overcome this. In this paper, we will describe the development of a prototype multimodal input system based on two gestural input channels. Results from extensive user trials of this system are presented. These trials showed that the physical and cognitive loads on the user can quickly become excessive and detrimental to the interaction. Designers of multimodal input systems need to be aware of this and perform regular user trials to minimize the problem.

Self-timed circuits relieve the designer of problems like clock distribution but introduce new constraints in the form of isochronic forks and equipotential regions. This paper shows how the combination of floor- and geometry-planning tools can be used to address these new problems. As a result, prototype self-timed circuits can be developed on conventional, clocked FPGAs without sacrificing performance. We also present a solution to the problem of designing arbiters on FPGAs.

This paper reports on ways of using digitised video from television cameras in user interfaces for computer systems. The DigitalDesk is built around an ordinary physical desk and can be used as such, but it has extra capabilities. A video camera mounted above the desk, pointing down at the work surface, is used to detect where the user is pointing and to read documents that are placed on the desk. A computer‐driven projector is also mounted above the desk, allowing the system to project electronic objects onto the work surface and onto real paper documents. The animated paper documents project is considering particular applications of the technology in electronic publishing. The goal is to combine electronic and printed documents to give a richer presentation than that afforded by either separate medium. This paper describes the framework that has been developed to assist with the preparation and presentation of these mixed‐media documents. The central component is a registry that associates physical locations on pieces of paper with actions. This is surrounded by a number of adaptors that assist with the creation of new documents either from scratch or by translating from conventional hypermedia, and also allow the documents to be edited. Finally the DigitalDesk itself identifies pieces of paper and animates them with the actions described in the registry.

The goal of 'Computer Augmented Environments' is to bring computational power to everyday objects with which users are already familiar, so that the user interface to this computational power becomes almost invisible.Video is a very important tool in creating Augmented Environments and recent camera-manufacturing techniques make it an economically viable proposition in the general marketplace.BrightBoard is an example system which uses a video camera and audio feedback to enhance the facilities of an ordinary whiteboard, allowing a user to control a computer through simple marks made on the board.We describe its operation in some detail, and discuss how it tackles some of the problems common to these 'Video-Augmented Environments'.

No abstract available.

A novel approach for the channel routing problem based on sorting is described. A router based on the bubble-sort for routing two point nets across a channel is presented. A lower and an upper bound on the channel width and an upperbound on the number of vias are given. The router is extended for handling multiterminal nets and single-sided connections. Results for multilayer channel routing are presented. The high-speed and predictable performance of these channel routing algorithms makes them valuable in assessing the difficulty of routing a particular configuration of logic blocks. This makes then suitable for use when calculating a cost metric in an iterative placement algorithm.>

An experimental workstation is described that supports high-quality colour raster graphics in a distributed environment. Hardware assistance for screen management is provided, using a method that allows rapid reconfiguration of screen images. Much of the workstation function is defined by high-level software and loadable microcode, thus permitting the exploration of different ways of using this kind of equipment.

The problem of quality in printed text produced by computer has received little attention. Three pieces of recent work at Cambridge which relate to different aspects of the problem are presented in this paper. The first is a letter design system, which enables the typographer to exploit not only the power of the computer but also the freedom which modern printing technology has made available. The second is an attack on the problem of high quality text for television transmission, especially where the text is itself computer generated. Third, is a system which enables the user to set and correct tabular text easily and flexibly.

Abstract Conventional operating systems have for many‐years used the ideas of standard stream interfaces to peripherals. Multi‐tasking has been used in order to give an efficient response while maintaining simplicity and flexibility for the user. This paper describes an extension to an operating system which makes similar facilities available for the control of interactive workstations, and also provides a command dispatcher which enables the straightforward implementation of arbitrary user‐defined command languages. The current system has been written in BCPL to run on a PDP 11/45 handling a Vector General display, but the design is suitable for use with a variety of terminals, computers and languages.