3D environments offer powerful capabilities for simulating and monitoring complex systems. However, they remain inaccessible to non-expert users who lack specialised training. We present the Layered Customisation System (LCS), a hybrid interface that combines GUI-based direct manipulation with LLM-powered natural language interaction for customising 3D scenes. Through a controlled study with 12 participants, each completing three sessions with different interfaces (GUI-only, LLM-only and hybrid), we investigated how interaction types affect performance, errors and user preferences. Key findings include a reduced learning effect across sessions for users starting with the LLM interface and a trend where participants who first experienced GUI-based interaction showed higher rates of recognising LLM errors later. We contribute empirical evidence of GUI-LLM modality trade-offs in 3D interaction, the selector-layer architecture for hybrid interfaces, and design recommendations, including GUI-first onboarding, enabling prompts as discovery tools and supporting prompt-then-refine workflows.

In the rapidly evolving landscape of cross-device computing and the Internet of Things (IoT), there is a need for intuitive and user-friendly solutions empowering end users to create and tailor their applications. To address this need, we analysed the different metaphors for End-User Development (EUD) of cross-device and IoT applications. A key observation is the lack of addressing end user’s mental models when designing interactions among devices, resulting in less intuitive EUD tools. To fill this gap, we introduce eSPACE, an end-user authoring tool facilitating the development of cross-device and IoT applications. In contrast to most existing tools, eSPACE is grounded on strong theoretical foundations, including a conceptual model, reference framework as well as design guidelines and suitable metaphors. The effectiveness of these theoretical foundations in creating an intuitive and user-friendly EUD platform has been validated in a user study. Our study confirms eSPACE’s potential as a useful and easy to use tool for end-user application development. In addition, we present potential future research directions, including automation functionality, intelligibility and human-AI interaction. In discussing these future directions, we aim to foster and advance EUD research towards more end-user-accessible authoring environments for cross-device and IoT applications.

With the rise of popular task automation or IoT platforms such as If This Then That (IFTTT), users can define rules to enable interactions between smart devices in their environment and thereby improve their daily lives.However, the rules authored via these platforms are usually tied to the platforms and sometimes even to specific devices for which they have been defined.Therefore, when a user wishes to move to a different environment controlled by a different platform and/or devices, they need to recreate their rules for the new environment.The rise in the number of smart devices further adds to the complexity of rule authoring since users will have to navigate an ever-changing landscape of IoT devices.In order to address this problem, we need humancomputer interaction that works across the boundaries of specific IoT platforms and devices.A step towards this human-computer interaction across platforms and devices is the introduction of a high-level semantic model for end-user IoT development, enabling users to create rules at a higher level of abstraction.However, many users who are used to the rule representation in their favourite tool might be unwilling to learn and adapt to a new representation.We present a method for translating proprietary rules to a high-level semantic model by using natural language processing techniques.Our translation enables users to work with their familiar rule representation language and tool, but then apply their rules across different IoT platforms and devices.

The tech industry is a fast-growing field, with many companies facing issues in finding skilled workers to fill their open vacancies.At the same time, many people have limited access to the quality education necessary to enter this job market.To address this issue, various small and often volunteer-run non-profit organisations have emerged to up-skill capable learners.However, these organisations face tight constraints and many challenges while trying to design and deliver high-quality education to their learners.In this position paper, we discuss some of these challenges and present a preliminary version of a curriculum packager addressing some of these issues.Our proposed solution, inspired by first-hand experience in these organisations as well as computing education research (CER), is based on a combination of micromaterials, study lenses and a companion mobile application.While our solution is designed for the specific context of small organisations providing vocational ICT training, it can also be applied to the broader domain of learning environments facing similar constraints.

The field of Computing Education Research (CER) produces important insights into learning and notable interventions, yet due to the research/practice divide these do not have the desired impact on learners or practitioners. Even within CER, Computing Education (CE) learning theories have limited influence on learning designs due to the theory/design divide, which is unfortunate given that the goal of CER is to impact learners and broaden access to computation.

Learning to program can often be a daunting task. The syntax of new programming languages might be intimidating and on top of that compilers are unforgiving. The combination of these things can be discouraging for new students. Recently there has been a push in the field of Computing Education Research to apply the Predict-Run-Investigate-Modify-Make (PRIMM) approach when teaching programming concepts. The aim is to push students to practise reading and investigating existing code before going on to write their own code, as a way to build self-confidence and boost their understanding. We present an educational game that aiming to help students practise their code prediction abilities. The game uses a simple story set in a medieval fantasy kingdom to create a welcoming, less intimidating environment. Users are presented with randomly generated pieces of JavaScript code that modify four core Boolean variables (shield, sword, cape and helmet). By making use of the state tables, students can manually walk through the code and predict the final result. A user ends the round by selecting the hero that matches the final state of all four variables. The presented heroes are further randomly assigned physical characteristics such as gender, skin tone and hair colour, promoting inclusivity and lowering the bar of entry via representation. The game allows users to indicate which language features they already know, thereby generating JavaScript code snippets that are understandable by a broad variety of students.

The learning of new knowledge and skills often requires previous knowledge, which can lead to some frustration if a teacher does not know a learner's exact knowledge and skills and therefore confronts them with exercises that are too difficult to solve.We present a solution to address this issue when teaching techniques and skills in the domain of table tennis, based on the concrete needs of trainers that we have investigated in a survey.We present a conceptual model for the representation of knowledge graphs as well as the level at which individual players already master parts of this knowledge graph.Our fine-grained model enables the automatic suggestion of optimal exercises in a player's so-called zone of proximal development, and our domain-specific application allows table tennis trainers to schedule their training sessions and exercises based on this information.In an initial evaluation of the presented solution for personalised learning environments, we received positive and promising feedback from trainers.We are currently investigating how our approach and conceptual model can be generalised to some more traditional educational settings and how the personalised learning environment might be further improved based on the expressive concepts of the presented model.

Hybrid positioning frameworks use various sensors and algorithms to enhance positioning through different types of fusion. The optimisation of the fusion process requires the testing of different algorithm parameters and optimal low-as well as high-level sensor fusion techniques. The presented OpenHPS open source hybrid positioning system is a modular framework managing individual nodes in a process network, which can be configured to support concrete positioning use cases or to adapt to specific technologies. This modularity allows developers to rapidly develop and optimise their positioning system while still providing them the flexibility to add their own algorithms. In this paper we discuss how a process network developed with OpenHPS can be used to realise a customisable indoor positioning solution with an offline and online stage, and how it can be adapted for high accuracy or low latency. For the demonstration and validation of our indoor positioning solution, we further compiled a publicly available dataset containing data from WLAN access points, BLE beacons as well as several trajectories that include IMU data.

In our position paper on a technology-enhanced smart learning environment, we propose the innovative combination of a knowledge graph representing what one has to learn and a learning path defining in which order things are going to be learned.In this way, we aim to identify students' weak spots or knowledge gaps in order to individually assist them in reaching their goals.Based on the performance of different learning paths, one might further identify the characteristics of a learning system that leads to successful students.In addition, by studying assessments and the different ways a particular problem can be solved, new methods for a multi-dimensional classification of assessments can be developed.The theoretical findings on learning paths in combination with the classification of assessments will inform the design and development of a smart learning environment.By combining a knowledge graph with different learning paths and the corresponding practical assessments we enable the creation of a smart learning tool.While the proposed approach can be applied to different educational domains and should lead to more effective learning environments fostering deep learning in schools as well as in professional settings, in this paper we focus on the domain of mathematics in primary and high schools as the main use case.

With the rapid increase of digital information we are dealing with in our daily work, we face significant document retrieval and discovery challenges. We present a novel document retrieval and discovery framework that addresses some of the limitations of existing solutions. An innovative aspect of our solution is the combination of implicit and explicit links between documents in the retrieval as well as in the visualisation process, in order to improve document retrieval and discovery. Our framework exploits implicit relationships between documents--defined by the similarity of their content as well as their metadata--and explicit links (hyperlinks) defined between documents based on a third-party link service. Further, the software framework can be extended with arbitrary third-party visualisations. Last but not least, our search query interface offers advanced features not available in most existing document retrieval systems.

School burnout refers to exhaustion at school combined with a cynical and detached attitude towards school. It often precedes Early School Leave (ESL), which has great impact on the socioeconomic situation of a person. Numerous local programs exist to deal with school burnout and prevent ESL, but they are all very labor-intensive. To come to a less labor-intensive solution, we proposed an ICT solution. The goal was to create an engaging software environment that youngsters would use voluntarily and that could re-engage them in learning. We developed a so-called playful digital environment, called TICKLE. It allows youngsters to collect digital cards by performing associated challenges in their surroundings. The challenges are small activities intended to reactivate the youngster for learning. Persuasive principles and gamification are used to motivate the youngsters to use the tool. In this paper, we discuss the design decisions that have been taken as well as the main components of the tool.

Over the last decade, research on adaptive and distributed user interfaces (DUIs) has increased. We also witness a growing number of Internet of Things (IoT) devices, allowing digital user interfaces (UIs) to communicate with physical objects and vice versa through so-called hybrid user interfaces. There exist various solutions to manage adaptive, distributed or hybrid UIs. However, none of them covers all three aspects and users have to deal with multiple applications and configurations when developing adaptive distributed hybrid user interfaces. We introduce the eSPACE reference framework and conceptual model unifying the domains of adaptive, distributed and hybrid interfaces. While our reference framework has been inspired by the CAMELEON reference framework, the conceptual model is based on the Resource-Selector-Link (RSL) hypermedia metamodel. We propose an approach for adaptive distributed hybrid user interfaces where users can author their user interfaces based on the different levels of abstraction introduced by our reference framework. We further present a use case illustrating the extensibility, flexibility and reusability offered by our unified approach and discuss potential future work.

The efficient management and retrieval of information via dedicated devices and data structures has been investigated since the early days of Vannevar Bush's seminal article As We May Think introducing the Memex. However, nowadays information is usually fragmented across different media types, devices as well as digital and physical environments, and we are often struggling to retrieve specific information. We discuss three main issues to be addressed when developing solutions for managing information in these co-called cross-media information spaces. First, we have a look at an extensible cross-media linking solution based on the resource-selector-link (RSL) hypermedia metamodel where information can be integrated across applications, devices as well as digital and physical information environments. We then outline some of the limitations of existing digital document formats which are often just a simulation of paper documents and their affordances on desktop computers, and discuss more flexible document representations for cross-media information spaces. Further, new forms of human-information interaction and cross-media user interfaces-including some recent work on dynamic data physicalisation-are discussed. A number of research artefacts are used to illustrate different aspects of the presented data-centric approach for cross-media information spaces and architectures. Last but not least, we provide an outlook on how the embedding of the presented concepts at the level of an operating system might ultimately lead to new possibilities for cross-media information management and innovative forms of human-information interaction.

Mainstream presentation tools such as Microsoft PowerPoint were originally built to mimic physical media like photographic slides and still exhibit the same characteristics. However, the state of the art in presentation tools shows that more recent solutions start to go beyond the classic presentation paradigms. For instance, presentations are becoming increasingly non-linear, content is quickly evolving beyond simple text and images and the way we author our presentations is becoming more collaborative. Nevertheless, existing presentation content models are often based on assumptions that do not apply to the current state of presentations any more, making them incompatible for some use cases and limiting the potential of end-user presentation solutions. In order to support state-of-the-art presentation functionality, we rethink the concept of a presentation and introduce a conceptual framework for presentation content. We then present a new content model for presentation solutions based on the Resource-Selector-Link (RSL) hypermedia metamodel. We further discuss an implementation of our model and show some example use cases. We conclude by outlining how design choices in the model address currently unmet needs with regards to extensibility, content reuse, collaboration, semantics, user access management, non-linearity, and context awareness, resulting in better support for the corresponding end-user functionality in presentation tools.

Audio recordings and the corresponding transcripts are often used as prosthetic memory (PM) after meetings and lectures. While current research is mainly developing novel features for prosthetic memory, less is known on how and why audio recordings and transcripts are used. We investigate how users interact with audio and transcripts as prosthetic memory, whether interaction strategies change over time, and analyse potential differences in accuracy and efficiency. In contrast to the subjective user perception, our results show that audio recordings and transcripts are equally efficient, but that transcripts are generally preferred due to their easily accessible contextual information. We further identified that prosthetic memory is not only used as a recall aid but frequently also consulted for verifying information that has been recalled from organic memory (OM). Our findings are summarised in a number of design implications for prosthetic memory solutions.

Nowadays, digital and paper documents are used simultaneously during daily tasks. While significant research has been carried out to support the re-finding of digital documents, less effort has been made to provide similar functionality for paper documents. In this paper, we present a solution that enables the design of cross-media Personal Information Management (PIM) user interfaces helping users in re-finding documents across digital and physical information spaces. We propose three main design requirements for the presented cross-media PIM user interfaces. Further, we illustrate how these design requirements have been applied in the development of three proof-of-concept applications and describe a software framework supporting the design of these interfaces. Finally, we discuss opportunities for future improvements of the presented cross-media PIM user interfaces.

Documents do often not exist in isolation but are implicitly or explicitly linked to parts of other documents. However, due to a multitude of proprietary document formats with rather simple link models, today's possibilities for creating hyperlinks between snippets of information in different document formats are limited. In previous work, we have presented a dynamically extensible cross-document link service overcoming the limitations of the simple link models supported by most existing document formats. Based on a plug-in mechanism, our link service enables the linking across different document types. In this paper, we assess the extensibility of our link service by integrating some document formats as well as third-party document viewers. We illustrate the flexibility of creating advanced hyperlinks across these document formats and viewers that cannot be realised with existing linking solutions or link models of existing document formats. A user study further investigates the user experience when creating and navigating cross-document hyperlinks.

In our daily lives we are witnessing a proliferation of digital devices including tablets, smartphones, digital cameras or wearable appliances. A major effort has been made to enable these devices to exchange information in intelligent spaces and collaborative settings. However, the arising technical challenges often manifest themselves to end users as limitations, inconsistencies or added complexity. A wide range of existing and emerging devices cannot be used with existing solutions for cross-device information exchange due to restrictions in terms of the supported communication protocols, hardware or media types. We present INFEX, a general and extensible framework for cross-device information exploration and exchange. While existing solutions often support a restricted set of devices and networking protocols, our unifying and extensible INFEX framework enables information exchange and exploration across arbitrary devices and also supports devices that cannot run custom software or do not offer their own I/O modalities. The plug-in based INFEX architecture allows developers to provide custom but consistent user interfaces for information exchange and exploration across a heterogeneous set of devices.

While popular digital maps support an unprecedented number of use cases, new reference map customization tools have been created for purposes for which those maps fall short. With the goal of informing the design of this new class of cartographic tools, we present the first study of naturalistic ("bespoke") map customization behavior. Through a mixed methods and mixed-media approach involving a survey, the analysis of a corpus of customized maps, and an interview with a power user, we find that bespoke map customization is a relatively common activity and identify frequent use cases as well as map customization strategies. We discuss these use cases and strategies in detail, and propose design implications for future customization tools, such as the use of templates for common use cases, adaptability for various customization styles and the support of multimedia interaction.

Context awareness plays an important role in recent smart environments and embedded interactions. In order to increase user satisfaction and acceptance, these context-aware solutions should be controllable by end users. Over the last few years we have therefore seen an emerging trend towards visual programming tools for context-aware applications based on simple "IF this THEN that" rules. However, existing solutions often do not support the simple reuse of the "this" part in order to define more sophisticated rules. Given that the desired level of control varies among individuals, we propose a unified multi-layered context modelling approach distinguishing between end users, expert users and programmers. Our Context Modelling Toolkit (CMT) consists of the necessary context modelling concepts and offers a rule-based context processing engine. We further illustrate how end users and expert users might interact with the CMT framework. Finally, we highlight some advantages of our Context Modelling Toolkit by discussing a number of use cases.

Over the last decade, we have witnessed an emergence of Personal Information Management (PIM) solutions. Despite the fact that paper documents still form a significant part of our daily working activities, existing PIM systems usually support the organisation and re-finding of digital documents only. While physical document tracking solutions such as RFID- or computer vision-based systems are recently gaining some attention, they usually focus on the paper document tracking and offer limited support for re-finding activities. We present PimVis, a solution for exploring and re-finding digital and paper documents in so-called cross-media information spaces. The PimVis user interface enables a unified organisation of digital and paper documents through the creation of bidirectional links between the digital and physical information space. The presented personal cross-media information management solution further supports the extension with alternative document tracking techniques as well as augmented reality solutions. A formative PimVis evaluation revealed the high potential of fully integrated cross-media PIM solutions.

Existing research in the field of information visualisation has shown that interactive data exploration and storytelling can significantly improve the extraction and transfer of knowledge from raw data.Established visualisation techniques help viewers to strengthen their mental model and improve the understanding of the underlying data.However, these techniques are not yet manifested in slide decks created by existing presentation tools which offer little to no support beyond static charts for transferring knowledge.Based on a detailed analysis of interactive and narrative data visualisation solutions and the shortcomings of existing presentation tools, we derived a set of requirements for interactive information visualisation in presentation tools.The presented prototype of a presentation tool for interactive data visualisation addresses these requirements and has been implemented as a plug-in for the MindXpres presentation platform.Our approach for rich presentation-based data exploration and storytelling enables the presenter to predefine a series of interactive views as support for their oral narrative, but also allows them to freely explore the data during presentation time.

The teaching of programming concepts and algorithms very much depends on the mental models developed by scholars when learning how to program. There is a rich body of research on how to best teach programming. Nevertheless, many instructors follow a presentation-based approach where existing slideware such as PowerPoint or Keynote is used to show a sequential series of slides with static pieces of source code. Such a presentation-based approach based on existing slideware tools might not be optimal for the authoring as well as the delivery of programming courses. We outline how presentation-based eduction in programming can be improved by paying attention to existing research on how to best teach programming. We derive a number of requirements for more efficient source code visualisation in presentation tools and present an architecture as well as an extensible prototype for enhanced presentation-based teaching of programming. The presented interactive source code visualisation plug-in for the MindXpres presentation tool can be seen as a step towards enhancing existing slideware in order to achieve a more efficient and interactive teaching of programming concepts and algorithms. The ultimate goal of the presented approach is to present source code in a way that reinforces a user’s mental model and thereby increases the knowledge transfer of presentations delivered in programming courses.

We present ArtVis, an application combining advanced visualisation techniques and tangible interaction to explore a large digital collection of almost 28 000 European artworks managed by the Web Gallery of Art. In order to get new insights by exploring, analysing and browsing the artworks, our graphical ArtVis user interface offers three complementary but synchronised visualisation components. We further developed a tangible ArtVis user interface for the playful exploration and seamless integration of the digital artwork collection with physical artefacts. A formative evaluation of the ArtVis prototype revealed that users are able to answer relatively difficult questions as well as get some new insights based on the vast amount of data. A second user evaluation of the tangible ArtVis interface has shown that this sort of physical interaction attracts users and stimulates them to further explore the digital artwork collection.

There exist various audience response systems (ARS) and other standalone solutions to enrich presentations with supplementary active learning content. We propose a unified communication platform for enriching and enhancing presentations with active learning components where third party solutions can easily be integrated via data adapters, activity plug-ins as well as a multi-directional communication channel based on a publish-subscribe pattern. We present how the MindXpres presentation platform has been extended with a communication module and discuss a number of activity plug-ins that have been realised. The presented unified communication platform in combination with the MindXpres presentation tool represents an ideal platform for the investigation and rapid prototyping of new forms of presentations that support enhanced forms of interaction between a presenter and their audience.

Water Ball Z is a novel interactive two-player water game that allows kids and young adults to "fight" in a virtual world with actual physical feedback. The body movement of a player is captured via an RGB-D sensor and analysed by a 3D gesture recognition engine. In order to enable tactile feedback without the need for wearable devices, a number of water nozzles are positioned around each user's play area. The idea is to translate the input gesture of one player to the corresponding water spray hitting the other player. Besides severely reducing the risk of injury in a fight, Water Ball Z engages people in a real and fun experience where a hit is physically manifested via a water spray. Furthermore, power up moves and a live scoreboard extension bring the virtual world of Dragon Ball Z and Mortal Kombat cartoons into real (augmented) life. In addition to a detailed description of the physically augmented game, we discuss two new control parameters for mapping gesture input to haptic output which, to the best of our knowledge, are not present in existing 3D gesture recognition approaches.

With the emergence of smart TVs, set-top boxes and public information screens over the last few years, there is an increasing demand to no longer use these appliances only for passive output. These devices can also be used to do text-based web search as well as other tasks which require some form of text input. However, the design of text entry interfaces for efficient input on such appliances represents a major challenge. With current virtual keyboard solutions we only achieve an average text input rate of 5.79 words per minute (WPM) while the average typing speed on a traditional keyboard is 38 WPM. Furthermore, so-called controller-free appliances such as Samsung's Smart TV or Microsoft's Xbox Kinect result in even lower average text input rates. We present SpeeG2, a multimodal text entry solution combining speech recognition with gesture-based error correction. Four innovative prototypes for the efficient controller-free text entry have been developed and evaluated. A quantitative evaluation of our SpeeG2 text entry solution revealed that the best of our four prototypes achieves an average input rate of 21.04 WPM (without errors), outperforming current state-of-the-art solutions for controller-free text input.

The advent of advanced mobile devices in combination with new interaction modalities and methods for the tracking of contextual information, opens new possibilities in the field of context-aware user interface adaptation. One particular research direction is the automatic context-aware adaptation of input modalities in multimodal mobile interfaces. We present existing adaptive multimodal mobile input solutions and position them within closely related research fields. Based on a detailed analysis of the state of the art, we propose eight design guidelines for adaptive multimodal mobile input solutions. The use of these guidelines is further illustrated through the design and development of an adaptive multimodal calendar application.

We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data sets. ViDaX enables the extraction of information from RDF data sources and offers functionality for the analysis of various data characteristics as well as the exploration of the corresponding ontology graph structure. In addition to some basic data mining features, our interactive semantic data visualisation and exploration tool offers various types of visualisations based on the type of data. In contrast to existing semantic data visualisation solutions, ViDaX also offers non-expert users the possibility to explore semantic data based on powerful automatic visualisation and interaction techniques without the need for any low-level programming. To illustrate some of ViDaX's functionality, we present a use case based on semantic data retrieved from DBpedia, a semantic version of the well-known Wikipedia online encyclopedia, which forms a major component of the emerging linked data initiative.

We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking. A modified version of the zoomable Dasher interface combines the input from Sphinx and the Kinect sensor. In contrast to existing speech error correction solutions with a clear distinction between a detection and correction phase, our innovative SpeeG text input system enables continuous real-time error correction. An evaluation of the SpeeG prototype has revealed that low error rates for a text input speed of about six words per minute can be achieved after a minimal learning phase. Moreover, in a user study SpeeG has been perceived as the fastest of all evaluated user interfaces and therefore represents a promising candidate for future controller-free text input.

In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic inference-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.

Pen and paper support the rapid production of sketches. However, the paper interface is not always optimal for collaborative sketching as seen in brainstorming sessions where multiple parties would often like to communicate and participate in the sketching synchronously. Novel interactive paper solutions may provide the answer by bridging the paper-digital divide and allowing users to sketch on paper simultaneously while capturing the actions digitally. We present an analysis of collaborative sketching activities in working environments with remote participation. After highlighting the importance of paper for natural interaction in these settings, we introduce PaperSketch, an interactive paper-digital tool for collaborative remote sketching. We discuss the collaborative development of ideas based on the prototype and outline how important feedback issues have been addressed by utilising spatial constraints and multimodal features.

Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple's iPhone or Microsoft's Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hard-coded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

Paper notes are still widely used during meetings for the capture and review of information created in meetings. However, personal notes are limited in terms of providing an overview of collaborative work practices and reflecting the evolution of data along successive meeting phases, especially taki

The first steps towards bridging the paper-digital divide have been achieved with the development of a range of technologies that allow printed documents to be linked to digital content and services. However, the static nature of paper and limited structural information encoded in classical paginated formats make it difficult to map between parts of a printed instance of a document and logical elements of a digital instance of the same document, especially taking document revisions into account. We present a solution to this problem based on a model that combines metadata of the digital and printed instances to enable a seamless mapping between digital documents and their physical counterparts on paper. We also describe how the model was used to develop iDoc, a framework that supports the authoring and publishing of interactive paper documents.

With the emergence of digital pen and paper interfaces, there is a need for gesture recognition tools for digital pen input. While there exists a variety of gesture recognition frameworks, none of them addresses the issues of support- ing application developers as well as the designers of new recognition algorithms and, at the same time, can be inte- grated with new forms of input devices such as digital pens. We introduce iGesture, a Java-based gesture recognition framework focusing on extensibility and cross-application reusability by providing an integrated solution that includes tools for gesture recognition as well as the creation and management of gesture sets for the evaluation and optimi- sation of new or existing gesture recognition algorithms. In addition to traditional screen-based interaction, iGesture provides a digital pen and paper interface.

Recent developments in digital pen and paper solutions enable, not only the digital capture of handwriting, but also paper to be used as an interactive medium that links to digital information and services. We present a tool that builds on technologies for interactive paper to enable PowerPoint presentations to be controlled from printed slide handouts. Furthermore, slides can be easily annotated during presentations by simply drawing on the printed version of the slide. As well as discussing the advantages of such a paper-based interface and initial findings on its use, we describe how we were also able to exploit it to provide a general prototyping tool for interactive paper applications.

Citations form the basis for a web of scientific publications. Search engines, embedded hyperlinks and digital libraries all simplify the task of finding publications of interest on the web and navigating to cited publications or web sites. However the actual reading of publications often takes place on paper and frequently on the move. We present a system Print-n-Link that uses technologies for interactive paper to enhance the reading process by enabling users to access digital information and/or searches for cited documents from a printed version of a publication using a digital pen for interaction. A special virtual printer driver automatically generates links from paper to digital services during the printing process based on an analysis of PDF documents. Depending on the user setting and interaction gesture, the system may retrieve metadata about the citation and inform the user through an audio channel or directly display the cited document on the user's screen.

Our demonstration is a paper-based interactive guide for visitors to the world's largest international arts festival that was developed as part of a project investigating new forms of context-aware information delivery and interaction in mobile environments. Information stored in a database is accessed from a set of interactive paper documents, including a printed festival brochure, a city map and a bookmark. Active areas are defined within the documents and selection of these using a special digital pen causes the corresponding query request along with context data to be sent to a festival application database and the response is returned to the visitor in the form of generated speech output. In addition to paper-based information browsing and transactions such as ticket booking, the digital pen can also be applied for data capture of event ratings and handwritten comments on events. The system integrates three main database components - a cross-media information platform, a content management framework for multi-channel context-aware publishing of data and the festival application database.

Paper persists as an important medium for documents and this has motivated the development of new technologies for interactive paper that enable actions on paper to be linked to digital actions. A major issue that remains is how to integrate these technologies into the document life cycle and, specifically, how to facilitate the authoring of links between printed documents and digital documents and services. We describe how we have extended a general web publishing framework to support the production of interactive paper documents, thereby integrating paper as a new web channel in a platform for multi-channel access.

Interaction design is a major issue for mobile information systems in terms of not only the choice of input-output channels and presentation of information, but also the application of context-awareness. To support experimentation with these factors, we have developed a platform that supports the rapid prototyping of multi-channel, multi-modal, context-aware applications. The paper presents the main components of the platform and describes how it was used to develop a tourist information system for an international arts festival where interaction was based on a combination of speech input-output and interactive paper.

Despite the wide-ranging recognition that paper remains a pervasive resource for human conduct and collaboration, there has been uncertain progress in developing technologies to bridge the paper-digital divide. In this essay we discuss the design of a technology that interweaves developments in new materials, electronics and software, and seeks to provide a cheap and accessible solution to creating new affinities between digital content, in whatever form, and ordinary paper. The technology and its design draws from a broad range of field studies, including research in classrooms and museums. These delineate the requirements and considerations that inform solutions to enhancing paper whilst preserving its integrity. The paper also discusses a naturalistic experiment, an evaluation in a museum, where we assessed the technology and the solution. We also chart the progressive development of this solution and the ways in which seemingly simple actions and issues became reconstituted as highly complex technical and analytic problems.

We present a general model and information server for the digital annotation of printed documents. The resulting annotation framework supports both informal and structured annotations as well as context-dependent services. A demonstrator application for mammography that features both enhanced writing and reading activities is described.

We present a general Web-based information infrastructure capable of supporting the rapid development of highly-interactive information environments that cater for widely varying requirements across application domains and all forms of fixed and mobile client devices. In particular, we describe how this infrastructure has been extended to support digitally augmented paper through a special transformation component that can map active areas of document pages to information objects so that user and context-dependent interaction can be supported. Our infrastructure is sufficiently general and flexible to adapt to, not only emerging and even unanticipated technologies in the area of interactive paper, but also the rapidly expanding interaction sphere of hypermedia.