This paper describes an approach to algorithmic music composition that takes narrative structures as input, allowing composers to create music directly from narrative elements. Creating narrative development in music remains a challenging task in algorithmic composition. Our system addresses this by combining leitmotifs to represent characters, generative grammars for harmonic coherence, and evolutionary algorithms to align musical tension with narrative progression. The system operates at different scales, from overall plot structure to individual motifs, enabling both autonomous composition and co-creation with varying degrees of user control. Evaluation with compositions based on tales demonstrated the system's ability to compose music that supports narrative listening and aligns with its source narratives, while being perceived as familiar and enjoyable.

The online shopping industry benefits from the usage of virtual assistants that are able to provide a 24/7 way of communication. This is normally done through the use of text based interactions. In order to improve the user experience and provide a more engaging user-to-customer interaction, we present a virtual shopping environment that emulates the physical store, providing the user with an experience similar to buying products in the actual store. This allows users to interact with the store items and have a full conversation with a virtual shopping assistant. This is all done through a virtual store where the user is able to walk, try-on and see the displayed items, while having a conversation with the store's assistant.

This paper presents and discusses the integration of human pose estimation techniques into an existing web-based multimodal video annotation tool, applying it to the sports context, where basketball is the first case study. The relevance of video analysis extends across many fields of work (e.g., professional sports, education). In sports, systematic, detailed analysis using videos of players and teams is vital to evaluate many aspects of both training and competition. MotionNotes annotation tool now combines human pose and motion information with existing traditional annotation mechanisms (e.g., text and drawings annotations), allowing users to add further details to their annotation work. The paper reports feedback from a pilot study based on a participatory workshop involving people with relevant competitive experience in basketball. Based on this use case feedback, we conclude with an outlook of future iterations for our video annotation tool.

This paper presents and discusses the introduction of 3D functionalities for an existing web-based multimodal video annotation tool. Over the past years, we have developed a multimodal web video annotation tool that now combines 3D models and 360º content with more traditional annotation types (e.g., text, drawings, images), offering users the possibility of adding extra information in their annotation work. We show how 3D models augment the annotation work and add advantages like viewing or exploring objects in detail and from different angles. The paper reports detailed feedback from a pilot study in form of a workshop with traditional dance experts to whom these new features were presented. We conclude with an outlook of future iterations of the video annotator based on the experts’ feedback.

We live surrounded by computing devices, but we are not yet combining them into a natural unified user experience. Therefore, we have designed and implemented a framework to generalize and support the development of multi-device applications, which distribute their User Interfaces (UI) across co-located devices. The framework is composed of: authentication and authorization services; a broker to sync information across application instances; background services to gather the capabilities of the devices; an integration module with an indoor positioning system to determine when to associate devices. It provides a library to determine the components to be displayed based on UI requirements and the capabilities of the available devices. It also includes elements to manage the distribution of UI components and application states that can be shared to enable collaboration. We present an application prototype and a user study which assesses how multi-device applications are perceived by users. The results are very positive, indicating that we should keep working on our research since the users received very well the prototype and the concept, which also validates the support provided by the framework.

Creativity and inspiration for problem-solving are critical skills in a group-based learning environment. Communication procedures have seen continuous adjustments over the years, with increased multimedia elements usage like videos to provide superior audience impact. Annotations are a valuable approach for remembering, reflecting, reasoning, and sharing thoughts on the learning process. However, it is hard to control playback flow and add potential notes during video presentations, such as in a classroom context. Teachers often need to move around the classroom to interact with the students, which leads to situations where they are physically far from the computer. Therefore, we developed a multimodal web video annotation tool that combines a voice interaction module with manual annotation capabilities for more intelligent natural interactions towards ubiquitous environments. We observed current video annotation practices and created a new set of principles to guide our research work. Natural language enables users to express their intended actions while interacting with the web video player for annotation purposes. We have developed a customized set of natural language expressions that map the user speech to specific software operations through studying and integrating new artificial intelligence techniques. Finally, the paper presents positive results gathered from a user study conducted to evaluate our solution.

We live surrounded by the most varied computing devices, which may give us the opportunity to combine them to form a unified and richer user experience. Considering this opportunity, we created the UnaxY Framework to support the development of applications with UI components distributed by co-located devices. This paper is focused on a feasibility study based on two prototype applications created using UnaxY. We performed user studies to evaluate concepts associated to this type of applications and the framework they were based on. We had a special interest in assessing how managing the application state and collaborating across devices would be perceived and received by users. The results are positive and clearly indicate that we should continue developing solutions that support a generalized implementation of applications with the user interaction spanning multiple devices.

In this paper we present a virtual reality application, where multiple users can observe and interact with a portion of geo-referenced terrain where a real wildfire took place. The application presents a layout with two maps, one is a three-dimensional view with terrain elevation and the other is a conventional two-dimensional view. The VR users can control different layers (roads, waterways, etc), control the wildfire’s playback, command vehicles to change positions and paint the terrain conveying information to one-another. This work explores how users interact with map visualizations and plan for a crisis management scenario within a virtual environment.

In an emergency situation, each response agent must act quickly and accurately. The support of a mobile device that can provide an appropriate insight of the surroundings and that allows users to exchange information with the other members of the emergency teams, can prevent harm and even save many lives. This paper presents a mobile application that integrates a georeferenced system with augmented reality techniques, in order to serve the needs of the operatives in emergency situations. The work intends to introduce solutions which optimize the performance with which the user takes advantage of the mobile application, such as the organization of the data flow that is displayed and the augmentation of the surrounding area. User studies were conducted with members of the National Navy. The results were positive although there are still some aspects that should be improved.

We currently live surrounded by many different computing devices. Therefore, it is important to take better advantage of those devices by coming up with smart ways of integrating and combining them. We have been exploring the possibility of building applications that present user interfaces pervasively distributed across different co-located devices. We designed the YanuX framework, which generalizes and supports the development of this new type of applications. A key issue is the automatic distribution of user interface (UI) components among co-located devices. We created the tools set needed to describe the capabilities of each of the devices present in the environment and the requirements of each of the applications' components as configured by the developers. Restrictions of the components should match the capabilities of the devices leading to a UI component distribution decision that should reflect developers' intentions and expectations of the users. Besides detailing YanuX's components, the paper also presents YouTube Viewer as a proof-of-concept application based on YanuX. The application was also used in a user study to evaluate the concept and the experience supported by the framework. The results presented here are positive and very promising.

Although public displays installations are easy to attract the attention of passers-by to specific content, they present an issue in targeting the right audience. We envision public displays installations working as platforms in which different interactive applications run with highly personalised content based on personal characteristics and interests of a person attending the public displays. Mobile devices can act as an important interaction tool that complements the public screen, providing a feeling of privacy, but they can also be used to get users' preferences in order to obtain their user profiles. This paper is focused on demonstrating how personalisation can be designed and applied to improve the user experience in public display systems, integrating mobile interaction to potentiate the approach. It describes a public display experiment deployed at a university campus and presents a user study composed of two phases: the system without personalisation and the system with personalisation.

In this demonstration a test bench is presented to test and exhibit the operation of a Switched Capacitor DC-DC converter, designed in 130nm CMOS technology. The converter generates a stable 0.9 V from an input voltage range of 1.2 to 2.3 V, that is provided by a supercapacitor power supply charged by a PV cell. The demonstration allows to select between different loads to change the output power level of the converter (up to 1 mW of output power). The voltages and currents of the input and output of the converter are measured in real time in order to create a real time graphical visualization of the several parameters of the converter.

Together with materials researchers at CENIMAT, a mobile application was developed in order to perform colorimetric analysis of a glucose paper biosensor to be used for the diagnostic of one of the most incident diseases in the world, Diabetes, using Image Processing Techniques and Machine Learning algorithms. This application was designed based on a User Story and various scenarios describing how users can interact with it. Testing showed that the app is capable of distinguishing high glucose concentrations from low concentrations, while a usability questionnaire made to potential users showed that they thought the application was easy to use, being capable of becoming a portable and fast mean of diagnostic.

We live surrounded by computing devices, but applications are mostly confined to run on a single device. It should be possible to make better use of the multiple devices around us by coming up with ways of integrating and combining them, while leveraging the specific strengths of some devices and minimizing the individual weaknesses of others. We want to explore the possibility of building applications that have their user interface seamlessly distributed across co-located devices. We created the YanuX Framework to provide the guidelines and tools needed by developers to build those applications. This paper presents the framework and its architecture, which is outlined along with the description of its components. We end by presenting an early evaluation of YanuX and by discussing the status of our work along with directions for further research.

With the widespread use of social networks, sharing emotions has become easy and accessible to anyone with a smartphone or a computer. In this paper, we present a system capable of generating automatic highlight videos of sports match TV broadcasts based on the emotions shared by the spectators during the match, the audio, the analysis of the movement and manual annotations (when available). Our system also allows for the user to query the video to extract specific clips, such as dangerous attacking plays of a certain team. Our preliminary results were encouraging, showing that the creation of these summaries can be successfully done in very short time.

This paper presents a working prototype of PaintMyEmotions, an interactive self-reflection instrument that allows individuals to assess, express and monitor their affective states in a fun and engaging way. Despite the ever-increasing popularity of eHealth technologies, online interventions and research studies often present severe problems of attrition (low adherence to the program and/or high drop-out rates). To increase the chances for success of our project and subsequent research, we decided to test early and often, involving users from the beginning, in an iterative design cycle. In this article, we describe the process and the results of an early exploratory user study, where we analyzed the performance and usability of a working prototype, studied its potential, and explored aspects like engagement and joy of use.

Tablet computers provide a natural support for digital annotation associated to any type of media, including live video. However, annotations of live video require real-time motion tracking, maintaining the association between annotations and moving objects, as well as interfaces that facilitate the annotation task of a live event. This work presents the initial users' feedback on a video annotator that includes two real-time trackers, Kinect and TLD, and on two annotation methods that aim to help annotating live video, "Hold and Overlay" and "Hold and Speed Up".

There is a need to deliver personalized experiences in terms of synchronization between television broadcasts and second screen applications to avoid frustrating experiences. However, within a second screen gaming scenario based on TV broadcasts, if players synchronize with a delay higher than the real delay of the TV broadcast, they will have an unfair advantage over the others. To solve this issue, we introduce a software-based solution called Sync2FairPlay that can be integrated in any second screen gaming application. It provides a universal synchronization mechanism that helps users to achieve a pleasant second screen experience, while automatically dealing with cheating to provide a fair gaming experience. We also describe the user tests performed to validate our idea and the different components of Sync2FairPlay.

This paper presents Haunted House, a game where a mobile device projects the interior of a virtual haunted house over a room. The users explore the house discovering its secrets. Pico projectors are becoming popular and smartphones are starting to be designed with this functionality. When projecting we can overcome many problems of the mobile device small display, giving room to new and more interesting applications. Even when a mobile device does not have the projecting functionality, it is possible to attach a pico-projector to it. By having this technology available for everybody with a smartphone, new forms of interaction will emerge. We aim at exploring the use of pico-projectors to create interactive experiences by using an infrastructure, adaptable to a space that uses existing video camera networks to support these experiences. We intend to develop new forms of interaction and social experiences using mobile devices and pico-projectors.

Image matching algorithms are used in image search, classification and retrieval but are also useful to show how urban structures evolve over time. Images have the power to illustrate and evoke past events and can be used to show the evolution of structures such as buildings and other elements present in the urban landscape. The paper describes a process and a tool to provide a chronological journey through time, given a set of photographs from different time periods. The developed tool provides the ability to generate visualizations of a geographic location, given a set of related images, taken at different periods in time. It automatically processes comparisons of images and establishes relationships between them. It also offers a semi-automated method to define relationships between parts of images.

This paper proposes a concept of exploring virtual worlds using a pico projector. To demonstrate this concept, a game, called Gone Fishing, was created on top of the infrastructure vuSpot. Gone Fishing is a game where a mobile device projects a virtual seabed over a room. Each user owns an aquarium and fish, flora and decorations can be collected and exchanged. Fish must be pursued to be captured. Pico projectors are becoming popular and more applications are needed. We intend to create applications that use pico projectors and mobile devices to play games and explore spaces using vuSpot, an infrastructure adaptable to a space that uses existing video camera networks to support interactive experiences. With this game we aim at exploring the application of pico projectors in gaming, investigating new forms of interaction and social experiences using mobile devices and pico projectors.

The paper describes a new paradigm for geometry learning applicable to other areas where children engage with the real world to find the meaning of abstract concepts. The target of the developments is a game or game series where image processing is used for geometric concepts learning. The learner has to match the concepts presented in the game challenges with its real world representations. Ubiquitous and pervasive computational devices become flexible tools that shape the learning experience as users engage in challenging, experiential activities with immediate feedback. The game concept was developed for mobile devices with an image processing library optimized for real time performance in interactive systems. The library and game features were tested at a school with children with very encouraging results.

This paper proposes a system, created with the goal of simplifying the design and development of interactive experiences to help explore a space. It has four components: XploreDescription, XploreBuilder, urSpace and an infrastructure.

We propose a platform with the goal of helping developers when they need to apply personalization to ubiquitous computing applications, ranging from simple mobile apps to rich multimodal systems. The platform’s core is a general personalization model, which provides the orientation and required tool

Summary form only given, as follows. The talk presents how different systems use image and graphics, as a part of more complex systems using diverse interaction techniques and sensors. The systems and applications that will be described belong to a range of fields such as cultural heritage and contemporary art and dance. Image analysis and other sensors are combined to produce responsive environments and natural interfaces in several domains. The talk also describes the process of setting up, specifying and developing interdisciplinary projects and the methodologies that are used.

This paper discusses the evaluation of a video annotator that supports multimodal annotation and is applied to contemporary dance as a creation tool. The Creation-Tool was conceived and designed to assist the creative processes of choreographers and dance performers, functioning as a digital notebook for personal annotations. The prototype, developed for Tablet PCs, allows video annotation in real-time, using a live video stream, or postevent, using a pre-recorded video stream. The tool also allows different video annotation modalities, such as annotation marks, text, audio, ink strokes and hyperlinks. In addition, the system enables different modes of annotation and video visualization. The development followed an iterative design process involving two choreographers, and a usability study was carried out, involving international dance performers participating in a contemporary dance "residence - workshop".

This paper proposes a framework to support interaction with virtual objects integrated in a real life scenario. In the proposed interaction concept, the user can reshape or re-design a real space using virtual objects using several pictures of the desired space. The images are analyzed for known features such as surfaces, floor or room orientation. Using these elements, it is possible to devise an augmented reality system where the user can add virtual objects to the scenario that react to its content. These are magnetic objects because they attach itself to elements on the scene such as floor or walls. This approach liberates the user from moving objects in three-dimensional space. Instead they just push around objects in a more natural interface. The paper studies the impact of adding these magnetism effects in the interaction with an augmented reality-editing interface.

Many exhibitions that occurred in the past in art museums left a large amount of records (pictures, videos and texts), which are distributed over several documents or places. This paper proposes a multimedia system to reconstruct the heritage and space from such exhibitions. The virtual reconstruction is based on collecting and recovering the documents and making them accessible using an integrated interface. The evolution of image and photograph visualization allows the virtual reconstruction of physical spaces, offering a new dimension of interaction with the user. Through image analysis techniques, the spatial relation between each photo in the archive is accessed allowing the semi-automatic construction of a virtual artwork exhibition.

One of the recurrent major environmental problems is the high energy consumption. It has been challenging to find methods of persuading people to have better habits on energy usage. We are proposing the use of personalization to enhance a mobile and pervasive-based gaming approach to better foster domestic energy awareness close to people. The game is based on real-time domestic energy consumption data and presents persuasive feedback information, beyond the competitive feeling of the game. The personalization mechanisms are provided by an external web service that is designed to serve different applications, even if from different systems or domains.

In this work we present an interactive prototype for an interface that supports the interaction with virtual objects integrated in a real life scenario. The user can reshape or re-design a real space with virtual objects using several pictures of the desired space. The images are analyzed for known features such as surfaces, edges, floor location and room orientation. Using these elements, it is possible to devise an augmented reality system where the user can add virtual objects to the scenario. The smart interface attaches the objects to the scene elements (e.g., floor) automatically. The current demo loads images from user files or takes a snapshot directly from the camera. The high-level features are automatically detected, but can be manually adjusted. A use-case example of an augmented reality application is presented.

This demonstration presents work done in the TimeMachine project with the goal to explore patterns in everyday life using a mobile application. This mobile application captures GPS data and uses this information to provide a subjective map of the user behavior using several visualizations. The project resorts to a rigorous data capture and processing approach with the goal of providing meaningful visualizations to the users. The visualizations depart from common map based representations as time is the main variable that is considered. Routines are represented through time based variables such as duration and frequency (for a given place or set of places) and a glimpse of the future is also provided considering what happened in the past. The demonstration presents the current version of the mobile application and desktop based tools used to extract information or prototype additional visualizations.

When using a tablet computer, sketching is a natural approach for users to annotate video scenes. However, when these annotations are done in real-time and overlaid in the video, their context can be lost due to changes in the scene being annotated. We propose an approach towards maintaining the annotations' context, by using object tracking to create anchors onto which further annotations can be attached. To this end, the annotator is capable of using different tracking methods, including a Kinect sensor and/or the TLD object tracking algorithm.

This paper proposes an application for real-time object recognition using mobile devices that runs locally without the need to communicate with a server. Both object detection and identification algorithms are performed by the mobile device avoiding the communication with the server. The paper presents results that show the effectiveness of the proposal.

This paper presents a video annotator that supports multimodal annotation and is applied to contemporary dance as a creation tool. The Creation-Tool was conceived and designed to assist the creative processes of choreographers, working as a digital notebook for personal annotations. The prototype, developed for Tablet PCs, is a real-time multimodal video annotator based on keyboard, pen and voice inputs. In addition, a remote control for mobile devices was added to the system. Two types of annotations were defined: annotation marks and regular annotations. The annotations marks are defined by a keyword and an icon, in contrast to regular annotations that do not have a pre-defined structure. Motion tracking defines the dynamic behavior of the annotations and voice input complements the other modalities.

[email protected] is an interactive installation that provides a bond between urban beings and the forest. Participants hug a real tree, triggering their registration in the virtual world for further interaction.

We all had the problem of forgetting about what we just read a few sentences before. This comes from the problem of attention and is more common with children and elderly. People feel either bored or distracted by something more interesting. This paper proposes an application to help people reading news by illustrating the news story. The application provides mechanisms to (1) select the best illustration for each scene and (2) to select the set of illustrations to improve the story sequence. The application proposed in this technical demo aims at improving the user's attention when reading news articles. The application implements several information processing techniques to generate an audio-visual presentation of the text news article.

RTiVISS is an exploratory project that proposes to investigate innovative concepts and design methods regarding environmental and sustainability issues. It is concerned with natural resources, specially forests, and their preservation, through critical research and experimental artistic approaches. The project proposes multiplatform access to real-time networked video and allows users to adopt selected forests under surveillance. The interactive system feeds a whole community that establishes connections by sharing "the emotion of real-time" and the challenge of uncertainty, while remotely monitoring natural environments for forests protection. This achievement enables artistic explorations with digital media in interactive installations that engage the audience senses in unconventional ways.

Face annotation is an important concept for personal memories retrieval. Using automatic face recognition to annotate and find people in those memories provides an improvement of a personal memories management system. However, its results are limited by the uncontrolled conditions inherent to personal memories. In this paper, we propose a face recognition method to address these limitations which includes techniques of skin detection and pose estimation. It is also proposed a gender classification method to provide more information about the detected faces. Experimental results are presented that show that these methods improve the overall performance of face recognition and gender classification in personal memories.

This paper presents the Memoria mobile interface, an application to share and access personal memories when visiting historical sites, museums or other points of interest. With the proposed interface people can navigate the memory space of the place they are visiting and, using their camera-phones or Personal Digital Assistants (PDA), view what has interested them or other people in past occasions. The system consists of a retrieval engine and a mobile user interface that allows capture and automatic annotation of images. Experimental results are presented to show the performance of the retrieval mechanisms and the usability of the interface.

In this paper we present a wearable user interface for ubiquitous gaming activities, discussing the design choices that led to the implementation of a first, fully functional prototype. The Gauntlet takes the form of an unobtrusive arm piece that allows gestures and manipulation of real objects to be included as game-play elements. When paired to a mobile device it can be used ubiquitously, without need of a location-dependent hardware framework. Public demonstration of the prototype by means of an interactive installation has allowed us to conduct observations of its usage and achieve preliminary conclusions on our approach.

This project develops tools to manage personal memories that include a multimedia retrieval system and user interfaces for different devices. This paper and demonstration presents the mobile interface which allows browsing, retrieving, and taking pictures that are automatically annotated with GPS data and audio information. The multimedia retrieval system uses multimodal information: visual content, GPS metadata and audio information. The interface was evaluated in a cultural heritage site.

This paper describes the current status of the VideoStore project. This project has the goal of developing a Web based system to store, transmit, search, and share the contents of courses captured in video. The current version includes a stable version of the architecture, and a set of advanced interfaces. The application synchronizes video streaming and slideshow presentations, and currently it provides several video based interfaces, and a video annotation system. The paper describes the system features, the architecture and the interfaces developed.

This paper describes HyperMem, a system to store and replay user experiences in mixed reality worlds. The experience is stored as a set of hypermedia nodes and links, with the information that was displayed along with the video of the real world that was navigated. It uses a generic hypermedia model implemented as software components developed to handle mixed reality environments. This approach is applied to an augmented museum, that includes information and gaming activities in the physical and virtual 3D worlds. The paper presents the goals of the system, the underlying hypermedia model, the type of worlds that we are considering, and the architecture and tools for replaying and repurposing stored information.

This paper describes a video annotation tool based on a new and flexible model, that gives several perspectives over the same video content. The model was designed in a way that allows having multiple views over the same video data, enabling users with different requirements to have the most appropriate interface. These views, video-lenses, highlight a specific aspect of the video content that is being annotated. Annotations are made using a timeline based interface with multiple tracks, where each track corresponds to a given video-lens. The format used to store and exchange the information is the MPEG-7 standard. The annotation tool (VAnnotator) is being developed in the scope of Vizard, an ambitious project that aims to define a new paradigm for video navigation, annotation, editing and retrieval. The Vizard project includes users, both from the production/archiving area and from the consumer electronics area, that help to define and validate the annotation requirements and functionality.

This paper describes the principles and a model for adding content and structure to existing video materials, based on annotation.Annotations in printed media promote active reading and, in a similar way, annotations in video promote active watching.The principles and model are illustrated by a prototype system for video annotation and browsing, named AntV (Annotations in Video).

The new media types used in advance user interfaces and interactive systems introduce time as a significant variable. This paper addresses the architectural support and programming tools that should be provided to the programmer to manage the time dependencies. The approach considers that the basic models and programming paradigms adopted in the manipulation and management of time should be isomorphic with the spatial models used in existing graphical user interfaces.