As a field, haptics encompasses a wide and diverse array of technologies, modalities, and applications, which makes it challenging to understand, classify, and compare comprehensively. In this paper, we introduce Haptree: a design space framework developed based on a review of existing literature and building on top of existing taxonomies. Unlike other haptic frameworks that focus on specific aspect of haptics, Haptree is intended to show the big picture of haptic experiences, connecting together aspects of haptic experience that used to be isolated in other frameworks. Haptree maps haptic experiences in five different stages, from hardware characteristics to end-user applications. It provides practical guidelines for designing new haptic experiences by referencing existing approaches and use cases, or pointing to a current gap in the literature. Furthermore, Haptree is intended to be a community-driven tool, designed for iterative refinement and contribution by the haptic research community. Finally, we outline future steps, including validating and enhancing Haptree through community engagement and empirical studies.

Since its early development in the 1990s, Virtual Reality (VR) technology, particularly head-mounted displays (HMDs), has seen significant advancements. In 1999, an empirical study demonstrated that passive haptics could significantly improve both user performance and preference in 2D tasks. In this paper, we replicate this experiment using modern VR hardware to investigate the influence of technological evolution on the relevance of passive haptics in similar scenarios. Our findings show that, for the tasks examined, performance in non-haptic conditions with current VR systems is comparable to that in haptic conditions from 1999, challenging the relevance of passive haptics for such tasks for nowadays standards. Our results imply that enhancements in visual fidelity, tracking and interaction design may have reduced the performance gap that passive haptics were previously used to address.

Buildings increasingly incorporate sensing and actuation techniques to automate the regulation of temperature, lighting, ventilation, and more. This trend seeks to minimize human intervention, justified by the promise of enhancing energy optimization. However, it has been widely acknowledged that loss of control over environmental conditions can lead to a diminished perception of comfort and compromised long-term user awareness and satisfaction. How can we envision building systems that can interact with building inhabitants and engage them at the “right” time and place? In this work, we address this challenge through three key contributions: 1) AirSpecs, a novel smart glasses-based system that enables holistic sensing of Indoor Environmental Quality (IEQ) and physiological data, 2) an objective method for assessing environmental awareness using a peripheral LED light and a proposed comfort awareness process, and 3) design implications for addressing the fluidity of comfort. Over five days, 30 participants across three continents used the AirSpecs device and its accompanying mobile application. Through a mixed-methods analysis of user interactions, post-experience surveys, interviews, and co-design sessions, we present findings and design scenarios that demonstrate the potential of our contributions to enhance occupant engagement and comfort in smart buildings in the future. • Novel eyewear and apps enable access-anywhere interaction and personal attachment. • Comfort awareness transits between subliminal, preconscious, and conscious states. • Focus, ambient aid, and reflection modes can be designed for three awareness states. • Users prefer to interact with a semi-automated building system.

Technologies for blind and low-vision (BLV) people have long been a focus of Human-Computer Interaction (HCI) and accessibility (ASSETS) research. To map and assess this cross-disciplinary field, prior literature reviews have focused on specific BLV research areas (e.g., navigation assistance) or study methodologies (e.g., qualitative methods). In this paper, we provide a more holistic examination, combining both quantitative bibliometric analyses with qualitative assessments. Using keyword queries of terms focused on the human (e.g., people) and their visual status (e.g., blind, low-vision), we first derived a dataset of 880 papers published between 2010-2022 from ACM and IEEE conferences and journals. We then apply a programmatic analysis of this dataset followed by a qualitative analysis of the 100 most-cited papers. Our findings highlight four major research areas: Accessibility at Home & on the Go, Non-Visual Interaction, Orientation & Mobility, and Education. We also capture the diversity of denominations used to refer to the BLV community and their co-occurrences, as well as computer systems targeting both blind and low-vision users with a focus on visual substitution. We close by suggesting areas for future work and hope to stimulate discussions in our field.

In a world where technological advancements are progressing at a vertiginous pace, social networks, online games, virtual worlds, streaming services, and remote work are part of everyday life. This is the case for the work environment, with the use of technological tools and home offices. In contrast, harmful aspects have been amplified, such as stress that affects occupational health. Lately, considerable interest has been gained in the affective domain in improving the occupational situation using empathic responses. In this work, we study the effect of machine empathic responses such as blue light, relaxing music, and the combination of light and music on people performing stressful tasks in an occupational environment. Thirty five participants tested different stimuli, eleven tested the music condition, twelve the light effect, and another twelve the combination of light and music. The monitoring of the heart rate variability along with psychological measures show that empathic responses can help reduce humans stress levels.

The communication between humans and artificial agents is becoming crucial and significant in daily life, especially with the advancements in the fields of human-robot and human-computer interaction. For these artificial agents to be recognized as social beings, they should exhibit emotional and empathic behaviors. However, there is no global agreement on measuring the empathic capabilities of these agents. For this reason, the scientific community has paid a significant focus on developing a standardized metric to perceive artificial agents' empathy. In this regard, this article provides a discussion on challenges in artificial empathy evaluation and researches the developments to discuss the factors and recommendations to design a globally accepted metric. It also discusses the qualities required for a globally accepted and standardized metric. Finally, an adaptation to an existing questionnaire is proposed for the evaluation of empathy in artificial agents.

In 2020, we have witnessed global experimentation of remote co-working, co-learning, and co-habiting, leading to the re-emergence of a collective search for platforms and paradigms that can optimally coalesce the virtual and physical settings - what has been studied as "hybrid models". In this context, we examine the opportunities that the advances in Spatial Audio techniques can create to improve hybrid meetings. Concretely, we present a controlled study in which 84 participants used an online platform to follow six pre-recorded semi-scripted dialogues. The videos were around two minutes long and each of them simulated a piece of conversation in the physical meeting room among three actors who played the role of co-located attendees. The six videos represented six conditions: three auditive formats, (x2) once co-located attendees wore face masks, and once without masks. We compared the experiences of the participants (remote attendees) in these six conditions. Analyzing three types of data, namely, comprehension/memory test results, self-reported ratings, and eye-tracking, we have found reinforcing evidence that demonstrates the benefits of binaural audio in hybrid settings.

Repeated exposure to poor air quality in indoor environments such as office, home, and classroom can have substantial adverse effects on our health and productivity. The problem is especially recognized in closed indoor spaces shared by several people. We have studied the evolution of carbon dioxide level in office-meeting spaces, during 1007 meeting sessions. The collected data is employed to examine machine learning models aimed to indicate the CO2 evolution pattern and to forecast when fresh air should be supplied. In addition, to gain insight into the relations and interdependencies of social factors in meetings that may influence the users’ perception of an interactive solution, we have conducted a series of online surveys. Building on the results of the two studies, a solution is proposed that predicts the evolution of air quality in naturally-ventilated meeting rooms and engages the users in preventive actions when risk is forecast.

Humans nowadays are tending to spend too much time in front of their screens. Direct interaction between humans is falling in numbers and people are losing their empathic behaviour. By integrating empathy and emotions in everyday objects researchers can address this problem. In addition we can have a positive effect on our lives, from physical and mental health through tackling many issues, like productivity, time wasting, stress and other problems. In this article, we tackle the productivity problem by presenting the Empathic Flower Companion (EFC) that will be using the expression of emotions to help the human through their working day. It will be monitoring their time and at the same time analysing the websites they will be surfing. The concept proposed will reduce the time wasted on unproductive websites. The results show an increase of 15% in the productivity of the testers, which shows that EFC was effective in reducing the amount of time wasted.

Since the introduction of the iconic Aware Home project [39] in 1999, the notion of “living laboratory” has been taken up and developed in HCI research. Many of the underpinning assumptions have evolved over the past two decades in various directions, while the same nomenclature is employed—inevitably in ambiguous ways. This contribution seeks to elicit an organized understanding of what we talk about and when we talk about living lab studies in HCI. This is accomplished through the methods of discourse analysis [66, 69], a combination of coding, hypothesis generation, and inferential statistics on the coded data. Analysing the discursive context within which the term living laboratory (or lab) appears in 152 SIGCHI and TOCHI articles, we extracted five divergent strands with overlapping but distinct conceptual frameworks, labeled as “Visited Places,” “Instrumented Places,” “Instrumented People,” “Lived-in Places,” and “Innovation Spaces.” In the first part of this article, we describe in detail the method and outcome of our analysis that draws out the five strands. Building on the results of the first part, in the second part of this article, each of the five types of living lab is discussed using some of the prototypical examples of that kind presented in the literature. Finally, we discuss the raison d’être and future position of the living lab as a method within HCI research and design and in relation to advances in sensing technologies and the emerging world of intelligent built environments (e.g., smart city and smart home).

Buildings and urban spaces increasingly incorporate artificial intelligence and new forms of interactivity, raising a wide span of research questions about the future of human experiences with, and within, built environments. We call this emerging area Human-Building Interaction (HBI) and introduce it as an interdisciplinary domain of research interfacing Human-Computer Interaction (HCI) with Architecture and Urban Design. HBI seeks to examine the involvement of HCI in studying and steering the evolution of built environments. Therefore, we need to ask foundational questions such as the following: what are the specific attributes of built environments that HCI researchers should take into account when shifting attention and scale from “artefacts” to “environments”? Are architecture and interaction design methods and processes compatible? Concretely, how can a team of interaction designers bring their tools to an architectural project, and collaborate with other stakeholders? Can and will architecture change the theory and practice of HCI? Furthermore, research in HBI should produce knowledge and practical guidelines by experimenting novel design instances that combine architecture and digital interaction. The primary aim of this article is to specify the mission, vision, and scope of research in HBI. As the introductory article to the TOCHI special issue, it also provides a summary of published manuscripts and describes their collective contribution to the development of this field.

This contribution exemplifies how the study of space perception and its impact on space-use behavior can inform sustainable architecture. We describe our attempt to integrate the methods of user research in an architectural project that was focused on optimization of space usage. In an office building, two large office rooms were refurbished to provide desk-sharing opportunities through hot-desking. We studied the space-use behavior of 33 office workers over eight weeks in those two rooms as well as their occasional presence in ten other areas (cafeteria, atrium, meeting rooms, etc.). Quantitative and qualitative analyses were performed to understand the nature and nuances of space occupancy at the scope of the building and within the refurbished offices. While at the scope of building the patterns of movements between rooms were found to be related to the professional profile of the users, at the scope of office the occupancy patterns were influenced by the spatial design of workspaces. More precisely, certain visual attributes of a workspace, namely Visual Exposure and Visual Openness, could determine whether or not it was regularly used. In this paper, we describe our findings in detail and discuss their implications for sustainable building design.

Video-based communication has become a common way of interacting with remote interlocutors, whether through complex videoconferencing systems or webcams integrated into consumer technologies. Ethnomethodology and conversation analysis (EM/CA) are sociological approaches that have been influential in Human–Computer Interaction for nearly three decades due to their focus on the situated organization of practical activities. In this article, we present a state-of-the-art review of empirical research on video-mediated social interaction studied from the perspective of EM/CA. We put forward an original organization of the findings on the interplay of talk, bodily behavior and spatial and material resources. The review underscores the ways in which technology enables and constrains interaction, shaping familiar and novel social activities. We also propose directions for future research and systems design.

Interacting with Computers, 2018:73–84. doi: 10.1093/iwc/iwx019 This article has been corrected to address an error in the reference list. The correct reference is: Pekarek Doehler, S., Wagner, J. and González-Martínez, E. (eds) (2018) Longitudinal Studies on the Organization of Social Interaction. Palgrave MacMillan, London, [in press].

Understanding how people use different spaces in a building can inform design interventions aimed at improving the utility of that building, but can also inform the design of future buildings. We studied space use in an office building following a method we have designed to reveal the occupancy rate and navigational patterns. Our method involves two key components: 1) a pervasive sensing system that is scalable for large buildings, and high number of occupants, and 2) participatory data analysis engaging stakeholders including interior architects and building performance engineers, to refine the questions and define the needs for further analyses through multiple iterations.

Several models and design spaces have been defined and are regularly used to describe how modalities can be fused together in an interactive multimodal system. However, models such as CASE, the CARE properties or TYCOON have all been defined more than two decades ago. In this paper, we start with a critical review of these models, which notably highlighted a confusion between how the user and the system side of a multimodal system were described. Based on this critical review, we define MMMM v1, an improved model for the description of multimodal fusion in interactive systems targeting completeness. A first user evaluation comparing the models revealed that MMMM v1 was indeed complete, but at the cost of user friendliness. Based on the results of this first evaluation, an improved version of MMMM, called MMMM v2 was defined. A second user evaluation highlighted that this model achieved a good balance between complexity, consistency and completeness compared to the state of the art.

This paper presents the concept of a lamp that allows displaying and collecting user's emotional states. In particular, it displays the emotional information changing colors and facial expressions; in fact, the lamp is characterized by anthropomorphic form and behavior in order to make the interaction more natural and spontaneous. The user can interact with the lamp through tangible gestures typically used in social interactions by humans. Two different scenarios involving the use of the lamp as a companion and for computer-mediated communication are presented.

More and more objects of our everyday environment are becoming smart and connected, offering us new interaction possibilities. Tangible interaction and gestural interaction are promising communication means with these objects in this post-WIMP interaction era. Although based on different principles, they both exploit our body awareness and our skills to provide a richer and more intuitive interaction. Occasionally, when user gestures involve physical artifacts, tangible interaction and gestural interaction can blend into a new paradigm, i.e., tangible gesture interaction [5]. This workshop fosters the comparison among these different interaction paradigms and offers a unique opportunity to discuss their analogies and differences, as well as the definitions, boundaries, strengths, application domains and perspectives of tangible gesture interaction. Participants from different backgrounds are invited.

"Eyes on the road, hands on the wheel" is a crucial principle to be taken into account designing interactions for current in-vehicle interfaces. Gesture interaction is a promising modality that can be implemented following this principle in order to reduce driver distraction and increase safety. We present the results of a user elicitation for gestures performed on the surface of the steering wheel. We asked to 40 participants to elicit 6 gestures, for a total of 240 gestures. Based on the results of this experience, we derived a taxonomy of gestures performed on the steering wheel. The analysis of the results offers useful suggestions for the design of in-vehicle gestural interfaces based on this approach.

Social awareness streams limit the expressivity of emotions in computer mediated communication. In this demo, we present a system that allows sharing emotional states in a social group with multimodal ambient feedback in order to provide a more immersive and natural interaction experience.

This paper examines how emotion feedback influences emotion awareness and gaze behavior. Simulating a videoconference setup, 36 participants watched 12 emotional video sequences that were selected from the SEMAINE database. All participants wore an eye-tracker to measure gaze behavior and were asked to rate the perceived emotion for each video sequence. 3 conditions were tested: (c1) no feedback, i.e., the original video-sequences, (c2) correct feedback, i.e., an emoticon is integrated in the video to show the emotion depicted by the person in the video and (c3) random feedback, i.e., the emoticon displays at random an emotional state that may or may not correspond to the one of the person. The results showed that emotion feedback had a significant influence on gaze behavior, e.g., over time random feedback led to a decrease in the frequency of episodes of gaze. No effect of emotion display was observed for emotion recognition. However, experiments on the automatic emotion recognition using gaze behavior provided good performance, with better score on arousal than valence, and a very good performance was obtained in the automatic recognition of the correctness of the emotion feedback.

In this paper, we present a novel opportunistic paradigm for in-vehicle gesture recognition. This paradigm allows using two or more subsystems in a synergistic manner: they can work in parallel but the lack of some of them does not compromise the functioning of the whole system. In order to segment and recognize micro-gestures performed by the user on the steering wheel, we combine a wearable approach based on the electromyography of the user's forearm muscles, with an environmental approach based on pressure sensors integrated directly on the steering wheel. We present and analyze several fusion methods and gesture segmentation strategies. A prototype has been developed and evaluated with data from nine subjects. The results prove that the proposed opportunistic system performs equal or better than each stand-alone subsystem while increasing the interaction possibilities.

In this paper, we present a research oriented open challenge focusing on multimodal gesture spotting and recognition from continuous sequences in the context of close human-computer interaction. We contextually outline the added value of the proposed challenge by presenting most recent and popular challenges and corpora available in the field. Then we present the procedures for data collection, corpus creation and the tools that have been developed for participants. Finally we introduce a novel single performance metric that has been developed to quantitatively evaluate the spotting and recognition task with multiple sensors.

Abstract We present a qualitative user study analyzing findings made while exploring changes over time in spatial interactions. We analyzed findings made by the study participants with flow maps, one of the most popular representations of spatial interactions, using animation and small‐multiples as two alternative ways of representing temporal changes. Our goal was not to measure the subjects’ performance with the two views, but to find out whether there are qualitative differences between the types of findings users make with these two representations. To achieve this goal we performed a deep analysis of the collected findings, the interaction logs, and the subjective feedback from the users. We observed that with animation the subjects tended to make more findings concerning geographically local events and changes between subsequent years. With small‐multiples more findings concerning longer time periods were made. Besides, our results suggest that switching from one view to the other might lead to an increase in the numbers of findings of specific types made by the subjects which can be beneficial for certain tasks.

Abstract Many origin‐destination datasets have become available in the recent years, e.g. flows of people, animals, money, material, or network traffic between pairs of locations, but appropriate techniques for their exploration still have to be developed. Especially, supporting the analysis of datasets with a temporal dimension remains a significant challenge. Many techniques for the exploration of spatio‐temporal data have been developed, but they prove to be only of limited use when applied to temporal origin‐destination datasets. We present Flowstrates, a new interactive visualization approach in which the origins and the destinations of the flows are displayed in two separate maps, and the changes over time of the flow magnitudes are represented in a separate heatmap view in the middle. This allows the users to perform spatial visual queries, focusing on different regions of interest for the origins and destinations, and to analyze the changes over time provided with the means of flow ordering, filtering and aggregation in the heatmap. In this paper, we discuss the challenges associated with the visualization of temporal origin‐destination data, introduce our solution, and present several usage scenarios showing how the tool we have developed supports them.

Revealing and being able to manipulate the structured content of PDF documents is a difficult task, requiring pre-processing and reverse engineering techniques. In this paper, we present OCD, an optimized, easy-to-process and canonical format for representing structured electronic documents. The system and methods used for reverse engineering PDF documents into the OCD format are presented as well as the techniques to optimize it. We finally expose concrete evaluations of our OCD format compactness and restructuring performances.

This article proposes an evaluation framework to benchmark the performance of multimodal fusion engines. The paper first introduces different concepts and techniques associated with multimodal fusion engines and further surveys recent implementations. It then discusses the importance of evaluation as a mean to assess fusion engines, not only from the user perspective, but also at a performance level. The article further proposes a benchmark and a formalism to build testbeds for assessing multimodal fusion engines. In its last section, our current fusion engine and the associated system HephaisTK are evaluated thanks to the evaluation framework proposed in this article. The article concludes with a discussion on the proposed quantitative evaluation, suggestions to build useful testbeds, and proposes some future improvements.

This article introduces HephaisTK, a toolkit for rapid prototyping of multimodal interfaces. After briefly discussing the state of the art, the architecture traits of the toolkit are displayed, along with the major features of HephaisTK: agent-based architecture, ability to plug in easily new input recognizers, fusion engine and configuration by means of a SMUIML XML file. Finally, applications created with the HephaisTK toolkit are discussed.

Fusion engines are fundamental components of multimodal inter-active systems, to interpret input streams whose meaning can vary according to the context, task, user and time. Other surveys have considered multimodal interactive systems; we focus more closely on the design, specification, construction and evaluation of fusion engines. We first introduce some terminology and set out the major challenges that fusion engines propose to solve. A history of past work in the field of fusion engines is then presented using the BRETAM model. These approaches to fusion are then classified. The classification considers the types of application, the fusion principles and the temporal aspects. Finally, the challenges for future work in the field of fusion engines are set out. These include software frameworks, quantitative evaluation, machine learning and adaptation.

This paper proposes a new approach for modelling, testing and prototyping pervasive, possibly mobile, and distributed applications. It describes a set of tools aimed at supporting designers in the conceptualisation of their application and in the software development stage, and proposes a method for checking the validity of their design. The article also presents a pervasive application implemented and evaluated using our approach. It concludes with propositions for improvements in order to build a complete modelling, prototyping and testing framework for pervasive applications.

Abstract The paper presents an extension to the Excentric Labeling, a labeling technique to dynamically show labels around a movable lens. Each labels refers to one object within the lens and is connected to it through a line. The original implementation has several known limitations and potential improvements that we address in this work, like: high density areas, uneven density distributions, and summary statistics. We describe the implemented extensions and present a think‐aloud user study. The study shows that users can naturally understand and easily operate the majority of the implemented function but label scrolling, which requires additional research. From the study we also gained unanticipated requirements and interesting directions for further research.

This paper reviews the challenges associated with the development of tangible and multimodal interfaces and exposes our experiences with the development of three different software architectures to rapidly prototype such interfaces. The article first reviews the state of the art, and further compares existing systems with our approaches. Finally, the article stresses the major issues associated with the development of toolkits allowing the creation of multimodal and tangible interfaces, and presents our future objectives.

The paper describes a method by which one could use the documents captured from low-resolution handheld devices to retrieve the originals of those documents from a document store. The method considers conjunctively two complementary feature sets. First, the geometrical distribution of the color in the document's 2D image plane is preferred. Secondly, the shallow layout features is considered due to the poor resolution of the captured documents. We propose in this article to fuse those two complementary feature sets in order to improve document identification performance. Finally, in order to test the influence of merging strategies on document identification performance, a synergic method is proposed and evaluated relative to a similar method in which feature sets are simply considered sequentially.

This paper proposes a multi-signature document identification method that works robustly with low-resolution documents captured from handheld devices. The proposed method is based on the extraction of a visual signature containing both (a) the color content distribution in the image plane of the document, i.e. the color signature, and (b) the shallow layout structure of the document, i.e. the layout signature. The color distribution is first considered, in order to filter documents with very dissimilar colors, and the identification is finally done on the remaining set using the layout signature. An evaluation, that compares our color and layout-based method with the layout signature alone, is finally presented.

Relationships that link static documents discussed during meetings to the corresponding speech transcripts can be of various kinds. The most important ones, thematic links, quotations and references are presented in this paper. Thematic links are detected via a thematic alignment process. However, quotations extraction is based on the detection of segments of documents that are quoted in the speech transcript. References, made by speakers to documents, are performed via a matching process between referring expressions detected in the speech transcript, and corresponding documents logical blocks. Finally, a framework that combines these links and an evaluation of the links complementarity are presented.

This article presents Xed, a reverse engineering tool for PDF documents, which extracts the original document layout structure. Xed mixes electronic extraction methods with state-of-the-art document analysis techniques and outputs the layout structure in a hierarchical canonical form, i.e. which is universal and independent of the document type. This article first reviews the major traps and tricks of the PDF format. It then introduces the architecture of Xed along with its main modules, and, in particular, the document physical structure extraction algorithm. Later on, a canonical format is proposed and discussed with an example. Finally the results of a practical evaluation are presented, followed by an outline of future works on the logical structure extraction.

In this paper, we describe a new method for a simultaneous thematic segmentation of the meeting dialogs and the documents discussed or visible throughout the meeting. This bi-modal method is suitable for multimodal applications that are centered on documents, such as meetings and lectures, where documents can be aligned with meeting dialogs. Bringing into play this alignment, our bi-modal segmentation method first transforms its results into a set of nodes in a 2D graph space, where the two axes represent respectively the document units and the meeting dialogs units. Secondly, via a clustering method, the most connected regions in the constituted bi-graph are detected. Finally, the denser clusters are projected on the two axes. The two sequences of segments, obtained on both axes, represent the thematic structure of the document and of the meeting dialogs respectively. We present in this article this bi-modal segmentation technique and its performance compared with two mono-modal segmentation methods. Categories and Subject Descriptors H.3.1 [Content Analysis and Indexing] indexing methods;

In this paper, we present (a) a method for identifying documents captured from low-resolution devices such as web-cams, digital cameras or mobile phones and (b) a technique for extracting their textual content without performing OCR. The first method associates a hierarchically structured visual signature to the low-resolution document image and further matches it with the visual signatures of the original high-resolution document images, stored in PDF form in a repository. The matching algorithm follows the signature hierarchy, which speeds-up the search by guiding it towards fruitful solution spaces. In a second step, the content of the original PDF document is extracted, structured, and matched with its corresponding high-resolution visual signature. Finally, the matched content is attached to the low-resolution document image's visual signature, which greatly enriches the document's content and indexing. We present in this article both these identification and extraction methods and evaluate them on various documents, resolutions and lighting conditions, using different capture devices.

In the context of a multimodal application, the article proposes an image-based method for bridging the gap between document excerpts and video extracts. The approach, called document image alignment, takes advantage of the observable events related to documents that are visible during meetings. In particular, the article presents a new method for detecting slide changes in slideshows, its evaluation, and a preliminary work on document identification.

The primary goal of this PhD thesis is to align printable documents with meetings' dialogs. This bi-modal alignment consists in bridging thematic links between documents' content and speech transcripts' content. An obvious application is a system that automatically link document parts with audio-video extracts of a meeting. Further, this bi-modal alignment is considered for thematically segmenting both meeting dialogs and documents discussed during this meeting.

This article proposes a multimodal approach for segmenting meeting recordings. This bi-modal method takes advantages of the alignment of speech transcript with documents, in the context of meetings or lectures, where documents are discussed. The method first displays the alignment results as a set of nodes in a 2D space, where the two axes represent respectively the documents content and the speech transcript. The most connected regions in this graph are detected using a clustering method. The final clusters are then projected on the speech axis. Finally, the obtained sequence of segments is considered as the thematic structure of the speech transcript. In this article, we present our bi-modal method and compare it with two other mono-modal thematic segmentation methods.

We present in this article a method for detecting similarity links between documents' content and speech recordings' content. This process, further called thematic alignment, is a novel research area that combines both document and speech analysis. This alignment will a) provide temporal indexes to documents, which are non-temporal data, and b) help discovering hidden thematic structures. This article first introduces a multi-layered document structure and quickly introduces the traditional speech structure. Further, it presents a simple similarity measure and various multi-level simple alignments between those two structures. Later, the meeting corpus is presented, as well as an evaluation of the implemented alignments. Finally, we present our future works on multi-alignments and thematic structure discovery.

Visual thinking tools are visualization-enabled mixed initiative systems that empower people in solving complex problems by engaging them in the entire resolution process, suggesting appropriate actions with visual cues, and reducing their cognitive load with visual representations of their tasks. At the same time, the visual interaction style provides an alternative to the dialog-based model employed in most mixed-initiative (MI) systems. Visual thinking tools avoid complex analyses of turn taking, and put users in control all the time. We are especially interested in implementing visual "affordances" in such systems and present three examples used in COMIND, a visual MI system that we have developed. We show how humans can more effectively concentrate on synthesizing problems, selecting resolution paths that were unseen by the machine, and reformulating problems if solutions cannot be found or are unsatisfactory. We further discuss our evaluation of the techniques at the end of the paper.