We characterize 16 challenges faced by those investigating and developing remote and synchronous collaborative experiences around visualization. Our work reflects the perspectives and prior research efforts of an international group of 29 experts from across human-computer interaction and visualization sub-communities. The challenges are anchored around five collaborative activities that exhibit a centrality of visualization and multimodal communication. These activities include exploratory data analysis, creative ideation, visualization-rich presentations, joint decision making grounded in data, and real-time data monitoring. The challenges also reflect the changing dynamics of these activities in the face of recent advances in extended reality (XR) and artificial intelligence (AI). As an organizing scheme for future research at the intersection of visualization and computer-supported cooperative work, we align the challenges with a sequence of four sets of research and development activities: technological choices, social factors, AI assistance, and evaluation.

This paper presents an attempt to read the success of a presentation in conveying information and gaining empathy from listener responses. The success of a presentation is strongly influenced not only by the speaker but also by the listener's reactions, such as nodding and other nonverbal behaviors and speech during the Q&A session. Therefore, we analyze the relationship between the participants' verbal and nonverbal behavior data obtained from online Bibliobattle and their voting results. Specifically, we focus on the amount of speech and nodding by the listeners, and in addition, we examine the relationship between their voting results and the results of questionnaires on empathy and comprehension. The results suggest that while some listeners expressed nodding as a signal of empathy, there were not a few who did not, and that these differences may be related to personality.

Navigation applications are becoming ubiquitous in our daily navigation experiences. With the intention to circumnavigate congested roads, their route guidance always follows the basic assumption that drivers always want the fastest route. However, it is unclear how their recommendations are followed and what factors affect their adoption. We present the results of a semi-structured qualitative study with 17 drivers, mostly from the Philippines and Japan. We recorded their daily commutes and occasional trips, and inquired into their navigation practices, route choices and on-the-fly decision-making. We found that while drivers choose a recommended route in urgent situations, many still preferred to follow familiar routes. Drivers deviated because of a recommendation's use of unfamiliar roads, lack of local context, perceived driving unsuitability, and inconsistencies with realized navigation experiences. Our findings and implications emphasize their personalization needs, and how the right amount of algorithmic sophistication can encourage behavioral adaptation.

This paper discusses the effects of multiple viewpoint videos for metacognition of experiences. We present a system for recording multiple users' collaborative experiences by wearable and environmental sensors, and another system for viewing multiple viewpoint videos automatically identified and extracted to associate to individual users. We designed an experiment to compare the metacognition of one's own experience between those based on memory and those supported by video viewing. The experimental results show that metacognitive descriptions related to one's own mind, such as feelings and preferences, are possible regardless whether a person is viewing videos, but such episodic descriptions as the content of someone's utterance and what s/he felt associated with it are strongly promoted by video viewing. We conducted another experiment where the same participants did identical metacognitive description tasks about half a year after the previous experiment. Through the experiments, we found the first-person view video is mostly used for confirming the episodic facts immediately after the experience, whereas after half a year, even one's own experience is often felt like the experiences of others therefore the videos capturing themselves from the conversation partners and environment become important for thinking back to the situations where they were placed.

This paper proposes a system, ComicQA, that provides a hyper-comic showing solutions to a user having problems. The ComicQA system identifies the user's situation through question-answering and represents the problem and its solution in comic stories, from which we expect the user to easily grasp not only the solution for the focused problem but also its context. We show two use cases of the system's application, namely troubleshooting for an electric appliance and navigating a museum exhibition tour.

In this study, we propose an in-car conversation sharing system. People frequently converse in a car. In these conversations, people often talk about points of interest that they have just passed. Because we believe that they contain useful information, the aim of our study is to share the conversations. To develop such a system, we needed to know about the characteristics of in-car conversations. Consequently we collected 120 in-car conversations with their locations over a 10-month period. Our analysis showed that many types of conversation take place in a car; but of the greatest interest is when the subject of conversation is a specific location or area. We discuss the requirements for an in-car conversation sharing system to guide our on-going research.

In this paper, we propose a novel way to encourage visitors to share their experiences and interests in exhibition spaces. To acquire information about the experiences of the visitors, we used a robot that inhabits the exhibition space. The robot acquires the visitor experience data and then shares this data with other visitors. We believe that this information (i.e., the experiences of previous visitors) will enhance the exhibition experience for subsequent visitors because the information may include interesting topics for the new visitors. However, there are some difficulties in communication between robots and humans. We used “PhotoChat”, which is an in-house photo communication software. PhotoChat is capable of communicating with others by taking photographs and adding annotations to each photograph. It also records the locations for coordination between the photographs and the statistical information contained in the annotations. We therefore used PhotoChat as the communication medium between the robot and the visitors. The robot acquires the visitor experiences and shares these experiences with other visitors through PhotoChat. The robot can also use bodily actions to express instructions to the visitors. We then developed a system that coordinates the photographs and their respective exhibition areas. We also implemented robot behavior (i.e., bodily actions and motions) that includes recommendations for photographs taken by others. We held workshops to perform data collection and manually classified the data into three content categories. We then performed experiments using the developed system to distribute the classified content. The results showed that the robot's behavior encouraged conversations between the visitors based on provided topics.

Communicative hand gestures play important roles in face-to-face conversations. These gestures are arbitrarily used depending on an individual; even when two speakers narrate the same story, they do not always use the same hand gesture (movement, position, and motion trajectory) to describe the same scene. In this paper, we propose a framework for the classification of communicative gestures in small group interactions. We focus on how many times the hands are held in a gesture and how long a speaker continues a hand stroke, instead of observing hand positions and hand motion trajectories. In addition, to model communicative gesture patterns, we use nonverbal features of participants addressed from participant gestures. In this research, we extract features of gesture phases defined by Kendon (2004) and co-occurring nonverbal patterns with gestures, i.e., utterance, head gesture, and head direction of each participant, by using pattern recognition techniques. In the experiments, we collect eight group narrative interaction datasets to evaluate the classification performance. The experimental results show that gesture phase features and nonverbal features of other participants improves the performance to discriminate communicative gestures that are used in narrative speeches and other gestures from 4% to 16%.

There are many consumer electronics placed on our home. When we control these consumer electronics remotely, we are often forced to switch between remote controllers. Further, the increased functions of electronics also increase a number of buttons of remote controllers. These complexities make it difficult for users to control consumer electronics. To simplify interaction between remote controllers and the electronics, we propose a user buildable remote controller device, called Blockon. The Blockon enables a user to build his own controller easily by placing button blocks on the board as building LEGO blocks. The Blockon not only supports placing buttons freely but also makes it possible to define macro commands by stacking button blocks. In this paper, we also present a prototype of the Blockon for technical verification along with our future plan.

Human activity recognition through the wearable sensor will enable a next-generation human-oriented ubiquitous computing. However, most of research on human activity recognition so far is based on small number of subjects, and non-public data. To overcome the situation, we have gathered 4897 accelerometer data with 116 subjects and compose them as HASC2011corpus. In the field of pattern recognition, it is very important to evaluate and to improve the recognition methods by using the same dataset as a common ground. We make the HASC2011corpus into public for the research community to use it as a common ground of the Human Activity Recognition. We also show several facts and results of obtained from the corpus.

This paper proposes a novel mining method for multimodal interactions to extract important patterns of group activities. These extracted patterns can be used as machine-readable event indices in developing an interaction corpus based on a huge collection of human interaction data captured by various sensors. The event indices can be used, for example, to summarize a set of events and to search for particular events because they contain various pieces of context information. The proposed method extracts simultaneously occurring patterns of primitive events in interaction, such as gaze and speech, that in combination occur more consistently than randomly. The proposed method provides a statistically plausible definition of interaction events that is not possible through intuitive top-down definitions. We demonstrate the effectiveness of our method for the data captured in an experimental setup of a poster-exhibition scene. Several interesting patterns are extracted by the method, and we examined their interpretations.

An interactive exhibition guide robot has been developed, that is integrated with ubiquitous sensors. The robot utilizes its human-like body properties to perform human-like gestures such as eye-contact and pointing. The ubiquitous sensors are embedded in an exhibition room or attached to people in the room, and they provide information not only for the robot's interactive guide but also for the development of the robot. The results of a two-days exhibition experiment show that the robot attracted visitors and successfully guided them in the environment. Moreover, the obtained interaction statistics, including video from the ubiquitous sensors, prove to be useful for analyzing the interaction between the robot and visitors, and this implies that the use of such interaction data is a promising approach for developing interactive robots.

We propose an intelligent environment for "co-experience communication" consisting of stationary and wearable sensors, and communication robots. The environment captures many events collaboratively from multiple viewpoints using a combination of stationary sensors that track people's behaviors and interactions, and wearable sensors that observe the viewpoints and areas of interest of individual persons. Humanoid communication robots also play a role as assistants to help visitors pay attention to significant events. First of all, the basic framework of the environment is described. Next, our developing exhibition room is introduced as an example of an intelligent environment.

This paper proposes a concept of robots called ubiquitous intelligent robot and introduces our research efforts toward establishing its fundamental technology. A ubiquitous intelligent robot is a kind of embodied intelligence that encompasses the sphere of people's activities, records these activities, and supports people. Under this concept, we have developed an intelligent environment for capturing our experiences and a wearable event recording system for medical care. These developments mainly focus on recording people's activities along with their environments.

This paper describes our ongoing attempts to build a communityware system by presenting a project of providing digital assistants to support participants in an academic conference. We provided participants at the conference with a personal assistant system with mobile and ubiquitous computing technologies and facilitated communication among the participants. We also made online services available via the Web to encourage the participants to continue their relationships even after the conference. In this paper, we show the system we provided for the project and report the results.

We present a method for visualizing the structure of a discussion in an electronic conference environment, and propose a prototype system called MRV (Multi-Relationship Viewer). MRV assists users in understanding the transitions of an argument and the whole argument structure by visualizing the relationship between the utterances from the view points of cause-and-effect and semantics. MRV provides the cause-and-effect view by using a tree structure, and the semantic view by using a spring model. The transition from one of the views to the other is seamless, enabling users to keep track of the flow of thinking. Results from a series of experiments proved that MRV can shorten the necessary time for users to grasp the transitions of an argument and the point at issue.

We present a method for visualizing a structure of discussion in an electronic conference environment, and propose a prototype system. The system assists users to understand the transition of the argument and the whole structure of the argument by visualizing the relationship between utterances from the view points of the cause-and-effect and the semantics. The system provides the cause-and-effect view by using a tree structure, and semantic view by using a spring model. The transition from one view to other is seamless so as to enable users to keep track of thinking.

We propose a method for presenting information based on individual user interests and knowledge. In previous work on recommending information, such as through WWW pages, the user could not always get his/her needed information, since the user's interests were represented only in keywords or key categories, and the contents of the presented information were previously formed by the information presenter; consequently, this information did not necessarily match the user's interests and knowledge. In the proposed method, the system acquires the user's implicit and/or explicit interests and knowledge from a question-and-answer interaction and presents the user with focus-of-interest information based on them. For personalizing the presentation and recommendation of information, the system provides each user with an information retriever using the user's operation records, a personal conceptual space, and control knowledge.