This paper describes a Tabletop Imaginative Play as Enactive Storytelling (TIPES) system for children that captures their tabletop enactment and transforms them into animated cartoons in real time. In this paper, we compare children’s storytelling support using TIPES (AR condition) and the raw tabletop-play recordings without using TIPES (TR condition). We conducted a study involving 18 pairs of 7-13 year-old children. Each pair engaged in creating two stories in the AR or TR conditions, respectively. We coded for actions and vocalizations and grouped these action-vocalization compounds into enactment idea units. We identified a statistically significant positive correlation between total actions and vocalizations of each story in AR but not in TR and a statistically significant negative correlation between action complexity (actions/idea unit) and vocal richness (vocalizations/idea unit) in TR but not in AR. It can be inferred that children manifest distinct engagement in constructing story enactment ideas by demonstrating a more balanced management of semantic loads between executing actions and uttering vocalizations in AR compared to TR. This implies varying levels of incorporating multimodality in expression across different conditions. Furthermore, the analysis of interview responses highlighted greater enjoyment in utilizing TIPES.

This research full paper describes the augmented reality (AR) application AR-Classroom that combines a physical and virtual environment to teach 3D geometric rotations in an engaging and simplified manner. The AR-Classroom contains a virtual workshop where users can perform rotations by manipulating the application's X, Y, and Z-axis sliders to rotate a virtual LEGO model and a physical workshop where users perform rotations using a physical LEGO model. Guided by previous findings and an iterative approach to usability, the present usability study focused on assessing the usability of the AR-Classroom in its most recent version using a new physical LEGO model (i.e., airplane) and reflecting on how discoverability and usability can be assessed using different types of user experience measures and qualitative analysis. Participants were 22 undergraduate students who completed a pre-test with demographic information, watched a video on geometric transformations, and were randomly assigned to interact with either workshop. While interacting with the AR app, participants were instructed to provide feedback and a single ease-of-use question score. Participants then completed a post-test with two measures of usability. Descriptive statistics of the UX measures were explored, and a thematic analysis was conducted to identify and code themes in human-computer interaction. Findings suggest that the AR-Classroom has reached satisfactory usability, and users can navigate the app's features effectively. Discussion includes insight into which aspects of the app need improvement, how to promote self-directed support in the app, the development of future app efficacy experiments, and how to evaluate the usability of AR technology for learning.

This research-to-practice paper investigates how undergraduates organically transition into researchers through the lens of Legitimate Peripheral Participation, Legitimate Peripheral Participation is a contextual learning phenomenon through which new members become core members of a community. These peripheral members begin with the execution of peripheral low-risk and simple activities, gradually accruing knowledge and experience as they take on more complex and essential activities. They gain legitimacy and access to more complex involvement through interaction with, and mentorship by, senior members of the community. Our research examines how seven undergraduate students, who are involved with a research lab, transition from peripheral roles to core membership through observation and peripheral engagement, skill development, community integration, increasing responsibility, and collaborative projects. These undergraduates interact with and are mentored by both graduate students and faculty members in an organic fashion, with no assigned mentorship roles existing between the undergraduate students and graduate students or faculty. The focus of our research is not on how one recruits undergraduates to work in a research lab, but on how the identity of an undergraduate as a researcher, both in their own mind and in those of their coworkers, is nurtured and developed. Through the process of surveys and interviews we extract the core elements of the undergraduates' experience and explore factors that might contribute to or hinder their transition into being more active researchers. The degree of involvement and identification as a researcher varied significantly between the undergraduate students. The key elements we discover at social engagement, physical presence, and near-peer interaction.

In this paper, we investigate blended physical-digital kinesthetic feedback (or blended haptics in short) as a means for controlled three-dimensional design ideation in mixed reality (MR) environments. We define blended haptics as a spatial interaction wherein a physical object (e.g. a 3D printed human head) provides a specific design context for a user to generate ideas through physical manipulation of and on the object (e.g. drawing a digital sketch of a helmet) in the virtual environment. Using 3D wire-frame modeling as a concrete digital prototyping context, we investigate this idea of blended haptics in terms of how it supports design cognition specifically in spatial user interfaces. For this, we implemented a modeling tool as an experimental setup that allows a user to directly create curve-networks (wire-frame models) on a physical object (i.e. a contextual proxy) with one hand while simultaneously controlling the object with the other hand using a tracked turn-table. The key idea is that the user can simultaneously experience kinesthetic feedback from both the physical objects as well as the digital wire-frame models. To systematically investigate our approach, we conducted a comparative user evaluation of conceptual design tasks performed by two groups of users, one with the blended haptics (e.g. physical head and digital helmet) and the other with purely digital haptic feedback (e.g. digital head and digital helmet). Our study shows that blended haptics required less physical effort in the design task and resulted in concepts with higher novelty score as compared to using purely digital haptic feedback.

Instructional discourse around physically-based material typically involves the use of gestures to focus attention in the joint visual space of the instructor and the student. Such gestures are performed in conjunction with speech, thus situating the discourse in the space of what the student is learning or doing. This kind of situated discourse is missing in distance learning. The lack of situatedness between conversants contributes to communication inefficiencies, as was recently highlighted when classes began using Zoom during COVID. For instruction that involves hands-on learning, the inability for instructors to index the student's space and vice versa is a more significant problem. The use of telepresence technologies allows this spatial gap to be bridged but can often leave a communication gap. Little evaluation has been made into how these mechanisms alter gesture and speech. This paper explores how giving the instructor situated gestural access to the student's workspace in distant learning settings alters gesture and speech, with the goal of guiding design of future systems that are mindful of how these systems work, not simply whether they work. Our research is based on a system that supports situated instructional discourse through augmented reality gesturing. The instructor is able to see the student's workspace and perform gestures that are superposed into the student's workspace. This shared gestural space is used in tandem with a traditional video chat. Two spatially-focused tasks were examined: a two-dimension problem that required varying levels of assistance, and a three-dimension problem with complete guidance by the instructor needed. We evaluated how gestures were used to augment verbal communication and the types of gestures and verbal utterances used. We found that static pointing G-hand deictic gestures were more common in the augmented reality mode than in the co-situated mode and complex spatial structuring gestures (e.g. orientation gestures) occurred more often in the co-situated mode than the augmented reality mode. Additionally, students provided a range of body postures and working orientations to encourage or discourage invasion of their personal workspace in co-situated settings, and several subjects indicated that the augmented reality made them feel more comfortable with sharing their gestural space. These results provide a framework to guide the design of future improvements of telepresence systems. Subjective statements of comfort by subjects also suggest this may be a profitable avenue for exploration with students who are reluctant to engage physically with instructors.

Mathematics is indispensable to engineering and computer education. Indeed, the reading, performance and sharing of mathematics is analogous to the reading, writing and speaking of natural languages: Both are prerequisites for other fields of study, and both provide solutions to the very practical problems of communicating ideas across space, culture and time. Without mathematics, members of these fields would struggle to formalize, systemize and communicate ideas central to their missions. And yet the inaccessibility of conventional mathematical notation inhibits Individuals with Blindness or Severe Low Vision (IBSLV) from fully participating in this discipline. Such notations were designed by the sighted, for the sighted: They allow for the offloading of working memory to the page (as information is always accessible through sight) and take advantage of our immense capacity to remember both what information is present and where it can be found. If IBSLV are to be “first class citizens” of engineering and computer education, researchers must design tools that allow them to fully participate in the reading, performance and sharing of mathematics as well. In this paper, we make inroads into these problems by analyzing, contextualizing and framing the results of three computer-human interaction studies of MathSTAAR, a system which conveys mathematics to IBSLV. In the first, we found that a system which implements Vanderheiden's touch-and-speak model (in which participants can touch portions of a mathematical expression and have it spoken to them) allowed participants to correctly read more than 90% of expressions at a middle-school level, the level at which the average IBSLV falls behind their sighted peers. In the second, we found that augmenting this system with “coarse visual cues” (the highlighting of what is touched to allow IBSLV to take advantage of any vision they retain), allowed participants' to read, locate and relocate mathematical information faster than a keyboard system while also reducing participants' self-reported Frustration and Effort. In the third, we found that two multimodal approaches-one instantiated in our MathSTAAR system, the other instantiated in the Math Melodies application-allowed IBSLV to solve more than 90% of problems correctly. However, MathSTAAR did so while reducing both Frustration and performance time. We contextualize these findings in two ways. First, we show that even though many of those with sight consider reading mathematical expressions a trivial task, for IBSLV, this is not the case. Second, we employ Polanyi's ideas of particular and gestalt knowledge to show that both the nature of one's vision, as well as the tools one uses, greatly impacts what an individual might know and when they might know it.

Writing and storytelling require both ideas of what is in the story, and how to use language skills to express the story. This paper addresses the problem of bringing story play into the orbit of formal writing by implementing digitally-augmented enactment to bridge the gap between concrete story ideas and the more abstract organizations for expression. Our approach provides children with an interactive experience and enables them to capture their imaginative play to support story expression (retelling or writing). We designed and developed a mixed-reality storytelling system that combines physical play and digital augmentation within a tabletop environment. The system tracks the physical figurines on the tabletop and produces an animated (cartoon) version of their story in real-time, which the children can then view and use as a memory aid and dynamic outline for writing. In addition, to support a viewpoint-dependent narrative thinking, we developed the concept of implementing a virtual camera that allows children to control the narrative perspective by physically manipulating how their story world is viewed. In this paper, we will present the design and development of our system and a pilot study we conducted with 6 children aged 7-12 where two children worked together on creating stories with and without using our system. We will present the preliminary results we obtained from the pilot study by analyzing the collected data including questionnaires, interviews, and the story videos and writings. The results show that children using our system exhibited greater motivation in creating stories and higher levels of collaboration and writing performance compared to traditional tabletop play.

“What are we learning this for?” is a common refrain heard in education. Showing how engineering practices are relevant to everyday needs like building science models in K-12 classes may lead to greater motivation for students in STEM learning. This paper investigates how horizontal integration of different streams of learning may enable students to see why and how each element of knowledge is relevant to the whole. By combining microcontroller-based electronics, programming, and 3D design and fabrication to build science models, students may develop a stronger appreciation of how the component technologies interrelate (internal contextualization) and how these integrated computational models contribute to the enterprise of science (external contextualization). Furthermore, students in science classrooms see how computation can make physical models dynamic to represent changes and relationships more effectively. The implicit representational science models and the incorporation of technology and computation into a regular science curriculum make the knowledge transferable beyond an individual lesson. Students learn the component technologies cumulatively through a sustained process of engagement across multiple science modeling projects. The major challenges of this approach are how the science modeling projects may be designed and what factors have to be considered in the designs to realize this learning progression. To answer these questions, this paper describes a rigorous design process for the Physical Computational Models (PCMs) that incorporates science model correctness, progression of learning across all streams, pedagogical efficacy, classroom dynamics and manufacturability. In addition, we implemented a year-long project using our Horizontal Learning (HL) approach at two local public schools. This paper reports in detail two PCMs we developed for authentic 5thand 6thgrade classrooms to illustrate the integration of the learning streams in science, computing, digital electronics, and 3D design and fabrication. By analyzing three lessons we developed, the result demonstrated that our approach did produce the incremental learning progressions characterized by two primary principles in these streams across multiple HL interventions.

This paper aims to analyze the practices in recruiting mentors for our STEM + Computing (STEM+C) MentorCorps project. We used quantitative research methods in this study. A total of 69 student mentors were entered into study analysis. We utilized descriptive statistics and conducted t-tests and ANOVA to compare mentors' prior teaching, experience with kids, computing experience, Arduino knowledge, and Making experience regarding gender and ethnicity. The results indicated that female mentors had more prior experience working with kids, while male mentors had more computing skills than their female counterparts. There are no significant differences between genders regarding previous teaching, Arduino knowledge, and Making. Further, no differences were found between Caucasian, Hispanic, Asian, and Black/ African mentors regarding these five variables.

Multimedia approaches retain a powerful ability to convey numerous streams of information simultaneously. When used appropriately, the uptake and comprehension of these multiple streams can be accomplished with relative ease, despite multimedia approaches typically conveying more information than their single medium counterparts. Given their power, this paper investigates how multimedia approaches can be leveraged to convey spatially arranged text. We do this with a special eye towards Individuals with Severe Low Vision, a group often forgotten in discussions of multimedia and who have a much-reduced capacity to receive information via the visual medium. We accomplish our investigation by comparing two tools for reading mathematical expressions, a form of spatially arranged text. The first is MathSTAAR, a novel system which treats mathematical text as a form of interactive multimedia. The second is JAWS, a screen reading tool which provides access to mathematical text via audio only. Researchers found that the multimedia approach reduced the amount of time it took for participants to relocate mathematical content while also reducing Frustration and Effort.

Supporting children’s writing activities is a primary concern in elementary-school grades. Using embodied means such as visual stimuli through pictures or motion pictures has been shown effective in supporting children’s writing. To guide children during the writing activity, contextual cues are usually provided in face-to-face classroom practice with interactive instruction from a teacher. This paper explores the design of an interactive online system for video-scaffolded writing with contextual cues. In our online study with 13 children, we explored two approaches to designing contextual cues: temporally situated and visually situated. Participants were divided into two groups, each using a web-based interface to watch an animated video and write the story guided by one of the cue designs. We analyzed the children’s writing process and outcome in both groups. Our results show that temporally situated cues support a stronger overall story structure, while visually situated cues support more descriptive and interpretive writing.

Many story authoring tools have been designed to support children’s creativity through story enactment. However, at school, children are encouraged to engage in formal modes of expression, such as writing. This paper aims to explore how we may design tools that support children in writing using body-based enactment as a scaffold. In our study, 17 children used an interactive story authoring system to enact and record stories as videos and then write the stories while viewing them. We compared the story enactment videos and the corresponding written story essays in terms of the narrative structure, content, and coherence. Our results show a significant difference in story structure in the two formats and a marginally significant difference in the coherence ratings. We present our findings in terms of learned lessons for designing children’s enactment-based authoring tools to encourage more substantial written outcomes.

This paper explores the psycho-physiological impact of remote touch in conjunction with conversations shared over video telecommunication. Interlocutors sent upper arm squeezes to each other through paired touch input and output devices in real time. Committed couples were recruited to use the devices in semi-structured discussions after they watched a video clip that contained emotionally charged moments. Users were not told how to use the devices and were free to define the purpose of their use. We examined how remote touch may impact skin conductance and heart rate variability during emotionally charged conversations. Our results revealed that irrespective of speech sentiment, co-verbal touch reduced arousal when compared to discourse without touch. However, touch was used more frequently in utterances that were judged negative and its reduction in arousal was more pronounced than co-verbal touch in positive utterances. These findings suggest that remote touch can be used to attenuate negative conversations or similar circumstances.

The promise of anyone being able to 3D print anywhere relies on both technological advances and incremental shifts in social organizations to trigger changes in human behavior. While much research has focused on how people learn aspects of predefined printing processes, such as expressively utilizing particular design-software (e.g. CAD) and fabrication-machinery (e.g. 3D Printers), this work explores how anyone may gain an understanding of what can be 3D printed through dynamic-processes in computationally-guided exploration of online resources and 3D printing facilities. Investigations surrounding online printing services reveal accessible 3D printing processes that do not require end-users to have experience with design-software or fabrication-machinery, only requiring end-users to specify printable ideas. We present these accessible printing processes alongside associated technologies in a meta-design framework for supporting end-users’ specification of 3D printing ideas. Informed by this framework and a series of formative studies, we designed the website HowDIY to introduce anyone to 3D printing by encouraging and facilitating the intelligent exploration of various online resources. HowDIY was deployed over several weeks with diverse newcomers to 3D printing, validating that intelligent user interfaces can support anyone to participate in the utilization and design of 3D printing tools and processes.

Storytelling is a critical step in the cognitive development of children. Particularly, this requires children to mentally project into the story context and to identify with the thoughts of the characters in their stories. We propose to support free imagination in creative storytelling through an enactment-based approach that allows children to embody an avatar and perform as the story character. We designed our story creation interface with two modes of avatar: the story-relevant avatar and the self-avatar, to investigate the effects of avatar design on the quality of children's creative products. In our study with 20 child participants, the results indicate that self-avatars can create a stronger sense of identification and embodied presence, while story-relevant avatars can provide a scaffold for mental projection.

Touch is a contextual medium for affect conveyance. It is difficult to assign specific meanings to each instance of touch in isolation. However, when engaged within context, touch becomes a powerful and indispensable means for humans to convey feelings of love, concern, and sympathy. This paper explores how remote touch may influence one's affective experience of music, and how this experience is dependent on the relationship between the originator and recipient of the touch. For our study, we designed: (1) A mediated touch system that provides kinesthetic touches through an armband device that is controlled by changes in force applied to an input device, and (2) A touch-media system by which one can record and playback a co-temporal stream of touches with a track of music. We examined: (1) How remote touch may influence a touch recipient's experience of the music, and (2) how the recipients' affective experiences may depend on whether the touch is believed to be generated by intimates (familiar touch) versus by a music performer as part of a performance (unfamiliar touch)? Our study's results revealed that there is a significant correlation between the conditions (familiar, unfamiliar, and no touches) and song types (happy, sad) in both positive and negative affective experiences.

Our work is concerned with how embodied communication involving speech and gestures may be mediated through mobile tele-robotics and augmented reality to support hands-on distance mentoring. Following work in the psycholinguistics of embodied communication (e.g., meaning is expressed through gesture, gaze, and speech), a four design-implement-test-deploy-evaluate study was undertaken. We investigated whether and how powerful multimodal language to support explanation and mentoring may be mediated over distance through the designs.

With increasing focus on integrating 3D printing in educational settings, more emphasis needs to be placed on how to introduce young students to the complexities of the 3D printing process. Inspired by the patrons of 3D printer services, we engage children in a simplified 3D printing process. We conducted a study with two 3rd grade public school science classes over 4 days, where students were tasked to print 3D designs they find online for use in a class presentation. Initial findings identify challenges within this process, and show indications of emerging interest towards 3D printing.

This paper extends the concept of digital literacy and applies it to Making. Through case descriptions, we contribute an understanding of how children can become or fail to become individuals literate in Making within a formal learning context. Our analysis draws from video recordings and other data sources of two 4th grade classrooms in which the students, who had already participated in 1.5 years of more structured 'makified activities', engaged in an open-ended, exploration-based, and playful task that was more in line with the spirit of Making. Student teams were classified as 'high in Making literacy' and 'low in Making literacy', revealing how Making literacy was expressed at the level of skills, mental models, and practices in various ways for different students. Our qualitative analysis demonstrates what burgeoning Making literacy may mean in a public elementary school classroom, paving the way for a vision of a time when Making becomes generalized practice.

Prior research investigating the effects of incorporating Making into educational contexts has been limited to snapshot studies. These studies however do not allow for the investigation of aspects that require longer-term development and nurture. We present a longitudinal study that investigates the effects of Making on children's degree of science self-efficacy, identity formation as possible scientists and engineers, and academic performance in science. Designed interactions with Making technology were integrated into the science curriculum of elementary school classrooms in a public school with a high proportion of students from minority populations for a year. Results showed significant differences between the "Making classrooms" and the control classrooms, and from pre- to post-test on the students' inclination towards science. The results support the promise and potential of incorporating Making into formal schooling on the growth and long-term attitudes of children towards science and STEM in general.

We present Making activities designed and observed within the formal environment of elementary school classrooms. Using a collaborative curriculum-matching design process with teachers, 8 Maker activities and lesson plans were developed, and implemented in Science and Language Arts classrooms of a school with a large percentage of students from underrepresented populations, over the course of 18 weeks during one semester. Coded videos revealed three categories of Maker activities: those that enabled learning, demonstrated learning, and provided learning of the concept itself. Experiences of teachers and students also revealed eagerness to participate, engagement, and exploration in the activities, as observed in a series of analyses. Other themes include the importance of multi-sensory exploration and ownership of self-constructed apparatuses with electronics. The resulting Maker activities and lesson plans offer strategies for familiarizing students with electronic tools and fostering tinkering while remaining true to the learning standards of the classroom.

The study of Making to support children's cognitive or social needs is gaining prominence rapidly. However, little research in Making focuses on children's motor skills and abilities, especially at younger ages. Arguing that Making provides greater benefits when introduced at elementary school-aged, this paper contributes to the understanding of how children aged 8 to 11 manipulate connectors, a key component of Maker kits. We present a review of connectors used in prior work, a theoretical foundation based on the mediated action theory of affordances, and an empirical study of children using 6 different types of connectors with various affordances. Based on quantitative analysis and qualitative video analysis, we present themes that may guide the design of components in future Maker kits for children, with a view towards usability aligned with children's capabilities and mental models.

Based on an intensive semester-long study of designing and implementing curriculum-based Maker activities in 6 classes, this paper presents themes derived from a qualitative analysis of experience data using the framework of Activity Theory. Insights generated contribute to the understanding of the integration of Making into elementary schools at a systemic level.

The purpose of this study is to investigate the role of materiality in tangible interaction design for children. We specifically target children aged 4 to 6 years old because of societal trend of early exposure to touch screen devices for children. This study compares three types of material (felt, wood, and plastic) for tangibles along with touch-based interaction and how the differences implicate child art creation on an iPad application. Through mixed-methods analysis of twenty-six participants' experiences, we use data sources of video recordings, drawings, and interview. The main finding looked at the hardness of materiality for physical and digital drawing tools and its influence to digital art drawings. The findings from this study may be applied to design tangible user interfaces for young children.

At around age nine when social awareness and self-evaluation heighten, children experience a precipitous slump in creative engagement. We propose an enactment-based approach grounded in embodied cognition theories to support children's creative self-efficacy and creative thinking in storytelling during the period of this slump. Our investigation of the approach with 20 children indicated that enactment-based animated authoring improves children's sense of self-efficacy in creating stories, particularly for children with low to medium extraversion, and enables children to produce richer stories, especially for children who scored low on the baseline creativity test.

The purpose of this paper is to investigate how young children associate materiality and meanings and how it can benefit tangible interaction design. To study this, we developed a research prototype, Stampies, that allows playful tangible interactions. Stampies consists of tangible objects made out of different materials (wood, felt, silicone, and plastic) and an iPad drawing application. We describe results from our empirical study involving 19 children aged 4 to 7. The study indicates that children associate materials with meanings through "material essences", feel, and tactile preference. We conclude with design implications for tangible interaction for children.

We propose a new action-transferred design approach by which the benefits of embodied cognition and activity can be realized to enhance spatial learning and usability for navigating virtual spaces. The action-transferred design approach is supported by theories of learning, action-perception, and neuropsychology. These theories help us understand how human action can be transferred to different body parts for improving the usability of interaction techniques and why the acquired spatial knowledge using the transferred action may remain the same independent of the used body parts. The finger-walking-in-place (FWIP) navigation technique is used as a design example to demonstrate the concept of the action-transferred design approach. Leveraging 3D immersive virtual reality technology, we performed an empirical study to evaluate the performance of the action-transferred FWIP navigation technique in terms of spatial knowledge acquisition. The FWIP navigation technique was compared with a full-body-based walking-like (sensor-fusion walking-in-place; SF-WIP) navigation technique and a well-known, convenient (Joystick) navigation technique using a common input device, that is, a wand with a joystick. Both the action-transferred and the full-body-based navigation techniques were more effective for spatial learning than the navigation technique using the common input device. However, only the action-transferred FWIP navigation technique can provide users with the convenience of navigating with their fingers. These results suggest that the action-transferred design approach is useful in designing a navigation technique supporting users' spatial learning performance more effectively and conveniently. Possible design implications for broader applications are discussed and indicate that the action-transferred design approach is worth further study.

Animated stories are challenging to produce for the novice adult, let alone for children. Research into how to facilitate the creation of animations for children is an ongoing effort. In this paper, we propose the concept of performative authoring which taps into the power of children's pretend play to allow them to create animations through body enactments. We review the literature on animation authoring systems focusing on children, propose the new concept of performative authoring that uses enactment as a mode of story authoring, and describe the design of DiME, an exemplar system embodying performative authoring. We report on a pilot study that was conducted with the system.

This paper investigates how the presentation of contextual visual images may influence a child's imagination during the use of authoring systems to create digital content. This issue is particularly significant to understand what supports the child's creativity in authoring systems in creative storytelling. We address specifically children aged 8 to 10 because of the "FourthGrade Slump" phenomenon whereby children experience a decrease in creative engagement at this period. We carried out a study in which children used a storytelling system that allows the user to physically enact stories in a digitallyaugmented space contextualized by either a contextual background image or a blank screen. Using methods of video coding and analysis, we uncovered themes relating differences between children's enactment in the presence of a digital background and without in terms of both the process and product of storytelling. We discuss implications that the themes have for the design of story authoring systems for children.

Slate-type devices allow Individuals with Blindness or Severe Visual Impairment (IBSVI) to read in place with the touch of their fingertip by audio-rendering the words they touch. Such technologies are helpful for spatial cognition while reading. However, users have to move their fingers slowly or they may lose place on screen. Also, IBSVI may wander between lines without realizing they did. In this paper, we address these two interaction problems by introducing dynamic speech-touch interaction model, and intelligent reading support system. With this model, the speed of the speech will dynamically change coping up with the user's finger speed. The proposed model is composed of: 1- Audio Dynamics Model, and 2- Off-line Speech Synthesis Technique. The intelligent reading support system predicts the direction of reading, corrects the reading word if the user drifts, and notifies the user using a sonic gutter to help her from straying off the reading line. We tested the new audio dynamics model, the sonic gutter, and the reading support model in two user studies. The participants' feedback helped us fine-tune the parameters of the two models. Finally, we ran an evaluation study where the reading support system is compared to other VoiceOver technologies. The results showed preponderance to the reading support system with its audio dynamics and intelligent reading support components.

The analysis of multimodal data benefits from meaningful search and retrieval. This paper investigates strategies of searching multimodal data for event patterns. Through three longitudinal case studies, we observed researchers exploring and identifying event patterns in multimodal data. The events were extracted from different multimedia signal sources ranging from annotated video transcripts to interaction logs. Each researcher's data has varying temporal characteristics (e.g., sparse, dense, or clustered) that posed several challenges for identifying relevant patterns. We identify unique search strategies and better understand the aspects that contributed to each.

How does the physicality of an object impact creativity? Our work investigates how objects that possess affordances varying in perceptual and manipulative specificity affect the imagination of the elementary school child. The use of three different kinds of objects was compared: cultural, physical and across three objects (frying pan, pickaxe, lantern) by children enacting parts of a story. We measure the child's 'broader imagination'. Results showed that the physical object type provides better support to extend the imagination of the 9-year old child. There were however gender differences that suggest differing affinity to specific objects. We discuss how our results inform the design of tangibles for enactment-based story authoring systems.

This paper investigates the effects of physical objects as support for imagination in the context of enactive storytelling. More specifically, we target nine-year-old children because of their general disengagement from creative activity, a phenomenon known as the Fourth-grade Slump that arises from a demotivational spiral brought on by social awareness. We study how enactment using physical objects may allow the child to better engage in story imagination. Our study compares the richness of the imagination under three main enactment conditions with objects that have varying degrees of fidelity to referent objects: Cultural objects (physical visual resemblance); Physical objects (similar physical affordances); Arbitrary objects (minimal physical and visual affordances). We employ a mixed-methods analysis to gauge the child's level of broader imagination from three data sources: Enactment videos, drawings and interviews with the children. We found that the object types significantly differ in their support of the imagination, with the object of highest specificity being most effective. Our findings can inform the design of embodied creativity-support systems for children.

In this paper we present the findings of three longitudinal case studies in which a new method for conducting multimodal analysis of human behavior is tested. The focus of this new method is to engage a researcher integrally in the analysis process and allow them to guide the identification and discovery of relevant behavior instances within multimodal data. The case studies resulted in the creation of two analysis strategies: Single-Focus Hypothesis Testing and Multi-Focus Hypothesis Testing. Each were shown to be beneficial to multimodal analysis through supporting either a single focused deep analysis or analysis across multiple angles in unison. These strategies exemplified how challenging questions can be answered for multimodal datasets. The new method is described and the case studies' findings are presented detailing how the new method supports multimodal analysis and opens the door for a new breed of analysis methods. Two of the three case studies resulted in publishable results for the respective participants.

In this paper we introduce the idea of making touch surfaces of mobile devices (e.g. touch phones and tablets) truly tangible for Individuals with Blindness or Severe Visual Impairment (IBSVI). We investigate how to enable IBSVI to fuse tangible landmark patterns with layout of page and location of lexical elements -- words, phrases, and sentences. We designed a tactile overlay that gives tangible feedback to IBSVI when using touch devices for reading. The overlay was tested in a usability study, and the results showed the role of tangibility in leveraging accessibility of touch devices and supporting reading for IBSVI.

ABSTRACT We present a crowd model informed by common ground theory to accommodate high‐level socially aware behavioral realism of characters in crowd simulations. In our approach, group members maintain group cohesiveness by communicating and adapting their behaviors to each other. The resulting character behaviors in animations form a consequential chain interpreted as a coherent story by observers. We demonstrate that our model produces more believable animations from the viewpoint of human observers through a series of user studies. Copyright © 2013 John Wiley & Sons, Ltd.

Human discourse is an embodied activity emerging from the embodied imagery and construction of our talk. Gesture and speech are coexpressive, conveying this imagery and meaning simultaneously. Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. Our goal is to assist Individuals who are Blind or Severely Visually Impaired (IBSVI) to access such instruction/communication. We employ a haptic glove interface to furnish the IBSVI with awareness of the deictic gestures performed by the instructor over the graphic in conjunction with speech. We present a series of studies spanning two years where we show how our Haptic Deictic System (HDS) can support learning in inclusive classrooms where IBSVI receive instruction alongside sighted students. We discuss how the introduction of the HDS was advantageous to all parties: IBSVI, instructor, and sighted students. The HDS created more learning opportunities, increasing mutual understanding and promoting greater engagement.

In this research we designed, developed, and tested a reading system that enables Individuals with Blindness or Severe Visual Impairment (IBSVI) to fuse audio, tactile landmarks, and spatial information in order to read. This system renders electronic text documents on iPad-type devices, and reads aloud each word touched by the user's finger. A tactile overlay on the iPad screen helps IBSVI to navigate a page, furnishing a framework of tactile landmarks to give IBSVI a sense of place on the page. As the user moves her finger along the tangible pattern of the overlay, the text on the iPad screen that is touched is rendered audibly using a text-to-speech bsynthesizer.

Visualizations often use spatial representations to aid understanding, but it is unclear what properties of a spatial information presentation are most important to effectively support cognitive processing. This research explores how spatial layout and view control impact learning and investigates the role of persistent visibility when working with large displays. We performed a controlled experiment with a learning activity involving memory and comprehension of a visually represented story. We compared performance between a slideshow-type presentation on a single monitor and a spatially distributed presentation among multiple monitors. We also varied the method of view control (automatic vs. interactive). Additionally, to separate effects due to location or persistent visibility with a spatially distributed layout, we controlled whether all story images could always be seen or if only one image could be viewed at a time. With the distributed layouts, participants maintained better memory of the associated locations where information was presented. However, learning scores were significantly better for the slideshow presentation than for the distributed layout when only one image could be viewed at a time.

This paper investigates whether and how digitally mediated social touch (remote touch) may influence the sense of connectedness toward a speaker and the emotional experience of what is being communicated. We employ an 'augmented' storytelling methodology where we manipulate the modality of an 'emotive' channel that accompanies the speech, and the contextual expectation of the listener. Comparing a remote upper-arm touch against a similarly timed flashing light, we explore the importance of the touch modality in affect conveyance. Our second manipulation involves two cover stories where the listener is told that the touch or flashing light is triggered either by the storyteller expressively squeezing a touch input device while speaking, or by measured 'high points' in the mental state of the storyteller. Our results show that the story accompanied by communicative touch resulted in a significant increase in the sense of connectedness with the storyteller over the speech-only condition, and a trend toward greater affective conveyance.

Technological devices are being rapidly adopted into schools for education, but we have limited understanding of the value and ways through which the devices can benefit learning. As opposed to research placing the value of these devices in terms of digitality, we make use of theories of embodiment to under-stand how the physicality of the devices can support learning and sensemaking. We conducted a month-long study to collect data on students' strategies, patterns, attitudes and behaviors toward the use of a suite of devices, for the completion of a course assignment. Themes uncovered include the objectification of information, the immediate awareness of possibilities, an expectation of interaction, coherence of interaction and territorialization of technology spaces. We present a model for the role of physicality of devices with regards to educational activities, and argue for the need to construct a digital ecology to provide a cohesive experience of learning.

In this paper, we introduce a novel interaction model for reading text documents depending on situated touch. This interaction modality targets Individuals with Blindness or Severe Visual Impairment (IBSVI). We aim to provide IBSVI with an effective reading tool that enables them to use their spatial abilities while reading. We used an iPad device and augmented it with a static tactile overlay to display the text to serve as a kind of spatial landmark space for the IBSVI. The text is rendered audibly in response to user's touch. Two user studies, with IBSVI participants, were conducted to test the system. The first study was a laboratory-controlled study, and the second one was a longitudinal study. These studies showed that while the approach is new to the users' experience, it is a promising direction to enable self-paced spatial reading for IBSVI.

There are a multitude of annotated behavior corpora (manual and automatic annotations) available as research expands in multimodal analysis of human behavior. Despite the rich representations within these datasets, search strategies are limited with respect to the advanced representations and complex structures describing human interaction sequences. The relationships amongst human interactions are structural in nature. Hence, we present Structural and Temporal Inference Search (STIS) to support search for relevant patterns within a multimodal corpus based on the structural and temporal nature of human interactions. The user defines the structure of a behavior of interest driving a search focused on the characteristics of the structure. Occurrences of the structure are returned. We compare against two pattern mining algorithms purposed for pattern identification amongst sequences of symbolic data (e.g., sequence of events such as behavior interactions). The results are promising as STIS performs well with several datasets.

This paper investigates a technique for the discovery of temporal behavior models within multimedia event data. Advancements in both technology and the marketplace present us the opportunity for research in analysis of situated human behavior using video and other sensor data (media streams). By situated analysis, we mean the study of behavior in time as opposed to looking at behavior in the form of aggregated data divorced from how they occur in context. Human and social scientists seek to model behavior captured in media, and these data may be represented in a multi-dimensional event data space derived from media streams. The knowledge of these scientists (experts) is a valuable resource which can be leveraged to search this space. We propose a solution that incorporates the expert in an iteratively, interactive data-driven discovery process to evolve a desired behavior model. We test our solution's accuracy on a multimodal meeting corpus with a progressive three tiered approach.

Large crowds are seldom made up solely of a mass of individuals. They typically also include large collections of small groups. However the most of existing approaches to crowd modeling treat a crowd either as a collection of isolated individuals, each maintaining its own goal, or as an aggregated entity in which large number of individuals share the same goal and behavior pattern.

This paper describes a sensor-fusion-based wireless walking-in-place (WIP) interaction technique. We devised a new human-walking detection algorithm that is based on a sensor-fusion using both acceleration and magnetic sensors integrated within a smart phone. Our sensor-fusion approach can be useful for the cases when the detection capability of a single sensor is limited to a certain range of walking speeds, when a system power source is limited, and/or when computation power is limited. The proposed algorithm is versatile enough to handle possible data-loss and random delay in the wireless communication environment, resulting in reduced wireless communication load and computation overhead. The initial study demonstrated that the algorithm can detect dynamic speeds of human walking. The algorithm can be implemented on any mobile device equipped with magnetic and acceleration sensors.

The Fourth Grade Slump, whereby children's creativity drops precipitously at around fourth grade, is a developmental phenomenon that begs for research to be done on ways to nurture children's creativity. We posit that due to its form and formal features, the use of the animated medium for creative activities can positively sustain the child's creativity throughout the Slump. We present a study that investigated how the animated medium (animation) mediates and influences the creative process of children in the third and fourth grade, using the digital print medium (storybook) as comparison. Analysis of the process was done at the structural and production level using mainly the method of discourse analysis.

Multimodal analysis of human behavior is ultimately situated. The situated context of an instance of a behavior phenomenon informs its analysis. Starting with some initial (user-supplied) descriptive model of a phenomenon, accessing and studying instances in the data that are matches or near matches to the model is essential to refine the model to account for variations in the phenomenon. This inquiry requires viewing the instances within-context to judge their relevance. In this paper, we propose an automatic processing approach that supports this need for situated analysis in multimodal data. We process events on a semi-interval level to provide detailed temporal ordering of events with respect to instances of a phenomenon. We demonstrate the results of our approach and how it facilitates and allows for situated multimodal analysis.

10.1145/1753326.1753424

As an important approach to ensure safety and naturalness, compliant motion is already implemented in large-size robots, but it is still an undeveloped area in child-oriented robots as it calls for a lightweight and compact solution compared with large-size robots. In this paper, we proposed the design of a social robot which aims at conducting safe and playful Human-Robot Interaction (HRI), especially with children. We built a teddy bear robot prototype based on hybrid passive-active compliant system which consists of flexible joints as passive part and compliant motion controller as active part. The compliant controller detects external perturbation through motor state variables, therefore force and torque sensors could be omitted to keep overall system compact. Experiments conducted in typical HRI scenarios showed that the hybrid passive-active compliant system enabled our robot to conduct safer and more interactive physical interaction compared with robot under traditional control method.

This work investigates using n-gram processing and a temporal relation encoding to providing relational information about events extracted from media streams. The event information is temporal and nominal in nature being categorized by a descriptive label or symbolic means and can be difficult to relationally compare and give ranking metrics. Given a parsed sequence of events, relational information pertinent to comparison between events can be obtained through the application of n-grams techniques borrowed from speech processing and temporal relation logic. The procedure is discussed along with results computed using a representative data set characterized by nominal event data.

Touch is a unique channel in affect conveyance. A significant aspect of this uniqueness is that the relation of touch to affect is immediate, without the need for symbolic encoding and decoding. However, most pioneering research work in developing remote touch technologies, result in the use of touch as a symbolic channel either by design or user decision. We present a review of relevant psychological and sociological literature of touch and propose a model of immediacy of the touch channel for conveyance of affect. We posit that the strategic provision of contextualizing channels will liberate touch to assume its role in affect conveyance. Armed with this analysis, we propose two design guidelines: first, the touch channel needs to be coupled with other communication channels to clarify its meaning; second, encourage the use touch as an immediate channel by not assigning any symbolic meaning to touch interactions. We proceed to describe our haptic interface design based on these guidelines. Our in-lab experiment shows that remote touch reinforces the meaning of a symbolic channel reducing sadness significantly and showing a trend to reduce general negative mood and to reinforce joviality.

We explore some of the characteristics of multimodal input interaction spaces for notification systems within a multi-tasking environment like a command and control center using two promising interaction methods: gestures and touch based input through a laboratory experiment comparing both techniques. Results of our study suggest that gestures are better suited for multi-tasking situations because they are less interruptive than touch interaction to users' primary tasks and are subjectively preferred by users in certain situations.

Tabletop hand tracking techniques have evolved much during the last few years from single to multiple cameras, offering users an improved interactive experience. MirrorTrack is one of such techniques. This paper demonstrates the comparison of accuracy between MirrorTrack and top-down approach, which is generally used for table top tasks. In this paper, we focus on the comparison of distance errors in finger trajectory, and clicking errors by manual monitoring.

Detection of meeting events is one of the most important tasks in multimodal analysis of planning meetings. Speaker detection is a key step for extraction of most meaningful meeting events. In this paper, we present an approach of speaker localization using combination of visual and audio information in multimodal meeting analysis. When talking, people make a speech accompanying mouth movements and hand gestures. By computing correlation of audio signals, mouth movements, and hand motion, we detect a talking person both spatially and temporally. Three kinds of features are extracted for speaker localization. Hand movements are expressed by hand motion efforts; audio features are expressed by computing 12 mel-frequency cepstral coefficients from audio signals, and mouth movements are expressed by normalized cross-correlation coefficients of mouth area between two successive frames. A time delay neural network is trained to learn the correlation relationships, which is then applied to perform speaker localization. Experiments and applications in planning meeting environments are provided.

iPhone and iPod Touch are multi-touch handheld devices that provide new possibilities for interaction techniques. We describe iPhone/iPod Touch implementation of a navigation interaction technique originally developed for a larger multi-touch device (i.e. Lemur). The interaction technique implemented on an iPhone/iPod Touch was used for navigation tasks in a CAVE virtual environment. We performed a pilot study to measure the control accuracy and to observe how human subjects respond to the interaction technique on the iPhone and iPod Touch devices. We used the preliminary results to improve the design of the interaction technique.

When combined with the organizational space provided by a simple table, physical notecards are a powerful organizational tool for information analysis. The physical presence of these cards affords many benefits but also is a source of disadvantages. For example, complex relationships among them are hard to represent. There have been a number of notecard software systems developed to address these problems. Unfortunately, the amount of visual details in such systems is lacking compared to real notecards on a large physical table; we look to alleviate this problem by providing a digital solution. One challenge with new display technology and systems is providing an efficient interface for its users. In this paper we look at comparing different interaction techniques of an emerging class of organizational systems that use high-resolution tabletop displays. The focus of these systems is to more easily and efficiently assist interaction with information. Using PDA, token, gesture, and voice interaction techniques, we conducted a within subjects experiment comparing these techniques over a large high-resolution horizontal display. We found strengths and weaknesses for each technique. In addition, we noticed that some techniques build upon and complement others.

The estimation of the direction of visual attention is critical to a large number of interactive systems. This paper investigates the cross-modal relation of the position of one's feet (or standing stance) to the focus of gaze. The intuition is that while one CAN have a range of attentional foci from a particular stance, one may be MORE LIKELY to look in specific directions given an approach vector and stance. We posit that the cross-modal relationship is constrained by biomechanics and personal style. We define a stance vector that models the approach direction before stopping and the pose of a subject's feet. We present a study where the subjects' feet and approach vector are tracked. The subjects read aloud contents of note cards in 4 locations. The order of `visits' to the cards were randomized. Ten subjects read 40 lines of text each, yielding 400 stance vectors and gaze directions. We divided our data into 4 sets of 300 training and 100 test vectors and trained a neural net to estimate the gaze direction given the stance vector. Our results show that 31% our gaze orientation estimates were within 5°, 51% of our estimates were within 10°, and 60% were within 15°. Given the ability to track foot position, the procedure is minimally invasive.

In this paper, we study the calibration of head pose estimation in stereo camera setting for meeting room video event analysis. Head pose information infers the direction of attention of the subjects in video, therefore is valuable for video event analysis/indexing, especially in meeting room scenario. We are developing a multi-modal meeting room data analyzing system for studying meeting room interaction dynamics, in which head pose estimation is one of the key components. As each subject in the meeting room can be observed by a pair of stereo cameras, we do 2D head tracking for the subject in each camera, and the 3D coordinate of the head can be obtained by triangulation. The 3D head pose is estimated in one of the camera coordinate system, we develop a procedure to accurately convert the estimated 3D pose in the camera coordinate system to that in the world coordinate system. In the experiment, visualization of the estimated head pose and location in world coordinate system verifies the soundness of our design. The estimated head pose and 3D location of the subjects in the meeting room allows further analysis of meeting room interaction dynamics, such as F-formation, floor-control, etc.

Students who are blind are typically one to three years behind their seeing counterparts in mathematics and science. We posit that a key reason for this resides in the inability of such students to access multimodal embodied communicative behavior of mathematics instructors. This impedes the ability of blind students and their teachers to maintain situated communication. In this paper, we set forth the relevant phenomenological analyses to support this claim. We show that mathematical communication and instruction are inherent embodied; that the blind are able to conceptualize visuo-spatial information; and argue that uptake of embodied behavior is critical to receiving relevant mathematical information. Based on this analysis, we advance an approach to provide students who are blind with awareness of their teachers' deictic gestural activity via a set of haptic output devices. We lay forth a set of open research question that researcher in multimodal interfaces may address.

Students who are blind are typically one to three years behind their seeing counterparts in mathematics and science. We posit that a key reason for this resides in the inability of such students to access multimodal embodied communicative behavior of mathematics instructors. This impedes the ability of blind students and their teachers to maintain situated communication. In this paper, we set forth the relevant phenomenological analyses to support this claim. We show that mathematical communication and instruction are inherent embodied; that the blind are able to conceptualize visuo-spatial information; and argue that uptake of embodied behavior is critical to receiving relevant mathematical information. Based on this analysis, we advance an approach to provide students who are blind with awareness of their teachers' deictic gestural activity via a set of haptic output devices. We lay forth a set of open research question that researcher in multimodal interfaces may address.

This paper presents an approach for registering aerial images taken at different time, viewpoints, or heights. Different from conventional image registration algorithms, our approach does not need image matching or correspondence. In this approach, we extract a number of corner features as the basis for registration and create a number of image patches with the corner points as centers on both reference and observed images. In order to let the corresponding patches cover same scene, we use a circle which the radius can be changed as the shape of the image patches. In this way, the image patches can handle the case in which there are rotation and scaling at the same time between reference and observed images. With the orientation differences of patches between these two images, we create an angle histogram with a voting procedure. The rotation angle between the two images can be determined by seeking the orientation difference that corresponds to the maximum peak in the histogram. Once we get the rotation angle, we seek back for the two corresponding patches which the value of orientation difference is the same as the rotation angle. The ratio of radii of these two patches is the value of the scaling. The proposed approach can handle the situation of large rotation and scaling between reference and observed images. It is applied to real aerial images and the results are very satisfying.

We present an adaptive skin color model for hand gesture tracking which is applied to cross-modal analysis of planning meetings. We build a skin color model with the Gaussian distribution and a skin color filter for each participant in meetings. By combining with the vector coherence mapping (VCM) algorithm, we track hand motion and obtain 3D trajectories. The hand gesture stream is extracted from hand motion trajectories. Different skin color models are created for different people to handle the differences of skin color. We update each model dynamically to adapt changes of environments. A parallel system has been implemented to track and extract hand motion trajectories. Examples of hand motion gesture tracking in meeting environments are provided. The applications of the adaptive skin color model can increase the speed of hand tracking

Gesture and speech are part of a single human language system. They are coexpressive and complementary channels in the act of speaking. Whereas speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, our work on oscillatory gestures and multimodal discourse is discussed. Our new techniques of analyzing hand gestures in the frequency domain are described. By tracking an individual's hands during a speech, hand motion trajectory signals are extracted from real video datasets. Our wavelet-based approach in gestural oscillation extraction is presented as frequency ridges in a frequency-time space. Wavelet ridges are extracted from responses of wavelet analysis. These wavelet ridges are employed to characterize frequency properties of hand motion trajectory signals. Hand motion oscillatory gestures can be extracted from these frequency properties. The potential of such computational cross-modal language analysis is motivated by performing a microanalysis of 2 video datasets. In the first dataset, a participant describes her living space to an interlocutor. In the second, a participant describes her action plan to an interlocutor. The ability of our algorithm to extract gestural oscillations is demonstrated, and the way that oscillatory gestures reveal portions of the discourse structure is shown.

In video based cross-model analysis of planning meeting, the meeting events are recorded by multiple cameras distributed in the entire meeting room. Subject's hand gestures, hand motion, head orientations, gaze targets, body poses are very important for the meeting event analysis. In order to register everything to the same global coordinate system, build 3D model, get 3D data from the video, we need to create a proper meeting room configuration and calibrate all cameras to obtain their intrinsic and extrinsic parameters. However, the calibration of multiple cameras distributed in the entire meeting room area is a challenging task because it is impossible to let all cameras in the meeting room see a reference object at the same time and wide field-of-view cameras suffer under radial distortion. In this paper, we propose a simple approach to create a good meeting room configuration and calibrate multiple cameras in the meeting room. The proposed approach includes several steps. First, we create stereo camera pairs according to the room configuration and the requirements of the targets, the participants of the meeting. Second, we apply Tsai's algorithm to calibrate each stereo camera pair and obtain the parameters in its own local coordinate system. Third, we use Vicon motion capture data to transfer all local coordinate systems of stereo camera pairs into a global coordinate system in the meeting room. We can obtain the positions, orientations, and parameters for all cameras in the same global coordinate system, so that we can register everything into this global coordinate system. Next, we do calibration error analysis for the current camera and meeting room configuration. We can obtain error distribution in the entire meeting room area. Finally, we improve the current camera and meeting room configuration according to the error distribution. By repeating these steps, we can obtain a good meeting room configuration and parameters of all cameras for this room configuration.

We describe an agent-based approach to the visual tracking of human hands and head that represents a very useful "middle ground" between the simple model-free tracking and the highly constrained model-based solutions. It combines the simplicity, speed, and flexibility of tracking without using explicit shape models with the ability to utilize domain knowledge and to apply various constraints characteristic of more elaborate model-based tracking approaches. One of the key contributions of our system, called AgenTrac, is that it unifies the power of data-fusion (cue integration) methodologies with a well-organized extended path-coherence-resolution approach designed to handle crossing trajectories of multiple objects. Both approaches are combined in an easily configurable framework. We are not aware of any path-coherence or data-fusion solution in the computer vision literature that equals the breadth, generality, and flexibility of our approach. The AgenTrac system is not limited to tracking only human motion; in fact, one of its main strengths is that it can be easily reconfigured to track many types of objects in video sequences. The multiagent paradigm simplifies the application of basic domain-specific constraints and makes the entire system flexible. The knowledge necessary for effective tracking can be easily encoded in agent hierarchies and agent interactions.

The study of embodied communication requires access to mul-tiple data sources such as multistream video and audio, various derived and meta-data such as gesture, head, posture, facial expression and gaze information. The common element that runs through these data is the co-temporality of the multiple modes of behavior. In this paper, we present the multimedia Visualization for Situated Temporal Analysis (MacVisSTA) system for the analysis of multimodal human communication through video, audio, speech transcriptions, and gesture and head orientation data. The system uses a multiple linked representation strategy in which different rep-resentations are linked by the current time focus. In this framework, the multiple display components associated with the disparate data types are kept in synchrony, each compo-nent serving as both a controller of the system as well as a display. Hence the user is able to analyze and manipulate the data from different analytical viewpoints (e.g. through the time-synchronized speech transcription or through motion segments of interest). MacVisSTA supports analysis of the synchronized data at varying timescales. It provides an annotation interface that permits users to code the data into 'music-score' objects, and to make and organize multimedia observa-tions about the data. Hence MacVisSTA integrates flexible visualization with annotation within a single framework. An XML database manager has been created for storage and search of annotation data. We compare the system with other existing annotation tools with respect to functionality and interface design. The software runs on Macintosh OS X computer systems.

This paper presents an approach for the extraction of vasculature from a volume of Magnetic Resonance Angiography (MRA) images by using a 3D wave propagation and traceback mechanism. We discuss both the theory and the implementation of the approach. Using a dual-sigmoidal filter, we label each voxel in the MRA volume with the likelihood that it is within a vessel. Representing the reciprocal of this likelihood image as an array of refractive indices, we propagate a digital wave through the volume from the base of the vascular tree. This wave 'washes' over the vasculature and extracts the vascular tree, ignoring local noise perturbations. While the approach is inherently SIMD we present an efficient sequential algorithm for the wave propagation, and discuss the traceback algorithm. We demonstrate the effectiveness of our integer image neighborhood-based algorithm and its robustness to image noise.

Vessel segmentation algorithms are critical components of circulatory blood vessel analysis systems. We present a survey of vessel extraction techniques and algorithms, putting the various approaches and techniques in perspective by means of a classification of the existing research. While we target mainly the extraction of blood vessels, neurovascular structure in particular we also review some of the segmentation methods for the tubular objects that show similar characteristics to vessels. We divide vessel segmentation algorithms and techniques into six main categories: (1) pattern recognition techniques, (2) model-based approaches, (3) tracking-based approaches, (4) artificial intelligence-based approaches, (5) neural network-based approaches, and (6) miscellaneous tube-like object detection approaches. Some of these categories are further divided into sub-categories. A table compares the papers against such criteria as dimensionality, input type, preprocessing, user interaction, and result type.

Curvature-based surface features are well suited for use in multimodal medical image registration. The accuracy of such feature-based registration techniques is dependent upon the reliability of the feature computation. The computation of curvature features requires second derivative information that is best obtained from a parametric surface representation. We present a method of explicitly parameterizing surfaces from volumetric data. Surfaces are extracted, without a global thresholding, using active contour models. A Mong basis for each surface patch is estimated and used to transform the patch into local, or parametric, coordinates. Surface patches are fit to first a bicubic polynomial and second to a Bezier polynomial. The bicubic polynomial is fit in local coordinates using least squares solved by singular value decomposition. Bezier polynomial is fit using de Casteljau algorithm. We tested our method by reconstructing surfaces from the surface model and analytically computing Gaussian and mean curvatures. The model was tested on analytical and medical data and the results of both methods are compared.

Gesture and speech are part of a single human language system. They are co-expressive and complementary channels in the act of speaking. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established contemporality of gesture and speech, we discuss our work on oscillatory gestures and speech. We present our wavelet-based approach in gestural oscillation extraction as geodesic ridges in frequency-time space. We motivate the potential of such computational cross-modal language analysis by performing a micro analysis of a video dataset in which a subject describes her living space. We demonstrate the ability of our algorithm to extract gestural oscillations and show how oscillatory gestures reveal portions of the discourse structure.

The catchment feature model (CFM) addresses two questions in multimodal interaction: how do we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We discuss the need for our model, motivate the CFM from psycholinguistic research, and present the model. In contrast to 'whole gesture' recognition, the CFM applies a feature decomposition approach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for CFM-based research, and cite three concrete examples of catchment features (CF), and propose new directions of multimodal research based on the model.

Gesture and speech are part of a single human language system. They are co-expressive and complementary channels in the act of speaking. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. Proceeding from the established cotemporality of gesture and speech, we discuss our work on oscillatory gestures and speech. We present our wavelet-based approach in gestural oscillation extraction as geodesic ridges in frequency-time space. We motivate the potential of such computational cross-modal language analysis by performing a micro analysis of a video dataset in which a subject describes her living space. We demonstrate the ability of our algorithm to extract gestural oscillations and show how oscillatory gestures reveal portions of the discourse structure.

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.

Computer supported cooperative work (CSCW) research is the discipline that explores how to utilize computing and networking technologies to facilitate cooperation and collaboration among people. A variety of research has been conducted on CSCW architecture. However, how to ensure CSCW system effectiveness and efficiency on supporting collaboration remains a challenge. In this paper we present a rule mitigated framework for CSCW applications. This research contributes to the state of the art by presenting a robust and comprehensive architecture to support distributed collaboration.

Curvature-based surface features are well suited for use in multimodal medical image registration. The accuracy of such feature-based registration techniques is dependent upon the reliability of the feature computation. The computation of curvature features requires second derivative information that is best obtained from a parametric surface representation. We present a method of explicitly parameterizing surfaces from volumetric data. Surfaces are extracted, without a global thresholding, using active contour models. A monge/spl acute/ basis for each surface patch is estimated and used to transform the patch into local, or parametric, coordinates. Surface patches are fit to a bicubic polynomial in local coordinates using least squares solved by singular value decomposition. We tested our method by reconstructing surfaces from the surface model and analytically computing Gaussian and mean curvatures. The model was tested on analytical and medical data.

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.

Parkinson’s disease (PD) belongs to a class of neurodegenerative diseases that affect the patient’s speech, motor, and cognitive capabilities. All three deÞcits affect the multimodal communication channels of speech and gesture. We presenta study on the changes in speech pause patterns and gesture holds before and after treatment. We present the results of a pilot study of two Idiopathic PD patients who have undergone Lee Silverman Voice Treatment (LSVT). We show that therewas a consistentchangein the location of pauses with respect to semantic sentential utterance units. After treatment, the number and duration of pauses within sentential units decreased while the inter sentential pauses increased. This indicates reduction in hesitation and increase in speech phrasing. We also found a decrease in the number of sentence repairs and the time spent in repairs. For gesture, we found that non-rest holds intersecting with within-sentence pauses appears to decline after treatment, as does the ratio of rest holds during speechagainst rest holds between sentences. While the work is preliminary, these patterns suggest that multimodal discourse characteristics might provide accessto the underlying cognitive state under the load of narrative discourse.

Human multimodal communicative behaviorsform a tightly integrated whole. By matching up gestural features with a carefully time-tagged transcription of the speech, we can observe how gesture features and discourse unit transitions cohere. Space usage SU is a key gestural component. We summarize the theory of SU. In our experimentswhere subjects make action plans around a terrain map, such SU become key organizational loci around which the discourse may be built. Our vision-based approach extracts ‘SU histograms’ from stereo video describing the locus of motion of a speaker’s dominant hand. An ‘fuzzy correlation’ of these histograms yields a correlation space in which similar SU is clustered. By locating the cluster transitions we can locate topical shiftsinthediscourse. Weshowresultsbycomparingthetransitions extracted from a sentential codingwith a psycholinguisticsemantic coding. We do the same with a uniform distributed time units and demonstratethe ability to recoverdiscoursetransitions.

Our approach is motivated by the conviction that gesture and speech are coexpressive of the underlying dynamic ideation that drives human communication. As such, transitions and cohesions is gestural behavior would inform us as to the discourse conceptualization. In this paper, we examine the role of motion symmetries of twohanded gestures in the structuring of speech. We employ a set of hand motion traces extracted from video and compute the correlation of these traces. The signs and magnitudes of the correlation coefÞcients computed in the cardinal directions of the subject’s torso (lateral and vertical in this work) characterize the symmetries. We employ a windowed computation approach that permits a balance between temporal resolution and robustness to noise. The resulting correlation proÞles are merged according to a temporal proximity rule. We apply this analysis to two conversational video sequences. A detailed analysis of the Þrst sequence reveals the persistence of gestural imagery between semantically-similar discourse pieces. A symmetry transition analysis is applied to the second dataset and compared against a manually generated discourse segmentation to demonstrate the potential of cross-modal discourse segmentation. 1.

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.

Parkinson's disease (PD) belongs to a class of neurodegenerative diseases that affect both the patient's speech and motor capabilities. To date, PD diagnosis and the determination of disease progress and treatment efficacy is based entirely on the subjective observation of a trained physician. We present the results of a pilot study of two Idiopathic PD patients who have undergone Lee Silverman Voice Treatment (LSVT). It has been observed subjectively that gestural performance of patients improve in tandem with speech improvements after LSVT. It is hypothesized that these improvements are taking place at a neurological level. Measurements of speech and gesture suggest that LSVT improves the quality of both gesticulation and speech.

Human multimodal communicative behaviors form a tightly integrated whole. We have present a paradigm multimodal analysis in natural discourse based on a feature decompositive psycholinguistically derived model that permits us to accessthe underlying structure and intent of multimodal communicative discourse. We outline the psycholinguistics that drive our paradigm, the Catchment concept that facilitates our getting a computational handle on discourse entities, and summarize some approaches and results that realize the vision. We show examples of such discoursestructuring features as handedness, types of symmetry, gaze-atinterlocutor, and hand ‘origos’. Such analysis is an alternative to the ‘recognition of one discrete gesture out of stylized whole gesturemodels’ paradigm.

This paper presents an approach for the extraction of vasculature from angiography images by using a wave propagation and traceback mechanism. We discuss both the theory and the implementation of the approach. Using a dual-sigmoidal filter, we label each pixel in an angiogram with the likelihood that it is within a vessel. Representing the reciprocal of this likelihood image as an array of refractive indexes, we propagate a digital wave through the image from the base of the vascular tree. This wave "washes" over the vasculature, ignoring local noise perturbations. The extraction of the vasculature becomes that of tracing the wave along the local normals to the waveform. While the approach is inherently single instruction stream multiple data stream (SIMD), we present an efficient sequential algorithm for the wave propagation and discuss the traceback algorithm. We demonstrate the effectiveness of our integer image neighborhood-based algorithm and its robustness to image noise.

Curvature-based surface features are well suited for use in multimodal medical image registration. The accuracy of such feature-based registration techniques is dependent upon the reliability of the feature computation. The computation of curvature features requires second derivative information that is best obtained from a parametric surface representation. The authors present a method of explicitly parametrizing surfaces from volumetric data. Surfaces are extracted, without a global thresholding, using active contour models. A monge basis for each surface patch is estimated and used to transform the patch into local, or parametric, coordinates. Surface patches are fit to a bicubic polynomial in local coordinates using least squares solved by singular value decomposition. The authors tested their method by reconstructing surfaces from the surface model and analytically computing gaussian and mean curvatures. The model was tested on analytical and medical data.

Psycholinguistic evidence has established the complementary nature of the verbal and non-verbal aspects of human expression. We present our findings in the detection of these cites in interaction. We use the psycholinguistic device known as the 'catchment' as the locus of integration of gesture, speech and gaze components. We videotape conversation elicitation experiments in which subjects convey complex spatial plans to an interlocutor using a calibrated three-camera setup. We extract the gestural motion of both hands, gaze direction, and voiced units in the discourse and compare these with transcripts generated by expert microanalysis of the video. Our results show the complementary nature of these communicative modalities. Where there is ambiguity in the structure of one modality (such as in haptologies or owing to noise in the audio signal), other modalities provide evidence for correct segmentation.

We propose a model to interpret neurovascular X-ray angiogram (XRA) images interactively. This attentionally based interactive model (AIM) exploits human interaction as part of the solution. AIM posits two channels of interaction: context ("what to look for") and focus-of-attention ("where to look") as the locus of spatial information exchange between the user and the machine. In an AIM system, the user specifies a context (e.g., a carotid vessel) and directs the attentional spotlight to focus machine processing. AIM involves the user with the computer as integral partners and facilitates varying degrees of human intervention in the process. A hierarchy of context abstractions permits the system to function more autonomously (doing high-level tasks like extracting an arterial vessel) in routine interpretation and to require more user intervention (e.g., locating arterial wall boundaries) as the image complexity increases. This is especially important in medical imaging where the medical professional must have ultimate control and confidence in the system. Such technology can have a significant impact on the design of radiological systems.

Presents a recursive inductive learning scheme that is able to acquire hand pose models in the form of disjunctive normal form expressions involving multivalued features. Based on an extended variable-valued logic, our rule-based induction system is able to abstract compact rule sets from any set of feature vectors describing a set of classifications. The rule bases which satisfy the completeness and consistency conditions are induced and refined through five heuristic strategies. A recursive induction learning scheme in the RIEVL algorithm is designed to escape local minima in the solution space. A performance comparison of RIEVL with other inductive algorithms, ID3, NewID, C4.5, CN2, and HCV, is given in the paper. In the experiments with hand gestures, the system produced the disjunctive normal form descriptions of each pose and identified the different hand poses based on the classification rules obtained by the RIEVL algorithm. RIEVL classified 94.4 percent of the gesture images in our testing set correctly, outperforming all other inductive algorithms.

Hand pose recognition for gestural systems must be able to handle variations in user hand anatomy, perspective effects, and the idiosyncrasies of individual gesture presentation. We present an inductive learning system that is able to derive a rulebase of disjunctive normal form formulae in which each DNF describes a hand pose, and each conjunct within the DNF constitutes a single rule. The system applies a suite of feature detectors during learning and is able to determine a salient subset of features for a given set of hand poses. Only the reduced feature set needs to be computed at recognition time. The rule-based induction system is capable of both exact and flexible matching. The latter produces a list of hand poses ordered by goodness of match when presented with a new hand pose image. We present the results of our experiments in which we trained our system with 931 instances of 20 different hand poses. Our system produced compact rule sets and had a recognition rate of 94%.

An abstraction-based paradigm that makes explicit the process of imposing assumptions on data is discussed. The units of abstraction are models in which levels of abstraction are determined by the degree of assumption necessary for their application. A general-to-specific refinement process provides a mechanism to proceed gracefully through the abstraction hierarchy. This strategy was applied to the recognition and pose determination of objects comprising simple and compound cylindrical and planar surfaces in dense range data. A method of computing reliable Gaussian and mean curvature sign-map descriptors from the polynomial approximations of surfaces is demonstrated. A means for determining the pose of constructed geometric forms whose algebraic surface descriptions are nonlinear in terms of their orienting parameters is developed. It is shown that biquadratic surfaces are suitable companion-linear forms for cylinder approximation and parameter estimation. The estimates provide the initial parametric approximations necessary for a nonlinear regression stage to fine tune the estimates by fitting the actual nonlinear form to the data.>

The authors present an interactive map conversion system which combines a human operator's high level reasoning with machine perception under the Human-Machine Perceptual Cooperation (HMPC) paradigm. HMPC defines two channels of interaction: the focus of attention (FOA) by which the user directs the attention of machine perception, and context. As the user moves the FOA across a raster map display via a pointing device, a smart cursor operates proactively on the data highlighting objects for extraction. The FOA permits foveal emphasis, enabling the user to vary motor precision with map clutter. HMPC provides for contexts at four levels of abstraction. This permits the efficiency of the system to degrade gracefully as data quality worsens. They also present a boundary-based line follower which computes line thickness, and an isolated symbol extractor based on feature-vectors.>

Phase portraits are a powerful mathematical model for describing oriented textures. An isotangent-based approach is presented which is a linear formulation to the problem, to locate the critical points and compute the parameter sets of this model for the nonsingular two-dimensional first-order phase portraits. The authors classify flow patterns by Jordan canonical forms of the characteristic matrix made up of the estimated parameters. For these systems, they prove that all the isotangent curves are straight lines which intersect at a critical point. They also apply least median of squares (LMS) estimators to find the isotangent lines and locate the critical point. A linear regression technique is used to estimate the parameters of the two-dimensional first-order phase portrait of a given flow pattern. Results of applying the algorithm to synthetic and real images are presented.>