Peer agents can supplement real-time collaborative learning in asynchronous online courses. Constructive Controversy (CC) theory suggests that humans deepen their understanding of a topic by confronting and resolving controversies. This study explores whether CC’s benefits apply to LLM-based peer agents, focusing on the impact of agents’ disputatious behaviors and disclosure of agents’ behavior designs on the learning process. In our mixed-method study (n=144), we compare LLMs that follow detailed CC guidelines (regulated) to those guided by broader goals (unregulated) and examine the effects of disclosing the agents’ design to users (transparent vs. opaque). Findings show that learners’ values influence their agent interaction: those valuing control appreciate unregulated agents’ willingness to cease push-back upon request, while those valuing intellectual challenges favor regulated agents for stimulating creativity. Additionally, design transparency lowers learners’ perception of agents’ abilities. Our findings lay the foundation for designing effective collaborative peer agents in isolated educational settings.

The Steering Law has long been a fundamental model in predicting movement time for tasks involving navigating through constrained paths, such as in selecting sub-menu options, particularly for straight and circular arc trajectories. However, this does not reflect the complexities of real-world tasks where curvatures can vary arbitrarily, limiting its applications. This study aims to address this gap by introducing the total curvature parameter K into the equation to account for the overall curviness characteristic of a path. To validate this extension, we conducted a mouse-steering experiment on fixed-width paths with varying lengths and curviness levels. Our results demonstrate that the introduction of K significantly improves model fitness for movement time prediction over traditional models. These findings advance our understanding of movement in complex environments and support potential applications in fields like speech motor control and virtual navigation.

Sensing touch on arbitrary surfaces has long been a goal of ubiquitous computing, but often requires instrumenting the surface.Depth camera-based systems have emerged as a promising solution for minimizing instrumentation, but at the cost of high touch-down detection error rates, high touch latency, and high minimum hover distance, limiting them to basic tasks.We developed HaloTouch, a vision-based system which exploits a multipath interference effect from an off-the-shelf time-of-flight depth camera to enable fast, accurate touch interactions on general surfaces.HaloTouch achieves a 99.2% touch-down detection accuracy across various materials, with a motion-to-photon latency of 150 ms.With a brief (20s) userspecific calibration, HaloTouch supports millimeter-accurate hover sensing as well as continuous pressure sensing.We conducted a user study with 12 participants, including a typing task demonstrating text input at 26.3 AWPM.HaloTouch shows promise for more robust, dynamic touch interactions without instrumenting surfaces or adding hardware to users.

University students engage in a substantial amount of multitasking in online classes despite being aware of its negative impacts on their learning. Depending on the learner’s goals, in-class multitasking can be a positive strategic behavior to increase productivity. In a formative pilot study (N=10), we established the structure and scope for our design by exploring students’ motivations, perceptions, and challenges in in-class multitasking and identified several promising design elements. Our design facilitates multitasking in online synchronous classes by providing a novel bichronous (blending of synchronous and asynchronous) learning environment manifested in Time-Turner that enables asynchronous guided accelerated viewing of past content during synchronous classes. A summative evaluation of our prototype showed significant improvement in learning outcomes when multitasking (N=20). Furthermore, 95% of users found Time-Turner helpful and expressed interest in having it in their online classes. Our findings show the great potential of supporting positive multitasking in synchronous online classes.

Exposure to diverse perspectives is helpful for bursting the filter bubble in online public video platforms. The recent advancement of Large Language Models (LLMs) illuminates the potential of creating a debate chatbot that prompts users to critically examine their stances on a topic formed by watching videos. However, whether the viewer is influenced by the chatbot may depend on its persona. In this paper, we investigated the effect of two relevant persona attributes - social identity and rhetorical styles - on critical thinking. In a mixed-methods study (n=36), we found that chatbots with outgroup (vs. ingroup) identity (t(33)=-2.33, p=0.03) and persuasive (vs. eristic) rhetoric (t(44)=1.98, p=0.05) induced critical thinking most effectively, making participants re-examine their arguments. However, participants’ stances remain largely unaffected, likely due to the chatbot’s lack of contextual knowledge and human touch. Our paper provides empirical groundwork for designing chatbot persona for remedying filter bubbles in online communities.

Videos are convenient resources for asynchronous learning, but they lack interpersonal interactions found in synchronous classrooms. Due to missed social connectedness, the isolated video-based learners experience low emotional, behavioral, and cognitive engagement. This work presents "Scripted Vicarious Dialogues" (SVD), a technique for engaging students in a pseudo-social experience of witnessing scripted dialogues between virtual characters (teaching assistants and students) around a video. We conducted a participatory design study to derive design guidelines for SVD. The findings indicate the need to distinguish the virtual components and to give students control of the dialogue’s pace. We then implemented an interactive prototype of SVD and evaluated it (N=40) against a non-social, direct-learning baseline. The results show that the preference for SVD versus the baseline is polarized (25 of 40 preferred SVD; no neutral preferences), and those who preferred SVD had significantly higher emotional and behavioral engagement with SVD compared to the baseline.

Finite element models (FEM) of the tongue have facilitated speech studies through analysis of internal muscle forces indirectly derived from imaging data. In this work, we build a uniform hexahedral FEM of a tongue atlas constructed from magnetic resonance imaging data of a healthy population. The FEM is driven by inverse internal tongue tissue kinematics of speakers temporally aligned and deformed into the same atlas space, while performing the speech task "a souk" allowing muscle activation predictions. This work aims to investigate the commonalities in tongue motor strategies in the articulation of "a souk" predicted by the inverse tongue atlas model. Our findings report variability among five speakers for estimated muscle activations with a similarity index using a dynamic time warp function. Two speakers show similarity index > 0.9 and two others < 0.7 with respect to a reference speaker for most tongue muscles. The relative motion tracking error of the model is less than 2% which is promising for speech study applications.

Recent neural rendering approaches greatly improve image quality, reaching near photorealism. However, the underlying neural networks have high runtime, precluding telepresence and virtual reality applications that require high resolution at low latency. The sequential dependency of layers in deep networks makes their optimization difficult. We break this dependency by caching information from the previous frame to speed up the processing of the current one with an implicit warp. The warping with a shallow network reduces latency and the caching operations can further be parallelized to improve the frame rate. In contrast to existing temporal neural networks, ours is tailored for the task of rendering novel views of faces by conditioning on the change of the underlying surface mesh. We test the approach on view-dependent rendering of 3D portrait avatars, as needed for telepresence, on established benchmark sequences. Warping reduces latency by 70% (from 49.4ms to 14.9ms on commodity GPUs) and scales frame rates accordingly over multiple GPUs while reducing image quality by only 1%, making it suitable as part of end-to-end view-dependent 3D teleconferencing applications.

Boredom is … a vital problem for the moralist, since half the sins of mankind are caused by the fear of it. (Bertrand Russell, “The Conquest of Happiness,” 1930).Humans are essentially storytellers...

Pull Requests (PRs) are a frequently used method for proposing changes to source code repositories. When discussing proposed changes in a PR discussion, stakeholders often reference a wide variety of information objects for establishing shared awareness and common ground. Previous work has not considered how the referential behavior impacts collaborative software development via PRs. This knowledge gap is the major barrier in evaluating the current support for referencing in PRs and improving them. We conducted an explorative analysis of \textasciitilde7K references, collected from 450 public PRs on GitHub, and constructed taxonomies of referent types and expressions. Using our annotated dataset, we identified several patterns in the use of references. Referencing source code elements was prevalent but the authoring interface lacks support for it. Three classes of contextual factors influence referencing behaviors: referent type, discussion thread, and project attributes. Referencing patterns may indicate PR outcomes (e.g., merged PRs frequently reference issues, users, and tests). We conclude with design implications to support more effective referencing in PR discussion interfaces.

We introduce Gripmarks, a system that enables users to opportunistically use objects they are already holding as input surfaces for mixed reality head-mounted displays (HMD). Leveraging handheld objects reduces the need for users to free up their hands or acquire a controller to interact with their HMD. Gripmarks associate a particular hand grip with the shape primitive of the physical object without the need of object recognition or instrumenting the object. From the grip pose and shape primitive we can infer the surface of the object. With an activation gesture, we can enable the object for use as input to the HMD. With five gripmarks we demonstrate a recognition rate of 94.2%; we show that our grip detection benefits from the physical constraints of holding an object. We explore two categories of input objects 1) tangible surfaces and 2) tangible tools and present two representative applications. We discuss the design and technical challenges for expanding the concept.

Fish Tank Virtual Reality (FTVR) displays provide compelling 3D experiences by rendering view-dependent imagery on a 2D screen. While users perceive a 3D object in space, they are actually looking at pixels on a 2D screen, thus, a perceptual duality exists between the object's pixels and the 3D percept potentially interfering with the experience. To investigate, we conducted an experiment to see whether the on-screen size of the 2D imagery affects the perceived object size in 3D space with different viewing conditions, including stereopsis. We found that the size of on-screen imagery significantly influenced object size perception, causing 83.3% under/overestimation of perceived size when viewing without stereopsis and reducing to 64.7% with stereopsis. Contrary to reality, objects look smaller when the viewer gets closer. Understanding the perceptual duality helps us to provide accurate perception of real-world objects depicted in the virtual environment and pave the way for 3D applications.

The articulatory geometric configurations of the vocal tract and the acoustic properties of the resultant speech sound are considered to have a strong causal relationship. This paper aims at finding a joint latent representation between the articulatory and acoustic domain for vowel sounds via invertible neural network models, while simultaneously preserving the respective domain-specific features. Our model utilizes a convolutional autoencoder architecture and normalizing flow-based models to allow both forward and inverse mappings in a semi-supervised manner, between the mid-sagittal vocal tract geometry of a two degrees-of-freedom articulatory synthesizer with 1D acoustic wave model and the Mel-spectrogram representation of the synthesized speech sounds. Our approach achieves satisfactory performance in achieving both articulatory-to-acoustic as well as acoustic-to-articulatory mapping, thereby demonstrating our success in achieving a joint encoding of both the domains.

Video-based learning (VBL) is widespread; however, there are numerous challenges when teaching and learning with video. For instructors, creating effective instructional videos takes considerable time and effort. For students, watching videos can be a passive learning activity. Artificial intelligence (AI) has the potential to improve the VBL experience for students and teachers. This half-day workshop will bring together multi-disciplinary researchers and practitioners to collaboratively envision the future of VBL enhanced by AI. This workshop will be comprised of a group discussion followed by a presentation session. The goal of the workshop is to facilitate the cross-pollination of design ideas and critical assessments of AI approaches to VBL.

We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatialtemporal classification problems.

Spherical fish tank virtual reality (FTVR) displays attempt to create a virtual "crystal ball" experience using head-tracked rendering. Almost all of these systems have omitted stereo cues, making them easy to build, but it is not clear how much this omission degrades the 3D experience. In this study, we evaluate performance and subjective effects of stereo on 3D perception and interaction tasks with a spherical FTVR display. To control for calibration error and tracking latency, we perform the evaluation on a simulated spherical display in VR. The results of our study provide a clear recommendation for the design and use of spherical FTVR displays: while omitting stereo may not be readily apparent for users, their performance will be significantly degraded (20% - 91% increase in median task time). Therefore, including stereo viewing in spherical displays is critical for use in FTVR.

As videos become increasingly ubiquitous, so is video-based commenting. To contextualize comments, people often reference specific audio/visual content within video. However, the literature falls short of explaining the types of video content people refer to, how they establish references and identify referents, how video characteristics (e.g., genre) impact referencing behaviors, and how references impact social engagement. We present a taxonomy for classifying video references by referent type and temporal specificity. Using our taxonomy, we analyzed 2.5K references with quotations and timestamps collected from public YouTube comments. We found: 1) people reference intervals of video more frequently than time-points, 2) visual entities are referenced more often than sounds, and 3) comments with quotes are more likely to receive replies but not more "likes". We discuss the need for in-situ dereferencing user interfaces, illustrate design concepts for typed referencing features, and provide a dataset for future studies.

Speech-related Brain Computer Interfaces (BCI) aim primarily at finding an alternative vocal communication pathway for people with speaking disabilities. As a step towards full decoding of imagined speech from active thoughts, we present a BCI system for subject-independent classification of phonological categories exploiting a novel deep learning based hierarchical feature extraction scheme. To better capture the complex representation of high-dimensional electroencephalography (EEG) data, we compute the joint variability of EEG electrodes into a channel cross-covariance matrix. We then extract the spatio-temporal information encoded within the matrix using a mixed deep neural network strategy. Our model framework is composed of a convolutional neural network (CNN), a long-short term network (LSTM), and a deep autoencoder. We train the individual networks hierarchically, feeding their combined outputs in a final gradient boosting classification step. Our best models achieve an average accuracy of 77.9% across five different binary classification tasks, providing a significant 22.5% improvement over previous methods. As we also show visually, our work demonstrates that the speech imagery EEG possesses significant discriminative information about the intended articulatory movements responsible for natural speech synthesis.

The simulation of two-dimensional (2D) wave propagation is an affordable computational task and its use can potentially improve time performance in vocal tracts' acoustic analysis. Several models have been designed that rely on 2D wave solvers and include 2D representations of three-dimensional (3D) vocal tract-like geometries. However, until now, only the acoustics of straight 3D tubes with circular cross-sections have been successfully replicated with this approach. Furthermore, the simulation of the resulting 2D shapes requires extremely high spatio-temporal resolutions, dramatically reducing the speed boost deriving from the usage of a 2D wave solver. In this paper, we introduce an in-progress novel vocal tract model that extends the 2D Finite-Difference Time-Domain wave solver (2.5D FDTD) by adding tube depth, derived from the area functions, to the acoustic solver. The model combines the speed of a light 2D numerical scheme with the ability to natively simulate 3D tubes that are symmetric in one dimension, hence relaxing previous resolution requirements. An implementation of the 2.5D FDTD is presented, along with evaluation of its performance in the case of static vowel modeling. The paper discusses the current features and limits of the approach, and the potential impact on computational acoustics applications.

Speech-related Brain Computer Interface (BCI) technologies provide effective vocal communication strategies for controlling devices through speech commands interpreted from brain signals. In order to infer imagined speech from active thoughts, we propose a novel hierarchical deep learning BCI system for subject-independent classification of 11 speech tokens including phonemes and words. Our novel approach exploits predicted articulatory information of six phonological categories (e.g., nasal, bilabial) as an intermediate step for classifying the phonemes and words, thereby finding discriminative signal responsible for natural speech synthesis. The proposed network is composed of hierarchical combination of spatial and temporal CNN cascaded with a deep autoencoder. Our best models on the KARA database achieve an average accuracy of 83.42% across the six different binary phonological classification tasks, and 53.36% for the individual token identification task, significantly outperforming our baselines. Ultimately, our work suggests the possible existence of a brain imagery footprint for the underlying articulatory movement related to different sounds that can be used to aid imagined speech decoding.

While video becomes increasingly prevalent in educational settings, current research has yet to investigate what feedback instructors need regarding their students' engagement and learning despite video technologies being equipped to provide viewing analytics and collect student feedback. In this paper we investigate instructors' requirements from video analytics. We used a Grounded Theory Approach and interviewed 16 instructors who teach using video to determine the advantages for using video in their teaching and the different requirements for analytics and feedback in their existing practice. Based on our analysis of the interviews, we found three categories of information that instructors want to inform their teaching. Instructors are looking to see if their students have watched their videos, how much they understood in those videos, and how useful the videos are to the students. These categories provide the foundations and design implications for instructor-centric educational video analytics interfaces.

Eye-contact is a key aspect of non-verbal human communication in everyday tasks [1]. It provides important social and emotional information that can increase the effectiveness of human communication [8]. In a conversation, eye-contact, or the lack thereof, is constantly evaluated by human brains. Conversation partners derive subjective judgment of others' credibility, focus and confidence [4] from it. Seeking another person's eye-contact is a signal for them that focus is put on that person and the main receptive senses are prepared to receive input from the other person. Likewise breaking eye-contact usually indicates distraction, loss of confidence, loss of interest or shifting focus to a different target.

Incorrect depth perception and lack of good evaluation systems are major barriers in clinical translation of augmented and mixed reality AR/MR. Thus, a systematic study of depth cues is necessary. Therefore, in the current paper we present a method to introduce the quantitative depth cue Motion Parallax (MP) in surgical scenes and study its effect on depth perception when combined with binocular disparity. In addition to this we present an innovative virtual tool method to evaluate depth. To introduce MP, we reconstructed the tissue surface using structure from motion technique. Then to get accurate absolute scale of the reconstructed surface stereo-triangulation was used. The simulated tumor was rendered beneath the reconstructed point-cloud by rendering a hole for `X-ray' like vision. The MP was introduced by rotating the entire scene from side-to-side with a tumor-surface-point as pivot for maximum impact. Finally for evaluation, we used a virtual surgical tool rendered using real-time da Vinci surgical API's forward kinematics data. In total, 12 subjects participated in a within-subjects-experiment design to study four cases, i.e., Stereo + MP ( S+MP), Mono + MP ( M+MP), Stereo + No MP (S+N-MP) and Mono + No MP (M+N-MP). The subjects significantly overestimated Judged Percentage of True Distance in M+MP when compared to M+N-MP (probability (p)=0.000, Number of Samples (N))=120) and S+N-MP cases ( p=0.001, N = 120). Furthermore, the observed VariableError was less in S+MP and S+N-MP cases when compared to M+MP and M+N-MP cases. The use of Motion Parallax in console interfaces for surgical robotics showed overestimation of judged distance. But to our knowledge it is the first work studying the effect of motion parallax and stereo in the surgical context. Therefore, its further study is warranted.

We developed a mixed-reality approach, called Crystal Palace, which provides prop-based passive haptic feedback for the 3DUI contest. In this system, we propose an interface of controlling virtual object with physical tools such as scissors, a spray and a screwdriver as game controllers. Our prop-based approach provides natural visual affordance and passive haptics by associating physical tools with various 3D tasks. Using 360 degrees spherical display, players will be able to use common real-world tools in an intuitive way to interact with 3D objects in a virtual escape room.

Pointing is a fundamental building block in human communication. While it is ubiquitous in our daily interactions within the real world, it is difficult to precisely interpret a virtual agent's pointing direction to the physical world, considering its complex and subtle gesture cues, such as the movements of the human hand and head. Fish Tank Virtual Reality (FTVR) display has the potential to provide accurate pointing cues as it creates a compelling 3D spatial effect by rendering perspective-corrected vision. In this paper, we conducted a study with pointing cues of three levels (Head-only, Hand-only, and Hand+Head) to evaluate how the head and hand gesture cues affect observers' performance in interpretation of where a virtual agent is pointing in a spherical FTVR display. The results showed that the hand gesture significantly helps people interpret the pointing both accurately and quickly for fine pointing (15°), with 19.4% higher accuracy and 1.42 seconds faster than the head cue. The combination of the head and hand yielded a small improvement on the accuracy (4.4%) with even slightly longer time (0.38 seconds) compared to the hand-only cue. However, for coarse pointing (30°), head cue appears to be sufficient with the accuracy of 90.2%. The result of this study provides guidelines on cues selection for designing pointing in the virtual environment.

Fish Tank Virtual Reality (FTVR) displays create a compelling 3D effect with the motion parallax cue using the head-coupled perspective. While the head-coupled viewpoint control provides natural visuomotor coupling, the motion parallax cue has been found to be underutilized with minimal head motion detected when manual input becomes available to users. We investigate whether users can effectively use head-coupling in conjunction with manual input in a mental rotation task involving inspection and comparison of a pair of 3D cubes. We found that participants managed to incorporate the head-coupled viewpoint control with the manual touch input in the task. They used the touch input as the primary input and the head as the secondary input with the input ratio of 4.2:1. The combined input approach appears to be sequential with only 8.63% duration when the head and manual input are co-activated. The result of this study provides insights for designing head-coupled interactions in many 3D interactive applications.

Fish Tank Virtual Reality (FTVR) displays create a compelling 3D spatial effect by rendering to the perspective of the viewer with head-tracking. Combining FTVR with a spherical display enhances the 3D experience with unique properties of the spherical screen such as the enclosing shape, consistent curved surface, and borderless views from all angles around the display. The ability to generate a strong 3D effect on a spherical display with head-tracked rendering is promising for increasing user's performance in 3D tasks. An unanswered question is whether these natural affordances of spherical FTVR displays can improve spatial perception in comparison to traditional flat FTVR displays. To investigate this question, we conducted an experiment to see whether users can perceive the depth and size of virtual objects better on a spherical FTVR display compared to a flat FTVR display on two tasks. Using the spherical display, we found significantly that users had 1cm depth accuracy compared to 6.5cm accuracy using the flat display on a depth-ranking task. Likewise, their performance on a size-matching task was also significantly better with the size error of 2.3mm on the spherical display compared to 3.1mm on the flat display. Furthermore, the perception of size-constancy is stronger on the spherical display than the flat display. This study indicates that the natural affordances provided by the spherical form factor improve depth and size perception in 3D compared to a flat display. We believe that spherical FTVR displays have potential as a 3D virtual environment to provide better task performance for various 3D applications such as 3D designs, scientific visualizations, and virtual surgery.

Video is an increasingly popular medium for education. Motivated by the problem of video as a one-way medium, this paper investigates the ways in which learners» active interaction with video materials contributes to active learning. In this study, we examine active viewing behaviors, specifically seeking and highlighting within videos, which may suggest greater levels of participation and learning. We deployed a system designed for active viewing to an undergraduate class for a semester. The analysis of online activity traces and interview data provided novel findings on video highlighting behavior in educational contexts.

Vocal tract configurations play a vital role in generating distinguishable speech sounds, by modulating the airflow and creating different resonant cavities in speech production. They contain abundant information that can be utilized to better understand the underlying speech production mechanism. As a step towards automatic mapping of vocal tract shape geometry to acoustics, this paper employs effective video action recognition techniques, like Long-term Recurrent Convolutional Networks (LRCN) models, to identify different vowel-consonant-vowel (VCV) sequences from dynamic shaping of the vocal tract. Such a model typically combines a CNN based deep hierarchical visual feature extractor with Recurrent Networks, that ideally makes the network spatio-temporally deep enough to learn the sequential dynamics of a short video clip for video classification tasks. We use a database consisting of 2D real-time MRI of vocal tract shaping during VCV utterances by 17 speakers. The comparative performances of this class of algorithms under various parameter settings and for various classification tasks are discussed. Interestingly, the results show a marked difference in the model performance in the context of speech classification with respect to generic sequence or video classification tasks.

Video-based learning is most effective when students are engaged with video content; however, the literature has yet to identify students' viewing behaviors and ground them in theory. This paper addresses this need by introducing a framework of active viewing, which is situated in an established model of active learning to describe students' behaviors while learning from video. We conducted a field study with 460 undergraduates in an Applied Science course using a video player designed for active viewing to evaluate how students engage in passive and active video-based learning. The concept of active viewing, and the role of interactive, constructive, active, and passive behaviors in video-based learning, can be implemented in the design and evaluation of video players.

In this work we describe the design, implementation and initial evaluation of a spherical Fish Tank Virtual Reality (FTVR) display for realistic eye-contact. We identify display shape, size, and depth cues as well as model fidelity as important considerations and challenges for setting up realistic eye-contact and package it into a reproducible framework. Based on the design, we implemented and evaluated the system to assess the effectiveness of the eye-contact. In our initial evaluation participants were able to identify eye-contact with an accuracy of 89.6%. Moreover eye-contact with a virtual character triggered changes in participant's social behavior that are in line with real world eye-contact scenarios. Taken together, these results provide practical guidelines for building displays for realistic eye-contact and can be applied to applications such as teleconferencing and VR treatment in psychology.

A mixed reality experience with a physical display, that situates 3D virtual content within the real world, has the potential to help people work and play with 3D information. However, almost all of such "fish tank virtual reality" (FTVR) systems have been isolated to a single-person experience, making them unsuitable for collaborative tasks. In this paper, we present a display system that allows two people to have unobstructed 3D perspective views into a spherical display while still being able to see and talk to one another. We evaluated the system through qualitative observation at a four-day exhibition and found it was effective for providing a convincing, shared 3D experience.

course Share on How to design, build, and perform with new musical interfaces Authors: Michael Lyons Ritsumeikan University Ritsumeikan UniversityView Profile , Sidney Fels University of British Columbia University of British ColumbiaView Profile Authors Info & Claims SA '17: SIGGRAPH Asia 2017 CoursesNovember 2017 Article No.: 9Pages 1–143https://doi.org/10.1145/3134472.3134473Published:27 November 2017Publication History 0citation151DownloadsMetricsTotal Citations0Total Downloads151Last 12 Months6Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We describe an auto-calibrated 3D perspective-corrected spherical display that uses multiple rear projected pico-projectors. The display system is auto-calibrated via 3D reconstruction of each projected pixel on the display using a single inexpensive camera. With the automatic calibration, the multiple-projector system supports a seamless blended imagery on the spherical screen. Furthermore, we incorporate head tracking with the display to present 3D content with motion parallax by rendering perspective-corrected images based on the viewpoint. To show the effectiveness of this design, we implemented a view-dependent application that allows walk-around visualization from all angles for a single head-tracked user. We also implemented a view-independent application that supports a wall-papered rendering for multi-user viewing. Thus, both view-dependent 3D VR content and spherical 2D content, such as a globe, can be easily experienced with this display.

We describe a novel automatic calibration method using a single camera for a multiple-projector spherical Fish Tank Virtual Reality (FTVR) display. Modeling the projector as an inverse camera, we estimate the intrinsic and extrinsic projector parameters automatically using a set of projected images on the spherical screen. A calibrated camera is placed beneath to observe partially visible projected patterns. Using the correspondence between the observed pattern and the projected pattern, we reconstruct the shape of the spherical display and finally recover the 3D position of each projected pixel on the display. Additionally we present a practical calibration evaluation method that estimates on-surface accuracy using the single camera. We use point mismatch as a metric to describe misalignment and line mismatch to describe distortion. We demonstrate our automatic approach can achieve an on-surface point mismatch less than 1mm and line mismatch less than 1 on a 30cm diameter spherical screen. Taken together, our calibration approach and evaluation method are automatic and accurate for a desktop spherical FTVR and can be applied to other multiple-projector displays with curved screens.

We introduce a biomechanical model of oropharyngeal structures that adds the soft-palate, pharynx, and larynx to our previous models of jaw, skull, hyoid, tongue, and face in a unified model. The model includes a comprehensive description of the upper airway musculature, using point-to-point muscles that may either be embedded within the deformable structures or operate exter- nally. The airway is described by an air-tight mesh that fits and deforms with the surrounding articulators, which enables dynamic coupling to our articulatory speech synthesizer. We demonstrate that the biomechanics, in conjunction with the skinning, supports a range from physically realistic to simplified vocal tract geometries to investigate different approaches to aeroacoustic modeling of vocal tract. Furthermore, our model supports inverse modeling to support investigation of plausible muscle activation patterns to generate speech.

We present a novel eye-camera positioning system with four degrees-of-freedom (DOF). The system has been designed to emulate human eye movements, including saccades, for anatomically accurate androids. The architecture of our system is similar to that of a universal joint in that a hollowed sphere (the eyeball), hosting a miniature CMOS color camera, takes the part of the cross shaft that connects a pair of hinges that are oriented at 90 degrees of each other. This concept allows the motors to remain static, enabling placing them in multiple configurations during the mechanical design stage facilitating the inclusion of other robotic parts into the robots head. Based on our evaluations, the robotic eye-camera has been shown to be suitable for perception experiments that require human-like eye motion.

In this poster we present Portable-Spheree, an interactive spherical rear-projected 3D-content-display that provides perspective-corrected views according to the user's head position, to provide parallax, shading and occlusion depth cues. Portable-Spheree is an evolution of the Spheree and it is developed in a smaller form factor, using more projectors and a dark-translucent screen with increased contrast. We present some preliminary results of this new configuration as well as applications with spatial interaction that might benefit from this new form factor.

A common challenge when performing surface-based registration of images is ensuring that the surfaces accurately represent consistent anatomical boundaries. Image segmentation may be difficult in some regions due to either poor contrast, low slice resolution, or tissue ambiguities. To address this, we present a novel non-rigid surface registration method designed to register two partial surfaces, capable of ignoring regions where the anatomical boundary is unclear. Our probabilistic approach incorporates prior geometric information in the form of a statistical shape model (SSM), and physical knowledge in the form of a finite element model (FEM). We validate results in the context of prostate interventions by registering pre-operative magnetic resonance imaging (MRI) to 3D transrectal ultrasound (TRUS). We show that both the geometric and physical priors significantly decrease net target registration error (TRE), leading to TREs of 2.35 ± 0.81 mm and 2.81 ± 0.66 mm when applied to full and partial surfaces, respectively. We investigate robustness in response to errors in segmentation, varying levels of missing data, and adjusting the tunable parameters. Results demonstrate that the proposed surface registration method is an efficient, robust, and effective solution for fusing data from multiple modalities.

In surface-based registration for image-guided interventions, the presence of missing data can be a significant issue. This often arises with real-time imaging modalities such as ultrasound, where poor contrast can make tissue boundaries difficult to distinguish from surrounding tissue. Missing data poses two challenges: ambiguity in establishing correspondences; and extrapolation of the deformation field to those missing regions. To address these, we present a novel non-rigid registration method. For establishing correspondences, we use a probabilistic framework based on a Gaussian mixture model (GMM) that treats one surface as a potentially partial observation. To extrapolate and constrain the deformation field, we incorporate biomechanical prior knowledge in the form of a finite element model (FEM). We validate the algorithm, referred to as GMM-FEM, in the context of prostate interventions. Our method leads to a significant reduction in target registration error (TRE) compared to similar state-of-the-art registration algorithms in the case of missing data up to 30%, with a mean TRE of 2.6 mm. The method also performs well when full segmentations are available, leading to TREs that are comparable to or better than other surface-based techniques. We also analyze robustness of our approach, showing that GMM-FEM is a practical and reliable solution for surface-based registration.

We present an investigation of two different visualizations of video history: Video Timeline and Video Tiles. Video Timeline extends the commonly employed list-based visualization for navigation history by applying size to indicate heuristics and occupying the full screen with a two-sided timeline. Video Tiles visualizes history items in a grid-based layout by following pre-defined templates based on items' heuristics and ordering, utilizing screen space more effectively at the expense of a clearer temporal location. The visualizations are compared against the state-of-the-art method (a filmstrip-based visualization), with ten participants tasked with sharing their previously-seen affective intervals. Our study shows that our visualizations are perceived as intuitive and both outperform and are strongly preferred to the current method. Based on these results, Video Timeline and Video Tiles provide an effective addition to video viewers to help manage the growing quantity of video. They provide users with insight into their navigation patterns, allowing them to quickly find previously-seen intervals, leading to efficient clip sharing, simpler authoring and video summarization.

Humans see things from various viewpoints but nobody attempts to see anything from every viewpoint owing to physical limitations and the great effort required. Intelligent interfaces for viewing multi-viewpoint videos may effectively remove these limitations and open up a new visual world to mankind. We have developed a multi-viewpoint video viewer that incorporates target-centered viewpoint switching. The viewer stabilizes an object at the center of the display field, which helps to focus the user's gaze on the target. We conducted a user study to analyze user behavior, especially eye movement, while watching a multi-viewpoint video on the viewer. Statistical analyses of the results indicated that the target-centered viewpoint switching encouraged the users to gaze at the center of the display where the target was located during the viewing. We believe that these are useful findings that pave the way for the design of even more intelligent viewers.

Driving is an integral part of our everyday lives, but it is also a time when people are uniquely vulnerable. Poor road condition, traffic congestion and long driving time may bring negative emotion to drivers and increase the chance of traffic accidents. We propose SAfeDJ, a situation-aware in-car music delivery application, which turns people's trips into pleasant journeys and driving into a safe and enjoyable activity. SAfeDJ aims at helping drivers to diminish fatigue and negative emotion. It is built on a vehicular healthcare platform that enables communications among drivers and integrates with multiple types of sensors to promote safe driving. Prototype implementation and initial results of SAfeDJ have demonstrated its desired functionality in drivers' daily lives and feasibility for real-world deployment.

we describe a novel robotic neck mechanism that supports realistic human head motion. Our design uses a 3-DOF spherical neck inspired by the 2-DOF spherical wrist of the Orthoglide 5-axis industrial robot. We use a gimbal-like mechanism to combine three 1-DOF motion components to rotate the head about a common point and around the three principal axes. Based on this design, we implemented and compared our neck in an android called Uma using human expressive neck motion specifications to ensure it is capable of human-like motion. Based on our evaluations, the neck has been shown to be suitable for perception experiments that require human-like head motion.

Human pose detection is the foundation for many applications, particularly those using gestures as part of a natural user interface. We introduce a novel task-based control method for human pose detection, encoding specialist knowledge in a descriptive abstraction for application by non-experts, such as developers, artists and students. The abstraction hides the details of a set of algorithms which specialise either in different estimations of pose (e.g. articulated, body part) or under different conditions (e.g. occlusion, clutter). Users describe the conditions of their problem, which is used to select the most suitable algorithm (and automatically set up the parameters). The task-based control is evaluated with images described using the abstraction. Expected outcomes are compared to results and demonstrate that describing the conditions is sufficient to allow the abstraction to produce the required result.

Recent years have seen a significant increase of interest in rich multi-modal user interfaces going beyond conventional mouse/keyboard/screen interaction. The new interface technologies are broadly impacting music technology and culture. New musical interfaces use a variety of sensing (and actuating) modalities to receive and present information to users, and often require techniques from signal processing and machine learning in order to extract and fuse high level information from noisy, high dimensional signals over time. Hence they pose many interesting signal processing challenges while offering fascinating possibilities for new research. At the same time the richness of possibilities for new forms of musical interaction requires a new approach to the design of musical technologies and has implications for performance aesthetics and music pedagogy. This tutorial begins with a general and gentle introduction to the theory and practice of the design of new technologies for musical creation and performance. It continues with an overview of signal processing and machine learning methods which are needed for more advanced work in new musical interface design.

Research into computer vision techniques has far outpaced the development of interfaces (such as APIs) to support the techniques' accessibility, especially to developers who are not experts in the field. We present a new description-based interface designed to be mainstream-developer-friendly while retaining sufficient power and flexibility to solve a wide variety of computer vision problems. The interface presents vision at the task level (hiding algorithmic detail) and uses a task-based description derived from definitions of vision problems. We show that after interpretation, the description can be used to invoke an appropriate method to provide the developer's requested result. Our implementation interprets the description and invokes various vision methods with automatically derived parameters, which we demonstrate on a range of tasks.

Advances in digital audio technologies have led to a situation where computers play a role in most music production and performance. Digital technologies offer unprecedented opportunities for the creation and manipulation of sound, however the flexibilty of these new technologies imply an often confusing array of choices for musical composers and performers. Some artists have faced this challenge by using computers directly to create music and leading to an explosion of new musical forms. However, most would agree that the computer is not a musical instrument, in the same sense as traditional instruments, and it is natural to ask 'how to play the computer' using interface technology appropriate for human brains and bodies. In 2001, we organized the first workshop on New Interfaces for Musical Expression (NIME), to attempt to answer this question by exploring connections with the better established field of human-computer interaction. This course summarizes what has been learned at NIME which has been held annually since that first workshop. We begin with an overview of the theory and practice of new musical interface design, asking what makes a good musical interface and whether there are any useful design principles or guidelines available. We will also discuss topics such as the mapping from human action to musical output, and control intimacy. Practical information about the tools for creating musical interfaces will be given, including an overview of sensors and microcontrollers, audio synthesis techniques, and communication protocols such as Open Sound Control (and MIDI). The remainder of the course will consist of several specific case studies representative of the major broad themes of the NIME conference, including augmented and sensor based instruments, mobile and networked music, and NIME pedagogy.

Abstract—Through a participatory design (PD) process with therapists at a neurological chronic care hospital that has no ICT based workflow processes we identified three main components where NUI technology can be deployed for physical therapy: patient stimulation, on-demand patient therapy tool control, and data logging and visualization. Based on our study, we created a scenario that encompasses these components for a specific physical therapy activity targeting head and torso movement for children suffering neurological disorders. The scenario includes a Kinect, large screen and mobile pervasive technologies controlling a game-like activity including data logging and visualization in a shared social intranet. Hospital staff liked the game-activity and logging, but expressed concerns about parallel use of technology. They also wanted to expand the use of the NUI technology in the therapy room leading to new scenarios that we are pursuing. We are using our participatory design process with therapists that are not tainted by the use of graphical user interfaces (GUIs) to establish scenarios where NUI designs can provide access to ICT tools to help us develop better NUI design principles.

There has been significant research interest over recent years in the use of public digital displays and in particular their capability to offer both interactivity and personalized content. Although a number of interaction technologies have been investigated, a promising approach has been the use of the ubiquitous cell phone, which not only offers a means to interact with displays but increasingly offers a small but high-quality screen to complement the larger public display. However, to date there has been little investigation into the impact on users when interfaces are distributed across this type of dual screen setup. This article reports on a series of experiments carried out to determine if there is a significant quantitative or qualitative effect on user performance when interaction is split across large public and smaller private screens.

Advances in digital audio technologies have led to a situation where computers play a role in most music production and performance. Digital technologies offer unprecedented opportunities for the creation and manipulation of sound. However, the flexibilty of these new technologies implies an often confusing array of choices for musical composers and performers. Some artists have faced this challenge by using computers directly to create music, leading to an explosion of new musical forms. However, most would agree that the computer is not a musical instrument - in the same sense as traditional instruments - and it is natural to ask 'how to play the computer' using interface technology appropriate for human brains and bodies. A decade ago we organized the first workshop on New Interfaces for Musical Expression (NIME) to attempt to answer this question by exploring connections with the established field of human-computer interaction. This course summarizes what has been learned at NIME. We begin with an overview of the theory and practice of new musical interface design, asking what makes a good musical interface and whether there are any useful design principles or guidelines available. We will also discuss topics such as the mapping from human action to musical output, and control intimacy. Practical information about the tools for creating musical interfaces will be given, including an overview of sensors and microcontrollers, audio synthesis techniques, and communication protocols such as Open Sound Control and MIDI. The remainder of the course will consist of several specific case studies representative of the major broad themes of the NIME conference, including augmented and sensor based instruments, mobile and networked music, and NIME pedagogy.

This paper introduces a system that automatically classifies image pairs based on the type of registration required to align them. The system uses support vector machines to classify between panoramas, high-dynamic-range images, focal stacks, super-resolution, and unrelated image pairs. A feature vector was developed to describe the images, and 1100 pairs were used to train and test the system with 5-fold cross validation. The system is able to classify the desired registration application using a 1: Many classifier with an accuracy of 91.18%. Similarly 1:1 classifiers were developed for each class with classification rates as follows: Panorama image pairs are classified at 93.15%, high-dynamic-range pairs at 97.56%, focal stack pairs at 95.68%, super-resolution pairs at 99.25%, and finally unrelated image pairs at 95.79%. An investigation into feature importance outlines the utility of each feature individually. In addition, the invariance of the classification system towards the size of the image used to calculate the feature vector was explored. The classification of our system remains level at ~91% until the image size is scaled to 10% (150 × 100 pixels), suggesting that our feature vector is image size invariant within this range.

The use of a dual mobile and large screen approach offers a number of intriguing possibilities including a potential solution to the problem of managing conflicts that arise when a large screen is shared in a public setting. Here, we report on a series of experiments carried out to determine quantitative or qualitative effects of user performance when interaction is split across large public and smaller private screens. Our position is that using mobile devices as an auxiliary device for interaction can boost user experience when interacting with large displays.

In this paper, we describe the design of a personal cubic display that offers novel interaction techniques for static and dynamic 3D content. We extended one-screen Fish Tank VR by arranging five small LCD panels into a box shape that is light and compact enough to be handheld. The display uses head-coupled perspective rendering and a real-time physics simulation engine to establish an interaction metaphor of having real objects inside a physical box that a user can hold and manipulate. We evaluated our prototype as a visualization tool and as an input device by comparing it with a conventional LCD display and mouse for a 3D tree-tracing task. We found that bimanual interaction with pCubee and a mouse offered the best performance and was most preferred by users. pCubee has potential in 3D visualization and interactive applications such as games, storytelling and education, as well as viewing 3D maps, medical and architectural data.

This paper describes the conceptual, aesthetic, hardware, and software design of Flow, a photo/media-based permanent public interactive artwork in Vancouver, Canada. The work is located at street level in a new local community centre at one of the city's oldest intersections. In addition to the community centre location, it has a related interactive web component. It involves the animation and projection of continually recombining photographic images onto a large, interactive 4x4 array of electronically controlled switch glass windows. Over the course of the day and night, these photographic tableaux appear on the glass in combinations that depend upon image-to-image relationships, time of day, season and weather. In addition, lighting elements including water effect gobos are integrated at selected times of day. Images disappear when viewers inside the building come within close proximity to the work: the interactive windows respond to movement by changing from translucent to clear. In the daytime, when the projected image is off at the site, the work continues on the project's website, offering the visitor an interaction with the work. The work aims to provide an experience of the flux of people, animals, landscape and urban environment over time. It addresses the way landscape has transformed in response to colonialism, capital and local pressures, where change is rapid and histories are lost and rewritten.

Due to advances in digital audio technologies, computers now play a role in most music production and performance. Digital technologies offer unprecedented opportunities for creation and manipulation of sound, but the flexibilty of these new technologies implies an often confusing array of choices for musical composers and performers. Some artists are using computers directly to create music and generate an explosion of new musical forms. However, most would agree that the computer is not a musical instrument, in the same sense as traditional instruments, and it is natural to wonder "how to play the computer" using interface technology appropriate for human brains and bodies.

We demonstrate for the first time a proof of concept projector with a secondary array of individually controllable, analog micromirrors added to improve the contrast and peak brightness of conventional projectors. The micromirrors reallocate the light of the projector lamp from the dark parts towards the light parts of the image, before it reaches the primary image modulator. Each element of the analog micromirror array can be tipped/tilted to divert portions of the light from the lamp in two dimensions. By directing these mirrors on an image-dependent basis, we can increase both the peak intensity of the projected image as well as its contrast. In this paper, we describe and analyze the optical design for projectors using this light reallocation approach. We also discuss software algorithms to compute the best light reallocation pattern for a given input image, using the constraints of real hardware. We perform extensive simulations of this process to evaluate image quality and performance characteristics of this process. Finally, we present a first proof-of-concept implementation of this approach using a prototype analog micromirror device.

In this paper, we explore the practice of using a whiteboard for multiple tasks, and specifically how users employ whiteboards to smoothly transition between related sets of tasks. Our study underscores several basic, but important affordances of whiteboards that support this practice, including visual persistence, flexibility of interaction primitives, and their situated physicality. We discuss the implications of these findings for the design of large display applications.

We are addressing two aspects of vision-based system development that are not fully exploited in current frameworks: abstraction over low-level details and high-level module reusability. Through an evaluation of existing frameworks, we relate these shortcomings to the lack of systematic classification of sub-tasks in vision-based system development. In this paper we present our work-in-progress which addresses these two issues by classifying vision into decoupled sub-tasks, hence defining a clear scope for a vision-based system development framework and its sub-components. Firstly, we decompose the task of vision system development into data management and processing. We then proceed to further decompose data management into three components: data access, conversion and transportation. We present the Vision Utility (VU) framework which provides abstraction over the vision system data management and verify this approach through an example vision system.

We describe a demonstration of four novel interaction techniques for a cubic head-coupled 3D display. The interactions illustrated include: viewing a static scene, navigating through a large landscape, playing with colliding objects inside a box, and stylus-based manipulation of objects. Users experience new interaction techniques for 3D scene manipulation in a cubic display.

We introduce a work-in-progress camera access scheme we call the unified camera framework. Attempts have been made in the past to provide simple access to cameras, however these are generally OS specific or lacking in functionality. We present a novel interface which works across operating systems, and provides access to native images through a descriptor. A unified configuration model is presented to allow manipulation of camera parameters to the level each camera supports. Validation of the ideas presented is given in the form of a proof-of-concept implementation called the all seeing eye.

Video slicing---a variant of slit scanning in photography---extracts a scan line from a video frame and successively adds that line to a composite image over time. The composite image becomes a time line, where its visual patterns reflect changes in a particular area of the video stream. We extend this idea of video slicing by allowing users to draw marks anywhere on the source video to capture areas of interest. These marks, which we call slit-tears, are used in place of a scan line, and the resulting composite timeline image provides a much richer visualization of the video data. Depending on how tears are placed, they can accentuate motion, small changes, directional movement, and relational patterns.

Processing a video stream to segment foreground objects from the background is a critical first step in many computer vision applications. Background subtraction (BGS) is a commonly used technique for achieving this segmentation. The popularity of BGS largely comes from its computational efficiency, which allows applications such as human-computer interaction, video surveillance, and traffic monitoring to meet their real-time goals. Numerous BGS algorithms and a number of post-processing techniques that aim to improve the results of these algorithms have been proposed. In this paper, we evaluate several popular, state-of-the-art BGS algorithms and examine how post-processing techniques affect their performance. Our experimental results demonstrate that post-processing techniques can significantly improve the foreground segmentation masks produced by a BGS algorithm. We provide recommendations for achieving robust foreground segmentation based on the lessons learned performing this comparative study.

We study schemes to tackle problems of optimizing multiple object tracking and best-view video synthesis. A novel linear relaxation method is proposed for the class of multiple object tracking problems where the inter-object interaction metric is convex and the intra-object term quantifying object state continuity may use any metric. This scheme models object tracking as multi-path searching. It explicitly models track interaction, such as object spatial layout consistency or mutual occlusion, and optimizes multiple object tracks simultaneously. The proposed scheme does not rely on track initialization and complex heuristics. It has much less average complexity than previous efficient exhaustive search methods such as extended dynamic programming and can find the global optimum with high probability. Given the tracking data from our method, optimizing best-view video synthesis using multiple-view videos is further studied, which is formulated as a recursive decision problem and optimized by a dynamic programming approach. The proposed object tracking and best-view synthesis methods have found successful applications in MyView - a system to enhance media content presentation of multiple-view video.

We propose a linear programming relaxation scheme for the class of multiple object tracking problems where the inter-object interaction metric is convex and the intra-object term quantifying object state continuity may use any metric. The proposed scheme models object tracking as a multi-path searching problem. It explicitly models track interaction, such as object spatial layout consistency or mutual occlusion, and optimizes multiple object tracks simultaneously. The proposed scheme does not rely on track initialization and complex heuristics. It has much less average complexity than previous efficient exhaustive search methods such as extended dynamic programming and is found to be able to find the global optimum with high probability. We have successfully applied the proposed method to multiple object tracking in video streams.

This paper presents our progress on OpenVL -a novel software architecture to address efficiency through facilitating hardware acceleration, reusability and scalability for computer vision. A logical image understanding pipeline is introduced to allow parallel processing. As well, we discuss our middleware -VLUT that enables applications to operate transparently over a heterogeneous collection of hardware implementations. OpenVL works as a state machine, with an event-driven mechanism to provide users with application-level interaction. Various explicit or implicit synchronization and communication methods are supported among distributed processes in the logical pipelines. The intent of OpenVL is to allow users to quickly and easily recover useful information from multiple scenes across various software environments and hardware platforms. We implement two different human tracking systems to validate the critical underlying concepts of OpenVL.

In this paper, we propose a multi-camera video surveillance system with automatic camera selection. A new confidence measure, quality-of-view (QOV), is defined to automatically evaluate the camera's view performance for each time instant. This measure takes into account view angle and distance from subjects. By comparing each camera's QOVs, the system can select the most appropriate cameras to perform specific tasks. We also present an approach to determine the minimum number of cameras and their layout in a convex polygonal room under specific QOV constraints. Finally, we implement an experimental surveillance system, to confirm the stability of our algorithm and validate the critical underlying concepts of QOV.

This paper describes an electro-mechanical and software architecture for the development of an anthropomorphic 6 DOF robotic jaw. The architecture comprises the aggregate of motion components needed to position a prosthetic jaw in 3D space. This architecture frees the jaw kinematics from the dependency of constraints in the mechanical assembly allowing for the fabrication of mechanical systems that can simulate jaw motions beyond human capabilities. The orthogonal and concurrent nature of the structural design makes systems based on this idea potentially the easiest to control. To illustrate our concept two distinct prototypes are produced. Construction of these models resulted in the first two modular anthropomorphic robotic jaws to be built with 6-DOF. Areas of application for this mechanical design include dentistry, speech research, and facial gesture affect research.

We report preliminary results of our ongoing field study of IT professionals who are involved in security management. We interviewed a dozen practitioners from five organizations to understand their workplace and tools. We analyzed the interviews using a variation of Grounded Theory and predesigned themes. Our results suggest that the job of IT security management is distributed across multiple employees, often affiliated with different organizational units or groups within a unit and responsible for different aspects of it. The workplace of our participants can be characterized by their responsibilities, goals, tasks, and skills. Three skills stand out as significant in the IT security management workplace: inferential analysis, pattern recognition, and bricolage.

This paper presents an Open Source Vision Library (OpenVL) for hardware acceleration of video-based surveillance systems and other computer vision applications to facilitate low latency, real-time response. Our approach is inspired by the success of OpenGL in promoting the development of hardware acceleration for computer graphics. Our goal is to create OpenVL as a standard interface for computer vision applications that can work seamlessly on different software and hardware platforms supporting distributed camera arrays. It allows users to easily recover useful information about real dynamic scenes quickly, and in a portable manner across various software and hardware platforms. Finally, we implement an example surveillance system, called a Local Positioning System (LPS), to validate the critical underlying concepts of OpenVL.

Cooking is a daily activity for many people. However, traditional text recipes are often prohibitively difficult to follow for people with language disorders, such as aphasia. We have developed a multi-modal application that leverages the retained ability of aphasic individuals to recognize image-based representations of objects, providing a presentation format that can be more easily followed than a traditional text recipe. Through a systematic approach to developing a visual language for cooking, and the subsequent case study evaluation of a prototype developed according to this language, we show that a combination of visual instructions and navigational structure can help individuals with relatively large language deficits to cook more independently.

We present a novel way of manipulating a spatial soundscape, one that encourages collaboration and exploration. Through a table-top display surrounded by speakers and lights, participants are invited to engage in peaceful play with Beluga whales shown through a live web camera feed from the Vancouver Aquarium in Canada. Eight softly glowing buttons and a simple interface encourage collaboration with others who are also enjoying the swirling Beluga sounds overhead.

Enabling group collaboration is important in computer graphics today. We have developed a framework that supports multiple pointing devices to explore the collaborative utility of multiple mice and laser pointer interaction in graphical environments. Because most pointing device comparisons are done in the context of single user performance, very little is known about the affordances of collaborating with multiple pointing devices. We present an experimental comparison of mouse pointer to laser pointer interaction in a problem-solving task involving groups of one, two, and three people. We show that collaborative performance is largely orthogonal to motor performance and that the interaction patterns are dependent on the task and on the group size. This suggests that the collaborative characteristics of a pointing device are just as important as the physical characteristics that are usually given the most attention, such as precision and accuracy.

In this paper we consider the general goal of supporting physical manipulation of digital audio in a specific context: the performance disk jockey (DJ) seeking to migrate from vinyl to digital media. We classify both the DJ's traditional processes and tools and the field's newest technology.D'Groove, our own technological contribution, is a force feedback turntable used to manipulate digital audio in novel ways. We present an observational study of professional DJ's using D'Groove, and discuss this approach's attributes and directions for future augmentation. Finally, we extend our conclusions about the DJ's emerging needs to the broader domain of digital audio manipulation.

No abstract available.

Contemporary musical instrument design using computers provides nearly limitless potential for designing the mapping between gesture and sound. When designing effective and expressive musical instruments, the types of relationship between musician/player and his instrument and the aesthetics of the relationships must be considered. This paper discusses four types of relationships and their aesthetics. A high degree of intimacy is achieved when the relationship reaches a level where the mapping between control and sound is transparent to the player, that is, the player embodies the device. Ultimately, this type of relationship allows intent and expression to flow through the player to the sound and, hence, create music. Three new interfaces for musical expression, the Iamascope, Sound Sculpting and Tooka, provide examples of how instruments may be designed to develop and explore intimacy and embodiment of new musical instruments.

In this paper, we discuss a new hand-gesture-based application, FIowField, which explores the use of pressure-sensitive multi-point touchpad (multi-touch) interaction. FIowField allows participants to interact using their whole hand with a flow of circulating particles, providing visual and auditory feedback. Limitations of the raw data mapping used in FIowField motivated work on applying interpolation techniques to improve the data representation. Thus far, bicubic interpolation provides the most effective method for initial processing. We observed two factors limiting the quality of the representation: the spacing of the sensors, and the pressure-distributing property of the touchpad surface.

Multi-feature analysis of human chromosome images is a major step towards classification of homologous chromosomes. An automatic quantitative classification method is proposed for homolog differentiation using multiple features. This method is based on mutual information maximization applied to an unsupervised neural network architecture. The neural network consists of separate modules which are trained to classify homologs using independent features. Mutual information is then maximized between the outputs of the modules forcing them to produce the same classification results, for a given chromosome. The proposed method was successfully applied to classify homologs of chromosome 16 with 100% accuracy.

People have aesthetic experiences when they manipulate objects skillfully. Highly skilled performance with an object requires forming a highly intimate relationship with it. Aesthetics flow from this intimacy. This paper discusses three works that illustrate how intimacy and embodiment can be useful concepts for designing human-machine interfaces. The three works include: Iamascope, Glove-TalkII and the Forklift Ballet.

EasyOGL is a Tcl/Tk extension that adds three new dynamically loadable packages: a new Tk 3D graphics widget, an OpenGL package and an OpenGLU package. The 3D graphics widget provides a context for drawing 3D graphics. The OpenGL/GLU packages provide interpretive access to the OpenGL/GLU libraries so that developers can script interactive 3D graphics that appear inside an active 3D drawing context. Since EasyOGL is embedded in Tcl, it provides an extensible, easy-to-use, fast prototyping environment for 3D multimedia applications.

This paper demonstrates how non-isomorphic rotational mappings and interaction techniques can be designed and used to build effective spatial 3D user interfaces. In this paper, we develop a mathematical framework allowing us to design non-isomorphic 3D rotational mappings and techniques, investigate their usability properties, and evaluate their user performance characteristics. The results suggest that non-isomorphic rotational mappings can be an effective tool in building high-quality manipulation dialogs in 3D interfaces, allowing our subjects to accomplish experimental tasks 13% faster without a statistically detectable loss in accuracy. The current paper will help interface designers to use non-isomorphic rotational mappings effectively.

Classification of homologous human chromosomes is essential to advanced studies of cancer genetics. This paper describes novel segmentation and classification algorithms to extract multiple features, from microscopy images of chromosomes, for classification purposes. Multicolour images of metaphase chromosomes prepared by applying PNA probes are used for this purpose. Centromeres are segmented using an iterative fuzzy algorithm as well as a gradient method. Moreover, telomere length measurements are performed on chromosome images and normalized for the image database. Multiple intensity features are calculated as a result of the developed algorithms. Heteromorphic chromosomes (such as 16 and 22) are then successfully classified into their parental homologues, based on the calculated multiple features, and used to verify the developed methods.

This paper presents an interactive video visualization technique called video cubism. With this technique, video data is considered to be a block of three dimensional data where frames of video data comprise the third dimension. The user can observe and manipulate a cut plane or cut sphere through the video data. An external real-time video source may also be attached to the video cube. The visualization leads to images that are aesthetically interesting as well as being useful for image analysis.

Glove-Talk II is a system which translates hand gestures to speech through an adaptive interface. Hand gestures are mapped continuously to ten control parameters of a parallel formant speech synthesizer. The mapping allows the hand to act as an artificial vocal tract that produces speech in real time. This gives an unlimited vocabulary in addition to direct control of fundamental frequency and volume. Currently, the best version of Glove-Talk II uses several input devices, a parallel formant speech synthesizer, and three neural networks. The gesture-to-speech task is divided into vowel and consonant production by using a gating network to weight the outputs of a vowel and a consonant neural network. The gating network and the consonant network are trained with examples from the user. The vowel network implements a fixed user-defined relationship between hand position and vowel sound and does not require any training examples from the user. Volume, fundamental frequency, and stop consonants are produced with a fixed mapping from the input devices. With Glove-Talk II, the subject can speak slowly but with far more natural sounding pitch variations than a text-to-speech synthesizer.

Glove-Talk II is a system which translates hand gestures to speech through an adaptive interface. Hand gestures are mapped continuously to ten control parameters of a parallel formant speech synthesizer. The mapping allows the hand to act as an artificial vocal tract that produces speech in real time. This gives an unlimited vocabulary in addition to direct control of fundamental frequency and volume. Currently, the best version of Glove-Talk II uses several input devices, a parallel formant speech synthesizer, and three neural networks. The gesture-to-speech task is divided into vowel and consonant production by using a gating network to weight the outputs of a vowel and a consonant neural network. The gating network and the consonant network are trained with examples from the user. The vowel network implements a fixed user-defined relationship between hand position and vowel sound and does not require any training examples from the user. Volume, fundamental frequency, and stop consonants are produced with a fixed mapping from the input devices. With Glove-Talk II, the subject can speak slowly but with far more natural sounding pitch variations than a text-to-speech synthesizer.

Article Glove-TalkII: an adaptive gesture-to-formant interface Share on Authors: Sidney Fels Department of Computer Science & Department of Computer Science, University of Toronto & University of Toronto, Toronto, ON, Canada, M5S 1A4 & Toronto, ON, Canada, M5S 1A4 Department of Computer Science & Department of Computer Science, University of Toronto & University of Toronto, Toronto, ON, Canada, M5S 1A4 & Toronto, ON, Canada, M5S 1A4View Profile , Geoffrey Hinton Department of Computer Science & Department of Computer Science, University of Toronto & University of Toronto, Toronto, ON, Canada, M5S 1A4 & Toronto, ON, Canada, M5S 1A4 Department of Computer Science & Department of Computer Science, University of Toronto & University of Toronto, Toronto, ON, Canada, M5S 1A4 & Toronto, ON, Canada, M5S 1A4View Profile Authors Info & Claims CHI '95: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMay 1995 Pages 456–463https://doi.org/10.1145/223904.223966Online:01 May 1995Publication History 24citation473DownloadsMetricsTotal Citations24Total Downloads473Last 12 Months18Last 6 weeks4 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Glove-TaikII is a system which translates hand gestures to speech through an adaptive interface. Hand gestures are mapped contin(cid:173) uously to 10 control parameters of a parallel formant speech syn(cid:173) thesizer. The mapping allows the hand to act as an artificial vocal tract that produces speech in real time. This gives an unlimited vocabulary in addition to direct control of fundamental frequency and volume. Currently, the best version of Glove-TalkII uses sev(cid:173) eral input devices (including a CyberGlove, a ContactGlove, a 3- space tracker, and a foot-pedal), a parallel formant speech synthe(cid:173) sizer and 3 neural networks. The gesture-to-speech task is divided into vowel and consonant production by using a gating network to weight the outputs of a vowel and a consonant neural network. The gating network and the consonant network are trained with examples from the user. The vowel network implements a fixed, user-defined relationship between hand-position and vowel sound and does not require any training examples from the user. Volume, fundamental frequency and stop consonants are produced with a fixed mapping from the input devices. One subject has trained to speak intelligibly with Glove-TalkII. He speaks slowly with speech quality similar to a text-to-speech synthesizer but with far more natural-sounding pitch variations. 844 S. Sidney Fe Is, Geoffrey Hinton

To illustrate the potential of multilayer neural networks for adaptive interfaces, a VPL Data-Glove connected to a DECtalk speech synthesizer via five neural networks was used to implement a hand-gesture to speech system. Using minor variations of the standard backpropagation learning procedure, the complex mapping of hand movements to speech is learned using data obtained from a single ;speaker' in a simple training phase. With a 203 gesture-to-word vocabulary, the wrong word is produced less than 1% of the time, and no word is produced about 5% of the time. Adaptive control of the speaking rate and word stress is also available. The training times and final performance speed are improved by using small, separate networks for each naturally defined subtask. The system demonstrates that neural networks can be used to develop the complex mappings required in a high bandwidth interface that adapts to the individual user.