Crowdsourcing enables human workers to perform designated tasks unbounded by time and location. As mobile devices and embedded cameras have become widely available, we deployed an image capture task globally for more geographically diverse images. Via our micro-crowdsourcing mobile application, users capture images of surrounding subjects, tag with keywords, and can choose to open source their work. We open-sourced 478,000 images collected from worldwide users as a dataset "Open Images Extended" that aims to add global diversity to imagery training data. We describe our approach and workers' feedback through survey responses from 171 global contributors to this task.

Developing interactive systems often involves a large set of callback functions for handling user interaction, which makes it challenging to manage UI behaviors, create descriptive documentation, and track code revisions. We developed Doppio, a tool that automatically tracks and visualizes UI flows and their changes based on source code. For each input event listener of a widget, e.g., onClick of an Android View class, Doppio captures and associates its UI output from a program execution with its code snippet from the codebase. It automatically generates a screenflow diagram organized by the callback methods and interaction flow, where developers can review the code and UI revisions interactively. Doppio, as an IDE plugin, is seamlessly integrated into a common development workflow. Our studies show that our tool is able to generate quality visual documentation and helped participants understand unfamiliar source code and track changes.

Cross-device interactions involve input and output on multiple computing devices. Implementing and reasoning about interactions that cover multiple devices with a diversity of form factors and capabilities can be complex. To assist developers in programming cross-device interactions, we created DemoScript, a technique that automatically analyzes a cross-device interaction program while it is being written. DemoScript visually illustrates the step-by-step execution of a selected portion or the entire program with a novel, automatically generated cross-device storyboard visualization. In addition to helping developers understand the behavior of the program, DemoScript also allows developers to revise their program by interactively manipulating the cross-device storyboard. We evaluated DemoScript with 8 professional programmers and found that DemoScript significantly improved development efficiency by helping developers interpret and manage cross-device interaction; it also encourages testing to think through the script in a development process.

Illustrations of human movements are used to communicate ideas and convey instructions in many domains, but creating them is time-consuming and requires skill. We introduce DemoDraw, a multi-modal approach to generate these illustrations as the user physically demonstrates the movements. In a Demonstration Interface, DemoDraw segments speech and 3D joint motion into a sequence of motion segments, each characterized by a key pose and salient joint trajectories. Based on this sequence, a series of illustrations is automatically generated using a stylistically rendered 3D avatar annotated with arrows to convey movements. During demonstration, the user can navigate using speech and amend or re-perform motions if needed. Once a suitable sequence of steps has been created, a Refinement Interface enables fine control of visualization parameters. In a three-part evaluation, we validate the effectiveness of the generated illustrations and the usability of DemoDraw. Our results show 4 to 7-step illustrations can be created in 5 or 10 minutes on average.

We present Weave, a framework for developers to create cross-device wearable interaction by scripting. Weave provides a set of high-level APIs, based on JavaScript, for developers to easily distribute UI output and combine sensing events and user input across mobile and wearable devices. Weave allows developers to focus on their target interaction behaviors and manipulate devices regarding their capabilities and affordances, rather than low-level specifications. Weave also contributes an integrated authoring environment for developers to program and test cross-device behaviors, and when ready, deploy these behaviors to its runtime environment on users' ad-hoc network of devices. An evaluation of Weave with 12 participants on a range of tasks revealed that Weave significantly reduced the effort of developers for creating and iterating on cross-device interaction.

Showing a live software demonstration during a talk can be engaging, but it is often not easy: presenters may struggle with (or worry about) unexpected software crashes and encounter issues such as mismatched screen resolutions or faulty network connectivity. Furthermore, it can be difficult to recall the steps to show while talking and operating the system all at the same time. An alternative is to present with pre-recorded screencast videos. It is, however, challenging to precisely match the narration to the video when using existing video players. We introduce DemoWiz, a video presentation system that provides an increased awareness of upcoming actions through glanceable visualizations. DemoWiz supports better control of timing by overlaying visual cues and enabling lightweight editing. A user study shows that our design significantly improves the presenters' perceived ease of narration and timing compared to a system without visualizations that was similar to a standard playback control. Furthermore, nine (out of ten) participants preferred DemoWiz over the standard playback control with the last expressing no preference.

Amateur instructional videos often show a single uninterrupted take of a recorded demonstration without any edits. While easy to produce, such videos are often too long as they include unnecessary or repetitive actions as well as mistakes. We introduce DemoCut, a semi-automatic video editing system that improves the quality of amateur instructional videos for physical tasks. DemoCut asks users to mark key moments in a recorded demonstration using a set of marker types derived from our formative study. Based on these markers, the system uses audio and video analysis to automatically organize the video into meaningful segments and apply appropriate video editing effects. To understand the effectiveness of DemoCut, we report a technical evaluation of seven video tutorials created with DemoCut. In a separate user evaluation, all eight participants successfully created a complete tutorial with a variety of video editing effects using our system.

Users of complex software applications often learn concepts and skills through step-by-step tutorials. Today, these tutorials are published in two dominant forms: static tutorials composed of images and text that are easy to scan, but cannot effectively describe dynamic interactions; and video tutorials that show all manipulations in detail, but are hard to navigate. We hypothesize that a mixed tutorial with static instructions and per-step videos can combine the benefits of both formats. We describe a comparative study of static, video, and mixed image manipulation tutorials with 12 participants and distill design guidelines for mixed tutorials. We present MixT, a system that automatically generates step-by-step mixed media tutorials from user demonstrations. MixT segments screencapture video into steps using logs of application commands and input events, applies video compositing techniques to focus on salient infor-mation, and highlights interactions through mouse trails. Informal evaluation suggests that automatically generated mixed media tutorials were as effective in helping users complete tasks as tutorials that were created manually.

Social media enables people to share personal experiences, often through real-time media such as chat. People also record their life experiences in media collections, with photos and video. However, today's social media force a choice between real-time communication, and authoring a coherent story illustrated with digital media. There is simply not enough time in real-time communication to select and compose coherent multimedia stories.

People who are not professional storytellers usually have difficulty composing travel photos and videos from a mundane slideshow into a coherent and engaging story, even when it is about their own experiences. However, consider putting the same person in a conversation with a friend - suddenly the story comes alive.

When editing a story from a large collection of media, such as photos and video clips captured from daily life, it is not always easy to understand how particular scenes fit into the intent for the overall story. Especially for novice editors, there is often a lack of coherent connections between scenes, making it difficult for the viewers to follow the story.

This case study in UbiComp technology and design presents a system for assisting parents and teachers to motivate kindergarten children to learn proper and thorough brushing skills. The system includes a vision-based motion tracker that recognizes different tooth brushing strokes and a tooth brushing game in which the child cleans a virtual, mirror picture of his/her dirty teeth by physically brushing his/her own teeth. The user study results suggest that Playful Toothbrush enhances the effectiveness of kindergarten children in brushing their teeth, as measured by number of brushing strokes, duration of brushing and thoroughness of teeth cleaning.