Document dewarping is crucial for many applications. However, existing learning-based methods rely heavily on supervised regression with annotated data without fully leveraging the inherent geometric properties of physical documents. Our key insight is that a well-dewarped document is defined by its axis-aligned feature lines. This property aligns with the inherent axis-aligned nature of the discrete grid geometry in planar documents. Harnessing this property, we introduce three synergistic contributions: for the training phase, we propose an axis-aligned geometric constraint to enhance document dewarping; for the inference phase, we propose an axis alignment preprocessing strategy to reduce the dewarping difficulty; and for the evaluation phase, we introduce a new metric, Axis-Aligned Distortion (AAD), that not only incorporates geometric meaning and aligns with human visual perception but also demonstrates greater robustness. As a result, our method achieves state-of-the-art performance on multiple existing benchmarks, improving the AAD metric by 18.2% to 34.5%.

Ding Xia, Xinyue Gui, Mark Colley, Fan Gao, Zhongyi Zhou, Dongyuan Li, Renhe Jiang, Takeo Igarashi. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Pedestrian–automated vehicle (AV) encounters in shared spaces often involve hesitation and ambiguity. Vehicle-mounted external human–machine interfaces (eHMIs) can help, but obscured or poorly timed communications create significant challenges. To address this, we present a mobile smart pole interaction unit (SPIU) with integrated cameras and LED displays, designed as a pedestrian-side system to deliver explicit cues (“WALK,” “STOP”). An in-the-wild evaluation of the SPIU (N = 21) using a four-factor analysis (CarBehavior, Mobility, eHMI, SPIU) showed that the SPIU improved understandability, trust, and perceived safety, and reduced workload compared with the baseline, with a combination (eHMI+SPIU) yielding the strongest results. Beyond these quantitative benefits, participants appreciated the mobility of the SPIU for its “clear” and “easy to decide” mediation. This work contributes to (1) a design and deployment framework for a mobile SPIU and (2) an in-the-wild evaluation protocol for pedestrian–AV interactions in nonsignalized spaces. Our work sparks discussions on real world evaluations involving detailed vehicle kinematics and accessible multimodality (e.g., audio), focusing on the role of personal robots as user-side eHMIs.

Field studies are irreplaceable but costly, time-consuming, and error-prone, which need careful preparation. Inspired by rapid-prototyping in manufacturing, we propose a fast, low-cost evaluation method using Vision-Language Model (VLM) personas to simulate outcomes comparable to field results. While LLMs show human-like reasoning and language capabilities, autonomous vehicle (AV)-pedestrian interaction requires spatial awareness, emotional empathy, and behavioral generation. This raises our research question: To what extent can VLM personas mimic human responses in field studies? We conducted parallel studies: 1) one real-world study with 20 participants, and 2) one video-study using 20 VLM personas, both on a street-crossing task. We compared their responses and interviewed five HCI researchers on potential applications. Results show that VLM personas mimic human response patterns (e.g., average crossing times of 5.25 s vs. 5.07 s) lack the behavioral variability and depth. They show promise for formative studies, field study preparation, and human data augmentation.

Animation production workflows often involve digital colorization of line art, where small unpainted regions (“gaps”) frequently occur and remain an underexplored challenge. We conducted a formative study in Japanese animation (anime) pipelines and found that while the paint bucket tool is widely used for base coloring, tiny enclosed areas are frequently overlooked, resulting in time-consuming manual detection and filling. We introduce GapFill, a tool grounded in professional practices that reduces the effort of gap detection, zooming, and color selection. Our deep-learning method suggests appropriate fill colors by referencing surrounding regions, leveraging the flat-color nature of anime-style images. In a user study with 13 professional colorists, our system improved performance and usability in gap-filling tasks over conventional methods. The study also suggested that prediction accuracy alone is not the primary factor for usability, that appropriate colors can be contextually ambiguous, and that GapFill can complement existing tools depending on users’ trust in new AI-powered assistance.

Despite widespread use, charts remain largely inaccessible for Low-Vision Individuals (LVI). Reading charts requires viewing data points within a global context, which is difficult for LVI who may rely on magnification or experience a partial field of vision. We aim to improve exploration by providing visual access to critical context. To inform this, we conducted a formative study with five LVI. We identified four fundamental contextual elements common across chart types: axes, legend, grid lines, and the overview. We propose two pointer-based interaction methods to provide this context: Dynamic Context, a novel focus+context interaction, and Mini-map, which adapts overview+detail principles for LVI. In a study with N=22 LVI, we compared both methods and evaluated their integration to current tools. Our results show that Dynamic Context had significant positive impact on access, usability, and effort reduction; however, worsened visual load. Mini-map strengthened spatial understanding, but was less preferred for this task. We offer design insights to guide the development of future systems that support LVI with visual context while balancing visual load.

Multiparty conversations are ubiquitous and indispensable in diverse social and collaborative contexts. However, current conversational agents (CAs) face significant challenges in effectively engaging in such interactions, particularly within text-based environments. While earlier limitations were often attributed to the inadequacies of AI models, recent advances in large language models now compel us to revisit both our understanding of multiparty conversation and the way we design CAs. This paper synthesizes findings from two complementary qualitative investigations and proposes a conceptual model for designing CAs that can genuinely participate, rather than merely function as tools or outsiders. The first study, employing retrospective think-aloud sessions (N=30) with users in text-based multiparty settings, uncovers 5 key interactional mechanisms (e.g., Turn-taking Management, Presence Management) that underpin successful human-human multiparty interactions, derived from participants’ articulated perceptions and reasoning. Subsequently, the second study, through semi-structured interviews (N=15), identifies user expectations for CA integration and key traits (e.g., proactivity, social authenticity) that shape an ideal CA persona perceived by users as a genuine participant. Drawing from these human-centric insights, we then derive design considerations, aiming to guide the development of CAs capable of more natural, effective, and socially intelligent participation in multiparty conversation.

Toggle (a.1) Zoom settings panel (a.2) Highlight settings panel (a) The VeasyGuide player (b) Zoomed-in view (c) VeasyGuide's personalization settings panelsFigure 1: VeasyGuide's interface includes: (a) a video player with highlighted areas (red rectangle and hand pointer), (a.1) zoom settings toggle, (a.2) highlight settings toggle, (b) a zoomed window (toggled with the Z key), and (c) zoom and highlight settings panels.While watching a video, VeasyGuide auto-highlights pointing, marking, and sketching activities.Users can zoom into highlighted portions with the Z key, adjust zoom with arrow keys, and customize highlight and zoom appearance in real time.

Remote meetings using 3D avatars in Augmented Reality (AR) allow effective communication and enable users to retain awareness of their surroundings. However, positioning 3D avatars effectively and consistently for all users in AR is challenging since most spaces, such as offices or living rooms, are not large enough to accommodate multiple life-sized avatars without interference. To address this issue, we contribute MiniMates—a novel approach leveraging miniature avatars, which make it possible to place multiple remote users in a limited physical space. We see MiniMates as complementary to traditional 2D video conferencing and immersive telepresence. Our approach automatically adjusts the formation of avatars and redirects users’ head and body orientation to facilitate communication. Results from our user study (n = 24) show that participants experience a higher sense of co-presence compared to video conferencing, and that MiniMates enabled them to communicate the direction of their interactions non-verbally as well as manage multiple simultaneous conversations.

International audience

International audience

Compliant mechanisms enable the creation of compact and easy-to-fabricate devices for tangible interaction. This work explores interconnected compliant mechanisms consisting of multiple joints and rigid bodies to transmit and process displacements as signals that result from physical interactions. As these devices are difficult to design due to their vast and complex design space, we developed a graph-based design algorithm and computational tool to help users program and customize such computational functions and procedurally model physical designs. When combined with active materials with actuation and sensing capabilities, these devices can also render and detect haptic interaction. Our design examples demonstrate the tool's capability to respond to relevant HCI concepts, including building modular physical interface toolkits, encrypting tangible interactions, and customizing user augmentation for accessibility. We believe the tool will facilitate the generation of new interfaces with enriched affordance.

Numerous recent works have utilized 3D Gaussian Splatting to represent high-fidelity digital avatars. However, none have enabled interactive multilayer Gaussian garments for virtual try-ons without relying on expensive hardware, such as a camera array and/or multiple GPUs. To enable affordable mix-and-match dressing—dressing 3D avatars with realistic and complex combinations of garments—it is crucial to handle the interactions between multiple layers of garments using consumer-level capturing hardware. To address this, we present a novel screenspace layer resolution method combined with physical simulation and Gaussian garments to enable realistic multilayer mix-and-match avatar dressing at interactive rates using low-cost hardware. As an offline process, we capture multiple static garments individually using only a single mobile camera on a static mannequin and then perform a dual reconstruction of Gaussians and simulation mesh. During runtime, these Gaussians are driven by a fast but simple physics simulator, whose output may contain inter-penetrations across garment layers. Our method fixes these in screenspace by rasterizing the simulation mesh from various camera views and culling the Gaussians that are skinned to unseen mesh triangles. We show the effectiveness of our approach by demonstrating mix-and-match dressing results at interactive rates using short-sleeves, long-sleeves, a fur vest, and a singlet. Additionally, we showcase a webcam-based interactive try-on application to further illustrate the capabilities of our system.

Automated vehicles (AVs) are gradually becoming part of our daily lives. However, effective communication between road users and AVs remains a significant challenge. Although various external human-machine interfaces (eHMIs) have been developed to facilitate interactions, psychological factors, such as a lack of trust and inadequate emotional signaling, may still deter users from confidently engaging with AVs in certain contexts. To address this gap, we propose TailCue, an exploration of how tail-based eHMIs affect user interaction with AVs. We first investigated mappings between tail movements and emotional expressions from robotics and zoology, and accordingly developed a motion-emotion mapping scheme. A physical robotic tail was implemented, and specific tail motions were designed based on our scheme. An online, video-based user study with 21 participants was conducted. Our findings suggest that, although the intended emotions conveyed by the tail were not consistently recognized, open-ended feedback indicated that the tail motion needs to align with the scenarios and cues. Our result highlights the necessity of scenario-specific optimization to enhance tail-based eHMIs. Future work will refine tail movement strategies to maximize their effectiveness across diverse interaction contexts.

We introduce NeRF-GS, a novel framework that jointly optimizes Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). This framework leverages the inherent continuous spatial representation of NeRF to mitigate several limitations of 3DGS, including sensitivity to Gaussian initialization, limited spatial awareness, and weak inter-Gaussian correlations, thereby enhancing its performance. In NeRF-GS, we revisit the design of 3DGS and progressively align its spatial features with NeRF, enabling both representations to be optimized within the same scene through shared 3D spatial information. We further address the formal distinctions between the two approaches by optimizing residual vectors for both implicit features and Gaussian positions to enhance the personalized capabilities of 3DGS. Experimental results on benchmark datasets show that NeRF-GS surpasses existing methods and achieves state-of-the-art performance. This outcome confirms that NeRF and 3DGS are complementary rather than competing, offering new insights into hybrid approaches that combine 3DGS and NeRF for efficient 3D scene representation.

As autonomous vehicles (AVs) enter pedestrian-centric environments, existing vehicle-mounted external human–machine interfaces (eHMIs) often fall short in shared spaces due to line-of-sight limitations, inconsistent signaling, and increased decision latency on pedestrians. To address these challenges, we introduce the Smart Pole Interaction Unit (SPIU), an infrastructure-based eHMI that decouples intent signaling from vehicles and provides context-aware, elevated visual cues. We evaluate SPIU using immersive VR-AWSIM simulations in four high-risk urban scenarios: four-way intersections, autonomous mixed traffic, blindspots, and nighttime crosswalks. The experiment was developed in Japan and replicated in Norway, where forty participants engaged in 32 trials each under both SPIU-present and SPIU-absent conditions. Behavioral (response time) and subjective (acceptance scale) data were collected. Results show that SPIU significantly improves pedestrian decision-making, with reductions ranging from 40% to over 80% depending on scenario and cultural context, particularly in complex or low-visibility scenarios. Cross-cultural analyses highlight SPIU’s adaptability across differing urban and social contexts. We release our open-source Smartpole-VR-AWSIM framework to support reproducibility and global advancement of infrastructure-based eHMI research through reproducible and immersive behavioral studies.

Abstract Recent deep learning methods can generate diverse graphic design layouts efficiently. However, these methods often create layouts with flaws, such as misalignment, unwanted overlaps, and unsatisfied containment. To tackle this issue, we propose an optimization‐based method called LayoutRectifier, which gracefully rectifies auto‐generated graphic design layouts to reduce these flaws while minimizing deviation from the generated layout. The core of our method is a two‐stage optimization. First, we utilize grid systems, which professional designers commonly use to organize elements, to mitigate misalignments through discrete search. Second, we introduce a novel box containment function designed to adjust the positions and sizes of the layout elements, preventing unwanted overlapping and promoting desired containment. We evaluate our method on content‐agnostic and content‐aware layout generation tasks and achieve better‐quality layouts that are more suitable for downstream graphic design tasks. Our method complements learning‐based layout generation methods and does not require additional training.

Abstract Per‐garment virtual try‐on methods collect garment‐specific datasets and train networks tailored to each garment to achieve superior results. However, these approaches often struggle with loose‐fitting garments due to two key limitations: (1) They rely on human body semantic maps to align garments with the body, but these maps become unreliable when body contours are obscured by loose‐fitting garments, resulting in degraded outcomes; (2) They train garment synthesis networks on a per‐frame basis without utilizing temporal information, leading to noticeable jittering artifacts. To address the first limitation, we propose a two‐stage approach for robust semantic map estimation. First, we extract a garment‐invariant representation from the raw input image. This representation is then passed through an auxiliary network to estimate the semantic map. This enhances the robustness of semantic map estimation under loose‐fitting garments during garment‐specific dataset generation. To address the second limitation, we introduce a recurrent garment synthesis framework that incorporates temporal dependencies to improve frame‐to‐frame coherence while maintaining real‐time performance. We conducted qualitative and quantitative evaluations to demonstrate that our method outperforms existing approaches in both image quality and temporal coherence. Ablation studies further validate the effectiveness of the garment‐invariant representation and the recurrent synthesis framework.

Traditional integral wood joints, despite their strength, durability, and elegance, remain rare in modern workflows due to the cost and difficulty of manual fabrication. CNC milling offers a scalable alternative, but directly milling traditional joints often fails to produce functional results because milling induces geometric deviations—such as rounded inner corners—that alter the target geometries of the parts. Since joints rely on tightly fitting surfaces, such deviations introduce gaps or overlaps that undermine fit or block assembly. We propose to overcome this problem by (1) designing a language that represent millable geometry, and (2) co-optimizing part geometries to restore coupling. We introduce Millable Extrusion Geometry (MXG), a language for representing geometry as the outcome of milling operations performed with flat-end drill bits. MXG represents each operation as a subtractive extrusion volume defined by a tool direction and drill radius. This parameterization enables the modeling of artifact-free geometry under an idealized zero-radius drill bit, matching traditional joint designs. Increasing the radius then reveals milling-induced deviations, which compromise the integrity of the joint. To restore coupling, we formalize tight coupling in terms of both surface proximity and proximity constraints on the mill-bit paths associated with mating surfaces. We then derive two tractable, differentiable losses that enable efficient optimization of joint geometry. We evaluate our method on 30 traditional joint designs, demonstrating that it produces CNC-compatible, tightly fitting joints that approximates the original geometry. By reinterpreting traditional joints for CNC workflows, we continue the evolution of this heritage craft and help ensure its relevance in future making practices.

We present the Mokume dataset for solid wood texturing consisting of 190 cube-shaped samples of various hard and softwood species documented by high-resolution exterior photographs, annual ring annotations, and volumetric computed tomography (CT) scans. A subset of samples further includes photographs along slanted cuts through the cube for validation purposes. Using this dataset, we propose a three-stage inverse modeling pipeline to infer solid wood textures using only exterior photographs. Our method begins by evaluating a neural model to localize year rings on the cube face photographs. We then extend these exterior 2D observations into a globally consistent 3D representation by optimizing a procedural growth field using a novel iso-contour loss. Finally, we synthesize a detailed volumetric color texture from the growth field. For this last step, we propose two methods with different efficiency and quality characteristics: a fast inverse procedural texture method, and a neural cellular automaton (NCA). We demonstrate the synergy between the Mokume dataset and the proposed algorithms through comprehensive comparisons with unseen captured data. We also present experiments demonstrating the efficiency of our pipeline's components against ablations and baselines. Our code, the dataset, and reconstructions are available via https://mokumeproject.github.io/.

Abstract This paper proposes a method to estimate the locations of grid handles in free-form deformation (FFD) while preserving the local shape characteristics of the 2D/3D input model embedded into the grid, named locality-preserving FFD (lp-FFD). Users first specify some vertex locations in the input model and grid handle locations. The system then optimizes all locations of grid handles by minimizing the distortion of the input model’s mesh elements. The proposed method is fast and stable, allowing the user to directly and indirectly make the deformed shape of the mesh model and grid. This paper shows some examples of deformation results to demonstrate the robustness of our lp-FFD. In addition, we conducted a user study and confirm our lp-FFD’s efficiency and effectiveness in shape deformation is higher than those of existing methods used in commercial software.

When employing autonomous driving technology in logistics, small autonomous delivery vehicles (aka delivery robots) encounter challenges different from passenger vehicles when interacting with other road users. We conducted an online video survey as a pre-study and found that autonomous delivery vehicles need external human-machine interfaces (eHMIs) to ask for help due to their small size and functional limitations. Inspired by everyday human communication, we chose arms as eHMI to show their request through limb motion and gesture. We held an in-house workshop to identify the arm’s requirements for designing a specific arm with shrink-ability (conspicuous when delivering messages but not affect traffic at other times). We prototyped a small delivery robot with a shrinkable arm and filmed the experiment videos. We conducted two studies (a video-based and a 360-degree-photo VR-based) with 18 participants. We demonstrated that arm-on-delivery robots can increase interaction efficiency by drawing more attention and communicating specific information.

Reconstructing 3D human poses from video has wide applications, such as character animation and sports analysis. Automatic 3D pose reconstruction methods have demonstrated promising results, but failure cases can still appear due to the diversity of human actions, capturing conditions, and depth ambiguities. Thus, manual intervention remains indispensable, which can be time-consuming and require professional skills. We thus present iPose, an interactive tool that facilitates intuitive human pose reconstruction from a given video. Our tool incorporates both human perception in specifying pose appearance to achieve controllability, and video frame processing algorithms to achieve precision and automation. A user manipulates the projection of a 3D pose via 2D operations on top of video frames, and the 3D poses are updated correspondingly while satisfying both kinematic and video frame constraints. The pose updates are propagated temporally to reduce user workload. We evaluate the effectiveness of iPose with a user study on the 3DPW dataset and expert interviews.

The integration of advanced technologies in healthcare necessitates the development of systems accommodating the daily routines in medical practices. Neurosurgeons, in particular, require extensive practice in microsurgical suturing in the long term, even in the busy routine of a medical practice. This study collaboratively developed a Mixed Reality system with neurosurgeons to support self-training in microscopic suturing. Based on the neurosurgeons’ opinions, we implemented a level-appropriate microsurgical suture training system. For novices, the system offers shadow-matching training to support the practice of precise movements under the high-sensitivity environment of the microscope. For intermediates, it provides a real-time feedback system, which allows users to practice attention to details. Evaluation involved testing the novice system on students with no medical background and the intermediate system on neurosurgery residents. The effectiveness of the system was demonstrated through the experimental results and subsequent discussion.

This study presents “Speed Labeling”, an image-labeling technique to increase the efficiency of easy binary labeling tasks where an annotator can choose a label instantly. We first conduct a formative study to identify the factors affecting the efficiency of easy image labeling: image layout and image transition. Based on these results, we designed a novel labeling technique using non- stop scrolling. In conventional image labeling, the system moves to the next image only after the user assigns a label to the previous image. To maximize efficiency, our technique continuously scrolls images without waiting for the completion of labeling, assuming that the user gives labels at a mostly constant speed. The system dynamically adjusts the scrolling speed based on the labeling speed. Subsequently, we conduct a user study to compare the proposed “non-stop scrolling” technique to the conventional “stop-and-go scrolling” technique in an easy image-labeling task. The results showed that speed labeling requires less time (faster by 7%, 305 more images labeled per man-hour) to complete the labeling task than the conventional technique without a significant increase in errors. In addition, the results showed that speed labeling makes the labeling task more enjoyable for crowd workers and makes them feel more attentive during tasks.

The popularity of virtual try-on methods has increased in recent years as they allow users to preview the appearance of garments on themselves without physically wearing them. However, existing image-based methods for general virtual try-on provide limited support to synthesize realistic and consistent garment images under different poses, due to two main difficulties: 1) the dataset used to train these methods contains a vast collection of garments, but they lack fine details of each garment; 2) they synthesize results by warping the front-view image of the target garment in a rest pose, which results in poor quality and detail for other viewpoints and poses. To overcome these drawbacks, per-garment virtual try-on methods train garment-specific networks that can produce high-quality results with fine-grained details for a particular target garment. However, existing per-garment virtual try-on methods require the use of a physical measurement garment, which limits their applicability. In this paper, we propose a novel per-garment virtual try-on method that leverages a virtual measurement garment, which eliminates the need for the physical measurement garment, to guide the synthesis of high-quality and temporally consistent garment images under various poses. Furthermore, we introduce a gap-filling module that effectively fills the gap between the synthesized garment and body parts. We conduct qualitative and quantitative evaluations against a state-of-the-art image-based virtual try-on method and ablation studies to demonstrate that our method achieves superior performance in terms of realism and consistency of the generated garment images.

While text-based external human-machine interface (eHMI) is widely accepted, one limitation is the lack of capability to communicate spatial information such as a different person or location. We built a mixed-eHMI using "eye" as a target-specifier when "text" shows the clear intention to their communication partners. We conducted a pre-experimental observation to develop two testbed scenarios, followed by a video-based user study via life-size projection with a real-car prototype mounted a text display and a set of robotic eyes. The results demonstrated that our proposed "text + eye" combination may represent geospatial information by increasing the success pick-up rate.

Humans use multiple communication channels to interact with each other. For instance, body gestures or facial expressions are commonly used to convey an intent. The use of such non-verbal cues has motivated the development of prediction models. One such approach is predicting arousal and valence (AV) from facial expressions. However, making these models accurate for human-robot interaction (HRI) settings is challenging as it requires handling multiple subjects, challenging conditions, and a wide range of facial expressions. In this paper, we propose a data augmentation (DA) technique to improve the performance of AV predictors using 3D morphable models (3DMM). We then utilize this approach in an HRI setting with a mediator robot and a group of three humans. Our augmentation method creates synthetic sequences for underrepresented values in the AV space of the SEWA dataset, which is the most comprehensive dataset with continuous AV labels. Results show that using our DA method improves the accuracy and robustness of AV prediction in realtime applications. The accuracy of our models on the SEWA dataset is 0.793 for arousal and valence.

Abstract We propose a method for inferring the internal anisotropic volumetric texture of a given wood block from annotated photographs of its external surfaces. The global structure of the annual ring pattern is represented using a continuous spatial scalar field referred to as the growth time field (GTF). First, we train a generic neural model that can represent various GTFs using procedurally generated training data. Next, we fit the generic model to the GTF of a given wood block based on surface annotations. Finally, we convert the GTF to an annual ring field (ARF) revealing the layered pattern and apply neural style transfer to render orientation‐dependent small‐scale features and colors on a cut surface. We show rendered results of various physically cut real wood samples. Our method has physical and virtual applications such as cut‐preview before subtractive fabricating solid wood artifacts and simulating object breaking.

Abstract Acquiring the desired font for various design tasks can be challenging and requires professional typographic knowledge. While previous font retrieval or generation works have alleviated some of these difficulties, they often lack support for multiple languages and semantic attributes beyond the training data domains. To solve this problem, we present FontCLIP – a model that connects the semantic understanding of a large vision‐language model with typographical knowledge. We integrate typography‐specific knowledge into the comprehensive vision‐language knowledge of a pretrained CLIP model through a novel finetuning approach. We propose to use a compound descriptive prompt that encapsulates adaptively sampled attributes from a font attribute dataset focusing on Roman alphabet characters. FontCLIP's semantic typographic latent space demonstrates two unprecedented generalization abilities. First, FontCLIP generalizes to different languages including Chinese, Japanese, and Korean (CJK), capturing the typographical features of fonts across different languages, even though it was only finetuned using fonts of Roman characters. Second, FontCLIP can recognize the semantic attributes that are not presented in the training data. FontCLIP's dual‐modality and generalization abilities enable multilingual and cross‐lingual font retrieval and letter shape optimization, reducing the burden of obtaining desired fonts.

Animations can come to life when they are synchronized with relevant sounds. Yet, synchronizing animations to audio requires tedious key-framing or programming, which is difficult for novice creators. There are existing tools that support audio-driven live animation, but they focus primarily on speech and have little or no support for non-speech sounds. We present SoundToons, an exemplar-based authoring tool for interactive, audio-driven animation focusing on non-speech sounds. Our tool enables novice creators to author live animations to a wide variety of non-speech sounds, such as clapping and instrumental music. We support two types of audio interactions: (1) discrete interaction, which triggers animations when a discrete sound event is detected, and (2) continuous, which synchronizes an animation to continuous audio parameters. By employing an exemplar-based iterative authoring approach, we empower novice creators to design and quickly refine interactive animations. User evaluations demonstrate that novice users can author and perform live audio-driven animation intuitively. Moreover, compared to other input modalities such as trackpads or foot pedals, users preferred using audio as an intuitive way to drive animation.

Attention guidance is used to address dataset bias in deep learning, where the model relies on incorrect features to make decisions. Focusing on image classification tasks, we propose an efficient human-in-the-loop system to interactively direct the attention of classifiers to regions specified by users, thereby reducing the effect of co-occurrence bias and improving the transferability and interpretability of a deep neural network (DNN). Previous approaches for attention guidance require the preparation of pixel-level annotations and are not designed as interactive systems. We herein present a new interactive method that allows users to annotate images via simple clicks. Additionally, we identify a novel active learning strategy that can significantly reduce the number of annotations. We conduct both numerical evaluations and a user study to evaluate the proposed system using multiple datasets. Compared with the existing non-active-learning approach, which typically relies on considerable amounts of polygon-based segmentation masks to fine-tune or train the DNNs, our system can obtain fine-tuned networks on biased datasets in a more time- and cost-efficient manner and offers a more user-friendly experience. Our experimental results show that the proposed system is efficient, reasonable, and reliable. Our code is publicly available at https://github.com/ultratykis/Guiding-DNNs-Attention.

Abstract Procedural models enable the generation of a large amount of diverse shapes by varying the parameters of the model. However, writing a procedural model for replicating a collection of reference shapes is difficult, requiring much inspection of the original and replicated shapes during the development of the model. In this paper, we introduce a data‐guided method for aiding a programmer in creating a procedural model to replicate a collection of reference shapes. The user starts by writing an initial procedural model, and the system automatically predicts the model parameters for reference shapes, also grouping shapes by how well they are approximated by the current procedural model. The user can then update the procedural model based on the given feedback and iterate the process. Our system thus automates the tedious process of discovering the parameters that replicate reference shapes, allowing the programmer to focus on designing the high‐level rules that generate the shapes. We demonstrate through qualitative examples and a user study that our method is able to speed up the development time for creating procedural models of 2D and 3D man‐made shapes.

Abstract Interface icons are prevalent in various digital applications. Due to limited time and budgets, many designers rely on informal evaluation, which often results in poor usability icons. In this paper, we propose a unique human‐in‐the‐loop framework that allows our target users, that is novice and professional user interface (UI) designers, to improve the usability of interface icons efficiently. We formulate several usability criteria into a perceptual usability function and enable users to iteratively revise an icon set with an interactive design tool, EvIcon. We take a large‐scale pre‐trained joint image‐text embedding (CLIP) and fine‐tune it to embed icon visuals with icon tags in the same embedding space (IconCLIP). During the revision process, our design tool provides two types of instant perceptual usability feedback. First, we provide perceptual usability feedback modelled by deep learning models trained on IconCLIP embeddings and crowdsourced perceptual ratings. Second, we use the embedding space of IconCLIP to assist users in improving icons' visual distinguishability among icons within the user‐prepared icon set. To provide the perceptual prediction, we compiled IconCEPT10K, the first large‐scale dataset of perceptual usability ratings over 10,000 interface icons, by conducting a crowdsourcing study. We demonstrated that our framework could benefit UI designers' interface icon revision process with a wide range of professional experience. Moreover, the interface icons designed using our framework achieved better semantic distance and familiarity, verified by an additional online user study.

The exact shape of intracranial aneurysms is critical in medical diagnosis and surgical planning. While voxel-based deep learning frameworks have been proposed for this segmentation task, their performance remains limited. In this study, we offer a two-step surface-based deep learning pipeline that achieves significantly better results. Our proposed model takes a surface model of an entire set of principal brain arteries containing aneurysms as input and returns aneurysm surfaces as output. A user first generates a surface model by manually specifying multiple thresholds for time-of-flight magnetic resonance angiography images. The system then samples small surface fragments from the entire set of brain arteries and classifies the surface fragments according to whether aneurysms are present using a point-based deep learning network (PointNet++). Finally, the system applies surface segmentation (SO-Net) to surface fragments containing aneurysms. We conduct a direct comparison of the segmentation performance of our proposed surface-based framework and an existing voxel-based method by counting voxels: our framework achieves a much higher Dice similarity (72%) than the prior approach (46%).

Segmenting and determining the 3D bounding boxes of objects of interest in RGB videos is an important task for a variety of applications such as augmented reality, navigation, and robotics. Supervised machine learning techniques are commonly used for this, but they need training datasets: sets of images with associated 3D bounding boxes manually defined by human annotators using a labelling tool. However, precisely placing 3D bounding boxes can be difficult using conventional 3D manipulation tools on a 2D interface. To alleviate that burden, we propose a novel technique with which 3D bounding boxes can be created by simply drawing 2D bounding rectangles on multiple frames of a video sequence showing the object from different angles. The method uses reconstructed dense 3D point clouds from the video and computes tightly fitting 3D bounding boxes of desired objects selected by back-projecting the 2D rectangles. We show concrete application scenarios of our interface, including training dataset creation and editing 3D spaces and videos. An evaluation comparing our technique with a conventional 3D annotation tool shows that our method results in higher accuracy. We also confirm that the bounding boxes created with our interface have a lower variance, likely yielding more consistent labels and datasets.

Manual audio segmentation is a time-consuming process, especially when there is more than one sound playing simultaneously that needs to be segmented and annotated (e.g., target and background sounds). In conventional audio annotation interfaces, users need to repeatedly pause and replay the audio to complete an overlap segmentation task, which is very inefficient. In this paper, we propose "SyncLabeling," a synchronized audio segmentation interface for smartphones that allows users to segment and annotate two overlapping sounds in a single audio stream at a time using a game-like labeling interface on mobile devices. We conducted a user study to compare the proposed SyncLabeling interface with a conventional audio annotation interface on four types of audio segmentation tasks. The results showed that the proposed interface is much more efficient than the conventional interface (2.4× faster) under comparable annotation accuracy in most tasks. In addition, more than half of the participants enjoyed using the proposed SyncLabeling interface and showed willingness to use it.

We propose Inkjet 4D Print, a self-folding fabrication method of 3D origami tessellations by printing 2D patterns on both sides of a heat-shrinkable base sheet, using a commercialized inkjet ultraviolet (UV) printer. Compared to the previous folding-based 4D printing approach using fused deposition modeling (FDM) 3D printers [An et al. 2018], our method has merits in (1) more than 1200 times higher resolution in terms of the number of self-foldable facets, (2) 2.8 times faster printing speed, and (3) optional full-color decoration. This paper describes the material selection, the folding mechanism, the heating condition, and the printing patterns to self-fold both known and freeform tessellations. We also evaluated the self-folding resolution, the printing and transformation speed, and the shape accuracy of our method. Finally, we demonstrated applications enabled by our self-foldable tessellated objects.

Power flow simulation produces a colossal amount of complex, integrated, and diverse data. An analyst can get lost in those irregular data when understanding the undergoing of the system. Developing a well-performed analysis method and corresponding visualization techniques is essential. This study proposed NEGraf, a system prototype for transforming data into useful information to assist decision-making in power system monitoring. It can explain the system operating status to support the analyst in recognizing the specific phenomenon in collapse before the blackout, an abnormal period. NEGraf contains a node-edge graph module for data representation, a customized edge-weighted PageRank inference algorithm for data analysis, and a colored graph explanatory interface for information display. We ran a user study for a blackout recognition task. The results show that our interface can better explain intuitively dynamic features in collapse and improve accuracy and recall in predicting blackout than a traditional bar visualization interface.

In deep learning application development, programmers tend to try different architectures and hyper-parameters until satisfied with the model performance. Nevertheless, program crashes due to tensor shape mismatch prohibit programmers, especially novice programmers, from smoothly going back and forth between neural network (NN) architecture editing and experimentation. We propose to leverage live programming techniques in NN architecture editing with an always-on visualization. When the user edits the program, the visualization can synchronously display tensor states and provide a warning message by continuously executing the program to prevent program crashes during experimentation. We implement the live visualization and integrate it into an IDE called ODEN that seamlessly supports the “edit→experiment→edit→···” repetition. With ODEN, the user can construct the neural network with the live visualization and transits into experimentation to instantly train and test the NN architecture. An exploratory user study is conducted to evaluate the usability, the limitations, and the potential of live visualization in ODEN.

Abstract Knitting and weaving patterns can be visually represented as pixel art. With hand knitting and weaving, human error (shifting, duplicating, or skipping pixels) can occur during manual fabrication. It is too costly to change already‐fabricated pixels, so experts often adapt pixels that have not yet been fabricated to make the errors less visible. This paper proposes an automatic adaptation process to minimize visual artifacts. The system presents multiple adaptation possibilities to the user, who can choose the proposed adaptation or untie and re‐fabricate their work. In typical handicraft fabrication, the design is complete before the start of fabrication and remains fixed during fabrication. Our system keeps updating the design during fabrication to tolerate human errors in the process. We implemented the proposed algorithm in a system that visualizes the knitting pattern, cross‐stitching and bead weaving processes.

The cover image is based on the Research Article Interactive texture editing for garment line drawings by Tsukasa Fukusato et al., https://doi.org/10.1002/cav.2117

Abstract Adding two‐dimensional (2D) textures to garment line drawings (e.g., cartoon characters) remains challenging in the production pipeline of comics and illustrations since garment line drawings often have self‐occluded wrinkles. Although several techniques that can automatically deform and map 2D texture patterns to 2D line drawings have been proposed, their qualities are insufficient for representing 3D‐like realistic garment designs and manual editing of UV coordinates, which is labor‐intensive. In this article, we introduce an interactive tool to efficiently edit UV coordinates of 2D garment line drawings on the modeling panel with curve and point handles. Our algorithm is simple to integrate into existing image authoring tools. We conduct a user study with novice users and confirm that the proposed tool can effectively handle texture mapping envisioned by the users.

We present a procedural framework for modeling the annual ring pattern of solid wood with knots. Although wood texturing is a well-studied topic, there have been few previous attempts at modeling knots inside the wood texture. Our method takes the skeletal structure of a tree log as input and produces a three-dimensional scalar field representing the time of added growth, which defines the volumetric annual ring pattern. First, separate fields are computed around each strand of the skeleton, i.e., the stem and each knot. The strands are then merged into a single field using smooth minimums. We further suggest techniques for controlling the smooth minimum to adjust the balance of smoothness and reproduce the distortion effects observed around dead knots. Our method is implemented as a shader program running on a GPU with computation times of approximately 0.5 s per image and an input data size of 600 KB. We present rendered images of solid wood from pine and spruce as well as plywood and cross-laminated timber (CLT). Our results were evaluated by wood experts, who confirmed the plausibility of the rendered annual ring patterns. Link to code: https://github.com/marialarsson/procedural_knots.

We propose a method to integrate more interactivity into automatic hyperparameter optimization systems to leverage the user’s prior knowledge on parameter distribution. In our method, the user continuously observes automatic optimization’s progress and dynamically specifies where to search in the parameter space. We present a prototype implementation of an interactive dashboard for an optimizer to show our method’s feasibility. The interactive dashboard’s main feature is “paintable timeline” where the user can not only observe the past parameter values tested as in standard timeline but also specify the range of future parameters to be tested with simple painting operations. We show three examples where user intervention might improve the performance of automatic optimizations. We run a user study with experts and the results show that, with prior knowledge about parameter distribution of the target problem, interactive optimization can reach better results compared to fully automatic optimization.

Non-expert annotators (who lack sufficient domain knowledge) are often recruited for manual image labeling tasks owing to the lack of expert annotators. In such a case, label quality may be relatively low. We propose leveraging the spatial layout for improving label quality in non-expert image annotation. In the proposed system, an annotator first spatially lays out the incoming images and labels them on an open space, placing related items together. This serves as a working space (spatial organization) for tentative labeling. During the process, the annotator observes and organizes the similarities and differences between the items. Finally, the annotator provides definitive labels to the images based on the results of the spatial layout. We ran a user study comparing the proposed method and a traditional non-spatial layout in an image labeling task. The results demonstrated that annotators can complete the labeling tasks more accurately using the spatial layout interface than the non-spatial layout interface.

How to handle gender with machine learning is a controversial topic. A growing critical body of research brought attention to the numerous issues transgender communities face with the adoption of current automatic gender recognition (AGR) systems. In contrast, we explore how such technologies could potentially be appropriated to support transgender practices and needs, especially in non-Western contexts like Japan. We designed a virtual makeup probe to assist transgender individuals with passing, that is to be perceived as the gender they identify as. To understand how such an application might support expressing transgender individuals gender identity or not, we interviewed 15 individuals in Tokyo and found that in the right context and under strict conditions, AGR based systems could assist transgender passing.

Reinforcement learning techniques successfully generate convincing agent behaviors, but it is still difficult to tailor the behavior to align with a user's specific preferences. What is missing is a communication method for the system to explain the behavior and for the user to repair it. In this paper, we present a novel interaction method that uses interactive explanations using templates of natural language as a communication method. The main advantage of this interaction method is that it enables a two-way communication channel between users and the agent; the bot can explain its thinking procedure to the users, and the users can communicate their behavior preferences to the bot using the same interactive explanations. In this manner, the thinking procedure of the bot is transparent, and users can provide corrections to the bot that include a suggested action to take, a goal to achieve, and the reasons behind these decisions. We tested our proposed method in a clone of the video game named Super Mario Bros., and the results demonstrate that our interactive explanation approach is effective at diagnosing and repairing bot behaviors.

Programming-by-example (PBE), can be a powerful tool to reduce manual work in repetitive data transformation tasks. However, few examples often leave ambiguity and may cause undesirable data transformation by the system. This ambiguity can be resolved by allowing the user to directly edit the synthesized programs; however, this is difficult for non-programmers. Here, we present a novel approach: data-centric disambiguation for data transformation, where users resolve the ambiguity in data transformation by examining and modifying the output rather than the program. The key idea is to focus on the given set of data the user wants to transform instead of pursuing the synthesized program’s generality or completeness. Our system provides visualization and interaction methods that allow users to efficiently examine and fix the transformed outputs, which is much simpler than understanding and modifying the program itself. The user study suggests that our system can successfully help non-programmers to more easily and efficiently process data.

Recent content creation systems allow users to generate various high-quality content (e.g., images, 3D models, and melodies) by just specifying a parameter set (e.g., a latent vector of a deep generative model). The task here is to search for an appropriate parameter set that produces the desired content. To facilitate this task execution, researchers have investigated user-in-the-loop optimization, where the system samples candidate solutions, asks the user to provide preferential feedback on them, and iterates this procedure until finding the desired solution. In this work, we investigate a novel approach to enhance this interactive process: allowing users to control the sampling behavior. More specifically, we allow users to adjust the balance between exploration (i.e., favoring diverse samples) and exploitation (i.e., favoring focused samples) in each iteration. To evaluate how this approach affects the user experience and optimization behavior, we implement it into a melody composition system that combines a deep generative model with Bayesian optimization. Our experiments suggest that this approach could improve the user’s engagement and optimization performance.

Virtual try-on is a promising application of computer graphics and human computer interaction that can have a profound real-world impact especially during this pandemic. Existing image-based works try to synthesize a try-on image from a single image of a target garment, but it inherently limits the ability to react to possible interactions. It is difficult to reproduce the change of wrinkles caused by pose and body size change, as well as pulling and stretching of the garment by hand. In this paper, we propose an alternative per garment capture and synthesis workflow to handle such rich interactions by training the model with many systematically captured images. Our workflow is composed of two parts: garment capturing and clothed person image synthesis. We designed an actuated mannequin and an efficient capturing process that collects the detailed deformations of the target garments under diverse body sizes and poses. Furthermore, we proposed to use a custom-designed measurement garment, and we captured paired images of the measurement garment and the target garments. We then learn a mapping between the measurement garment and the target garments using deep image-to-image translation. The customer can then try on the target garments interactively during online shopping. The proposed workflow requires certain manual labor, but we believe that the cost is acceptable given that the retailers are already paying significant costs for hiring professional photographers and models, stylists, and editors to take photographs for promotion. Our method can remove the need of hiring these costly professionals. We evaluated the effectiveness of the proposed system with ablation studies and quality comparison with previous virtual try-on methods. We perform a user study to show our promising virtual try-on performances. Moreover, we also demonstrate that we use our method for changing virtual costumes in video conferences. Finally, we provide the collected dataset as the cloth dataset parameterized by various viewing angles, body poses, and sizes.

Writing high-performance image processing code is challenging and labor-intensive. To address this, we propose a programming support method called “guided optimization.” Guided optimization provides programmers a set of valid optimization options and interactive feedback about their current choices, which enables them to comprehend and efficiently optimize image processing code without the time-consuming trial-and-error process of traditional text editors. We implemented a proof-of-concept system, Roly-poly, which integrates guided optimization, program visualization, and schedule cost estimation to support the comprehension and development of efficient image processing code. We conducted a user study with novice Halide programmers and confirmed that Roly-poly and its guided optimization was informative, increased productivity, and resulted in higher-performing schedules in less time.

Abstract Generative image modeling techniques such as GAN demonstrate highly convincing image generation result. However, user interaction is often necessary to obtain desired results. Existing attempts add interactivity but require either tailored architectures or extra data. We present a human‐in‐the‐optimization method that allows users to directly explore and search the latent vector space of generative image modelling. Our system provides multiple candidates by sampling the latent vector space, and the user selects the best blending weights within the subspace using multiple sliders. In addition, the user can express their intention through image editing tools. The system samples latent vectors based on inputs and presents new candidates to the user iteratively. An advantage of our formulation is that one can apply our method to arbitrary pre‐trained model without developing specialized architecture or data. We demonstrate our method with various generative image modelling applications, and show superior performance in a comparative user study with prior art iGAN [ZKSE16].

Abstract This paper introduces a generative model for 3D surfaces based on a representation of shapes with mean curvature and metric, which are invariant under rigid transformation. Hence, compared with existing 3D machine learning frameworks, our model substantially reduces the influence of translation and rotation. In addition, the local structure of shapes will be more precisely captured, since the curvature is explicitly encoded in our model. Specifically, every surface is first conformally mapped to a canonical domain, such as a unit disk or a unit sphere. Then, it is represented by two functions: the mean curvature half‐density and the vertex density, over this canonical domain. Assuming that input shapes follow a certain distribution in a latent space, we use the variational autoencoder to learn the latent space representation. After the learning, we can generate variations of shapes by randomly sampling the distribution in the latent space. Surfaces with triangular meshes can be reconstructed from the generated data by applying isotropic remeshing and spin transformation, which is given by Dirac equation. We demonstrate the effectiveness of our model on datasets of man‐made and biological shapes and compare the results with other methods.

Specifying precise relationships among graphic elements is often a time-consuming process with traditional alignment tools. Automatic beautification of roughly designed layouts can provide a more efficient solution but often lead to undesired results due to ambiguity problems. To facilitate ambiguity resolution in layout beautification, we present a novel user interface for visualizing and editing inferred relationships through an automatic global layout beautification process. First, our interface provides a preview of the beautified layout with inferred constraints without directly modifying an input layout. In this way, the user can easily keep refining beautification results by interactively repositioning and/or resizing elements in the input layout. Second, we present a gestural interface for editing automatically inferred constraints by directly interacting with the visualized constraints via simple gestures. Our technique is applicable to both 2D and 3D global layout beautification, supported by efficient system implementation that provides instant user feedback. Our user study validates that our tool is capable of creating, editing, and refining layouts of graphic elements, and is significantly faster than the standard snap-dragging or command-based alignment tools for both 2D and 3D layout tasks.

Interactive reinforcement learning (RL) has been successfully used in various applications in different fields, which has also motivated HCI researchers to contribute in this area. In this paper, we survey interactive RL to empower human-computer interaction (HCI) researchers with the technical background in RL needed to design new interaction techniques and propose new applications. We elucidate the roles played by HCI researchers in interactive RL, identifying ideas and promising research directions. Furthermore, we propose generic design principles that will provide researchers with a guide to effectively implement interactive RL applications.

The development of a statistical modeling program requires example data to observe and verify the behavior of the program. Such example data are either taken from an existing dataset or synthesized using commands. Programmers may want to directly design an arbitrary dataset or modify it interactively, but it is difficult to do so in current development environments. We therefore propose combining a code editor with an interactive scatter plot editor to efficiently understand the behavior of statistical modeling algorithms. The user interactively creates and modifies the dataset on the scatter plot editor, while the system continuously executes the code in the editor, taking the data as input, and shows the result in the editor. This paper presents the design rationale behind the system and introduces several usage scenarios.

Medicine is an important application area for deep learning models. Research in this field is a combination of medical expertise and data science knowledge. In this paper, instead of 2D medical images, we introduce an open-access 3D intracranial aneurysm dataset, IntrA, that makes the application of points-based and mesh-based classification and segmentation models available. Our dataset can be used to diagnose intracranial aneurysms and to extract the neck for a clipping operation in medicine and other areas of deep learning, such as normal estimation and surface reconstruction. We provide a large-scale benchmark of classification and part segmentation by testing state-of-the-art networks. We also discuss the performance of each method and demonstrate the challenges of our dataset. The published dataset can be accessed here: https://github.com/intra2d2019/IntrA.

Game designers take into account the wide range of play-styles and skill levels of players to create enjoyable experiences. One important step in the game design process involves playtests with professional testers; this process is time-consuming and expensive. Hence, there exist several methods to create synthetic testers to test a game automatically. However, one shortcoming is the lack of realistic-playing with different play-styles and skill levels. In this paper, we propose a game level authoring tool that incorporates synthetic testers, which enable the control of play-styles and skill levels. Furthermore, we utilize visualization techniques to help assess the difficulty level of each part of the stage. Our user studies confirmed that our tool was effective for designing game stages appropriate for a particular type of player.

We present Tsugite - an interactive system for designing and fabricating wood joints for frame structures. To design and manually craft such joints is difficult and time consuming. Our system facilitates the creation of custom joints by a modeling interface combined with computer numerical control (CNC) fabrication. The design space is a 3D grid of voxels that enables efficient geometrical analysis and combinatorial search. The interface has two modes: manual editing and gallery. In the manual editing mode, the user edits a joint while receiving real-time graphical feedback and suggestions provided based on performance metrics including slidability, fabricability, and durability with regard to the direction of fiber. In the gallery mode, the user views and selects feasible joints that have been pre-calculated. When a joint design is finalized, it can be manufactured with a 3-axis CNC milling machine using a specialized path planning algorithm that ensures joint assemblability by corner rounding. This system was evaluated via a user study and by designing and fabricating joint samples and functional furniture.

Generative models based on deep neural networks often have a high-dimensional latent space, ranging sometimes to a few hundred dimensions or even higher, which typically makes them hard for a user to explore directly. We propose differential subspace search to allow efficient iterative user exploration in such a space, without relying on domain- or data-specific assumptions. We develop a general framework to extract low-dimensional subspaces based on a local differential analysis of the generative model, such that a small change in such a subspace would provide enough change in the resulting data. We do so by applying singular value decomposition to the Jacobian of the generative model and forming a subspace with the desired dimensionality spanned by a given number of singular vectors stochastically selected on the basis of their singular values, to maintain ergodicity. We use our framework to present 1D subspaces to the user via a 1D slider interface. Starting from an initial location, the user finds a new candidate in the presented 1D subspace, which is in turn updated at the new candidate location. This process is repeated until no further improvement can be made. Numerical simulations show that our method can better optimize synthetic black-box objective functions than the alternatives that we tested. Furthermore, we conducted a user study using complex generative models and the results show that our method enables more efficient exploration of high-dimensional latent spaces than the alternatives.

Repetitive patterns are ubiquitous in natural and human-made objects, and can be created with a variety of tools and methods. Manual authoring provides unmatched degree of freedom and control, but can require significant artistic expertise and manual labor. Computational methods can automate parts of the manual creation process, but are mainly tailored for discrete pixels or elements instead of more general continuous structures. We propose an example-based method to synthesize continuous curve patterns from exemplars. Our main idea is to extend prior sample-based discrete element synthesis methods to consider not only sample positions (geometry) but also their connections (topology). Since continuous structures can exhibit higher complexity than discrete elements, we also propose robust, hierarchical synthesis to enhance output quality. Our algorithm can generate a variety of continuous curve patterns fully automatically. For further quality improvement and customization, we also present an autocomplete user interface to facilitate interactive creation and iterative editing. We evaluate our methods and interface via different patterns, ablation studies, and comparisons with alternative methods.

We introduce dynamically warping grids for adaptive liquid simulation. Our primary contributions are a strategy for dynamically deforming regular grids over the course of a simulation and a method for efficiently utilizing these deforming grids for liquid simulation. Prior work has shown that unstructured grids are very effective for adaptive fluid simulations. However, unstructured grids often lead to complicated implementations and a poor cache hit rate due to inconsistent memory access. Regular grids, on the other hand, provide a fast, fixed memory access pattern and straightforward implementation. Our method combines the advantages of both: we leverage the simplicity of regular grids while still achieving practical and controllable spatial adaptivity. We demonstrate that our method enables adaptive simulations that are fast, flexible, and robust to null-space issues. At the same time, our method is simple to implement and takes advantage of existing highly-tuned algorithms.

While tree trunks are standardized as lumber, branches are typically chipped or burned. This paper proposes a workflow to upcycle such mundane and diverse natural material to architectural elements. Introducing an online design interface, we let users participate in the design and fabrication workflow from collecting branches to CNC milling. The branches are first scanned, and then key geometrical features are extracted and uploaded to the online game "BranchConnect". This application lets multiple non-expert users create 2D-layouts. At the point of intersection between two branches, the geometry of a lap joint and its cutting path are calculated on-the-fly. A CNC router mills out the joints accordingly, and the branches are assembled manually. Through this workflow, users go back-and-forth between physical and digital representations of tree branches. The process was validated with two case studies.

Vascular network reconstruction is an essential aspect of the daily practice of medical doctors working with vascular systems. Accurately representing vascular networks, not only graphically but also in a way that encompasses their structure, can be used to run simulations, plan medical procedures or identify real-life diseases, for example. A vascular network is thus reconstructed from a 3D medical image sequence via segmentation and skeletonization. Many automatic algorithms exist to do so but tend to fail for specific corner cases. On the other hand, manual methods exist as well but are tedious to use and require a lot of time. In this paper, we introduce an interactive vascular network reconstruction system called INVANER that relies on a graph-like representation of the network's structure. A general skeleton is obtained with an automatic method and medical practitioners are allowed to manually repair the local defects where this method fails. Our system uses graph-related tools with local effects and introduces two novel tools, dedicated to solving two common problems arising when automatically extracting the centerlines of vascular structures: so-called "Kissing Vessels" and a type of phenomenon we call "Dotted Vessels."

Manual image labeling (selecting an appropriate “category” for an image) is very tedious and time consuming especially when selecting labels from a large number of categories. In this study, we propose a hierarchical assignment of labeling tasks where the labelers recursively classify images in a category group into sub category groups, working on a single level at a time. This significantly makes each labeler's task easier, reducing the number of choices from 1,000 to 27 on average. In the user study, we compared our hierarchical assignment to a normal (non-hierarchical) assignment for a labeling task. The results show that the hierarchical assignment requires less total time to complete the labeling task. In addition, the learning effect in the labeling process is more profound in the hierarchical assignment.

Abstract A set diagram represents the membership relation among data elements. It is often visualized as secondary information on top of primary information, such as the spatial positions of elements on maps and charts. Visualizing the temporal evolution of such set diagrams as well as their primary features is quite important; however, conventional approaches have only focused on the temporal behavior of the primary features and do not provide an effective means to highlight notable transitions within the set relationships. This paper presents an approach for generating a stepwise animation between set diagrams by decomposing the entire transition into atomic changes associated with individual data elements. The key idea behind our approach is to optimize the ordering of the atomic changes such that the synthesized animation minimizes unwanted set occlusions by considering their depth ordering and reduces the gaze shift between two consecutive stepwise changes. Experimental results and a user study demonstrate that the proposed approach effectively facilitates the visual identification of the detailed transitions inherent in dynamic set diagrams.

The cover image is based on the Special Issue Paper Sketch2VF: Sketch-based Flow Design with Conditional Generative Adversarial Network by Xie Haoran et al., https://doi.org/10.1002/cav.1889.

Abstract We present an interactive user interface to support sketch‐based fluid design with a perceptual understanding of human sketches. In particular, the proposed system generates a 2D fluid animation from hand‐drawn sketches. The proposed system utilizes a conditional generative adversarial network model to generate stationary velocity fields from a sketch input. The network model is trained with hand‐drawn strokes and corresponding 2D velocity fields. On the basis of the generated velocity field, the system calculates fluid dynamics using a semi‐Lagrangian method. We ran a user study of the proposed system and confirmed that the proposed interface is effective for a 2D fluid design and that the system achieves good results based on user input.

We propose a fast, interactive real-time 3DCG deformable simulation prototype for preoperative virtual practice of brain aneurysm clipping surgery, controlled by Position Based Dynamics (PBD). Blood vessels are reconstructed from their central lines, connected to the brain by automatically generated thin threads "virtual trabeculae", and colored by automatically estimated their dominant region.

Abstract The Fluid Implicit Particle method (FLIP) reduces numerical dissipation by combining particles with grids. To improve performance, the subsequent narrow band FLIP method (NB‐FLIP) uses a FLIP‐based fluid simulation only near the liquid surface and a traditional grid‐based fluid simulation away from the surface. This spatially‐limited FLIP simulation significantly reduces the number of particles and alleviates a computational bottleneck. In this paper, we extend the NB‐FLIP idea even further, by allowing a simulation to transition between a FLIP‐like fluid simulation and a grid‐based simulation in arbitrary locations, not just near the surface. This approach leads to even more savings in memory and computation, because we can concentrate the particles only in areas where they are needed. More importantly, this new method allows us to seamlessly transition to smooth implicit surface geometry wherever the particle‐based simulation is unnecessary. Consequently, our method leads to a practical algorithm for avoiding the noisy surface artifacts associated with particle‐based liquid simulations, while simultaneously maintaining the benefits of a FLIP simulation in regions of dynamic motion.

We introduce a new advection scheme for fluid animation. Our main contribution is the use of long-term temporal changes in pressure to extend the commonly used semi-Lagrangian scheme further back along the time axis. Our algorithm starts by tracing sample points along a trajectory following the velocity field backwards in time for many steps. During this backtracing process, the pressure gradient along the path is integrated to correct the velocity of the current time step. We show that our method effectively suppresses numerical diffusion, retains small-scale vorticity, and provides better long-term kinetic energy preservation.

Molding is a popular mass production method, in which the initial expenses for the mold are offset by the low per-unit production cost. However, the physical fabrication constraints of the molding technique commonly restrict the shape of moldable objects. For a complex shape, a decomposition of the object into moldable parts is a common strategy to address these constraints, with plastic model kits being a popular and illustrative example. However, conducting such a decomposition requires considerable expertise, and it depends on the technical aspects of the fabrication technique, as well as aesthetic considerations. We present an interactive technique to create such decompositions for two-piece molding, in which each part of the object is cast between two rigid mold pieces. Given the surface description of an object, we decompose its thin-shell equivalent into moldable parts by first performing a coarse decomposition and then utilizing an active contour model for the boundaries between individual parts. Formulated as an optimization problem, the movement of the contours is guided by an energy reflecting fabrication constraints to ensure the moldability of each part. Simultaneously the user is provided with editing capabilities to enforce aesthetic guidelines. Our interactive interface provides control of the contour positions by allowing, for example, the alignment of part boundaries with object features. Our technique enables a novel workflow, as it empowers novice users to explore the design space, and it generates fabrication-ready two-piece molds that can be used either for casting or industrial injection molding of free-form objects.

In this article, we introduce an efficient and versatile numerical aerodynamics model for general three-dimensional geometry shapes in potential flow. The proposed model has low computational cost and achieves an accuracy of moderate fidelity for the aerodynamic loads for a given glider shape. In the geometry preprocessing steps of our model, lifting-wing surfaces are recognized, and wake panels are generated automatically along the trailing edges. The proposed aerodynamics model improves the potential theory-based panel method. Furthermore, a new quadratic expression for aerodynamic forces and moments is proposed. It consists of geometry-dependent aerodynamic coefficient matrices and has a continuous representation for the drag/lift-force coefficients. Our model enables natural and real-time aerodynamics simulations combined with general rigid-body simulators for interactive animation. We also present a design system for original gliders. It uses an assembly-based modeling interface and achieves interactive feedback by leveraging the partwise precomputation enabled by our method. We illustrate that one can easily design various flyable gliders using our system.

Self-driving technologies have been increasingly developed and tested in recent years (e.g., Volvo's and Google's self-driving cars). However, only a limited number of investigations have so far been conducted into communication between self-driving cars and pedestrians. For example, when a pedestrian is about to cross a street, that pedestrian needs to know the intension of the approaching self-driving car. In the present study, we designed a novel interface known as "Eyes on a Car" to address this problem. We added eyes onto a car so as to establish eye contact communication between that car and pedestrians. The car looks at the pedestrian in order to indicate its intention to stop. This novel interface design was evaluated via a virtual reality (VR) simulated environment featuring a street-crossing scenario. The evaluation results show that pedestrians can make the correct street-crossing decision more quickly if the approaching car has the novel interface "eyes" than in the case of normal cars. In addition, the results show that pedestrians feel safer with regard to crossing a street if the approaching car has eyes and if the eyes look at them.

Recent developments in 3D scanning make it easy to obtain 3D geometry of a human and produce a 3D replica using a 3D printer. However, the setting for full body scanning enforces low-quality of the facial area due to the lack of image resolution. To solve this issue, artists retouch the raw scanned full body model with a 3D sculpting tool, but it takes a lot of time and effort. In this paper, we present a system to automate the retouching process by transferring a retouched result by an expert to an arbitrary target face model. Our method first identifies facial features, such as the eyes and nose of the 3D models by exploiting 2D face detection. For each facial part, the method establishes dense correspondences between the exemplar and the target using the detection result, and then transfers the geometry of the exemplar to the target by coating transfer. We show that our method improves not only the geometry of model but also the 3D printed replica.

Experimentation plays an essential role in exploratory programming, and programmers apply version control operations when switching the part of the source code back to the past state during experimentation. However, these operations, which we refer to as micro-versioning, are not well supported in current programming environments. We first examined previous studies to clarify the requirements for a micro-versioning tool. We then developed a micro-versioning tool that displays visual cues representing possible micro-versioning operations in a textual code editor. Our tool includes a history model that generates meaningful candidates by combining a regional undo model and tree-structured undo model. The history model uses code executions as a delimiter to segment text edit operations into meaning groups. A user study involving programmers indicated that our tool satisfies the above-mentioned requirements and that it is useful for exploratory programming.

We propose a drawing method to create large-scale pictures in public space. We use a property of anisotropic reflection to show images on the grass field. We created a prototype of roller type device which can control the angle of grass. We observed that our system entertains people in public exhibition.

Sound information in videos plays an important role in shaping the user experience. When sound is not accessible in videos, text captions can provide sound information. However, conventional text captions are not very expressive for nonverbal sounds because they are designed to visualize speech sounds. Here, we present a framework to automatically transform nonverbal video sounds into animated sound words and position them near the sound source objects in the video for visualization. This provides natural visual representation of nonverbal sounds with rich information about the sound category and dynamics. To evaluate how the animated sound words generated by our framework affect the user experience, we implemented an experimental system and conducted a user study involving over 300 participants from an online crowdsourcing service. The results of the user study show that the animated sound words can effectively and naturally visualize the dynamics of sound while clarifying the position of the sound source as well as contribute to making video-watching more enjoyable and increasing the visual impact of videos.

Parameter tweaking is a common task in various design scenarios. For example, in color enhancement of photographs, designers tweak multiple parameters such as "brightness" and "contrast" to obtain the best visual impression. Adjusting one parameter is easy; however, if there are multiple correlated parameters, the task becomes much more complex, requiring many trials and a large cognitive load. To address this problem, we present a novel extension of Bayesian optimization techniques, where the system decomposes the entire parameter tweaking task into a sequence of one-dimensional line search queries that are easy for human to perform by manipulating a single slider. In addition, we present a novel concept called crowd-powered visual design optimizer , which queries crowd workers, and provide a working implementation of this concept. Our single-slider manipulation microtask design for crowdsourcing accelerates the convergence of the optimization relative to existing comparison-based microtask designs. We applied our framework to two different design domains: photo color enhancement and material BRDF design, and thereby showed its applicability to various design domains.

To realize 3D spatial sound rendering with a two-channel headphone, one needs head-related transfer functions (HRTFs) tailored for a specific user. However, measurement of HRTFs requires a tedious and expensive procedure. To address this, we propose a fully perceptual-based HRTF fitting method for individual users using machine learning techniques. The user only needs to answer pairwise comparisons of test signals presented by the system during calibration. This reduces the efforts necessary for the user to obtain individualized HRTFs. Technically, we present a novel adaptive variational AutoEncoder with a convolutional neural network. In the training, this AutoEncoder analyzes publicly available HRTFs dataset and identifies factors that depend on the individuality of users in a nonlinear space. In calibration, the AutoEncoder generates high-quality HRTFs fitted to a specific user by blending the factors. We validate the feasibilities of our method through several quantitative experiments and a user study.

Color enhancement is a very important aspect of photo editing. Even when photographers have tens of or hundreds of photographs, they must enhance each photo one by one by manually tweaking sliders in software such as brightness and contrast, because automatic color enhancement is not always satisfactory for them. To support this repetitive manual task, we present self-reinforcing color enhancement, where the system implicitly and progressively learns the user's preferences by training on their photo editing history. The more photos the user enhances, the more effectively the system supports the user. We present a working prototype system called SelPh, and then describe the algorithms used to perform the self-reinforcement. We conduct a user study to investigate how photographers would use a self-reinforcing system to enhance a collection of photos. The results indicate that the participants were satisfied with the proposed system and strongly agreed that the self-reinforcing approach is preferable to the traditional workflow.

We present a method of image sequence interpolation, which can generate a sequence of continuous intermediate frames between two input images. This method is based on a path framework that describes the motion information in the images. A path which starts from one input image, and ends at another input image is constructed for each pixel in the images. The main contribution of this paper is that we take the feature points into consideration. By calculating the position deviation out of the feature points, information and guidance can be given to the process of path optimization, making the interpolation result more plausible and natural. We also increase the conditions and restrictions in the optimization procedure, hence the time and memory cost can be effectively decreased.

I will introduce our research project (design interface project) aiming at the development of various design tools for end-users. We live in a mass-production society today and everyone buy and use same things all over the world. This might be economically efficient, but not necessarily ideal for individual persons. We envision that computer tools that help people to design things by themselves can enrich their lives. To that end, we develop innovative interaction techniques for end users to (1) create rich graphics such as three-dimensional models and animations by simple sketching (2) design their own real-world, everyday objects such as clothing and furniture with real-time physical simulation integrated in a simple geometry editor, and (3) design the behavior of their personal robots and give instructions to them to satisfy their particular needs.

An interactive method for segmentation and isosurface extraction of medical volume data is proposed. In conventional methods, users decompose a volume into multiple regions iteratively, segment each region using a threshold, and then manually clean the segmentation result by removing clutter in each region. However, this is tedious and requires many mouse operations from different camera views. We propose an alternative approach whereby the user simply applies painting operations to the volume using tools commonly seen in painting systems, such as flood fill and brushes. This significantly reduces the number of mouse and camera control operations. Our technical contribution is in the introduction of the threshold field, which assigns spatially-varying threshold values to individual voxels. This generalizes discrete decomposition of a volume into regions and segmentation using a constant threshold in each region, thereby offering a much more flexible and efficient workflow. This paper describes the details of the user interaction and its implementation. Furthermore, the results of a user study are discussed. The results indicate that the proposed method can be a few times faster than a conventional method.

Abstract We present an interactive design system for designing free‐formed bamboo‐copters, where novices can easily design free‐formed, even asymmetric bamboo‐copters that successfully fly. The designed bamboo‐copters can be fabricated using digital fabrication equipment, such as a laser cutter. Our system provides two useful functions for facilitating this design activity. First, it visualizes a simulated flight trajectory of the current bamboo‐copter design, which is updated in real time during the user's editing. Second, it provides an optimization function that automatically tweaks the current bamboo‐copter design such that the spin quality—how stably it spins—and the flight quality—how high and long it flies—are enhanced. To enable these functions, we present non‐trivial extensions over existing techniques for designing free‐formed model airplanes [ UKSI14 ], including a wing discretization method tailored to free‐formed bamboo‐copters and an optimization scheme for achieving stable bamboo‐copters considering both spin and flight qualities.

Iris folding is an art-form consisting of layered strips of paper, forming a spiral pattern behind an aperture, which can be used to make cards and gift tags. This paper describes an interactive computational tool to assist in the design and construction of original iris folding patterns. The design of iris folding patterns is formulated as the calculation of a circumscribed polygonal sequence around a seed polygon. While it is possible to compute the positions of vertices analytically for a regular polygon, it is not straightforward to do so for irregular polygons. We give a numerical method for irregular polygons, which can be applied to arbitrary convex seed polygons. The user can quickly experiment with various patterns using the system prior to constructing the art-form.

We propose subtle foot-based gestures named foot plantar-based (FPB) gestures that are used with sock-placed pressure sensors. In this system, the user can control a computing device by changing his or her foot plantar distributions, e.g., pressing the floor with his/her toe. Because such foot movement is subtle, it is suitable for use especially in a public space such as a crowded train. In this study, we first conduct a guessability study to design a user-defined gesture set for interaction with a computing device. Then, we implement a gesture recognizer with a machine learning technique. To avoid unexpected gesture activations, we also collect foot plantar pressure patterns made during daily activities such as walking, as negative training data. Additionally, we evaluate the unobservability of FPB gestures by using crowdsourcing. Finally, we conclude with several applications to further illustrate the utility of FPB gestures.

We present a video annotation system called ``AnnoTone', which can embed various contextual information describing a scene, such as geographical location. Then the system allows the user to edit the video using this contextual information, enabling one to, for example, overlay with map or graphical annotations. AnnoTone converts annotation data into high-frequency audio signals (which are inaudible to the human ear), and then transmits them from a smartphone speaker placed near a video camera. This scheme makes it possible to add annotations using standard video cameras with no requirements for specific equipment other than a smartphone. We designed the audio watermarking protocol using dual-tone multi-frequency signaling, and developed a general-purpose annotation framework including an annotation generator and extractor. We conducted a series of performance tests to understand the reliability and the quality of the watermarking method. We then created several examples of video-editing applications using annotations to demonstrate the usefulness of Annotone, including an After Effects plug-in.

Many graphic applications rely on command-based arrangement tools to achieve precise layouts. Traditional tools are designed to operate on a single group of elements that are distributed consistently with the arrangement axis implied by a command. This often demands a process with repeated element selections and arrangement commands to achieve 2D layouts involving multiple rows and/or columns of well aligned and/or distributed elements. Our work aims to reduce the numbers of selection operation and command invocation, since such reductions are particularly beneficial to professional designers who design lots of layouts. Our key idea is that an issued arrangement command is in fact very informative, instructing how to automatically decompose a 2D layout into multiple 1D groups, each of which is compatible with the command. We present a parameter-free, command-driven grouping approach so that users can easily predict our grouping results. We also design a simple user interface with pushpins to enable explicit control of grouping and arrangement. Our user study confirms the intuitiveness of our technique and its performance improvement over traditional command-based arrangement tools.

I will introduce our research project (design interface project) aiming at the development of various design tools for end-users. We live in a mass-production society today and everyone buy and use same things all over the world. This is cheap, but not necessarily ideal for individual persons. We envision that computer tools that help people to design things by themselves can enrich their lives. To that end, we develop innovative interaction techniques for end users to (1) create rich graphics such as three-dimensional models and animations by simple sketching (2) design their own real-world, everyday objects such as clothing and furniture with real-time physical simulation integrated in a simple geometry editor, and (3) design the behavior of their personal robots and give instructions to them to satisfy their particular needs.

Text captions are important means to provide sound information in videos when the sound is not accessible. However, conventional text captions are far less expressive for non-verbal sounds since they are designed to visualize speech sound. To address this problem, we propose a method for automatically transforming non-verbal video sounds to animated sound words, and positioning them near the sound source objects in the video for visualization. This provides natural visual representation of non-verbal sounds with rich information about the sound category and dynamics. We conducted a user study with over 300 participants using an online crowdsourcing service. The results showed that animated sound words could not only effectively and naturally visualize the dynamics of sound while clarify the position of the sound source, but also contribute to making video watching more enjoyable and increasing the visual impact of the video.

Abstract We presentfacetons, geometric modeling primitives designed for building architectural models especially effective for a virtual environment where six degrees of freedom input devices are available. Afacetonis an oriented point floating in the air and defines a plane of infinite extent passing through the point. The polygonal mesh model is constructed by taking the intersection of the planes associated with thefacetons. With the simple interaction offaceton, users can easily create 3D architecture models. Thefacetonprimitive and its interaction reduce the overhead associated with standard polygonal mesh modeling, where users have to manually specify vertexes and edges which could be far away. Thefacetonrepresentation is inspired by the research on boundary representations (B‐rep) and constructive solid geometry, but it is driven by a novel adaptive bounding algorithm and is specifically designed for 3D modeling activities in an immersive virtual environment. We describe the modeling method and our current implementation. The implementation is still experimental but shows potential as a viable alternative to traditional modeling methods. Copyright © 2014 John Wiley & Sons, Ltd.

Recent digital fabrication tools have opened up accessibility to personalized rapid prototyping; however, such tools are limited to product-scale objects. The materials currently available for use in 3D printing are too fine for large-scale objects, and CNC gantry sizes limit the scope of printable objects. In this paper, we propose a new method for printing architecture-scale objects. Our proposal includes three developments: (i) a construction material consisting of chopsticks and glue, (ii) a handheld chopstick dispenser, and (iii) a printing guidance system that uses projection mapping. The proposed chopstickglue material is cost effective, environmentally sustainable, and can be printed more quickly than conventional materials. The developed handheld dispenser enables consistent feeding of the chopstickglue material composite. The printing guidance system --- consisting of a depth camera and a projector --- evaluates a given shape in real time and indicates where humans should deposit chopsticks by projecting a simple color code onto the form under construction. Given the mechanical specifications of the stickglue composite, an experimental pavilion was designed as a case study of the proposed method and built without scaffoldings and formworks. The case study also revealed several fundamental limitations, such as the projector does not work in daylight, which requires future investigations.

This paper presents a method that employs parametric surfaces as surface geometry representations at any stage of a computational process to compute self-supporting surfaces. This approach can be differentiated from existing relevant methods because such methods represent surfaces by a triangulated mesh surface or a network consisting of lines. The proposed method is based on the theory of Airy stress functions. Although some existing methods are also based on this theory, they apply its discrete version to discrete geometries. The proposed method simultaneously applies the theory to parametric surfaces directly and the discrete theory to the edges of parametric patches. The discontinuous boundary between continuous patches naturally corresponds to ribs seen in traditional vault masonry buildings. We use nonuniform rational B-spline surfaces in this study; however, the basic idea can be applied to other parametric surfaces. A variety of self-supporting surfaces obtained by the proposed computational scheme is presented.

We present AutoConnect, an automatic method that creates customized, 3D-printable connectors attaching two physical objects together. Users simply position and orient virtual models of the two objects that they want to connect and indicate some auxiliary information such as weight and dimensions. Then, AutoConnect creates several alternative designs that users can choose from for 3D printing. The design of the connector is created by combining two holders, one for each object. We categorize the holders into two types. The first type holds standard objects such as pipes and planes. We utilize a database of parameterized mechanical holders and optimize the holder shape based on the grip strength and material consumption. The second type holds free-form objects. These are procedurally generated shell-gripper designs created based on geometric analysis of the object. We illustrate the use of our method by demonstrating many examples of connectors and practical use cases.

Presents the introductory editorial for this issue of the publication.

We present a system called Aibiki, which can support users in practicing the shamisen, a three-stringed Japanese musical instrument, via an automatic and adaptive score scroll. We chose Nagauta, as an example of a type of shamisen music. Each piece typically lasts 10-40 min; furthermore, both hands are required to play the shamisen, and it is not desirable to turn pages manually during a performance. In addition, there are some characteristic issues that are particular to the shamisen, including the variable tempo of the music and the unique timbre of the instrument, which makes pitch detection difficult using standard techniques. In this work, we describe an application that automatically scrolls through a musical score, initially at a predefined tempo. Because there is often a difference between the predefined tempo and tempo with which the musician plays the piece, the application adjusts speed of the score scroll based on the input from a microphone. We evaluated the performance of the application via a user study. We find that the system was able to scroll the score in time to the actual performance, and that the system was useful for practicing and playing the shamisen.

We present an animation creation workflow for integrating offline physical, painted media into the digital authoring of Flash-style animations. Generally, animators create animations with standardized digital authoring software. However, the results tend to lack the individualism or atmosphere of physical media. In contrast, illustrators have skills in painting physical media but have limited experience in animation. To incorporate their skills, we present a workflow that integrates the offline painting and digital animation creation processes in a labor-saving manner. First, a user makes a rough sketch of the visual elements and defines their movements using our digital authoring software with a sketch interface. Then these images are exported to printed pages, and users can paint using offline physical media. Finally, the work is scanned and imported back into the digital content, forming a composite animation that combines digital and physical media. We present an implementation of this system to demonstrate its workflow. We also discuss the advantages of using physical media in digital animations through design evaluations.

We live in a mass-production society today and everyone buy and use same things all over the world. This is cheap, but not necessarily ideal for individual persons. We envision that computer tools that help people to design things by themselves can enrich their lives. To that end, we have developed innovative interaction techniques for end users to (1) create rich graphics such as three-dimensional models and animations by simple sketching (2) design their own real-world, everyday objects such as clothing and furniture with realtime physical simulation integrated in a simple geometry editor, and (3) design the behavior of their personal robots and give instructions to them to satisfy their particular needs. I will introduce these result with live demonstrations.

In this paper, we propose a to-do list interface for sharing tasks between human and multiple agents including robots and software personal assistants. While much work on software architectures aims to achieve efficient (semi-)autonomous task coordination among human and agents, little work on user interfaces can be found for user-oriented flexible task coordination. Instead, most of the existing human-agent interfaces are designed to command a single agent to handle specific kinds of tasks. Meanwhile, our interface is designed to be a platform to share any kinds of tasks between users and multiple agents. When agents can handle the task, they ask for details and permission to execute it. Otherwise, they try supporting users or just keep silent. New tasks can be registered not only by humans but also by agents when errors occur that can only be fixed by human users. We present the interaction design and implementation of the interface, Sharedo, with three example agents, followed by brief user feedback collected from a preliminary user study.

The creation of interactive computer-controlled characters in interactive media is a challenging and multi-faceted task requiring the skills and effort of professionals from many fields. This work addresses authoring the interactive aspect of these characters' behaviors -- how characters act automatically in response to a dynamic user-controlled character. We present PaintBoard, a system that enables users to prototype and test discrete, real-time, interactive movements in a 2D grid environment simply by digitally painting a storyboard. We designed and developed a novel authoring technique for creating behaviors (painting storyboards) and a novel algorithm based on machine-learning, that analyzes a storyboard to create a behavior that works beyond situations provided in the input storyboard. We conducted two exploratory studies that grounded the prototype design, and present the results of a proof-of-concept workshop with game developers. Finally, we performed a comparison of machine learning algorithms' performance on our storyboard data.

In this paper, we propose an electric power management system for delivering power to home appliances with mobile robots. With our system, the user places an appliance without power cables freely within the home, and a robot provides the appliance with the electric power required for its operation. In addition to providing explanations of a usage scenario and a theoretical analysis, we demonstrate a prototype implementation. The robot of the prototype autonomously locates the target appliance, transfers its battery power to the appliance, and returns to the home position to recharge its battery. The validation study showed our proof-of-concept prototype worked as expected by the theory.

One of the difficulties with standard route maps is accessing to multi-scale routing information. The user needs to display maps in both a large scale to see details and a small scale to see an overview, but this requires tedious interaction such as zooming in and out. We propose to use a hierarchical structure for a route map, called a "Route Tree", to address this problem, and describe an algorithm to automatically construct such a structure. A Route Tree is a hierarchical grouping of all small route segments to allow quick access to meaningful large and small-scale views. We propose two Route Tree applications, "RouteZoom" for interactive map browsing and "TreePrint" for route information printing, to show the applicability and usability of the structure. We conducted a preliminary user study on RouteZoom, and the results showed that RouteZoom significantly lowers the interaction cost for obtaining information from a map compared to a traditional interactive map.

Current DJ interfaces lack direct support for typical digital communication common in social media. We present a novel DJ interface for live internet broadcast performances with remote audience feedback integration. Our multi-touch interface is designed for a table top display, featuring a time-line based visualization. Two studies are presented involving seven DJs, culminating in four live broadcasts gathering and analyzing data to better understand both the DJ and audience perspective. This study is one of the first to look closer at DJs and remote audiences. We present useful insight for future interaction design between DJs and remote audiences, and interface integrated audience feedback.

Parameter tweaking is one of the fundamental tasks in the editing of visual digital contents, such as correcting photo color or executing blendshape facial expression control. A problem with parameter tweaking is that it often requires much time and effort to explore a high-dimensional parameter space. We present a new technique to analyze such high-dimensional parameter space to obtain a distribution of human preference. Our method uses crowdsourcing to gather pairwise comparisons between various parameter sets. As a result of analysis, the user obtains a goodness function that computes the goodness value of a given parameter set. This goodness function enables two interfaces for exploration: Smart Suggestion, which provides suggestions of preferable parameter sets, and VisOpt Slider, which interactively visualizes the distribution of goodness values on sliders and gently optimizes slider values while the user is editing. We created four applications with different design parameter spaces. As a result, the system could facilitate the user's design exploration.

We devised a display technology that utilizes the phenomenon whereby the shading properties of fur change as the fibers are raised or flattened. One can erase drawings by first flattening the fibers by sweeping the surface by hand in the fiber's growth direction, and then draw lines by raising the fibers by moving the finger in the opposite direction. These material properties can be found in various items such as carpets in our living environments. We have developed three different devices to draw patterns on a "fur display" utilizing this phenomenon: a roller device, a pen device and pressure projection device. Our technology can turn ordinary objects in our environment into rewritable displays without requiring or creating any non-reversible modifications to them. In addition, it can be used to present large-scale image without glare, and the images it creates require no running costs to maintain.

Automatic global beautification methods have been proposed for sketch-based interfaces, but they can lead to undesired results due to ambiguity in the user's input. To facilitate ambiguity resolution in layout beautification, we present a novel user interface for visualizing and editing inferred relationships. First, our interface provides a preview of the beautified layout with inferred constraints, without directly modifying the input layout. In this way, the user can easily keep refining beautification results by interactively repositioning and/or resizing elements in the input layout. Second, we present a gestural interface for editing automatically inferred constraints by directly interacting with the visualized constraints via simple gestures. Our efficient implementation of the beautification system provides the user instant feedback. Our user studies validate that our tool is capable of creating, editing and refining layouts of graphic elements and is significantly faster than the standard snap-dragging and command-based alignment tools.

We present a series of projects for end-user authoring of interactive robotic behaviors, with a particular focus on the style of those behaviors: we call this approach Style-by-Demonstration (SBD). We provide an overview introduction of three different SBD platforms: SBD for animated character interactive locomotion paths, SBD for interactive robot locomotion paths, and SBD for interactive robot dance. The primary contribution of this article is a detailed cross-project SBD analysis of the interaction designs and evaluation approaches employed, with the goal of providing general guidelines stemming from our experiences, for both developing and evaluating SBD systems. In addition, we provide the first full account of our Puppet Master SBD algorithm, with an explanation of how it evolved through the projects.

This paper presents a novel three dimensional (3D) flower modeling technique that utilizes an X-ray computed tomography (CT) system and real-world flowers. Although a CT system provides volume data that captures the internal structures of flowers, it is difficult to accurately segment them into regions of particular organs and model them as smooth surfaces because a flower consists of thin organs that contact one another. We thus introduce a semi-automatic modeling technique that is based on a new active contour model with energy functionals designed for flower CT. Our key idea is to approximate flower components by two important primitives, a shaft and a sheet. Based on our active contour model, we also provide novel user interfaces and a numerical scheme to fit these primitives so as to reconstruct realistic thin flower organs efficiently. To demonstrate the feasibility of our technique, we provide various flower models reconstructed from CT volumes.

This paper introduces novel interactive techniques for designing original hand-launched free-flight glider airplanes which can actually fly. The aerodynamic properties of a glider aircraft depend on their shape, imposing significant design constraints. We present a compact and efficient representation of glider aerodynamics that can be fit to real-world conditions using a data-driven method. To do so, we acquire a sample set of glider flight trajectories using a video camera and the system learns a nonlinear relationship between forces on the wing and wing shape. Our acquisition system is much simpler to construct than a wind tunnel, but using it we can efficiently discover a wing model for simple gliding aircraft. Our resulting model can handle general free-form wing shapes and yet agrees sufficiently well with the acquired airplane flight trajectories. Based on this compact aerodynamics model, we present a design tool in which the wing configuration created by a user is interactively optimized to maximize flight-ability. To demonstrate the effectiveness of our tool for glider design by novice users, we compare it with a traditional design workflow.

This paper introduces an input and output device that enables illumination, bi-directional data communication, and position sensing on a soft cloth. This "LightCloth" is woven from diffusive optical fibers. Since the fibers are arranged in parallel, the cloth has one-dimensional position information. Sensor-emitter pairs attached to bundles of contiguous fibers enable bundle-specific light input and output. We developed a prototype system that allows full-color illumination and 8-bit data input by infrared signals. We present as an application a chair with a LightCloth cover whose illumination pattern is specified using an infrared light pen. Here we describe the implementation details of the device and discuss possible interactions using the device.

Current programming environments use textual or symbolic representations. While these representations are appropriate for describing logical processes, they are not appropriate for representing raw values such as human and robot posture data, which are necessary for handling gesture input and controlling robots. To address this issue, we propose Picode, a text-based development environment integrated with visual representations: photos of human and robots. With Picode, the user first takes a photo to bind it to posture data. S/he then drag-and-drops the photo into the code editor, where it is displayed as an inline image. A preliminary in-house user study implied positive effects of taking photos on the programming experience.

We propose a technique called voice augmented manipulation (VAM) for augmenting user operations in a mobile environment. This technique augments user interactions on mobile devices, such as finger gestures and button pressing, with voice. For example, when a user makes a finger gesture on a mobile phone and voices a sound into it, the operation will continue until stops making the sound or makes another finger gesture. The VAM interface also provides a button-based interface, and the function connected to the button is augmented by voiced sounds. Two experiments verified the effectiveness of the VAM technique and showed that repeated finger gestures significantly decreased compared to current touch-input techniques, suggesting that VAM is useful in supporting user control in a mobile environment.

The availability of low-cost digital fabrication devices enables new groups of users to participate in the design and fabrication of things. However, software to assist in the transition from design to actual fabrication is currently overlooked. In this paper, we introduce PacCAM, a system for packing 2D parts within a given source material for fabrication using 2D cutting machines. Our solution combines computer vision to capture the source material shape with a user interface that incorporates 2D rigid body simulation and snapping. A user study demonstrated that participants could make layouts faster with our system compared with using traditional drafting tools. PacCAM caters to a variety of 2D fabrication applications and can contribute to the reduction of material waste.

We present faceton, a geometric modeling primitive designed for building architectural models, using a six degrees of freedom (DoF) input device in a virtual environment (VE). A faceton is given as an oriented point floating in the air and defines a plane of infinite extent passing through the point. The polygonal mesh model is constructed by taking the intersection of the planes associated with the facetons. With the simple drag-and-drop and group interaction of faceton, users can easily create 3D architecture models in the VE. The faceton primitive and its interaction reduce the overhead associated with standard polygonal mesh modeling in VE, where users have to manually specify vertexes and edges which could be far away. The faceton representation is inspired by the research on boundary representations (B-rep) and constructive solid geometry (CSG), but it is driven by a novel adaptive bounding algorithm and is specifically designed for the 3D modeling activities in an immersive virtual environment.

Abstract Image storyboards of films and videos are useful for quick browsing and automatic video processing. A common approach for producing image storyboards is to display a set of selected key‐frames in temporal order, which has been widely used for 2D video data. However, such an approach cannot be applied for 3D animation data because different information is revealed by changing parameters such as the viewing angle and the duration of the animation. Also, the interests of the viewer may be different from person to person. As a result, it is difficult to draw a single image that perfectly abstracts the entire 3D animation data. In this paper, we propose a system that allows users to interactively browse an animation and produce a comic sequence out of it. Each snapshot in the comic optimally visualizes a duration of the original animation, taking into account the geometry and motion of the characters and objects in the scene. This is achieved by a novel algorithm that automatically produces a hierarchy of snapshots from the input animation. Our user interface allows users to arrange the snapshots according to the complexity of the movements by the characters and objects, the duration of the animation and the page area to visualize the comic sequence. Our system is useful for quickly browsing through a large amount of animation data and semi‐automatically synthesizing a storyboard from a long sequence of animation.

Abstract We present an algorithm for artistically modifying physically based shadows. With our tool, an artist can directly edit the shadow boundaries in the scene in an intuitive fashion similar to freeform curve editing. Our algorithm then makes these shadow edits consistent with respect to varying light directions and scene configurations, by creating a shadow mesh from the new silhouettes. The shadow mesh helps a modified shadow volume algorithm cast shadows that conform to the artistic shadow boundary edits, while providing plausible interaction with dynamic environments, including animation of both characters and light sources. Our algorithm provides significantly more fine‐grained local and direct control than previous artistic light editing methods, which makes it simple to adjust the shadows in a scene to reach a particular effect, or to create interesting shadow shapes and shadow animations. All cases are handled with a single intuitive interface, be it soft shadows, or (self‐)shadows on arbitrary receivers.

Abstract We introduce soft folding, a new interactive method for designing and exploring thin‐plate forms. A user specifies sharp and soft folds as two‐dimensional(2D) curves on a flat sheet, along with the fold magnitude and sharpness of each. Then, based on the soft folds, the system computes the three‐dimensional(3D) folded shape. Internally, the system first computes a fold field, which defines local folding operations on a flat sheet. A fold field is a generalization of a discrete fold graph in origami, replacing a graph with sharp folds with a continuous field with soft folds. Next, local patches are folded independently according to the fold field. Finally, a globally folded 3D shape is obtained by assembling the locally folded patches. This algorithm computes an approximation of 3D developable surfaces with user‐defined soft folds at an interactive speed. The user can later apply nonlinear physical simulation to generate more realistic results. Experimental results demonstrated that soft folding is effective for producing complex folded shapes with controllable sharpness.

In this article we present a multipart formal design and evaluation of the style-by-demonstration (SBD) approach to creating interactive robot behaviors: enabling people to design the style of interactive robot behaviors by providing an exemplar. We first introduce our Puppet Master SBD algorithm that enables the creation of interactive robot behaviors with a focus on style: Users provide an example demonstration of human–robot interaction and Puppet Master uses this to generate real-time interactive robot output that matches the demonstrated style. We further designed and implemented original interfaces for demonstrating interactive robot style and for interacting with the resulting robot behaviors. Following, we detail a set of studies we performed to appraise users' reactions to and acceptance of the SBD interaction design approach, the effectiveness of the underlying Puppet Master algorithm, and the usability of the demonstration interfaces. Fundamentally, this article investigates the broad questions of how people respond to SBD interaction, how they engage SBD interfaces, how SBD can be practically realized, and how the SBD approach to social human–robot interaction can be employed in future interaction design.

As robots continue to enter people's everyday spaces, we argue that it will be increasingly important to consider the robots' movement style as an integral component of their interaction design. That is, aspects of the robot's movement which are not directly related to a task at hand (e.g., pick up a ball) can have a strong impact on how people perceive that action (e.g., aggressively or hesitantly). We call these elements the movement style. We believe that perceptions of this kind of style will be highly dependent on the culture, group, or individual, and so people will need to have the ability to customize their robot. Therefore, in this work we use Style by Demonstration, a style focus on the more-traditional programming by demonstration technique, and present the Puppet Dancer system, an interface for constructing paired and interactive robotic dances. In this paper we detail the Puppet Dancer interface and interaction design, explain our new algorithms for teaching dance by demonstration, and present the results from a formal qualitative study.

There are many toolkits for physical UIs, but most physical UI applications are not locomotive. When the programmer wants to make things move around in the environment, he faces difficulty related to robotics. Toolkits for robot programming, unfortunately, are usually not as accessible as those for building physical UIs. To address this interdisciplinary issue, we propose Phybots, a toolkit that allows researchers and interaction designers to rapidly prototype applications with locomotive robotic things. The contributions of this research are the combination of a hardware setup, software API, its underlying architecture and a graphical runtime debug tool that supports the whole prototyping activity. This paper introduces the toolkit, applications and lessons learned from three user studies.

Vignette is an interactive system that facilitates texture creation in pen-and-ink illustrations. Unlike existing systems, Vignette preserves illustrators' workflow and style: users draw a fraction of a texture and use gestures to automatically fill regions with the texture. We currently support both 1D and 2D synthesis with stitching. Our system also has interactive refinement and editing capabilities to provide a higher level texture control, which helps artists achieve their desired vision. A user study with professional artists shows that Vignette makes the process of illustration more enjoyable and that first time users can create rich textures from scratch within minutes.

PINOKY is a wireless ring-like device that can be externally attached to any plush toy as an accessory that animates the toy by moving its limbs. A user is thus able to instantly convert any plush toy into a soft robot. The user can control the toy remotely or input the movement desired by moving the plush toy and having the data recorded and played back. Unlike other methods for animating plush toys, PINOKY is non-intrusive, so alterations to the toy are not required. In a user study, 1) the roles of plush toys in the participants' daily lives were examined, 2) how participants played with plush toys without PINOKY was observed, 3) how they played with plush toys with PINOKY was observed, and their reactions to the device were surveyed. On the basis of the results, potential applications were conceptualized to illustrate the utility of PINOKY.

We present Assignment Space Planning, a new efficient robot multi-agent coordination algorithm for the PSPACE-hard problem of multi-robot multi-object push rearrangement. In both simulated and real robot experiments, we demonstrate that our method produces optimal solutions for simple problems and exhibits novel emergent behaviors for complex scenarios. Assignment Space takes advantage of the domain structure by splitting the planning up into three stages, effectively reducing the search space size and enabling the planner to produce optimized plans in seconds. Our algorithm finds solutions of comparable quality to complete configuration space search while reducing the computing time to seconds, which allows our approach to be applied in practical scenarios in real-time.

We propose an AR-based painting interface that enables users to design an illumination distribution for a real room using an array of computer-controlled lights. Users specify an illumination distribution of the room by painting on the image obtained by a camera mounted in the room. The painting result is overlaid on the camera image as contour lines of the target illumination intensity. The system runs an optimization interactively to calculate light parameters to deliver the requested illumination condition. In this implementation, we used actuated lights that can change the lighting direction to generate the requested illumination condition more accurately and efficiently than static lights. We built a miniature-scale experimental environment and ran a user study to compare our method with a standard direct manipulation method using widgets. The results showed that the users preferred our method for informal light control.

We present universal earphones that use both a proximity sensor and a skin conductance sensor and we demonstrate several implicit interaction techniques they achieve by automatically detecting the context of use. The universal earphones have two main features. The first involves detecting the left and right sides of ears, which provides audio to either ear, and the second involves detecting the shared use of earphones and this provides mixed stereo sound to both earphones. These features not merely free users from having to check the left and right sides of earphones, but they enable them to enjoy sharing stereo audio with other people.

The style in which a robot moves, expressed through its gait or locomotion, can convey effective messages to people. For example, a robot could move aggressively in reaction to a person's actions, or alternatively react using a set of careful, submissive movements. Designing, implementing and programming robotic interfaces that react to users' actions with properly styled movements can be a difficult, daunting, and time consuming technical task. On the other hand, most people can easily perform such stylistic tasks and movements, for example, through acting them out.

Existing 3D sketching methods typically allow the user to draw in empty space which is imprecise and lacks tactile feedback. We introduce a shape-stamping interface where users can model with tangible 3D primitive shapes. Each of these shapes represents a copy or a fragment of the construction material. Instead of modeling in empty space, these shapes allow us to use the real-world environment and other existing objects as a tangible guide during 3D modeling. We call this approach Situated Modeling: users can create new real-sized 3D objects directly in 3D space while using the nearby existing objects as the ultimate reference. We also describe a two-handed shape-stamping technique for stamping with tactile feedback. We show a variety of doit-yourself furniture and household products designed with our system, and perform a user study to compare our method with a related AR-based modeling system.

This paper introduces DressUp, a computerized system for designing dresses with 3D input using the form of the human body as a guide. It consists of a body-sized physical mannequin, a screen, and tangible prop tools for drawing in 3D on and around the mannequin. As the user draws, he/she modifies or creates pieces of digital cloth, which are displayed on a model of the mannequin on the screen. We explore the capacity of our 3D input tools to create a variety of dresses. We also describe observations gained from users designing actual physical garments with the system.

We study uncertainty in graphical-based interaction (with special attention to sketches). We argue that a comprehensive model for the problem must include the interaction participants (and their current beliefs), their possible actions and their past sketches. It's yet unclear how to frame and solve the former problem, considering all the latter elements. We suggest framing the problem as a game and solving it with a game-theoretical solution, which leads to a framework for the design of new two-way, sketch-based user interfaces. In special, we use the framework to design a game that can progressively learn visual models of objects from user sketches, and use the models in real-world interactions. Instead of an abstract visual criterion, players in this game learn models to optimize interaction (the game's duration). This two-way sketching game addresses problems essential in emerging interfaces (such as learning and how to deal with interpretation errors). We review possible applications in robotic sketch-to-command, hand gesture recognition, media authoring and visual search, and evaluate two. Evaluations demonstrate how players improve performance with repeated play, and the influence of interaction aspects on learning.

We have developed a simple skin-like user interface that can be easily attached to curved as well as flat surfaces and used to measure tangential force generated by pinching and dragging interactions. The interface consists of several photoreflectors that consist of an IR LED and a phototransistor and elastic fabric such as stocking and rubber membrane. The sensing method used is based on our observation that photoreflectors can be used to measure the ratio of expansion and contraction of a stocking using the changes in transmissivity of IR light passing through the stocking. Since a stocking is thin, stretchable, and nearly transparent, it can be easily attached to various types of objects such as mobile devices, robots, and different parts of the body as well as to various types of conventional pressure sensors without altering the original shape of the object. It can also present natural haptic feedback in accordance with the amount of force exerted. A system using several such sensors can determine the direction of a two-dimensional force. A variety of example applications illustrated the utility of this sensing system.

Abstract We propose 2D stick figures as a unified medium for visualizing and searching for human motion data. The stick figures can express a wide range or human motion, and they are easy to be drawn by people without any professional training. In our interface, the user can browse overall motion by viewing the stick figure images generated from the database and retrieve them directly by using sketched stick figures as an input query. We started with a preliminary survey to observe how people draw stick figures. Based on the rules observed from the user study, we developed an algorithm converting motion data to a sequence of stick figures. The feature‐based comparison method between the stick figures provides an interactive and progressive search for the users. They assist the user's sketching by showing the current retrieval result at each stroke. We demonstrate the utility of the system with a user study, in which the participants retrieved example motion segments from the database with 102 motion files by using our interface.

Abstract We propose an efficient and light‐weight solution for rendering smooth shadow boundaries that do not reveal the tessellation of the shadow‐casting geometry. Our algorithm reconstructs the smooth contours of the underlying mesh and then extrudes shadow volumes from the smooth silhouettes to render the shadows. For this purpose we propose an improved silhouette reconstruction using the vertex normals of the underlying smooth mesh. Then our method subdivides the silhouette loops until the contours are sufficiently smooth and project to smooth shadow boundaries. This approach decouples the shadow smoothness from the tessellation of the geometry and can be used to maintain equally high shadow quality for multiple LOD levels. It causes only a minimal change to the fill rate, which is the well‐known bottleneck of shadow volumes, and hence has only small overhead.

We introduce the interactive system "Beady" to assist the design and construction of customized 3D beadwork. The user first creates a polygonal mesh model called the design model that represents the overall structure of the beadwork. Each edge of the mesh model corresponds to a bead in the beadwork. We provide two methods to create the design model. One is interactive modeling from scratch. The user defines the mesh topology with gestural interaction and the system continuously adjusts edge lengths by considering the physical constraints among neighboring beads. The other is automatic conversion that takes an existing polygonal model as input and generates a near-hexagonal mesh model with a near-uniform edge length as output. The system then converts the design model into a beadwork model with the appropriate wiring. Computation of an appropriate wiring path requires careful consideration, and we present an algorithm based on face stripification of the mesh. The system also provides a visual step-by-step guide to assist the manual beadwork construction process. We show several beadwork designs constructed by the authors and by test users using the system.

Geometric modeling and the physical validity of shapes are traditionally considered independently. This makes creating aesthetically pleasing yet physically valid models challenging. We propose an interactive design framework for efficient and intuitive exploration of geometrically and physically valid shapes. During any geometric editing operation, the proposed system continuously visualizes the valid range of the parameter being edited. When one or more constraints are violated after an operation, the system generates multiple suggestions involving both discrete and continuous changes to restore validity. Each suggestion also comes with an editing mode that simultaneously adjusts multiple parameters in a coordinated way to maintain validity. Thus, while the user focuses on the aesthetic aspects of the design, our computational design framework helps to achieve physical realizability by providing active guidance to the user. We demonstrate our framework on plank-based furniture design with nail-joint and frictional constraints. We use our system to design a range of examples, conduct a user study, and also fabricate a physical prototype to test the validity and usefulness of the system.

Character pose design is one of the most fundamental processes in computer graphics authoring. Although there are many research efforts in this field, most existing design tools consider only character body structure, rather than its interaction with the environment. This paper presents an intuitive sketching interface that allows the user to interactively place a 3D human character in a sitting position on a chair. Within our framework, the user sketches the target pose as a 2D stick figure and attaches the selected joints to the environment (e.g., the feet on the ground) with a pin tool. As reconstructing the 3D pose from a 2D stick figure is an ill-posed problem due to many possible solutions, the key idea in our paper is to reduce solution space by considering the interaction between the character and environment and adding physics constraints, such as balance and collision. Further, we formulated this reconstruction into a nonlinear optimization problem and solved it via the genetic algorithm (GA) and the quasi-Newton solver. With the GPU implementation, our system is able to generate the physically correct and visually pleasing pose at an interactive speed. The promising experimental results and user study demonstrates the efficacy of our method.

As various home robots come into homes, the need for efficient robot task management tools is arising. Current tools are designed for controlling individual robots independently, so they are not ideally suitable for assigning coordinated action among multiple robots. To address this problem, we developed a management tool for home robots with a graphical editing interface. The user assigns instructions by selecting a tool from a toolbox and sketching on a bird's-eye view of the environment. Layering supports the management of multiple tasks in the same room. Layered graphical representation gives a quick overview of and access to rich information tied to the physical environment. This paper describes the prototype system and reports on our evaluation of the system.

We present an actuated handheld puppet system for controlling the posture of a virtual character. Physical puppet devices have been used in the past to intuitively control character posture. In our research, an actuator is added to each joint of such an input device to provide physical feedback to the user. This enhancement offers many benefits. First, the user can upload pre-defined postures to the device to save time. Second, the system is capable of dynamically adjusting joint stiffness to counteract gravity, while allowing control to be maintained with relatively little force. Third, the system supports natural human body behaviors, such as whole-body reaching and joint coupling. This paper describes the user interface and implementation of the proposed technique and reports the results of expert evaluation. We also conducted two user studies to evaluate the effectiveness of our method.

Photograph is a very useful tool for describing configurations of real-world objects to others. People immediately understand various pieces of information such as "what is the target object" and "where is the target position" by looking at a photograph, even without verbal descriptions. Our goal was to leverage these features of photographs to enrich human-robot interactions. We propose to use photographs as a front-end between a human and a home robot system. We named this method "Snappy". The user takes a photo to remember the target in a real-world situation involving a task and shows it to the system to make it physically execute the task. We developed a prototype system in which the user took a photo of a dish layout on a table and showed it to the system later to then have robots deliver and arrange the dishes in the same way.

Object delivery by pushing objects with mobile robots on a flat surface has been successfully demonstrated. However, existing methods can push objects that have a circular or rectangular shape. In this paper, we introduce a learning-based approach for pushing objects of any irregular shape to user-specified goal locations. We first automatically collect a set of data on how an irregular-shaped object moves given the robot's relative position and pushing direction. We collect this data with a randomized approach, and we demonstrate that this approach can successfully collect useful data. Object delivery is achieved by using the collected data with a non-parametric regression method. We demonstrate our approach with a number of irregular-shaped objects.

Abstract—We present a method of instructing a sequential task to a household robot using a hand-held augmented reality device. The user decomposes a high-level goal such as “prepare a drink ” into steps such as delivering a mug under a kettle and pouring hot water into the mug. The user takes a photograph of each step using the device and annotates it with necessary information via touch operation. The resulting sequence of annotated photographs serves as a reference for review and reuse at a later time. We created a working prototype system with various types of robots and appliances. I.

Human-robot interaction (HRI) tasks in everyday environments will require people to direct or lead a robot as they walk in close proximity to it. Tasks that exemplify this interaction include a robotic porter, carrying heavy suitcases, or a robot carrying groceries. As many users may not be robotics experts, we argue that such interaction schemes must be accessible, easy to use and understand. In this paper, we present a dog-leash interface that enables a person to lead a robot simply by holding the leash, following a dog-leash interaction metaphor. We introduce variants on dog-leash robotic interaction, present our original interface implementation, and detail our formal qualitative evaluation, exploring how users perceive and accept the dog leash robotic interaction.

Beadwork is the art of connecting beads together by wires. While common beadwork is two-dimensional (2D), three-dimensional (3D) beadwork is also popular in oriental regions such as Japan and China. However, the design and construction of 3D beadwork is very difficult. The final shape is defined by the complicated three-dimensional interaction between beads and wires, thus making it very difficult to design manually. One also needs to specify an appropriate wire path to hold the beads together and to manually insert the wire into the beads one by one following the path to construct the beadwork. Careful observation of existing beadwork structures shows several geometrically interesting problems, which make beadwork design an interesting technical challenge.

Route setting is the process of creating climbing routes for the sport of free climbing. By fastening climbing holds to artificial climbing walls, a route emerges that provides a challenge for climbers. It is a difficult process usually requiring either a very skilled climber or a lot of time and test climbs. We propose a climbing route designer that aims to enable even novice climbers to create interesting and challenging routes by designing them on the computer using a simulated climber to analyze the route and visualize the result.

SketchChair is an application that allows novice users to control the entire process of designing and building their own chairs. Chairs are designed using a simple 2D sketch-based interface and design validation tools, and are then fabricated from sheet materials, cut by a laser cutter or CNC milling machine. This paper presents the concepts and details of SketchChair, and both miniature and full-sized chairs are designed using the application. We conclude with results and insights from a workshop in which novice users designed their own model chairs.

SketchChair is an application that allows novice users to take part in the entire process of designing and fabricating their own full-sized usable chairs from scratch. During the studio, participants will be able to design, build and take home their own scale model paper chair. Participants will be introduced to SketchChair whilst learning how to use a paper cutting plotter to make slice form models. Furthermore we will discuss the considerations around designing tools for customization and CNC production.

We present the FuwaFuwa sensor module, a round, hand-size, wireless device for measuring the shape deformations of soft objects such as cushions and plush toys. It can be embedded in typical soft objects in the household without complex installation procedures and without spoiling the softness of the object because it requires no physical connection. Six LEDs in the module emit IR light in six orthogonal directions, and six corresponding photosensors measure the reflected light energy. One can easily convert almost any soft object into a touch-input device that can detect both touch position and surface displacement by embedding multiple FuwaFuwa sensor modules in the object. A variety of example applications illustrate the utility of the FuwaFuwa sensor module. An evaluation of the proposed deformation measurement technique confirms its effectiveness.

Abstract We propose a method for interactive cloning of 3D surface geometry using a paintbrush interface, similar to the continuous cloning brush popular in image editing. Existing interactive mesh composition tools focus on atomic copy‐and‐paste of preselected feature areas, and are either limited to copying surface displacements, or require the solution of variational optimization problems, which is too expensive for an interactive brush interface. In contrast, our GeoBrush method supports real‐time continuous copying of arbitrary high‐resolution surface features between irregular meshes, including topological handles. We achieve this by first establishing a correspondence between the source and target geometries using a novel generalized discrete exponential map parameterization. Next we roughly align the source geometry with the target shape using Green Coordinates with automatically‐constructed cages. Finally, we compute an offset membrane to smoothly blend the pasted patch with C continuity before stitching it into the target. The offset membrane is a solution of a bi‐harmonic PDE, which is computed on the GPU in real time by exploiting the regular parametric domain. We demonstrate the effectiveness of GeoBrush with various editing scenarios, including detail enrichment and completion of scanned surfaces.

Although there is an abundance of 3D models available, most of them exist only in virtual simulation and are not immediately usable as physical objects in the real world. We solve the problem of taking as input a 3D model of a man-made object, and automatically generating the parts and connectors needed to build the corresponding physical object. We focus on furniture models, and we define formal grammars for IKEA cabinets and tables. We perform lexical analysis to identify the primitive parts of the 3D model. Structural analysis then gives structural information to these parts, and generates the connectors (i.e. nails, screws) needed to attach the parts together. We demonstrate our approach with arbitrary 3D models of cabinets and tables available online.

We present a novel interactive tool for garment design that enables, for the first time, interactive bidirectional editing between 2D patterns and 3D high-fidelity simulated draped forms. This provides a continuous, interactive, and natural design modality in which 2D and 3D representations are simultaneously visible and seamlessly maintain correspondence. Artists can now interactively edit 2D pattern designs and immediately obtain stable accurate feedback online, thus enabling rapid prototyping and an intuitive understanding of complex drape form.

This paper presents a lightweight sketching system that enables interactive illustration of complex fluid systems. Users can sketch on a 2.5-dimensional (2.5D) canvas to design the shapes and connections of a fluid circuit. These input sketches are automatically analyzed and abstracted into a hydraulic graph, and a new hybrid fluid model is used in the background to enhance the illustrations. The system provides rich simple operations for users to edit the fluid system incrementally, and the new internal flow patterns can be simulated in real time. Our system is used to illustrate various fluid systems in medicine, biology, and engineering. We asked professional medical doctors to try our system and obtained positive feedback from them.

We propose a cooking system that operates in an open environment. The system cooks a meal by pouring various ingredients into a boiling pot on an induction heating cooker and adjusts the heating strength according to the user's instructions. We then describe how the system incorporates robotic- and human-specific elements in a shared workspace so as to achieve a cooperative rudimentary cooking capability. First, we use small mobile robots instead of built-in arms to save space, improve flexibility and increase safety. Second, we use detachable visual markers to allow the user to easily configure the real-world environment. Third, we provide a graphical user interface to display detailed cooking instructions to the user. We hope insights obtained in this experiment will be useful for the design of other household systems in the future.

Personal photographs are important media for communication in our daily lives. People take photos to remember things about themselves and show them to others to share the experience. We expect that a photograph can be useful tool for teaching a task to a robot. We propose a novel human-robot interaction using photographs. The user takes a photo to remember the target in a real-world situation involving a task and shows it to the system to make it physically execute the task. We developed a prototype system in which the user took a photo of a dish arrangement on a table and showed it to the system later to then have a small robot deliver and arrange the dishes in the same way.

This paper proposes a physical user interface for a user to teach a robot to navigate a home environment. The user places small devices containing infrared based communication functionality as landmarks in the environment. The robot follows these landmarks to navigate to a goal landmark. Active landmarks communicate with each other to map their spatial relationships. Our method allows the user to start using the system immediately after placing the landmarks without installing any global position sensing system or prior mapping by the robot.

Robots are poised to enter our everyday environments such as our homes and offices, contexts that present unique questions such as the style of the robot's actions. Style-oriented characteristics are difficult to define programmatically, a problem that is particularly prominent for a robot's interactive behaviors, those that must react accordingly to dynamic actions of people. In this paper, we present a technique for programming the style of how a robot should follow a person by demonstration, such that non-technical designers and users can directly create the style of following using their existing skill sets. We envision that simple physical interfaces like ours can be used by non-technical people to design the style of a wide range of robotic behaviors.

This paper introduces a simple algorithm for non-prehensile object transportation by a pushing robot on a flat surface. We assume that the global position and orientation of the robot and objects are known. The system computes a dipole field around the object and moves the robot along the field. This simple algorithm resolves many subtle issues in implementing reliable pushing behaviors, such as collision avoidance, error recovery, and multi-robot coordination. We verify the effectiveness of the algorithm via several experiments with varying robot and object form factors. Although object delivery by pushing and motion control by a vector field are not new, the proposed algorithm offers easier implementation with fewer parameter adjustments because of its mode-less definition and scale-invariant formulation.

We propose a simple drag-and-drop interface on a touch screen to give instructions to an object-delivery robot. The user uses a finger to drag a target object and drop it to a target location on the screen that shows an image of the floor provided by a ceiling camera. The system then autonomously executes a delivery task without continuous control by the user. We employ user-guided computer vision methods to identify and track an object in a scene without prior registration. We conducted a user study, which showed that the participants completed separate tasks more successfully when using our method than when using a remote controller.

Spatial Sketch is a three-dimensional (3D) sketch application that bridges between physical movement and the fabrication of objects in the real world via cut planar materials. This paper explores the rationale and details behind the development of the Spatial Sketch application, and presents our observations from user testing and a hands-on lamp shade design workshop. Finally we reflect upon the relevance of embodied forms of human computer interaction for use in digital fabrication.

We introduce layer operations for single-view 3D deformable object manipulation, in which the user can control the depth order of layered 3D objects resting on a flat ground with simple clicks and drags, as in 2D drawing systems. We present two interaction techniques based on this idea and describe their implementation. The first technique is explicit layer swap. The user clicks the target layer, and the system swaps the layer with the one directly underneath it. The second technique is layer-aware dragging. As the user drags the object, the system adjusts its depth automatically to pass over or under a colliding object in the screen space, according to user control. Although the user interface is 2.5D, all scene representations are true 3D, and thus the system naturally supports local layering, self-occlusions, and folds. Internally, the system dynamically computes the apparent layer structure in the current configuration and makes appropriate depth adjustments to obtain the desired results. We demonstrate the effectiveness of this approach in cloth and rope manipulation systems.

We present a new sketch-based modeling approach in which models are interactively designed by drawing their 2D silhouettes from different views. The core idea of our paper is to limit the input to 2D silhouettes, removing the need to explicitly create or position 3D elements. Arbitrarily complex models can be constructed by assembling them out of parts defined by their silhouettes, which can be combined using CSG operations. We introduce a new simplified algorithm to compute CSG solids that leverages special properties of silhouette cylinders to convert the 3D CSG problem into one that can be handled entirely with 2D operations, making implementation simpler and more robust. We evaluate our approach by modeling a random sampling of man-made objects taken from the words in WordNet, and show that all of the tested man-made objects can be modeled quickly and easily using our approach.

We present a way to bring cartoon objects and characters into the third dimension, by giving them the ability to rotate and be viewed from any angle. We show how 2D vector art drawings of a cartoon from different views can be used to generate a novel structure, the 2.5D cartoon model, which can be used to simulate 3D rotations and generate plausible renderings of the cartoon from any view. 2.5D cartoon models are easier to create than a full 3D model, and retain the 2D nature of hand-drawn vector art, supporting a wide range of stylizations that need not correspond to any real 3D shape.

The modeling of volumetric objects is still a difficult problem. Solid texture synthesis methods enable the design of volumes with homogeneous textures, but global features such as smoothly varying colors seen in vegetables and fruits are difficult to model. In this paper, we propose a representation called diffusion surfaces (DSs) to enable modeling such objects. DSs consist of 3D surfaces with colors defined on both sides, such that the interior colors in the volume are obtained by diffusing colors from nearby surfaces. A straightforward way to compute color diffusion is to solve a volumetric Poisson equation with the colors of the DSs as boundary conditions, but it requires expensive volumetric meshing which is not appropriate for interactive modeling. We therefore propose to interpolate colors only locally at user-defined cross-sections using a modified version of the positive mean value coordinates algorithm to avoid volumetric meshing. DSs are generally applicable to model many different kinds of objects with internal structures. As a case study, we present a simple sketch-based interface for modeling objects with rotational symmetries that can also generate random variations of models. We demonstrate the effectiveness of our approach through various DSs models with simple non-photorealistic rendering techniques enabled by DSs.

Numerous robots have been developed, and some of them are already being used in homes, institutions, and workplaces. Despite the development of useful robot functions, the focus so far has not been on user interfaces of robots. General users of robots find it hard to understand what the robots are doing and what kind of work they can do. This paper presents an interface for the commanding home robots by using stroke gestures on a computer screen. This interface allows the user to control robots and design their behaviors by sketching the robot's behaviors and actions on a top-down view from ceiling cameras. To convey a feeling of directly controlling the robots, our interface employs the live camera view. In this study, we focused on a house-cleaning task that is typical of home robots, and developed a sketch interface for designing behaviors of vacuuming robots.

Typical Human Robot Interaction (HRI) assumes that the user explicitly interacts with robots. However, explicit control with robots can be unnecessary or even undesirable in certain cases, such as dealing with domestic services (or housework). In this paper, we propose an alternative strategy of interaction: the user implicitly controls a robot by issuing commands on corresponding real world objects and the environment. Robots then discover these commands and complete them in the background. We implemented a paper-tag-based interface to support such implicit robot control in a sensor-augmented home environment. Our initial user studies indicated that the paper-tag-based interface is particularly simple to use and provides users with flexibility in planning and controlling their housework tasks in a simulated home environment.

Designing a digital font requires not only artistic but also technical skills. However, it is time-consuming and requires professional skills to create a nice font with consistency looks and plausible details. We present a font model that blends structures and outlines of template fonts individually, yet maintains the original style. We also present an algorithm to automatically generate font by blending the template fonts. It takes the desired outline of one character as an input and infer blending weights.

In this paper we present an algorithm for watertight meshing of closed, sketched curves. The sketch is resampled as a piecewise linear (PWL) curve and placed onto a triangular grid. A small boundary (seed) that describes a closed path along grid points is placed inside the sketch and grown until it resembles the sketch. Vertices of the evolved grid boundary are projected onto the stroke to establish a bijective, ordered mapping. Finally, valences along the boundary are optimized while retaining the previously established mapping. The resulting mesh patch can be duplicated, stitched and inflated to generate a new shape, or used to fill a hole in an existing shape. We have implemented our algorithm in FiberMesh, an interactive sketch based interface for designing freeform surfaces, where it is used for the all mesh generation processes. The triangulation generated with our algorithm improves the quality of the model by reducing the number of irregular vertices, while running at real time rates.

Abstract We developed an interactive system to design a customized cover for a given three‐dimensional (3D) object such as a camera, teapot, or car. The system first computes the convex hull of the input geometry. The user segments it into several cloth patches by drawing on the 3D surface. This paper provides two technical contributions. First, it introduces a specialized flattening algorithm for cover patches. It makes each two‐dimensional edge in the flattened pattern equal to or longer than the original 3D edge; a smaller patch would fail to cover the object, and a larger patch would result in extra wrinkles. Second, it introduces a mechanism to verify that the user‐specified opening would be large enough for the object to be removed. Starting with the initial configuration, the system virtually “pulls” the object out of the cover while avoiding excessive stretching of cloth patches. We used the system to design real covers and confirmed that it functions as intended.

Abstract Animations of characters with flexible bodies such as jellyfish, snails, and, hearts are difficult to design using traditional skeleton‐based approaches. A standard approach is keyframing, but adjusting the shape of the flexible body for each key frame is tedious. In addition, the character cannot dynamically adjust its motion to respond to the environment or user input. This paper introduces a new procedural deformation framework (ProcDef) for designing and driving animations of such flexible objects. Our approach is to synthesize global motions procedurally by integrating local deformations. ProcDef provides an efficient design scheme for local deformation patterns; the user can control the orientation and magnitude of local deformations as well as the propagation of deformation signals by specifying line charts and volumetric fields. We also present a fast and robust deformation algorithm based on shape‐matching dynamics and show some example animations to illustrate the feasibility of our framework.

Abstract We propose a system that allows the user to design a continuous flow animation starting from a still fluid image. The basic idea is to apply the fluid motion extracted from a video example to the target image. The system first decomposes the video example into three components, an average image, a flow field and residuals. The user then specifies equivalent information over the target image. The user manually paints the rough flow field, and the system automatically refines it using the estimated gradients of the target image. The user semi‐automatically transfers the residuals onto the target image. The system then approximates the average image and synthesizes an animation on the target image by adding the transferred residuals and warping them according to the user‐specified flow field. Finally, the system adjusts the appearance of the resulting animation by applying histogram matching. We designed animations of various pictures, such as rivers, waterfalls, fires, and smoke.

Abstract In the context of non‐photorealistic imaging, such as digital cel animation, lighting is symbolic and stylized to depict the scene's mood and the geometric or physical features of the objects in the scene. Stylized light and shade should therefore be intentionally animated rather than rigorously simulated. However, it is difficult to achieve smooth animation of light and shade that are stylized with a user's intention, because such stylization cannot be achieved using just conventional 3D lighting. To address this problem, we propose a 3D stylized lighting method, focusing on several stylized effects including straight lighting, edge lighting, and detail lighting which are important features in hand‐drawn cartoon animation. Our method is an extension of the conventional cartoon shader and introduces a light coordinate system for light shape control with smooth animations of light and shade. We also extend a toon mapping process for detailed feature lighting. Having these algorithms in a real‐time cartoon shader, our prototype system allows the interactive creation of stylized lighting animations. We show several animation results obtained by our method to illustrate usefulness and effectiveness of our method. Copyright © 2009 John Wiley & Sons, Ltd.

We present a system for 3D modeling of free-form surfaces from 2D sketches. Our system frees users to create 2D sketches from arbitrary angles using their preferred tool, which may include pencil and paper. A 3D model is created by placing primitives and annotations on the 2D image. Our primitives are based on commonly used sketching conventions and allow users to maintain a single view of the model. This eliminates the frequent view changes inherent to existing 3D modeling tools, both traditional and sketch-based, and enables users to match input to the 2D guide image. Our annotations---same-lengths and angles, alignment, mirror symmetry, and connection curves---allow the user to communicate higher-level semantic information; through them our system builds a consistent model even in cases where the original image is inconsistent. We present the results of a user study comparing our approach to a conventional "sketch-rotate-sketch" workflow.

We present a demonstration-based system for automatically generating succinct step-by-step visual tutorials of photo manipulations. An author first demonstrates the manipulation using an instrumented version of GIMP that records all changes in interface and application state. From the example recording, our system automatically generates tutorials that illustrate the manipulation using images, text, and annotations. It leverages automated image labeling (recognition of facial features and outdoor scene structures in our implementation) to generate more precise text descriptions of many of the steps in the tutorials. A user study comparing our automatically generated tutorials to hand-designed tutorials and screen-capture video recordings finds that users are 20--44% faster and make 60--95% fewer errors using our tutorials. While our system focuses on tutorial generation, we also present some initial work on generating content-dependent macros that use image recognition to automatically transfer selection operations from the example image used in the demonstration to new target images. While our macros are limited to transferring selection operations we demonstrate automatic transfer of several common retouching techniques including eye recoloring, whitening teeth and sunset enhancement.

We propose the "ninja cursor" to improve the performance of target acquisition, particularly on large screens. This technique uses multiple distributed cursors to reduce the average distance to targets. Each cursor moves synchronously following mouse movement. We present the design and implementation of the proposed technique, including a method to resolve the ambiguity that results when multiple cursors indicate different targets simultaneously. We also conducted an experiment to assess the performance of the ninja cursor. The results indicate that it can generally reduce movement time. However, the performance is greatly affected by the number of cursors and target density. Based on these results, we discuss how our technique can be put into practical use. In addition to presenting a novel method to improve pointing performance, our study is the first to explore a variable number of cursors for performing pointing tasks.

A history-of-user-operations function helps make applications easier to use. For example, users may have access to an operation history list in an application to undo or redo a past operation. To provide an overview of a long operation history and help users find target interactions or application states quickly, visual representations of operation history have been proposed. However, most previous systems are tightly integrated with target applications and difficult to apply to new applications. We propose an application-independent method that can visualize the operation history of arbitrary GUI applications by monitoring the input and output GUI events from outside of the target application. We implemented a prototype system that visualizes operation sequences of generic Java Awt/Swing applications using an annotated comic strip metaphor. We tested the system with various applications and present results from a user study.

Abstract A knitted animal is made of a closed surface consisting of several knitted patches knitted out of yarn and stuffed with cotton ( Fig. 1 ). We introduce a system to create a knitting pattern from a given 3D surface model (mainly designed for rotund animal models). A knitting pattern is an instructional diagram describing how to knit yarn to obtain a desired shape. Since the creation of knitting patterns requires special skill, this is difficult for nonprofessionals. Our system automates the process and allows anyone to obtain his or her original knitting patterns from a 3D model. The system first covers the surface of the model with parallel winding strips of constant width. The system then samples the strip at constant intervals to convert it into a knitting pattern. The result is presented in a standard visual format so that the user can easily refer it during actual knitting. We show several examples of knitted animals created using the system. Example of a knitted animal. A typical knitted animal consists of several circular and cylindrical patches. image

Abstract We present a method for synthesizing two dimensional (2D) element arrangements from an example. The main idea is to combine texture synthesis techniques based‐on a local neighborhood comparison and procedural modeling systems based‐on local growth. Given a user‐specified reference pattern, our system analyzes neigh‐borhood information of each element by constructing connectivity. Our synthesis process starts with a single seed and progressively places elements one by one by searching a reference element which has local features that are the most similar to the target place of the synthesized pattern. To support creative design activities, we introduce three types of interaction for controlling global features of the resulting pattern, namely a spray tool, a flow field tool, and a boundary tool. We also introduce a global optimization process that helps to avoid local error concentrations. We illustrate the feasibility of our method by creating several types of 2D patterns.

We present a method for representing solid objects with spatially-varying oriented textures by repeatedly pasting solid texture exemplars. The underlying concept is to extend the 2D texture patch-pasting approach of lapped textures to 3D solids using a tetrahedral mesh and 3D texture patches. The system places texture patches according to the user-defined volumetric tensor fields over the mesh to represent oriented textures. We have also extended the original technique to handle nonhomogeneous textures for creating solid models whose textural patterns change gradually along the depth fields. We identify several texture types considering the amount of anisotropy and spatial variation and provide a tailored user interface for each. With our simple framework, large-scale realistic solid models can be created easily with little memory and computational cost. We demonstrate the effectiveness of our approach with several examples including trees, fruits, and vegetables.

Construction of a 3-D scene consisting of multiple objects can be tedious work. Existing 3-D editing tools require the user to choose an appropriate model in a database first and then carefully place it in the scene at a desired position combining various operations such as translation, rotation, and scaling. To simplify the process, we propose a system that takes simple 2D sketches of models in a scene as input for 3D scene construction. The system then automatically identifies corresponding models in a database and puts them in the appropriate location and posture so that their appearance matches the user's input sketches. The system combines a 3-D model search and a 3-D posture estimation to obtain the result. This system allows the user to construct a prototype of a 3-D scene quickly and intuitively.

In this paper we present a presentation training system that observes a presentation rehearsal and provides the speaker with recommendations for improving the delivery of the presentation, such as to speak more slowly and to look at the audience. Our system "Presentation Sensei" is equipped with a microphone and camera to analyze a presentation by combining speech and image processing techniques. Based on the results of the analysis, the system gives the speaker instant feedback with respect to the speaking rate, eye contact with the audience, and timing. It also alerts the speaker when some of these indices exceed predefined warning thresholds. After the presentation, the system generates visual summaries of the analysis results for the speaker's self-examinations. Our goal is not to improve the content on a semantic level, but to improve the delivery of it by reducing inappropriate basic behavior patterns. We asked a few test users to try the system and they found it very useful for improving their presentations. We also compared the system's output with the observations of a human evaluator. The result shows that the system successfully detected some inappropriate behavior. The contribution of this work is to introduce a practical recognition-based human training system and to show its feasibility despite the limitations of state-of-the-art speech and video recognition technologies.

We present an appearance-based user interface for artists to efficiently design customized image-based lighting environments. 1 Our approach avoids typical iterations of parameter editing, rendering, and confirmation by providing a set of intuitive user interfaces for directly specifying the desired appearance of the model in the scene. Then the system automatically creates the lighting environment by solving the inverse shading problem. To obtain a realistic image, all-frequency lighting is used with a spherical radial basis function (SRBF) representation. Rendering is performed using precomputed radiance transfer (PRT) to achieve a responsive speed. User experiments demonstrated the effectiveness of the proposed system compared to a previous approach.

We present the boomerang technique, which makes it possible to suspend and resume drag-and-drop operations. A throwing gesture while dragging an object suspends the operation, anytime and anywhere. A drag-and-drop interaction, enhanced with our technique, allows users to switch windows, invoke commands, and even drag other objects during a drag-and-drop operation without using the keyboard or menus. We explain how a throwing gesture can suspend drag-and-drop operations, and describe other features of our technique, including grouping, copying, and deleting dragged objects. We conclude by presenting prototype implementations and initial feedback on the proposed technique.

Cameras are a useful source of input for many interactive applications, but computer vision programming is difficult and requires specialized knowledge that is out of reach for many HCI practitioners. In an effort to learn what makes a useful computer vision design tool, we created Eyepatch, a tool for designing camera-based interactions, and evaluated the Eyepatch prototype through deployment to students in an HCI course. This paper describes the lessons we learned about making computer vision more accessible, while retaining enough power and flexibility to be useful in a wide variety of interaction scenarios.

Spatial layout is frequently used for managing loosely organized information, such as desktop icons and digital ink. To help users organize this type of information efficiently, we propose an interface for manipulating spatial aggregations of objects. The aggregated objects are automatically recognized as a group, and the group structure is visualized as a two-dimensional bubble surface that surrounds the objects. Users can drag, copy, or delete a group by operating on the bubble. Furthermore, to help pick out individual objects in a dense aggregation, the system spreads the objects to avoid overlapping when requested. This paper describes the design of this interface and its implementation. We tested our technique in icon grouping and ink relocation tasks and observed improvements in user performance.

We introduce Plushie, an interactive system that allows nonprofessional users to design their own original plush toys. To design a plush toy, one needs to construct an appropriate two-dimensional (2D) pattern. However, it is difficult for non-professional users to appropriately design a 2D pattern. Some recent systems automatically generate a 2D pattern for a given three-dimensional (3D) model, but constructing a 3D model is itself a challenge. Furthermore, an arbitrary 3D model cannot necessarily be realized as a real plush toy, and the final sewn result can be very different from the original 3D model. We avoid this mismatch by constructing appropriate 2D patterns and applying simple physical simulation to it on the fly during 3D modeling. In this way, the model on the screen is always a good approximation of the final sewn result, which makes the design process much more efficient. We use a sketching interface for 3D modeling and also provide various editing operations tailored for plush toy design. Internally, the system constructs a 2D cloth pattern in such a way that the simulation result matches the user's input stroke. Our goal is to show that relatively simple algorithms can provide fast, satisfactory results to the user whereas the pursuit of optimal layout and simulation accuracy lies outside this paper's scope. We successfully demonstrated that non-professional users could design plush toys or balloon easily using Plushie.

This paper presents a system for designing freeform surfaces with a collection of 3D curves. The user first creates a rough 3D model by using a sketching interface. Unlike previous sketching systems, the user-drawn strokes stay on the model surface and serve as handles for controlling the geometry. The user can add, remove, and deform these control curves easily, as if working with a 2D line drawing. The curves can have arbitrary topology; they need not be connected to each other. For a given set of curves, the system automatically constructs a smooth surface embedding by applying functional optimization. Our system provides real-time algorithms for both control curve deformation and the subsequent surface optimization. We show that one can create sophisticated models using this system, which have not yet been seen in previous sketching or functional optimization systems.

Recent progress in non-photorealistic rendering (NPR) has led to many stylized shading techniques that efficiently convey visual information about the objects depicted. Another crucial goal of NPR is to give artists simple and direct ways to express the abstract ideas born of their imaginations. In particular, the ability to add intentional, but often unrealistic, shading effects is indispensable for many applications. We propose a set of simple stylized shading algorithms that allow the user to freely add localized light and shade to a model in a manner that is consistent and seamlessly integrated with conventional lighting techniques. The algorithms provide an intuitive, direct manipulation method based on a paint-brush metaphor, to control and edit the light and shade locally as desired. Our prototype system demonstrates how our method can enhance both the quality and range of applicability of conventional stylized shading for offline animation and interactive applications.

It is tedious to handwrite long passages of text by hand. To make this process more efficient, we propose predictive handwriting that provides input predictions when the user writes by hand. A predictive handwriting system presents possible next words as a list and allows the user to select one to skip manual writing. Since it is not clear if people are willing to use prediction, we first run a user study to compare handwriting and selecting from the list. The result shows that, in Japanese, people prefer to select, especially when the expected performance gain from using selection is large. Based on these observations, we designed a multimodal input system, called speech-pen, that assists digital writing during lectures or presentations with background speech and handwriting recognition. The system recognizes speech and handwriting in the background and provides the instructor with predictions for further writing. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the result of the instructor's speech recognition is sent to the audience to support their own note-taking. Our preliminary study shows the effectiveness of this system and the implications for further improvements.

Abstract We present an interactive modeling system for flower composition that supports seamless transformation from an initial sketch to a detailed three‐dimensional (3D) model. To begin, the user quickly sketches the overall appearance of the desired model as a collection of two‐dimensional (2D) strokes on hierarchical billboards. Then the user iteratively replaces the coarse sketch with a detailed 3D model referring to the initial sketch as a guide. Since a flower model consists of many repetitive components, the system helps the user to reuse 3D components to facilitate the modeling process. The global view of the entire model is always shown in a separate window to visualize how local modifications affect the global appearance. Our system helps the user make appropriate design decisions to keep the model consistent with the initial design, which is difficult in traditional bottom‐up plant modeling systems in which the global view only emerges after all of the details are specified. Categories and Subject Descriptors (according to ACM CCS): I.3.6 [Computer Graphics]: Methodology and Techniques – Interaction Techniques.

It is difficult to obtain a specific region within unsegmented volume data (region of interest, ROI). The user must first segment the volume, a task which itself involves significant user intervention, and then chooses a desired target within the 3D space. This paper proposes a simple and intuitive user interface for the task: the user traces the contour of the target region using a 2D free form stroke on the screen, and the system instantly returns a plausible 3D region inside the stroke by applying a segmentation algorithm. The main contribution is that the system infers the depth information of the ROI automatically by analyzing the data, whereas existing systems require the user to provide the depth information explicitly. Our system first computes the 3D location of the user-specified 2D stroke based on the assumption that the user traced the silhouette of the ROI, that is, the curve where the gradient is perpendicular to the viewing direction. The system then places constraint points around the 3D stroke to guide the following segmentation. Foreground constraints are placed inside the stroke and background constraints are placed outside the stroke. We currently use the statistical region-merging algorithm of Nock et al. [Nock and Nielsen 2004a] to perform the segmentation. We tested our system with real-world examples to verify the effectiveness of our approach.

We present an interactive system that lets a user move and deform a two-dimensional shape without manually establishing a skeleton or freeform deformation (FFD) domain beforehand. The shape is represented by a triangle mesh and the user moves several vertices of the mesh as constrained handles. The system then computes the positions of the remaining free vertices by minimizing the distortion of each triangle. While physically based simulation or iterative refinement can also be used for this purpose, they tend to be slow. We present a two-step closed-form algorithm that achieves real-time interaction. The first step finds an appropriate rotation for each triangle and the second step adjusts its scale. The key idea is to use quadratic error metrics so that each minimization problem becomes a system of linear equations. After solving the simultaneous equations at the beginning of interaction, we can quickly find the positions of free vertices during interactive manipulation. Our approach successfully conveys a sense of rigidity of the shape, which is difficult in space-warp approaches. With a multiple-point input device, even beginners can easily move, rotate, and deform shapes at will.

We present a system for modeling flowers in three dimensions quickly and easily while preserving correct botanical structures. We use floral diagrams and inflorescences , which were developed by botanists to concisely describe structural information of flowers. Floral diagrams represent the layout of floral components on a single flower, while inflorescences are arrangements of multiple flowers. Based on these notions, we created a simple user interface that is specially tailored to flower editing, while retaining a maximum variety of generable models. We also provide sketching interfaces to define the geometries of floral components. Separation of structural editing and editing of geometry makes the authoring process more flexible and efficient. We found that even novice users could easily design various flower models using our technique. Our system is an example of application-customized sketching, illustrating the potential power of a sketching interface that is carefully designed for a specific application.

This paper presents an interactive system for designing and browsing volumetric illustrations. Volumetric illustrations are 3D models with internal textures that the user can browse by cutting the models at desired locations. To assign internal textures to a surface mesh, the designer cuts the mesh and provides simple guiding information to specify the correspondence between the cross-section and a reference 2D image. The guiding information is stored with the geometry and used during the synthesis of cross-sectional textures. The key idea is to synthesize a plausible cross-sectional image using a 2D texture-synthesis technique, instead of sampling from a complete 3D RGB volumetric representation directly. This simplifies the design interface and reduces the amount of data, making it possible for non-experts to rapidly design and use volumetric illustrations. We believe that our system can enrich human communications in various domains, such as medicine, biology, and geology.

This paper describes a framework for introducing visually smooth surfaces into sketch-based freeform modeling systems. An existing sketch-based freeform modeling system generates rough polygonal meshes with uneven triangulations after each operation. Our approach generates a dense, visually smooth polygonal mesh by beautifying and refining the original rough mesh. A beautification process generates near-equilateral triangles with a near-uniform distribution of vertices to mask the noise and bad sampling of the uneven mesh. The vertices are distributed on a smoothed surface that approximately interpolates the original mesh. Refinement generates a smooth, dense mesh by subdividing the beautified mesh and moving the vertices to the interpolative surface. The smooth interpolative surface is computed via implicit quadratic surfaces that best fit the mesh locally in a least-squares sense.

We will present a user interface for quickly and easily modeling three-dimensional (3D) botanical trees from freehand sketches. The system generates 3D geometry from a two-dimensional (2D) sketch based on the assumption that trees spread their branches uniformly. Several editing operations are implemented, such as adding, cutting, and erasing branches. Our system also predicts, and generates, branches automatically. The user can continue editing a 3D tree by selecting the predicted branches. The interface of our system allows novices to design reasonably natural-looking trees quickly and interactively as compared with the interfaces of rule-based systems designed for expert users (e.g., L-systems).

Cascading menus are commonly seen in most GUI systems. However, people sometimes choose the wrong items by mistake, or become frustrated when submenus pop up unnecessarily. This paper proposes two methods for improving the usability of cascading menus. The first uses the direction of cursor movement to change the menu behavior: horizontal motion opens/closes submenus, while vertical motion changes the highlight within the current menu. This feature can reduce cursor movement errors. The second causes a submenu to pop up at the position where horizontal motion occurs. This is expected to reduce the length of the movement path for menu traversal. A user study showed that our methods reduce menu selection times, shorten search path lengths, and prevent unexpected submenu appearance and disappearance.

No abstract available.

This paper presents interaction techniques (and the underlying implementations) for putting clothes on a 3D character and manipulating them. The user paints freeform marks on the clothes and corresponding marks on the 3D character; the system then puts the clothes around the body so that corresponding marks match. Internally, the system grows the clothes on the body surface around the marks while maintaining basic cloth constraints via simple relaxation steps. The entire computation takes a few seconds. After that, the user can adjust the placement of the clothes by an enhanced dragging operation. Unlike standard dragging where the user moves a set of vertices in a single direction in 3D space, our dragging operation moves the cloth along the body surface to make possible more flexible operations. The user can apply pushpins to fix certain cloth points during dragging. The techniques are ideal for specifying an initial cloth configuration before applying a more sophisticated cloth simulation.

We present a method for dynamically generating an efficient texture bitmap and its associated UV-mapping in an interactive texture painting system for 3D models. Typical 3D texture painting programs require the user to explicitly define the underlying UV-mapping from 3D geometry to 2D bitmap prior to painting. This mapping is unchanged by the painting process. However, a predefined UV-mapping can cause distortion at arbitrary locations and waste bitmap memory in unpainted areas. To solve these problems, we propose an adaptive unwrapping mechanism where the system dynamically creates a tailored UV-mapping for newly painted polygons during the interactive painting process. This eliminates the distortion of brush strokes, and the resulting texture bitmap is more compact because the system allocates texture space only for the painted polygons. In addition, this dynamic texture allocation allows the user to paint smoothly at any zoom level. This technique can be efficiently implemented using standard 3D rendering engines, and the painted models can be stored as standard textured polygonal models. We implemented a prototype system, called Chameleon, and our users ’ experiences suggest that our technique is very useful for simple painting by casual users.

We describe the use of non-verbal features in voice for direct control of interactive applications. Traditional speech recognition interfaces are based on an indirect, conversational model. First the user gives a direction and then the system performs certain operation. Our goal is to achieve more direct, immediate interaction like using a button or joystick by using lower-level features of voice such as pitch and volume. We are developing several prototype interaction techniques based on this idea, such as "control by continuous voice", "rate-based parameter control by pitch," and "discrete parameter control by tonguing." We have implemented several prototype systems, and they suggest that voice-as-sound techniques can enhance traditional voice recognition approach.

This paper introduces a new type of interface for 3D drawings that improves the usability of gestural interfaces and augments typical command-based modeling systems. In our suggestive interface, the user gives hints about a desired operation to the system by highlighting related geometric components in the scene. The system then infers possible operations based on the hints and presents the results of these operations as small thumbnails. The user completes the editing operation simply by clicking on the desired thumbnail. The hinting mechanism lets the user specify geometric relations among graphical components in the scene, and the multiple thumbnail suggestions make it possible to define many operations with relatively few distinct hint patterns. The suggestive interface system is implemented as a set of suggestion engines working in parallel, and is easily extended by adding customized engines. Our prototype 3D drawing system, Chateau, shows that a suggestive interface can effectively support construction of various 3D drawings.

This paper describes the software architecture for our pen-based electronic whiteboard system, called Flatland. The design goal of Flatland is to support various activities on personal office whiteboards, while maintaining the outstanding ease of use and informal appearance of conventional whiteboards. The GUI framework of existing window systems is too complicated and heavy-weight to achieve this goal, and so we designed a new architecture that works as a kind of window system for penbased applications. Our architecture is characterized by its use of freeform strokes as the basic primitive for both input and output, flexible screen space segmentation, pluggable applications that can operate on each segment, and built-in history management mechanisms. This architecture is carefully designed to achieve simple, unified coding and high extensibility, which was essential to the iterative prototyping of the Flatland interface. While the current implementation is optimized for large office whiteboards, this architecture is useful for the implementation of a range of various pen-based systems.

A number of recent systems have provided rich facilities for manipulating the timelines of applications. Such timelines represent the history of an application's use in some session, and captures the effects of the user's interactions with that application. Applications can use timeline manipulation techniques prosaically as a way to provide undo and redo within an application context; more interestingly, they can use these same techniques to make an application's history directly manipulable in richer ways by users. This paper presents a number of extensions to current techniques for representing and managing application timelines. The first extension captures causal relationships in timelines via a nested transaction mechanism. This extension addresses a common problem in history-based applications, namely, how to represent application state as a set of atomic, incremental operations. The second extension presents a model for "multi-level" time, in which the histories of a set of inter-related artifacts can be represented by both "local" and "global" timelines. This extension allows the histories of related objects in an application to be manipulated independently from one another.

We propose a navigation technique for browsing large documents that integrates rate-based scrolling with automatic zooming.The view automatically zooms out when the user scrolls rapidly so that the perceptual scrolling speed in screen space remains constant.As a result, the user can efficiently and smoothly navigate through a large document without becoming disoriented by extremely fast visual flow.By incorporating semantic zooming techniques, the user can smoothly access a global overview of the document during rate-based scrolling.We implemented several prototype systems, including a web browser, map viewer, image browser, and dictionary viewer.An informal usability study suggests that for a document browsing task, most subjects prefer automatic zooming and the technique exhibits approximately equal performance time to scroll bars, suggesting that automatic zooming is a helpful alternative to traditional scrolling when the zoomed out view provides appropriate visual cues.

Flatland is an augmented whiteboard interface designed for informal office work. Our research investigates approaches to building an augmented whiteboard in the context of continuous, long term office use. In particular, we pursued three avenues of research based on input from user studies: techniques for the management of space on the board, the ability to flexibly apply behaviors to support varied application semantics, and mechanisms for managing history on the board. Unlike some previously reported whiteboard systems, our design choices have been influenced by a desire to support long-term, informal use in an individual office setting.

Article Teddy: a sketching interface for 3D freeform design Share on Authors: Takeo Igarashi University of Tokyo University of TokyoView Profile , Satoshi Matsuoka Tokyo Institute of Technology Tokyo Institute of TechnologyView Profile , Hidehiko Tanaka University of Tokyo University of TokyoView Profile Authors Info & Claims SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniquesJuly 1999 Pages 409–416https://doi.org/10.1145/311535.311602Published:01 July 1999 830citation3,369DownloadsMetricsTotal Citations830Total Downloads3,369Last 12 Months47Last 6 weeks13 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Diagram manipulation in conventional CAD systems requires frequent mode switching and explicit placement of the pivot for rotation and scaling. In order to simplify this process, we propose an interaction technique called integrated manipulation, where the user can move, rotate, and scale without mode switching. In addition, the pivot for rotation and scaling automatically snaps to a contact point during moving operation. We performed a user study is performed using our prototype system and a commercial CAD system. The results showed that users could perform a diagram manipulation task much more rapidly using our technique.

The information presented in a document often consists of primary content as well as supporting material such as explanatory notes, detailed derivations, illustrations, and the like.We introduce a class of user interface techniques for fluid documents that supports the reader's shift to supporting material while maintaining the context of the primary material.Our approach initially minimizes the intrusion of supporting material by presenting it as a small visual cue near the annotated primary material.When the user expresses interest in the annotation, it expands smoothly to a readable size.At the same time, the primary material makes space for the expanded annotation.The expanded supporting material must be given space to occupy, and it must be made salient with respect to the surrounding primary material.These two aspects, space and salience, are subject to a negotiation between the primary and supporting material.This paper presents the components of our fluid document techniques and describes the negotiation architecture for ensuring that the presentations of both primary and supporting material are honored.

No abstract available.

We propose interactive beautification, a technique for rapid geometric design, and introduce the technique and its algorithm with a prototype system Pegasus.The motivation is to solve a problem with current drawing systems: too many complex commands and unintuitiveprocedures to satisfy geometric constraints.The Interactive beautification system receives the user'sfree stroke and beautifies it by considering geometric constraints among segments.A single stroke is beautified one after another, preventing accumulation of recognition errors or catastrophic deformation.Supported geometric constraints include perpendicularity, congruence, symmetry, etc., which were not seen in existing free stroke recognition systems.In addition, the system generates multiple candidates as a result of beautification to solve the problem of ambiguity.Using this technique, the user can draw precise diagrams rapidly satisfying geometric relations without using any editing commands.Interactive beautification is achieved by three sequential processes: 1) inferring underlining geometric constraints based on the spatial relationships among the input stroke and the existing segments, 2) generating multiple candidates by combining inferred constraints appropriately, and 3) evaluating the candidates to find the most plausible candidate and to remove the inappropriate candidates.A user study was performed using the prototypesystem, a commercial CAD tppl, and an OObased drawing system.The result showed that users can draw required diagrams more rapidly and moreprecisely using the prototype system.