Senior architects are pivotal in shaping architectural projects, yet integrating Generative AI (GenAI) into their workflows presents notable challenges. A formative study (N=11) identified key pain points in their early concept design process. To address these, we developed EarlyArchi, a GenAI-driven system supporting automated concept generation and evaluation. In a within-subject study (N=13), participants used EarlyArchi for early-stage design tasks. Results showed enhanced perceived creativity, improved design competency, and more efficient ideation. However, concerns emerged regarding controllability and domain-specific accuracy, highlighting the need for features that preserve professional autonomy and trust. Further analysis revealed three GenAI involvement modes—fully AI-driven, GenAI-led, and human-led—emphasizing the importance of adaptive role allocation in balancing creative exploration with expert leadership. These findings offer insights into supporting senior architects through GenAI while identifying key considerations for designing future human–AI co-creation systems.

Block-building activities are crucial for developing children's spatial reasoning and mathematical skills, yet parents often lack the expertise to guide these activities effectively. BrickSmart, a pioneering system, addresses this gap by providing spatial language guidance through a structured three-step process: Discovery & Design, Build & Learn, and Explore & Expand. This system uniquely supports parents in 1) generating personalized block-building instructions, 2) guiding parents to teach spatial language during building and interactive play, and 3) tracking children's learning progress, altogether enhancing children's engagement and cognitive development. In a comparative study involving 12 parent-child pairs children aged 6-8 years) for both experimental and control groups, BrickSmart demonstrated improvements in supportiveness, efficiency, and innovation, with a significant increase in children's use of spatial vocabularies during block play, thereby offering an effective framework for fostering spatial language skills in children.

Creative Problem-Solving (CPS) promotes creative and critical thinking while enhancing real-world problem-solving skills, making it essential for middle school education.However, providing personalized mentorship in CPS projects at scale is challenging due to resource constraints and diverse student needs.To address this, we developed Mentigo, an AI-driven mentor agent designed to guide middle school students through the CPS process.Using a dataset of real classroom interactions, we encoded CPS task stages, adaptive guidance strategies, and personalized feedback mechanisms to inform Mentigo's dynamic mentoring framework powered by large language models (LLMs).A comparative experiment with 12 students and evaluations from five expert educators demonstrated improved student engagement, creativity, and task performance.Our findings highlight design implications for using LLM-based AI mentors to enhance CPS learning in educational environments.

Online collaborative learning is increasingly important, yet children still face challenges communicating and working together virtually, limiting their engagement in long-term teamwork. To address this, we designed the Children’s Online Long-term Program (COLP), a 16-week online project-based learning program grounded in multiple learning theories. The program was implemented with 67 upper primary school students (Grades 3–6, ages 8–13) across five provinces in China. Results show that over one-third of participants sustained engagement in online teamwork. Interviews with children and their parents further revealed key communication channels, benefits, and challenges. Notably, parents played multiple roles in supporting their children’s collaboration, especially through modeling and guidance. This study contributes to the design of long-term online collaborative learning interventions for children within computer-supported collaborative learning (CSCL) communities.

Olfactory interfaces are pivotal in HCI, yet their development is hindered by limited application scenarios, stifling the discovery of new research opportunities. This challenge primarily stems from existing design tools focusing predominantly on odor display devices and the creation of standalone olfactory experiences, rather than enabling rapid adaptation to various contexts and tasks. Addressing this, we introduce Mul-O, a novel task-oriented development platform crafted to aid semi-professionals in navigating the diverse requirements of potential application scenarios and effectively prototyping ideas. Mul-O facilitates the swift association and integration of olfactory experiences into functional designs, system integrations, and concept validations. Comprising a web UI for task-oriented development, an API server for seamless third-party integration, and wireless olfactory display hardware, Mul-O significantly enhances the ideation and prototyping process in multisensory tasks. This was verified by a 15-day workshop attended by 30 participants. The workshop produced seven innovative projects, underscoring Mul-O’s efficacy in fostering olfactory innovation.

Numerous studies have shown that integrating scents into movies enhances viewer engagement and immersion. However, creating such olfactory experiences often requires professional perfumers to match scents, limiting their widespread use. To address this, we propose OdorAgent which combines a LLM with a text-image model to automate video-odor matching. The generation framework is in four dimensions: subject matter, emotion, space, and time. We applied it to a specific movie and conducted user studies to evaluate and compare the effectiveness of different system elements. The results indicate that OdorAgent possesses significant scene adaptability and enables inexperienced individuals to design odor experiences for video and images.

Conversational Agents (CAs) can facilitate information elicitation in various scenarios, such as semi-structured interviews. Current CAs can ask predetermined questions but lack skills for asking follow-up questions. Thus, we designed three approaches for CAs to automatically ask follow-up questions, i.e., follow-ups on concepts, follow-ups on related concepts, and general follow-ups. To investigate their effects, we conducted a user study (N=26) in which a CA interviewer asked follow-up questions generated by algorithms and crafted by human wizards. Our results showed that the CA's follow-up questions were readable and effective in information elicitation. The follow-ups on concepts and related concepts achieved a lower drop rate and better relevance, while the general follow-ups elicited more informative responses. Further qualitative analysis of the human-CA interview data revealed algorithm drawbacks and identified follow-up question techniques used by the human wizards. We provided design implications for improving information elicitation of future CAs based on the results.

Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants’ PEI of the CA, and the empathetic responses from the CA helped alleviate some participants’ negative emotions. Our work provides implications for designing future CAs with better PEI.

Constructing olfactory interfaces on demand requires significant design proficiency and engineering effort. The absence of powerful and convenient tools that reduced innovation complexity posed obstacles for future research in the area. To address this problem, we proposed O&O, a modular olfactory interface DIY toolkit. The toolkit consists of: (1) a scent generation kit, a set of electronics and accessories that supported three common scent vaporization techniques; (2) a module construction kit, a set of primitive cardboard modules for assembling permutable functional structures; (3) a design manual, a step-by-step design thinking framework that directs the decision-making and prototyping process. We organized a formal workshop with 19 participants and four solo DIY trials to evaluate the capability of the toolkit, the overall user engagement, the creations in both sessions, and the iterative suggestions. Finally, design implications and future opportunities were discussed for further research.

Implementing computational methods for preservation, inheritance, and promotion of Cultural Heritage (CH) has become a research trend across the world since the 1990s. In China, generations of scholars have dedicated themselves to studying the country’s rich CH resources; there are great potential and opportunities in the field of computational research on specific cultural artefacts or artforms. Based on previous works, this paper proposes a systematic framework for Chinese Cultural Heritage Computing that consists of three conceptual levels which are Chinese CH protection and development strategy, computing process, and computable cultural ecosystem. The computing process includes three modules: (1) data acquisition and processing, (2) digital modeling and database construction, and (3) data application and promotion. The modules demonstrate the computing approaches corresponding to different phases of Chinese CH protection and development, from digital preservation and inheritance to presentation and promotion. The computing results can become the basis for the generation of cultural genes and eventually the formation of computable cultural ecosystem Case studies on the Mogao caves in Dunhuang and the art of Guqin, recognized as world’s important tangible and intangible cultural heritage, are carried out to elaborate the computing process and methods within the framework. With continuous advances in data collection, processing, and display technologies, the framework can provide constructive reference for building up future research roadmaps in Chinese CH computing and related fields, for sustainable protection and development of Chinese CH in the digital age.

The use of digital examples plays a critical role in mobile UI design. Yet, it remains unclear how UX/UI designers manage (i.e., collect, archive, and utilize) examples to facilitate their design processes at different stages, and what possible challenges are imposed on the design of proper tools to support these practices. In this paper, we conduct a qualitative interview study with mobile UI/UX designers (12 experts and 12 novices), deriving the commonality in practices and analyzing possible differences across four design phases (Discover, Define, Develop, and Deliver) and expertise. In brief, we find that there is more diverse and frequent use of examples in the Discover and Develop phases, and that experts take more diverse advantage of the information from examples compared to novices. We further identify the challenges faced by designers when using existing example management services, and propose potential design implications for the development of more supportive design tools in the future.

Recently, researchers have become increasingly interested in finding new input methods for olfactory interfaces. Physical odor capture is a potential solution to this issue and, in order to make it more accessible for users, we designed a portable and fast smell capture prototype based on headspace technology and inspired by point-and-shoot cameras. We conducted a two-week diary study with 13 participants, in which they were allowed to freely use the prototype for odor capture activities. Through diary and interview feedback, we summarized factors such as the motivations of capturing, the collected odor types, and perceptual effects of odor replay. We found that the capture activities can positively affect user emotions, memory, or perception. User preferences on device parameters were also gathered to guide further design iterations. Physical odor capture has many potential applications in daily-life and other implications resulting from the study have been proposed for further research.

Designing a tactile image for blind people is a significant challenge due to the difficulty of recognizing objects on a 2D line drawing image by touch compared to vision. In this paper, we proposed ”tactile accuracy”, a new criterion to evaluate the performance of recognizing 242 raised line images of common objects for 30 subjects (10 blindfolded sighted subjects, 10 congenitally blind subjects, and 10 late blind subjects), instead of the conventional ”naming accuracy” used in the visual image recognition tasks. We used multi-level evaluation criteria including ”tactile accuracy” to systematically analyze the design factors in tactile images. The results showed that using multi-level evaluate criteria could help unveil the tactile cognitive preferences of different types of subjects for personalized learning. Moreover, we reported important design factors that affect tactile image recognition, thus providing guidelines on the design of tactile images.

In Dunhuang Mogao Grottoes, unique Buddhist murals of ancient China are preserved. Unfortunately, the exquisite murals are suffering from degradation. Experts have been trying to enhance public's awareness of mural protection, but there's no efficacious means to attract interest and popularize knowledge yet. In this paper, we propose RestoreVR, an interactive virtual reality (VR) system engaging users to experience Dunhuang mural restoration in a digital tour in the cave. Based on an online survey with the public and in-depth interviews with five Dunhuang experts, we derive a set of design requirements for generating embodied knowledge and situated experience in VR to bridge the gap between highly specialized experts and general audiences. Accordingly, we design RestoreVR and conduct a between-subjects user study to compare our system with traditional methods. The results suggest that RestoreVR significantly improves user experience and awareness of CH protection over existing methods.

This paper presents WalkingBot, a modular robot system that allows non-expert users to build a multi-legged robot in various morphologies using a set of building blocks with sensors and actuators embedded. The kinematic model of the built robot is interpreted automatically and revealed in a customized GUI through an integrated hardware and software design, so that users can understand, control, and program the robot easily. A Model Predictive Control (MPC) scheme is introduced to generate a control policy for various motions (e.g. moving forward, turning left) corresponding to the sensed robot structure, affording rich robot motions right after assembling. Targeting different levels of programming skill, two programming methods, visual block programming and events programming, are also presented to enable users to create their own interactive legged robot.

We present IRelics, a tangible interaction platform for the popularization of field archeology. IRelics allows users to experience archaeological field work activities as a serious game by using a set of tangible tools. We developed an innovative LWIR (Long Wavelength Infrared Rays) sensing system, which implements the design of tangible tools that provide real manipulation experiences. By interacting with IRelics, a player may experience different archaeological activities such as excavation and cleaning. We conducted two observations to evaluate the usability and effectiveness at archeology popularizing. Findings suggest that the IRelics platform can enhance the engagement of the participants by providing a positive and interactive environment while teaching them unfamiliar knowledge.

Abstract Brick elements are very popular and have been widely used in many areas, such as toy design and architectural fields. Designing a vivid brick sculpture to represent a three‐dimensional (3D) model is a very challenging task, which requires professional skills and experience to convey unique visual characteristics. We introduce an automatic system to convert an architectural model into a LEGO sculpture while preserving the original model's shape features. Unlike previous legolization techniques that generate a LEGO sculpture exactly based on the input model's voxel representation, we extract the model's visual features, including repeating components, shape details and planarity. Then, we translate these visual features into the final LEGO sculpture by employing various brick types. We propose a deformation algorithm in order to resolve discrepancies between an input mesh's continuous 3D shape and the discrete positions of bricks in a LEGO sculpture. We evaluate our system on various architectural models and compare our method with previous voxelization‐based methods. The results demonstrate that our approach successfully conveys important visual features from digital models and generates vivid LEGO sculptures.

The use of armatures is a convenient way of deforming and animating 3D digital models. However, interact with an armature is usually time-consuming, and often requires professional skills. Tangible interfaces, such as building blocks, while having improved the accessibility of digital construction, are still lacking in flexibility and present difficulties in dealing with curved armatures. This paper introduces TwistBlocks, a pluggable and twistable modular TUI that improves the accessibility of 3D modeling and animating by physical armature interaction. TwistBlocks is capable of creating complex armatures with dense branches, and supports a high DOF (Degree of Freedom) in physical manipulation. In addition, a set of software tools are provided for novice users to easily create, rig, and animate models. The global-posture sensing network sensing scheme can also measure the rotation and movement of the physical armature, and enables interaction between multiple models.

Taking a famous Chinese painting for a case study, the paper presents a virtual exhibition platform. Through the platform, users can walk into the scenes in the painting with virtual candles in hands, know the scenes which are endowed vitality by attaching actor performances, and see every detail of the artwork. The scenes change their light, shades and shadows in real time by the candles, just as real scenes. For real-time candle-moving and light-changing interaction, in implementation, we render the light effects at densely sampled user positions offline, and extract the light, shades and shadows as masks; during online processing, the system merges the artwork with masks chosen by the positions of candles. The system, novel in both design and techniques, has been partially used in the Palace Museum (Beijing).

Visual-textual presentation layout (e.g., digital magazine cover, poster, Power Point slides, and any other rich media), which combines beautiful image and overlaid readable texts, can result in an eye candy touch to attract users’ attention. The designing of visual-textual presentation layout is therefore becoming ubiquitous in both commercially printed publications and online digital magazines. However, handcrafting aesthetically compelling layouts still remains challenging for many small businesses and amateur users. This article presents a system to automatically generate visual-textual presentation layouts by investigating a set of aesthetic design principles, through which an average user can easily create visually appealing layouts. The system is attributed with a set of topic-dependent layout templates and a computational framework integrating high-level aesthetic principles (in a top-down manner) and low-level image features (in a bottom-up manner). The layout templates, designed with prior knowledge from domain experts, define spatial layouts, semantic colors, harmonic color models, and font emotion and size constraints. We formulate the typography as an energy optimization problem by minimizing the cost of text intrusion, the utility of visual space, and the mismatch of information importance in perception and semantics, constrained by the automatically selected template and further preserving color harmonization. We demonstrate that our designs achieve the best reading experience compared with the reimplementation of parts of existing state-of-the-art designs through a series of user studies.

With the proliferation of mobile devices, people are taking videos of the same events anytime and anywhere. Even though these crowdsourced videos are uploaded to the cloud and shared, the viewing experience is very limited due to monotonous viewing, visual redundancy, and bad audio-video quality. In this paper, we present a fully automatic mobile video mashup system that works in the cloud to combine recordings captured by multiple devices from different view angles and at different time slots into a single yet enriched and professional looking video-audio stream. We summarize a set of computational filming principles for multicamera settings from a formal focus study. Based on these principles, given a set of recordings of the same event, our system is able to synchronize these recordings with audio fingerprints, assess audio and video quality, detect video cut points, and generate video and audio mashups. The audio mashup is the maximization of audio quality under the less switching principle, while the video mashup is formalized as maximizing video quality and content diversity, constrained by the summarized filming principles. Our system is different from any existing work in this field in three ways: 1) our system is fully automatic; 2) the system incorporates a set of computational domain-specific filming principles summarized from a formal focus study; and 3) in addition to video, we also consider audio mashup that is a key factor of user experience (UX) yet often overlooked in existing research. Evaluations show that our system achieves performance results that are superior to state-of-the-art video mashup techniques, thus providing a better UX.

We present style-aware image cloning, a novel image editing approach for artworks, which allows users to seamlessly insert any photorealistic or artificial objects into an artwork to create a new image that shares the same artistic style with the original artwork. To this end, a real-time image transfer algorithm is developed to stylize the cloned object according to a distance metric based on the artistic styles and semantic information. Several interactive functions, such as layering, shadowing, semantic labeling, and direction field editing, are provided to enhance the harmonization of the composite image. Extensive experimental results demonstrate the effectiveness of our method.

We introduce our interdisciplinary studies on information art & design, including 1) the interdisciplinary master program of information design at Tsinghua University that is a new platform to integrate art, new media and technology, to educate students through both theoretical and practical training on cutting-edge and social impact research projects; 2) e-Heritage, which is to apply the latest information technologies to aid the preservation, interpretation, and dissemination of cultural and natural heritages, in which we are working together with museums and cultural heritage sites to develop immersive multimedia exhibition systems, to recover old paintings, and to animate some art contents; 3) new media design and its applications of digital museum and gallery; 4) discussion on the future of information art & design.

We present an interactive video stylization system for transforming an input video into a painterly animation. The system consists of two phases: a content extraction phase to obtain semantic objects, i.e., recognized content, in a video and establish dense feature correspondences, and a painterly rendering phase to select, place, and propagate brush strokes for stylized animations based on the semantic content and object motions derived from the first phase. Compared with the previous work, the proposed method has the following three advantages. First, we propose a two-pass rendering strategy and brush strokes with mixed colors in order to render expressive visual effects. Second, the brush strokes are warped according to global object deformations, so that the strokes appear to be naturally attached to the object surfaces. Third, we propose a deferred rendering and backward completion method to draw brush strokes on emerging regions and simulate a damped system to reduce stroke scintillation effect. Moreover, we discuss the graphics processing unit-based implementation of our system, which is demonstrated to greatly improve the efficiency of producing stylized videos. In experiments, we verify this system by applying it to a number of video clips to produce expressive oil-painting animations and compare it with the state-of-the-art approaches.

Mobile devices are becoming ubiquitous. People take pictures via their phone cameras to explore the world on the go. In many cases, they are concerned with the picture-related information. Understanding user intent conveyed by those pictures therefore becomes important. Existing mobile applications employ visual search to connect the captured picture with the physical world. However, they only achieve limited success due to the ambiguity nature of user intent in the picture-one picture usually contains multiple objects. By taking advantage of multitouch interactions on mobile devices, this paper presents a prototype of interactive mobile visual search, named TapTell, to help users formulate their visual intent more conveniently. This kind of search leverages limited yet natural user interactions on the phone to achieve more effective visual search while maintaining a satisfying user experience. We make three contributions in this work. First, we conduct a focus study on the usage patterns and concerned factors for mobile visual search, which in turn leads to the interactive design of expressing visual intent by gesture. Second, we introduce four modes of gesture-based interactions (crop, line, lasso, and tap) and develop a mobile prototype. Third, we perform an in-depth usability evaluation on these different modes, which demonstrates the advantage of interactions and shows that lasso is the most natural and effective interaction mode. We show that TapTell provides a natural user experience to use phone camera and gesture to explore the world. Based on the observation and conclusion, we also suggest some design principles for interactive mobile visual search in the future.

We present an approach to synthesize the subtle 3D relief and texture of oil painting brush strokes from a single photograph. This task is unique from traditional synthesize algorithms due to its mixed modality between the input and output; i.e., our goal is to synthesize surface normals given an intensity image input. To accomplish this task, we propose a framework that first applies intrinsic image decomposition to produce a pair of initial normal maps. These maps are combined into a conditional random field (CRF) optimization framework that incorporates additional information derived from a training set consisting of normals captured using photometric stereo on oil paintings with similar brush styles. Additional constraints are incorporated into the CRF framework to further ensures smoothness and preserve brush stroke edges. Our results show that this approach can produce compelling reliefs that are often indistinguishable from results captured using photometric stereo.

Mobile devices are becoming ubiquitous. People use their phones as a personal concierge discovering and making decisions anywhere and anytime. Understanding user intent on the go therefore becomes important for task completion on the phone. While existing efforts have predominantly focused on understanding the explicit user intent expressed by a textual or voice query, this paper presents an approach to context-aware and personalized entity recommendation which understands the implicit intent without any explicit user input on the phone. The approach, highly motivated from a large-scale mobile click-through analysis, is able to rank both the entity types and the entities within each type (here an entity is a local business, e.g., "I love sushi," while an entity type is a category, e.g., "restaurant"). The recommended entity types and entities are relevant to both user context (past behaviors) and sensor context (time and geo-location). Specifically, it estimates the generation probability of an entity by a given user conditioned on the current context in a probabilistic framework. A random-walk propagation is then employed to refine the estimated probability by mining the temporal patterns among entities. We deploy a recommendation application based on the proposed approach on Window Phone 7 devices. We evaluate recommendation performance on 10 thousand mobile clicks, as well as user experience through subjective user studies. We show that the application is effective to facilitate the exploration and discovery of surroundings for mobile users.

Abstract Beam tracing combines the flexibility of ray tracing and the speed of polygon rasterization. However, beam tracing so far only handles linear transformations; thus, it is only applicable to linear effects such as planar mirror reflections but not to non‐linear effects such as curved mirror reflection, refraction, caustics and shadows. In this paper, we introduce non‐linear beam tracing to render these non‐linear effects. Non‐linear beam tracing is highly challenging because commodity graphics hardware supports only linear vertex transformation and triangle rasterization. We overcome this difficulty by designing a non‐linear graphics pipeline and implementing it on top of a commodity GPU. This allows beams to be non‐linear where rays within the same beam do not have to be parallel or intersect at a single point. Using these non‐linear beams, real‐time GPU applications can render secondary rays via polygon streaming similar to how they render primary rays. A major strength of this methodology is that it naturally supports fully dynamic scenes without the need to pre‐store a scene database. Utilizing our approach, non‐linear ray tracing effects can be rendered in real‐time on a commodity GPU under a unified framework.

Color and tone adjustments are among the most frequent image enhancement operations. We define a color and tone style as a set of explicit or implicit rules governing color and tone adjustments. Our goal in this paper is to learn implicit color and tone adjustment rules from examples. That is, given a set of examples, each of which is a pair of corresponding images before and after adjustments, we would like to discover the underlying mathematical relationships optimally connecting the color and tone of corresponding pixels in all image pairs. We formally define tone and color adjustment rules as mappings, and propose to approximate complicated spatially varying nonlinear mappings in a piecewise manner. The reason behind this is that a very complicated mapping can still be locally approximated with a low-order polynomial model. Parameters within such low-order models are trained using data extracted from example image pairs. We successfully apply our framework in two scenarios, low-quality photo enhancement by transferring the style of a high-end camera, and photo enhancement using styles learned from photographers and designers.

Due to the rapid increase in video capture technology, more and more tourist videos are captured every day, creating a challenge for organization and association with metadata. In this paper, we present a novel system for annotating and navigating tourist videos. Placing annotations in a video is difficult because of the need to track the movement of the camera. Navigation of a regular video is also challenging due to the sequential nature of the media. To overcome these challenges, we introduce a system for registering videos to geo-referenced 3D models and analyzing the video contents. We also introduce a novel scheduling algorithm for showing annotations in video. We show results in automatically annotated videos and in a map-based application for browsing videos. Our user study indicates the system is very useful.

In this paper, we present a system for exhibiting a Chinese landscape painting about 900 years old. There are three parts in our system: (1) we allocate a voice dubbing or background music, which is treated as a point sound source, onto the 2D painting and obtain its position in the 2D space. All of the audio data are then located in a 3D hidden space, by projecting their 2D positions to the 3D space through a projection model. (2) A two-layer directed graph structure is proposed to well organize the audio data in a 4D space (with 1D temporal and 3D spatial). (3) The exhibition is defined as an active exploration in a viewpoint space, which faces both the image and the 3D world where the sound sources reside. The 3D space and the two-layer graph structure generate a natural and meaningful stereo audio field. Meanwhile, compared to videos with guided walk through, the active exploration makes the exhibition more attractive.

It is often important for designers and photographers to convey or enhance desired color themes in their work. A color theme is typically defined as a template of colors and an associated verbal description. This paper presents a data-driven method for enhancing a desired color theme in an image. We formulate our goal as a unified optimization that simultaneously considers a desired color theme, texture-color relationships as well as automatic or user-specified color constraints. Quantifying the difference between an image and a color theme is made possible by color mood spaces and a generalization of an additivity relationship for two-color combinations. We incorporate prior knowledge, such as texture-color relationships, extracted from a database of photographs to maintain a natural look of the edited images. Experiments and a user study have confirmed the effectiveness of our method.

We present an accelerated depth peeling algorithm for order-independent transparency rendering on graphics hardware. Unlike traditional depth peeling which only peels one layer of transparent pixels per rendering pass, our algorithm peels multiple layers simultaneously per rendering pass. Our acceleration is achieved via our fragment program which sorts and writes multiple fragment colors and depths via MRT. A notable feature of our algorithm is that it is robust against the unreliable parallel read-after-write behavior in current graphics hardware, guaranteeing correct transparency ordering. For ordinary scenes rendered under RGBA8 color precision, we achieve up to 8x speed-up over conventional depth peeling with current generation graphics hardware. Our algorithm is simple to implement on current GPU without any hardware modification. In addition, it does not require applications to perform any pre-sorting of transparent geometry.

We investigate exploiting the class specific information in the conventional perceptual edge grouping for the task of object extraction, since the domain information is usually available in practice. Instead of applying the classical Gestalt principles, we turn to learn a class specific probabilistic structure model from training images. During the learning, both geometrical and photometric features such as color and texture are fused. Experiments show the model is fairly robust to the intra-class variations of object as well as background clutters. Moreover, we design a novel saliency measure for the grouping based on the probabilistic structure model. The object extraction is formulated as an optimization problem which can be efficiently solved by the recently developed ratio contour algorithm. The effectiveness of the proposed method is demonstrated by the experiments on real images.

This article presents an intuitive and easy-to-use system for interactively posing 3D facial expressions. The user can model and edit facial expressions by drawing freeform strokes, by specifying distances between facial points, by incrementally editing curves on the face, or by directly dragging facial points in 2D screen space. Designing such an interface for 3D facial modeling and editing is challenging because many unnatural facial expressions might be consistent with the user's input. We formulate the problem in a maximum a posteriori framework by combining the user's input with priors embedded in a large set of facial expression data. Maximizing the posteriori allows us to generate an optimal and natural facial expression that achieves the goal specified by the user. We evaluate the performance of our system by conducting a thorough comparison of our method with alternative facial modeling techniques. To demonstrate the usability of our system, we also perform a user study of our system and compare with state-of-the-art facial expression modeling software (Poser 7).

This paper presents a sketch-based password authentication system called Scribble-a-Secret as a graphical password scheme in which free-form drawings are used as a means to authenticate users. Unlike existing schemes, this approach requires no input of graphical passwords in particular sequences of strokes. Moreover, the system allows for a modicum of variation when users recreate their passwords. Our technique uses edge orientations extracted from sketch images to discern one user from another. Our experiments show that our recognition technique is robust for recognizing sketches while differentiating from others with both a false acceptance rate and false rejection rate of less than 1%.

A family photo album based cartoon personalization system, EasyToon, is proposed in this paper. Using state of the art computer vision and graphics technologies and effective UI design, the interactive tool can quickly generate a personalized cartoon storyboard, which naturally blends a real face chosen from the family photo album into a cartoon picture. The personalized cartoon image is easily and quickly obtained in two main steps. First, the best face candidate is selected from the album interactively. Then a personalized cartoon image is automatically synthesized by blending the selected face into the interesting cartoon image. Experiments show that most users express great interest in our system. Without any art background, they can make a personalized cartoon of high quality using the EasyToon within minutes.

Panning and zooming interfaces for exploring very large images containing billions of pixels (gigapixel images) have recently appeared on the internet. This paper addresses issues that arise when creating and rendering auditory and textual annotations for such images. In particular, we define a distance metric between each annotation and any view resulting from panning and zooming on the image. The distance then informs the rendering of audio annotations and text labels. We demonstrate the annotation system on a number of panoramic images.

In this article, we introduce sketching reality , the process of converting a freehand sketch into a realistic-looking model. We apply this concept to architectural designs. As the sketch is being drawn, our system periodically interprets its 2.5D-geometry by identifying new junctions, edges, and faces, and then analyzing the extracted topology. The user can add detailed geometry and textures through sketches as well. This is possible through the use of databases that match partial sketches to models of detailed geometry and textures. The final product is a realistic texture-mapped 2.5D-model of the building. We show a variety of buildings that have been created using this system.

In this paper, we describe a new system for converting a user's freehand sketch of a tree into a full 3D model that is both complex and realistic-looking. Our system does this by probabilistic optimization based on parameters obtained from a database of tree models. The best matching model is selected by comparing its 2D projections with the sketch. Branch interaction is modeled by a Markov random field, subject to the constraint of 3D projection to sketch. Our system then uses the notion of self-similarity to add new branches before finally populating all branches with leaves of the user's choice. We show a variety of natural-looking tree models generated from freehand sketches with only a few strokes.

In this paper, we introduce an interactive tool for local color transfer. The new technique is based on the observation that color transfer operations are local in nature while at the same time should adhere global consistency. We introduce a brush by which the user specifies the source and destination image regions for color transfer. Color statistics in the source region are transferred to the destination region. A global optimization is then applied to eliminate vi- sual discontinuities that may result by the local operations. We demonstrate that our tool is easy to use yet effective in quickly generating diverse artistic effects.

Cosmetic makeup is used worldwide as a means to enhance beauty and express moods. An art form in its own right, cosmetic styles continuously change and evolve to reflect cultural and societal trends. While countless magazines and books are dedicated to demonstrating cosmetic art, the actual application of makeup still remains a physical endeavor. In this paper, we describe a procedure to apply cosmetic makeup to the image of a person's face with the click of a mouse. Our approach works from before- and-after example images created by professional makeup artists. Using our "cosmetic-transfer" procedure, we can realistically transfer the cosmetic style captured in the example-pair to another person's face. This greatly reduces the time and effort needed to demonstrate a cosmetic style on a new person's face. In addition, our approach can be used to mix-and- match, and even fine-tune, example styles, all virtually, without the need for any physical makeup.

In this paper, we present a shape constrained Markov network for accurate face alignment. The global face shape is defined as a set of weighted shape samples which are integrated into the Markov network optimization. These weighted samples provide structural constraints to make the Markov network more robust to local image noise. We propose a hierarchical Condensation algorithm to draw the shape samples efficiently. Specifically, a proposal density incorporating the local face shape is designed to generate more samples close to the image features for accurate alignment, based on a local Markov network search. A constrained regularization algorithm is also developed to weigh favorably those points that are already accurately aligned. Extensive experiments demonstrate the accuracy and effectiveness of our proposed approach.

Harmonic colors are sets of colors that are aesthetically pleasing in terms of human visual perception. In this paper, we present a method that enhances the harmony among the colors of a given photograph or of a general image, while remaining faithful, as much as possible, to the original colors. Given a color image, our method finds the best harmonic scheme for the image colors. It then allows a graceful shifting of hue values so as to fit the harmonic scheme while considering spatial coherence among colors of neighboring pixels using an optimization technique. The results demonstrate that our method is capable of automatically enhancing the color "look-and-feel" of an ordinary image. In particular, we show the results of harmonizing the background image to accommodate the colors of a foreground image, or the foreground with respect to the background, in a cut-and-paste setting. Our color harmonization technique proves to be useful in adjusting the colors of an image composed of several parts taken from different sources.

This article proposes a technique to animate a Chinese style painting given its image. We first extract descriptions of the brush strokes that hypothetically produced it. The key to the extraction process is the use of a brush stroke library, which is obtained by digitizing single brush strokes drawn by an experienced artist. The steps in our extraction technique are first to segment the input image, then to find the best set of brush strokes that fit the regions, and, finally, to refine these strokes to account for local appearance. We model a single brush stroke using its skeleton and contour, and we characterize texture variation within each stroke by sampling perpendicularly along its skeleton. Once these brush descriptions have been obtained, the painting can be animated at the brush stroke level. In this article, we focus on Chinese paintings with relatively sparse strokes. The animation is produced using a graphical application we developed. We present several animations of real paintings using our technique.

Lighting condition is an important factor in face analysis and synthesis, which has received extensive study in both computer vision and computer graphics. Motivated by the work on multilinear model, we propose a learning-based algorithm for relighting based on tensor framework, which explicitly accounts for the interaction of the identity factor and the lighting factor. The major contribution of our work is that we develop a novel algorithm based on a two-stage decomposition scheme to simultaneously and robustly solve for the identity parameter and the lighting parameter which are both unknown. Equipped with the decomposition algorithm, the capability of the tensor model is significantly extended. Experiment results illustrate the effectiveness of our algorithm.

We present a method that preserves the recognizability of key object interactions in a vector animation. The method allows an artist to author an animation once, and then output it to any display device. We specifically target mobile devices with small screen sizes. In order to adapt an animation, the author specifies an importance value for objects in the animation. The algorithm then identifies and categorizes the vector graphics objects that comprise the animation, leveraging the implicit relationship between extensible Markup Language (XML) and scalable vector graphics (SVG). Based on importance, the animation can then be automatically retargeted for any display using artistically motivated resizing and grouping algorithms that budget size and spatial detail for each object.

In this paper, we present an intelligent image editing and face synthesis system that automatically removes eyeglasses from an input frontal face image. Although conventional image editing tools can be used to remove eyeglasses by pixel-level editing, filling in the deleted eyeglasses region with the right content is a difficult problem. Our approach works at the object level where the eyeglasses are automatically located, removed as one piece, and the void region filled. Our system consists of three parts: eyeglasses detection, eyeglasses localization, and eyeglasses removal. First, an eye region detector, trained offline, is used to approximately locate the region of eyes, thus the region of eyeglasses. A Markov-chain Monte Carlo method is then used to accurately locate key points on the eyeglasses frame by searching for the global optimum of the posterior. Subsequently, a novel sample-based approach is used to synthesize the face image without the eyeglasses. Specifically, we adopt a statistical analysis and synthesis approach to learn the mapping between pairs of face images with and without eyeglasses from a database. Extensive experiments demonstrate that our system effectively removes eyeglasses.

We describe a system for transforming an input video into a highly abstracted, spatio-temporally coherent cartoon animation with a range of styles. To achieve this, we treat video as a space-time volume of image data. We have developed an anisotropic kernel mean shift technique to segment the video data into contiguous volumes. These provide a simple cartoon style in themselves, but more importantly provide the capability to semi-automatically rotoscope semantically meaningful regions.In our system, the user simply outlines objects on keyframes. A mean shift guided interpolation algorithm is then employed to create three dimensional semantic regions by interpolation between the keyframes, while maintaining smooth trajectories along the time dimension. These regions provide the basis for creating smooth two dimensional edge sheets and stroke sheets embedded within the spatio-temporal video volume. The regions, edge sheets, and stroke sheets are rendered by slicing them at particular times. A variety of styles of rendering are shown. The temporal coherence provided by the smoothed semantic regions and sheets results in a temporally consistent non-photorealistic appearance.

We present a framework for knitwear modeling and rendering that accounts for characteristics that are particular to knitted fabrics. We first describe a model for animation that considers knitwear features and their effects on knitwear shape and interaction. With the computed free-form knitwear configurations, we present an efficient procedure for realistic synthesis based on the observation that a single cross section of yarn can serve as the basic primitive for modeling entire articles of knitwear. This primitive, called the lumislice, describes radiance from a yarn cross section that accounts for fine-level interactions among yarn fibers. By representing yarn as a sequence of identical but rotated cross sections, the lumislice can effectively propagate local microstructure over arbitrary stitch patterns and knitwear shapes. The lumislice accommodates varying levels of detail, allows for soft shadow generation, and capitalizes on hardware-assisted transparency blending. These modeling and rendering techniques together form a complete approach for generating realistic knitwear.

In this paper, we present PicToon, a cartoon system which can generate a personalized cartoon face from an input Picture. PicToon is easy to use and requires little user interaction. Our system consists of three major components: an image-based Cartoon Generator, an interactive Cartoon Editor for exaggeration, and a speech-driven Cartoon Animator. First, to capture an artistic style, the cartoon generation is decoupled into two processes: sketch generation and stroke rendering. An example-based approach is taken to automatically generate sketch lines which depict the facial structure. An inhomogeneous non-parametric sampling plus a flexible facial template is employed to extract the vector-based facial sketch. Various styles of strokes can then be applied. Second, with the pre-designed templates in Cartoon Editor, the user can easily make the cartoon exaggerated or more expressive. Third, a real-time lip-syncing algorithm is also developed that recovers a statistical audio-visual mapping between the character's voice and the corresponding lip configuration. Experimental results demonstrate the effectiveness of our system.

In this paper, we present a system that automatically generates caricatures from input face images. From example caricatures drawn by an artist, our caricature system learns how an artist draws caricatures. In our approach, we decouple the process of caricature generation into two parts, i.e., shape exaggeration and texture style transferring. The exaggeration of a caricature is accomplished by a prototype-based method that captures the artist's understanding of what are distinctive features of a face and the exaggeration style. Such prototypes are learnt by analyzing the correlation between the image caricature pairs using partial least-squares (PLS). Experimental results demonstrate the effectiveness of our system.

In this paper we present an approach that synthesizes a dynamic sequence from another related sequence, and apply it to a virtual conductor: to synthesize linked figure animation from an input music track. We propose that the mapping between two dynamic sequences can be modeled with a Kernel-based Hidden Markov model, or KHMM. A KHMM is an HMM for which the kernel-based functions are used to model the state observation density of the joint input and output distribution. Specifically, the state observation density is estimated by employing a likelihood-weighted sampling scheme. Our KHMM model is ideal for dynamic sequence synthesis because the global dynamics are learned by the HMM, and subtle details in the dynamic mapping are kept in the kernel-based state density. We demonstrate our virtual conductor by synthesizing extensive animation sequences from input music sequences with different styles and beat patterns.

Human hair modeling is a difficult task. This paper presents a constructive hair modeling system with which users can sculpt a wide variety of hairstyles. Our Multiresolution Hair Modeling (MHM) system is based on the observed tendency of adjacent hair strands to form clusters at multiple scales due to static attraction. In our system, initial hair designs are quickly created with a small set of hair clusters. Refinements at finer levels are achieved by subdividing these initial hair clusters. Users can edit an evolving model at any level of detail, down to a single hair strand. High level editing tools support curling, scaling, and copy/paste, enabling users to rapidly create widely varying hairstyles. Editing ease and model realism are enhanced by efficient hair rendering, shading, antialiasing, and shadowing algorithms.

We present techniques for realistic modeling and rendering of feathers and birds. Our approach is motivated by the observation that a feather is a branching structure that can be described by an L-system. The parametric L-system we derived allows the user to easily create feathers of different types and shapes by changing a few parameters. The randomness in feather geometry is also incorporated into this L-system. To render a feather realistically, we have derived an efficient form of the bidirectional texture function (BTF), which describes the small but visible geometry details on the feather blade. A rendering algorithm combining the L-system and the BTF displays feathers photorealistically while capitalizing on graphics hardware for efficiency. Based on this framework of feather modeling and rendering, we developed a system that can automatically generate appropriate feathers to cover different parts of a bird's body from a few "key feathers" supplied by the user, and produce realistic renderings of the bird.

We propose a new physically-based model for real-time animation of draped cloth which not only speeds up the rendering greatly, but also maintains visually appealing results. Our simplified model is represented as a grid object composed of mass points connected by semi-rigid rods, whose behavior is governed by non-rigid dynamics. The longitudinal (vertical) and latitudinal (horizontal) directions of our model are decoupled and processed separately, and later combined to generate the final cloth. Moreover, we provide a uniform treatment of internal and external forces.

In this paper, we present an example-based facial sketch system. Our system automatically generates a sketch from an input image, by learning from example sketches drawn with a particular style by an artist. There are two key elements in our system: a non-parametric sampling method and a flexible sketch model. Given an input image pixel and its neighborhood, the conditional distribution of a sketch point is computed by querying the examples and finding all similar neighborhoods. An "expected sketch image" is then drawn from the distribution to reflect the drawing style. Finally, facial sketches are obtained by incorporating the sketch model. Experimental results demonstrate the effectiveness of our techniques.

In this paper, we present a cartoon face animation system for multimedia HCI applications. We animate face cartoons not only from input speech, but also based on emotions derived from speech signal. Using a corpus of over 700 utterances from different speakers, we have trained SVMs (support vector machines) to recognize four categories of emotions: neutral, happiness, anger and sadness. Given each input speech phrase, we identify its emotion content as a mixture of all four emotions, rather than classifying it into a single emotion. Then, facial expressions are= generated from the recovered emotion for each phrase, by morphing different cartoon templates that correspond to various emotions. To ensure smooth transitions in the animation, we apply low-pass filtering to the recovered (and possibly jumpy) emotion sequence. Moreover, lip-syncing is applied to produce the lip movement from speech, by recovering a statistical audio-visual mapping. Experimental results demonstrate that cartoon animation sequences generated by our system are of good and convincing quality.

We present a method for efficient synthesis of photorealistic free-form knitwear. Our approach is motivated by the observation that a single cross-section of yarn can serve as the basic primitive for modeling entire articles of knitwear. This primitive, called the lumislice, describes radiance from a yarn cross-section based on fine-level interactions — such as occlusion, shadowing, and multiple scattering — among yarn fibers. By representing yarn as a sequence of identical but rotated cross-sections, the lumislice can effectively propagate local microstructure over arbitrary stitch patterns and knitwear shapes. This framework accommodates varying levels of detail and capitalizes on hardware-assisted transparency blending. To further enhance realism, a technique for generating soft shadows from yarn is also introduced.

Abstract We present a method for rendering knitwear on free‐form surfaces. This method has three main advantages. First, it renders yarn microstructure realistically and efficiently. Second, the rendering efficiency of yarn microstructure does not come at the price of ignoring the interactions between the neighboring yarn loops. Such interactions are modeled in our system to further enhance realism. Finally, our approach gives the user intuitive control on a few key aspects of knitwear appearance: the fluffiness of the yarn and the irregularity in the positioning of the yarn loops. The result is a system that efficiently produces highly realistic rendering of free‐form knitwear with user control on key aspects of visual appearance. Copyright © 2001 John Wiley & Sons, Ltd.

We present an algorithm for synthesizing textures from an input sample. This patch-based sampling algorithm is fast and it makes high-quality texture synthesis a real-time process. For generating textures of the same size and comparable quality, patch-based sampling is orders of magnitude faster than existing algorithms. The patch-based sampling algorithm works well for a wide variety of textures ranging from regular to stochastic. By sampling patches according to a nonparametric estimation of the local conditional MRF density function, we avoid mismatching features across patch boundaries. We also experimented with documented cases for which pixel-based nonparametric sampling algorithms cease to be effective but our algorithm continues to work well.