Artificial intelligence (AI) technologies offer tremendous potential for product and service innovation, yet finding good use cases remains challenging. Currently, AI projects largely fail due to breakdowns in early stage ideation and problem formulation. Drawing on HCI research that used AI capabilities and examples to facilitate AI concept ideation, this paper investigates how these approaches might be operationalized in industry settings. We collaborated with cross-functional industry teams in insurance, accounting, and consultancy. We conducted a series of AI Design Sprints, where innovators simultaneously consider AI capabilities and human needs. All teams perceived the ideation method highly valuable both for rapidly exploring use cases and building AI literacy within teams. We detail our process, the challenges, and artifacts that proved effective. We share insights on how AI projects get initiated, and how innovation teams identify use cases. Reflecting on these case studies, we discuss opportunities for improving early stage AI innovation.

Humans are able to convey different messages using only touch. Equipping robots with the ability to under-stand social touch adds another modality in which humans and robots can communicate. In this paper, we present a social gesture recognition system using a fabric-based, large-scale tactile sensor placed onto the arms of a humanoid robot. We built a social gesture dataset using multiple participants and extracted temporal features for classification. By collecting tactile data on a humanoid robot, our system provides insights into human-robot social touch, and displays that the use of fabric based sensors could be a potential way of advancing the development of spHRI systems for more natural and effective communication.

Lightweight, mesh-level models of knit fabric behavior are useful for both interactive pattern editing and initialization of yarn-level simulations. However, existing mesh-level simulation methods abstract knitting as a homogeneous material, which prevents them from capturing more complicated mixed structures. Furthermore, these methods require different simulation parameters depending on the knit pattern, or arrangement of stitches within the knit. Thus, fitting these parameters to physical examples must be done for each new pattern, even when the same types of stitches are used. To address this, we observe that physical behavior of a stitch is determined not only by its individual structure but also by the stitch types that surround it. In our work, we extend the stitch mesh model to allow for neighbor-aware material properties at the stitch level. Using structural analysis of stitch connections, we derive a finite set of four-way kernels that combine to create general knit-purl patterns for relaxation. From this, we generate a set of reference patterns that can be measured to infer the rest-lengths of the kernels using a linear model. After knitting and measuring these reference patterns, we used the derived kernel rest lengths to run relaxation on our stitch mesh models with mixtures of knits and purls that we then validated against physical examples. Our results show that the 4 neighbors of each stitch is sufficient to account for much of the neighborhood-dependent deformation, while remaining simple enough to directly fit to measured data with a set of 11 basis swatches. This allows our relaxation method to efficiently estimate the rest shape of mixed knit-purl patterns, which enables fast fabric preview and more accurate yarn-level simulation.

We present an algorithm that canonicalizes the algebraic representations of the topological semantics of machine knitting programs. Machine knitting is a staple technology of modern textile production where hundreds of mechanical needles are manipulated to form yarn into interlocking loop structures. Our semantics are defined using a variant of a monoidal category, and they closely correspond to string diagrams. We formulate our canonicalization as an Abstract Rewriting System (ARS) over words in our category, and prove that our algorithm is correct and runs in polynomial time.

Advances in artificial intelligence (AI) have enabled unprecedented capabilities, yet innovation teams struggle when envisioning AI concepts. Data science teams think of innovations users do not want, while domain experts think of innovations that cannot be built. A lack of effective ideation seems to be a breakdown point. How might multidisciplinary teams identify buildable and desirable use cases? This paper presents a first hand account of ideating AI concepts to improve critical care medicine. As a team of data scientists, clinicians, and HCI researchers, we conducted a series of design workshops to explore more effective approaches to AI concept ideation and problem formulation. We detail our process, the challenges we encountered, and practices and artifacts that proved effective. We discuss the research implications for improved collaboration and stakeholder engagement, and discuss the role HCI might play in reducing the high failure rate experienced in AI innovation.

Clinical practice guidelines, care pathways, and protocols are designed to support evidence-based practices for clinicians; however, their adoption remains a challenge. We set out to investigate why clinicians deviate from the “Wake Up and Breathe” protocol, an evidence-based guideline for liberating patients from mechanical ventilation in the intensive care unit (ICU). We conducted over 40 hours of direct observations of live clinical workflows, 17 interviews with frontline care providers, and 4 co-design workshops at three different medical intensive care units. Our findings indicate that unlike prior literature suggests, disagreement with the protocol is not a substantial barrier to adoption. Instead, the uncertainty surrounding the application of the protocol for individual patients leads clinicians to deprioritize adoption in favor of tasks where they have high certainty. Reflecting on these insights, we identify opportunities for technical systems to help clinicians in effectively executing the protocol and discuss future directions for HCI research to support the integration of protocols into clinical practice in complex, team-based healthcare settings.

Prior robot painting and drawing work, such as FRIDA, has focused on decreasing the sim-to-real gap and expanding input modalities for users, but the interaction with these systems generally exists only in the input stages. To support interactive, human-robot collaborative painting, we introduce the Collaborative FRIDA (CoFRIDA) robot painting framework, which can co-paint by modifying and engaging with content already painted by a human collaborator. To improve text-image alignment–FRIDA’s major weakness–our system uses pre-trained text-to-image models; however, pre-trained models in the context of real-world co-painting do not perform well because they (1) do not understand the constraints and abilities of the robot and (2) cannot perform co-painting without making unrealistic edits to the canvas and overwriting content. We propose a self-supervised fine-tuning procedure that can tackle both issues, allowing the use of pre-trained state-of-the-art text-image alignment models with robots to enable co-painting in the physical world. Our open-source approach, CoFRIDA, creates paintings and drawings that match the input text prompt more clearly than FRIDA, both from a blank canvas and one with human created work. More generally, our fine-tuning procedure successfully encodes the robot’s constraints and abilities into a foundation model, showcasing promising results as an effective method for reducing sim-to-real gaps. https://pschaldenbrand.github.io/cofrida/

We introduce solid knitting, a new fabrication technique that combines the layer-by-layer volumetric approach of 3D printing with the topologically-entwined stitch structure of knitting to produce solid 3D objects. We define the basic building blocks of solid knitting and demonstrate a working prototype of a solid knitting machine controlled by a low-level instruction language, along with a volumetric design tool for creating machine-knittable patterns. Solid knitting uses a course-wale-layer structure, where every loop in a solid-knit object passes through both a loop from the previous layer and a loop from the previous course. Our machine uses two beds of latch needles to create stitches like a conventional V-bed knitting machine, but augments these needles with a pair of rotating hook arrays to provide storage locations for all of the loops in one layer of the object. It can autonomously produce solid-knit prisms of arbitrary length, although it requires manual intervention to cast on the first layer and bind off the final row. Our design tool allows users to create solid knitting patterns by connecting elementary stitches; objects designed in our interface can---after basic topological checks and constraint propagation---be exported as a sequence of instructions for fabrication on the solid knitting machine. We validate our solid knitting hardware and software on prism examples, detail the mechanical errors which we have encountered, and discuss potential extensions to the capability of our solid knitting machine.

Programming low-level controls for knitting machines is a meticulous, time-consuming task that demands specialized expertise. Recently, there has been a shift towards automatically generating low-level knitting machine programs from high-level knit representations that describe knit objects in a more intuitive, user-friendly way. Current high-level systems trade off expressivity for ease-of-use, requiring ad-hoc trapdoors to access the full space of machine capabilities, or eschewing completeness in the name of utility. Thus, advanced techniques either require ad-hoc extensions from domain experts, or are entirely unsupported. Furthermore, errors may emerge during the compilation from knit object representations to machine instructions. While the generated program may describe a valid machine control sequence, the fabricated object is topologically different from the specified input, with little recourse for understanding and fixing the issue. To address these limitations, we introduce instruction graphs , an intermediate representation capable of capturing the full range of machine knitting programs. We define a semantic mapping from instruction graphs to fenced tangles, which make them compatible with the established formal semantics for machine knitting instructions. We establish a semantics-preserving bijection between machine knittable instruction graphs and knit programs that proves three properties - upward, forward, and ordered (UFO) - are both necessary and sufficient to ensure the existence of a machine knitting program that can fabricate the fenced tangle denoted by the graph. As a proof-of-concept, we implement an instruction graph editor and compiler that allows a user to transform an instruction graph into UFO presentation and then compile it to a machine program, all while maintaining semantic equivalence. In addition, we use the UFO properties to more precisely characterize the limitations of existing compilers. This work lays the groundwork for more expressive and reliable automated knitting machine programming systems by providing a formal characterization of machine knittability.

Advances in artificial intelligence have enabled unprecedented technical capabilities, yet making these advances useful in the real world remains challenging. We engaged in a Research through Design process to improve the ideation of AI products and services. We developed a design resource capturing AI capabilities based on 40 AI features commonly used across various domains. To probe its usefulness, we created a set of slides illustrating AI capabilities and asked designers to ideate AI-enabled user experiences. We also incorporated capabilities into our own design process to brainstorm concepts with domain experts and data scientists. Our research revealed that designers should focus on innovations where moderate AI performance creates value. We reflect on our process and discuss research implications for creating and assessing resources to systematically explore AI’s problem-solution space.

A scarf is inherently reconfigurable: wearers often use it as a neck wrap, a shawl, a headband, a wristband, and more. We developed uKnit, a scarf-like soft sensor with scarf-like reconfigurability, built with machine knitting and electrical impedance tomography sensing. Soft wearable devices are comfortable and thus attractive for many human-computer interaction scenarios. While prior work has demonstrated various soft wearable capabilities, each capability is device- and location-specific, being incapable of meeting users’ various needs with a single device. In contrast, uKnit explores the possibility of one-soft-wearable-for-all. We describe the fabrication and sensing principles behind uKnit, demonstrate several example applications, and evaluate it with 10-participant user studies and a washability test. uKnit achieves 88.0%/78.2% accuracy for 5-class worn-location detection and 80.4%/75.4% accuracy for 7-class gesture recognition with a per-user/universal model. Moreover, it identifies respiratory rate with an error rate of 1.25 bpm and detects binary sitting postures with an average accuracy of 86.2%.

Painting is an artistic process of rendering visual content that achieves the high-level communication goals of an artist that may change dynamically throughout the creative process. In this paper, we present a Framework and Robotics Initiative for Developing Arts (FRIDA) that enables humans to produce paintings on canvases by collaborating with a painter robot using simple inputs such as language descriptions or images. FRIDA introduces several technical innovations for computationally modeling a creative painting process. First, we develop a fully differentiable simulation environment for painting, adopting the idea of real to simulation to real (real2sim2real). We show that our proposed simulated painting environment is higher fidelity to reality than existing simulation environments used for robot painting. Second, to model the evolving dynamics of a creative process, we develop a planning approach that can continuously optimize the painting plan based on the evolving canvas with respect to the high-level goals. In contrast to existing approaches where the content generation process and action planning are performed independently and sequentially, FRIDA adapts to the stochastic nature of using paint and a brush by continually re-planning and re-assessing its semantic goals based on its visual perception of the painting progress. We describe the details on the technical approach as well as the system integration. FRIDA software is freely available at: https://github.com/cmubig/Frida.

Tactile sensing is essential for robots to perceive and react to the environment. However, it remains a challenge to make large-scale and flexible tactile skins on robots. Industrial machine knitting provides solutions to manufacture customiz-able fabrics. Along with functional yarns, it can produce highly customizable circuits that can be made into tactile skins for robots. In this work, we present RobotSweater, a machine-knitted pressure-sensitive tactile skin that can be easily applied on robots. We design and fabricate a parameterized multi-layer tactile skin using off-the-shelf yarns, and characterize our sensor on both a flat testbed and a curved surface to show its robust contact detection, multi-contact localization, and pressure sensing capabilities. The sensor is fabricated using a well-established textile manufacturing process with a programmable industrial knitting machine, which makes it highly customizable and low-cost. The textile nature of the sensor also makes it easily fit curved surfaces of different robots and have a friendly appearance. Using our tactile skins, we conduct closed-loop control with tactile feedback for two applications: (1) human lead-through control of a robot arm, and (2) human-robot interaction with a mobile robot.

Weaving is a fabrication process that is grounded in mathematics and engineering: from the binary, matrix-like nature of the pattern drafts weavers have used for centuries, to the punch card programming of the first Jacquard looms. This intersection of disciplines provides an opportunity to ground abstract mathematical concepts in a concrete and embodied art, viewing this textile art through the lens of engineering. Currently, available looms are not optimized to take advantage of this opportunity to increase mathematics learning by providing hands-on interdisciplinary learning in collegiate classrooms. In this work, we present SPEERLoom: an open-source, robotic Jacquard loom kit designed to be a tool for interweaving cloth fabrication, mathematics, and engineering to support interdisciplinary learning in the classroom. We discuss the design requirements and subsequent design of SPEERLoom. We also present the results of a pilot study in a post-secondary class finding that SPEERLoom supports hands-on, interdisciplinary learning of math, engineering, and textiles.

Machine knitting is a well-established fabrication technique for complex soft objects, and both companies and researchers have developed tools for generating machine knitting patterns. However, existing representations for machine knitted objects are incomplete (do not cover the complete domain of machine knittable objects) or overly specific (do not account for symmetries and equivalences among knitting instruction sequences). This makes it difficult to define correctness in machine knitting, let alone verify the correctness of a given program or program transformation. The major contribution of this work is a formal semantics for knitout, a low-level Domain Specific Language for knitting machines. We accomplish this by using what we call the fenced tangle , which extends concepts from knot theory to allow for a mathematical definition of knitting program equivalence that matches the intuition behind knit objects. Finally, using this formal representation, we prove the correctness of a sequence of rewrite rules; and demonstrate how these rewrite rules can form the foundation for higher-level tasks such as compiling a program for a specific machine and optimizing for time/reliability, all while provably generating the same knit object under our proposed semantics. By establishing formal definitions of correctness, this work provides a strong foundation for compiling and optimizing knit programs.

HCI research has explored AI as a design material, suggesting that designers can envision AI’s design opportunities to improve UX. Recent research claimed that enterprise applications offer an opportunity for AI innovation at the user experience level. We conducted design workshops to explore the practices of experienced designers who work on cross-functional AI teams in the enterprise. We discussed how designers successfully work with and struggle with AI. Our findings revealed that designers can innovate at the system and service levels. We also discovered that making a case for an AI feature’s return on investment is a barrier for designers when they propose AI concepts and ideas. Our discussions produced novel insights on designers’ role on AI teams, and the boundary objects they used for collaborating with data scientists. We discuss the implications of these findings as opportunities for future research aiming to empower designers in working with data and AI.

Knitting can efficiently fabricate stretchable and durable soft surfaces. These surfaces are often designed to be worn on solid objects as covers, garments, and accessories. Given a 3D model, we consider a knit for it wearable if the knit not only reproduces the shape of the 3D model but also can be put on and taken off from the model without deforming the model. This "wearability" places additional constraints on surface design and fabrication, which existing machine knitting approaches do not take into account. We introduce the first practical automatic pipeline to generate knit designs that are both wearable and machine knittable. Our pipeline handles knittability and wearability with two separate modules that run in parallel. Specifically, given a 3D object and its corresponding 3D garment surface, our approach first converts the garment surface into a topological disc by introducing a set of cuts. The resulting cut surface is then fed into a physically-based unclothing simulation module to ensure the garment's wearability over the object. The unclothing simulation determines which of the previously introduced cuts could be sewn permanently without impacting wearability. Concurrently, the cut surface is converted into an anisotropic stitch mesh. Then, our novel, stochastic, any-time flat-knitting scheduler generates fabrication instructions for an industrial knitting machine. Finally, we fabricate the garment and manually assemble it into one complete covering worn by the target object. We demonstrate our method's robustness and knitting efficiency by fabricating models with various topological and geometric complexities. Further, we show that our method can be incorporated into a knitting design tool for creating knitted garments with customized patterns.

Machine knitting is an increasingly accessible fabrication technology for producing custom soft goods. However, recent machine knitting research has focused on knit shaping, or on adapting hand-knitting patterns. We explore a capability unique to machine knitting: producing multilayer spacer fabrics. These fabrics consist of two face layers connected by a monofilament filler yarn which gives the structure stiffness and volume. We show how to vary knit patterning and yarn parameters in spacer fabrics to produce tactile materials with embedded functionality for forming soft actuated mechanisms and sensors with tunable density, stiffness, material bias, and bristle properties. These soft mechanisms can be rapidly produced on a computationally-controlled v-bed knitting machine and integrated directly into soft objects.

We present an inexpensive tabletop loom that offers fully computational patterning while maintaining the flexibility of handweaving. Our loom can be assembled for under US$200 with 3D printed parts, and it can be controlled straightforwardly over USB. Our loom is explicitly a hand loom: that is, a weaver is required to operate the weaving process and may mediate row-by-row patterning and material specifics like yarn tension. Our approach combines the flexibility of fully analog handweaving with the computational affordances of digital fabrication: it enables the incorporation of special techniques and materials, as well as allowing for the possibility of computational and creative interventions in the weaving process itself. In taking this approach, we aim to serve a range of end users including artisans and researchers, whether for skill-building, for rapid prototyping, or for creative reflection.

Industrial knitting machines create fabric by manipulating loops held on hundreds of needles. A core problem in pattern making for these machines is transfer planning – coming up with a sequence of low-level operations that move loops to the appropriate needles so that knitting through those loops produces the correct final structure. Since each loop is connected to the larger piece in progress, transfer plans must account for not only loop position, but the way strands of yarn tangle around each other.We present the first complete, discrete representation of the machine’s loop-tangling process. Our representation combines a braid from the Artin Braid Group with an array of explicit loop positions to fully capture loop crossings. By storing braids in the Symmetric Normal Form, states can be quickly compared and updated incrementally with machine operations. This representation can be used to verify the equivalence of transfer operations, providing an important tool in optimizing knit manufacturing.We improve on prior work in transfer planning algorithms, which can only solve certain subclasses of problems and are frequently suboptimal in terms of fabrication time, by introducing a novel A* search heuristic and state-collapsing mechanism, which we show finds optimal transfer plans for a large benchmark set of small transfer planning problems.

Digital fabrication tools have transformed how people work in micro- and small-scale manufacturing settings. While increasing efficiency and precision, these tools raise concerns around user agency and control. This paper describes an exploratory study investigating the felt work experience and desired futures of professionals who use fabrication tools. We conducted co-design workshops with 23 professionals who use 3D printers, laser cutters, and CNC routers. We probed about current practices; machine awareness and autonomy; and user agency. Our findings reveal that current tools are not very professional. They are unreliable and untrustworthy. Participants desired smarter tools that can actively prevent errors and perform self-calibration and self-maintenance. They had few concerns that more intelligence would impact agency. They desired tools that could negotiate trade-offs between time, cost, and quality; and that can operate as super-human shop assistants. We discuss the implications of these findings as opportunities for research that can improve professionals' work experience.

Many digital design tasks require a user to set a large number of parameters. Gallery-based interfaces provide a way to quickly evaluate examples and explore the space of potential designs, but require systems to predict which designs from a high-dimensional space are the right ones to present to the user. In this paper we present the design adjectives framework for building parameterized design tools in high dimensional design spaces. The framework allows users to create and edit design adjectives, machine-learned models of user intent, to guide exploration through high-dimensional design spaces. We provide a domain-agnostic implementation of the design adjectives framework based on Gaussian process regression, which is able to rapidly learn user intent from only a few examples. Learning and sampling of the design adjective occurs at interactive rates, making the system suitable for iterative design workflows. We demonstrate use of the design adjectives framework to create design tools for three domains: materials, fonts, and particle systems. We evaluate these tools in a user study showing that participants were able to easily explore the design space and find designs that they liked, and in professional case studies that demonstrate the framework's ability to support professional design concepting workflows.

Expressive robots are useful in many contexts, from industrial to entertainment applications. However, designing expressive robot behaviors requires editing a large number of unintuitive control parameters. We present an interactive, data-driven system that allows editing of these complex parameters in a semantic space. Our system combines a physics-based simulation that captures the robot's motion capabilities, and a crowd-powered framework that extracts relationships between the robot's motion parameters and the desired semantic behavior. These relationships enable mixed-initiative exploration of possible robot motions. We specifically demonstrate our system in the context of designing emotionally expressive behaviors. A user-study finds the system to be useful for more quickly developing desirable robot behaviors, compared to manual parameter editing.

We present CATS, a digital painting system that synthesizes textures from live video in real-time, short-cutting the typical brush- and texture- gathering workflow. Through the use of boundary-aware texture synthesis, CATS produces strokes that are non-repeating and blend smoothly with each other. This allows CATS to produce paintings that would be difficult to create with traditional art supplies or existing software. We evaluated the effectiveness of CATS by asking artists to integrate the tool into their creative practice for two weeks; their paintings and feedback demonstrate that CATS is an expressive tool which can be used to create richly textured paintings.

Knitting creates complex, soft fabrics with unique texture properties that can be used to create interactive objects.However, little work addresses the challenges of designing and using knitted textures computationally. We present KnitPick: a pipeline for interpreting hand-knitting texture patterns into KnitGraphs which can be output to machine and hand-knitting instructions. Using KnitPick, we contribute a measured and photographed data set of 472 knitted textures. Based on findings from this data set, we contribute two algorithms for manipulating KnitGraphs. KnitCarving shapes a graph while respecting a texture, and KnitPatching combines graphs with disparate textures while maintaining a consistent shape. KnitPick is the first system to bridge the gap between hand- and machine-knitting when creating complex knitted textures.

Abstract We introduce a method to design lightweight shell objects that are structurally robust under the external forces they may experience during use. Given an input 3D model and a general description of the external forces, our algorithm generates a structurally‐sound minimum weight shell object. Our approach works by altering the local shell thickness repeatedly based on the stresses that develop inside the object. A key issue in shell design is that large thickness values might result in self‐intersections on the inner boundary creating a significant computational challenge during optimization. To address this, we propose a shape parametrization based on the solution to the Laplace's equation that guarantees smooth and intersection‐free shell boundaries. Combined with our gradient‐free optimization algorithm, our method provides a practical solution to the structural design of hollow objects with a single inner cavity. We demonstrate our method on a variety of problems with arbitrary 3D models under complex force configurations and validate its performance with physical experiments.

Industrial knitting machines are commonly used to manufacture complicated shapes from yarns; however, designing patterns for these machines requires extensive training. We present the first general visual programming interface for creating 3D objects with complex surface finishes on industrial knitting machines. At the core of our interface is a new, augmented, version of the stitch mesh data structure. The augmented stitch mesh stores low-level knitting operations per-face and encodes the dependencies between faces using directed edge labels. Our system can generate knittable augmented stitch meshes from 3D models, allows users to edit these meshes in a way that preserves their knittability, and can schedule the execution order and location of each face for production on a knitting machine. Our system is general, in that its knittability-preserving editing operations are sufficient to transform between any two machine-knittable stitch patterns with the same orientation on the same surface. We demonstrate the power and flexibility of our pipeline by using it to create and knit objects featuring a wide range of patterns and textures, including intarsia and Fair Isle colorwork; knit and purl textures; cable patterns; and laces.

From smart toys and household appliances to personal robots, electromechanical devices play an increasingly important role in our daily lives. Rather than relying on gadgets that are mass-produced, our goal is to enable casual users to custom-design such devices based on their own needs and preferences. To this end, we present a computational design system that leverages the power of digital fabrication and the emergence of affordable electronics such as sensors and microcontrollers. The input to our system consists of a 3D representation of the desired device's shape, and a set of user-preferred off-the-shelf components. Based on this input, our method generates an optimized, 3D printable enclosure that can house the required components. To create these designs automatically, we formalize a new spatio-temporal model that captures the entire assembly process, including the placement of the components within the device, mounting structures and attachment strategies, the order in which components must be inserted, and collision-free assembly paths. Using this model as a technical core, we then leverage engineering design guidelines and efficient numerical techniques to optimize device designs. In a user study, which also highlights the challenges of designing such devices, we find our system to be effective in reducing the entry barriers faced by casual users in creating such devices. We further demonstrate the versatility of our approach by designing and fabricating three devices with diverse functionalities.

We present the first computational approach that can transform three-dimensional (3D) meshes, created by traditional modeling programs, directly into instructions for a computer-controlled knitting machine. Knitting machines are able to robustly and repeatably form knitted 3D surfaces from yarn but have many constraints on what they can fabricate. Given user-defined starting and ending points on an input mesh, our system incrementally builds a helix-free, quad-dominant mesh with uniform edge lengths, runs a tracing procedure over this mesh to generate a knitting path, and schedules the knitting instructions for this path in a way that is compatible with machine constraints. We demonstrate our approach on a wide range of 3D meshes.

We introduce a lightweight structure optimization approach for problems in which there is uncertainty in the force locations. Such uncertainty may arise due to force contact locations that change during use or are simply unknown a priori. Given an input 3D model, regions on its boundary where arbitrary normal forces may make contact, and a total force-magnitude budget, our algorithm generates a minimum weight 3D structure that withstands any force configuration capped by the budget. Our approach works by repeatedly finding the most critical force configuration and altering the internal structure accordingly. A key issue, however, is that the critical force configuration changes as the structure evolves, resulting in a significant computational challenge. To address this, we propose an efficient critical instant analysis approach. Combined with a reduced order formulation, our method provides a practical solution to the structural optimization problem. We demonstrate our method on a variety of models and validate it with mechanical tests.

RFID tags can be used to add inexpensive, wireless, batteryless sensing to objects. However, quickly and accurately estimating the state of an RFID tag is difficult. In this work, we show how to achieve low-latency manipulation and movement sensing with off-the-shelf RFID tags and readers. Our approach couples a probabilistic filtering layer with a monte-carlo-sampling-based interaction layer, preserving uncertainty in tag reads until they can be resolved in the context of interactions. This allows designers' code to reason about inputs at a high level. We demonstrate the effectiveness of our approach with a number of interactive objects, along with a library of components that can be combined to make new designs.

We introduce a new form of low-cost 3D printer to print interactive electromechanical objects with wound in place coils. At the heart of this printer is a mechanism for depositing wire within a five degree of freedom (5DOF) fused deposition modeling (FDM) 3D printer. Copper wire can be used with this mechanism to form coils which induce magnetic fields as a current is passed through them. Soft iron wire can additionally be used to form components with high magnetic permeability which are thus able to shape and direct these magnetic fields to where they are needed. When fabricated with structural plastic elements, this allows simple but complete custom electromagnetic devices to be 3D printed. As examples, we demonstrate the fabrication of a solenoid actuator for the arm of a Lucky Cat figurine, a 6-pole motor stepper stator, a reluctance motor rotor and a Ferrofluid display. In addition, we show how printed coils which generate small currents in response to user actions can be used as input sensors in interactive devices.

Industrial knitting machines can produce finely detailed, seamless, 3D surfaces quickly and without human intervention. However, the tools used to program them require detailed manipulation and understanding of low-level knitting operations. We present a compiler that can automatically turn assemblies of high-level shape primitives (tubes, sheets) into low-level machine instructions. These high-level shape primitives allow knit objects to be scheduled, scaled, and otherwise shaped in ways that require thousands of edits to low-level instructions. At the core of our compiler is a heuristic transfer planning algorithm for knit cycles, which we prove is both sound and complete. This algorithm enables the translation of high-level shaping and scheduling operations into needle-level operations. We show a wide range of examples produced with our compiler and demonstrate a basic visual design interface that uses our compiler as a backend.

We present a new type of 3D printer that can form precise, but soft and deformable 3D objects from layers of off-the-shelf fabric. Our printer employs an approach where a sheet of fabric forms each layer of a 3D object. The printer cuts this sheet along the 2D contour of the layer using a laser cutter and then bonds it to previously printed layers using a heat sensitive adhesive. Surrounding fabric in each layer is temporarily retained to provide a removable support structure for layers printed above it. This process is repeated to build up a 3D object layer by layer. Our printer is capable of automatically feeding two separate fabric types into a single print. This allows specially cut layers of conductive fabric to be embedded in our soft prints. Using this capability we demonstrate 3D models with touch sensing capability built into a soft print in one complete printing process, and a simple LED display making use of a conductive fabric coil for wireless power reception.

Light field displays allow viewers to see view-dependent 3D content as if looking through a window; however, existing work on light field display interaction is limited. Yet, they have the potential to parallel 2D pen and touch screen systems, which present a joint input and display surface for natural interaction. We propose a 4D display and interaction space using a dual-purpose lenslet array, which combines light field display and light field pen sensing, and allows us to estimate the 3D position and 2D orientation of the pen. This method is simple, fast (150Hz), with position accuracy of 2-3mm and precision of 0.2-0.6mm from 0-350mm away from the lenslet array, and orientation accuracy of 2 degrees and precision of 0.2-0.3 degrees within a 45 degree field of view. Further, we 3D print the lenslet array with embedded baffles to reduce out-of-bounds cross-talk, and use an optical relay to allow interaction behind the focal plane. We demonstrate our joint display/sensing system with interactive light field painting.

Abstract Increasingly, companies are creating product advertisements and catalog images using computer renderings of 3D scenes. A common goal for these companies is to create aesthetically appealing compositions that highlight objects of interest within the context of a scene. Unfortunately, this goal is challenging, not only due to the need to balance the trade‐off among aesthetic principles and design constraints, but also because of the huge search space induced by possible camera parameters, object placement, material choices, etc. Previous methods have investigated only optimization of camera parameters. In this paper, we develop a tool that starts from an initial scene description and a set of high‐level constraints provided by a stylist and then automatically generates an optimized scene whose 2D composition is improved. It does so by locally adjusting the 3D object transformations, surface materials, and camera parameters. The value of this tool is demonstrated in a variety of applications motivated by product catalogs, including rough layout refinement, detail image creation, home planning, cultural customization, and text inlay placement. Results of a perceptual study indicate that our system produces images preferable for product advertisement compared to a more traditional camera‐only optimization.

Abstract Traditional methods for creating dynamic objects and characters from static drawings involve careful tweaking of animation curves and/or simulation parameters. Sprite sheets offer a more drawing‐centric solution, but they do not encode timing information or the logic that determines how objects should transition between poses and cannot generalize outside the given drawings. We present an approach for creating dynamic sprites that leverages sprite sheets while addressing these limitations. In our system, artists create a drawing, deform it to specify a small number of example poses, and indicate which poses can be interpolated. To make the object move, we design a procedural simulation to navigate the pose manifold in response to external or user‐controlled forces. Powerful artistic control is achieved by allowing the artist to specify both the pose manifold and how it is navigated, while physics is leveraged to provide timing and generality. We used our method to create sprites with a range of different dynamic properties. Copyright © 2014 John Wiley & Sons, Ltd.

Traditional methods for creating dynamic objects and characters from static drawings involve careful tweaking of animation curves and/or simulation parameters. Sprite sheets offer a more drawing-centric solution, but they do not encode timing information or the logic that determines how objects should transition between poses and cannot generalize outside the given drawings. We present an approach for creating dynamic sprites that leverages sprite sheets while addressing these limitations. In our system, artists create a drawing, deform it to specify a small number of example poses, and indicate which poses can be interpolated. To make the object move, we design a procedural simulation to navigate the pose manifold in response to external or user-controlled forces. Powerful artistic control is achieved by allowing the artist to specify both the pose manifold and how it is navigated, while physics is leveraged to provide timing and generality. We used our method to create sprites with a range of different dynamic properties.

Abstract Physical simulation and other procedural methods are increasingly popular tools in interactive applications because they generate complex and reactive behaviors given only a few parameter settings. This automation accelerates initial implementation, but also introduces a need to tune the available parameters until the desired behaviors emerge. These adjustments are typically performed iteratively, with the designer repeatedly running—and interacting with—the procedural animation with different parameter settings. Such a process is inaccurate, time consuming, and requires deep understanding and intuition, as parameters often have complex, nonlinear effects. Instead, we propose that designers construct physics storyboards to accelerate the process of tuning interactive, procedural animations. Physics storyboards are collections of space‐time snapshots that highlight critical events and outcomes. They can be used to summarize the effects of parameter changes (without requiring the designer to perform extensive play‐testing); and—when augmented with designer‐provided evaluation functions—allow automatic parameter selection. We describe our implementation of this method, including how we use sampling to ensure that our automatically‐selected parameters generalize, and how we time‐warp user input to adapt it to changing parameters. We validate our implementation by using it to perform various design tasks in three example games.

Abstract In this paper, we present a continuous approach to ordering 2D images when compositing. Previous methods for stacking image layers require them to appear in a single (though possibly different) order at every point in the image. Our soft stacking approach removes this restriction — allowing layers to stack as if they were volumes of fog, appearing partially in front of and partially in back of other layers within the same pixel, and moving smoothly through other layers across the image. Our approach involves augmenting each pixel with stacking coefficients— a necessary and sufficient representation for sub‐pixel stacking complexity. These stacking coefficients arise naturally when considering sub‐pixel stacking complexity, much as continuous (alpha) transparency arises when considering sub‐pixel coverage complexity. While the number of stacking coefficients required to represent all possible sub‐pixel stacking arrangements is factorial in the number of layers in the stack, in many practical situations only a small subset of the stacking coefficients are nonzero. We use this sparsity as the basis of a prototype that allows artists to interactively paint stacking adjustments into composites. Additionally, we demonstrate how to generate optimally‐stacked images under a generalized notion of stacking consistency.

Given multiple time sequences with missing values, we propose DynaMMo which summarizes, compresses, and finds latent variables. The idea is to discover hidden variables and learn their dynamics, making our algorithm able to function even when there are missing values.We performed experiments on both real and synthetic datasets spanning several megabytes, including motion capture sequences and chlorine levels in drinking water. We show that our proposed DynaMMo method (a) can successfully learn the latent variables and their evolution; (b) can provide high compression for little loss of reconstruction accuracy; (c) can extract compact but powerful features for segmentation, interpretation, and forecasting; (d) has complexity linear on the duration of sequences.

In a conventional 2d painting or compositing program, graphical objects are stacked in a user-specified global order, as if each were printed on an image-sized sheet of transparent film. In this paper we show how to relax this restriction so that users can make stacking decisions on a per-overlap basis, as if the layers were pictures cut from a magazine. This allows for complex and visually exciting overlapping patterns, without painstaking layer-splitting, depth-value painting, region coloring, or mask-drawing. Instead, users are presented with a layers dialog which acts locally. Behind the scenes, we divide the image into overlap regions and track the ordering of layers in each region. We formalize this structure as a graph of stacking lists, define the set of orderings where layers do not interpenetrate as consistent, and prove that our local stacking operators are both correct and sufficient to reach any consistent stacking. We also provide a method for updating the local stacking when objects change shape or position due to user editing - this scheme prevents layer updates from producing undesired intersections. Our method extends trivially to both animation compositing and local visibility adjustment in depth-peeled 3d scenes; the latter of which allows for the creation of impossible figures which can be viewed and manipulated in real-time.

We present an image editing program which allows artists to paint in the gradient domain with real-time feedback on megapixel-sized images. Along with a pedestrian, though powerful, gradient-painting brush and gradient-clone tool, we introduce an edge brush designed for edge selection and replay. These brushes, coupled with special blending modes, allow users to accomplish global lighting and contrast adjustments using only local image manipulations --- e.g. strengthening a given edge or removing a shadow boundary. Such operations would be tedious in a conventional intensity-based paint program and hard for users to get right in the gradient domain without real-time feedback. The core of our paint program is a simple-to-implement GPU multigrid method which allows integration of megapixel-sized full-color gradient fields at over 20 frames per second on modest hardware. By way of evaluation, we present example images produced with our program and characterize the iteration time and convergence rate of our integration method.

In game environments, animated character motion must rapidly adapt to changes in player input - for example, if a directional signal from the player's gamepad is not incorporated into the character's trajectory immediately, the character may blithely run off a ledge. Traditional schemes for data-driven character animation lack the split-second reactivity required for this direct control; while they can be made to work, motion artifacts will result. We describe an on-line character animation controller that assembles a motion stream from short motion fragments, choosing each fragment based on current player input and the previous fragment. By adding a simple model of player behavior we are able to improve an existing reinforcement learning method for precalculating good fragment choices. We demonstrate the efficacy of our model by comparing the animation selected by our new controller to that selected by existing methods and to the optimal selection, given knowledge of the entire path. This comparison is performed over real-world data collected from a game prototype. Finally, we provide results indicating that occasional low-quality transitions between motion segments are crucial to high-quality on-line motion generation; this is an important result for others crafting animation systems for directly-controlled characters, as it argues against the common practice of transition thresholding.

Newton is a library-based CAD tool with an analytical synthesis engine which has been developed to support the direct synthesis of the physical design and an electromechanically equivalent model of RF-MEMS resonators based on process parameters and performance metrics. Newton provides accuracy comparable to finite element analysis while requiring a fraction of the computation and design time. A comparison of results from synthesis with Newton, design with FEA, and test results from fabricated devices is presented.