Large image generation and vision models, combined with differentiable rendering technologies, have become powerful tools for generating paths that can be drawn or painted by a robot. However, these tools often overlook the intrinsic physicality of the human drawing/writing act, which is usually executed with skillful hand/arm gestures. Taking this into account is important for the visual aesthetics of the results and for the development of closer and more intuitive artist-robot collaboration scenarios. We present a method that bridges this gap by enabling gradient-based optimization of natural human-like motions guided by cost functions defined in image space. To this end, we use the sigma-lognormal model of human hand/arm movements, with an adaptation that enables its use in conjunction with a differentiable vector graphics (DiffVG) renderer. We demonstrate how this pipeline can be used to generate feasible trajectories for a robot by combining image-driven objectives with a minimum-time smoothing criterion. We demonstrate applications with generation and robotic reproduction of synthetic graffiti as well as image abstraction.

Abstract We present an adaptive, semantics‐based abstraction approach that balances aesthetic quality and structural coherence within the practical constraints of robotic painting. We apply panoptic segmentation with color‐based over‐segmentation to partition images into meaningful regions aligned with semantic objects, while providing flexible abstraction levels. Automatic parameter selection for region merging is enabled by semantic saliency maps, derived from Out‐of‐Distribution segmentation techniques in combination with machine learning methods for feature detection. This preserves the boundaries of salient objects while simplifying less prominent regions. A graph‐based community detection step further refines the abstraction by grouping regions according to local connectivity and semantic coherence. The runtime of our method outperforms optimization‐based image vectorization methods, enabling the efficient generation of multiple abstraction levels that can serve as hierarchical layers for robotic painting. We demonstrate the quality of our method by showing abstraction results, robotic paintings with the e‐David robot, and a comparison to other abstraction methods.

We integrate smoothing B-splines into a standard differentiable vector graphics (DiffVG) pipeline through linear mapping, and show how this can be used to generate smooth and arbitrarily long paths within image-based deep learning systems. We take advantage of derivative-based smoothing costs for parametric control of fidelity vs. simplicity tradeoffs, while also enabling stylization control in geometric and image spaces. The proposed pipeline is compatible with recent vector graphics generation and vectorization methods. We demonstrate the versatility of our approach with four applications aimed at the generation of stylized vector graphics: stylized space-filling path generation, stroke-based image abstraction, closed-area image abstraction, and stylized text generation.

Until recently, Virtual Reality (VR) applications relied on controllers to enable user interaction in virtual environments. With advances in tracking technology, HMDs are now able to track the movements of users’ hands in real-time with significantly greater accuracy, allowing us to interact with the digital world directly with our hands. However, it is not entirely clear how hand tracking affects users’ performance. In this study, we investigate user performance using an in-game analytics-based assessment methodology for a VR memory puzzle task. We conducted a within-subjects experiment with 30 participants in three conditions: 1- Hand-tracking, 2-Controller Without Haptics, and 3- Controller With Haptics. In all our measurements (correct order and pattern, correct pattern only, and trial completion) except for the initial selection time, participants performed best with hand tracking. The use of controllers with haptics did not outperform controllers without haptics in most measures, possibly because other feedback cues compensated for the lack of haptics. This study helps us better understand the three selected interactivity methods when used in VR, as well as the importance of naturalistic experience in interaction design.

In this article, we present a survey of mixed-initiative methods for the creation of content for video games. We also propose a definition of what mixed initiative implies, as the term lacks a clear specification. The survey includes works not directly aimed at video games but which create content that can potentially be used in games, such as art programs utilizing mixed initiative. Furthermore, we highlight research areas that overlap wholly or partly with mixed initiative, such as casual creators, explainable artificial intelligence, or interactive evolutionary computation. We examine these and several other topics in the context of mixed initiative. Finally, we provide a catalogue of typical techniques and challenges connected with mixed initiative before considering future directions.

We develop a method to automatically segment a font’s glyphs into a set of overlapping and intersecting strokes with the aim of generating artistic stylizations. The segmentation method relies on a geometric analysis of the glyph’s outline, its interior, and the surrounding areas and is grounded in perceptually informed principles and measures. Our method does not require training data or templates and applies to glyphs in a large variety of input languages, writing systems, and styles. It uses the medial axis, curvilinear shape features that specify convex and concave outline parts, links that connect concavities, and seven junction types. We show that the resulting decomposition in strokes can be used to create variations, stylizations, and animations in different artistic or design-oriented styles while remaining recognizably similar to the input font.

Based on the Method of Loci, the following experiment compares the effect of two different virtual environments on participants' memory performance. The primary task consists of remembering a sequence of random playing cards. Each virtual environment is based on a different architectural style with a different layout. One is inspired by a Palladian style architecture, and the other by a Modern curved architecture.

MOTIVATION: The 3D structure of chromatin in the nucleus is important for gene expression and regulation. Chromosome conformation capture techniques, such as Hi-C, generate large amounts of data showing interaction points on the genome but these are hard to interpret using standard tools. RESULTS: We have developed CSynth, an interactive 3D genome browser and real-time chromatin restraint-based modeller to visualize models of any chromosome conformation capture (3C) data. Unlike other modelling systems, CSynth allows dynamic interaction with the modelling parameters to allow experimentation and effects on the model. It also allows comparison of models generated from data in different tissues/cell states and the results of third-party 3D modelling outputs. In addition, we include an option to view and manipulate these complicated structures using Virtual Reality (VR) so scientists can immerse themselves in the models for further understanding. This VR component has also proven to be a valuable teaching and a public engagement tool. AVAILABILITYAND IMPLEMENTATION: CSynth is web based and available to use at csynth.org. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Abstract We present a review of methods for procedurally generating the morphology of virtual creatures. We include a range of methods, with the main groups being from ALife over art to video games. Even though at times these groups overlap, for clarity we have kept this distinction. By including the word virtual, we mean that we focus on methods for simulation in silico, and not physical robots. We also include a historical perspective, with information on methods such as cellular automata, L‐systems and a focus on earlier pioneers in the field.

While the games industry is moving towards procedural content generation (PCG) with tools available under popular platforms such as Unreal, Unity or Houdini, and video game titles like No Man’s Sky and Horizon Zero Dawn taking advantage of PCG, the gap between academia and industry is as wide as it has ever been, in terms of communication and sharing methods. The authors have worked on both sides of this gap and in an effort to shorten it and increase the synergy between the two sectors have identified three design pillars for PCG using mixed-initiative interfaces. The three pillars are respect designer control, respect the creative process and respect existing work processes. Respecting designer control is about creating a tool that gives enough control to bring out the designer’s vision. Respecting the creative process concerns itself with having a feedback loop that is short enough, that the creative process is not disturbed. Respecting existing work processes means that a PCG tool should plug in easily to existing asset pipelines. As academics and communicators, it is surprising that publications often do not describe ways for developers to use our work or lack considerations for how a piece of work might fit into existing content pipelines.

In this paper two swarm intelligence algorithms are used, the first leading the “attention” of the swarm and the latter responsible for the tracing mechanism. The attention mechanism is coordinated by agents of Stochastic Diffusion Search where they selectively attend to areas of a digital canvas (with line drawings) which contains (sharper) corners. Once the swarm's attention is drawn to the line of interest with a sharp corner, the corresponding line segment is fed into the tracing algorithm, Dispersive Flies Optimisation which “consumes” the input in order to generate a “swarmic sketch” of the input line. The sketching process is the result of the “flies” leaving traces of their movements on the digital canvas which are then revisited repeatedly in an attempt to re-sketch the traces they left. This cyclic process is then introduced in the context of autopoiesis, where the philosophical aspects of the autopoietic artist are discussed. The autopoetic artist is described in two modalities: gluttonous and contented. In the Gluttonous Autopoietic Artist mode, by iteratively focussing on areas-of-rich-complexity, as the decoding process of the input sketch unfolds, it leads to a less complex structure which ultimately results in an empty canvas; therein reifying the artwork's “death”. In the Contented Autopoietic Artist mode, by refocussing the autopoietic artist's reflections on “meaning” onto different constitutive elements, and modifying her reconstitution, different behaviours of autopoietic creativity can be induced and therefore, the autopoietic processes become less likely to fade away and more open-ended in their creative endeavour.

We present an approach to generate rapid and fluid drawing movements on a compliant Baxter robot, by taking advantage of the kinematic redundancy and torque control capabilities of the robot. We concentrate on the task of reproducing graffiti-stylised letter-forms with a marker. For this purpose, we exploit a compact lognormal-stroke based representation of movement to generate natural drawing trajectories. An Expectation-Maximisation (EM) algorithm is used to iteratively improve tracking performance with low gain feedback control. The resulting system captures the aesthetic and dynamic features of the style under investigation and permits its reproduction with a compliant controller that is safe for users surrounding the robot.

We study the use of the generative systems known as computational ecosystems to convey artistic and narrative aims. These are virtual worlds running on computers, composed of agents that trade units of energy and emulate cycles of life and behaviors adapted from biological life forms. In this article we propose a conceptual framework in order to understand these systems, which are involved in processes of authorship and interpretation that this investigation analyzes in order to identify critical instruments for artistic exploration. We formulate a model of narrative that we call system stories (after Mitchell Whitelaw), characterized by the dynamic network of material and conceptual processes that define these artefacts. They account for narrative constellations with multiple agencies from which meaning and messages emerge. Finally, we present three case studies to explore the potential of this model within an artistic and generative domain, arguing that this understanding expands and enriches the palette of the language of these systems.

A novel biological software approach to define and evolve 3D computer art forms is described based on a re-implementation of the FormGrow system produced by Latham and Todd at IBM in the early 1990’s. This original work is extended by using DNA sequences as the input to generate complex organic-like forms. The translation of the DNA data to 3D graphic form is performed by two contrasting processes, one intuitive and one informed by the biochemistry. The former involves the development of novel, but simple, look-up tables to generate a code list of functions such as the twisting, bending, stacking, and scaling and their associated parametric values such as angle and scale. The latter involves an analysis of the biochemical properties of the proteins encoded by genes in DNA, which are used to control the parameters of a fixed FormGrow structure. The resulting 3D data sets are then rendered using conventional techniques to create visually appealing art forms. The system maps DNA data into an alternative multi-dimensional space with strong graphic visual features such as intricate branching structures and complex folding. The potential use in scientific visualisation is illustrated by two examples. Forms representing the sickle cell anaemia mutation demonstrate how a point mutation can have a dramatic effect. An animation illustrating the divergent evolution of two proteins with a common ancestor provides a compelling view of an evolutionary process lost in millions of years of natural history.

We introduce the notion of the medial scaffold, a hierarchical organization of the medial axis of a 3D shape in the form of a graph constructed from special medial curves connecting special medial points. A key advantage of the scaffold is that it captures the qualitative aspects of shape in a hierarchical and tightly condensed representation. We propose an efficient and exact method for computing the medial scaffold based on a notion of propagation along the scaffold itself, starting from initial sources of the flow and constructing the scaffold during the propagation. We examine this method specifically in the context of an unorganized cloud of points in 3D, e.g., as obtained from laser range finders, which typically involve hundreds of thousands of points, but the ideas are generalizable to data arising from geometrically described surface patches. The computational bottleneck in the propagation-based scheme is in finding the initial sources of the flow. We thus present several ideas to avoid the unnecessary consideration of pairs of points which cannot possibly form a medial point source, such as the "visibility" of a point from another given a third point and the interaction of clusters of points. An application of using the medial scaffold for the representation of point samplings of real-life objects is also illustrated.

A prototype system for the automatic evolution of biomimetic structures using structural automata is described and its utility for generating digital sculpture is demonstrated. Sculptures are generated from a primordial shape which is represented in terms of a triangular mesh and sculpture is created by extending the original surface using tetrahedral structural elements. Recursively applicable rules or equivalently, automata, are defined which allow the sculptor to generate a volumetric scaffold from the original surface. This scaffold is generated using the stated rules for inserting and connecting together the tetrahedral elements. The software is operated as a generative process where sculptures are grown from an original triangular surface mesh as a sequence of layers. Each layer is created as a 2-step process. In step 1, we populate the surface with tetrahedral structures where the base of each tetrahedron coincides with a surface triangle. Step 2 re-triangulates the apexes of the tetrahedra from step 1 creating an offset and deformed version of the original surface mesh. The sculptor has artistic control of the process at all points and may assign or change rules to generate different biomimetic behaviors, i.e., structures which tend to replicate natural phenomena.

The Mid-Ocean Studio, Brown University's SHAPE lab, and Goldsmiths College are collaborating on a prototype system for the automatic evolution of biomimetic sculpture using structural automata. This collaboration is resulting in effective, computerized means to autogenerate large, increasingly complex works of art, and allowing for a long-anticipated development of the desire to create works that reflect and respond to the environment they are in. We propose to create an installation that allows visitors to a site at Goldsmiths College to experience and interact with the development of our structures.

The shock scaffold is a hierarchical organization of the medial axis in 3D consisting of special medial points and curves connecting these points, thereby forming a geometric directed graph, which is key in applications such as object recognition. In this paper we describe a method for computing the shock scaffold of realistic datasets, which involve tens or hundreds of thousands of points, in a practical time frame. Our approach is based on propagation along the scaffold from initial sources of flow by considering pairs of input points. We present seven principles which avoid the consideration of those pairs of points which cannot possibly lead to a shock flow; they involve: (i) the "visibility" of a point from another, (ii) the clustering of points, (iii) the visibility of a cluster from another, (iv) the convex hull of a cluster, (v) the vertices of such convex hulls as "virtual" points, (vi) a multi-resolution framework, and, finally, (vii) a search strategy organized in layers.

The usefulness of the 3D medial axis (MA) is dependent on both the availability of accurate and stable methods for computing individual MA points and on schemes for deriving the local structure and connectivity among these points. We review the leading approaches for computing the 3D MA and the different fields of application for this representation of 3D shapes. We also present the shock scaffold which combines the advantages of exact bisector computations used in computational geometry on the one hand, and the local nature of propagation-based algorithms on the other, but without the computational complexity, connectivity, added dimensionality, and post processing issues commonly found in other approaches.

REALISE was designed to extract from sequences of images, acquired with a moving camera, the information necessary for determining the 3D (CAD-like) structure of a real-life scene together with information about the radiometric signatures of surfaces bounding the extracted 3D objects (e.g. reflectance behaviour). The retrieved information is then integrated in a virtual reality (VR) software environment. The R&D work is been performed principally in the following areas of computer vision and computer graphics: structure from motion, recovery of geometries, recovery of photometric and texture information, highly realistic rendering on the basis of empirically-based reflectance models, and the design and development of improved rendering processes together with a new VR system. Beside this innovative R&D work another key aspect of REALISE is to have computer vision & computer graphics cooperate to produce realistic 3D data efficiently.

The problems of segmenting a noisy intensity image and tracking a nonrigid object in the plane are discussed. In evaluating these problems, a technique based on an active contour model commonly called a snake is examined. The technique is applied to cell locomotion and tracking studies. The snake permits both the segmentation and tracking problems to be simultaneously solved in constrained cases. A detailed analysis of the snake model, emphasizing its limitations and shortcomings, is presented, and improvements to the original description of the model are proposed. Problems of convergence of the optimization scheme are considered. In particular, an improved terminating criterion for the optimization scheme that is based on topographic features of the graph of the intensity image is proposed. Hierarchical filtering methods, as well as a continuation method based on a discrete sale-space representation, are discussed. Results for both segmentation and tracking are presented. Possible failures of the method are discussed.>

A method for shape description of planar objects that integrates both region and boundary features is presented. The method is an implementation of a 2D dynamic grassfire that relies on a distance surface on which elastic contours minimize an energy function. The method is based on an active contour model. Numerous implementation aspects of the shape description method were optimized. A Euclidean metric was used for optimal accuracy, and the active contour model permits bypassing some of the discretization limitations inherent in using a digital grid. Noise filtering was performed on the basis of both contour feature measures and region measures, that is, curvature extremum significance and ridge support, respectively, to obtain robust shape descriptors. Other improvements and variations of the algorithmic implementation are proposed.>