Large image generation and vision models, combined with differentiable rendering technologies, have become powerful tools for generating paths that can be drawn or painted by a robot. However, these tools often overlook the intrinsic physicality of the human drawing/writing act, which is usually executed with skillful hand/arm gestures. Taking this into account is important for the visual aesthetics of the results and for the development of closer and more intuitive artist-robot collaboration scenarios. We present a method that bridges this gap by enabling gradient-based optimization of natural human-like motions guided by cost functions defined in image space. To this end, we use the sigma-lognormal model of human hand/arm movements, with an adaptation that enables its use in conjunction with a differentiable vector graphics (DiffVG) renderer. We demonstrate how this pipeline can be used to generate feasible trajectories for a robot by combining image-driven objectives with a minimum-time smoothing criterion. We demonstrate applications with generation and robotic reproduction of synthetic graffiti as well as image abstraction.

Abstract We present an adaptive, semantics‐based abstraction approach that balances aesthetic quality and structural coherence within the practical constraints of robotic painting. We apply panoptic segmentation with color‐based over‐segmentation to partition images into meaningful regions aligned with semantic objects, while providing flexible abstraction levels. Automatic parameter selection for region merging is enabled by semantic saliency maps, derived from Out‐of‐Distribution segmentation techniques in combination with machine learning methods for feature detection. This preserves the boundaries of salient objects while simplifying less prominent regions. A graph‐based community detection step further refines the abstraction by grouping regions according to local connectivity and semantic coherence. The runtime of our method outperforms optimization‐based image vectorization methods, enabling the efficient generation of multiple abstraction levels that can serve as hierarchical layers for robotic painting. We demonstrate the quality of our method by showing abstraction results, robotic paintings with the e‐David robot, and a comparison to other abstraction methods.

We integrate smoothing B-splines into a standard differentiable vector graphics (DiffVG) pipeline through linear mapping, and show how this can be used to generate smooth and arbitrarily long paths within image-based deep learning systems. We take advantage of derivative-based smoothing costs for parametric control of fidelity vs. simplicity tradeoffs, while also enabling stylization control in geometric and image spaces. The proposed pipeline is compatible with recent vector graphics generation and vectorization methods. We demonstrate the versatility of our approach with four applications aimed at the generation of stylized vector graphics: stylized space-filling path generation, stroke-based image abstraction, closed-area image abstraction, and stylized text generation.

A word-as-image is a semantic typography technique where a word illustration presents a visualization of the meaning of the word, while also preserving its readability. We present a method to create word-as-image illustrations automatically. This task is highly challenging as it requires semantic understanding of the word and a creative idea of where and how to depict these semantics in a visually pleasing and legible manner. We rely on the remarkable ability of recent large pretrained language-vision models to distill textual concepts visually. We target simple, concise, black-and-white designs that convey the semantics clearly. We deliberately do not change the color or texture of the letters and do not use embellishments. Our method optimizes the outline of each letter to convey the desired concept, guided by a pretrained Stable Diffusion model. We incorporate additional loss terms to ensure the legibility of the text and the preservation of the style of the font. We show high quality and engaging results on numerous examples and compare to alternative techniques. Code and demo will be available at our project page.

We develop a method to automatically segment a font’s glyphs into a set of overlapping and intersecting strokes with the aim of generating artistic stylizations. The segmentation method relies on a geometric analysis of the glyph’s outline, its interior, and the surrounding areas and is grounded in perceptually informed principles and measures. Our method does not require training data or templates and applies to glyphs in a large variety of input languages, writing systems, and styles. It uses the medial axis, curvilinear shape features that specify convex and concave outline parts, links that connect concavities, and seven junction types. We show that the resulting decomposition in strokes can be used to create variations, stylizations, and animations in different artistic or design-oriented styles while remaining recognizably similar to the input font.

We present an approach to generate rapid and fluid drawing movements on a compliant Baxter robot, by taking advantage of the kinematic redundancy and torque control capabilities of the robot. We concentrate on the task of reproducing graffiti-stylised letter-forms with a marker. For this purpose, we exploit a compact lognormal-stroke based representation of movement to generate natural drawing trajectories. An Expectation-Maximisation (EM) algorithm is used to iteratively improve tracking performance with low gain feedback control. The resulting system captures the aesthetic and dynamic features of the style under investigation and permits its reproduction with a compliant controller that is safe for users surrounding the robot.