Similar to the human’s multiple perception system, the robot can also benefit from cross-modal learning. The connection between visual input and tactile perception is potentially important for automated operations. However, establishing an algorithmic mapping of the visual modal to the tactile modal is a challenging task. In this work, we use the framework of GANs to propose a cross-modal imaging method for estimating the tactile physical properties values based on the Gramian Summation Angular Field, combined with visual-tactile embedding cluster fusion and feature matching methods. The approach estimates 15 tactile properties. In particular, the task attempts to predict unknown surface properties based on "learned knowledge". Our results surpass the state-of-the-art approach on most tactile dimensions of the publicly available dataset. Additionally, we conduct a robustness study to verify the effect of angle and complex environment on the network prediction performance.

Abstract We propose a method for converting a single image of a transparent object into multi‐view photo that enables users observing the object from multiple new angles, without inputting any 3D shape. The complex light paths formed by refraction and reflection makes it challenging to compute the lighting effects of transparent objects from a new angle. We construct an encoder–decoder network for normal reconstruction and texture extraction, which enables synthesizing novel views of transparent object from a set of new views and new environment maps using only one RGB image. By simultaneously considering the optical transmission and perspective variation, our network learns the characteristics of optical transmission and the change of perspective as guidance to the conversion from RGB colours to surface normals. A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation. We test our method using 3D objects within and without our training data, including real 3D objects that exists in our lab, and completely new environment maps that we take using our phones. The results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single‐view image.

Reconstructing a 3D human body mesh from a monocular image is a challenging inverse problem because of occlusion and complicated human articulations. Recent deep learning-based methods have made significant progress in single-image human reconstruction. Most of these works are either model-based methods or model-free methods. However, model-based methods always suffer detail losses due to the limited parameter space, and model-free methods are hard to directly recover satisfactory results from images due to the use of a shared global feature for all vertices and the domain gap between 2D regular images and 3D irregular meshes. To resolve these issues, we propose a hybrid model, which combines the advantages of both model based approach and model-free approach to estimate a 3D human mesh in a coarse-to fine manner. Initially, we utilize a convolutional neural network (CNN) to estimate the parameters of a Skinned Multi-Person Linear Model (SMPL), which allows us to generate a coarse human mesh. After that, the vertex coordinates of the coarse human mesh are further refined by a graph convolutional neural network (GCN). Unlike previous GCN-based methods, whose vertex coordinates are recovered from a shared global feature, we propose a LOcal CorRespondence-Aware (LOCRA) module to extract local special features for each vertex. To make the local features related to the human pose, we also add a keypoint-related loss to supervise the training process of the LOCRA module. Experiments demonstrate that our hybrid model with the LOCRA module outperforms existing methods on multiple public benchmarks.

Non-contact measurement of human body height can be very difficult under some circumstances.In this paper we address the problem of accurately estimating the height of a person with arbitrary postures from a single depth image. By introducing a novel part-based intermediate representation plus a four-stage increasingly complex deep neural network, we manage to achieve significantly higher accuracy than previous methods. We first describe the human body in the form of a segmentation of human torso as four nearly rigid parts and then predict their lengths respectively by 3 CNNs. Instead of directly adding the lengths of these parts together, we further construct another independent developing CNN that combines the intermediate representation, part lengths and depth information together to finally predict the body height results.Here we develop an increasingly complex network architecture and adopt a hybrid pooling to optimize training process. To the best of our knowledge, this is the first method that estimates height only from a single depth image. In experiments our average accuracy reaches at 99.1% for people in various positions and postures.

Sketch recognition using deep neural networks have become a recent trend. However, traditional pixel (image) based convolutional neural networks show poor recognizing performance on low resolution (LR) sketch image due to the loss of image details. To solve this problem, we propose a joint pixel and point convolutional neural network for LR sketch image recognition. The network, equipped with both image convolution and point convolution, can simultaneously handle both the image and point representation of sketches. Furthermore, we propose a hybrid classifier, a corresponding loss function, and a training scheme to better extract features for recognition. Experimental results show that our method outperforms state-of-art deep neural networks.

We present a novel sketch-based system for generating digital bas-relief sculptures. All existing computational methods for generating digital bas-reliefs first require the input of a three-dimensional (3D) scene, thus preventing artists from freely creating or exploring designs when 3D data are not available. Motivated by this limitation, we propose a generative adversarial network (GAN)-based sketch modeling system for generating digital bas-reliefs from freehand user sketches (see Figure 1, 5). The basic tool underpinning the interface is a conditional GAN (cGAN) that digitally learns a functional map from a contour image to a 3D model for any given viewpoint of the corresponding bas-relief model. When using our system for designing bas-reliefs, the user only needs to draw 2D sketch lines without having to designate any additional hints on the lines. The interface returns bas-relief results in interactive time (500 ms per bas-relief on average). We tested the quality and robustness of our approach with extensive and comprehensive experiments. By carefully analyzing the results, we verified that our system can faithfully reconstruct bas-reliefs from a test dataset and can generate completely new reliefs from raw amateur sketches.

With the increasing gap between transistor feature size and lithography manufacturing capability, the detection of lithography hotspots becomes a key stage of physical verification flow to enhance manufacturing yield. Although machine learning approaches are distinguished for their high detection efficiency, they still suffer from problems such as large-scale layout and class imbalance. In this article, we develop a hotspot detection model based on machine learning with high performance. In the proposed model, we first apply an Fast Fourier Transform--based feature extraction method that can compress large-scale layout to a multi-dimensional representation with much smaller size while preserving the discriminative layout pattern information to improve the detection efficiency. Second, addressing the class imbalance problem, we propose a new technique called imbalanced learning rate and embed it into the convolutional neural network model to further reduce false alarms without accuracy decay. Compared with the results of current state-of-the-art approaches on ICCAD 2012 Contest benchmarks, our proposed model can achieve better solutions in many evaluation metrics, including the official metrics.

Freehand sketches are an important medium for expressing and communicating ideas. However creating a meaningful and understandable sketch drawing is not always an easy task especially for unskillful users. Existing methods for rendering 3D shape into line drawings such as Suggestive Contours, only consider the geometry-dependent and view-dependent information thus leads to over-regular or over-perfect results which doesn't look like a human freehand drawing. For this challenge we address the problem of producing freehand line drawing sketches from a 3D object under a given viewpoint automatically. The core solution here is a recurrent generative deep neural network, which learns a functional mapping from the suggestive contours of a 3D shape to a more abstract sketch representation. We drop the encoder of the generator, i.e., use only a decoder to achieve better stability of the sketch structure. Users can tune the level of freehand style of the generated sketches by changing a single parameter. Experiments show that our results are expressive enough to faithfully describe the input shape and at the same time be with the style of freehand drawings created by a real human. We also perform a comparative user study to verify the quality and style of generated sketch results over existing methods. We also retrain our network using several different mingled dataset to test the extendibility of our method for this particular application. As far as our knowledge this work is the first research effort to automate the generation of human-like freehand sketches directly from 3D shapes.

Decorative patterns are observed in many forms of art, typically enriching the visual aspect of otherwise simple shapes. Such patterns are especially difficult to create, as they often exhibit intricate structural details and at the same time have to precisely match the size and shape of the underlying geometry. In the field of Computer Graphics, several approaches have been proposed to automatically synthesize a decorative pattern along a curve, from an example. This empowers non expert users with a simple brush metaphor, allowing them to easily paint complex structured decorations. We extend this idea to the space of design and fabrication. The major challenge is to properly account for the topology of the produced patterns. In particular, our technique ensures that synthesized patterns will be made of exactly one connected component, so that once printed they form a single object. To achieve this goal we propose a two steps synthesis process, first synthesizing the topology of the pattern and later synthesizing its exact geometry. We introduce topology descriptors that efficiently capture the topology of the pattern synthesized so far. We propose several applications of our method, from designing objects using synthesized patterns along curves and within rectangles, to the decoration of surfaces with a dedicated smooth frame interpolation. Using our technique, designers paint structured patterns that can be fabricated into solid, tangible objects, creating unusual and surprising designs of lamps, chairs and laces from examples.

Abstract Many algorithms in Computer Graphics require to synthesize a pattern along a curve. This is for instance the case with line stylization, to decorate objects with elaborate patterns (chains, laces, scratches), or to synthesize curvilinear features such as mountain ridges, rivers or roads. We describe a simple yet effective method for this problem. Our method addresses the main challenge of maintaining the continuity of the pattern while following the curve. It allows some freedom to the synthesized pattern: It may locally diverge from the curve so as to allow for a more natural global result. This also lets the pattern escape areas of overlaps or fold‐overs. This makes our method particularly well suited to structured, detailed patterns following complex curves. Our synthesizer copies tilted pieces of the exemplar along the curve, following its orientation. The result is optimized through a shortest path search, with dynamic programming. We speed up the process by an efficient parallel implementation. Finally, since discontinuities may always remain we propose an optional post‐processing step optimally deforming neighboring pieces to smooth the transitions.

Revealing the sketching sequence of a line drawing can be visually intriguing and used for video-based storytelling. Typically this is enabled based on tedious recording of artists' drawing process. We demonstrate that it is often possible to estimate a reasonable drawing order from a static line drawing with clearly defined shape geometry, which looks plausible to a human viewer. We map the key principles of drawing order from drawing cognition to computational procedures in our framework. Our system produces plausible animated constructions of input line drawings, with no or little user intervention. We test our algorithm on a range of input sketches, with varying degree of complexity and structure, and evaluate the results via a user study. We also present applications to gesture drawing synthesis and drawing animation creation especially in the context of video scribing.

We present an easy-to-use image retouching technique for realistic reshaping of human bodies in a single image. A model-based approach is taken by integrating a 3D whole-body morphable model into the reshaping process to achieve globally consistent editing effects. A novel body-aware image warping approach is introduced to reliably transfer the reshaping effects from the model to the image, even under moderate fitting errors. Thanks to the parametric nature of the model, our technique parameterizes the degree of reshaping by a small set of semantic attributes, such as weight and height. It allows easy creation of desired reshaping effects by changing the full-body attributes, while producing visually pleasing results even for loosely-dressed humans in casual photographs with a variety of poses and shapes.

The essence of a 3D shape can often be well captured by its salient feature curves. In this paper, we explore the use of salient curves in synthesizing intuitive, shape-revealing textures on surfaces. Our texture synthesis is guided by two principles: matching the direction of the texture patterns to those of the salient curves, and aligning the prominent feature lines in the texture to the salient curves exactly. We have observed that textures synthesized by these principles not only fit naturally to the surface geometry, but also visually reveal, even reinforce, the shape's essential characteristics. We call these feature-aligned shape texturing . Our technique is fully automatic, and introduces two novel technical components in vector-field-guided texture synthesis: an algorithm that orients the salient curves on a surface for constrained vector field generation, and a feature-to-feature texture optimization.