3D generation methods have shown visually compelling results powered by diffusion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We condition the visual texture generation on high-resolution tactile normals and guide the patch-based tactile texture refinement with a customized TextureDreambooth. We further present a multi-part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities of vision and touch.

Deep generative models have various content creation applications such as graphic design, e-commerce, and virtual try-on. However, current works mainly focus on synthesizing realistic visual outputs, often ignoring other sensory modalities, such as touch, which limits physical interaction with users. In this work, we leverage deep generative models to create a multi-sensory experience where users can touch and see the synthesized object when sliding their fingers on a haptic surface. The main challenges lie in the significant scale discrepancy between vision and touch sensing and the lack of explicit mapping from touch sensing data to a haptic rendering device. To bridge this gap, we collect high-resolution tactile data with a GelSight sensor and create a new visuotactile clothing dataset. We then develop a conditional generative model that synthesizes both visual and tactile outputs from a single sketch. We evaluate our method regarding image quality and tactile rendering accuracy. Finally, we introduce a pipeline to render high-quality visual and tactile outputs on an electroadhesion-based haptic device for an immersive experience, allowing for challenging materials and editable sketch inputs.

The ability to simultaneously distinguish objects, materials, and their associated physical properties is one fundamental function of the sense of touch. Recent advances in the development of tactile sensors and machine learning techniques allow more accurate and complex modelling of robotic tactile sensations. However, many state-of-the-art (SotA) approaches focus solely on constructing black-box models to achieve ever higher classification accuracy and fail to adapt across sensors with unique spatial-temporal data formats. In this work, we propose an Explainable and Sensor-Adaptable Recurrent Networks (ExSARN) model for tactile texture representation. The ExSARN model consists of a two-stage recurrent networks fed by a sensor-specific header network. The first stage recurrent network emulates our human touch receptors and decouples sensor-specific tactile sensations into different frequency response bands, while the second stage codes the overall temporal signature as a variational recurrent autoen-coder. We infuse the latent representation with ternary labels to qualitatively represent texture properties (e.g. roughness and stiffness), which facilitates representation learning and provide explainability to the latent space. The ExSARN model is tested on texture datasets collected with two different tactile sensors. Our results show that the proposed model not only achieves higher accuracy, but also provides adaptability across sensors with different sampling frequencies and data formats. The addition of the crudely obtained qualitative property labels offers a practical approach to enhance the interpretability of the latent space, facilitate property inference on unseen materials, and improve the overall performance of the model.

The sense of touch is an essential sensing modality for a robot to interact with the environment as it provides rich and multimodal sensory information upon contact. It enriches the perceptual understanding of the environment and closes the loop for action generation. One fundamental area of perception that touch dominates over other sensing modalities, is the understanding of the materials that it interacts with, for example, glass versus plastic. However, unlike the senses of vision and audition which have standardized data format, the format for tactile data is vastly dictated by the sensor manufacturer, which makes it difficult for large-scale learning on data collected from heterogeneous sensors, limiting the usefulness of publicly available tactile datasets. This paper investigates the joint learnability of data collected from two tactile sensors performing a touch sequence on some common materials. We propose a supervised recurrent autoencoder framework to perform joint material classification task to improve the training effectiveness. The framework is implemented and tested on the two sets of tactile data collected in sliding motion on 20 material textures using the iCub RoboSkin tactile sensors and the SynTouch BioTac sensor respectively. Our results show that the learning efficiency and accuracy improve for both datasets through the joint learning as compared to independent dataset training. This suggests the usefulness for large-scale open tactile datasets sharing with different sensors.

Touch is arguably the most important sensing modality in physical interactions. However, tactile sensing has been largely under-explored in robotics applications owing to the complexity in making perceptual inferences until the recent advancements in machine learning or deep learning in particular. Touch perception is strongly influenced by both its temporal dimension similar to audition and its spatial dimension similar to vision. While spatial cues can be learned episodically, temporal cues compete against the system's re-sponse/reaction time to provide accurate inferences. In this paper, we propose a fast tactile-based texture classification framework which makes use of the spiking neural network to learn from the neural coding of the conventional tactile sensor readings. The framework is implemented and tested on two independent tactile datasets collected in sliding motion on 20 material textures. Our results show that the framework is able to make much more accurate inferences ahead of time as compared to that by the state-of-the-art learning approaches.

Classification of non-tumorous facial pigmentation disorders is an important but overlooked problem. Recently, a voting-based probabilistic linear discriminant analysis (V-PLDA) method was developed to address this problem by extracting hand-craft features from a given image set of rather small size, with limited classification accuracy. In this paper, we propose an improved Synthetic Minority Over-sampling Technique (improved SMOTE) with several parameters tuned to fully utilize the available images. Moreover, transfer learning is applied to reduce the data size requirement of the deep learning model. By combining the improved SMOTE and transfer learning, a classification accuracy gain (10%) is attained compared to the state-of-the-art V-PLDA method.

Non-tumorous facial pigmentation, though not fatal, adversely affects one's quality of life and may indicate concurrence of systemic diseases. Automatic diagnosis method such as voting-based probabilistic discriminant analysis (V-PLDA) has been explored, but the accuracy of classification is not satisfactory due to the limited number of data for training. This paper proposes to use the pre-trained deep learning network of Inception-ResNet-v2 so that information from similar datasets can be utilized. Furthermore, data augmentation using synthetic minority over-sampling technique (SMOTE) is also applied to make full use of available training data. A clinical dataset of five most common types of non-tumorous facial pigmentation disorders in Asia, namely freckles, lentigines, melasma, Hori's nevus, and nevus of Ota, is used for training and testing. The classification accuracy has shown significant improvement (> 7%) compared to the state-of-the-art method.