We present a new multimodal face image generation method that converts a text prompt and a visual input, such as a semantic mask or scribble map, into a photorealistic face image. To do this, we combine the strengths of Generative Adversarial networks (GANs) and diffusion models (DMs) by employing the multimodal features in the DM into the latent space of the pretrained GANs. We present a simple mapping and a style modulation network to link two models and convert meaningful representations in feature maps and attention maps into latent codes. With GAN inversion, the estimated latent codes can be used to generate 2D or 3D-aware facial images. We further present a multi-step training strategy that reflects textual and structural representations into the generated image. Our proposed network produces realistic 2D, multi-view, and stylized face images, which align well with inputs. We validate our method by using pretrained 2D and 3D GANs, and our results outperform existing methods. Our project page is available at https://github.com/1211sh/Diffusion-driven_GAN-Inversion/.

Recently, panoramic imaging system has been attracting a lot of attention in various real-world applications due to its all-around sensing abilities. Despite the success of semantic segmentation, the performance of panoramic segmentation is still poor because the number of annotated panoramic datasets is insufficient and existing methods cannot handle the structural distortions in panoramic images caused by wide FoV. In this paper, we present a novel PAnoramic Segmentation Transformers (PASTs) trained by a knowledge distillation strategy with teacher-student branches. We first train the teacher using labeled pinhole images. The knowledge learned from the teacher is transferred to the student via feature distillation. To this end, we exploit the distorted pinhole images to force the attention and the prediction from the teacher consistent with those from the student. In addition, we adopt the entropy loss to train the student with unlabeled panoramic images. Experimental results demonstrate the effectiveness of our method, both qualitatively and quantitatively.

In this paper, we formulate image edit propagation as a task of machine learning to handle a high resolution image efficiently. Conventional graph-based methods solve the edit propagation by minimizing an energy function which considers the relationship between a reference pixel and its spatially neighboring ones. It is becoming a time-consuming and memory-requiring task due to the increase of the image size. Inspired by the observation that similar features get analogous edits, the edit propagation is casted as a classification problem using support vector machines in the feature space. A classifier is trained with initial sparse edits given by user interaction, and then the rest of the features are classified and manipulated. In experiments, the proposed method is applied to an image recoloring to verify the performance. Experimental results show that the proposed method gives competitive editing results comparing to other state-of-the-art methods.