Composed image retrieval (CIR) utilizes a visual and a textual query to retrieve a target image. Prior work often abstracts the visual query into textual representations, which may discard instance-level visual cues; however, directly using these raw visual queries is empirically unreliable for improving the retrieval accuracy. Thus, we ask: why are visual cues difficult to leverage reliably in CIR, and when can their potential be realized? To answer this question, we conduct analysis with a reranking-based framework by reranking the top-K candidate pool and measuring per-query ground truth rank changes with different cues. Experiments on CIRR and CIRCO datasets show that raw query images frequently induce degradations, whereas controlled cues (e.g., edited images and generated captions) yield more reliable improvements and provide complementary evidence across modalities, with their impact being most pronounced when the initial ranking is uncertain.

Causal self-attention models such as SASRec are widely used in sequential recommendation, where learned attention weights are often assumed to provide crucial importance weighting over past interactions. Yet it is unclear when predictive performance truly depends on such non-uniform weighting. We study a controlled SASRec variant that replaces learned attention weights with uniform aggregation and is trained under an otherwise identical block structure and training recipe. Across fourteen benchmark datasets, this modification often yields performance comparable to the original model, with clear dataset-dependent exceptions. To explain this heterogeneity, we introduce a stage-wise norm-based decomposition that quantifies self-preserving vs. cross-position mixing within attention blocks. Across datasets, we find distinct regimes: low mixing yields robustness to uniformization; higher mixing tends to coincide with sensitivity, while some datasets exhibit substantial mixing without dependence on learned weighting. Our results provide a practical diagnostic for identifying when attention weighting is functionally utilized in sequential recommendation. The code is available at: https://github.com/keito0329/revisiting-sasrec.

This study aims to extend the applicability of stylized motion generation methods to be robust for large and diverse motions akin to those found in real-world data. Specifically, we introduce metadata-independent learning alongside style-focused learning, thereby enabling training from motions absent in motion-style datasets. In addition, we construct a novel motion dataset containing both various motions and stylized motions by unifying the multiple datasets to effectively train the model. Our novel learning method and dataset enable stylized motion generation methods to learn from both various motion knowledge and motion-style relations and improve their generalized performance. In downstream tasks, we address motion style transfer and text-to-stylized-motion, validating the enhancement of generalization abilities for each task. Compared to conventional methods, the proposed method demonstrates superior performance in generating and reflecting style, particularly under conditions featuring larger and more diverse motions. The code for this paper and the dataset will be released.

We present a novel Clustered Federated Learning (CFL) approach that efficiently shares knowledge among clusters to address non-Independent and Identically Distributed (non-IID) settings. Although conventional CFL has demonstrated strong performance in non-IID settings, a fundamental challenge, the lack of knowledge sharing among clusters remains as a problem to be solved for improvement of generalization. To overcome this limitation, we utilize personalized layers for specific tasks while retaining globally shared layers after clustering. By separating the layers of the model according to their functions, our method enables the model to be well personalized without sacrificing generalizability. Furthermore, we cluster clients based on the gradients of specific layers. Our efficient CFL method can better characterize the data of each client, reduce the memory demand of the client, and mitigate the computational burden of the FL server. We evaluate the effectiveness of our method in various data distribution settings.

Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.

We propose a novel continual self-supervised learning method (CSSL) considering medical domain knowledge in chest CT images. Our approach addresses the challenge of sequential learning by effectively capturing the relationship between previously learned knowledge and new information at different stages. By incorporating an enhanced dark experience replay (DER) into CSSL and maintaining both diversity and representativeness within the rehearsal buffer of DER, the risk of data interference during pretraining is reduced, enabling the model to learn more richer and robust feature representations. In addition, we incorporate a mixup strategy and feature distillation to further enhance the model’s ability to learn meaningful representations. We validate our method using chest CT images obtained under two different imaging conditions, demonstrating superior performance compared to state-of-the-art methods.

This paper presents a novel adversarial defense method that exploits non-transferability of attack across foundation models. Existing adversarial training methods have insufficient robustness to adversarial examples under powerful adversarial attacks in a white-box setting. We clarify that there is no attack transferability between various foundation models through preliminary experiments and propose a defense method that combines the non-transferability with adversarial training. The proposed method constructs a common embedding space of different models and detects adversarial examples by leveraging the different outputs from each model. Based on the detection of adversarial examples, the proposed method can adaptively select a robust model depending on the input images based on non-transferability of attack between models. Experimental results show that the proposed method improves the accuracy of zero-shot class classification for adversarial examples against each model.

Model generalization that prevents overfitting to the training data is critical to the robustness and reliability of image classification models. Diffusion models are rapidly developing in generating photorealistic images from text, yet it is difficult for a simple text to generate a variety of images that reflect the training data, and there is potential for model generalization. In this paper, we propose a novel method for model generalization with generated images via diffusion models. We make the diffusion model recognize the training data distribution in both the latent and the image space, and obtain generated images reflecting the distribution. In the latent space, we synthesize new features from the training data distribution and decode them into images using the diffusion model. In the image space, we perform selection of unintentionally generated images (out-of-distribution samples) based on the semantic information of the training data. By considering the training data distribution not only in the latent space but also in the image space, we eliminate out-of-distribution samples generated by the diffusion model. Experiments demonstrate that our generated images are faithful to the training data distribution and enhance model generalization performance.

This paper investigates the adversarial defense for foundation models without data centralization. To deploy foundation models in safety-critical applications, they must be robust against diverse attacks. In the defense against a single attack, foundation models require numerous adversarial examples, leading to a substantial increase in data and computational demands when addressing multiple attacks. However, traditional adversarial defense must centralize all types of adversarial examples, and there has been insufficient investigation into adversarial defense for foundation models without data centralization. Accordingly, we focus on model merging, which integrates fine-tuned models for a specific task to construct a merged model adaptable to any task. Through model merging, we aim to integrate individual models’ defensive capabilities to enhance robustness against various attacks. The main contribution of this paper is demonstrating that defense methods leveraging model merging can achieve classification accuracy approaching that of the defense method with data centralization under multiple attacks.

This paper presents a new domain-specific representation learning method, exponential dissimilarity-dispersion family (EDDF), a novel distribution family that includes a dissimilarity function and a global dispersion parameter. In generative models, variational autoencoders (VAEs) has a solid theoretical foundation based on variational inference in visual representation learning and are also used as one of core components of other generative models. This paper addresses the issue where conventional VAEs, with the commonly adopted Gaussian settings, tend to experience performance degradation in generative modeling for high-dimensional data. This degradation is often caused by their excessively limited model family. To tackle this problem, we propose EDDF, a new domain-specific method introducing a novel distribution family with a dissimilarity function and a global dispersion parameter. A decoder using this family employs dissimilarity functions for the evidence lower bound (ELBO) reconstruction loss, leveraging domain-specific knowledge to enhance high-dimensional data modeling. We also propose an ELBO optimization method for VAEs with EDDF decoders that implicitly approximates the stochastic gradient of the normalizing constant using log-expected dissimilarity. Empirical evaluations of the generative performance show the effectiveness of our model family and proposed method. Our framework can be integrated into any VAE-based generative models in representation learning. The code and model are available at https://github.com/ganmodokix/eddf-vae.

Disentangled representation learning aims at obtaining an independent latent representation without supervisory signals. However, the independence of a representation does not guarantee interpretability to match human intuition in the unsupervised settings. In this article, we introduce conceptual representation learning, an unsupervised strategy to learn a representation and its concepts. An antonym pair forms a concept, which determines the semantically meaningful axes in the latent space. Since the connection between signifying words and signified notions is arbitrary in natural languages, the verbalization of data features makes the representation make sense to humans. We thus construct Conceptual VAE (ConcVAE), a variational autoencoder (VAE)-based generative model with an explicit process in which the semantic representation of data is generated via trainable concepts. In visual data, ConcVAE utilizes natural language arbitrariness as an inductive bias of unsupervised learning by using a vision-language pretraining, which can tell an unsupervised model what makes sense to humans. Qualitative and quantitative evaluations show that the conceptual inductive bias in ConcVAE effectively disentangles the latent representation in a sense-making manner without supervision. Code is available at https://github.com/ganmodokix/concvae.

Recommender systems personalize user experiences by filtering large volumes of information, and further shape user behavior. Collaborative filtering (CF), a widely used algorithm in this domain, learns user preferences from user-item interactions. However, recent studies indicate that certain CF algorithms can excessively amplify inherent data biases, manifesting as bias disparity. In this study, we examine the latent factor model (LFM), a state-of-the-art CF method that represents users and items as vectors in a shared latent space. By applying linear dimensionality reduction techniques with strong interpretability (such as principal component analysis) to LFM embeddings trained on real-world data, we identify specific axes that encode biases. We then demonstrate the application of these linear relationships in mitigating bias disparity. Experimental results show that our proposed approach can reduce bias disparity with only a slight decrease in recommendation accuracy-an average of$5-8\%$with principal component analysis and$1-3\%$with independent component analysis.

To alleviate the negative impacts of noisy labels, most of the noisy label learning (NLL) methods dynamically divide the training data into two types, "clean samples" and "noisy samples", in the training process. However, the conventional selection of clean samples heavily depends on the features learned in the early stages of training, making it difficult to guarantee the cleanliness of the selected samples in scenarios where the noise ratio is high. In addition, their optimization processes are based on the supervised loss including noisy labels, and effective representation cannot be obtained in the presence of a large number of noisy labels. To address these problems, we propose an effective method capable of robustly performing NLL under extremely high noise ratios. In the proposed method, by introducing the prior knowledge of a pre-trained vision and language model, we can effectively select clean samples since it does not depend on the learning process of NLL. Moreover, the introduction of an unsupervised contrastive learning approach enables the acquisition of noise-robust feature representations in SSL. Experiments with synthetic label noise on CIFAR-10 and CIFAR-100, the benchmark datasets in NLL, demonstrate that the proposed method significantly outperforms state-of-the-art methods.

This paper presents a personalized text-to-image diffusion model for multiple object editing that can improve visual fidelity of the target image and editing ability with a segmentation-based restriction and continual learning. Multiple personalization tasks face the problem of destabilization, especially when the number of targets increases and the concepts of the targets are similar. The proposed method introduces a segmentation guide into continual learning to improve performance for multiple objects. The segmentation guide helps to separate each concept by restricting the regions of target objects during both training and inference. The proposed method learns these concepts by continual learning with Elastic Weight Consolidation, and achieves the output of multiple target objects with concept separation while maintaining visual fidelity. Experimental results demonstrate that the proposed method successfully maintains visual fidelity for multiple target objects.

This paper presents a new task of caption unification and a novel caption unification method for multi-view lifelogging images based on in-context learning with heterogeneous semantic contents. Most of the existing image captioning models target a single image and do not consider the common semantic contents among multiple images. Therefore, they suffer from inconsistent captioning for multi-view lifelogging images taken in the same scene, i.e., the images that have common semantic contents regardless of the viewpoints. The proposed method enables training the common semantic contents of multiple images and generating captions that comprehensively contain the semantic contents of the images based on in-context learning. The proposed method has the following two contributions. First, we identify the problem of existing image captioning models and consider caption unification as a new task for multi-view lifelogging images. Second, we generate unified captions based on in-context learning with heterogeneous semantic contents and evaluate the captions from two perspectives, uniformity and retainability. The experimental results demonstrate that the proposed method realizes successful caption unification.

We present a novel prompt-based personalized federated learning (pFL) method to address data heterogeneity and privacy concerns in traditional medical visual question answering (VQA) methods. Specifically, we regard medical datasets from different organs as clients and use pFL to train personalized transformer-based VQA models for each client. To address the high computational complexity of client-to-client communication in previous pFL methods, we propose a succinct information sharing system by introducing prompts that are small learnable parameters. In addition, the proposed method introduces a reliability parameter to prevent the negative effects of low performance and irrelevant clients. Finally, extensive evaluations on various heterogeneous medical datasets attest to the effectiveness of our proposed method.

In this paper, we propose a cross-domain few-shot in-context learning method based on the multimodal large language model (MLLM) for enhancing traffic sign recognition (TSR). We first construct a traffic sign detection network based on Vision Transformer Adapter and an extraction module to extract traffic signs from the original road images. To reduce the dependence on training data and improve the performance stability of cross-country TSR, we introduce a cross-domain few-shot in-context learning method based on the MLLM. To enhance MLLM’s fine-grained recognition ability of traffic signs, the proposed method generates corresponding description texts using template traffic signs. These description texts contain key information about the shape, color, and composition of traffic signs, which can stimulate the ability of MLLM to perceive fine-grained traffic sign categories. By using the description texts, our method reduces the cross-domain differences between template and real traffic signs. Our approach requires only simple and uniform textual indications, without the need for large-scale traffic sign images and labels. We perform comprehensive evaluations on the German traffic sign recognition benchmark dataset, the Belgium traffic sign dataset, and two real-world datasets taken from Japan. The experimental results show that our method significantly enhances the TSR performance.

This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. Deep learning classification models are often trained using datasets that mirror real-world scenarios. In this training process, because learning is based solely on correlations with labels, there is a risk that models may learn spurious relationships, such as an overreliance on features not central to the subject, like background elements in images. However, due to the black-box nature of the decision-making process in deep learning models, identifying and addressing these vulnerabilities has been particularly challenging. We introduce a novel framework for reinforcing the classification models, which consists of a two-stage process. First, we identify model weaknesses by testing the model using the counterfactual image dataset, which is generated by perturbed image captions. Subsequently, we employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model. Through extensive experiments on several classification models across various datasets, we revealed that fine-tuning with a small set of counterfactual images effectively strengthens the model.

This paper proposes a novel zero-shot composed image retrieval (CIR) method considering the query-target relationship by masked image-text pairs. The objective of CIR is to retrieve the target image using a query image and a query text. Existing methods use a textual inversion network to convert the query image into a pseudo word to compose the image and text and use a pre-trained visual-language model to realize the retrieval. However, they do not consider the query-target relationship to train the textual inversion network to acquire information for retrieval. In this paper, we propose a novel zero-shot CIR method that is trained end-to-end using masked image-text pairs. By exploiting the abundant image-text pairs that are convenient to obtain with a masking strategy for learning the query-target relationship, it is expected that accurate zero-shot CIR using a retrieval-focused textual inversion network can be realized. Experimental results show the effectiveness of the proposed method.

We propose a novel graph convolutional network (GCN)-based recommendation model that incorporates a contrastive loss. Although GCN-based recommendation models achieve high recommendation performance, existing models suffer from over-fitting since they explicitly encode the interactions as a graph. To mitigate this problem, while the models perform dropout which just randomly drops edges in the graph, this can miss the essential information that represents users’ preferences. The proposed method improves and extends the dropout via the contrastive loss and enhances the robustness while preserving the essential information. The contrastive loss can correlate the representations in different perturbed graphs and promote their consistency. Conceptually, our method captures users’ preferences which are invariant even in the perturbed interactions. The experimental results demonstrate that our method improves the recommendation performance in real-world datasets.

This paper presents a method for estimation of visual cognitive contents from human brain signals via a newly derived visual question answering (VQA) model. The proposed method can estimate a wide range of cognitive contents from functional magnetic resonance imaging data when subjects viewed images via the VQA model which can effectively utilize low-level and high-level image features extracted from the viewed images. To make use of multiple image features, we newly introduce brain-specific attention into the VQA model. The brain-specific attention can adaptively determine the significance of image features at each level depending on the complexity of the cognitive contents (e.g., category, pattern, number and color). Therefore, we enable flexible and extensive estimation of various cognitive contents. Experimental results show that the proposed method can significantly improve the performance of the visual content estimation.

We propose a novel binauralization method that is robust to camera rotation. Since binaural audio can bring a 3D sensation to the listener, it can enhance the immersive experience of the video. Researchers have been explored binaural audio generation from monaural audio to deepen the experience of already captured videos without special recording devices. However, this binauralization on real-world videos can be a challenging task due to camera rotation. Camera rotation makes it difficult to predict the exact sound source position due to the motion of the sound sources and the background in videos. To tackle this problem, we propose a training data generation pipeline using 360° videos for binauralization. We generate monocular videos and binaural audio with camera rotation from 360° videos for the training of binauralization. Additionally, we newly construct a binauralization framework that conducts multi-task learning with camera localization. The camera localization predicts the camera rotation and helps the binauralization. Experimental results show that our method can achieve the binauralization on videos with camera rotation.

This paper presents a novel video-music retrieval method for videos containing humans. Our method constructs a cross-modal common embedding space of video features based on human motion and music features and aligns the embedding space based on the beat and genre information. The beat and genre are distinctive properties of music, and music relates motion features better than features extracted from the whole of videos. Therefore, our method realizes the cross-modal retrieval focusing on the motion-music relation and improves the retrieval performance by utilizing motion-based video features and the music property-based embedding space alignment. This is the first task which relates human motion to music for the video-music retrieval. In the experiments, we compare our method with the state-of-the-art video-music and human motion-music retrieval methods and verify that our method achieves more than twice the retrieval performance of the conventional methods in Recall@1.

This paper proposes a novel text-guided facial image manipulation approach to improve robustness against the diversity of input images. Conventional text-guided facial image manipulation methods have achieved highly accurate image manipulation by restricting the aspect ratio and composition of the input image. In practical face image manipulation, images provided by the user often include regions other than the face, such as the upper body, and thus conventional methods have limitations in their application. To solve these problems, we tackle a new task of text-guided facial image manipulation with no restrictions on the composition and aspect ratio of the input image. As a base structure, we employ the face swapping method, which is robust to the diversity of input images and allows to swap faces. To achieve text-guided image manipulation based on the face swapping method, we also employ the manipulation direction, which focuses on the direction of changes between the image and its corresponding text that occurs before and after image manipulation. Experimental results show the effectiveness of the proposed method on wild images that include regions other than faces and have different aspect ratios.

We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model’s decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.

Multimedia content recommendation needs to consider users' preferences for each content. Conventional recommender systems consider them with wearable sensors, however, wearing such sensors can lead to a burden on users. In this paper, we construct a recommender system that can explicitly estimate users' preferences without wearable sensors. Specifically, by constructing lightweight but strong machine learning models suitable for our system, the users' interest levels for contents can be estimated from facial images obtained from a widely used webcam. In addition, through the interaction that the user selects displayed contents, our system finds the tendency of personal preferences for recommending contents with high user satisfaction. Our system is available on https://www.lmd-demo.org/2022/start_eng.html.

The global outbreak of the Coronavirus 2019 (COVID-19) has overloaded worldwide healthcare systems. Computer-aided diagnosis for COVID-19 fast detection and patient triage is becoming critical. This paper proposes a novel self-knowledge distillation based self-supervised learning method for COVID-19 detection from chest X-ray images. Our method can use self-knowledge of images based on similarities of their visual features for self-supervised learning. Experimental results show that our method achieved an HM score of 0.988, an AUC of 0.999, and an accuracy of 0.957 on the largest open COVID-19 chest X-ray dataset.

Domain shift causes performance drop in cross-domain object detection. To alleviate the domain shift, a prevailing approach is global feature alignment with adversarial learning. However, such simple feature alignment has defects of unawareness of fore-ground/background regions and well-aligned/poorly-aligned regions. To remedy the defects, in this paper, we propose a novel divergence-guided feature alignment method for cross-domain object detection. Specifically, we generate source-like images of the target domain and seek cues of foreground regions and poorly-aligned regions from prediction divergence of the source-like and original images. The feature alignment is guided by the divergence maps and consequently results in adaptation performance superior to alignment unaware of the cues. Different from most previous studies focusing on two-stage object detection, this paper is devoted to adapting one-stage object detectors which have simpler and faster inference. We validated the effectiveness of our method by conducting experiments in cross-weather, cross-camera, and synthetic-to-real adaptation scenarios.

This paper proposes a novel self-supervised learning method for learning better representations with small batch sizes. Many self-supervised learning methods based on certain forms of the siamese network have emerged and received significant attention. However, these methods need to use large batch sizes to learn good representations and require heavy computational resources. We present a new triplet network combined with a triple-view loss to improve the performance of self-supervised representation learning with small batch sizes. Experimental results show that our method can drastically outperform state-of-the-art self-supervised learning methods on several datasets in small-batch cases. Our method provides a feasible solution for self-supervised learning with real-world high-resolution images that uses small batch sizes.

This paper proposes a novel generative adversarial network to improve the performance of image manipulation using natural language descriptions that contain desired attributes. Text-guided image manipulation aims to semantically manipulate an image aligned with the text description while preserving text-irrelevant regions. To achieve this, we newly introduce referring image segmentation into the generative adversarial network for image manipulation. The referring image segmentation aims to generate a segmentation mask that extracts the text-relevant region. By utilizing the feature map of the segmentation mask in the network, the proposed method explicitly distinguishes the text-relevant and irrelevant regions and has the following two contributions. First, our model can pay attention only to the text-relevant region and manipulate the region aligned with the text description. Second, our model can achieve an appropriate balance between the generation of accurate attributes in the text-relevant region and the reconstruction in the text-irrelevant regions. Experimental results show that the proposed method can significantly improve the performance of image manipulation.

In this paper, we focus on a problem remaining to be studied: multi-source model adaptation, which is derived from multi-source unsupervised domain adaptation and replaces the source-domain data with source-domain pre-trained models. Pre-trained models are always easier to share than training data so that multiple source-domain models are available in many practical scenarios. Therefore, the problem setting of multi-source domain adaptation is practical in real-world applications. In this setting, we challenge the task of semantic segmentation which is difficult also in the traditional unsupervised domain adaptation due to the pixel-level knowledge transfer. Our method takes full advantage of the multiple source-domain models by learning model-invariant features, which aims to obtain target-domain features with similar distributions from the models pre-trained in different source domains. The adaptation models trained with the model-invariant feature learning benefit from the diversity of the source-domain models and can thus produce more generalizable features to the target domain. Experimental results in several adaptation settings validate the effectiveness and superiority of our method.

We propose a novel assessment approach to the performance of image manipulation using natural language descriptions in this paper. Text-guided image manipulation aims to modify an input image aligned with the text description and has recently become a hot topic for its usefulness. For the assessment, we focus on the similarity between "the change in text features" and "the change in image features" and define the direction of the changes as Manipulation Direction (MD). By capturing the changes in each different modality, MD can take into account the direction of image manipulation and quantify the extent to which the image is manipulated aligned with the text description. To the best of our knowledge, this is the first metric specialized in text-guided image manipulation. Experimental results show that MD can calculate evaluation scores that are correlated with subjective scores toward the manipulated images.

We propose an image attribute editing method with the mask-based retention loss. Although conventional image attribute editing methods can edit a particular attribute, they cannot retain non-editing attributes including unknown attributes before and after editing, which causes unexpected changes in the edited images. We solve this problem by dividing the pre- and post-edited images into the editing and non-editing regions and increasing the image similarity in the non-editing regions. In this paper, we introduce the novel mask-based retention loss to retain the non-editing regions. To compute the mask-based retention loss, we divide the images into the editing and non-editing regions by using a binary mask generated from the difference between the pre- and post-edited images. Experimental results show that our proposed method is qualitatively and quantitatively superior to state-of-the-art methods.

For developing valuable rubber materials, machine learning-based computer-aided analysis systems have been attracting a lot of attention. However, these systems mainly focus on analyzing the table and textual data, and the electron microscope images including the rich material information have not been enough analyzed. By effectively using these electron microscope images, further support for the material discovery is realized. In this paper, we present a material information retrieval system via electron microscope image space. Our system aims to support visually and comprehensively grasping the relationships between various rubber materials and those properties. By effectively using the electron microscope image space for material information retrieval, it is expected that the advances in material development are further accelerated.

Cross-modal image-retrieval methods retrieve desired images from a query text by learning relationships between texts and images. Such a retrieval approach is one of the most effective ways of achieving the easiness of query preparation. Recent cross-modal image-retrieval methods are convenient and accurate when users input a query text that can be used to uniquely identify the desired image. However, in reality, users frequently input ambiguous query texts, and these ambiguous queries make it difficult to obtain desired images. To overcome these difficulties, in this study, we propose a novel interactive cross-modal image-retrieval method based on question answering. The proposed method analyzes candidate images and asks users questions to obtain information that can narrow down retrieval candidates. By only answering questions generated by the proposed method, users can reach their desired images, even when using an ambiguous query text. Experimental results show the proposed method’s effectiveness.

Unpaired image-to-image (I2I) translation methods have been developed for several years. Present methods do not take into consideration semantic information of the original image, which may perform well on simple datasets of uncomplicated scenes, however, fail in complex datasets of scenes involving abundant objects, such as urban scenes. To tackle this problem, in this paper, we reasonably modify the previous problem setting and present a novel semantic-aware method. Specifically, in training, we use additional semantic label maps of training images, while in the test, no labels are required. We originally adopt a semantic knowledge distillation strategy to acquire semantic information from the labels and construct a particular normalization layer to introduce semantic information. Being aware of the pixel-level semantic information, our method can realize better I2I translation than the previous methods. Experiments are conducted on benchmark datasets of urban scenes to validate the effectiveness of our method.

We propose a novel approach that improves text-guided image manipulation performance in this paper. Text-guided image manipulation aims at modifying some parts of an input image in accordance with the user’s text description by semantically associating the regions of the image with the text description. We tackle the conventional methods’ problem of modifying undesired parts caused by differences in representation ability between text descriptions and images. Humans tend to pay attention primarily to objects corresponding to the foreground of images, and text descriptions by humans mostly represent the foreground. Therefore, it is necessary to introduce not only a foreground-aware bias based on text descriptions but also a background-aware bias that the text descriptions do not represent. We introduce an image segmentation network into the generative adversarial network for image manipulation to solve the above problem. Comparative experiments with three state-of-the-art methods show the effectiveness of our method quantitatively and qualitatively.

Unsupervised learning of disentangled representations is a core task for discovering interpretable factors of variation in an image dataset. We propose a novel method that can learn disentangled representations with semantic explanations on natural image datasets. In our method, we guide the representation learning of a variational autoencoder (VAE) via reconstruction in a visual-semantic embedding (VSE) space to leverage the semantic information of image data and explain the learned latent representations in an unsupervised manner. We introduce a semantic sub-encoder and a linear semantic sub-decoder to learn word vectors corresponding to the latent variables to explain factors of variation in the language form. Each basis vector (column) of the linear semantic sub-decoder corresponds to each latent variable, and we can interpret the basis vectors as word vectors indicating the meanings of the latent representations. By introducing the sub-encoder and the sub-decoder, our model can learn latent representations that are not just disentangled but interpretable. Comparing with other state-of-the-art unsupervised disentangled representation learning methods, we observe significant improvements in the disentanglement and the transferability of latent representations.

Image retrieval from a given text query (text-to-image retrieval) is one of the most essential systems, and it is effectively utilized for databases (DBs) on the Web. To make them more versatile and familiar, a retrieval system that is adaptive even for personal DBs such as images in smartphones and lifelogging devices should be considered. In this paper, we present a novel text-to-image retrieval system that is specialized for personal DBs. With the cross-modal scheme and the question-answering scheme, the developed system enables users to obtain the desired image effectively even from personal DBs. Our demo is available at https://sites.google.com/view/ir-questioner/.

We propose an approach that enhances arbitrary existing cross-modal image retrieval performance. Most of the cross-modal image retrieval methods mainly focus on direct computation of similarities between a text query and candidate images in an accurate way. However, their retrieval performance is affected by the ambiguity of text queries and the bias of target databases (DBs). Dealing with ambiguous text queries and DBs with bias will lead to accurate cross-modal image retrieval in real-world applications. A DB-adaptive re-ranking method using modality-driven spaces, which can extend arbitrary cross-modal image retrieval methods for enhancing their performance, is proposed in this paper. The proposed method includes two approaches: "DB-adaptive re-ranking'' and "modality-driven clue information extraction''. Our method estimates clue information that can effectively clarify the desired image from the whole set of a target DB and then receives user's feedback for the estimated information. Furthermore, our method extracts more detailed information of a query text and a target DB by focusing on modality-driven spaces, and it enables more accurate re-ranking. Our method allows users to reach their desired single image by just answering questions. Experimental results using MSCOCO, Visual Genome and newly introduced datasets including images with a particular object show that the proposed method can enhance the performance of state-of-the-art cross-modal image retrieval methods.

Unsupervised domain adaptation, which leverages label information from other domains to solve tasks on a domain without any labels, can alleviate the problem of the scarcity of labels and expensive labeling costs faced by supervised semantic segmentation. In this paper, we utilize adversarial learning and semi-supervised learning simultaneously to solve the task of unsupervised domain adaptation in semantic segmentation. We propose a new approach that trains two segmentation models with the adversarial learning symmetrically and further introduces the consistency between the outputs of the two models into the semi-supervised learning to improve the accuracy of pseudo labels which significantly affect the final adaptation performance. We achieve state-of-the-art semantic segmentation performance on the GTA5-to-Cityscapes scenario, a widely used benchmark setting in unsupervised domain adaptation.

This paper presents a soft-label anonymous gastric X-ray image distillation method based on a gradient descent approach. The sharing of medical data is demanded to construct high-accuracy computer-aided diagnosis (CAD) systems. However, the large size of the medical dataset and privacy protection are remaining problems in medical data sharing, which hindered the research of CAD systems. The idea of our distillation method is to extract the valid information of the medical dataset and generate a tiny distilled dataset that has a different data distribution. Different from model distillation, our method aims to find the optimal distilled images, distilled labels and the optimized learning rate. Experimental results show that the proposed method can not only effectively compress the medical dataset but also anonymize medical images to protect the patient's private information. The proposed approach can improve the efficiency and security of medical data sharing.

Unsupervised domain adaptation, which transfers supervised knowledge from a labeled domain to an unlabeled domain, remains a tough problem in the field of computer vision, especially for semantic segmentation. Some methods inspired by adversarial learning and semi-supervised learning have been developed for unsupervised domain adaptation in semantic segmentation and achieved outstanding performances. In this paper, we propose a novel method for this task. Like adversarial learning-based methods using a discriminator to align the feature distributions from different domains, we employ a variational autoencoder to get to the same destination but in a non-adversarial manner. Since the two approaches are compatible, we also integrate an adversarial loss into our method. By further introducing pseudo labels, our method can achieve state-of-the-art performances on two benchmark adaptation scenarios, GTA5-to-CITYSCAPES and SYNTHIA-to-CITYSCAPES.

We propose an estimation method for free-form Visual Question Answering (VQA) from human brain activity, brain decoding VQA. The task of VQA in the field of computer vision is generating an answer given an image and a question about its contents. The proposed method can realize answering arbitrary visual questions about images from brain activity measured by functional Magnetic Resonance Imaging (fMRI) while viewing the same images. We enable estimating various information from brain activity via a unique VQA model, which can realize a more detailed understanding of images and complex reasoning. In addition, we newly make use of un-labeled images not used in the training phase to improve the performance of the transformation, since fMRI datasets are generally small. The proposed method can answer a visual question from a little amount of fMRI data measured while subjects are viewing images.

Generation of human cognitive contents based on the analysis of functional magnetic resonance imaging (fMRI) data has been actively researched. Cognitive contents such as viewed images can be estimated by analyzing the relationship between fMRI data and semantic information of viewed images. In this paper, we propose a new method generating captions for viewed images from human brain activity via a novel robust regression scheme. Unlike conventional generation methods using image feature representations, the proposed method makes use of more semantic text feature representations, which are more suitable for the caption generation. We construct a text latent space with unlabeled images not used for the training, and the fMRI data are regressed to the text latent space. Besides, we newly make use of unlabeled images not used for the training phase to improve caption generation performance. Finally, the proposed method can generate captions from the fMRI data measured while subjects are viewing images. Experimental results show that the proposed method enables accurate caption generation for viewed images.

We present a new multimodal image-to-image translation model for the generation of gastritis images using X-ray and blood inspection results. In clinical situations, clinicians estimate the prognosis of the target disease by considering multiple inspection results. Similarly, we take a multimodal approach in the task of gastric cancer risk prediction. Visual characteristics of the gastric X-ray image and blood index values are highly related in the evaluation of gastric cancer risk. If we can generate a prediction image from blood index values, it contributes to the clinicians' sophisticated and integrated diagnosis. Hence, we learn a model that can map non-gastritis images to gastritis images based on the blood index values. Although this is a challenging multimodal task in medical image analysis, experimental results showed the effectiveness of our model.

A new approach that improves text-based image retrieval (hereinafter referred to as TBIR) performance is proposed in this paper. TBIR methods aim to retrieve a desired image related to a query text. Especially, recent TBIR methods allow us to retrieve images considering word relationships by using a sentence as a query. In these TBIR methods, it is necessary to uniquely identify a desired image from similar images using a single query sentence. However, the diverse expressive styles for a query sentence make it difficult to uniquely identify a desired image. In this paper, we propose a novel TBIR method with paraphrasing on multiple representation spaces. Specifically, by paraphrasing a query sentence on lingual and visual representation spaces, the proposed method can retrieve a desired image from various perspectives and then it can uniquely identify a desired image from similar images. Comprehensive experimental results show the effectiveness of the proposed method.

Cross-modal retrieval methods retrieve desired images from a query text by learning relationships between texts and images. This retrieval approach is one of the most effective ways in the easiness of query preparation. Recent cross-modal retrieval is convenient and accurate when users input a query text that can uniquely identify the desired image. Meanwhile, users frequently input ambiguous query texts, and these ambiguous queries make it difficult to obtain the desired images. To alleviate these difficulties, in this paper, we propose a novel interactive cross-modal retrieval method based on question answering (QA) with users. The proposed method analyses candidate images and asks users about information that can narrow retrieval candidates effectively. By only answering the questions generated by the proposed method, users can reach their desired images even from an ambiguous query text. Experimental results show the effectiveness of the proposed method.

This paper presents a method for gastritis detection from gastric X-ray images via fine-tuning techniques using a deep convolutional neural network (DCNN). DCNNs can learn parameters to capture high-dimensional features which express semantic contents of images by training on a large number of labeled images. However, lack of gastric X-ray images for training often occurs. To realize accurate detection with a small number of gastric X-ray images, the proposed method adopts fine-tuning techniques and newly introduces simple annotation of stomach regions to gastric X-ray images used for training. The proposed method fine-tunes a pre-trained DCNN with patches and three kinds of patch-level class labels considering not only the image-level ground truth ("gastritis"/"non-gastritis") but also the regions of a stomach since the outside of the stomach is not related to the image-level ground truth. In the test phase, by estimating the patch-level class labels with the fine-tuned DCNN, the proposed method enables the image-level class label estimation which excludes the effect of the unnecessary regions. Experimental results show the effectiveness of the proposed method.

We present a new scene retrieval method based on text-to-image Generative Adversarial Network (GAN) and its application to query-based video summarization. Text-to-image GAN is a deep learning method that can generate images from their corresponding sentences. In this paper, we reveal a characteristic that deep learning-based visual features extracted from images generated by text-to-image GAN include semantic information sufficiently. By utilizing the generated images as queries, the proposed method achieves higher scene retrieval performance than those of the state-of-the-art methods. In addition, we introduce a novel architecture that can consider order relationship of the input sentences to our method for realizing a target video summarization. Specifically, the proposed method generates multiple images thorough text-to-image GAN from multiple sentences summarizing target videos. Their summarized video can be obtained by performing the retrieval of corresponding scenes from the target videos according to the generated images with considering the order relationship. Experimental results show the effectiveness of the proposed method in the retrieval and summarization performance.

With the development of convolutional neural networks (CNNs), CNN-based methods for medical image analysis have achieved more accurate performance than conventional machine learning methods using hand-crafted features. Although these methods utilize a large number of training images and realize high performance, lack of the training images often occurs in medical image analysis due to several reasons. This paper presents a novel image generation method to construct a dataset for gastritis detection from gastric X-ray images. The proposed method effectively utilizes two kinds of training images (gastritis and non-gastritis images) to generate images of each domain by introducing label conditioning into a generative model. Experimental results using real-world gastric X-ray images show the effectiveness of the proposed method.

This paper presents a method of semi-supervised learning based on tri-training for gastritis classification using gastric X-ray images. The proposed method is constructed based on the tri-training architecture, and the strategies of label smoothing regularization and random erasing augmentation are utilized in the method to enhance the performance. Although the task of gastritis classification is challenging, we report that the proposed semi-supervised learning method using only a small number of labeled data achieves 0.888 harmonic mean of sensitivity and specificity on test data composed of 615 patients.

Text-to-image Generative Adversarial Network (GAN) is a deep learning model that generates an image from an input sentence. It is expressly attracting attentions because of its applicability of the generated images. However, many existing studies have still focused on generation of high-quality images, and there are few studies focusing on application of the generated images since text-to-image GANs still cannot produce visually pleasing images in the complicated tasks. In this paper, we apply a text-to-image GAN as a generator of query images for a scene retrieval task to show availability of the visually non-pleasant images. The proposed method utilizes a low-resolution generated image that focuses on a sentence and a high-resolution generated image that focuses on each word of the sentence to retrieve a desired scene. With this mechanism, the proposed method realizes a high-accuracy scene retrieval from a sentence input. Experimental results show the effectiveness of our method.

This paper presents an anonymous gastritis image generation method based on a generative adversarial network approach. Since clinical individual data include highly confidential information, they must be handled carefully. Although data sharing is demanded to construct large-scale medical image datasets for deep learning-based recognition tasks, managing and annotating these data have been conducted manually. The proposed method enables the generation of anonymous images by an adversarial learning approach. Experimental results show that generated images by our method contribute to a gastritis recognition task. This will be helpful for constructing large-scale medical image datasets effectively.