Referring Image Segmentation (RIS) is a task aimed at segmenting objects expressed in natural language within an image. This task requires an understanding of the relationship between vision and language, along with precise segmentation capabilities. In the field of computer vision, CLIP and Segment anything model (SAM) have gained significant attention for their classification and the segmentation capabilities. Given that both models possess essential skills for RIS, combining them seems to be an effective strategy. In this paper, we propose a model that integrates CLIP and SAM to enhance RIS. Since SAM lacks classification capabilities, we developed a module that supplies the SAM mask decoder with features that specify the target object. We introduce a new module, which is trained on additional instance segmentation tasks. The features utilized and derived from this module serve as inputs for the SAM decoder. With these inputs, SAM is expected to effectively segment areas corresponding to the given naturallanguage expressions. We conducted experiments using the traditional RefCOCO/+/g, as well as the recently introduced gRefCOCO and Refrom datasets, demonstrating the advantages of our approach. Code will be available on https://github.com/hitachi-rd-cv/dfam.

To accelerate open science, it is encouraged to publish and share research artifacts, e.g., scholarly papers, datasets, and code, with their metadata. Although the metadata facilitates the discovery of research artifacts, the metadata is not always complete. In this study, we verify the feasibility of estimating metadata that has not yet been filled (unfilled metadata) using metadata that has been filled (filled-in metadata) for realizing automatic completion of metadata. We implement an estimation method and evaluate its performance through experiments. The method takes filled-in metadata expressed in text as input and estimates unfilled metadata using a BERT-based model. In our experiments, the filled-in metadata was set as the name of the research artifacts, and the unfilled metadata was set as their type. Experimental results showed a high estimation performance.

In Japan, the number of single-person households, particularly among the elderly, is increasing. Consequently, opportunities for people to narrate are being reduced. To address this issue, conversational agents, e.g., communication robots and smart speakers, are expected to play the role of the listener. To realize these agents, this paper describes the collection of conversational responses by listeners that demonstrate attentive listening attitudes toward narrative speakers, and a method to annotate existing narrative speech with responsive utterances is proposed. To summarize, 148,962 responsive utterances by 11 listeners were collected in a narrative corpus comprising 13,234 utterance units. The collected responsive utterances were analyzed in terms of response frequency, diversity, coverage, and naturalness. These results demonstrated that diverse and natural responsive utterances were collected by the proposed method in an efficient and comprehensive manner. To demonstrate the practical use of the collected responsive utterances, an experiment was conducted, in which response generation timings were detected in narratives.

Narrating is a fundamental human desire. In order to satisfy the desire, it is more effective to have multiple listeners listen to narratives than to have one listener does. Conversational agents, e.g., communication robots and smart speakers, are expected to listen to narratives, instead of human listeners. For such agents to be recognized as narrative listeners, it is necessary to generate responsive utterances at appropriate timings. This paper proposes a method to estimate the generation timings of responsive utterances by active listeners of spoken narratives. The proposed method estimates them by encoding acoustic and linguistic information extracted from narratives with a transformer-based approach. To evaluate the performance of the proposed method, an experiment was conducted using 131,616 responsive utterances to narratives. The experimental results demonstrated that the proposed method was effective for the estimation and both the acoustic and linguistic information was useful.

We propose an audio-visual speech enhancement (AVSE) method conditioned both on the speaker’s lip motion and on speaker-discriminative embeddings. We particularly explore a method of extracting the embeddings directly from noisy audio in the AVSE setting without an enrollment procedure. We aim to improve speech-enhancement performance by conditioning the model with the embedding. To achieve this goal, we devise an AV voice activity detection (AV-VAD) module and a speaker identification module for the AVSE model. The AV-VAD module assesses reliable frames from which the identification module can extract a robust embedding for achieving an enhancement with the lip motion. To effectively train our modules, we propose multi-task learning between the AVSE, speaker identification, and VAD. Experimental results show that (1) our method directly extracted robust speaker embeddings from the noisy audio without an enrollment procedure and (2) improved the enhancement performance compared with the conventional AVSE methods.

A service robot is expected to provide proactive service for visitors who require its help. In contrast to passive service, e.g., providing service only after being spoken to, proactive service initiates an interaction at an early stage, e.g., talking to potential visitors who need the robot’s help in advance. This paper addresses how to anticipate the start of user interaction. We propose an approach using only a single RGB camera that anticipates whether a visitor will come to the robot for interaction or just pass it by. In the proposed approach, we (i) utilize the visitor’s pose information from captured images incorporating facial information, (ii) train a CNN-LSTM–based model in an end-to-end manner with an exponential loss for early anticipation, and (iii) during the training, the network branch for facial keypoints acquired as the part of the human pose information is taught to mimic the branch trained with the face image from a specialized face detector with a human verification. By virtue of (iii), at the inference, we can run our model in an embedded system processing only the pose information without an additional face detector and typical accuracy drop.We evaluated the proposed approach on our collected real world data with a real service robot and publicly available JPL interaction dataset and found that it achieved accurate anticipation performance.

Nowadays, spoken dialogue agents such as communication robots and smart speakers listen to narratives of humans. In order for such an agent to be recognized as a listener of narratives and convey the attitude of attentive listening, it is necessary to generate responsive utterances. Moreover, responsive utterances can express empathy to narratives and showing an appropriate degree of empathy to narratives is significant for enhancing speaker’s motivation. The degree of empathy shown by responsive utterances is thought to depend on their type. However, the relation between responsive utterances and degrees of the empathy has not been explored yet. This paper describes the classification of responsive utterances based on the degree of empathy in order to explain that relation. In this research, responsive utterances are classified into five levels based on the effect of utterances and literature on attentive listening. Quantitative evaluations using 37,995 responsive utterances showed the appropriateness of the proposed classification.