Music generation has developed remarkably, and research on music generation from text has progressed. However, there are cases where it is difficult to use text as the query when users want to obtain music that is suitable for images, while image-to-music generation is still unexplored. There is a gap between images, which are visual content, and music, and it has been difficult to associate them directly. To address this problem, we realize an image-to-music generation method through musical captions which describe the characteristics of the music. Musical captions possess an enhanced capability to steer the process of music generation. Therefore, if musical captions effectively convey the intended message of the image, they serve as an excellent intermediary between the images and music. The proposed method connects these two different modalities through the medium of musical captions that describe the specialized content of the music. By generating images through musical captions using multi-modal large language models, we construct an image-musical caption pair dataset. Using query-similar images and their paired musical captions in the dataset, in-context learning on a multi-modal large language model is conducted to generate the musical caption corresponding to the target image. The generated musical caption is then input into a text-to-music generative model, and thus, the proposed method enables high-quality image-to-music generation. We conducted experiments to evaluate the quality of the generated music and the consistency with target images through both subjective and objective metrics. The results confirm the effectiveness of our proposed method. The code can be found at https://github.com/lsllsls/CAI2M.

In this paper, we present a method of feature integration via back-projection ordering multi-modal Gaussian process latent variable model (BPomGP) for rating prediction. In the proposed method, to extract features reflecting the users' interest, we use the known ratings assigned to the viewed contents and users' behavior information while viewing the contents which is related to the users' interest. BPomGP has two important approaches. Unlike the training phase, where the above two types of heterogeneous information are available, behavior information is not given in the test phase. Thus, the projection that transforms content features into integrated features is trained by using both content and behavior features. Furthermore, to reflect the unique characteristics of ratings, their ordering, we constrain the distance between integrated features in a latent space based on the difference in the degree of known ratings. Finally, experimental results show that the proposed method can calculate features that are useful for rating prediction.

Multimedia content recommendation needs to consider users' preferences for each content. Conventional recommender systems consider them with wearable sensors, however, wearing such sensors can lead to a burden on users. In this paper, we construct a recommender system that can explicitly estimate users' preferences without wearable sensors. Specifically, by constructing lightweight but strong machine learning models suitable for our system, the users' interest levels for contents can be estimated from facial images obtained from a widely used webcam. In addition, through the interaction that the user selects displayed contents, our system finds the tendency of personal preferences for recommending contents with high user satisfaction. Our system is available on https://www.lmd-demo.org/2022/start_eng.html.

This paper presents a method of feature integration via semi-supervised ordinally multi-modal Gaussian process latent variable model (Semi-OMGP). The proposed method transforms multimodal features into common latent variables suitable for users' interest level estimation. For dealing with the multi-modal features, the proposed method newly derives Semi-OMGP. Semi-OMGP has two contributions. First, Semi-OMGP is suitable for integration between heterogeneous modalities with different distributions by assuming that the similarity matrices of these modalities as observations are generated from latent variables. Second, Semi-OMGP can efficiently use label information by introducing an operator considering the ordinal grade into the prior distribution of latent variables when obtained label information is partially given. Semi-OMGP can simultaneously realize the above contributions, and successful multi-modal feature integration becomes feasible. Experimental results show the effectiveness of the proposed method.