In this paper, we propose a multimodal contrastive learning model for music information processing that integrates three modalities ---MIDI, audio, and text--- while effectively handling cases where some modalities are missing. Previous research has largely focused on integrating two modalities, such as audio-text or MIDI-text. To the best of our knowledge, no prior studies have explored the integration of more than two modalities in the context of music. Given that MIDI is widely used in music, and audio and text are even more prevalent, our goal is to increase the practical applicability of our approach in the music domain by integrating MIDI, audio, and text modalities. We utilize a pretrained CLAP audio encoder and text encoder, along with a dataset that aligns MIDI with text captions, to enable common embeddings across these three modalities. Due to the limited availability of MIDI-audio-aligned datasets, we avoid direct contrastive learning between MIDI and audio. Instead, we integrate MIDI-text alignment into the audio-text embedding space, allowing MIDI and audio embeddings to be indirectly aligned through the text modality. Additionally, we incorporate Content-Aware Loss Weighting into the loss function to improve the effectiveness of text-aligned multimodal contrastive learning. This technique adjusts weights to mitigate the adverse effects of coincidentally co-occurring word tokens between anchor and negative samples. Experimental results show that the proposed model outperforms existing methods in both cross-modal retrieval and zero-shot classification tasks.

Abstract In our study, we propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents, enabling greater flexibility and the ability to estimate more coherent topics compared to traditional topic models. On the other hand, some self-supervised learning methods use a joint embedding architecture with two identical networks that produce similar representations for two augmented versions of the same input. Regularizations are applied to these representations to prevent collapse, which would otherwise result in the networks outputting constant or redundant representations for all inputs. Our model enhances topic quality by explicitly regularizing latent topic representations of anchor and positive samples. We also introduced an adversarial data augmentation method to replace the heuristic sampling method. We further developed several variation models including those on the basis of an NTM that incorporates contrastive learning with both positive and negative samples. Experimental results on three datasets showed that our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.

Nonparametric topic models such as hierarchical Dirichlet processes (HDP) have been attracting more and more attentions for multimedia data analysis. However, the existing models for multimedia data are unsupervised ones that purely cluster semantically or characteristically related features into a specific latent topic without considering side information such as class information. In this paper, we present a novel supervised sequential symmetric correspondence HDP (Sup-SSC-HDP) model for multi-class video classification, where the empirical topic frequencies learned from multimodal video data are modeled as a predictor of video class. Qualitative and quantitative assessments demonstrate the effectiveness of Sup-SSC-HDP.

In recent years, many researchers have taken keen interest in analyzing various kinds of relational data, such as social networks and financial networks. These data can be expressed as a graph or network where each vertex or node is an entity and each edge or link is a relation between a pair of entities. Moreover, each link is often associated with continuous and/or discrete relational attributes, such as in financial networks, the interest rate for a transaction and whether the transaction is international or intranational. In this paper we focus on max-margin latent feature relational models (called Med-LFRM) that are based on Indian buffet process (IBP) and maximum entropy discrimination (MED). For the estimation of model parameters, the Bayesian estimation is deemed equivalent to minimizing an objective function, which involves misclassification errors. We focus on link prediction problem for the networks with continuous and discrete relational attributes. We also focused on the time dependent analysis for the networks, and therefore, we estimated the model parameters considering the observations in the previous time interval. We demonstrate, through experiments with interbank financial networks, the effectiveness of the above model in terms of the link prediction performance.

Video emotion recognition as an emerging research field has been attracting more and more focus in recent years. However, such work is quite challenging, since human emotions are hard to differentiate precisely due to its complexity and diversity, moreover, the expressions of sentiment in a content-rich video are sparse. Previous studies presented a number of approaches to try to learn human emotions on video level by exploiting various video features. However, most of works just used simple low-level video features such as hand-crafted image features, and they also did not consider the further latent connections among different multimodal data within a video. To tackle these problems, we develop a novel Bayesian non-parametric multimodal data modeling framework to learn the emotions from video, where the adopted image data are deep features extracted from key frames of video via convolutional neural networks (CNNs), and the adopted audio data are Mel-frequency cepstral coefficient (MFCC) features. In this framework, we then use a symmetric correspondence hierarchical Dirichlet processes (Sym-cHDP) model to mine their latent emotional events (topics) between image features and audio features. Finally, the effectiveness of our framework is demonstrated via comprehensive experimentations.

There are increasing demands for improved analysis of multimodal data that consist of multiple representations, such as multilingual documents and text-annotated images. One promising approach for analyzing such multimodal data is latent topic models. In this paper, we propose conditionally independent generalized relational topic models (CI-gRTM) for predicting unknown relations across different multiple representations of multimodal data. We developed CI-gRTM as a multimodal extension of discriminative relational topic models called generalized relational topic models (gRTM). We demonstrated through experiments with multilingual documents that CI-gRTM can more effectively predict both multilingual representations and relations between two different language representations compared with several state-of-the-art baseline models that enable to predict either multilingual representations or unimodal relations.

Multimedia data mining based on topic models as an emerging technique has become a very popular research topic in recent years. In this paper, we propose a novel topic model named sequential correspondence hierarchical Dirichlet Processes (Seq-cHDP) to learn the hidden structure within video data. The Seq-cHDP model can be considered as an extended hierarchical Dirichlet processes (HDP) model containing two important features: one is the time-dependency mechanism that connects neighboring video frames on the basis of a time dependent Markovian assumption, and the other is the data correspondence mechanism that provides a solution for dealing with the multimodal data such as the mixture of visual words and speech words extracted from video files. We present a comprehensive evaluation for Seq-cHDP through experimentation and finally demonstrate that our model outperforms than other baseline models.

Topic modeling is a widely used approach to analyzing large text collections. A small number of multilingual topic models have recently been explored to discover latent topics among parallel or comparable documents, such as in Wikipedia. Other topic models that were originally proposed for structured data are also applicable to multilingual documents. Correspondence Latent Dirichlet Allocation (CorrLDA) is one such model; however, it requires a pivot language to be specified in advance. We propose a new topic model, Symmetric Correspondence LDA (SymCorrLDA), that incorporates a hidden variable to control a pivot language, in an extension of CorrLDA. We experimented with two multilingual comparable datasets extracted from Wikipedia and demonstrate that SymCorrLDA is more effective than some other existing multilingual topic models.

No abstract available.

This paper gives an overview of the evaluation method used for the Web Retrieval Task in the Third NTCIR Workshop, which is currently in progress. In the Web Retrieval Task, we try to assess the retrieval effectiveness of each Web search engine system using a common data set, and attempt to build a re-usable test collection suitable for evaluating Web search engine systems. With these objectives, we have built 100gigabyte and 10-gigabyte document sets, mainly gathered from the ‘.jp ’ domain. Relevance judgment is performed on the retrieved documents, which are written in Japanese or English.

In this paper, we propose a new method of information retrieval which combines adaptive and incremental query expansion with cluster-based browsing. The proposed method attempts to accurately learn users' interests from their relevance judgments on clustered search results instead of individual documents, reducing users' loads for the judgments. The use of adaptive relevance feedback leads to the capability for tracking vague or dynamically shifting goals of users. Incrementally expanded and refined queries can be used in re-searching to improve the retrieval effectiveness. We apply the proposed method to the information retrieval on the World Wide Web and demonstrate its effectiveness through basic experiments.

No abstract available.