Grounded on pre-trained language models (PLMs), dense retrieval has been studied extensively on plain text. In contrast, there has been little research on retrieving data with multiple aspects using dense models. In the scenarios such as product search, the aspect information plays an essential role in relevance matching, e.g., category: Electronics, Computers, and Pet Supplies. A common way of leveraging aspect information for multi-aspect retrieval is to introduce an auxiliary classification objective, i.e., using item contents to predict the annotated value IDs of item aspects. However, by learning the value embeddings from scratch, this approach may not capture the various semantic similarities between the values sufficiently. To address this limitation, we leverage the aspect information as text strings rather than class IDs during pre-training so that their semantic similarities can be naturally captured in the PLMs. To facilitate effective retrieval with the aspect strings, we propose mutual prediction objectives between the text of the item aspect and content. In this way, our model makes more sufficient use of aspect information than conducting undifferentiated masked language modeling (MLM) on the concatenated text of aspects and content. Extensive experiments on two real-world datasets (product and mini-program search) show that our approach can outperform competitive baselines both treating aspect values as classes and conducting the same MLM for aspect and content strings. Code and related dataset will be available at the URL \footnotehttps://github.com/sunxiaojie99/ATTEMPT.

Candidate retrieval is a key part of the modern search engines whose goal is to find candidate items that are semantically related to the query from a large item pool. The core difference against the later ranking stage is the requirement of low latency. Hence, two-tower structure with two parallel yet independent encoder for both query and item is prevalent in many systems. In these efforts, the semantic information of a query and a candidate item is fed into the corresponding encoder and then use their representations for retrieval. With the popularity of pre-trained semantic models, the state-of-the-art for semantic retrieval tasks has achieved the significant performance gain.

Network embedding (NE), aiming to embed a network into a low dimensional latent representation while preserving the inherent structural properties of the network, has attracted considerable attention recently. Variational Autoencoder (VAE) has been widely studied for NE. Existing VAE based methods let the network follow a unimodal distribution, that is, they typically use some fixed distribution as the prior, e.g. Gaussian distribution. However, in reality networks often contain many complicated structural properties [5], [6] (such as the first/second order proximity, the motif or community structures, power-law, etc). The latent representation from unimodal and fixed distribution is not capable of describing such multi-modal characteristic of networks. To address this issue, we develop a new VAE method for NE, named Normalizing Flow Variational Graph Autoencoder (NF-VGA). We design a prior-generative module based on normalizing flows to generate flexible, multi-modal distribution as the prior of the latent representation. To make the generated prior better describe the coupling relationship between nodes, we further utilize network local structures to guide the prior generation. Extensive experiments on some real-world networks show a superior performance of the new approach over some state-of-the-art methods on some popular network embedding tasks.

Overlapping community detection has triggered recent research in network analysis. One of the promising techniques for finding overlapping communities is the popular stochastic models, which, unfortunately, have some common drawbacks. They do not support an important observation that highly connected nodes are more likely to reside in the overlapping regions of communities in the network. These methods are in essence not truly unsupervised, since they require a threshold on probabilistic memberships to derive overlapping structures and need the number of communities to be specified a priori . We develop a new method to address these issues for overlapping community detection. We first present a stochastic model to accommodate the relative importance and the expected degree of every node in each community. We then infer every overlapping community by ranking the nodes according to their importance. Second, we determine the number of communities under the Bayesian framework. We evaluate our method and compare it with five state-of-the-art methods. The results demonstrate the superior performance of our method. We also apply this new method to two applications, showing its superb performance on practical problems.