Topic modeling plays a critical role in organizing and understanding large-scale web content. While neural topic models (NTMs) based on variational autoencoders (VAEs) have achieved notable success in analyzing textual data, they remain limited in addressing the multimodal nature of modern web content. Existing unimodal or multimodal extensions often suffer from posterior collapse and fail to capture the directional semantics inherent in both text and images, resulting in incoherent topics and limited interpretability. To address these challenges, we propose MM-vNTM (MultiModal Neural Topic Model with von Mises-Fisher Mixtures), a framework for web-scale topic discovery over multimodal data. MM-vNTM leverages pre-aligned cross-modal embeddings as inputs and jointly models document-level representations of text and image modalities in a shared hyperspherical latent space. Furthermore, it defines topics as mixtures of von Mises-Fisher (vMF) distributions in the L2-normalized word embedding space, explicitly capturing directional similarity. Experiments on multimedia web datasets demonstrate that MM-vNTM consistently outperforms state-of-the-art unimodal and multimodal baselines in terms of overall topic quality, highlighting its effectiveness for real-world web scenarios.

In recent years, axial data, where observations are treated as axes of direction, has gained prominence in a range of complex tasks, including gene expression data clustering, blind speech separation, and depth image analysis. However, prevailing methods for axial data modeling mainly rely on shallow probabilistic models, which often overlook the hidden and hierarchical dependencies in the latent space. These methods also require a separate, human-engineered feature extractor to obtain features from raw axial data for downstream tasks. This work introduces a novel framework, Axial Variational Autoencoders (AVAEs), for modeling and representation learning of axial data by leveraging a deep generative model, the Variational Autoencoder (VAE). Unlike existing approaches, our method can autonomously learn more expressive representations from axial data by designing a VAE that uses the Watson distribution as the latent prior. Furthermore, we introduce a tailored reparameterization technique to support stable training. We validate the effectiveness of our model through experiments on simulated axial datasets and a real-world application.

Clustering high-dimensional directional data (i.e., L2 normalized vectors) presents significant challenges due to the intricate spherical representations of latent embeddings and the limitations of classical (non-deep) clustering techniques. Moreover, dynamically inferring the number of clusters remains a fundamental issue in existing deep clustering methods, especially those involving complex model-selection criteria. This paper addresses these challenges by introducing a novel deep nonparametric clustering framework that employs hyperspherical latent embeddings within a Variational Autoencoder architecture, enhanced by an infinite Von Mises-Fisher Mixture Model as a dynamic prior. This approach enables automatic adaptation of cluster numbers during training, eliminating the need for predefined clusters and traditional model selection processes. Our scalable architecture effectively integrates In-vMFMM with hyperspherical embeddings to tackle the complexities of directional data. Utilizing a joint training strategy, our method alternates between updating neural network parameters and adjusting mixture model priors via nonparametric variational Bayes. Empirical evaluations on benchmark datasets, including complex ImageNet-50, demonstrate that our approach significantly outperforms state-of-the-art deep nonparametric clustering methods. It also robustly estimates the number of clusters, showcasing its effectiveness and versatility in handling high-dimensional directional data.

We introduce a mixture of mixtures of bounded asymmetric generalized Gaussian and uniform distributions. Based on this framework, we propose model-based classification and model-based clustering algorithms. We develop an objective function for the minimum message length (MML) model selection criterion to discover the optimal number of clusters for the unsupervised approach of our proposed model. Given the crucial attention received by Explainable AI (XAI) in recent years, we introduce a method to interpret the predictions obtained from the proposed model in both learning settings by defining their boundaries in terms of the crucial features. Integrating Explainability within our proposed algorithm increases the credibility of the algorithm’s predictions since it would be explainable to the user’s perspective through simple If-Then statements using a small binary decision tree. In this paper, the proposed algorithm proves its reliability and superiority to several state-of-the-art machine learning algorithms within the following real-world applications: fault detection and diagnosis (FDD) in chillers, occupancy estimation and categorization of residential energy consumers.

We consider the problem of anomaly detection in proportional data by investigating the Libby-Novick Beta-Liouville distribution, a novel distribution merging the salient characteristics of Liouville and Libby-Novick Beta distributions. Its main benefit, compared to the typical distributions dedicated to proportional data such as Dirichlet and Beta-Liouville, is its adaptability and explanatory power when dealing with this kind of data. Our goal is to exploit this appropriateness for modeling proportional data to achieve great performance in the anomaly detection task. First, we develop generative models, namely finite mixture models of Libby-Novick Beta-Liouville distributions. Then, we propose two discriminative techniques: Normality scores based on selecting the given distribution to approximate the softmax output vector of a deep classifier and an improved version of Support Vector Machine (SVM) by suggesting a feature mapping approach. We demonstrate the benefits of the presented approaches through a variety of experiments on both image and non-image datasets. The results demonstrate that the proposed anomaly detectors based on the Libby-Novick Beta-Liouville distribution outperform the classical distributions as well as the baseline techniques.

We introduce Knowledge-enhanced Spherical Representation Learning (K-SRL), a generative probabilistic model of text documents that combines word embeddings and knowledge graph embeddings to effectively encode the semantic information of text and the related background knowledge into a low-dimensional representation. More specifically, the proposed model represents each text document as a combination of both words and entities linked to an external large knowledge graph and models them as points on the unit hypersphere using the von Mises-Fisher distribution. Furthermore, we develop an efficient variational Bayesian inference algorithm to learn unsupervised text embeddings in the spherical space. Experimental results on multiple benchmark datasets demonstrate that our model outperforms existing probabilistic models on common text classification tasks, including text categorization and sentiment analysis.

In topic models, collections are organized as documents where they arise as mixtures over latent clusters called topics. A topic is a distribution over the vocabulary. In large-scale applications, parametric or finite topic mixture models such as LDA (latent Dirichlet allocation) and its variants are very restrictive in performance due to their reduced hypothesis space. In this article, we address the problem related to model selection and sharing ability of topics across multiple documents in standard parametric topic models. We propose as an alternative a BNP (Bayesian nonparametric) topic model where the HDP (hierarchical Dirichlet process) prior models documents topic mixtures through their multinomials on infinite simplex. We, therefore, propose asymmetric BL (Beta-Liouville) as a diffuse base measure at the corpus level DP (Dirichlet process) over a measurable space. This step illustrates the highly heterogeneous structure in the set of all topics that describes the corpus probability measure. For consistency in posterior inference and predictive distributions, we efficiently characterize random probability measures whose limits are the global and local DPs to approximate the HDP from the stick-breaking formulation with the GEM (Griffiths-Engen-McCloskey) random variables. Due to the diffuse measure with the BL prior as conjugate to the count data distribution, we obtain an improved version of the standard HDP that is usually based on symmetric Dirichlet (Dir). In addition, to improve coordinate ascent framework while taking advantage of its deterministic nature, our model implements an online optimization method based on stochastic, at document level, variational inference to accommodate fast topic learning when processing large collections of text documents with natural gradient. The high value in the predictive likelihood per document obtained when compared to the performance of its competitors is also consistent with the robustness of our fully asymmetric BL-based HDP. While insuring the predictive accuracy of the model using the probability of the held-out documents, we also added a combination of metrics such as the topic coherence and topic diversity to improve the quality and interpretability of the topics discovered. We also compared the performance of our model using these metrics against the standard symmetric LDA. We show that online HDP-LBLA (Latent BL Allocation)’s performance is the asymptote for parametric topic models. The accuracy in the results (improved predictive distributions of the held out) is a product of the model’s ability to efficiently characterize dependency between documents (topic correlation) as now they can easily share topics, resulting in a much robust and realistic compression algorithm for information modeling.

In this paper, we propose a clustering-based online news topic detection and tracking (TDT) approach based on hierarchical Bayesian nonparametric framework that allows topics to be shared across different news stories in a corpus. Our approach is formulated using the hierarchical Pitman-Yor process mixture model with the inverted Beta-Liouville (IBL) distribution as its component density, which has shown superior performance in modeling text data than the widely used Gaussian distribution. Moreover, we theoretically develop a convergence-guaranteed online learning algorithm that can effectively learn the proposed TDT model from a stream of news stories based on varational Bayes. The merits of our TDT approach are illustrated by comparing it with other well-defined clustering-based TDT approaches on different news data sets.

Burstiness and overdispersion phenomena of count vectors pose significant challenges in modeling such data accurately. While the dependency assumption of the multinomial distribution causes its failure to model frequency vectors in several machine learning and data mining applications, researchers found that by extending the multinomial distribution to the Dirichlet Compound multinomial (DCM), both phenomena modeling can be addressed. However, Dirichlet distribution is not the best choice, as a prior, given its negative-correlation and equal-confidence requirements. Thus, we propose to use a flexible generalization of the Dirichlet distribution, namely, the shifted-scaled Dirichlet, as a prior to the multinomial, which grants the model a capability to better fit real data, and we call the new model the Multinomial Shifted-Scaled Dirichlet (MSSD). Given that the likelihood function plays a key role in statistical inference, e.g., in maximum likelihood estimation and Fisher information matrix investigation, we propose to improve the efficiency of computing the MSSD log-likelihood by approximating its function based on Bernoulli polynomials where the log-likelihood function is computed using the proposed mesh algorithm. Moreover, given the sparsity and high-dimensionality nature of count vectors, we propose to improve its computation efficiency by approximating the novel MSSD as a member of the exponential family of distribution, which we call EMSSD. The clustering is based on mixture models, and for learning a model, selection approach is seamlessly integrated with the estimation of the parameters. The merits of the proposed approach are validated via challenging real-world applications such as hate speech detection in Twitter, real-time recognition of criminal action, and anomaly detection in crowded scenes. Results reveal that the proposed clustering frameworks offer a good compromise between other state-of-the-art techniques and outperform other approaches previously used for frequency vectors modeling. Besides, comparing to the MSSD, the approximation EMSSD has reduced the computational complexity in high-dimensional feature spaces.

A fast exact nearest neighbor search algorithm over large scale data is proposed based on semi-convex hull tree, where each node represents a semi-convex hull, which is made of a set of hyper planes. When performing the task of nearest neighbor queries, unnecessary distance computations can be greatly reduced by quadratic programming. GPUs are also used to accelerate the query process. Experiments conducted on both Intel(R) HD Graphics 4400 and Nvidia Geforce GTX1050 TI, as well as theoretical analysis show that the proposed algorithm yields significant improvements and outperforms current k-d tree based nearest neighbor query algorithms and others.

In this paper, we introduce data clustering based on infinite mixture models for spherical patterns. This particular clustering is based on Langevin distribution which has been shown to be effective to model this kind of data. The proposed learning algorithm is tackled using a fully Bayesian approach. In contrast to classical Bayesian approaches, which suppose an unknown finite number of mixture components, proposed approach assumes an infinite number of components and have witnessed considerable theoretical and computational advances in recent years. In particular, we have developed a Markov Chain Monte Carlo (MCMC) algorithm to sample from the posterior distributions associated with the selected priors for the different model parameters. Moreover, we propose an infinite framework that allows simultaneous feature selection selection and parameter estimation. The usefulness of the developed framework has been shown via topic novelty detection application.

Clustering has been a subject of extensive research in data mining, pattern recognition, and other areas for several decades. The main goal is to assign samples, which are typically non-Gaussian and expressed as points in high-dimensional feature spaces, to one of a number of clusters. It is well known that in such high-dimensional settings, the existence of irrelevant features generally compromises modeling capabilities. In this paper, we propose a variational inference framework for unsupervised non-Gaussian feature selection, in the context of finite generalized Dirichlet (GD) mixture-based clustering. Under the proposed principled variational framework, we simultaneously estimate, in a closed form, all the involved parameters and determine the complexity (i.e., both model an feature selection) of the GD mixture. Extensive simulations using synthetic data along with an analysis of real-world data and human action videos demonstrate that our variational approach achieves better results than comparable techniques.

The work proposed in this paper is motivated by the need to develop powerful models and approaches to classify and learn proportional data. Indeed, an abundance of interesting data in several applications occur naturally in this form. Our goal is to discover and capture the intrinsic nature of the data by proposing some approaches that combine the major advantages of generative models namely finite mixtures and discriminative techniques namely support vector machines (SVMs). Indeed, SVMs often rely on classic kernels which are not generally meaningful for proportional data. One serious limitation of these kernels is that they do not take into account the nature of data to classify and choosing a suitable kernel continues to be a formidable challenge for data mining and machine learning researchers. Our approach builds on selecting accurate kernels generated from finite mixtures of Dirichlet, generalized Dirichlet and Beta-Liouville distributions which chief advantage is their flexibility and explanatory capabilities in the case of heterogenous proportional data. Using extensive simulations and a number of experiments involving scene modeling and classification, and automatic image orientation detection, we show the merits of the proposed mixture models and the accuracy of the generated kernels.

In recent years, an increasing number of security threats have brought a serious risk to the internet and computer networks. Intrusion Detection System (IDS) plays a vital role in detecting various kinds of attacks. Developing adaptive and flexible oriented IDSs remains a challenging and demanding task due to the incessantly appearance of new types of attacks and sabotaging approaches. In this paper, we propose a novel unsupervised statistical approach for detecting network based attacks. In our approach, patterns of normal and intrusive activities are learned through finite generalized Dirichlet mixture models, in the context of Bayesian variational inference. Under the proposed variational framework, the parameters, the complexity of the mixture model, and the features saliency can be estimated simultaneously, in a closed-form. We evaluate the proposed approach using the popular KDD CUP 1999 data set. Experimental results show that this approach is able to detect many different types of intrusions accurately with a low false positive rate.

In this paper, we consider the problem of unsupervised discrete feature selection/weighting. Indeed, discrete data are an important component in many data mining, machine learning, image processing, and computer vision applications. However, much of the published work on unsupervised feature selection has concentrated on continuous data. We propose a probabilistic approach that assigns relevance weights to discrete features that are considered as random variables modeled by finite discrete mixtures. The choice of finite mixture models is justified by its flexibility which has led to its widespread application in different domains. For the learning of the model, we consider both Bayesian and information-theoretic approaches through stochastic complexity. Experimental results are presented to illustrate the feasibility and merits of our approach on a difficult problem which is clustering and recognizing visual concepts in different image data. The proposed approach is successfully applied also for text clustering.

In this paper, we examine the problem of count data clustering. We analyze this problem using finite mixtures of distributions. The multinomial distribution and the multinomial Dirichlet distribution (MDD) are widely accepted to model count data. We show that these two distributions cannot be the best choice in all the applications, and we propose another model called the multinomial generalized Dirichlet distribution (MGDD) that is the composition of the generalized Dirichlet distribution and the multinomial, in the same way that the MDD is the composition of the Dirichlet and the multinomial. The estimation of the parameters and the determination of the number of components in our model are based on the deterministic annealing expectation-maximization (DAEM) approach and the minimum description length (MDL) criterion, respectively. We compare our method to standard approaches such as multinomial and multinomial Dirichlet mixtures to show its merits. The comparison involves different applications such as spatial color image databases indexing, handwritten digit recognition, and text document clustering.

This paper proposes an unsupervised algorithm for learning a finite Dirichlet mixture model. An important part of the unsupervised learning problem is determining the number of clusters which best describe the data. We extend the minimum message length (MML) principle to determine the number of clusters in the case of Dirichlet mixtures. Parameter estimation is done by the expectation-maximization algorithm. The resulting method is validated for one-dimensional and multidimensional data. For the one-dimensional data, the experiments concern artificial and real SAP image histograms. The validation for multidimensional data involves synthetic data and two real applications: shadow detection in images and summarization of texture image databases for efficient retrieval. A comparison with results obtained for other selection criteria is provided