Deep clustering nowadays has proven to significantly surpass the classical clustering method, so it has been widely used in diverse applications. One current branch of deep clustering methods enhances the primary task through auxiliary tasks, among which the most prevalent is over-clustering, i.e., jointly training clustering with different numbers of clusters in a multi task manner. However, existing approaches typically treat these auxiliary tasks in isolation and neglect the inherent correlations among their cluster assignments. In this paper, we interpret the cluster assignment memberships of samples generated by all clustering tasks as correlated pseudo-labels. Motivated by this observation, we propose to explicitly exploit such correlation knowledge to improve clustering performance. To achieve this, we can formulate the collection of samples with pseudo-labels as a pseudo-multi-label learning problem, and solve it by employing any off-the-shelf multi-label learning methods which enable to capture correlations between pseudo-labels. Based on this idea, beyond the clustering tasks, we propose a correlation learning auxiliary task, namely Self-supervised Multi-Label Learning (SMLL); and we then specify a novel deep clustering method with SMLL, namely DCSL3. We conduct several experiments to examine the performance of DCSL3on benchmark datasets. Empirical results demonstrate the superiority of DCSL3over the existing deep clustering baseline methods.

In the domain of model-based diagnosis (MBD), the identification of the most probable faulty components entails the initial computation of candidate diagnoses across all system elements, followed by the application of Bayesian inference to derive their posterior failure probabilities. However, this conventional approach necessitates the extraction of minimal conflict sets (MCSs) for all components—a prerequisite for generating candidate diagnoses—and subsequently solving for minimal hitting sets (MHSs) of the MCSs. Both tasks are inherently NP-hard, imposing prohibitive computational complexity as system scale increases. Even most advanced diagnostic algorithms encounter significant challenges in enumerating all diagnoses, or even a cardinality-minimal solution, within tractable time constraints for large-scale systems. To address these limitations, this work introduces a novel incremental methodology for efficiently approximating posterior component failure probabilities. A foundational framework is first proposed, leveraging structural relationships inherent to hitting sets to probabilistically characterize component fault likelihoods. Building upon this foundation, two minimization theorems are formally established, accompanied by closed-form parameterizations to optimize computational efficiency. Crucially, the proposed method bypasses the explicit enumeration of diagnoses by directly inferring the most probable faulty components from conflict set analyses. Empirical evaluations demonstrate that the approach not only sustains diagnostic accuracy exceeding 95% but also achieves a substantial computational acceleration—surpassing contemporary state-of-the-art algorithms by multiple orders of magnitude.

ABSTRACT Active domain adaptation (active DA) provides an effective solution by selectively labelling a limited number of target samples to significantly enhance adaptation performance. However, existing active DA methods often struggle in real‐world scenarios where, due to data privacy concerns, only a pre‐trained source model is available, rather than the source samples. To address this issue, we propose a novel method called the structure‐based uncertainty estimation model (SUEM) for source‐free active domain adaptation (SFADA). To be specific, we introduce an innovative active sample selection strategy that combines both uncertainty and diversity sampling to identify the most informative samples. We assess the uncertainty in target samples using structure‐wise probabilities and implement a diversity selection method to minimise redundancy. For the selected samples, we not only apply standard‐supervised loss but also conduct interpolation consistency training to further explore the structural information of the target domain. Extensive experiments across four widely used datasets demonstrate that our method is comparable to or outperforms current UDA and active DA methods.

Variational Inference (VI) has gained popularity as a flexible approximate inference scheme for computing posterior distributions in Bayesian models. Original VI methods use Kullback-Leibler (KL) divergence to construct variational objectives. However, KL divergence has zero-forcing behavior and is completely agnostic to the metric of the underlying data distribution, resulting in bad approximations. To alleviate this issue, we propose a new variational objective by using Optimal Transport (OT) distance, which is a metric-aware divergence, to measure the difference between approximate posteriors and priors. The superior performance of OT distance enables us to learn more accurate approximations. We further enhance the objective by gradually including the OT term using a hyperparameter λ for over-parameterized models. We develop a Variational inference method with OT (VOT) which presents a gradient-based black-box framework for solving Bayesian models, even when the density function of approximate distribution is not available. We provide the consistency analysis of approximate posteriors and demonstrate the practical effectiveness on Bayesian neural networks and variational autoencoders.

Aspect-based sentiment analysis (ABSA), a fine-grained sentiment classification task, has received much attention recently. Many works investigate sentiment information through opinion words, such as "good'' and "bad''. However, implicit sentiment data widely exists in the ABSA dataset, whose sentiment polarity is hard to determine due to the lack of distinct opinion words. To deal with implicit sentiment, this paper proposes an ABSA method that integrates explicit sentiment augmentations (ABSA-ESA) to add more sentiment clues. We propose an ABSA-specific explicit sentiment generation method to create such augmentations. Specifically, we post-train T5 by rule-based data and employ three strategies to constrain the sentiment polarity and aspect term of the generated augmentations. We employ Syntax Distance Weighting and Unlikelihood Contrastive Regularization in the training procedure to guide the model to generate the explicit opinion words with the same polarity as the input sentence. Meanwhile, we utilize the Constrained Beam Search to ensure the augmentations are aspect-related. We test ABSA-ESA on two ABSA benchmarks. The results show that ABSA-ESA outperforms the SOTA baselines on implicit and explicit sentiment accuracy.

Positive and Unlabeled (PU) learning refers to a special case of binary classification, and technically, it aims to induce a binary classifier from a few labeled positive training instances and loads of unlabeled instances. In this paper, we derive a theorem indicating that the probability boundary of the asymmetric disambiguation-free expected risk of PU learning is controlled by its asymmetric penalty, and we further empirically evaluated this theorem. Inspired by the theorem and its empirical evaluations, we propose an easy-to-implement two-stage PU learning method, namely **P**ositive and **U**nlabeled **L**earning with **C**ontrolled **P**robability **B**oundary **F**ence (**PULCPBF**). In the first stage, we train a set of weak binary classifiers concerning different probability boundaries by minimizing the asymmetric disambiguation-free empirical risks with specific asymmetric penalty values. We can interpret these induced weak binary classifiers as a probability boundary fence. For each unlabeled instance, we can use the predictions to locate its class posterior probability and generate a stochastic label. In the second stage, we train a strong binary classifier over labeled positive training instances and all unlabeled instances with stochastic labels in a self-training manner. Extensive empirical results demonstrate that PULCPBF can achieve competitive performance compared with the existing PU learning baselines.

Wasserstein barycenter, built on the theory of Optimal Transport (OT), provides a powerful framework to aggregate probability distributions, and it has increasingly attracted great attention within the machine learning community. However, it is often intractable to precisely compute, especially for high dimensional and continuous settings. To alleviate this problem, we develop a novel regularization by using the fact that c-cyclical monotonicity is often necessary and sufficient conditions for optimality in OT problems, and incorporate it into the dual formulation of Wasserstein barycenters. For efficient computations, we adopt a variational distribution as the approximation of the true continuous barycenter, so as to frame the Wasserstein barycenters problem as an optimization problem with respect to variational parameters. Upon those ideas, we propose a novel end-to-end continuous approximation method, namely Variational Wasserstein Barycenters with c-Cyclical Monotonicity Regularization (VWB-CMR), given sample access to the input distributions. We show theoretical convergence analysis and demonstrate the superior performance of VWB-CMR on synthetic data and real applications of subset posterior aggregation.

Partial Label (PL) learning refers to the task of learning from the partially labeled data, where each training instance is ambiguously equipped with a set of candidate labels but only one is valid. Advances in the recent deep PL learning literature have shown that the deep learning paradigms, e.g., self-training, contrastive learning, or class activate values, can achieve promising performance. Inspired by the impressive success of deep Semi-Supervised (SS) learning, we transform the PL learning problem into the SS learning problem, and propose a novel PL learning method, namely Partial Label learning with Semi-supervised Perspective (PLSP). Specifically, we first form the pseudo-labeled dataset by selecting a small number of reliable pseudo-labeled instances with high-confidence prediction scores and treating the remaining instances as pseudo-unlabeled ones. Then we design a SS learning objective, consisting of a supervised loss for pseudo-labeled instances and a semantic consistency regularization for pseudo-unlabeled instances. We further introduce a complementary regularization for those non-candidate labels to constrain the model predictions on them to be as small as possible. Empirical results demonstrate that PLSP significantly outperforms the existing PL baseline methods, especially on high ambiguity levels. Code available: https://github.com/changchunli/PLSP.

Learning from only few samples is a challenging problem and meta-learning is an effective approach to solve it. Meta-learning model aims to learn by training a large number of other samples. When encountering target task, the model can quickly adapt and obtain better performance with only few labeled samples. However, general meta-learning only provides a universal model that has certain generalization ability for all unknown tasks, which causes limited effects on specific target tasks. In this paper, we propose a Few-shot Directed Meta-learning (FSDML) model to specialize and solve the target task by using few labeled samples of the target task to direct the meta-learning process. FSDML divides model parameters into shared parameters and target adaptation parameters to store prior knowledge and determine the update direction. These two parts of the parameters are updated in different stages of training. We conduct experiments of image classification task on miniImageNet and Omniglot and the results show that FSDML has better performance.

Weakly-supervised text classification aims to train predictive models with unlabeled texts and a few representative words of classes, referred to as category words, rather than labeled texts. These weak supervisions are much more cheaper and easy to collect in real-world scenarios. To resolve this task, we propose a novel deep classification model, namely Weakly-supervised Text Classification with Wasserstein Barycenter Regularization (WTC-WBR). Specifically, we initialize the pseudo-labels of texts by using the category word occurrences, and formulate a weakly self-training framework to iteratively update the weakly-supervised targets by combining the pseudo-labels with the sharpened predictions. Most importantly, we suggest a Wasserstein barycenter regularization with the weakly-supervised targets on the deep feature space. The intuition is that the texts tend to be close to the corresponding Wasserstein barycenter indicated by weakly-supervised targets. Another benefit is that the regularization can capture the geometric information of deep feature space to boost the discriminative power of deep features. Experimental results demonstrate that WTC-WBR outperforms the existing weakly-supervised baselines, and achieves comparable performance to semi-supervised and supervised baselines.

Optimal Transport (OT) has become a powerful tool to compare probability distributions. However, it suffers from a severe computational burden for high dimensional and continuous distributions. To this end, we develop two novel methods for the Kantorovich and Monge formulations, which are the fundamental problems in OT. First, we learn the optimal joint distribution in the Kantorovich formulation and propose an algorithm, namely Cop-OT, which transforms the primal objective of the Kantorovich problem into a tractable objective with respect to the copula parameter. Second, based on the copula formulation of the joint distribution, we learn the optimal map in the Monge problem and propose an algorithm, namely Map-OT, which describes the optimal map using a parameterized function estimated by approximating the barycentric projection of the optimal joint distribution and then obtains a tractable objective with respect to parameters of interest. Both of them can be solved by stochastic optimization with a stable optimizing process. Empirical results demonstrate that Cop-OT and Map-OT can gain more accurate approximations of the Kantorovich and Monge problems compared with the baseline methods.

Mean-field variational inference, built on fully factorisations, can be efficiently solved; however, it ignores the dependencies between latent variables, resulting in lower performance. To address this, the copula variational inference (CVI) method is proposed by using the well-established copulas to effectively capture posterior dependencies, leading to better approximations. However, it suffers from a computational issue, where the optimisation for big models with massive latent variables is quite time-consuming. This is mainly caused by the expensive sampling when forming noisy Monte Carlo gradients in CVI. For CVI speedup, in this paper we propose a novel fast CVI (abbr. FCVI). In FCVI, we derive the gradient of CVI objective by an expectation of the mean-field factorisation. Therefore, we can achieve a much efficient sampling from the D-dimensional mean-field factorisation, enabling to reduce the sampling complexity from O(D2) to O(D). To evaluate FCVI, we compare it against baseline methods on modelling performance and runtime. Experimental results demonstrate that FCVI is on a par with CVI, but runs much faster.

Changchun Li, Ximing Li, Jihong Ouyang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Ambiguous Label Learning (ALL), as an emerging paradigm of weakly supervised learning, aims to induce the prediction model from training datasets with ambiguous supervision, where, specifically, each training instance is annotated with a set of candidate labels but only one is valid. To handle this task, the existing shallow methods mainly disambiguate the candidate labels by leveraging various regularization techniques. Inspired by the great success of deep generative adversarial networks, we apply it to perform effective candidate label disambiguation from a new instance-pivoted perspective. Specifically, for each ALL instance, we recombine its feature representation with each of candidate labels to generate a set of candidate instances, where only one is real and all others are fake. We formulate a unified adversarial objective with respect to three players, i.e., a discriminator, a generator, and a classifier. The discriminator is used to detect the fake candidate instances, so that the classifier can be trained without them. With this insight, we develop a novel ALL method, namely Adversarial Ambiguous Label Learning with Candidate Instance Detection (A2L2CID). Theoretically, we analyze that there is a global equilibrium point between the three players. Empirically, extensive experimental results indicate that A2L2CID outperforms the state-of-the-art ALL methods.

Neural topic modeling provides a flexible, efficient, and powerful way to extract topic representations from text documents. Unfortunately, most existing models cannot handle the text data with network links, such as web pages with hyperlinks and scientific papers with citations. To resolve this kind of data, we develop a novel neural topic model , namely Layer-Assisted Neural Topic Model (LANTM), which can be interpreted from the perspective of variational auto-encoders. Our major motivation is to enhance the topic representation encoding by not only using text contents, but also the assisted network links. Specifically, LANTM encodes the texts and network links to the topic representations by an augmented network with graph convolutional modules, and decodes them by maximizing the likelihood of the generative process. The neural variational inference is adopted for efficient inference. Experimental results validate that LANTM significantly outperforms the existing models on topic quality, text classification and link prediction..

In real-world scenarios, the data are widespread that are annotated with a set of candidate labels but a single ground-truth label per-instance. The learning paradigm with such data, formally referred to as Partial Label (PL) learning, has recently drawn much attention. The traditional PL methods estimate the confidences being the ground-truth label of candidate labels with various regularizations and constraints, however, they only consider the local information, resulting in potentially less accurate estimations as well as worse classification performance. To alleviate this problem, we propose a novel PL method, namely PArtial label learNing by simultaneously leveraging GlObal and Local consIsteNcies (Pangolin). Specifically, we design a global consistency regularization term to pull instances associated with similar labeling confidences together by minimizing the distances between instances and label prototypes, and a local consistency term to push instances marked with no same candidate labels away by maximizing their distances. We further propose a nonlinear kernel extension of Pangolin, and employ the Taylor approximation trick for efficient optimization. Empirical results demonstrate that Pangolin significantly outperforms the existing PL baseline methods.

Wasserstein distance, defined as the cost (measured by word embeddings) of optimal transport plan for moving between two histograms, has been proven effective in tasks of natural language processing. In this paper, we extend Nonnegative Matrix Factorization (NMF) to a novel Wasserstein topic model, namely Semantics-Assisted Wasserstein Learning (SAWL), with simultaneous learning of topics and word embeddings. In Sawl, we formulate an NMF-like unified objective that integrates the regularized Wasserstein distance loss with a context factorization of word context information. Therefore, Sawl can refine the word embeddings for capturing corpus-specific semantics, enabling to boost topics and word embeddings each other. We analyze Sawl, and provide its dimensionality-dependent generalization bounds of reconstruction errors. Experimental results indicate that Sawl outperforms the state-of-the-art baseline models.

Conventional topic models suffer from a severe sparsity problem when facing extremely short texts such as social media posts. The family of Dirichlet multinomial mixture (DMM) can handle the sparsity problem, however, they are still very sensitive to ordinary and noisy words, resulting in inaccurate topic representations at the document level. In this paper, we alleviate this problem by preserving local neighborhood structure of short texts, enabling to spread topical signals among neighboring documents, so as to correct the inaccurate topic representations. This is achieved by using variational manifold regularization, constraining the close short texts should have similar variational topic representations. Upon this idea, we propose a novel Laplacian DMM (LapDMM) topic model. During the document graph construction, we further use the word mover’s distance with word embeddings to measure document similarities at the semantic level. To evaluate LapDMM, we compare it against the state-of-theart short text topic models on several traditional tasks. Experimental results demonstrate that our LapDMM achieves very significant performance gains over baseline models, e.g., achieving even about 0.2 higher scores on clustering and classification tasks in many cases.

Optimal Transport (OT) formulates a powerful framework by comparing probability distributions, and it has increasingly attracted great attention within the machine learning community. However, it suffers from severe computational burden, due to the intractable objective with respect to the distributions of interest. Especially, there still exist very few attempts for continuous OT, i.e., OT for comparing continuous densities. To this end, we develop a novel continuous OT method, namely Copula OT (Cop-OT). The basic idea is to transform the primal objective of continuous OT into a tractable form with respect to the copula parameter, which can be efficiently solved by stochastic optimization with less time and memory requirements. Empirical results on real applications of image retrieval and synthetic data demonstrate that our Cop-OT can gain more accurate approximations to continuous OT values than the state-of-the-art baselines.

Automatically classifying extremely short texts, such as social media posts and web page titles, plays an important role in a wide range of content analysis applications. However, traditional classifiers based on bag-of-words (BoW) representations often fail in this task. The underlying reason is that the document similarity can not be accurately measured under BoW representations due to the extreme sparseness of short texts. This results in significant difficulty to capture the generality of short texts. To address this problem, we use a better regularized word mover's distance (RWMD), which can measure distances among short texts at the semantic level. We then propose a RWMD-based centroid classifier for short texts, named RWMD-CC. Basically, RWMD-CC computes a representative semantic centroid for each category under the RWMD measure, and predicts test documents by finding the closest semantic centroid. The testing is much more efficient than the prior art of K nearest neighbor classifier based on WMD. Experimental results indicate that our RWMD-CC can achieve very competitive classification performance on extremely short texts.

Short texts, such as Twitter social media posts, have become increasingly popular on the Internet. Inferring topics from massive numbers of short texts is important to many real-world applications. A single short text often contains a few words, making traditional topic models less effective. A recently developed biterm topic model (BTM) effectively models short texts by capturing the rich global word co-occurrence information. However, in the sparse short-text context, many highly related words may never co-occur. BTM may lose many potential coherent and prominent word co-occurrence patterns that cannot be observed in the corpus. To address this problem, we propose a novel relational BTM (R-BTM) model, which links short texts using a similarity list of words computed employing word embeddings. To evaluate the effectiveness of R-BTM, we compare it against the existing short-text topic models on a variety of traditional tasks, including topic quality, clustering and text similarity. Experimental results on real-world datasets indicate that R-BTM outperforms baseline topic models for short texts.

Digital cameras that use Color Filter Arrays (CFA) entail a demosaicking procedure to form full RGB images. To digital camera industry, demosaicking speed is as important as demosaicking accuracy, because camera users have been accustomed to viewing captured photos instantly. Moreover, the cost associated with demosaicking should not go beyond the cost saved by using CFA. For this purpose, we revisit the classical Hamilton-Adams (HA) algorithm, which outperforms many sophisticated techniques in both speed and accuracy. Our analysis shows that the HA pipeline is highly efficient to exploit the originally captured data, but its oversimplified inter- and intra-channel smoothness formulation hinders its accuracy. We therefore propose a very low cost edge sensing scheme, which guides demosaicking by a logistic functional of the difference between directional variations. We extensively compare our algorithm with 27 demosaicking algorithms by running their open source codes on benchmark datasets. Compared to methods of similar computational cost, our method achieves substantially higher accuracy; Whereas compared to methods of similar accuracy, our method has significantly lower cost. On test images of currently popular resolution, the quality of our algorithm is comparable to top performers, yet our speed is tens of times faster. Source code for this work will be released with paper publication.

Recently, dataless text classification has attracted increasing attention. It trains a classifier using seed words of categories, rather than labeled documents that are expensive to obtain. However, a small set of seed words may provide very limited and noisy supervision information, because many documents contain no seed words or only irrelevant seed words. In this paper, we address these issues using document manifold, assuming that neighboring documents tend to be assigned to a same category label. Following this idea, we propose a novel Laplacian seed word topic model (LapSWTM). In LapSWTM, we model each document as a mixture of hidden category topics, each of which corresponds to a distinctive category. Also, we assume that neighboring documents tend to have similar category topic distributions. This is achieved by incorporating a manifold regularizer into the log-likelihood function of the model, and then maximizing this regularized objective. Experimental results show that our LapSWTM significantly outperforms the existing dataless text classification algorithms and is even competitive with supervised algorithms to some extent. More importantly, it performs extremely well when the seed words are scarce.

Overdispersed black-box variational inference employs importance sampling to reduce the variance of the Monte Carlo gradient in black-box variational inference. A simple overdispersed proposal distribution is used. This paper aims to investigate how to adaptively obtain better proposal distribution for lower variance. To this end, we directly approximate the optimal proposal in theory using a Monte Carlo moment matching step at each variational iteration. We call this adaptive proposal moment matching proposal (MMP). Experimental results on two Bayesian models show that the MMP can effectively reduce variance in black-box learning, and perform better than baseline inference algorithms.

In this paper, we develop a generic black-box expectation propagation (BBEP) algorithm that can be directly applied to Bayesian models without model-specific derivations. BBEP is built on the spirit of using Monte Carlo estimates, where the moment matching step in EP is replaced with Monte Carlo approximations. To avoid high variance, we employ importance sampling for variance reduction and analyze how to find an optimal proposal distribution. We compare BBEP against the state-of-the-art black-box algorithms on both synthetic and real-world data sets. The experimental results indicate that BBEP can reach better predictive performance than baseline algorithms, and even can be on a par with analytical solutions in some settings.

As an efficient sampling algorithm for latent dirichlet allocation SparseLDA uses cache strategy to improve the time and space efficiency of its standard gibbs sampling algorithm (StdGibbs) by recycling previous computation. However, SparseLDA cannot further improve the time-efficiency of StdGibbs, since the amount of recycled computation is limited. This is because the word types of two adjacent tokens are usually different and the previous computation cannot be further recycled easily. To solve this problem, in this paper we propose a new algorithm named Efficient SparseLDA (ESparseLDA) based on SparseLDA. The main idea of ESparseLDA is to first rearrange the tokens within one text according to the word types so that the tokens of the same word type are aggregated together and then recycle more computation while making no approximation and ensuring the exactness. In this paper, we make detailed theoretical explanations and comparative experimental analyses on the correctness, exactness and time-efficiency of ESparseLDA. In detail, the statistical significance tests on perplexities strictly show that ESparseLDA is correct and exact. In addition, the running time results show that the time-efficiency of ESparseLDA is the higher than SparseLDA in varying degrees from 5.06% to 31.85% on the different datasets used in experiments.

In topic modelling, the main computational problem is to approximate the posterior distribution given an observed collection. Commonly, we must resort to variational methods for approximations; however, we do not know which variational variant is the best choice under certain settings. In this paper, we focus on four topic modelling inference methods, including mean-field variation Bayesian, collapsed variational Bayesian, hybrid variational-Gibbs and expectation propagation, and aim to systematically compare them. We analyse them from two perspectives, i.e. the approximate posterior distribution and the type of -divergence; and then empirically compare them on various data-sets by two popular metrics. The empirical results are almost matching our analysis, where they indicate that CVB0 may be the best variational variant for topic models.

Supervised topic modeling algorithms have been successfully applied to multi-label document classification tasks. Representative models include labeled latent Dirichlet allocation (L-LDA) and dependency-LDA. However, these models neglect the class frequency information of words (i.e., the number of classes where a word has occurred in the training data), which is significant for classification. To address this, we propose a method, namely the class frequency weight (CF-weight), to weight words by considering the class frequency knowledge. This CF-weight is based on the intuition that a word with higher (lower) class frequency will be less (more) discriminative. In this study, the CF-weight is used to improve L-LDA and dependency-LDA. A number of experiments have been conducted on real-world multi-label datasets. Experimental results demonstrate that CF-weight based algorithms are competitive with the existing supervised topic models.

Gaussian LDA integrates topic modeling with word embeddings by replacing discrete topic distribution over word types with multivariate Gaussian distribution on the embedding space. This can take semantic information of words into account. However, the Euclidean similarity used in Gaussian topics is not an optimal semantic measure for word embeddings. Acknowledgedly, the cosine similarity better describes the semantic relatedness between word embeddings. To employ the cosine measure and capture complex topic structure, we use von Mises-Fisher (vMF) mixture models to represent topics, and then develop a novel mix-vMF topic model (MvTM). Using public pre-trained word embeddings, we evaluate MvTM on three real-world data sets. Experimental results show that our model can discover more coherent topics than the state-of-the-art baseline models, and achieve competitive classification performance.

Topic modeling algorithms such as the latent Dirichlet allocation (LDA) play an important role in machine learning research. Fitting LDA using Gibbs sampler-related algorithms involves a sampling process over K topics. We can use the sparsity in LDA to accelerate this expensive topic sampling process even for very large K values. However, LDA gradually loses sparsity as the number of documents increases. Motivated by the goal of fast LDA inference with large numbers of both topics and documents, in this paper we propose the novel sparse hybrid variational-Gibbs (SHVG) algorithm. The SHVG algorithm divides the topic sampling probability into a sparse term that scales linearly with the number of per-document instantiated topics Kd, and a dense term that uses the Alias method to reduce the time cost to constant O(1) time. This will lead to a significant improvement on efficiency. Using stochastic optimization techniques, we further develop an online version of SHVG for streaming documents. Experimental results on corpora with a wide range of sizes demonstrate the efficiency and effectiveness of the proposed SHVG algorithm.

The centroid-based classifier is both effective and efficient for document classification. However, it suffers from over-fitting and linear inseparability problems caused by its fundamental assumptions. To address these problems, we propose a kernel-based hypothesis margin centroid classifier (KHCC). First, KHCC optimises the class centroids via minimising hypothesis margin under structural risk minimisation principle; second, KHCC uses the kernel method to relieve the problem of linear inseparability in the original feature space. Given the radial basis function, we further discuss a guideline for tuning the value of its parameter. The experimental results on four well-known data-sets indicate that our KHCC algorithm outperforms the state-of-the-art algorithms, especially for the unbalanced data-set.

This paper develops a novel online algorithm, namely moving average stochastic variational inference (MASVI), which applies the results obtained by previous iterations to smooth out noisy natural gradients. We analyze the convergence property of the proposed algorithm and conduct a set of experiments on two large-scale collections that contain millions of documents. Experimental results indicate that in contrast to algorithms named ‘stochastic variational inference’ and ‘SGRLD’, our algorithm achieves a faster convergence rate and better performance.

Modeling large-scale document collections is a significant direction in machine learning research. Online LDA uses stochastic gradient optimization technology to speed the convergence; however the large noise of stochastic gradients leads to slower convergence and worse performance. In this paper, we employ the momentum term to smooth out the noise of stochastic gradients, and propose an extension of Online LDA, namely Momentum Online Lda (MOLDA). We collect a large-scale corpus consisting of 2M documents to evaluate our model. Experimental results indicate that MOLDA achieves faster convergence and better performance than the state-of-the-art.

This paper deals with a novel local arc length estimator for curves in gray-scale images. The method first estimates a cubic spline curve fit for the boundary points using the gray-level information of the nearby pixels, and then computes the sum of the spline segments’ lengths. In this model, the second derivatives and y coordinates at the knots are required in the computation; the spline polynomial coefficients need not be computed explicitly. We provide the algorithm pseudo code for estimation and preprocessing, both taking linear time. Implementation shows that the proposed model gains a smaller relative error than other state-of-the-art methods.