Benchmarks have emerged as the central approach for evaluating Large Language Models (LLMs).The research community often relies on a model's average performance across the test prompts of a benchmark to evaluate the model's performance.This is consistent with the assumption that the test prompts within a benchmark represent a random sample from a real-world distribution of interest.We note that this is generally not the case; instead, we hold that the distribution of interest varies according to the specific use case.We find that (1) the correlation in model performance across test prompts is non-random, (2) accounting for correlations across test prompts can change model rankings on major benchmarks, (3) explanatory factors for these correlations include semantic similarity and common LLM failure points.

Classical approaches for approximate inference depend on cleverly designed variational distributions and bounds. Modern approaches employ amortized variational inference, which uses a neural network to approximate any posterior without leveraging the structures of the generative models. In this paper, we propose Amortized Conjugate Posterior (ACP), a hybrid approach taking advantages of both types of approaches. Specifically, we use the classical methods to derive specific forms of posterior distributions and then learn the variational parameters using amortized inference. We study the effectiveness of the proposed approach on the Noisy-OR model and compare to both the classical and the modern approaches for approximate inference and parameter learning. Our results show that the proposed method outperforms or are at par with other approaches.

Text data co-clustering is the process of partitioning the documents and words simultaneously. This approach has proven to be more useful than traditional one-sided clustering when dealing with sparsity. Among the wide range of co-clustering approaches, Non-Negative Matrix Tri-Factorization (NMTF) is recognized for its high performance, flexibility and theoretical foundations. One important aspect when dealing with text data, is to capture the semantic relationships between words since documents that are about the same topic may not necessarily use exactly the same vocabulary. However, this aspect has been overlooked by previous co-clustering models, including NMTF. To address this issue, we rely on the distributional hypothesis stating that words which co-occur frequently within the same context, e.g., a document or sentence, are likely to have similar meanings. We then propose a new NMTF model that maps frequently co-occurring words roughly to the same direction in the latent space to reflect the relationships between them. To infer the factor matrices, we derive a scalable alternating optimization algorithm, whose convergence is guaranteed. Extensive experiments, on several real-world datasets, provide strong evidence for the effectiveness of the proposed approach, in terms of co-clustering.

Several recent studies have shown the benefits of combining language and perception to infer word embeddings.These multimodal approaches either simply combine pre-trained textual and visual representations (e.g.features extracted from convolutional neural networks), or use the latter to bias the learning of textual word embeddings.In this work, we propose a novel probabilistic model to formalize how linguistic and perceptual inputs can work in concert to explain the observed word-context pairs in a text corpus.Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data.Extensive experimental studies validate the proposed model.Concretely, on the tasks of assessing pairwise word similarity and image/caption retrieval, our approach attains equally competitive or stronger results when compared to other state-of-the-art multimodal models.

We present a novel generative mixture model for co-clustering text data. This model, the Sparse Poisson Latent Block Model (SPLBM), is based on the Poisson distribution, which arises naturally for contingency tables, such as document-term matrices. The advantages of SPLBM are two-fold. First, it is a rigorous statistical model which is also very parsimonious. Second, it has been designed from the ground up to deal with data sparsity problems. Extensive experiments on various real-world datasets of different size and structure provide strong evidence for the effectiveness of the proposed approach.

Semi-Non Negative Matrix Factorization (Semi-NMF) is one of the most popular extensions of NMF, it extends the applicable range of NMF models, to data having mixed signs, as well as strengthens their relation to clustering. However, Semi-NMF has been found to perform somewhat less than NMF, in terms of clustering, when applied to positive data such as text, which we focus on. Inspired by the recent success of neural word embedding models, e.g., word2vec, in learning high quality real valued vector representations of words, we propose to integrate a word embedding model into Semi-NMF. This allows Semi-NMF to capture more semantic relationships among words and, thereby, to infer document factors that are even better for clustering. The combination of Semi-NMF and word embedding noticeably improves the performance of NMF models, in terms of both clustering and embedding, as illustrated in our experiments.

Document clustering is central in modern information retrieval applications. Among existing models, non-negative-matrix factorization (NMF) approaches have proven effective for this task. However, NMF approaches, like other models in this context, exhibit a major drawback, namely they use the bag-of-word representation and, thus, do not account for the sequential order in which words occur in documents. This is an important issue since it may result in a significant loss of semantics. In this paper, we aim to address the above issue and propose a new model which successfully integrates a word embedding model, word2vec, into an NMF framework so as to leverage the semantic relationships between words. Empirical results, on several real-world datasets, demonstrate the benefits of our model in terms of text document clustering as well as document/word embedding.

Over the last decades, several studies have demonstrated the importance of co-clustering to simultaneously produce groups of objects and features. Even to obtain object clusters only, using co-clustering is often more effective than one-way clustering, especially when considering sparse high dimensional data. In this paper, we present a novel generative mixture model for co-clustering such data. This model, the Sparse Poisson Latent Block Model (SPLBM), is based on the Poisson distribution, which arises naturally for contingency tables, such as document-term matrices. The advantages of SPLBM are two-fold. First, it is a rigorous statistical model which is also very parsimonious. Second, it has been designed from the ground up to deal with data sparsity problems. As a consequence, in addition to seeking homogeneous blocks, as other available algorithms, it also filters out homogeneous but noisy ones due to the sparsity of the data. Experiments on various datasets of different size and structure show that an algorithm based on SPLBM clearly outperforms state-of-the-art algorithms. Most notably, the SPLBM-based algorithm presented here succeeds in retrieving the natural cluster structure of difficult, unbalanced datasets which other known algorithms are unable to handle effectively.

We present Coclus, a novel diagonal co-clustering algorithm which is able to effectively co-cluster binary or contingency matrices by directly maximizing an adapted version of the modularity measure traditionally used for networks. While some effective co-clustering algorithms already exist that use network-related measures (normalized cut, modularity), they do so by using spectral relaxations of the discrete optimization problems. In contrast, Coclus allows to get even better co-clusters by directly maximizing modularity using an iterative alternating optimization procedure. Extensive comparative experiments performed on various document-term datasets demonstrate that our algorithm is very effective, stable and outperforms other co-clustering algorithms.