Large Language Models (LLMs) have demonstrated remarkable performance across various tasks but remain prone to hallucinations.Detecting hallucinations is essential for safetycritical applications, and recent methods leverage attention map properties to this end, though their effectiveness remains limited.In this work, we investigate the spectral features of attention maps by interpreting them as adjacency matrices of graph structures.We propose the LapEigvals method, which utilizes the topk eigenvalues of the Laplacian matrix derived from the attention maps as an input to hallucination detection probes.Empirical evaluations demonstrate that our approach achieves stateof-the-art hallucination detection performance among attention-based methods.Extensive ablation studies further highlight the robustness and generalization of LapEigvals, paving the way for future advancements in the hallucination detection domain.

Large language models (LLMs) have revolutionized natural language processing, yet their tendency to hallucinate poses serious challenges for reliable deployment.Despite numerous hallucination detection methods, their evaluations often rely on ROUGE, a metric based on lexical overlap that misaligns with human judgments.Through comprehensive human studies, we demonstrate that while ROUGE exhibits high recall, its extremely low precision leads to misleading performance estimates.In fact, several established detection methods show performance drops of up to 45.9% when assessed using human-aligned metrics like LLM-as-Judge.Moreover, our analysis reveals that simple heuristics based on response length can rival complex detection techniques, exposing a fundamental flaw in current evaluation practices.We argue that adopting semantically aware and robust evaluation frameworks is essential to accurately gauge the true performance of hallucination detection methods, ultimately ensuring the trustworthiness of LLM outputs.

Knowledge-intensive tasks pose a significant challenge for Machine Learning (ML) techniques. Commonly adopted methods, such as Large Language Models (LLMs), often exhibit limitations when applied to such tasks. Nevertheless, there have been notable endeavours to mitigate these challenges, with a significant emphasis on augmenting LLMs through Knowledge Graphs (KGs). While KGs provide many advantages for representing knowledge, their development costs can deter extensive research and applications. Addressing this limitation, we introduce a framework for enriching embeddings of small-scale domain-specific Knowledge Graphs with well-established general-purpose KGs. Adopting our method, a modest domain-specific KG can benefit from a performance boost in downstream tasks when linked to a substantial general-purpose KG. Experimental evaluations demonstrate a notable enhancement, with up to a 44% increase observed in the Hits@10 metric. This relatively unexplored research direction can catalyze more frequent incorporation of KGs in knowledge-intensive tasks, resulting in more robust, reliable ML implementations, which hallucinates less than prevalent LLM solutions.

Despite impressive advancements in multilingual corpora collection and model training, developing large-scale deployments of multilingual models still presents a significant challenge. This is particularly true for language tasks that are culture-dependent. One such example is the area of multilingual sentiment analysis, where affective markers can be subtle and deeply ensconced in culture.This work presents the most extensive open massively multilingual corpus of datasets for training sentiment models. The corpus consists of 79 manually selected datasets from over 350 datasets reported in the scientific literature based on strict quality criteria. The corpus covers 27 languages representing 6 language families. Datasets can be queried using several linguistic and functional features. In addition, we present a multi-faceted sentiment classification benchmark summarizing hundreds of experiments conducted on different base models, training objectives, dataset collections, and fine-tuning strategies.

The availability of compute and data to train larger and larger language models increases the demand for robust methods of benchmarking the true progress of LM training. Recent years witnessed significant progress in standardized benchmarking for English. Benchmarks such as GLUE, SuperGLUE, or KILT have become a de facto standard tools to compare large language models. Following the trend to replicate GLUE for other languages, the KLEJ benchmark\ (klej is the word for glue in Polish) has been released for Polish. In this paper, we evaluate the progress in benchmarking for low-resourced languages. We note that only a handful of languages have such comprehensive benchmarks. We also note the gap in the number of tasks being evaluated by benchmarks for resource-rich English/Chinese and the rest of the world.In this paper, we introduce LEPISZCZE (lepiszcze is the Polish word for glew, the Middle English predecessor of glue), a new, comprehensive benchmark for Polish NLP with a large variety of tasks and high-quality operationalization of the benchmark.We design LEPISZCZE with flexibility in mind. Including new models, datasets, and tasks is as simple as possible while still offering data versioning and model tracking. In the first run of the benchmark, we test 13 experiments (task and dataset pairs) based on the five most recent LMs for Polish. We use five datasets from the Polish benchmark and add eight novel datasets. As the paper's main contribution, apart from LEPISZCZE, we provide insights and experiences learned while creating the benchmark for Polish as the blueprint to design similar benchmarks for other low-resourced languages.

The self-supervised learning (SSL) paradigm is an essential exploration area, which tries to eliminate the need for expensive data labeling. Despite the great success of SSL methods in computer vision and natural language processing, most of them employ contrastive learning objectives that require negative samples, which are hard to define. This becomes even more challenging in the case of graphs and is a bottleneck for achieving robust representations. To overcome such limitations, we propose a framework for self-supervised graph representation learning — Graph Barlow Twins, which utilizes a cross-correlation-based loss function instead of negative samples. Moreover, it does not rely on non-symmetric neural network architectures — in contrast to state-of-the-art self-supervised graph representation learning method BGRL. We show that our method achieves as competitive results as the best self-supervised methods and fully supervised ones while requiring fewer hyperparameters and substantially shorter computation time (ca. 30 times faster than BGRL).

Kamil Kanclerz, Alicja Figas, Marcin Gruza, Tomasz Kajdanowicz, Jan Kocon, Daria Puchalska, Przemyslaw Kazienko. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

The prevalence of fake news could be observed in circumstances of emotion-causing events, like elections or pandemics. In fear of the potential impact, many fact-checking organisations were established. However, fact-checking requires a large amount of human labor, and hence there is a strong demand for complete automation of this process. Nevertheless, this milestone has not been achieved yet, even for English. The problem grows for the less popular languages that suffer from a scarcity of available resources. To address this problem for the Polish language domain, we propose a solution for automating one of the fact-checking stages - relevance assessment, which is crucial when searching for evidence. Leveraging recent advancements in natural language processing, we have acquired relevant data and developed classifiers of evidence relevance with respect to claims in Polish. Our approach can assess the evidence relevance with a performance at a level of a 0.778 F1-score.

Analysis of subjective texts like offensive content or hate speech is a great challenge, especially regarding annotation process. Most of current annotation procedures are aimed at achieving a high level of agreement in order to generate a high quality reference source. However, the annotation guidelines for subjective content may restrict the annotators’ freedom of decision making . Motivated by a moderate annotation agreement in offensive content datasets, we hypothesize that personalized approaches to offensive content identification should be in place. Thus, we propose two novel perspectives of perception: group-based and individual. Using demographics of annotators as well as embeddings of their previous decisions (annotated texts), we are able to train multimodal models (including transformer-based) adjusted to personal or community profiles. Based on the agreement of individuals and groups, we experimentally showed that annotator group agreeability strongly correlates with offensive content recognition quality. The proposed personalized approaches enabled us to create models adaptable to personal user beliefs rather than to agreed offensiveness understanding. Overall, our individualized approaches to offensive content classification outperform classic data-centric methods that generalize offensiveness perception and it refers to all six tested models. Additionally, we developed requirements for annotation procedures, personalization and content processing to make the solutions human-centered.

Signed distance field (SDF) is a prominent implicit representation of 3D meshes. Methods that are based on such representation achieved state-of-the-art 3D shape reconstruction quality. However, these methods struggle to reconstruct non-convex shapes. One remedy is to incorporate a constructive solid geometry framework (CSG) that represents a shape as a decomposition into primitives. It allows to embody a 3D shape of high complexity and non-convexity with a simple tree representation of Boolean operations. Nevertheless, existing approaches are supervised and require the entire CSG parse tree that is given upfront during the training process. On the contrary, we propose a model that extracts a CSG parse tree without any supervision - UCSG-Net. Our model predicts parameters of primitives and binarizes their SDF representation through differentiable indicator function. It is achieved jointly with discovering the structure of a Boolean operators tree. The model selects dynamically which operator combination over primitives leads to the reconstruction of high fidelity. We evaluate our method on 2D and 3D autoencoding tasks. We show that the predicted parse tree representation is interpretable and can be used in CAD software.

The scikit-multilearn is a Python library for performing multi-label classification. It is compatible with the scikit-learn and scipy ecosystems and uses sparse matrices for all internal operations; provides native Python implementations of popular multi-label classification methods alongside a novel framework for label space partitioning and division and includes modern algorithm adaptation methods, network-based label space division approaches, which extracts label dependency information and multi-label embedding classifiers. The library provides Python wrapped access to the extensive multi-label method stack from Java libraries and makes it possible to extend deep learning single-label methods for multi-label tasks. The library allows multi-label stratification and data set management. The implementation is more efficient in problem transformation than other established libraries, has good test coverage and follows PEP8. Source code and documentation can be downloaded from http://scikit.ml and also via pip The project is BSD-licensed.

In recent years, many studies on urban mobility based on large data sets have been published: most of them are based on crowdsourced GPS data or smart-card data. We present, what is to the best of our knowledge, the first exploration of public transport delay data harvested from a large-scale, official public transport positioning system, provided by the Wrocław municipality. We introduce the methodology to analyze the distribution of delays in public transport, enabling the improvement of timetables by making them more realistic, and thus improve passenger comfort. We evaluate the method considering the characteristics of delays between stops in relation to the direction, time, and delay variance of 1648 stop pairs from 16-mln delay reports. We construct a normalized feature matrix of likelihood of a given delay change happening at a given hour on the edge between two stops. We then calculate the distances between such matrices using the earth mover's distance and cluster them using hierarchical agglomerative clustering with Vor Hees's linkage method. As a result, we obtained six profiles of delay changes in Wrocław: edges nearly not impacting the delay at all, these not impacting the delay significantly, likely to cause strong increase of delay, these causing increase of delay, edges likely to cause strong decrease of delay, and finally these likely to cause decrease of delay (i.e., when a public transport vehicle is speeding). We analyze the spatial and mode of transport properties of each cluster and provide insights into reasons of delay change patterns in each of the detected profiles. Such insights can be successfully utilized in traffic structure optimization and transport model split.

Currently available artificial network generation models are characterized by consistency and low variance due to the rigidity of models' underlying assumptions. Networks generated from these models are usually too regular and do not contain noise and imbalance inherent in networks induced by human behavior. An important consequence is that much research on social network analysis presented in recent years used idealistic artificial networks that did not conform to reality. In order to alleviate this problem we introduce a new network generation model capable of modeling a broad spectrum of networks. In our model, the network formation process is not hard-coded into the model. Rather, we propose a simple mechanism for network creation based on priority ranking, and we encode the guiding principle of network formation as a distance function. By only changing the distance function definition and using the same priority ranking mechanism we are able to model very diverse networks. Our preliminary results show that we can mimic the behavior of popular artificial network generation models, such as the Erdös-Rényi random network model, the Watts-Strogatz small world model, or the Albert-Barabási preferential attachment model, but we can generate new types of networks as well. Following the principles of Open Science we publish the source code used to perform experiments and publish results in a public repository.

It is a common problem that cost of extracting data for network analysis could be very high. Also sometimes in the Internet is it hard to find graph with desired features such as node degree or clustering level. Because of that graph generators can than be very helpful. In the past bunch of models of such generators was developed: random graphs, small worlds and scale free networks. All of these generators were developed to quickly and efficiently create networks with desired parameters. However all of this models produce single layer graphs. Domain of multiplexes or multilayer graphs has not already been so deeply analysed, also because it is hard to collect multilayer data among real datasets or there is hard to define what kind of information layers exactly should represent. Proposed MuNeG --- Multilayer Network Generator can produce, based on set of input parameters, multiplex networks - networks where each node has its counterpart in each layer. The carried out experiments proved that MuNeG graphs have different network and social parameters depends on input values. This feature gives user a very handful tool to generate multiplex networks on purpose of social network or complex network analysis. Generator features, input parameters and their influence on so called graph theory measures such as: node degree, average shortest path, diameter or clustering are described in the following article.

It has been shown in this paper that simplistic Bag of Words (BoW) lexicon methods for sentiment polarity assignment with ensemble classifiers are much faster than a supervised approach to sentiment classification while yielding similar accuracy. BoW methods also proved to be efficient and fast across all examined datasets. Moreover, a new approach to lexicon extraction that can be successfully used for sentiment polarity assignment is presented in the paper. It has been shown that accuracy obtained from such lexicons outperforms other lexicon based approaches.

More and more large data collections are gathered worldwide in various IT systems. Many of them possess a networked nature and need to be processed and analysed as graph structures. Due to their size they very often require the usage of a parallel paradigm for efficient computation. Three parallel techniques have been compared in the paper: MapReduce, its map-side join extension and Bulk Synchronous Parallel (BSP). They are implemented for two different graph problems: calculation of single source shortest paths (SSSP) and collective classification of graph nodes by means of relational influence propagation (RIP). The methods and algorithms are applied to several network datasets differing in size and structural profile, originating from three domains: telecommunication, multimedia and microblog. The results revealed that iterative graph processing with the BSP implementation always and significantly, even up to 10 times outperforms MapReduce, especially for algorithms with many iterations and sparse communication. The extension of MapReduce based on map-side join is usually characterized by better efficiency compared to its origin, although not as much as BSP. Nevertheless, MapReduce still remains a good alternative for enormous networks, whose data structures do not fit in local memories.

The relational label propagation problem for large data sets using MapReduce programming model was considered in the paper. The method we propose estimates class probability in relational domain in the networks. The method was examined on large real telecommunication data set. The results indicated that it could be used successfully to classify networks’ nodes and, thanks to that, new offerings or tariffs might be proposed to customers who belong to other providers. Moreover, basic properties of relational label propagation were examined and reported.

In relational learning tasks such as within network classification the main problem arises from the inference of nodes' labels based on the the ground true labels of remaining nodes. The problem becomes even harder if the nodes from initial network do not have any labels assigned and they have to be acquired. However, labels of which nodes should be obtained in order to provide fair classification results? Active learning and inference is a practical framework to study this problem. The method for active learning and inference in within network classification based on node selection is proposed in the paper. Based on the structure of the network it is calculated the utility score for each node, the ranking is formulated and for selected nodes the labels are acquired. The paper examines several distinct proposals for utility scores and selection methods reporting their impact on collective classification results performed on various real-world networks.

The problem of automated video categorization in large datasets is considered in the paper. A new Iterative Multi-label Propagation (IMP) algorithm for relational learning in multi-label data is proposed. Based on the information of the already categorized videos and their relations to other videos, the system assigns suitable categories—multiple labels to the unknown videos. The MapReduce approach to the IMP algorithm described in the paper enables processing of large datasets in parallel computing. The experiments carried out on 5-million videos dataset revealed the good efficiency of the multi-label classification for videos categorization. They have additionally shown that classification of all unknown videos required only several parallel iterations.

Knowledge exchange between employees is an essential feature of recent commercial organisations on the competitive market. Based on the data gathered by various information technology (IT) systems, social links can be extracted and exploited in knowledge exchange systems of a new kind. Users of such a system ask their queries and the system recommends known and unknown experts selected out of user's friends. The friends either provide the solution or forward the query to their friends. By means of the established social paths to experts, the system facilitates informal learning and exchange of latent knowledge between organisation members in their workplace. The overall concept, limitations and detailed features of this novel knowledge exchange system are discussed in the article.

All online sharing systems gather data that reflects users' collective behavior and their shared activities. This data can be used to extract different kinds of relationships which can be grouped into layers and which are basic components of the multidimensional social network (MSN) proposed in the paper. The layers are created on the basis of two types of relations between humans, i.e., direct and object-based ones which, respectively, correspond to either social or semantic links between individuals. For better understanding of the complexity of the social network structure, layers and their profiles were identified and studied on two, spanned in time, snapshots of the `Flickr' population. Additionally, for each layer, a separate strength measure was proposed. The experiments on the `Flickr' photo sharing system revealed that the relationships between users result either from semantic links between objects they operate on or from social connections of these users. Moreover, the density of the social network increases in time. The second part of this paper is devoted to building a social recommender system that supports the creation of new relations between users in a multimedia sharing system. Its main goal is to generate personalized suggestions that are continuously adapted to users' needs depending on the personal weights assigned to each layer in the MSN. The conducted experiments confirmed the usefulness of the proposed model.