A careful analysis of the Large Language Model (LLM) results, generated through anonymized representations of the original dataset, is crucial to precisely evaluate the data-sharing procedure's limitations and facilitate valuable collaborations among Internet-based cognitive behavioral therapy (ICBT) companies and third parties. This paper presents an experimental study of fine-tuning 27 LMs for a multiclass classification task to identify depression severity using 40,191 tweets labeled by human annotators. We fine-tune 14 Bidirectional Encoder Representations from Transformers (BERT), 6 Robustly Optimized BERT Pretraining Approaches (RoBerta), 3 Generative Pretraining (GPT), and 4 Text-to-Text Transfer Transformer (T5) based LMs to classify confidential and anonymized tweets. We report that T5, through conditional generation, outperforms widely adopted BERT, RoBerta, and GPT types for classifying confidential and anonymized tweets. Anonymizing personal information safeguards user privacy and often increases LM performance. Case sensitivity can potentially improve or harm the performance of domain-specific LMs for original and anonymized text.

Transformer-based architectures have advanced text summarization, yet their quadratic complexity limits scalability on long documents. This paper introduces BiSparse-AAS (Bilinear Sparse Attention with Adaptive Spans), a novel framework that combines sparse attention, adaptive spans, and bilinear attention to address these limitations. Sparse attention reduces computational costs by focusing on the most relevant parts of the input, while adaptive spans dynamically adjust the attention ranges. Bilinear attention complements both by modeling complex token interactions within this refined context. BiSparse-AAS consistently outperforms state-of-the-art baselines in both extractive and abstractive summarization tasks, achieving average ROUGE improvements of about 68.1% on CNN/DailyMail and 52.6% on XSum, while maintaining strong performance on OpenWebText and Gigaword datasets. By addressing efficiency, scalability, and long-sequence modeling, BiSparse-AAS provides a unified, practical solution for real-world text summarization applications. For reproducibility, our source code is available at this link11https://osf.io/enyv5/?view _only=079db437e94147a489626f275bed90c7.

This paper explores the development and evaluation of a multimodal system for survival prediction in clinical settings, leveraging both structured electronic health records and unstructured clinical notes. The core objective is to enhance the accuracy and reliability of survival predictions in Intensive Care Units by integrating diverse data types through advanced machine learning models. The system combines the novel architecture of Tabular Transformers, adapted to process structured data such as patient demographics, medical history, and diagnoses, with Multi-Layer Perceptrons for text embeddings obtained from BioClinicalBERT, a specialized model for clinical narratives. These models’ integration aims to capture the complex and multifaceted nature of patient profiles, thereby improving prediction performance. The finalized system, a Logistic Regression instance that aggregates the obtained predictions, demonstrates superior performance on evaluation metrics, highlighting the system’s ability to identify high-risk patients. Comprehensive benchmarking against decision trees, standalone MLPs, and various configurations underscored the robustness of the proposed system. This research highlights the transformative potential of multimodal data integration in medical predictive modeling. Thus, medical professionals can guide their efforts and prioritization of patient care, enabling more efficient and targeted allocation of resources during triage.

This work aims to explore the use of machine learning techniques, particularly clustering and cluster evolution tracking, to analyze travel patterns in public transportation in a city and provide valuable insights for urban transit planning and optimization. Clustering involves identifying and grouping similar objects, such as passengers with different ticket types, and distinguishing them from dissimilar objects in other groups. Over time, groups can change, so tracking this change can provide more detailed and valuable insights than analyzing data in aggregates. Clustering and cluster evolution tracking can reveal groups of passengers that are more or less affected by changes such as seasonality or fare increases. We propose a framework called DUGET (Dynamic User Grouping and Evolution Tracking), which clusters anonymized users based on their ticket choices and temporal travel patterns using a multi-step approach. The clusters are then tracked over time using Jaccard similarity based on memberships, allowing for the analysis and visualization of changes. Our experiments using a real-world public transportation dataset collected in Stockholm, Sweden, show the feasibility of tracking change over time in public transportation by examining passenger behavior as a temporal aggregate. The framework we propose is generalizable and can be used for future projects to understand trends in groups of objects.

Numerous text classification tasks inherently possess hierarchical structures among classes, often overlooked in traditional classification paradigms. This study introduces novel approaches for hierarchical text classification using Large Language Models (LLMs), exploiting taxonomies to improve accuracy and traceability in a zero-shot setting. We propose two hierarchical classification methods, namely (i) single-path and (ii) path-traversal, which all leverage the hierarchical class structures inherent in the target classes (e.g., a bird is a type of animal that belongs to a species) and improve naïve hierarchical text classification from literature. We implement them as prompts for generative models such as OpenAI GPTs and benchmark them against discriminative language models (BERT and RoBERTa). We measure the classification performance (precision, recall, and F1-score) vs. computational efficiency (time and cost). Throughout the evaluations of the classification methods on two diverse datasets, namely ComFaSyn, containing mental health patients’ diary entries, and DBpedia, containing structured information extracted from Wikipedia, we observed that our methods, without any form of fine-tuning and few-shot examples, achieve comparable results to flat classification and existing methods from literature with minimal increases in the prompts and processing time.

In wood science, accurate segmentation of non-cellular elements in microscopy images is critical for assessing wood quality and understanding growth patterns. Yet, it is challenging due to the complex morphology of wood components. This work explores the development of a semi-supervised deep learning model for segmenting non-cellular elements in wood microscopy images of Norway spruce, an essential source for construction materials in Europe, addressing the challenge of manual annotation’s labor intensity and expertise requirement. The segmentation model employs advanced deep learning architectures, including Convolutional Neural Networks and a Vision Transformer, to capture the intrinsic patterns embedded in wood structures. We proposed a Pixel-level Guided Mean-Teacher (PG-MT) framework as an improvement to the Mean-Teacher semi-supervised learning technique. Our framework enables pixel-level guided correction to enhance segmentation accuracy and model robustness with limited labeled datasets. Our experimental evaluations show that the proposed PG-MT framework improved the Dice score for medullary ray segmentation by 0.95% and the IoU score by 1.14% over the Uncertainty-Aware Mean-Teacher (UA-MT) framework. Additionally, the integration with laboratory instruments emphasizes the model’s effectiveness in accurately estimating cross-sectional cell wall thickness, demonstrating a strong correlation with X-ray measurements. This result validates the model’s practical applicability in laboratory settings, enhancing the analysis of wood properties. This work provides a robust semi-supervised DL framework for segmenting non-cellular elements in wood microscopy images, significantly reducing the annotation burden and paving the way for more automated and precise wood property analysis.

Transformer models have great potential in Graph Representation Learning (GRL) for efficiently scaling the learning process on large datasets and solving many challenges presented in Graph Neural Networks, e.g., oversmoothing and suspended animation. To represent each node of a graph, Transformer models as input usually take a node together with the node context, i.e., a set of other nodes that serve as learning context for the target node. However, current GRL Transformer models mainly consider the graph topology when selecting the node context for each target node. In this work, we demonstrate the important role of node features in selecting the node context. Specifically, we propose a hybrid approach for selecting node context that considers both the graph topology and the semantic similarities between node features. Through the empirical evaluations, we show the advantages of our hybrid node context selection method for a downstream classification task on various datasets compared to selection methods that only consider graph topology or semantic similarities. The best classification accuracy improvements of our proposed hybrid methods over the baseline methods on each dataset range from 0.77% to 6.05%.

Deteriorating air quality is a growing concern that has been linked to many health-related issues. Its monitoring is a good first step to understanding the problem. However, it is not always possible to collect air quality data from every location. Various data interpolation techniques are used to assist with populating sparse maps with more context, but many of these algorithms are computationally expensive. This work introduces a three-step Chain Mail algorithm that uses kriging (without any modifications to the base algorithm) and achieves up to ×100 execution time improvement with minimal accuracy loss (relative RMSE of 3%) by running concurrent interpolation executions. This approach can be described as a multiple-step parallel interpolation algorithm that includes specific regional border data manipulation for achieving greater accuracy. It does so by interpolating geographically defined data chunks in parallel and sharing the results with their neighboring nodes to provide context and compensate for lack of knowledge of the surrounding areas. Combined with a serverless cloud architecture, this approach opens doors to interpolating large data sets in a matter of minutes while remaining cost-efficient. The effectiveness of the three-step Chain Mail approach depends on the equal point distribution among all nodes and the resolution of the parallel configuration. In general, it offers a good balance between execution speed and accuracy.

Predictive modeling has the potential to improve risk stratification of cancer patients and thereby contribute to optimized treatment strategies and better outcomes for patients in clinical practice. To develop robust predictive models for decision-making in healthcare, sensitive patient-level data is often required when developing the training models. Consequently, data privacy is an important aspect to consider when building these predictive models and in subsequent communication of the results. In this study we have used Graph Neural Networks for survival prediction, and compared the accuracy to state-of-the-art prediction models after applying Differential Privacy and k-Anonymity, i.e. two privacy-preservation solutions. By using two different data sources we demonstrated that Graph Neural Networks and Survival Forests are the two most well-performing survival prediction methods when used in combination with privacy preservation solutions. Furthermore, when the predictive model was built using clinical expertise in the specific area of interest, the prediction accuracy of the proposed knowledge based graph model drops by at most 10% when used with privacy preservation solutions. Our proposed knowledge based graph is therefore more suitable to be used in combination with privacy preservation solutions as compared to other graph models.

Privacy preservation plays a vital role in health care applications as the requirements for privacy preservation are very strict in this domain. With the rapid increase in the amount, quality and detail of health data being gathered with smart devices, new mechanisms are required that can cope with the challenges of large scale and real-time processing requirements. Federated learning (FL) is one of the conventional approaches that facilitate the training of AI models without access to the raw data. However, recent studies have shown that FL alone does not guarantee sufficient privacy. Differential privacy (DP) is a well-known approach for privacy guarantees, however, because of the noise addition, DP needs to make a trade-off between privacy and accuracy. In this work, we design and implement an end-to-end pipeline using DP and FL for the first time in the context of health data streams. We propose a clustering mechanism to leverage the similarities between users to improve the prediction accuracy as well as significantly reduce the model training time. Depending on the dataset and features, our predictions are no more than 0.025% far off the ground-truth value with respect to the range of value. Moreover, our clustering mechanism brings a significant reduction in the training time, with up to 49% reduction in prediction accuracy error in the best case, as compared to training a single model on the entire dataset. Our proposed privacy preserving mechanism at best introduces a decrease of ≈ 2% in the prediction accuracy of the trained models. Furthermore, our proposed clustering mechanism reduces the prediction error even in highly noisy settings by as much as 38% as compared to using a single federated private model.

Graphs are a vastly useful and widely used form of modeling and representation of systems, processes, entities, events, objects, components etc., in various domains of discourse, that reflects relations or connections of modeled entities. Graphs are vital to diverse data mining applications, as they capture relationships between data items, such as dependencies or interactions, and graph analysis can reveal valuable insights for many application domains including machine learning, anomaly detection, clustering, recommendations, social influence analysis, bioinformatics, and others. The analysis of the evolutionary behavior of dynamic graphs provides the means to continuously predict the appearance, and also, the disappearance of new graph links, i.e., to perform the Dynamic Link Prediction Task. Dynamic Link Prediction has been explored widely in the past years; however, the majority of these works focus on discovering new edges (by implicitly assuming ever growing dynamic networks). However, very few works focus on the repeating edges, i.e., links that continuously vanish and reappear in the dynamic network, but which size (in terms of number of nodes and edges) does not significantly change over long periods of time. In this work, we first study the literature for link prediction in the static settlement, then, we focus on dynamic link prediction, underlining the strengths and weaknesses of every approach studied. We discover that traditional methods do not work well with repeating links as they are unable to encode temporal patterns associated with the edges while also considering the topological graph features. We propose a novel method, Temporal Edge Embedding Neural Network (TEEN), which is based on a deep learning architecture that jointly optimizes the prediction of the correct edge labels as well as the proximity of two nodes' pairs in their latent space at every time step. Our solution benefits of node embeddings created with deep encoders from where an edge embedding is created for every time step. Our evaluation experiments on transactional graphs show that TEEN is able to outperform state-of-the-art models by over 8% on AUC and over 7% on F1-Score. We show that our approach brings significant improvements in the scenario of transactional graphs.

Neural networks are used for different machine learning tasks, such as spatial time-series forecasting. Accurate modelling of a large and complex system requires large datasets to train a deep neural network that causes a challenge of scale as training the network and serving the model are computationally and memory intensive. One example of a complex system that produces a large number of spatial time-series is a large road sensor infrastructure deployed for traffic monitoring. The goal of this work is twofold: 1) To model large amount of spatial time-series from road sensors; 2) To address the scalability problem in a real-life task of large-scale road traffic prediction which is an important part of an Intelligent Transportation System.We propose a partitioning technique to tackle the scalability problem that enables parallelism in both training and prediction: 1) We represent the sensor system as a directed weighted graph based on the road structure, which reflects dependencies between sensor readings, and weighted by sensor readings and inter-sensor distances; 2) We propose an algorithm to automatically partition the graph taking into account dependencies between spatial time-series from sensors; 3) We use the generated sensor graph partitions to train a prediction model per partition. Our experimental results on traffic density prediction using Long Short-Term Memory (LSTM) Neural Networks show that the partitioning-based models take 2x, if run sequentially, and 12x, if run in parallel, less training time, and 20x less prediction time compared to the unpartitioned model of the entire road infrastructure. The partitioning-based models take 100x less total sequential training time compared to single sensor models, i.e., one model per sensor. Furthermore, the partitioning-based models have 2x less prediction error (RMSE) compared to both the single sensor models and the entire road model.

Copernicus is the European programme for monitoring the Earth.It consists of a set of systems that collect data from satellites and in-situ sensors, process this data and provide users with reliable and up-to-date information on a range of environmental and security issues.The data and information processed and disseminated puts Copernicus at the forefront of the big data paradigm, giving rise to all relevant challenges, the so-called 5 Vs: volume, velocity, variety, veracity and value.In this short paper, we discuss the challenges of extracting information and knowledge from huge archives of Copernicus data.We propose to achieve this by scale-out distributed deep learning techniques that run on very big clusters offering virtual machines and GPUs.We also discuss the challenges of achieving scalability in the management of the extreme volumes of information and knowledge extracted from Copernicus data.The envisioned scientific and technical work will be carried out in the context of the H2020 project ExtremeEarth which starts in January 2019.

Graph partitioning is an essential yet challenging task for massive graph analysis in distributed computing. Common graph partitioning methods scan the complete graph to obtain structural characteristics offline, before partitioning. However, the emerging need for low-latency, continuous graph analysis led to the development of online partitioning methods. Online methods ingest edges or vertices as a stream, making partitioning decisions on the fly based on partial knowledge of the graph. Prior studies have compared offline graph partitioning techniques across different systems. Yet, little effort has been put into investigating the characteristics of online graph partitioning strategies. In this work, we describe and categorize online graph partitioning techniques based on their assumptions, objectives and costs. Furthermore, we employ an experimental comparison across different applications and datasets, using a unified distributed runtime based on Apache Flink. Our experimental results showcase that model-dependent online partitioning techniques such as low-cut algorithms offer better performance for communication-intensive applications such as bulk synchronous iterative algorithms, albeit higher partitioning costs. Otherwise, model-agnostic techniques trade off data locality for lower partitioning costs and balanced workloads which is beneficial when executing data-parallel single-pass graph algorithms.

Efficient processing of large-scale graphs in distributed environments has been an increasingly popular topic of research in recent years. Inter-connected data that can be modeled as graphs appear in application domains such as machine learning, recommendation, web search, and social network analysis. Writing distributed graph applications is inherently hard and requires programming models that can cover a diverse set of problems, including iterative refinement algorithms, graph transformations, graph aggregations, pattern matching, ego-network analysis, and graph traversals. Several high-level programming abstractions have been proposed and adopted by distributed graph processing systems and big data platforms. Even though significant work has been done to experimentally compare distributed graph processing frameworks, no qualitative study and comparison of graph programming abstractions has been conducted yet. In this survey, we review and analyze the most prevalent high-level programming models for distributed graph processing, in terms of their semantics and applicability. We review 34 distributed graph processing systems with respect to the graph processing models they implement and we survey applications that appear in recent distributed graph systems papers. Finally, we discuss trends and open research questions in the area of distributed graph processing.

While cluster computing frameworks are continuously evolving to provide real-time data analysis capabilities, Apache Spark has managed to be at the forefront of big data analytics. Recent studies propose scale-in clusters with in-storage processing devices to process big data analytics with Spark However the proposal is based solely on the memory bandwidth characterization of in-memory data analytics and also does not shed light on the specification of host CPU and memory. Through empirical evaluation of in-memory data analytics with Apache Spark on an Ivy Bridge dual socket server, we have found that (i) simultaneous multi-threading is effective up to 6 cores (ii) data locality on NUMA nodes can improve the performance by 10% on average, (iii) disabling next-line L1-D prefetchers can reduce the execution time by up to 14%, (iv) DDR3 operating at 1333 MT/s is sufficient and (v) multiple small executors can provide up to 36% speedup over single large executor.