Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of training samples and scheduling them accordingly.In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn.We propose a four-quadrant categorisation of difficulty signalshuman vs. model and task-agnostic vs. task-dependent -and systematically analyse their interactions on a natural language understanding dataset.We find that task-agnostic features behave largely independently and that only task-dependent features align.These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.

7253

7089

The proliferation of textual data containing sensitive personal information across various domains requires robust anonymization techniques to protect privacy and comply with regulations, while preserving data usability for diverse and crucial downstream tasks. This survey provides a comprehen-sive overview of current trends and recent advances in text anonymization techniques. We begin by discussing foundational approaches, primarily centered on Named Entity Recognition, before examining the transformative impact of Large Language Models, detailing their dual role as sophisticated anonymizers and potent de-anonymization threats. The survey further ex-plores domain-specific challenges and tailored solutions in critical sectors such as healthcare, law, finance, and education. We investigate advanced methodologies incorporating formal privacy models and risk-aware frameworks, and address the specialized subfield of authorship anonymization. Additionally, we review evaluation frameworks, comprehensive metrics, benchmarks, and practical toolkits for real-world deployment of anonymization solutions. This review consolidates current knowledge, identifies emerging trends and persistent challenges, including the evolving privacy-utility trade-off, the need to address quasi-identifiers, and the implications of LLM capabilities, and aims to guide future research directions for both academics and practitioners in this field.

Surgical gauze is an essential part of surgical procedures, which is primarily used for controlling bleeding and absorbing bodily fluids. The post-surgical retention of gauze can lead to serious complications in the patient's health and necessitate additional surgery for gauze removal. In the wake of data scarcity, the research on gauze segmentation on the real-world surgical data remains underexplored. In this work, we investigate the use of deep learning methods for gauze segmentation in robotassisted minimally invasive abdominal surgeries, utilizing an inhouse surgical dataset prepared at a university hospital. The training data reflects a realistic surgical setting and extensive diversity in spatial, morphological, and visual attributes of three different gauze categories. We have investigated prevalently used segmentation architectures, including CNN-based, transformer-based, and hybrid architectures, to provide a proof-of-concept for gauze segmentation in a realistic setting. Besides, we investigate the influence of additional sub-optimally annotated, auto-tracked segmentation masks to address the bottleneck of data scarcity and performance optimization. Our results demonstrate the efficacy of real-world data to counter the main challenge reported by prior works - the trade-off between blood presence and gauze detection. The incorporation of auto-track annotations enables performance enhancements, particularly in generic cases. The integration of effective segmentation approaches will benefit robotguided surgical procedures and various downstream applications by providing a precise delineation of foreign objects, enhancing patient safety and surgical outcomes.

Despite Arabic being one of the most widely spoken languages, there is a scarcity of available dialectal Arabic data. In this paper, we address this challenge by proposing a novel approach to data collection through the main use of video captions from TikTok, and other resources such as dictionaries and articles, resulting in the creation of the ArDia dataset. To the best of our knowledge, the ArDia dataset is the largest labeled dialectal Arabic dataset, containing over 900,000 examples, each labeled with its respective dialect. We further leverage this dataset to pretrain transformer-based models, ArDiaBERT and ArDiaGPT. Due to a lack of research on the Arabic models, we present a comprehensive study of Arabic dialect identification using the ArDia dataset on the dialect identification task.

End-to-end robot policies achieve high performance through neural networks trained via reinforcement learning (RL). Yet, their black box nature and abstract reasoning pose challenges for human-robot interaction (HRI), because humans may experience difficulty in understanding and predicting the robot’s navigation decisions, hindering trust development. We present a virtual reality (VR) interface that visualizes explainable AI (XAI) outputs and the robot’s lidar perception to support intuitive interpretation of RL-based navigation behavior. By visually highlighting objects based on their attribution scores, the interface grounds abstract policy explanations in the scene context. This XAI visualization bridges the gap between obscure numerical XAI attribution scores and a human-centric semantic level of explanation. A within-subjects study with 24 participants evaluated the effectiveness of our interface for four visualization conditions combining XAI and lidar. Participants ranked scene objects across navigation scenarios based on their importance to the robot, followed by a questionnaire assessing subjective understanding and predictability. Results show that semantic projection of attributions significantly enhances non-expert users’ objective understanding and subjective awareness of robot behavior. In addition, lidar visualization further improves perceived predictability, underscoring the value of integrating XAI and sensor for transparent, trustworthy HRI.

Abstract Graph neural networks (GNNs) are among the most widely used methods for node classification in graphs. A common strategy to improve their predictive performance is to enrich nodes with additional features. A weakness of this method is that the set of appropriate features can vary from graph to graph. We address this shortcoming by proposing a novel method. In a preprocessing step, a first GNN is trained on a set of graphs with varying structural properties, using a candidate set of node features fixed in advance. The resulting GNN model is then used to predict the most relevant features from the candidate set for unseen target graphs, which are later processed for node classification. For each target graph, a second GNN is trained on the graph, which is enriched with the node feature vectors calculated for the features selected by the first GNN. A key advantage of the proposed method is that the features are selected without computing the candidate features for the target graph. Our experimental results on synthetic and real-world graphs show that even a few features selected in this way is sufficient to significantly improve the predictive performance of GNNs that use either none or all of the candidate features. Moreover, the time needed to learn the second GNN for the target graph can be reduced by up to two orders of magnitude.

Ever-larger language models with ever-increasing capabilities are by now well-established text processing tools. Alas, information extraction tasks such as named entity recognition are still largely unaffected by this progress as they are primarily based on the previous generation of encoder-only transformer models. Here, we propose a simple yet effective approach, Informed Named Entity Recognition Decoding (iNERD), which treats named entity recognition as a generative process. It leverages the language understanding capabilities of recent generative models in a future-proof manner and employs an informed decoding scheme incorporating the restricted nature of information extraction into open-ended text generation, improving performance and efficiency and eliminating any risk of hallucinations. We coarse-tune our model on a merged named entity corpus to strengthen its performance, evaluate five generative language models on eight named entity recognition datasets, and achieve remarkable results, especially in an environment with an unknown entity class set, demonstrating the adaptability of the approach.

Accurate detection of dementia is crucial for timely intervention and care, and leveraging multimodal data holds significant potential for improving diagnostic accuracy. In this study, we explore deep learning approaches for dementia classification using the Pitt corpus, which includes brief participant descriptions of a cookie theft scene. We analyze 242 control and 307 dementia audio clips to investigate various representation learning techniques. Our best-performing approach fuses audio spectrograms with advanced language models, including Whisper model transcriptions and transformer-based feature extraction. We rigorously evaluate these models and find that our multimodal approach with an F1-score of 86.42% eclipses other single modality approaches by a considerable margin. Our findings underscore the promise of multimodal deep learning techniques in advancing the reliability of dementia detection through audio analysis, possibly paving the way for more robust and accessible diagnostic tools.

Named Entity Recognition (NER) in the legal domain presents unique challenges due to specialized terminology and complex linguistic structures inherent in legal texts. While large language models (LLMs) like GPT-4, Llama-3, and others have significantly advanced natural language processing, their effectiveness in domain-specific tasks like legal Named Entity Recognition remains underexplored. This study conducts a comprehensive comparative analysis of eleven state-of-the-art LLMs on legal NER tasks across seven diverse datasets in five languages, namely English, Portuguese, German, Turkish, and Ukrainian. We evaluate the models’ performance using F1scores, focusing on their ability to accurately identify and classify legal entities. Our findings reveal significant variability in LLM performance across different languages and legal contexts, with proprietary models like GPT-4 achieving the highest overall scores. The results highlight the influence of model architecture, dataset characteristics, and prompt design on the effectiveness of legal NER tasks. This study provides valuable benchmarks for legal NER applications and offers insights into the strengths and limitations of current LLMs, guiding future research and development in legal natural language processing.

We explore the use of Large Language Models (LLMs) for automating the extraction of Key Performance Indicators (KPIs) from diverse financial reports without any additional fine-tuning. We focus on evaluating various proprietary and open-source LLMs to address the joint named entity recognition and relation extraction tasks essential for accurately linking KPIs to their corresponding values and attributes. Our study highlights the technical challenges involved in the extraction process and presents a comprehensive evaluation of the models’ effectiveness. Our results reveal significant insights into handling these LLMs in such a crucial environment and showcase the transformative potential of LLMs in enhancing financial analysis and decision-making.

Model-agnostic explanation methods for deep learning models are flexible regarding usability and availability. However, due to the fact that they can only manipulate input to see changes in output, they suffer from weak performance when used with complex model architectures. For models with large inputs as, for instance, in object detection, sampling-based methods like KernelSHAP are inefficient due to many computation-heavy forward passes through the model. In this work, we present a framework for using sampling-based explanation methods in a computer vision context shown for body part relevance assessment for pedestrian detection. Furthermore, we introduce a novel sampling-based method similar to KernelSHAP that shows more robustness for lower sampling sizes and, thus, is more efficient for explainability analyses on large-scale datasets. We demonstrate our relevance assessment method on simulation data acquired with the CARLA simulator. In the end, our method enables the benchmarking and performance comparison of various pedestrian detection models based on human-interpretable semantic regions.

Thanks to the progress in precision agriculture and remote sensing, automatic visual disease severity scoring of largescale fields is getting more efficient. This potentially allows the investigation and observation of disease spread dynamics and infection incidence in real-world field trials. Furthermore, phytopathological research has been obtaining comprehensive knowledge about pathogens and epidemiology for decades. In this work, we want to leverage modern data-driven Machine Learning (ML) methods and connect them with available knowledge incorporated by simulation techniques. As a result, we show how to create infection datasets from disease severity annotations and environmental data that can serve as training data for a transformer-based ML model. First preliminary results show coherent, interpretable results that motivate for further optimization and improvements of our modeling strategy.

Table structure recognition (TSR), the task of inferring the layout of tables, including the row, column, and cell structure, is a surprisingly complex task. With the growing amount and importance of digital documents, it has become an increasingly relevant problem, which nonetheless has not yet been solved adequately and still presents a very active area of research. In recent years, a growing number of deep-learning-based approaches to table parsing have been proposed.This paper presents a novel deep-learning-based table structure recognition method that can predict row, column, and cell bounds for table images with a high degree of accuracy. To achieve this goal, a multi-stage pipeline incorporating a Vision-Transformer-based Autoencoder model was devised. This model was trained to predict cell regions for table images, from which accurate cell bounds can be inferred, including spanning cells which cover multiple rows or columns. The goal was to obtain a model that generalizes well and can return accurate predictions on various tables of differing complexity, even if they contain little initial structural information.An additional modification to the model architecture presented in the Masked Autoencoder (MAE) approach was also evaluated.

Leichte Sprache", the German counterpart to Simple English, is a regulated language aiming to facilitate complex written language that would otherwise stay inaccessible to different groups of people.We present a new sentencealigned monolingual corpus for Simple German -German.It contains multiple documentaligned sources which we have aligned using automatic sentence-alignment methods.We evaluate our alignments based on a manually labelled subset of aligned documents.The quality of our sentence alignments, as measured by the F1-score, surpasses previous work.

The auditing of financial documents, historically a labor-intensive process, stands on the precipice of transformation. AI-driven solutions have made inroads into streamlining this process by recommending pertinent text passages from financial reports to align with the legal requirements of accounting standards. However, a glaring limitation remains: these systems commonly fall short in verifying if the recommended excerpts indeed comply with the specific legal mandates. Hence, in this paper, we probe the efficiency of publicly available Large Language Models (LLMs) in the realm of regulatory compliance across different model configurations. We place particular emphasis on comparing cutting-edge open-source LLMs, such as Llama-2, with their proprietary counterparts like OpenAI’s GPT models. This comparative analysis leverages two custom datasets provided by our partner PricewaterhouseCoopers (PwC) Germany. We find that the open-source Llama-2 70 billion model demonstrates outstanding performance in detecting non-compliance or true negative occurrences, beating all their proprietary counterparts. Nevertheless, proprietary models such as GPT-4 perform the best in a broad variety of scenarios, particularly in non-English contexts.

Correct identification and correction of contradictions and inconsistencies within financial reports constitute a fundamental component of the audit process. To streamline and automate this critical task, we introduce a novel approach leveraging large language models and an embedding-based paragraph clustering methodology. This paper assesses our approach across three distinct datasets, including two annotated datasets and one unannotated dataset, all within a zero-shot framework. Our findings reveal highly promising results that significantly enhance the effectiveness and efficiency of the auditing process, ultimately reducing the time required for a thorough and reliable financial report audit.

Auditing financial documents is a very tedious and time-consuming process. As of today, it can already be simplified by employing AI-based solutions to recommend relevant text passages from a report for each legal requirement of rigorous accounting standards. However, these methods need to be fine-tuned regularly, and they require abundant annotated data, which is often lacking in industrial environments. Hence, we present ZeroShotALI, a novel recommender system that leverages a state-of-the-art large language model (LLM) in conjunction with a domain-specifically optimized transformer-based text-matching solution. We find that a two-step approach of first retrieving a number of best matching document sections per legal requirement with a custom BERT-based model and second filtering these selections using an LLM yields significant performance improvements over existing approaches.

During the pre-training step of natural language models, the main objective is to learn a general representation of the pre-training dataset, usually requiring large amounts of textual data to capture the complexity and diversity of natural language. Contrasting this, in most cases, the size of the data available to solve the specific downstream task is often dwarfed by the aforementioned pre-training dataset, especially in domains where data is scarce. We introduce controlled randomness, i.e. noise, into the training process to improve fine-tuning language models and explore the performance of targeted noise in addition to the parameters of these models. We find that adding such noise can improve the performance in our two downstream tasks of joint named entity recognition and relation extraction and text summarization.

Informed machine learning describes the injection of prior knowledge into learning systems. It can help to improve generalization, especially when training data is scarce. However, the field is so application-driven that general analyses about the effect of knowledge injection are rare. This makes it difficult to transfer existing approaches to new applications, or to estimate potential improvements. Therefore, in this paper, we present a framework for quantifying the value of prior knowledge in informed machine learning. Our main contributions are threefold. Firstly, we propose a set of relevant metrics for quantifying the benefits of knowledge injection, comprising in-distribution accuracy, out-of-distribution robustness, and knowledge conformity. We also introduce a metric that combines performance improvement and data reduction. Secondly, we present a theoretical framework that represents prior knowledge in a function space and relates it to data representations and a trained model. This suggests that the distances between knowledge and data influence potential model improvements. Thirdly, we perform a systematic experimental study with controllable toy problems. All in all, this helps to find general answers to the question how knowledge injection helps in informed machine learning.

in "A Realism Metric for Generated LiDAR Point Clouds" by Triess et al. In this work, a new metric is proposed that measures the quality of LiDAR point clouds that are generated, e.g., by a generative network. The metric can be used as an early indicator which assesses whether the generated training data will improve a down-stream task like semantic point cloud segmentation. Brissman et al. propose in "Recurrent Graph Neural Networks for Video Instance Segmentation" a very efficient approach for video instance segmentation. The approach tracks and segments multiple objects on-line and in real-time. Temporal information is also used in "Spatio-Temporal Outdoor Lighting Aggregation on Image Sequences using Transformer Networks" by Lee et al. to estimate outdoor illumination more consistently and without the need of an additional post-processing step.

Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning.

We introduce KPI-Check, a novel system that automatically identifies and cross-checks semantically equivalent key performance indicators (KPIs), e.g. "revenue" or "total costs", in real-world German financial reports. It combines a financial named entity and relation extraction module with a BERT-based filtering and text pair classification component to extract KPIs from unstructured sentences before linking them to synonymous occurrences in the balance sheet and profit & loss statement. The tool achieves a high matching performance of 73.00% micro F1on a hold out test set and is currently being deployed for a globally operating major auditing firm to assist the auditing procedure of financial statements.

Variational quantum circuits build the foundation for various classes of quantum algorithms. In a nutshell, the weights of a parametrized quantum circuit are varied until the empirical sampling distribution of the circuit is sufficiently close to a desired outcome. Numerical first-order methods are applied frequently to fit the parameters of the circuit, but most of the time, the circuit itself, that is, the actual composition of gates, is fixed. Methods for optimizing the circuit design jointly with the weights have been proposed, but empirical results are rather scarce. Here, we consider a simple evolutionary strategy that addresses the trade-off between finding appropriate circuit ar-chitectures and parameter tuning. We evaluate our method both via simulation and on actual quantum hardware. Our benchmark problems include the transverse field Ising Hamiltonian and the Sherrington-Kirkpatrick spin model. Despite the shortcomings of current noisy intermediate-scale quantum hardware, we find only a minor slowdown on actual quantum machines compared to simulations. Moreover, we investigate which mutation operations most significantly contribute to the optimization. The results provide intuition on how randomized search heuristics behave on actual quantum hardware and layout a path for further refinement of evolutionary quantum gate circuits.

Many applications in automated auditing and the analysis and consistency check of financial documents can be formulated in part as the subset sum problem: Given a set of numbers and a target sum, find the subset of numbers that sums up to the target. The problem is NP-hard and classical solving algorithms are therefore not practical to use in many real applications.We tackle the problem as a QUBO (quadratic unconstrained binary optimization) problem and show how gradient descent on Hopfield Networks reliably finds solutions for both artificial and real data. We outline how this algorithm can be applied by adiabatic quantum computers (quantum annealers) and specialized hardware (field programmable gate arrays) for digital annealing and run experiments on quantum annealing hardware.

We introduce KPI-EDGAR, a novel dataset for Joint Named Entity Recognition and Relation Extraction building on financial reports uploaded to the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system, where the main objective is to extract Key Performance Indicators (KPIs) from financial documents and link them to their numerical values and other attributes. We further provide four accompanying baselines for benchmarking potential future research. Additionally, we propose a new way of measuring the success of said extraction process by incorporating a word-level weighting scheme into the conventional F1score to better model the inherently fuzzy borders of the entity pairs of a relation in this domain.

We present KPI-BERT, a system which employs novel methods of named entity recognition (NER) and relation extraction (RE) to extract and link key performance indicators (KPIs), e.g. "revenue" or "interest expenses", of companies from real-world German financial documents. Specifically, we introduce an end-to-end trainable architecture that is based on Bidirectional Encoder Representations from Transformers (BERT) combining a recurrent neural network (RNN) with conditional label masking to sequentially tag entities before it classifies their relations. Our model also introduces a learnable RNN-based pooling mechanism and incorporates domain expert knowledge by explicitly filtering impossible relations. We achieve a substantially higher prediction performance on a new practical dataset of German financial reports, outperforming several strong baselines including a competing state-of-the-art span-based entity tagging approach.

Modern society is heavily dependent on large scale client-server systems with applications ranging from Internet and Communication Services to sophisticated logistics and deployment of goods. To maintain and improve such a system, a careful study of client and server dynamics is needed – e.g. response/service times, aver-age number of clients at given times, etc. To this end, one traditionally relies, within the queuing theory formalism,on parametric analysis and explicit distribution forms.However, parametric forms limit the model’s expressiveness and could struggle on extensively large datasets. We propose a novel data-driven approach towards queuing systems: the Deep Generative Service Times. Our methodology delivers a flexible and scalable model for service and response times. We leverage the representation capabilities of Recurrent Marked Point Processes for the temporal dynamics of clients, as well as Wasserstein Generative Adversarial Network techniques, to learn deep generative models which are able to represent complex conditional service time distributions. We provide extensive experimental analysis on both empirical and synthetic datasets, showing the effectiveness of the proposed models

We consider Automated List Inspection (ALI), a content-based text recommendation system that assists auditors in matching relevant text passages from notes in financial statements to specific law regulations. ALI follows a ranking paradigm in which a fixed number of requirements per textual passage are shown to the user. Despite achieving impressive ranking performance, the user experience can still be improved by showing a dynamic number of recommendations. Besides, existing models rely on a feature-based language model that needs to be pre-trained on a large corpus of domain-specific datasets. Moreover, they cannot be trained in an end-to-end fashion by jointly optimizing with language model parameters. In this work, we alleviate these concerns by considering a multi-label classification approach that predicts dynamic requirement sequences. We base our model on pre-trained BERT that allows us to fine-tune the whole model in an end-to-end fashion, thereby avoiding the need for training a language representation model. We conclude by presenting a detailed evaluation of the proposed model on two German financial datasets.

One-vs-Rest (OVR) classification aims to distinguish a single class of interest from other classes. The concept of novelty detection and robustness to dataset shift becomes crucial in OVR when the scope of the rest class extends from the classes observed during training to unseen and possibly unrelated classes. In this work, we propose a novel architecture, namely Decoupling Autoencoder (DAE) to tackle the common issue of robustness w.r.t. out-of-distribution samples which is prevalent in classifiers such as multi-layer perceptrons (MLP) and ensemble architectures. Experiments on plain classification, outlier detection, and dataset shift tasks show DAE to achieve robust performance across these tasks compared to the baselines, which tend to fail completely, when exposed to dataset shift. While DAE and the baselines yield rather uncalibrated predictions on the outlier detection and dataset shift task, we found that DAE calibration is more stable across all tasks. Therefore, calibration measures applied to the classification task could also improve the calibration of the outlier detection and dataset shift scenarios for DAE.

We present a comparative study on toxicity detection, focusing on the problem of identifying toxicity types of low prevalence and possibly even unobserved at training time.For this purpose, we train our models on a dataset that contains only a weak type of toxicity, and test whether they are able to generalize to more severe toxicity types.We find that representation learning and ensembling exceed the classification performance of simple classifiers on toxicity detection, while also providing significantly better generalization and robustness.All models benefit from a larger training set size, which even extends to the toxicity types unseen during training.

The problem of uncovering different dynamical regimes is of pivotal importance in time series analysis. Switching dynamical systems provide a solution for modeling physical phenomena whose time series data exhibit different dynamical modes. In this work we propose a novel variational RNN model for switching dynamics allowing for both non-Markovian and nonlinear dynamical behavior between and within dynamic modes. Attention mechanisms are provided to inform the switching distribution. We evaluate our model on synthetic and empirical datasets of diverse nature and successfully uncover different dynamical regimes and predict the switching dynamics.

In this paper we ask for the main factors that determine a classifier's decision making process and uncover such factors by studying latent codes produced by auto-encoding frameworks. To deliver an explanation of a classifier's behaviour, we propose a method that provides series of examples highlighting semantic differences between the classifier's decisions. These examples are generated through interpolations in latent space. We introduce and formalize the notion of a semantic stochastic path, as a suitable stochastic process defined in feature (data) space via latent code interpolations. We then introduce the concept of semantic Lagrangians as a way to incorporate the desired classifier's behaviour and find that the solution of the associated variational problem allows for highlighting differences in the classifier decision. Very importantly, within our framework the classifier is used as a black-box, and only its evaluation is required.

Natural Language Inference, and specifically Contradiction Detection, is still an unexplored topic with respect to German text. In this paper, we apply Recurrent Neural Network (RNN) methods to learn contradiction-specific sentence embeddings. Our data set for evaluation is a machine-translated version of the Stanford Natural Language Inference (SNLI) corpus. The results are compared to a baseline using unsupervised vectorization techniques, namely tf-idf and Flair, as well as state-of-the art transformer-based (MBERT) methods. We find that the end-to-end models outperform the models trained on unsupervised embeddings, which makes them the better choice in an empirical use case. The RNN methods also perform superior to MBERT on the translated data set.

Artificial intelligence for autonomous driving must meet strict requirements on safety and robustness, which motivates the thorough validation of learned models. However, current validation approaches mostly require ground truth data and are thus both cost-intensive and limited in their applicability. We propose to overcome these limitations by a model agnostic validation using a-priori knowledge from street maps. In particular, we show how to validate semantic segmentation masks and demonstrate the potential of our approach using OpenStreetMap. We introduce validation metrics that indicate false positive or negative road segments. Besides the validation approach, we present a method to correct the vehicle's GPS position so that a more accurate localization can be used for the street-map based validation. Lastly, we present quantitative results on the Cityscapes dataset indicating that our validation approach can indeed uncover errors in semantic segmentation masks.

Decision trees excel at interpretability of their prediction results. To achieve required prediction accuracies, however, often large ensembles of decision trees - random forests - are considered, reducing interpretability due to large size. Additionally, their size slows down inference on modern hardware and restricts their applicability in low-memory embedded devices. We introduce Decision Snippet Features, which are obtained from small subtrees that appear frequently in trained random forests. We subsequently show that linear models on top of these features achieve comparable and sometimes even better predictive performance than the original random forest, while reducing the model size by up to two orders of magnitude.

In this paper, we describe the combination of machine learning and simulation towards a hybrid modelling approach. Such a combination of data-based and knowledge-based modelling is motivated by applications that are partly based on causal relationships, while other effects result from hidden dependencies that are represented in huge amounts of data. Our aim is to bridge the knowledge gap between the two individual communities from machine learning and simulation to promote the development of hybrid systems. We present a conceptual framework that helps to identify potential combined approaches and employ it to give a structured overview of different types of combinations using exemplary approaches of simulation-assisted machine learning and machine-learning assisted simulation. We also discuss an advanced pairing in the context of Industry 4.0 where we see particular further potential for hybrid systems.

Our goal with this paper is to elucidate the close connection between Hopfield networks and adiabatic quantum computing. Focusing on their use in problem solving, we point out that the energy functions minimized by Hopfield networks are essentially identical to those minimized by adiabatic quantum computers. To practically illustrate this, we consider a simple textbook problem, namely the k-rooks problem, and discuss how to set it up for solution via a Hopfield network or adiabatic quantum computing.

Deep neural network models represent the state-of-the-art methodologies for natural language processing. Here we build on top of these methodologies to incorporate temporal information and model how review data changes with time. Specifically, we use the dynamic representations of recurrent point process models, which encode the history of how business or service reviews are received in time, to generate instantaneous language models with improved prediction capabilities. Simultaneously, our methodologies enhance the predictive power of our point process models by incorporating summarized review content representations. We provide recurrent network and temporal convolution solutions for modeling the review content. We deploy our methodologies in the context of recommender systems, effectively characterizing the change in preference and taste of users as time evolves. Source code is available at [1].

Despite the successful application of Deep Reinforcement Learning (DRL) in a wide range of complex tasks, agents either often learn sub-optimal behavior due to the sparse/deceptive nature of rewards or require a lot of interactions with the environment. Recent methods combine a class of algorithms known as Novelty Search (NS), which circumvents this problem by encouraging exploration towards novel behaviors. Even without exploiting any environment rewards, they are capable of learning skills that yield competitive results in several tasks. However, to assign novelty scores to policies, these methods rely on neighborhood models that store behaviors in an archive set. Hence they do not scale and generalize to complex tasks requiring too many policy evaluations. Addressing these challenges, we propose a function approximation paradigm to instead learn sparse representations of agent behaviors using auto-encoders, which are later used to assign novelty scores to policies. Experimental results on benchmark tasks suggest that this way of novelty-guided exploration is a viable alternative to classic novelty search methods.

The combinatorial problem of max-sum diversification asks for a maximally diverse subset of a given set of data. Here, we show that it can be expressed as an Ising energy minimization problem. Given this result, max-sum diversification can be solved on adiabatic quantum computers and we present proof of concept simulations which support this claim. This, in turn, suggests that quantum computing might play a role in data mining. We therefore discuss quantum computing in a tutorial like manner and elaborate on its current strengths and weaknesses for data analysis.

A Triple in knowledge-graph takes a form that consists of head, relation, tail. Triple Classification is used to determine the truth value of an unknown Triple. This is a hard task for 1-to-N relations using the vector-based embedding approach. We propose a new region-based embedding approach using fine-grained type chains. A novel geometric process is presented to extend the vectors of pre-trained entities into n-balls (n-dimensional balls) under the condition that head balls shall contain their tail balls. Our algorithm achieves zero energy cost, therefore, serves as a case study of perfectly imposing tree structures into vector space. An unknown Triple (h,r,x) will be predicted as true, when x’s n-ball is located in the r-subspace of h’s n-ball, following the same construction of known tails of h. The experiments are based on large datasets derived from the benchmark datasets WN11, FB13, and WN18. Our results show that the performance of the new method is related to the length of the type chain and the quality of pre-trained entityembeddings, and that performances of long chains with welltrained entity-embeddings outperform other methods in the literature. Source codes and datasets are located at https: //github.com/GnodIsNait/mushroom.

We present a tool that enables benchmarking of companies by means of automatic extraction of key performance indicators from publicly available financial reports. Our tool monitors companies of interest so that their reports are automatically downloaded as soon as they become available. After tables and paragraphs have been extracted from the documents using a table detection module based on convolutional neural networks, relevant key performance indicators are stored in a central database. The extracted values are finally displayed in a user-friendly web application where the user can compare time series of key performance indicators against arbitrary available companies.

We present the Automated List Inspection (ALI) tool that utilizes methods from machine learning, natural language processing, combined with domain expert knowledge to automate financial statement auditing. ALI is a content based context-aware recommender system, that matches relevant text passages from the notes to the financial statement to specific law regulations. In this paper, we present the architecture of the recommender tool which includes text mining, language modeling, unsupervised and supervised methods that range from binary classification models to deep recurrent neural networks. Next to our main findings, we present quantitative and qualitative comparisons of the algorithms as well as concepts for how to further extend the functionality of the tool.

We present a simple, two step procedure that selects central and extremal prototypes from a given set of data. The key idea is to identify minima of the function that characterizes the interior of a kernel minimum enclosing ball of the data. We discuss how to efficiently compute kernel minimim enclosing balls using the Frank-Wolfe algorithm and show that, for Gaussian kernels, the sought after prototypes can be naturally found via a variant of the mean shift procedure. Practical results demonstrate that prototypes found this way are descriptive, meaningful, and interpretable.

We present an online learning algorithm to extract extremal prototypes from a set of data. As an online algorithm, our method can continue to learn during the application phase of a system. However, as a greedy update procedure, it may be sensitive to outliers. We therefore consider the use of extreme value theory for self-assessment and discuss how to incorporate Weibull statistics so as to increase robustness. We evaluate our approaches on synthetic as well as real world datasets to perform profiling. Our empirical results show that incorporating self-assessment not only results in better data representations but also reveals interpretable insights about the analyzed dataset.

Recent years have seen a deluge of behavioral data from players hitting the game industry. Reasons for this data surge are many and include the introduction of new business models, technical innovations, the popularity of online games, and the increasing persistence of games. Irrespective of the causes, the proliferation of behavioral data poses the problem of how to derive insights therefrom. Behavioral data sets can be large, time-dependent and high-dimensional. Clustering offers a way to explore such data and to discover patterns that can reduce the overall complexity of the data. Clustering and other techniques for player profiling and play style analysis have, therefore, become popular in the nascent field of game analytics. However, the proper use of clustering techniques requires expertise and an understanding of games is essential to evaluate results. With this paper, we address game data scientists and present a review and tutorial focusing on the application of clustering techniques to mine behavioral game data. Several algorithms are reviewed and examples of their application shown. Key topics such as feature normalization are discussed and open problems in the context of game analytics are pointed out.

The articles in this special section ddress various flavors of the diverse field of game analytics. It covers topics ranging from player profiling, behavioral prediction, metrics extraction from gameplay recordings, behavioral analysis, retention analysis, and more.

Motivated by the center-surround mechanism in the human visual attention system, we propose to use average contrast maps for the challenge of pedestrian detection in street scenes due to the observation that pedestrians indeed exhibit discriminative contrast texture. Our main contributions are the first to design a local statistical multichannel descriptor to incorporate both color and gradient information. Second, we introduce a multidirection and multiscale contrast scheme based on grid cells to integrate expressive local variations. Contributing to the issue of selecting most discriminative features for assessing and classification, we perform extensive comparisons with respect to statistical descriptors, contrast measurements, and scale structures. By this way, we obtain reasonable results under various configurations. Empirical findings from applying our optimized detector on the INRIA and Caltech pedestrian datasets show that our features yield state-of-the-art performance in pedestrian detection.

Automatic pedestrian detection for advanced driver assistance systems (ADASs) is still a challenging task. Major reasons are dynamic and complex backgrounds in street scenes and variations in clothing or postures of pedestrians. We propose a simple yet effective detector for robust pedestrian detection. Observing that pedestrians usually appear upright in video data, we employ a statistical model of the upright human body in which the head, upper body, and lower body are treated as three distinct components. Our main contribution is to systematically design a pool of rectangular features that are tailored to this shape model. As we incorporate different kinds of low-level measurements, the resulting multimodal and multichannel Haar-like features represent characteristic differences between parts of the human body but are robust against variations in clothing or environmental settings. Our approach avoids exhaustive searches over all possible configurations of rectangular features nor does it rely on random sampling. It thus marks a middle ground among recently published techniques and yields efficient low-dimensional yet highly discriminative features. Experimental results on the well-established INRIA, Caltech, and KITTI pedestrian data sets show that our detector reaches state-of-the-art performance at low computational costs and that our features are robust against occlusions.

We propose a simple yet effective detector for pedestrian detection. The basic idea is to incorporate common sense and everyday knowledge into the design of simple and computationally efficient features. As pedestrians usually appear up-right in image or video data, the problem of pedestrian detection is considerably simpler than general purpose people detection. We therefore employ a statistical model of the up-right human body where the head, the upper body, and the lower body are treated as three distinct components. Our main contribution is to systematically design a pool of rectangular templates that are tailored to this shape model. As we incorporate different kinds of low-level measurements, the resulting multi-modal & multi-channel Haar-like features represent characteristic differences between parts of the human body yet are robust against variations in clothing or environmental settings. Our approach avoids exhaustive searches over all possible configurations of rectangle features and neither relies on random sampling. It thus marks a middle ground among recently published techniques and yields efficient low-dimensional yet highly discriminative features. Experimental results on the INRIA and Caltech pedestrian datasets show that our detector reaches state-of-the-art performance at low computational costs and that our features are robust against occlusions.

We analyze time series which indicate how collective attention to social media services or Web-based businesses evolves over time. Data was gathered from Goolge Trends and consists of discrete time series of varying duration. Following the related literature, we fit Weibull distributions to the data. Given the two parameters of its fitted model, we embed each time series in a low-dimensional space and apply kernel archetypal analysis based on the Kullback-Leibler divergence for clustering. Our results reveal strong regularities in the dynamics of collective attention to social media and thus illustrate the potential of advanced pattern recognition techniques in the emerging area of Web science.

We introduce a novel set of features for a challenging image analysis task in agriculture where cell phone camera images of beet leaves are analyzed as to the presence of plant diseases. Aiming at minimal computational costs on the cellular device and highly accurate prediction results, we present an efficient detector of potential disease regions and a robust classification method based on texture features. We evaluate several first- and second-order statistical features for classifying textures of leaf spots and we find that a combination of descriptors derived on multiple erosion bands of the RGB color channels, as well as, the local binary patterns of gradient magnitudes of the extracted regions accurately distinguish between symptoms caused by five diseases, including infections of the fungi Cercospora beticola, Ramularia beticola, Uromyces betae, and Phoma betae, and the bacterium Pseudomonas syringae pv. aptata.

Inspired by the human vision system, in this paper we propose a specifically organized kind of center-surround contrast features and show their suitability for pedestrian detection. These contrasts are computed from a novel combination of both local color and gradient statistics aggregated quickly for arbitrary sized square cells. We exploit our contrast features in a rich multi-scale and -direction fashion between each central cell and its neighbors and boost the significant ones for pedestrian detection. Experimental results on the INRIA and Caltech pedestrian datasets show that our method achieves state-of-the-art performance.

This paper addresses the problem of video alignment. We present efficient approaches that allow for spatiotemporal alignment of two sequences. Unlike most related works, we consider independently moving cameras that capture a 3D scene at different times. The novelty of the proposed method lies in the adaptation and extension of an efficient information retrieval framework that casts the sequences as an image database and a set of query frames, respectively. The efficient retrieval builds on the recently proposed quad descriptor. In this context, we define the 3D Vote Space (VS) by aggregating votes through a multiquerying (multiscale) scheme and we present two solutions based on VS entries; a causal solution that permits online synchronization and a global solution through multiscale dynamic programming. In addition, we extend the recently introduced ECC image-alignment algorithm to the temporal dimension that allows for spatial registration and synchronization refinement with subframe accuracy. We investigate full search and quantization methods for short descriptors and we compare the proposed schemes with the state of the art. Experiments with real videos by moving or static cameras demonstrate the efficiency of the proposed method and verify its effectiveness with respect to spatiotemporal alignment accuracy.

Pre-symptomatic drought stress prediction is of great relevance in precision plant protection, ultimately helping to meet the challenge of "How to feed a hungry world?". Unfortunately, it also presents unique computational problems in scale and interpretability: it is a temporal, large-scale prediction task, e.g., when monitoring plants over time using hyperspectral imaging, and features are `things' with a `biological' meaning and interpretation and not just mathematical abstractions computable for any data. In this paper we propose Dirichlet-aggregation regression (DAR) to meet the challenge. DAR represents all data by means of convex combinations of only few extreme ones computable in linear time and easy to interpret.Then, it puts a Gaussian process prior on the Dirichlet distributions induced on the simplex spanned by the extremes. The prior can be a function of any observed meta feature such as time, location, type of fertilization, and plant species. We evaluated DAR on two hyperspectral image series of plants over time with about 2 (resp. 5.8) Billion matrix entries. The results demonstrate that DAR can be learned efficiently and predicts stress well before it becomes visible to the human eye.

Spectral hashing (SH) seeks compact binary codes of data points so that Hamming distances between codes correlate with data similarity. Quickly learning such codes typically boils down to principle component analysis (PCA). However, this is only justified for normally distributed data. For proportional data (normalized histograms), this is not the case. Due to the sum-to-unity constraint, features that are as independent as possible will not all be uncorrelated. In this paper, we show that a linear-time transformation efficiently copes with sum-to-unity constraints: first, we select a small number K of diverse data points by maximizing the volume of the simplex spanned by these prototypes; second, we represent each data point by means of its cosine similarities to the K selected prototypes. This maximum volume hashing is sensible since each dimension in the transformed space is likely to follow a von Mises (vM) distribution, and, in very high dimensions, the vM distribution closely resembles a Gaussian distribution. This justifies to employ PCA on the transformed data. Our extensive experiments validate this: maximum volume hashing outperforms spectral hashing and other state of the art techniques.

Recognizing human actions in videos has become a rapidly growing area of research. Most existing research has focused only on a single aspect i.e. recognition of actions. However, humans tend to perform different actions in their own styles. In this paper, we deal with the problem of simultaneously identifying actions and the underlying styles (actors) in videos. We propose a hierarchical approach based on conventional action recognition and asymmetric bilinear modeling. Our approach is solely based on dynamics of the underlying activity. Results on the multi-actor multi-action data set IX-MAS show a high recognition rate.

Gait recognition is receiving increasing attention from computer vision researchers for its applicability in areas such as visual surveillance, access control, or smart interfaces. Most existing research attempts to model individual gait patterns as sequences of temporal templates either by determining gait cycles or by aggregating spatio-temporal information into a 2D signature. This paper presents a simple yet efficient and effective approach to gait recognition based on a contour-distance feature and key pose learning. Unlike existing work, gait patterns are modelled as a non-temporal collection of key poses distributed over gait cycles. Experimental results on a large multi-view benchmark data set exhibit high recognition accuracy and robustness against changes in viewpoint. Consequently, this paper establishes that non-temporal methods can accomplish efficient and accurate gait recognition.

Early stress recognition is of great relevance in precision plant protection. Pre-symptomatic water stress detection is of particular interest, ultimately helping to meet the challenge of “How to feed a hungry world?”. Due to the climate change, this is of considerable political and public interest. Due to its large-scale and temporal nature, e.g., when monitoring plants using hyper-spectral imaging, and the demand of physical meaning of the results, it presents unique computational problems in scale and interpretability. However, big data matrices over time also arise in several other real-life applications such as stock market monitoring where a business sector is characterized by the ups and downs of each of its companies per year or topic monitoring of document collections. Therefore, we consider the general problem of embedding data matrices into Euclidean space over time without making any assumption on the generating distribution of each matrix. To do so, we represent all data samples by means of convex combinations of only few extreme ones computable in linear time. On the simplex spanned by the extremes, there are then natural candidates for distributions inducing distances between and in turn embeddings of the data matrices. We evaluate our method across several domains, including synthetic, text, and financial data as well as a large-scale dataset on water stress detection in plants with more than 3 billion matrix entries. The results demonstrate that the embeddings are meaningful and fast to compute. The stress detection results were validated by a domain expert and conform to existing plant physiological knowledge.

Low-rank approximations which are computed from selected rows and columns of a given data matrix have attracted considerable attention lately. They have been proposed as an alternative to the SVD because they naturally lead to interpretable decompositions which was shown to be successful in application such as fraud detection, fMRI segmentation, and collaborative filtering. The CUR decomposition of large matrices, for example, samples rows and columns according to a probability distribution that depends on the Euclidean norm of rows or columns or on other measures of statistical leverage. At the same time, there are various deterministic approaches that do not resort to sampling and were found to often yield factorization of superior quality with respect to reconstruction accuracy. However, these are hardly applicable to large matrices as they typically suffer from high computational costs. Consequently, many practitioners in the field of data mining have abandon deterministic approaches in favor of randomized ones when dealing with today's large-scale data sets. In this paper, we empirically disprove this prejudice. We do so by introducing a novel, linear-time, deterministic CUR approach that adopts the recently introduced Simplex Volume Maximization approach for column selection. The latter has already been proven to be successful for NMF-like decompositions of matrices of billions of entries. Our exhaustive empirical study on more than 30 synthetic and real-world data sets demonstrates that it is also beneficial for CUR-like decompositions. Compared to other deterministic CUR-like methods, it provides comparable reconstruction quality but operates much faster so that it easily scales to matrices of billions of elements. Compared to sampling-based methods, it provides competitive reconstruction quality while staying in the same run-time complexity class.

Tragically, mass gatherings such as music festivals, sports events or pilgrimage quite often end in terrible crowd disasters with many victims. In the past, research focused on developing physical models that model human behavior in order to simulate pedestrian flows and to identify potentially hazardous locations. However, no automatic systems for detection of dangerous motion behavior in crowds exist. In this paper, we present an automatic system for the detection and early warning of dangerous situations during mass events. It is based on optical flow computations and detects patterns of crowd motion that are characteristic for hazardous congestions. By applying an online change-point detection algorithm, the system is capable of identifying changes in pedestrian flow and thus alarms security personnel to take necessary actions.

There have recently been considerable advances in fast inference for (online) latent Dirichlet allocation (LDA). While it is widely recognized that the scheduling of documents in stochastic optimization and in turn in LDA may have significant consequences, this issue remains largely unexplored. Instead, practitioners schedule documents essentially uniformly at random, due perhaps to ease of implementation, and to the lack of clear guidelines on scheduling the documents.

We introduce and study the spectral evolution model, which characterizes the growth of large networks in terms of the eigenvalue decomposition of their adjacency matrices: In large networks, changes over time result in a change of a graph's spectrum, leaving the eigenvectors unchanged. We validate this hypothesis for several large social, collaboration, authorship, rating, citation, communication and tagging networks, covering unipartite, bipartite, signed and unsigned graphs. Following these observations, we introduce a link prediction algorithm based on the extrapolation of a network's spectral evolution. This new link prediction method generalizes several common graph kernels that can be expressed as spectral transformations. In contrast to these graph kernels, the spectral extrapolation algorithm does not make assumptions about specific growth patterns beyond the spectral evolution model. We thus show that it performs particularly well for networks with irregular, but spectral, growth patterns.

Matrix factorization methods are among the most common techniques for detecting latent components in data. Popular examples include the Singular Value Decomposition or Non-negative Matrix Factorization. Unfortunately, most methods suffer from high computational complexity and therefore do not scale to massive data. In this paper, we present a linear time algorithm for the factorization of gigantic matrices that iteratively yields latent components. We consider a constrained matrix factorization s.t.~the latent components form a simplex that encloses most of the remaining data. The algorithm maximizes the volume of that simplex and thereby reduces the displacement of data from the space spanned by the latent components. Hence, it also lowers the Frobenius norm, a common criterion for matrix factorization quality. Our algorithm is efficient, well-grounded in distance geometry, and easily applicable to matrices with billions of entries. In addition, the resulting factors allow for an intuitive interpretation of data: every data point can now be expressed as a convex combination of the most extreme and thereby often most descriptive instances in a collection of data. Extensive experimental validations on web-scale data, including 80 million images and 1.5 million twitter tweets, demonstrate superior performance compared to related factorization or clustering techniques.

In this paper, we present a novel approach to age recognition from facial images. The method we propose, combines several established features in order to characterize facial characteristics and aging patterns. Since we explicitly consider age recognition in the wild, i.e. vast amounts of unconstrained Internet images, the methods we employ are tailored towards speed and efficiency. For evaluation, we test different classifiers on common benchmark data and a new data set of unconstrained images harvested from the Internet. Extensive experimental evaluation shows state of the art performance on the benchmarks, very high accuracy for the novel data set, and superior runtime performance; to our knowledge, this is the first time that automatic age recognition is carried out on a large Internet data set.

We consider the problem of finding points of interest along local curves of binary images. Information theoretic vector quantization is a clustering algorithm that shifts cluster centers towards the modes of principal curves of a data set. Its runtime characteristics, however, do not allow for efficient processing of many data points. In this paper, we show how to solve this problem when dealing with data on a 2D lattice. Borrowing concepts from signal processing, we adapt information theoretic clustering to the quantization of binary images and gain significant speedup.

Recognizing human actions is of vital interest in video surveillance or ambient assisted living. We consider an action as a sequence of body poses which are themselves a linear combination of body parts. In an offline procedure, nonnegative tensor factorization is used to extract basis images that represent body parts. The weighting coefficients are obtained by filtering a frame with the set of basis images. Since the basis images are obtained from nonnegative tensor factorization, they are separable and filtering can be implemented efficiently. The weighting coefficients encode dynamics and are used for action recognition. In the proposed action recognition framework, neither explicit detection and tracking of humans nor background subtraction are needed. Furthermore, for recognizing location specific actions, we implicitly take scene objects into account.

This paper considers the topic of automatic font recognition. The task is to recognize a specific font from a text snippet. Unlike previous contributions, we evaluate, how the frequencies of certain letters or words influence automatic recognition systems. The evaluation provides estimates on the general feasibility of font recognition under various changing conditions. Results on a data-set containing 747 different fonts shows that precision can vary between 16% and 94%, dependent on (i) which letters are provided, (ii) how many letters are provided, and (iii) which language is used - as these factors considerably influence the text snippet statistics. As a second contribution, we introduce a novel bag-of-features based approach to font recognition.

This paper presents a distributed Support Vector Machine (SVM) algorithm in order to detect malicious software (malware) on a network of mobile devices. The light-weight system monitors mobile user activity in a distributed and privacy-preserving way using a statistical classification model which is evolved by training with examples of both normal usage patterns and unusual behavior. The system is evaluated using the MIT reality mining data set. The results indicate that the distributed learning system trains quickly and performs reliably. Moreover, it is robust against failures of individual components.

In this paper, we consider the problem of automatically reading graffiti tags. As a preparatory step, we create a large set of synthetic graffiti-like characters, generated from publicly available true type fonts. For each character in the database, we extract a number of scale independent local binary descriptors. Then, using binary non negative matrix factorization, a sufficient number of basis functions are learned. Basis function coefficients of novel images can then be directly used for hashing characters from the database of prototypes. Finally, graffiti tags are recognized by means of a localized, spatial voting scheme.

Automatic classification of the aesthetic content of a picture is one of the challenges in the emerging discipline of computational aesthetics. Any suitable solution must cope with the facts that aesthetic experiences are highly subjective and that a commonly agreed upon theory of their psychological constituents is still missing. In this paper, we present results obtained from an empirical basis of several thousand images. We train SVM based classifiers to predict aesthetic adjectives rather than aesthetic scores and we introduce a probabilistic post processing step that alleviates effects due to misleadingly labeled training data. Extensive experimentation indicates that aesthetics classification is possible to a large extent. In particular, we find that previously established low-level features are well suited to recognize beauty. Robust recognition of unseemliness, on the other hand, appears to require more high-level analysis.

Collaborative tagging services (folksonomies) have been among the stars of theWeb 2.0 era. They allow their users to label diverse resources with freely chosen keywords (tags). Our studies of two real-world folksonomies unveil that individual users develop highly personalized vocabularies of tags. While these meet individual needs and preferences, the considerable differences between personal tag vocabularies (personomies) impede services such as social search or customized tag recommendation. In this paper, we introduce a novel user-centric tag model that allows us to derive mappings between personal tag vocabularies and the corresponding folksonomies. Using these mappings, we can infer the meaning of user-assigned tags and can predict choices of tags a user may want to assign to new items. Furthermore, our translational approach helps in reducing common problems related to tag ambiguity, synonymous tags, or multilingualism. We evaluate the applicability of our method in tag recommendation and tag-based social search. Extensive experiments show that our translational model improves the prediction accuracy in both scenarios.

Non-negative matrix factorization (NMF) has recently received a lot of attention in data mining, information retrieval, and computer vision. It factorizes a non-negative input matrix V into two non-negative matrix factors V = WH such that W describes "clusters" of the datasets. Analyzing genotypes, social networks, or images, it can be beneficial to ensure V to contain meaningful "cluster centroids", i.e., to restrict W to be convex combinations of data points. But how can we run this convex NMF in the wild, i.e., given millions of data points? Triggered by the simple observation that each data point is a convex combination of vertices of the data convex hull, we propose to restrict W further to be vertices of the convex hull. The benefits of this convex-hull NMF approach are twofold. First, the expected size of the convex hull of, for example, n random Gaussian points in the plane is ¿(¿log n), i.e., the candidate set typically grows much slower than the data set. Second, distance preserving low-dimensional embeddings allow one to compute candidate vertices efficiently. Our extensive experimental evaluation shows that convex-hull NMF compares favorably to convex NMF for large data sets both in terms of speed and reconstruction quality. Moreover, we show that our method can easily be applied to large-scale, real-world data sets, in our case consisting of 1.6 million images respectively 150 million votes on World of Warcraft ® guilds.

This paper presents a system that combines mobile services, ubiquitous computing, and augmented reality concepts in order to bring the kind of information-rich environment, which is currently limited to computer screens, to the physical world. This objective is achieved by constructing an architectu

We analyse the corpus of user relationships of the Slashdot technology news site. The data was collected from the Slashdot Zoo feature where users of the website can tag other users as friends and foes, providing positive and negative endorsements. We adapt social network analysis techniques to the problem of negative edge weights. In particular, we consider signed variants of global network characteristics such as the clustering coefficient, node-level characteristics such as centrality and popularity measures, and link-level characteristics such as distances and similarity measures. We evaluate these measures on the task of identifying unpopular users, as well as on the task of predicting the sign of links and show that the network exhibits multiplicative transitivity which allows algebraic methods based on matrix multiplication to be used. We compare our methods to traditional methods which are only suitable for positively weighted edges.

We extend the resistance distance kernel to the domain of signed dissimilarity values, and show how it can be applied to collaborative rating prediction. The resistance distance is a graph kernel inspired by electrical network models where edges of a graph are interpreted as electrical resistances. We model the similarity between users of a large collaborative rating database using this signed resistance distance, generalizing the previously known regular resistance distance kernel which is limited to nonnegative values. We show that the signed resistance distance kernel can be computed effectively using the Moore-Penrose pseudoinverse of the Laplacian matrix of the bipartite rating graph, leading to fast computation based on the eigenvalue decomposition of the Laplacian matrix. We apply this technique to collaborative rating prediction on the Netflix Prize corpus, and show how our new kernel can replace the traditional Pearson correlation for rating prediction.

We present a framework intended to assist users in the task of tagging pictures with content descriptors. Histogram- or correlogram features of manually indicated regions of interest are extracted from a few training images; probabilistic diffusion over these prototypes is used to analyze further images. Since speed is pivotal in interactive applications, we apply a fast algorithm for computing local correlograms; moreover, our diffusion-based classifier trains almost instantaneously. Experiments with images downloaded from flickr.com indicate that our method achieves good results even when trained with a single image only.

Compared to only a few years ago, today there is an abundance of annotated image data available on the Internet. For researchers on image retrieval, this is an unforseen but welcome consequence of the rise of Web 2.0 technologies. Popular social networking and content sharing services seem to hold the key to the integration of context and semantics into retrieval. However, at least for now, it appears that this promise has to be taken with a grain of salt. In this paper, we present preliminary empirical results on the tagging behavior of power users of content sharing and social bookmarking services. Our findings suggest different promising research directions for image retrieval and we briefly discuss some of them.

We present a distributed machine learning framework based on support vector machines that allows classification problems to be solved iteratively through parallel update algorithms with minimal communication overhead. Decomposing the main problem into multiple relaxed subproblems allows them to be simultaneously solved by individual computing units operating in parallel and having access to only a subset of the data. A sufficient condition is derived under which a synchronous, discrete-time gradient update algorithm converges to the approximate solution. We apply the proposed distributed learning framework in the context of automatic image tagging as a first processing layer. Initial results from corresponding experiments indicate that he proposed framework has favorable properties including efficiency, configurability, robustness, suitability for online learning, and low communication overhead.

This paper presents a stochastic diffusion approach to prototype-based classification. Relations between exemplary objects and their features are modeled in a bipartite graph. A Bayesian interpretation of the model leads to a Markov chain over the set of objects. In contrast to related graph diffusion approaches, our dual treatment of objects and features easily copes with out of sample objects. Applied to problems in color object localization in unconstrained images, our method performs robust and yields promising results.

Given a bipartite graph of collaborative ratings, the task of recommendation and rating prediction can be modeled with graph kernels. We interpret these graph kernels as the inverted squared Euclidean distance in a space defined by the underlying graph and show that this inverted squared Euclidean similarity function can be replaced by other similarity functions. We evaluate several such similarity functions in the context of collaborative item recommendation and rating prediction, using the exponential diffusion kernel, the von Neumann kernel, and the random forest kernel as a basis. We find that the performance of graph kernels for these tasks can be increased by using these alternative similarity functions.

We propose a method for the detection of trends in social bookmarking systems. Compared to other work in this emerging field, our approach has a more sound statistical basis. In order to cope with the problem of vanishing probabilities due to data sparsity, we apply smoothing and show that it allows for an easy calibration of our trend detector resulting in better generalization and scalability. We test our approach on a collection of 105, 000, 000 bookmarks collected from the del.icio.us bookmarking service. To our knowledge, this is the largest corpus of a real world bookmarking service analyzed in this context. The results show that our method outperforms previously proposed methods and successfully detects trends in the data.

We introduce a lightweight biometric solution for user authentication over networks using online handwritten signatures. The algorithm proposed is based on a modified Hausdorff distance and has favorable characteristics such as low computational cost and minimal training requirements. Furthermore, we investigate an information theoretic model for capacity and performance analysis for biometric authentication which brings additional theoretical insights to the problem. A fully functional proof-of-concept prototype that relies on commonly available off-the-shelf hardware is developed as a client-server system that supports Web services. Initial experimental results show that the algorithm performs well despite its low computational requirements and is resilient against over-the-shoulder attacks.

We propose an approach to image retrieval that does not require any distance computations. The idea is to represent images and corresponding image features by means of the two sets of vertices of a bipartite graph. Even though in such a graph the images are not directly related, the degrees to which the features are present in an image allow for defining partial orders. If the degrees of presence are normalized such that they form probability distributions, similarity rankings result from the stationary distributions of stochastic diffusion processes over the graph. The method is closely related to recent approaches to ranking on manifolds but does not involve the computation of parameterized affinity and Laplacian matrices. Experiments with a standard image retrieval data set demonstrate the efficacy of the approach. Compared to a corresponding distance-based approach, it yields a higher overall precision. 1

The World Wide Web, which has started as a document repository, is rapidly transforming to a full fledged virtual environment that facilitates services, interaction, and communication. Under this light, the Semantic Web and Web 2.0 movements can be seen as intermediate steps of a natural evolution towards a new paradigm, the 3D Internet. We provide an overview of the concept 3D Internet and discuss why it is a goal worth pursuing, what it does entail, and how one can realize it. Our goal in this paper is to discuss a research agenda and raise interest in networking, security, distributed computing, and machine learning communities. We explore first the motivation for the 3D Internet and the possibilities it brings. Subsequently, we investigate the specific system level and research challenges that need to be addressed in order to make the 3D Internet a reality.

Tensor-based approaches to visual object detection can drastically reduce the number of parameters in the training process. Compared to their vector-based counterparts, tensor methods therefore train faster, better manage noisy or corrupted training samples, and are less prone to over-fitting. In this paper, we show how to incorporate the kernel trick into tensor-based filter design. Dealing with object detection in cluttered natural environments, the method is shown to cope with substantially varying training data and a cascade of only two kernel tensor-filters is demonstrated to provide very reliable results.

This paper presents an expert peering system for information exchange in the knowledge society. Our system realizes an intelligent, real-time search engine for enterprise Intranets or online communities that automatically relays user queries to knowledgable specialists. According to its very nature, the system requires a sound integration of concepts drawn from various areas of Computer Science. In addition to our solutions to problems in scalable processing, data transfer, and networking, we also address issues of interface design and usability, as well as aspects of machine intelligence. Results obtained from extensive experiments demonstrate the efficiency and robustness of our system and convey its potential for next generation web services.

Some of the established approaches to evaluating text clustering algorithms for information retrieval show theoretical flaws. In this paper, we analyze these flaws and introduce a new evaluation measure to overcome them. Based on a simple yet rigorous mathematical analysis of the effect of certain parameters in cluster based retrieval, we show that certain conclusions drawn in the recent literature must be taken with a grain of salt. Our new measure, in contrast, accounts for statistical biases that have to be expected according to our analysis. A series of experiments and a comparison with results reported recently underlines that this measure is a more suitable performance indicator that allows for more meaningful interpretations.

We propose a hierarchical approach to document categorization that requires no pre-configuration and maps the semantic document space to a predefined taxonomy. The utilization of search engines to train a hierarchical classifier makes our approach more flexible than existing solutions which rely on (human) labeled data and are bound to a specific domain. We show that the structural information given by the taxonomy allows for a context aware construction of search queries and leads to higher tagging accuracy. We test our approach on different benchmark datasets and evaluate its performance on the single- and multi-tag assignment tasks. The experimental results show that our solution is as accurate as supervised classifiers for web page classification and still performs well when categorizing domain specific documents.

Ensemble methods such as bootstrap, bagging or boosting have had a considerable impact on recent developments in machine learning, pattern recognition and computer vision. Theoretical and practical results alike have established that, in terms of accuracy, ensembles of weak classifiers generally outperform monolithic solutions. However, this comes at the cost of an extensive training process. The work presented in this paper results from projects on advanced human machine interaction. In scenarios like ours, online learning is a major requirement, and lengthy training is prohibitive. We therefore propose a different approach to ensemble learning. Instead of a set of weak classifiers, we combine strong, separable, multilinear discriminant functions. These are especially suited for computer vision: they train very quickly and allow for rapid classification of image content. Training different classifiers for different contexts or on semantically organized data provides ensembles of experts. We collapse a set of experts into a single multilinear function and thus achieve the same runtime for arbitrarily many classifiers as for a single one. Moreover, carrying out the classification in the frequency domain results in faster framerates. Experiments with image sequences recorded in typical home environments show that our ensemble training schemes yield high accuracy on unconstrained and cluttered data.

This paper reports on tree-based shape encoding and classification. We present an approach that combines characteristics from the theories of R-trees known from data base indexing and regression trees known from pattern recognition. The resulting shape representations are highly storage efficient. As they immediately transform into scale invariant signatures, we apply the Earth Mover's distance for computing shape similarities. Experimental results underline the efficacy of this approach. The required computations are simple and fast but allow for robust shape classification and clustering.

Interactivity is a key concept in modern content-based retrieval. Therefore, in addition to the ability to learn from user generated data, easy and intuitive to use interfaces are an important area of research in (multi)media retrieval. In this contribution, we focus on the latter aspect and present how different modalities like speech and gestures on super sized touch screen facilities may be integrated to accomplish the goal of intuitive interaction. In order to evaluate our approach, we conducted a series of usability experiments. Their results demonstrate that our multimodal user interface allows for both, comfortable and successful interactive image retrieval

This paper presents a fast method to compute homeomorphisms between 2D lattices and shapes found in binary images. Unlike many other methods, this mapping is not restricted to simply connected shapes but applies to arbitrary topologies. Moreover, it provides an avenue to the embedding of shapes in vectorspaces over R and C and thus enables robust shape recognition.

As wearable sensors and computing hardware are becoming a reality, new and unorthodox approaches to seamless human-computer interaction can be explored. This paper presents the prototype of a wearable, head-mounted device for advanced human-machine interaction that integrates speech recognition and computer vision with head gesture analysis based on inertial sensor data. We will focus on the innovative idea of integrating visual and inertial data processing for interaction. Fusing head gestures with results from visual analysis of the environment provides rich vocabularies for human-machine communication because it renders the environment into an interface: if objects or items in the surroundings are being associated with system activities, head gestures can trigger commands if the corresponding object is being looked at. We will explain the algorithmic approaches applied in our prototype and present experiments that highlight its potential for assistive technology. Apart from pointing out a new direction for seamless interaction in general, our approach provides a new and easy to use interface for disabled and paralyzed users in particular.

Graphs and graph matching are powerful mechanisms for knowledge representation, pattern recognition and machine learning. Especially in computer vision their application is manifold. Graphs can characterize relations among image features like points or regions but they may also represent symbolic object knowledge. Hence, graph matching can accomplish recognition tasks on different levels of abstraction. In this contribution, we demonstrate that graphs may also bridge the gap between different levels of knowledge representation. We present a system for visual assembly monitoring that integrates bottom-up and top-down strategies for recognition and automatically generates and learns graph models to recognize assembled objects. Data-driven processing is subdived into three stages: first, elementary objects are recognized from low-level image features. Then, clusters of elementary objects are analyzed syntactically; if an assembly structure is found, it is translated into a graph that uniquely models the assembly. Finally, symbolic models like this are stored in a database so that individual assemblies can be recognized by means of graph matching. At the same time, these graphs enable top-down knowledge propagation: they are transformed into graphs which represent relations between image features and thus describe the visual appearance of the recently found assembly. Therefore, due to model-driven knowledge propagation assemblies may subsequently be recognized from graph matching on a lower computational level and tedious bottom-up processing becomes superfluous.

Given the size of todays professional image databases, the stan-dard approach to object- or theme-related image retrieval is to in-teractively navigate through the content. But as most users of such databases are designers or artists who do not have a technical back-ground, navigation interfaces must be intuitive to use and easy to learn. This paper reports on efforts towards this goal. We present a system for intuitive image retrieval that features different moda-lities for interaction. Apart from conventional input devices like mouse or keyboard it is also possible to use speech or haptic gesture to indicate what kind of images one is looking for. Seeing a selection of images on the screen, the user provides relevance feedback to narrow the choice of motifs presented next. This is done either by scoring whole images or by choosing cer-tain image regions. In order to derive consistent reactions from multimodal user input, asynchronous integration of modalities and probabilistic reasoning based on Bayesian networks are applied. After addressing technical details, we will discuss a series of usability experiments, which we conducted to examine the impact of multimodal input facilities on interactive image retrieval. The results indicate that users appreciate multimodality. While we ob-served little decrease in task performance, measures of contentment exceeded those for conventional input devices.

The capability to coordinate and interrelate speech and vision is a virtual prerequisite for adaptive, cooperative, and flexible interaction among people. It is therefore fair to assume that human-machine interaction, too, would benefit from intelligent interfaces for integrated speech and image processing. We first sketch an interactive system that integrates automatic speech processing with image understanding. Then, we concentrate on performance assessment which we believe is an emerging key issue in multimodal interaction. We explain the benefit of time scale analysis and usability studies and evaluate our system accordingly.

Many computer vision tasks rely on feature extraction. Interest points are such features. This paper shows that interest points are geometrically stable under different transformations and have high information content (distinctiveness). These two properties make interest points very successful in the contest of image matching. To measure these two properties quantitatively, we introduce two evaluation criteria: repeatability rate and information content. The quality of the interest points depends on the detector used. In this paper several detectors are compared according to the criteria specified above. We determine which detector gives the best results and show that it satisfies the criteria well.