Information extraction and analysis in healthcare have benefited significantly from advances in the AI and NLP community in recent years, often directly impacting human lives or at least their well-being. However, processing and analyzing mostly unstructured and semi-structured health and biomedical information presents substantial challenges to researchers. The inherent complexity of the domain, limited data availability, data ambiguity, and the high cost of involving medical experts make supporting healthcare difficult. With the recent revolution in AI, particularly in Large Language Models (LLMs), the quality of analyses of these data has increased significantly - though it has not yet reached the level required by medical practitioners. In this work, we address the challenge of information reasoning and question answering, with a particular focus on healthcare. We conduct a detailed analysis of PubMedQA, one of the most challenging and prominent benchmarks. We design, implement, and evaluate Med-RCQ (Medical Reasoning by Concluding and Questioning), a novel LLM-based method to analyze health and biomedical information with the objective of supporting informed decisions. Our PubMedQA benchmark results show that MedRCQ outperforms most competing open approaches in terms of result quality and even provides similar results compared to large commercial models, while outperforming all other benchmark results in an ‘accuracy per number of model parameters’ metric.

Multimedia data, such as images and videos, con-tinue to be recorded and created in ever increasing quantities. Multimedia collections on social media, but also private collec-tions of personal vacation and home videos are being collected at a higher rate than ever. This rapid growth makes it difficult, and in many cases impossible, to effectively use this kind of data without the use of automated analysis methods. While many methods have been developed to automate the analysis of multimedia data, users are unable to access the information extracted through this analysis without appropriate interfaces. Although purpose-built user interfaces are important even for use by trained experts, they are essential in facilitating the interaction with large multimedia data collections for laypersons and novices. In this paper, we describe a new and improved version of vitrivr- VR, a prototype multimedia analytics system built for immersive interfaces, in the form in which it will participate in the Video Retrieval for Beginners (VR4B) evaluation campaign. vitrivr- VR aims to improve the video browsing experience for novices through the additional affordances provided by immer-sive interfaces. We describe the current state of the system with a focus on features and adjustments made to support accessibility of large scale multimedia analytics for novices.

The vitrivr multimedia retrieval stack has been shown to be versatile and effective in many areas and application domains. In this paper, we present vitrivr's new retrieval engine, the vitrivr-engine, a flexible and highly configurable information extraction and query processing solution with support for a broad range of media types.

The field of information retrieval, especially when targeting multimodal content, has found ways of satisfying a broad range of information needs, which can be expressed in a multitude of ways. In contrast to related fields, such as relational databases, no universal way of representing the queries to be answered by a retrieval system has emerged. In this paper, we present an initial proposal for a universal query representation mechanism for multimodal information retrieval. The proposed approach imperatively expresses arbitrary information needs, using a DAG of query primitives. We show how such a representation can be used for both feature extraction and query processing pipelines and how it can serve as a foundation towards a query language for information retrieval.

Performance evaluation in multimedia retrieval, as in the information retrieval domain at large, relies heavily on retrieval experiments, employing a broad range of techniques and metrics. These can involve human-in-the-loop and machine-only settings for the retrieval process itself and the subsequent verification of results. Such experiments can be elaborate and use-case-specific, which can make them difficult to compare or replicate. In this article, we present a formal model to express all relevant aspects of such retrieval experiments, as well as a flexible open-source evaluation infrastructure that implements the model. These contributions intend to make a step towards lowering the hurdles for conducting retrieval experiments and improving their reproducibility.

With the rapidly increasing amount of multimedia content encountered in every day life, interactive multimedia retrieval continues to rise in importance. While a lot of progress has been made in making interactive multimedia retrieval accessible even to novice users using conventional desktop interfaces, novel interface technologies, such as immersive interfaces, have not received enough attention. With the rise in popularity and the increasing affordability of mixed and virtual reality (VR) devices, new methods to enable interactive multimedia retrieval, especially for novice users, are needed. In this paper, we describe a version of the vitrivr-VR virtual reality multimedia retrieval system enhanced to increase usability by novices. By providing simple and intuitive interactions and interfaces with the most common functionality prominently accessible, and more advanced functionality available for those who want to use it, we expect to be able to provide an effective and enjoyable search experience for novice users and experts alike. Specifically, we simplify the search interface to a single search bar, provide convenient and intuitive VR text input methods, and support simple and familiar results browsing interfaces ideal for novice users.

Within the domain of multimedia formats, 3D models – alongside images, videos, and texts – are rapidly gaining prominence in applications and research. As the uses of 3D models increase and it becomes easier and more accessible to capture and digitally create 3D models, methods are required to allow automatic analysis and retrieval within large collections. While a lot of previous work has focused on geometry-based retrieval, these methods usually require an example 3D model to query, and often struggle to capture appearance information expressed through textures and materials.In this paper, we propose a novel view-based approach that enables embedding of 3D models into multi-modal embedding spaces, by rendering 2D planar projections. Furthermore, our approach addresses the challenge of viewpoint selection for 3D model rendering, in order to maximize its semantic recognition. We demonstrate the capability of our approach using two pre-trained multi-modal embedding models applied to a large collection of modern 3D models.

Virtual reality (VR) applications are increasingly permeating our lives. The immersion provided by VR enables novel interactions with data that would be impossible in conventional environments. Especially regarding multimedia data, VR could overcome existing limitations when browsing videos to find specific scenes. This paper introduces a virtual multimedia drawer, tailored to VR environments, to enable novel ways to interact with videos. A within-subjects design experiment (N=24) was conducted to evaluate the multimedia drawer on user experience, efficiency, and effectiveness. Results show that the multimedia drawer, while taking slightly longer to locate a particular scene within a video for certain types of tasks (i.e., sequence tasks), provides statistically significantly higher levels of enjoyment, novelty, and stimulation and is preferred over conventional timeline-based approaches. Implications of quantitative and qualitative results regarding the design and features of the multimedia drawer as a video browsing method in VR are critically discussed.

This tutorial provides an overview of multimedia retrieval in the context of eXtended Reality (XR), including using virtual and augmented/mixed reality as a user interface for multimedia retrieval, as well as multimedia search tasks addressing content needs for the creation of XR experiences.It will discuss the opportunities and limitations of XR-based search, the evaluation of XR-based multimedia retrieval systems, the demonstration of selected research systems, and open research challenges.

The way we create, consume and interact with multimedia content has changed significantly in recent years with the advent of affordable recording devices and easy sharing and access in the form of mobile phones. With the imminent wave of affordable devices that enable mixed reality experiences and the large variety of devices on the market, interaction with multimedia content is expected to continue to evolve rapidly. This will also drastically affect the entire area of multimedia information retrieval in eXtended Reality (XR), for instance by novel ways to express user needs in VR, result presentation that takes the specific capabilities of XR devices into account, and/or result feedback. This tutorial on Multimedia Retrieval in XR discusses and demonstrates existing solutions and highlights key challenges in this evolving field.

Video retrieval still offers many challenges which can so far only be effectively mediated through interactive, human-in-the-loop retrieval approaches. The vitrivr multimedia retrieval stack offers a broad range of query mechanisms to enable users to perform such interactive retrieval. While these multiple mechanisms offer various options to experienced users, they can be difficult to use for novices. In this paper, we present a minimal user interface geared towards novice users that only exposes a subset of vitrivr’s functionality but simplifies user interaction.

Interactive retrieval with user-friendly and performant interfaces remains a necessity for video retrieval, even in light of significant gains in retrieval performance through multi-modal encoders. In recent years, novel interaction modalities such as virtual reality (VR) and augmented reality (AR) have gained popularity, but the best way to adapt paradigms from traditional retrieval interfaces, especially for result browsing and interaction, remains an open research question. In this paper, we compare two video retrieval interfaces in a controlled setting to gain insight into the differences in video browsing between VR and desktop interfaces. We formulate hypotheses explaining why there might be performance differences between the two interfaces, define metrics to test the hypotheses, and show results based on data gathered at an evaluation campaign. Our results show that VR interfaces can be competitive in browsing performance and indicate that there can even be an advantage when browsing larger result sets in VR.

The past decades have seen an exponential growth in the amount of data which is produced by individuals. Smartphones which capture images, videos and sensor data have become commonplace, and wearables for fitness and health are growing in popularity. Lifelog retrieval systems aim to aid users in finding and exploring their personal history. We present two systems for lifelog retrieval: vitrivr and vitrivr-VR, which share a common retrieval model and backend for multi-modal multimedia retrieval. They differ in the user interface component, where vitrivr relies on a traditional desktop-based user interface and vitrivr-VR has a Virtual Reality user interface. Their effectiveness is evaluated at the Lifelog Search Challenge 2021, which offers an opportunity for interactive retrieval systems to compete with a focus on textual descriptions of past events. Our results show that the conventional user interface outperformed the VR user interface. However, the format of the evaluation campaign does not provide enough data for a thorough assessment and thus to make robust statements about the difference between the systems. Thus, we conclude by making suggestions for future interactive evaluation campaigns which would enable further insights.

In real-world datasets, specifically in TV recordings, videos are often multi-person and multi-angle, which poses significant challenges for gesture recognition and retrieval. In addition to being of interest to linguists, gesture retrieval is a novel and challenging application for multimedia retrieval. In this paper, we propose a novel method for spatio-temporal gesture retrieval based on visual and pose information which can retrieve similar gestures in multi-person scenes through continuous shots. The attention-aware features, extracted from human pose key-points, together with a sophisticated pre-processing module, alleviate the susceptibility of gesture retrieval to background noise and occlusion. We have evaluated our method on a subset of the NewsScape Dataset. Our experimental results demonstrate the effectiveness of the proposed method in retrieving similar results in occluded scenes as measured by the quality of the top 5 results.

The harvesting, management, and analysis of thematic document collections is a major challenge in a wide variety of applications. While the criteria for compiling such collections are individual, the entire process is largely standardized. Therefore, it is not efficient to build new systems over and over again to take over these tasks. In this work, we introduce Seer-Dock, a novel and easy-to deploy general-purpose dockerized framework to build a scholarly document harvesting and management system. It is based on CiteSeerX, the most widely used scholarly search engine. Seer-Dock uses docker containers for all components and thus enables its users to rapidly deploy a full-fledged document collection and management system on any operating system platform and tailor it to the specific needs of an application domain. Moreover, it is easy to scale, orchestrate, maintain, and recover. In this resource paper, we introduce the architecture of Seer-Dock and its components. Like its kernel CiteSeerX, Seer-Dock is available under an Apache 2 open source license.

In medical imaging, modality refers to the technique and process used to create visual representations of a particular part of the body, organs, or tissues. Conventional modalities include X-ray, CT-scan, Magnetic Resonance Imaging (MRI), Ultrasound, and Positron Emission Tomography (PET). Depending on the modality used, the same disease can be detected differently, making a modality an essential filter in evaluating the relevance of search results when retrieving similar medical images. Traditionally, texture features have been used for content-based medical image retrieval. However, texture features are limited in capturing the semantic similarity between medical images, let alone their modalities. This paper explores deep features (features extracted by deep convolutional neural networks (CNN)) and analyzes their effectiveness in retrieving similar medical images, semantic-wise and modality-wise, from a collection with different medical image modalities. We have examined CNNs of different architectures pre-trained in natural images and CNNs we fine-tuned and fully-trained from scratch in medical images to extract deep features. Based on retrieval performance evaluation, we show that deep features, even though extracted by CNN pre-trained in natural images, still outperform texture features. On the other end, we show that deep features extracted by a smaller, simpler, and yet computationally efficient CNN we trained in medical images can compete with large and complex ImageNet CNNs fine-tuned or fully trained in medical images.

In recent years, Big Data has become an important topic in many areas of our daily lives, including sports. Almost all professional clubs analyze matches to improve the performance of their teams. However, events are still predominantly captured manually, although many sensor-based and video-based tracking systems exist which provide the positions of the players and the ball in real-time. This manual process is tedious and error-prone. In this paper, we present STREAMTEAM-FOOTBALL, an open source football analysis application, that fills this gap. STREAMTEAM-FOOTBALL allows to analyze football matches fully automatically and in real-time on the basis of tracked position data using a data stream analysis approach. Our evaluation confirms the effectiveness of our automated analysis.

Systems evaluations are an important part of empirical research in computer science. Such evaluations encompass the systematic assessment of the run-time characteristics of systems based on one or several parameters. Considering all possible parameter settings is often a very tedious and time-consuming task with many manual activities, or at least the manual creation of evaluation scripts. Ideally, the thorough evaluation of a complete evaluation space can be fully automated. This includes the set-up of the evaluation, its execution, and the subsequent analysis of the results. In this paper, we introduce Chronos, a system for the automation of the entire evaluation workflow. While Chronos has originally been built for database systems evaluations, its generic approach also allows its usage in other domains. We show how Chronos can be deployed for a concrete database evaluation, the comparative performance analysis of different storage engines in MongoDB.

Gestures play an important role in our daily communications. However, recognizing and retrieving gestures in-the-wild is a challenging task which is not explored thoroughly in literature. In this paper, we explore the problem of identifying and retrieving gestures in a large-scale video dataset provided by the computer vision community and based on queries recorded in-the-wild. Our proposed pipeline, I3DEF, is based on the extraction of spatio-temporal features from intermediate layers of an I3D network, a state-of-the-art network for action recognition, and the fusion of the output of feature maps from RGB and optical flow input. The obtained embeddings are used to train a triplet network to capture the similarity between gestures. We further explore the effect of a person and body part masking step for improving both retrieval performance and recognition rate. Our experiments show the ability of I3DEF to recognize and retrieve gestures which are similar to the queries independently of the depth modality. This performance holds both for queries taken from the test data, and for queries using recordings from different people performing relevant gestures in a different setting.

Multimedia retrieval and analysis are two important areas in "Big data" research. They have in common that they work with feature vectors as proxies for the media objects themselves. Together with metadata such as textual descriptions or numbers, these vectors describe a media object in its entirety, and must therefore be considered jointly for both storage and retrieval.

The steady growth of multimedia collections - both in terms of size and heterogeneity - necessitates systems that are able to conjointly deal with several types of media as well as large volumes of data. This is especially true when it comes to satisfying a particular information need, i.e., retrieving a particular object of interest from a large collection. Nevertheless, existing multimedia management and retrieval systems are mostly organized in silos and treat different media types separately. Hence, they are limited when it comes to crossing these silos for accessing objects. In this paper, we present vitrivr, a general-purpose content-based multimedia retrieval stack. In addition to the keyword search provided by most media management systems, vitrivr also exploits the object's content in order to facilitate different types of similarity search. This can be done within and, most importantly, across different media types giving rise to new, interesting use cases. To the best of our knowledge, the full vitrivr stack is unique in that it seamlessly integrates support for four different types of media, namely images, audio, videos, and 3D models.

Big data in sports comes with two closely related challenges: first, the online analysis of continuous data streams to identify characteristic events and second, advanced retrieval in video collections and/or event data that help game analysts to search for characteristic video scenes. For both challenges, dedicated big data stream processing and retrieval systems have been developed. However, there is no infrastructure yet that integrates retrieval and automatic online data stream analysis. In this paper, we close this gap by seamlessly combining STREAMTEAM, our real-time team sports analysis system, and SPORTSENSE, our team sports video retrieval system, to an integrated team sports analysis infrastructure that (i) automatically detects (collaborative) events and generates statistics in real-time based on a continuous stream of raw positions, (ii) visualizes the analysis results in real-time, (iii) stores the analysis results persistently for offline activities, and (iv) leverages the stored analysis results for intuitive sketch-based video retrieval.

Cloud providers are more and more confronted with very diverse and heterogeneous requirements their customers impose on the management of data. First, these requirements stem from service-level agreements that specify a desired degree of availability and a guaranteed latency. As a consequence, Cloud providers replicate data across data centers or availability zones and/or partition data and place it close to the location of their customers. Second, the workload at each Cloud data center or availability zone is diverse and may significantly change over time - e. g., an OLTP workload during regular business hours and OLAP analyzes over night. For this, polystore and multistore databases have recently been introduced as they are intrinsically able to cope with such mixed and varying workloads. While the problem of heterogeneous requirements on data management in the Cloud is either addressed at global level by replicating and partitioning data across data centers or at local level by providing polystore systems in a Cloud data center, there is no integrated solution that leverages the benefits of both approaches. In this paper, we present the Polypheny-DB vision of a distributed polystore system that seamlessly combines replication and partitioning with local polystores and that is able to dynamically adapt all parts of the system when the workload changes. We present the basic building blocks for both parts of the system and we discuss open challenges towards the implementation of the Polypheny-DB vision.

More and more companies move their data to the Cloud which is able to cope with the high scalability and availability demands due to its pay-as-you-go cost model. For this, databases in the Cloud are distributed and replicated across different data centers. According to the CAP theorem, distributed data management is governed by a trade-off between consistency and availability. In addition, the stronger the provided consistency level, the higher is the generated coordination overhead and thus the impact on system performance. Nevertheless, many OLTP applications demand strong consistency and use ROWA(A) for replica synchronization. ROWA(A) protocols eagerly update all (or all available) replicas and thus generate a high overhead for update transactions. In contrast, quorum-based protocols consider only a subset of sites for eager commit. This reduces the overhead for update transactions at the cost of reads, as the latter also need to access several sites. Existing quorum-based protocols do not consider the load of sites when determining the quorums; hence, they are not able to adapt at run-time to load changes. In this paper, we present QuAD, an adaptive quorum-based replication protocol that constructs quorums by dynamically selecting the optimal quorum configuration w.r.t. load and network latency. Our evaluation of QuAD based on Amazon EC2 shows that it considerably outperforms both static quorum protocols and dynamic protocols that neglect site properties in the quorum construction process.

The last years have seen a vast diversification on the database market. In contrast to the “one-size-fits-all” paradigm according to which systems have been designed in the past, today's database management systems (DBMS) are tuned for particular workloads. This has led to DBMSs optimized for high performance, high throughput read/write workloads in online transaction processing (OLTP) and systems optimized for complex analytical queries (OLAP). However, this approach reaches a limit when systems have to deal with mixed workloads that are neither pure OLAP nor pure OLTP workloads. In such cases, multistores are increasingly gaining popularity. Rather than supporting one single database paradigm and addressing one particular workload, multistores encompass several DBMSs that store data in different schemas and allow to route requests on a per-query level to the most appropriate system. In this paper, we introduce the multistore ICARUS. In our evaluation based on a workload that combines OLTP and OLAP elements, we show that ICARUS is able to speed-up queries up to a factor of three by properly routing queries to the best underlying DBMS.

The IMOTION system is a multimodal content-based video search and browsing application offering a rich set of query modes on the basis of a broad range of different features. It is able to scale with the size of the collection due to its underlying flexible polystore called ADAMpro and its very effective retrieval engine Cineast, optimized for multi-feature fusion. IMOTION is simultaneously geared towards precision-focused searches, i.e., known-item search with image or text queries, and recall-focused, exploratory searches. In this demo, we will present the 2017 IMOTION system deployed on the IACC.3 collection consisting of 600 hours of Internet Archive video, which was also used in the TRECVID 2016 Ad-Hoc Video Search and in the 2017 Video Browser Showdown (VBS) challenge in which IMOTION ranked first. Conference attendees will have the chance to interact with the 2017 IMOTION system and quickly solve various retrieval tasks.

Applications deployed in the Cloud usually come with dedicated performance and availability requirements. This can be achieved by replicating data across several sites and/or by partitioning data. Data replication allows to parallelize read requests and thus to decrease data access latency, but induces significant overhead for the synchronization of updates. Partitioning, in contrast, is highly beneficial if all the data accessed by an application is located at the same site, but again necessitates coordination if distributed transactions are needed to serve applications. In this paper, we analyze three protocols for distributed data management in the Cloud, namely Read-One-Write-All-Available (ROWAA), Majority Quorum (MQ) and Data Partitioning (DP)-all in a configuration that guarantees strong consistency. We introduce BEOWULF, a meta protocol based on a comprehensive cost model that integrates the three protocols and that dynamically selects the protocol with the lowest latency for a given workload. In the evaluation, we compare the prediction of the BEOWULF cost model with a baseline evaluation. The results nicely show the effectiveness of the analytical model and the precision in selecting the best suited protocol for a given workload.

In this paper, we present PAN , a generic middleware for distributed real-time complex event detection (CED) which is able to analyze multiple distributed data streams. In PAN , CED applications are defined as workflows and are executed by dedicated workers in a distributed way in a P2P network. In consequence, PAN is scalable in terms of the number of data streams and the complexity of the analyses. Evaluations based on an extended version of the ACM DEBS 2013 Grand Challenge scenario show the effectiveness and efficiency of PAN .

Mobile devices have become ubiquitous in the last years. With their powerful built-in cameras and sensors, they are intensively used as devices for capturing multimedia content. However, at the same time, the support of these mobile devices for searching within multimedia collections is still quite limited. These limitations are particularly obvious in the search paradigm(s) that are usually supported on mobile devices which mainly consist of traditional keyword search, if, at all, they might at most also support query-by-example content-based queries. In this paper, we present City-Stories, a powerful application that helps users to capture multimedia content and that in particular provides a rich set of different search paradigms. The search paradigms of City-Stories include temporal search (when a photo was taken), spatial search (where a photo was taken), and content-based similarity. For the latter, query-by-example (QbE) and query-by-sketch (QbS) are available. All these search paradigms can be freely mixed, either simultaneously (e.g., search for spatial and content similarity), or sequentially (e.g., to refine the query results obtained from one query mode with another one). In this paper, we introduce the City-Stories system and present the results of an evaluation which shows the gain in effectiveness when combining different search paradigms on a mobile device in comparison to using these paradigms individually.

vitrivr is an open source full-stack content-based multimedia retrieval system with focus on video. Unlike the majority of the existing multimedia search solutions, vitrivr is not limited to searching in metadata, but also provides content-based search and thus offers a large variety of different query modes which can be seamlessly combined: Query by sketch, which allows the user to draw a sketch of a query image and/or sketch motion paths, Query by example, keyword search, and relevance feedback. The vitrivr architecture is self-contained and addresses all aspects of multimedia search, from offline feature extraction, database management to frontend user interaction. The system is composed of three modules: a web-based frontend which allows the user to input the query (e.g., add a sketch) and browse the retrieved results (vitrivr-ui), a database system designed for interactive search in large-scale multimedia collections (ADAM), and a retrieval engine that handles feature extraction and feature-based retrieval (Cineast). The vitrivr source is available on GitHub under the MIT open source (and similar) licenses and is currently undergoing several upgrades as part of the Google Summer of Code 2016.

Cloud environments usually feature several geographically distributed data centers. In order to increase the scalability of applications, many Cloud providers partition data and distribute these partitions across data centers to balance the load. However, if the partitions are not carefully chosen, it might lead to distributed transactions. This is particularly expensive when applications require strong consistency guarantees. The additional synchronization needed for atomic commitment would strongly impact transaction throughput and could even completely undo the gain that can be achieved by load balancing. Hence, it is beneficial to avoid distributed transactions as much as possible by partitioning the data in such a way that transactions can be executed locally. As access patterns of characteristic transaction workloads may change over time, the partitioning also needs to be dynamically updated. In this paper we introduce Cumulus, an adaptive data partitioning approach which is able to identify characteristic access patterns of transaction mixes, to determine data partitions based on these patterns, and to dynamically re-partition data if the access patterns change. In the evaluation based on the TPC-C benchmark, we show that Cumulus significantly increases the overall system performance in an OLTP setting compared to static data partitioning approaches. Moreover, we show that Cumulus is able to adapt to workload shifts at runtime by generating partitions that match the actual workload and to re-configure the system on the fly.

Most applications deployed in a Cloud require a high degree of availability. For the data layer, this means that data have to be replicated either within a data center or across Cloud data centers. While replication also allows to increase the performance of applications if data is read as the load can be distributed across replica sites, updates need special coordination among the sites and may have an adverse effect on the overall performance. The actual effects of data replication depend on the replication protocol used. While ROWAA (read-one-write-all-available) prefers read operations, quorum-based replication protocols tend to prefer write operations as not all replica sites need to be updated synchronously. In this paper, we provide a detailed evaluation of ROWAA and quorum-based replication protocols in an Amazon AWS Cloud environment on the basis of the TPC-C benchmark and different transaction mixes. The evaluation results for single data center and multi data center environments show that in general the influence of transaction coordination significantly grows with the number of update sites and a growing number of update transactions. However, not all quorum-based protocols are well suited for high update loads as they may create a hot spot that again significantly impacts performance.

With the advent of data Clouds that come with nearly unlimited storage capacity combined with low storage costs, the well-established update-in-place paradigm for data management is more and more replaced by a multi-version approach. Especially in a Cloud environment with several geographically distributed data centers that act as replica sites, this allows to keep old versions of data and thus to provide a rich set of read operations with different semantics (e.g., read most recent version, read version not older than, read data as of, etc.). A combination of multi-version data management, replication, and partitioning allows to redundantly store several or even all versions of data items without significantly impacting each single site. However, in order to avoid that single sites in such partially replicated data Clouds are overloaded when processing archive queries that access old versions, query optimization has to jointly consider version selection and load balancing (site selection). In this paper, we introduce ARCTIC, a novel cost-aware index for version and site selection for a broad range of query types including both fresh data and archive data. We describe in detail the interplay between the different parts of the index and their implementation. Moreover, we present the results of the evaluation of the combined version and replica index in a Cloud environment that shows a significant gain in query throughput compared to a monolithic index.

Despite the tremendous importance and availability of large video collections, support for video retrieval is still rather limited and is mostly tailored to very concrete use cases and collections. In image retrieval, for instance, standard keyword search on the basis of manual annotations and content-based image retrieval, based on the similarity to query image (s), are well established search paradigms, both in academic prototypes and in commercial search engines. Recently, with the proliferation of sketch-enabled devices, also sketch-based retrieval has received considerable attention. The latter two approaches are based on intrinsic image features and rely on the representation of the objects of a collection in the feature space. In this paper, we present Cineast, a multi-feature sketch-based video retrieval engine. The main objective of Cineast is to enable a smooth transition from content-based image retrieval to content-based video retrieval and to support powerful search paradigms in large video collections on the basis of user-provided sketches as query input. Cineast is capable of retrieving video sequences based on edge or color sketches as query input and even supports one or multiple exemplary video sequences as query input. Moreover, Cineast also supports a novel approach to sketch-based motion queries by allowing a user to specify the motion of objects within a video sequence by means of (partial) flow fields, also specified via sketches. Using an emergent combination of multiple different features, Cineast is able to universally retrieve video (sequences) without the need for prior knowledge or semantic understanding. The evaluation with a general purpose video collection has shown the effectiveness and the efficiency of the Cineast approach.

The tremendous increase of multimedia data in recent years has heightened the need for systems that not only allow to search with keywords, but that also support content-based retrieval in order to effectively and efficiently query large collections. In this paper, we introduce ADAM, a system that is able to store and retrieve multimedia objects by seamlessly combining aspects from databases and information retrieval. ADAM is able to work with both structured and unstructured data and to jointly provide Boolean retrieval and similarity search. To efficiently handle large volumes of data it makes use of a signature-based indexing and the distribution of the collection to multiple shards that are queried in a MapReduce style. We present ADAM in the setting of a sketch-based image retrieval application using the ImageNet collection containing 14 million images.

Searching for scenes in team sport videos is a task that recurs very often in game analysis and other related activities performed by coaches. In most cases, queries are formulated on the basis of specific motion characteristics the user remembers from the video. Providing sketching interfaces for graphically specifying query input is thus a very natural user interaction for a retrieval application. However, the quality of the query (the sketch) heavily depends on the memory of the user and her ability to accurately formulate the intended search query by transforming this 3D memory of the known item(s) into a 2D sketch query. In this paper, we present an auto-suggest search feature that harnesses spatiotemporal data of team sport videos to suggest potential directions containing relevant data during the formulation of a sketch-based motion query. Users can intuitively select the direction of the desired motion query on-the-fly using the displayed visual clues, thus relaxing the need for relying heavily on memory to formulate the query. At the same time, this significantly enhances the accuracy of the results and the speed at which they appear. A first evaluation has shown the effectiveness and efficiency of our approach.

We present SportSense, a system for interactive sports video retrieval using sketch-based motion queries. SportSense is based on sports videos of games, enriched with an overlay of metadata that incorporates spatio-temporal information about various events and movements. We present how sketch-based motion queries are formulated and executed, as well as the use of various intuitive input interfaces to acquire the query object. The system uses spatio-temporal index structures to facilitate interactive response times.

The enormous increase of digital image collections urgently necessitates effective, efficient, and in particular highly flexible approaches to image retrieval. Different search paradigms such as text search, query-by-example, or query-by-sketch need to be seamlessly combined and integrated to support different information needs and to allow users to start (and subsequently refine) queries with any type of object. In this paper, we present QUEST (Query by Example, Sketch and Text), a novel flexible multi-modal content-based image retrieval (CBIR) framework. QUEST seamlessly integrates and blends multiple modes of image retrieval, thereby accumulating the strengths of each individual mode. Moreover, it provides several implementations of the different query modes and allows users to select, combine and even superimpose the mode(s) most appropriate for each search task. The combination of search paradigms is by itself done in a very flexible way: either sequentially, where one query mode starts with the result set of the previous one (i.e., for incrementally refining and/or extending a query) or by supporting different paradigms at the same time (e.g., creating an artificial query image by superimposing a query image with a sketch, thereby directly integrating query-by-example and query-by-sketch). We present the overall architecture of QUEST and the dynamic combination and integration of the query modes it supports. Furthermore, we provide first evaluation results that show the effectiveness and the gain in efficiency that can be achieved with the combination of different search modes in QUEST.

The advent of pen-based user interfaces has facilitated several natural ways for human-computer interaction. One example is sketch-based retrieval, i.e., the search for (multimedia) objects on the basis of sketches as query input. So far, work has focused mainly on sketch-based image retrieval. However, more and more application domains also benefit from sketches as query input for searching in video collections. Enabling spatial search in videos, in the form of sketch-based motion queries, is increasingly demanded by coaches and analysts in team sports as a novel and innovative tool for game analysis. Even though game analysis is already a major activity in this domain, it is still mostly based on manual selection of video sequences. In this paper, we present Sport Sense, a first approach to enabling intuitive and efficient video retrieval using sketch-based motion queries. This is accomplished by using videos of games in team sports, together with an overlay of meta data that incorporates spatio-temporal information about various events. Sport Sense exploits spatio-temporal databases to store, index, and retrieve the tracked information at interactive response times. Moreover, it provides first intuitive user input interfaces for sketches representing motion paths. A particular challenge is to convert the users' sketches into spatial queries and to execute these queries in a flexible way that allows for some controlled deviation between the sketched path and the actual movement of the players and/or the ball. The evaluation results of Sport Sense show that this approach to sketch-based retrieval in sports videos is both very effective and efficient.

During the last decade, overlay networks based on distributed hash tables have become the de facto standard for data management in Peer-to-Peer (P2P) systems, with Chord being its most prominent representative. Essentially, with its fully decentralized approach, Chord avoids any bottleneck and single point of failure while guaranteeing data to be retrieved in O(log N) hops in a network consisting of N nodes. By optimizing the number of hops for data access, Chord implicitly assumes that all connections between nodes have comparable bandwidth and latency characteristics. However, in heterogeneous, mobile P2P systems that consist of both mobile and fixed nodes, this is not the case. Moreover, due to the mobility of nodes, connection parameters can dynamically change. Especially in mobile P2P applications where low latency for data access is essential, such as in emergency management, routing should aim at reducing the overall latency, rather than the number of hops in the network. In this paper, we present COMPASS, a protocol for efficient data access in heterogeneous mobile Chord-based P2P systems. COMPASS takes into account that the network latency of nodes in a mobile P2P network may significantly differ and thus aims at minimizing the overall latency, even if this necessitates more hops in the network. This is done by probing the network and by maintaining, in addition to Chord's finger table, at each peer a data structure called COMPASS table. We present in detail the initialization and maintenance of the COMPASS table that dynamically adapts to changing node characteristics. Evaluation results show that COMPASS outperforms standard Chord-based routing and reduces the overall latency in heterogeneous P2P networks consisting of fixed and mobile nodes.

Clouds are becoming the preferred platforms for large-scale applications. Currently, Cloud environments focus on high scalability and availability by relaxing consistency. Weak consistency's considered to be sufficient for most of the currently deployed applications in the Cloud. However, the Cloud is increasingly being promoted as environment for running a wide range of different types of applications on top of replicated data - of which not all will be satisfied with weak consistency. Strong consistency, even though demanded by applications, decreases availability and is costly to enforce from both a performance and monetary point of view. On the other hand, weak consistency may generate high costs due to the access to inconsistent data. In this paper, we present a novel approach, called cost-based concurrency control (C3), that allows to dynamically and adaptively switch at runtime between different consistency levels of transactions. C3 has been implemented in a Data-as-a-Service Cloud environment and considers all costs that incur during execution. These costs are determined by infrastructure costs for running a transaction in a certain consistency level (called consistency costs) and, optionally, by additional application-specific costs for compensating the effects of accessing inconsistent data (called inconsistency costs).C3 considers transaction mixes running different consistency levels at the same time while enforcing the inherent consistency guarantees of each of these protocols. The main contribution of this paper is threefold. First, it thoroughly analyzes the consistency costs of the most common concurrency control protocols; second, it specifies a set of rules that allow to dynamically select the most appropriate consistency level with the goal of minimizing the overall costs (consistency and inconsistency costs);third, it provides a protocol that guarantees that anomalies in the transaction mixes supported by C3 are avoided and that enforces the correct execution of all transactions in a transaction mix. We have evaluated C3 on the basis of real infrastructure costs, derived from Amazon's EC2. The results demonstrate the feasibility of the cost model and show that C3 leads to a reduction of the overall costs of transactions compared to a fixed consistency level.

We present a novel and innovative user interface for query-by-sketching based image retrieval that exploits emergent interactive paper and digital pen technology. Users can draw sketches with a digital pen on interactive paper in a user-friendly way. The pen is able to capture the stroke vectors and to interactively stream them to the underlying content-based image retrieval (CBIR) system via the pen's Bluetooth interface. We present the integration of interactive paper/digital pen technology with QbS, our CBIR system tailored to Query-by-Sketching, and we demonstrate the use of the paper and pen interface together with QbS for three different collections: MIRFLICKR-25K, a cartoon collection, and a collection of medieval paper watermarks.

Service-oriented architectures (SOAs) provide a successful model for structuring complex distributed software systems, as they reduce the cost of ownership and ease the creation of new applications by composing existing services. However, currently, the development of service-oriented applications requires many manual tasks and prevailing infrastructure is often based on centralized components that are central points of failure and easily become bottlenecks. In this paper, we promote self-organizing SOA as a new approach to overcome these limitations. Self-organizing SOA integrates research results in the areas of autonomic and service oriented computing. We consider self-organizing features for the whole life-cycle of a service-oriented application, from the creation to the execution, optimization, and monitoring.

Data clouds are newly emerging environments in which commercial providers manage large volumes of data with individual quality of service (QoS) guarantees per customer. These guarantees mainly include keeping several replicas of each data item in different distributed data centers for availability purposes. However, as the cost of maintaining several updateable replicas per data object is very high, cloud providers rather offer only a limited number of synchronously updated replicas (i.e., replicas that are always up-to-date) together with several read-only replicas that are updated in a lazy way and thus might hold stale data. QoS agreements may also include the maintenance of dedicated archives (copies of data which are frozen at some point in time). Stale data allow cloud providers to offer a variety of read operations with different semantics, e.g., read the most recent data, read data not older than / not younger than some timestamp t, or read data produced between t1and t2, or read data exactly as of t. These read operations can be supported by a read-only site using a stale replica. In this paper we present our approach to cloud data management, based on a recent protocol for data grids. We discuss in detail how the refresh of individual replicas is provided in a completely distributed way. Finally, we present the results of a performance evaluation in a data cloud setting.

With the increasingly growing size of digital image collections, known image search is gaining more and more importance. Especially in collections where individual objects are not tagged with metadata describing their content, content-based image retrieval (CBIR) is a promising approach, but usually suffers from the unavailability of query images that are good enough to express the user's information need. In this paper, we present the QbS system that provides CBIR based on user-drawn sketches. The QbS system combines angular radial partitioning for the extraction of features in the user-provided sketch, taking into account the spatial distribution of edges, and the image distortion model. This combination offers several highly relevant invariances that allow the query sketch to slightly deviate from the searched image in terms of rotation, translation, relative size, and/or unknown objects in the background. To illustrate the benefits of the approach, we present search results from the evaluation of the QbS system on the basis of the MIRFLICKR collection with 25,000 objects and compare the retrieval results of pure metadata-driven approaches, pure content-based retrieval using different sketches, and combinations thereof.

In this paper we report on our work towards a novel system that aims at the combination of classical failure handling methods for composite service execution and support for automatic handling of semantic application level failures. In particular, we formulate the notion of Control Flow Intervention; a generalization for handling those two types of failure categories. When combined with automatic service composition it constitutes an optimistic forward recovery method. The method adopts expressive semantic service annotations provided by OWL resp. OWL-S.

Abstract Building digital libraries (DLs) on top of data grids while facilitating data access and minimizing access overheads is challenging. To achieve this, replication in a Grid has to provide dedicated features that are only partly supported by existing Grid environments. First, it must provide transparent and consistent access to distributed data. Second, it must dynamically control the creation and maintenance of replicas. Third, it should allow higher replication granularities, i.e. beyond individual files. Fourth, users should be able to specify their freshness demands, i.e. whether they need most recent data or are satisfied with slightly outdated data. Finally, all these tasks must be performed efficiently. This paper presents an approach that will finally allow one to build a fully integrated and self‐managing replication subsystem for data grids that will provide all the above features. Our approach is to start with an accepted replication protocol for database clusters, namely PDBREP, and to adapt it to the grid. Copyright © 2008 John Wiley & Sons, Ltd.

The proliferation of software and hardware sensors which continuously create large amounts of data has significantly facilitated novel types of applications such as healthcare telemonitoring or roadside traffic management. All these applications demand new mechanisms for online processing and analysis of relevant data coming from multiple data streams. Especially telemonitoring applications in healthcare require a high degree of reliability and must be able to be deployed in a distributed environment. We present OSIRIS-SE, an infrastructure for reliable data stream management in a failure-prone distributed setting including resource-limited mobile devices. OSIRIS-SE supports the combination of different data stream operators into stream processes and offers efficient coordinated operator check pointing for the execution of these stream processes. In order to support mobile devices, OSIRIS-SE is able to deal with multiple failures, offers fine-grained reliability at operator level, and supports decentralized stream process orchestration in a peer-to-peer fashion. Moreover, OSIRIS-SE is fully implemented in Java and thus can be run on different platforms. The demo shows the reliable execution of stream processes in a health monitoring application including a wearable ECG sensor, a Bluetooth enabled blood pressure sensor, and a web cam as data sources. Operators are hosted at mobile devices (PDAs, smart phones) of a patient and at a laptop computer which also acts as base station. An important feature of the demo is to show that sensor data can losslessly be processed by seamlessly migrating stream processing to other devices in the network even in case of multiple failures.

Business processes executing in peer-to-peer environments usually invoke Web services on different, independent peers. Although peer-to-peer environments inherently lack global control, some business processes nevertheless require global transactional guarantees, i.e., atomicity and isolation applied at the level of processes. This paper introduces a new decentralized serialization graph testing protocol to ensure concurrency control and recovery in peer-to-peer environments. The uniqueness of the proposed protocol is that it ensures global correctness without relying on a global serialization graph. Essentially, each transactional process is equipped with partial knowledge that allows the transactional processes to coordinate. Globally correct execution is achieved by communication among dependent transactional processes and the peers they have accessed. In case of failures, a combination of partial backward and forward recovery is applied. Experimental results exhibit a significant performance gain over traditional distributed locking-based protocols with respect to the execution of transactions encompassing Web service requests.

Standards like SOAP, WSDL, and UDDI facilitate the proliferation of services. Based on these technologies, processes are a means to combine services to applications and to provide new value-added services. For large information systems, a centralized process engine is no longer appropriate due to limited scalability. Instead, in this paper, we propose a distributed and decentralized process engine that routes process instances directly from one peer to the next. Such a peer-to-peer process execution promises good scalability characteristics since it is able to dynamically balance the load of processes and services among all available service providers. Therefore, navigation costs only accumulate on peers that are directly involved in the execution. However, this requires sophisticated strategies for the replication of meta-data for peer-to-peer process execution. Especially, replication mechanisms should avoid frequent accesses to global information repositories. In our system, called OSIRIS (Open Service Infrastructure for Reliable and Integrated Process Support), we deploy a publish/subscribe-based replication scheme together with freshness predicates to significantly reduce replication costs. This way, OSIRIS can support process-based applications in a dynamically evolving system without limiting scalability and correctness. Experiments have shown very promising results with respect to scalability. In addition, OSIRIS provides a flexible infrastructure that can be extended seamlessly in a modular way. This paper demonstrates the extension towards distributed concurrency control.

New sensor technologies, powerful mobile devices, and wireless communication standards strongly proliferate ubiquitous and pervasive computing. This is particularly true for healthcare applications. Modern non-invasive unobtrusive sensors enable sophisticated monitoring of the human body. As a result of this monitoring a growing amount of elderly people suffering from chronic diseases will benefit from an enhanced quality of life and improved treatment. Healthcare applications process a vast amount of continuously generated data, this has become a serious challenge. Data stream management addresses this problem in general. However, healthcare applications have additional requirements, e.g., flexibility and reliability. As a consequence, we propose the combination of a sophisticated information infrastructure, called hyperdatabase, and data stream management. Hyperdatabases already allow for reliable process management in a peer-to-peer fashion. In this paper, we elaborate the close relation between distributed process management and data stream management. Further, we show that data stream processing can significantly benefit from an infrastructure for reliable process management. We present the architecture of our prototype infrastructure that supports processing of continuous data streams with a high degree of flexibility and reliability.

The functionality of applications is increasingly being made available by services. General concepts and standards like SOAP, WSDL, and UDDI support the discovery and invocation of single Web services. State-of-the-art process management is conceptually based on a centralized process manager. The resources of this coordinator limit the number of concurrent process executions, especially since the coordinator has to persistently store each state change for recovery purposes. In this paper, we overcome this limitation by executing processes in a peer-to-peer way exploiting all nodes of the system. By distributing the execution and navigation costs, we can achieve a higher degree of scalability allowing for a much larger throughput of processes compared to centralized solutions. This paper describes our prototype system OSIRIS, which implements such a true peer-to-peer process execution. We further present very promising results verifying the advantages over centralized process management in terms of scalability.

Processes are increasingly being used to make complex application logic explicit. Programming using processes has significant advantages but it poses a difficult problem from the system point of view in that the interactions between processes cannot be controlled using conventional techniques. In terms of recovery, the steps of a process are different from operations within a transaction. Each one has its own termination semantics and there are dependencies among the different steps. Regarding concurrency control, the flow of control of a process is more complex than in a flat transaction. A process may, for example, partially roll back its execution or may follow one of several alternatives. In this article, we deal with the problem of atomicity and isolation in the context of processes. We propose a unified model for concurrency control and recovery for processes and show how this model can be implemented in practice, thereby providing a complete framework for developing middleware applications using processes.

In this paper, we propose process locking, a dynamic scheduling protocol based on ideas of ordered shared locks, that allows for the correct concurrent and fault-tolerant execution of transactional processes. Transactional processes are well defined, complex structured collections of transactional services. The process structure comprises flow of control between single process steps and also considers alternatives for failure handling purposes. Moreover, the individual steps of a process may have different termination characteristics, i.e., they cannot be compensated once they have committed. All these constraints have to be taken into consideration when deciding how to interleave processes. However, due to the higher level semantics of processes, standard locking techniques based on shared and exclusive locks on data objects cannot be applied. Yet, process locking addresses both atomicity and isolation simultaneously at the appropriate level, the scheduling of processes, and accounts for the various constraints imposed by processes. In addition, process locking aims at providing a high degree of concurrency while, at the same time, minimizing execution costs. This is done by allowing cascading aborts for rather simple processes white this is prevented for complex, long-running processes within the same framework.

When relational database systems were introduced twenty years ago (1980), they were an infrastructure and main platform for application development. With today's information systems, the database system is a storage manager, far away from the applications. Our vision is that hyperdatabases become available that move up and extend database concepts to a higher level, closer to the applications. A hyperdatabase manages distributed objects and software components as well as workflows, in analogy to a database system that manages data and transactions. In short, hyperdatabases, also called "higher order databases", will provide "higher order data independence", e.g., immunity of applications against changes in the implementation of components and workload transparency. They will be the infrastructure for distributed information systems engineering of the future, and they are an abstraction from the host of current infrastructures and middleware technology. The article elaborates on this vision and outlines concrete projects at ETHZ such as PowerDB, a database cluster project. It shows how an efficient document engine can be built on top of a database cluster. A further project studies transactional process management as a layer on top of database transactions. Image similarity and multimedia components is another project where a hyperdatabase coordinates specialized components such as feature extraction and indexing services in a distributed environment.

The most important phase in e-commerce interactions is the payment, due to the transfer of sensitive information (e.g., credit card numbers). A couple of requirements exist both from the point of view of a customer and from the merchant's perspective. This set of requirements is even enlarged when complex interactions are considered in which a customer purchases goods originating from different merchants within one single e-commerce transaction. We show how all these different requirements of payment interactions can be seamlessly integrated in transactional payment processes. These processes are generated automatically based on the customer's specification of the e-commerce transaction (involved participants, means of payment, etc.). We present the basic structure of such payment processes, how the requirements are mapped into these processes and how they can be generated automatically. Furthermore, we present the architecture of a payment coordinator that has been implemented within the INVENT project. This payment coordinator controls the execution of transactional payment processes, thereby keeping track of the interactions with the various participants.

Composite systems are collections of autonomous, heterogeneous and distributed software applications. In these systems, data dependencies are continuously violated by local operations, and therefore coordination processes are necessary to guarantee overall correctness and consistency. Such coordination processes must be endowed with some form of execution guarantees, which require the participating subsystems to have certain database functionality (such as atomicity of local operations, order preservation, and either compensation of operations or the deferment of their commit). However, this functionality is not present in many applications and must be implemented by a transactional coordination agent coupled with the application. In this paper, we discuss the requirements to be met by the applications and their associated transactional coordination agents. We identify a minimal set of functionalities which the applications must provide in order to participate in transactional coordination processes, and we also discuss how the missing database functionality can be added to arbitrary applications using transactional coordination agents. Then, we identify the structure of a generic transactional coordination agent and provide an implementation example of a transactional coordination agent tailored to SAP R/3.

The unified theory of concurrency control and recovery integrates atomicity and isolation within a common framework, thereby avoiding many of the shortcomings resulting from treating them as orthogonal problems.This theory can be applied to the traditional read/write model as well as to semantically rich operations.In this paper, we extend the unified theory by applying it to generalized process structures, i.e., arbitrary partially ordered sequences of transaction invocations.lJsing the extended unified theory, our goal is to provide a more flexible handling of concurrent processes while allowing: as much parallelism as possible.Unlike in the original unified theory, we take into account that not all activities of a process might be compensatable and the fact that these process structures require transactional properties more general than in traditional ACID transactions.We provide a correctness criterion for transactional processes and identity the key points in which the more flexible structure of transactional processes implies differences from traditional transactions.