Non-verbal cues are essential in communication, and eye contact and gaze awareness are crucial components of these cues. However, video-based communication lacks many of these non-verbal cues. To address this issue, we have developed a low-cost prototype allowing natural eye contact during one-to-one video calls. In a study focused on telepresence in remote psychotherapy, we demonstrate that our prototype enhances the perception of genuine and realistic communication with higher physical and social presence compared to classical video meeting setups with the camera placed on the screen.

Performance optimisation continues to be a relevant topic both in hardware and software development, with video games producing fully rendered images every 16 or 34 ms, depending on the desired framerate. Human observers close their eyes for about 300 ms an average of twelve times per minute, which means many frames will never be observed. This paper aimed to examine if it would be possible to reduce rendering time by detecting and skipping these unobserved frames. Blinks were identified during runtime by detecting the eye aspect ratio of the observer in low-quality web camera footage. A prototype using this method was tested on a small group of subjects to determine if footage watched this way was perceived as distracting or of lesser quality than unaltered images. Results from a questionnaire suggest that the altered footage did not impact the subjects’ opinions, with no participant reporting any visual disturbances. Because this test used video footage, skipping frames was substituted by a lower resolution render. Altered frames were rendered an average of five percent faster than their unaltered counterparts.

Gastrointestinal (GI) endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.

Games are often defined as engines of experience, and they are heavily relying on emotions, they arouse in players. In this paper, we present a dataset called Toadstool as well as a reproducible methodology to extend on the dataset. The dataset consists of video, sensor, and demographic data collected from ten participants playing Super Mario Bros, an iconic and famous video game. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. In addition to the dataset and the methodology for data collection, we present a set of baseline experiments which show that we can use video game frames together with the facial expressions to predict the blood volume pulse of the person playing Super Mario Bros. With the dataset and the collection methodology we aim to contribute to research on emotionally aware machine learning algorithms, focusing on reinforcement learning and multimodal data fusion. We believe that the presented dataset can be interesting for a manifold of researchers to explore exciting new interdisciplinary questions.

The BioMedia 2019 ACM Multimedia Grand Challenge is the first in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year's challenge, the participants are asked to develop efficient algorithms which automatically detect a variety of findings commonly identified in the gastrointestinal (GI) tract (a part of the human digestive system). The purpose of this task is to develop methods to aid medical doctors performing routine endoscopy inspections of the GI tract. In this paper, we give a detailed description of the four different tasks of this year's challenge, present the datasets used for training and testing, and discuss how each submission is evaluated both qualitatively and quantitatively.

demonstration Share on Alternative inputs for games and AR/VR applications: deep headbanging on the web Authors: Philipp Moll Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität KlagenfurtView Profile , Andreas Leibetseder Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität KlagenfurtView Profile , Sabrina Kletz Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität KlagenfurtView Profile , Mathias Lux Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität KlagenfurtView Profile , Bernd Muenzer Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität KlagenfurtView Profile Authors Info & Claims MMSys '19: Proceedings of the 10th ACM Multimedia Systems ConferenceJune 2019 Pages 320–323https://doi.org/10.1145/3304109.3323832Published:18 June 2019Publication History 2citation167DownloadsMetricsTotal Citations2Total Downloads167Last 12 Months20Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper introduces a novel image descriptor for content based image retrieval tasks that integrates contour and color information into a compact vector. Loosely inspired by the human visual system and its mechanisms in efficiently identifying visual saliency, operations are performed on a fixed lattice of discrete positions by a set of edge detecting kernels that calculate region derivatives at different scales and orientation. The description method utilizes a weighted edge histogram where bins are populated on the premise of whether the regions contain edges belonging to the salient contours, while the discriminative power is further enhanced by integrating regional quantized color information. The proposed technique is both efficient and adaptive to the specifics of each depiction, while it does not need any training data to adjust parameters. Experimental evaluation conducted on seven benchmarking datasets against 13 well known global descriptors along with SIFT, SURF implementations (both in VLAD and BOVW), highlight the effectiveness and efficiency of the proposed descriptor.

Automatic detection of diseases is a growing field of interest, and machine learning in form of deep learning neural networks are frequently explored as a potential tool for the medical video analysis. To both improve the "black box"-understanding and assist in the administrative duties of writing an examination report, we release an automated multimedia reporting software dissecting the neural network to learn the intermediate analysis steps, i.e., we are adding a new level of understanding and explainability by looking into the deep learning algorithms decision processes. The presented open-source software can be used for easy retrieval and reuse of data for automatic report generation, comparisons, teaching and research. As an example, we use live colonoscopy as a use case which is the gold standard examination of the large bowel, commonly performed for clinical and screening purposes. The added information has potentially a large value, and reuse of the data for the automatic reporting may potentially save the doctors large amounts of time.

In the future, medical doctors will to an increasing degree be assisted by deep learning neural networks for disease detection during examinations of patients. In order to make qualified decisions, the black box of deep learning must be opened to increase the understanding of the reasoning behind the decision of the machine learning system. Furthermore, preparing reports after the examinations is a significant part of a doctors work-day, but if we already have a system dissecting the neural network for understanding, the same tool can be used for automatic report generation. In this demo, we describe a system that analyses medical videos from the gastrointestinal tract. Our system dissects the Tensorflow-based neural network to provide insights into the analysis and uses the resulting classification and rationale behind the classification to automatically generate an examination report for the patient's medical journal.

This paper presents an open-source classification tool for image and video frame classification. The classification takes a search-based approach and relies on global and local image features. It has been shown to work with images as well as videos, and is able to perform the classification of video frames in real-time so that the output can be used while the video is recorded, playing, or streamed. OpenSea has been proven to perform comparable to state-of-the-art methods such as deep learning, at the same time performing much faster in terms of processing speed, and can be therefore seen as an easy to get and hard to beat baseline. We present a detailed description of the software, its installation and use. As a use case, we demonstrate the classification of polyps in colonoscopy videos based on a publicly available dataset. We conduct leave-one-out-cross-validation to show the potential of the software in terms of classification time and accuracy.

In this paper, we present LireSolr, an open source image retrieval server, build on top of the LIRE library and the Apache Solr search server. With LireSolr, visual information retrieval can be run on a server, which allows better distribution of workloads and simplifies applications in several areas including mobile and web. Furthermore, we showcase several example scenarios how LireSolr can be used to point out the broad range of possibilities and applications. The system is easy to install and setup, and the large number of retrieval tools either provided by LIRE or by other Apache Solr is made easily available on the search server. Moreover, our tool demonstrates how predictions from CNNs can easily be used to extend the visual information retrieval functionality.

Automatic detection of diseases by use of computers is an important, but still unexplored field of research. Such innovations may improve medical practice and refine health care systems all over the world. However, datasets containing medical images are hardly available, making reproducibility and comparison of approaches almost impossible. In this paper, we present KVASIR, a dataset containing images from inside the gastrointestinal (GI) tract. The collection of images are classified into three important anatomical landmarks and three clinically significant findings. In addition, it contains two categories of images related to endoscopic polyp removal. Sorting and annotation of the dataset is performed by medical doctors (experienced endoscopists). In this respect, KVASIR is important for research on both single- and multi-disease computer aided detection. By providing it, we invite and enable multimedia researcher into the medical domain of detection and retrieval.

Bowel preparation (cleansing) is considered to be a key precondition for successful colonoscopy (endoscopic examination of the bowel). The degree of bowel cleansing directly affects the possibility to detect diseases and may influence decisions on screening and follow-up examination intervals. An accurate assessment of bowel preparation quality is therefore important. Despite the use of reliable and validated bowel preparation scales, the grading may vary from one doctor to another. An objective and automated assessment of bowel cleansing would contribute to reduce such inequalities and optimize use of medical resources. This would also be a valuable feature for automatic endoscopy reporting in the future. In this paper, we present Nerthus, a dataset containing videos from inside the gastrointestinal (GI) tract, showing different degrees of bowel cleansing. By providing this dataset, we invite multimedia researchers to contribute in the medical field by making systems automatically evaluate the quality of bowel cleansing for colonoscopy. Such innovations would probably contribute to improve the medical field of GI endoscopy.

Health care has a long history of adopting technology to save lives and improve the quality of living. Visual information is frequently applied for disease detection and assessment, and the established fields of computer vision and medical imaging provide essential tools. It is, however, a misconception that disease detection and assessment are provided exclusively by these fields and that they provide the solution for all challenges. Integration and analysis of data from several sources, real-time processing, and the assessment of usefulness for end-users are core competences of the multimedia community and are required for the successful improvement of health care systems. We have conducted initial investigations into two use cases surrounding diseases of the gastrointestinal (GI) tract, where the detection of abnormalities provides the largest chance of successful treatment if the initial observation of disease indicators occurs before the patient notices any symptoms. Although such detection is typically provided visually by applying an endoscope, we are facing a multitude of new multimedia challenges that differ between use cases. In real-time assistance for colonoscopy, we combine sensor information about camera position and direction to aid in detecting, investigate means for providing support to doctors in unobtrusive ways, and assist in reporting. In the area of large-scale capsular endoscopy, we investigate questions of scalability, performance and energy efficiency for the recording phase, and combine video summarization and retrieval questions for analysis.

With an annual growth rate of 16.2% of taken photos a year, researchers predict an almost unbelievable number of 4.9 trillion stored images in 2017. Nearly 80% of these photos in 2017 will be taken with mobile phones. To be able to cope with this immense amount of visual data in a fast and accurate way, a visual information retrieval systems are needed for various domains and applications. LIRE, short for Lucene Image Retrieval, is a light weight and easy to use Java library for visual information retrieval. It allows developers and researchers to integrate common content based image retrieval approaches in their applications and research projects. LIRE supports global and local image features and can cope with millions of images using approximate search and distributing indexes on the cloud. In this demo we present a novel tool called F-search that emphasize the core strengths of LIRE: lightness, speed and accuracy.

In this paper, we present the dataset Right Inflight developed to support the exploration of the match between video content and the situation in which that content is watched. Specifically, we look at videos that are suitable to be watched on an airplane, where the main assumption is that that viewers watch movies with the intent of relaxing themselves and letting time pass quickly, despite the inconvenience and discomfort of flight. The aim of the dataset is to support the development of recommender systems, as well as computer vision and multimedia retrieval algorithms capable of automatically predicting which videos are suitable for inflight consumption. Our ultimate goal is to promote a deeper understanding of how people experience video content, and of how technology can support people in finding or selecting video content that supports them in regulating their internal states in certain situations. Right Inflight consists of 318 human-annotated movies, for which we provide links to trailers, a set of pre-computed low-level visual, audio and text features as well as user ratings. The annotation was performed by crowdsourcing workers, who were asked to judge the appropriateness of movies for inflight consumption.

In this paper, we present the computer-aided diagnosis part of the EIR system [9], which can support medical experts in the task of detecting diseases and anatomical landmarks in the gastrointestinal (GI) system. This includes automatic detection of important findings in colonoscopy videos and marking them for the doctors. EIR is designed in a modular way so that it can easily be extended for other diseases. For this demonstration, we will focus on polyp detection, as our system is trained with the ASU-Mayo Clinic polyp database [5].

In the field of medical endoscopy more and more surgeons are changing over to record and store videos of their endoscopic procedures, such as surgeries and examinations, in long-term video archives. In order to support surgeons in accessing these endoscopic video archives in a content-based way, we propose a simple yet effective signature-based approach: the Signature Matching Distance based on adaptive-binning feature signatures. The proposed distance-based similarity model facilitates an adaptive representation of the visual properties of endoscopic images and allows for matching these properties efficiently. We conduct an extensive performance analysis with respect to the task of linking specific endoscopic images with video segments and show the high efficacy of our approach. We are able to link more than 88% of the endoscopic images to their corresponding correct video segments, which improves the current state of the art by one order of magnitude.

Videos are becoming more and more a tool of communication. There are how-to videos, people are discussing actions of others based on their recorded performance, e.g., in soccer, or they simply record videos of great moments and show them to friends and family. In this paper we focus on very specific how-to videos and present a novel, web based annotation tool, that combines (i) zoom, (ii) drawing, and (iii) temporal social bookmarking in video streams. Moreover, we present a short study on the usefulness of the tool to communicate general concepts of a specific video game based on a captured game session.

In this paper we present Loki+Lire, a framework for the creation of web-based interfaces for search, annotation and presentation of multimedia data. The framework provides tools to ingest, transcode, present, annotate and index different types of media such as images, videos, audio files and textual documents. The front-end is compliant with the latest HTML5 standards, while the back-end allows system administrators to create processing pipelines that can be adapted for different tasks and purposes.

Parallel recordings made at the same event with different devices, e.g. by visitors of a concert, contain semantically the same content but do not run at the same speed when played back in parallel on a computer, which makes their synchronization difficult. This effect, time drift, concerns all current consumer multimedia recording devices and results from their internal clocks not running at the same speed, leading to deviations from their nominal sampling rates. We present a mobile application capable of conducting instant measurements of this time drift, thus helping in determining devices that go well together, or correcting the speed differences in post processing.

The power of crowds, leveraging a large number of human contributors and the capabilities of human computation, has enormous potential to address key challenges in the area of multimedia research. This power is, however, of difficult exploitation: challenges arise from the fact that a community of users or workers is a complex and dynamic system highly sensitive to changes in the form and the parameterization of their activities. Since 2012, the International ACM Workshop on Crowdsourcing for Multimedia \emph{CrowdMM} has been the venue for collecting new insights on the effective deployment of crowdsourcing towards boosting Multimedia research. In its third edition, CrowdMM14 especially focuses on contributions that propose solutions for the key challenges that face widespread adoption of crowdsourcing paradigms in the multimedia research community. These include: identification of optimal crowd members (e.g., user expertise, worker reliability), providing effective explanations (i.e., good task design), controlling noise and quality in the results, designing incentive structures that do not breed cheating, adversarial environments, gathering necessary background information about crowd members without violating privacy, controlling descriptions of task.

In this paper, we introduce the concept of intentional framing, defined as the sum of the choices that a photographer makes on how to portray the subject matter of an image. We carry out analysis experiments that demonstrate the existence of a correspondence between image similarity that is calculated automatically on the basis of global feature representations, and image similarity that is perceived by humans at the level of intentional frames. Intentional framing has profound implications: The existence of a fundamental image-interpretation principle that explains the importance of global representations in capturing human-perceived image semantics reaches beyond currently dominant assumptions in multimedia research. The ability of fast global-feature approaches to compete with more `sophisticated' approaches, which are computationally more complex, is demonstrated using a simple search method (SimSea) to classify a large (2M) collection of social images by tag class. In short, intentional framing provides a principled connection between human interpretations of images and lightweight, fast image processing methods. Moving forward, it is critical that the community explicitly exploits such approaches, as the social image collections that we tackle, continue to grow larger.

In this paper we present a multimedia art installation which uses human computer interaction to give the user the possibility to experience how blind people perceive the world and how hard it can be. The installation follows the classical co-operative game pattern, where a group of people - in this case two - have to work together to achieve a goal. With these basic characteristics in mind we consider our submission a serious game. With this serious game we show how multimedia can be used for making people more attentive for problems of other people and that multimedia, and in this case serious games, are indeed a useful tool for letting people experience situations they have not been aware of before.

Streaming video has recently surpassed peer-to-peer networks in terms of network capacity hunger.Reports estimate a share of 40 % of peak network capacity dedicated to entertainment, mostly streaming video.A large share of this traffic originates from web-based services.YouTube alone takes up to 8 % of the prime time Internet traffic.While transmission currently works in a best effort system, multimedia information systems on the web are far from being perfect.Retrieval, annotation, validated and useful metadata, reliable and trusted services, and user interaction and context-based adaptation are still under discussion and issues to research.Currently, the Web itself faces dramatic changes, looking for example at the spread of social networks, like Facebook, Twitter or the YouTube community, which become part of everyday life of more and more people.Technologies like Linked Data, HTML5, WebM, or WebRTC are enablers that make these changes possible and let people experience the Web without knowing details about technology.Also the availability of the Web has changed a lot.Many people create multimedia data on the go, put videos and photos on microblogs, or share audiovisual data with their loved ones or colleagues.These activities also have a deep effect on multimedia data formats and usage, multimedia systems and multimedia service providers."Multimedia everywhere" was a common concept before, but this has changed to "relevant media at your finger tips", so new methods for finding, annotating, sharing, remixing, summarizing, transmitting, storing and consuming multimedia data have to be found.

Content based image retrieval has been around for some time. There are lots of different test data sets, lots of published methods and techniques, and manifold retrieval challenges, where content based image retrieval is of interest. LIRE is a Java library, that provides a simple way to index and retrieve millions of images based on the images' contents. LIRE is robust and well tested and is not only recommended by the websites of ImageCLEF and MediaEval, but is also employed in industry. This paper gives an overview on LIRE, its use, capabilities and reports on retrieval and runtime performance.

Video annotation is a tedious task. But especially in medical domain the knowledge of experts for the interpretation of videos is of high value. Typically medical doctors do not have time for extensive annotation, but are used to manual notes, speech recordings, and pointing. In this demo paper we present an application for annotation of medical videos, focusing on endoscopic surgery. We adopt common interaction method of medical experts to mobile computing and provide a tool for experts to annotate videos by drawing on the video and recording speech annotations.

Manual synchronization of audio and video recordings is a very annoying and time consuming task, especially if the tracks are very long and/or of large quantity. If the tracks aren't just short clips (of a few seconds or minutes) and recorded from heterogeneous sources, an additional problem comes into play - time drift - which arises if different recording devices aren't synchronized. This demo paper presents the experimental software Audio Align, which aims to simplify the manual synchronization process with the ultimate goal to automate it altogether. It gives a short introduction to the topic, discusses the approach, method, implementation and preliminary results and gives an outlook at possible improvements.

Many photo-taking scenarios have emotional triggers such as happiness, joy, or in general a "good feeling". In this paper we postulate that if a photo was taken to preserve a good feeling, it will most likely trigger a good feeling when re-visiting the photo later on. We present an approach to classify photos and decide automatically whether they have been taken to preserve a good feeling or not. We employ an annotated data set to create a model and apply this model to enhance classical image search. The result is a novel search approach, demonstrated by a proof-of-concept application capable of performing keyword-based search and filtering out the results that are considered unlikely to convey a good feeling to the viewer.

Visual information retrieval (VIR) is an active and vibrant research area, which attempts at providing means for organizing, indexing, annotating, and retrieving visual information (images and videos) form large, unstructured repositories. The goal of VIR is to retrieve the highest number of relevant matches to a given query (often expressed as an example image and/or a series of keywords). In its early years (1995-2000) the research efforts were dominated by content-based approaches contributed primarily by the image and video processing community. During the past decade, it was widely recognized that the challenges imposed by the semantic gap (the lack of coincidence between an image's visual contents and its semantic interpretation) required a clever use of textual metadata (in addition to information extracted from the image's pixel contents) to make image and video retrieval solutions efficient and effective. The need to bridge (or at least narrow) the semantic gap has been one of the driving forces behind current VIR research. Additionally, other related research problems and market opportunities have started to emerge, offering a broad range of exciting problems for computer scientists and engineers to work on. In this tutorial, we present an overview of visual information retrieval (VIR) concepts, techniques, algorithms, and applications. Several topics are supported by examples written in Java, using Lucene (an open-source Java-based indexing and search implementation) and LIRE (Lucene Image REtrieval), an open-source Java-based library for content-based image retrieval (CBIR) written by Mathias Lux.

LIRe (Lucene Image Retrieval) is an open source library for content based image retrieval. Besides providing multiple common and state of the art retrieval mechanisms it allows for easy use on multiple platforms. LIRe is actively used for research, teaching and commercial applications. Due to its modular nature it can be used on process level (e.g. index images and search) as well as on image feature level. Developers and researchers can easily extend and modify LIRe to adapt it to their needs.

We present a new approach for video browsing using visualization of motion direction and motion intensity statistics by color and brightness variations. Statistics are collected from motion vectors of H.264/AVC encoded video streams, so full video decoding is not required. By interpreting visualized motion patterns of video segments, users are able to quickly identify scenes similar to a prototype scene or identify potential scenes of interest. We give some examples of motion patterns with different semantic value, including camera zooms, hill jumps of ski-jumpers, and the repeated appearance of a news speaker. In a user study we show that certain scenes of interest can be found significantly faster using our video browsing tool than using a video player with VCR-like controls.

MPEG-7 is an extensive multimedia metadata standard covering a huge number of aspects of metadata. However, as with most metadata standards details of usage and application of the standards are - at least partially - open to interpretation. In case of MPEG-7 storage and transmission of high level metadata on concept level are defined but retrieval methods are not proposed. So if for instance a user annotates photos using the MPEG-7 semantic description scheme, there are no standardized ways to retrieve the photos based on the annotation. In this paper we propose metrics for retrieval based on the MPEG-7 semantic description scheme and evaluate them in a digital photo retrieval scenario.

Results of internet searches are typically presented as lists. When searching for digital photos different search result presentations however offer different benefits. If users are primarily interested in the visual content of images a thumbnail grid may be more appropriate than a list. For people searching photos taken at a specific place image metadata in the result presentation is of interest too. In this paper we present an application which monitors a user's behavior while searching for digital photos and classifies the user's intention. Based on the intention, the result is adapted to support the user in an optimal way.

Caliph & Emir are Java-based applications for image annotation and retrieval. They implement a large part of MPEG-7 descriptors and support annotation and retrieval based on the descriptors. Manual annotation is based on text and the MPEG-7 semantic description scheme. Automatic extraction of low level features and existing metadata is also supported. Retrieval features include: linear search, content based image retrieval, textual metadata and graph indexing, and two-dimensional repository visualization.

We present an overview of the workshop on Common Sense Knowledge and Goal-Oriented Interfaces held at the 2008 Intelligent User Interfaces conference. Six papers were accepted from diverse research groups, each offering innovative new research on interfaces that incorporate common sense knowledge and that are oriented around the goals of their users.

LIRe (Lucene Image Retrieval) is a light weight open source Java library for content based image retrieval. It provides common and state of the art global image features and offers means for indexing and retrieval. Due to the fact that it is based on a light weight embedded text search engine, it can be integrated easily in applications without relying on a database server.

An encyclopedia provides a written compendium of knowledge consisting of articles concisely and exhaustively covering topics of interest. Most general encyclopedias are structured in an alphabetical manner and favor keyword queries and link-based navigation as the primary form of access. On closer examination, many encyclopedia articles can directly or indirectly be associated with geospatial references. This publication explores ways of anchoring encyclopedia articles to geospatial references and presents a Web-based 3D interface which allows navigation of the German Brockhaus encyclopedia through a geospatial metaphor