We introduce token-consistent dropout in vision transformers, which improves network calibration without causing any severe drop in performance. We use linear layers with token-consistent stochastic parameters inside the multilayer perceptron blocks, without altering the architecture of the transformer. The stochastic parameters are sampled from the uniform distribution, both during training and inference. The applied linear operations preserve the topological structure, formed by the set of tokens passing through the shared multilayer perceptron. This operation encourages the learning of the recognition task to rely on the topological structures of the tokens, instead of their values, which in turn offers the desired behavior. We compare our method to established baselines and applicable state-of-the-art alternatives, to demonstrate its impact on the calibration of vision transformers.

In most scenarios, conditional image generation can be thought of as an inversion of the image understanding process. Since generic image understanding involves solving multiple tasks, it is natural to aim at generating images via multi-conditioning. However, multi-conditional image generation is a very challenging problem due to the heterogeneity and the sparsity of the (in practice) available conditioning labels. In this work, we propose a novel neural architecture to address the problem of heterogeneity and sparsity of the spatially multi-conditional labels. Our choice of spatial conditioning, such as by semantics and depth, is driven by the promise it holds for better control of the image generation process. The proposed method uses a transformer-like architecture operating pixel-wise, which receives the available labels as input tokens to merge them in a learned homogeneous space of labels. The merged labels are then used for image generation via conditional generative adversarial training. In this process, the sparsity of the labels is handled by simply dropping the input tokens corresponding to the missing labels at the desired locations, thanks to the proposed pixel-wise operating architecture. Our experiments on three benchmark datasets demonstrate the clear superiority of our method over the state-of-the-art and compared baselines. The source code can be found at https://github.com/96ritika/TLAM.

The contextual information plays a core role in semantic segmentation. As for video semantic segmentation, the contexts include static contexts and motional contexts, corresponding to static content and moving content in a video clip, respectively. The static contexts are well exploited in image semantic segmentation by learning multi-scale and global/long-range features. The motional contexts are studied in previous video semantic segmentation. However, there is no research about how to simultaneously learn static and motional contexts which are highly correlated and complementary to each other. To address this problem, we propose a Coarse-to-Fine Feature Mining (CFFM) technique to learn a unified presentation of static contexts and motional contexts. This technique consists of two parts: coarse-to-fine feature assembling and cross-frame feature mining. The former operation prepares data for further processing, enabling the subsequent joint learning of static and motional contexts. The latter operation mines useful information/contexts from the sequential frames to enhance the video contexts of the features of the target frame. The enhanced features can be directly applied for the final prediction. Experimental results on popular benchmarks demonstrate that the proposed CFFM performs favorably against state-of-the-art methods for video semantic segmentation. Our implementation is available at https://github.com/GuoleiSun/VSS-CFFM.

Monocular depth reconstruction of complex and dynamic scenes is a highly challenging problem. While for rigid scenes learning-based methods have been offering promising results even in unsupervised cases, there exists little to no literature addressing the same for dynamic and deformable scenes. In this work, we present an unsupervised monocular framework for dense depth estimation of dynamic scenes, which jointly reconstructs rigid and nonrigid parts without explicitly modelling the camera motion. Using dense correspondences, we derive a training objective that aims to opportunistically preserve pairwise distances between reconstructed 3D points. In this process, the dense depth map is learned implicitly using the as-rigid-as-possible hypothesis. Our method provides promising results, demonstrating its capability of reconstructing 3D from challenging videos of non-rigid scenes. Furthermore, the proposed method also provides unsupervised motion segmentation results as an auxiliary output.

Digitization is advancing rapidly in many prevalently analogue domains such as healthcare. For the latter domain, the synergies with modern information technologies (IT) have become an integral part regarding communication and collaboration. For this reason, a comprehensible language is of importance in order to allow a frictionless exchange of information between domain experts. The Business Process Model and Notation (BPMN) 2.0 represents a promising notation that may be applied as lingua franca. Although the BPMN 2.0 is widespread applied by experts in business and industry, little experience exists how BPMN 2.0 is adopted in healthcare. In order to assess how BPMN 2.0 is deployed in healthcare, we conducted a preliminary eye tracking study, in which n=16 professionals from healthcare comprehended a particular BPMN 2.0 process model. The results indicate that BPMN 2.0 might be a candidate for a lingua franca to foster the comprehensible exchange of information as well as collaboration between healthcare and IT.

We define the concept of CompositeTasking as the fusion of multiple, spatially distributed tasks, for various aspects of image understanding. Learning to perform spatially distributed tasks is motivated by the frequent availability of only sparse labels across tasks, and the desire for a compact multi-tasking network. To facilitate CompositeTasking, we introduce a novel task conditioning model – a single encoder-decoder network that performs multiple, spatially varying tasks at once. The proposed network takes an image and a set of pixel-wise dense task requests as inputs, and performs the requested prediction task for each pixel. Moreover, we also learn the composition of tasks that needs to be performed according to some CompositeTasking rules, which includes the decision of where to apply which task. It not only offers us a compact network for multitasking, but also allows for task-editing. Another strength of the proposed method is demonstrated by only having to supply sparse supervision per task. The obtained results are on par with our baselines that use dense supervision and a multi-headed multi-tasking design. The source code will be made publicly available at www.github.com/nikola3794/composite-tasking.

We introduce Task Switching Networks (TSNs), a task-conditioned architecture with a single unified encoder/decoder for efficient multi-task learning. Multiple tasks are performed by switching between them, performing one task at a time. TSNs have a constant number of parameters irrespective of the number of tasks. This scalable yet conceptually simple approach circumvents the overhead and intricacy of task-specific network components in existing works. In fact, we demonstrate for the first time that multi-tasking can be performed with a single task-conditioned decoder. We achieve this by learning task-specific conditioning parameters through a jointly trained task embedding network, encouraging constructive interaction between tasks. Experiments validate the effectiveness of our approach, achieving state-of-the-art results on two challenging multi-task benchmarks, PASCAL-Context and NYUD. Our analysis of the learned task embeddings further indicates a connection to task relationships studied in the recent literature.

The utilization of mobile technology in the field of medicine and healthcare has become a decisive aspect. The entire field is denoted as mobile health (mHealth). For mHealth, the development and use of mobile applications are crucial. The purposes and goals of mHealth apps, in turn, are manifold. As a consequence, a plethora of mHealth apps can be found in the app stores. Interestingly, for patients, users, and health care providers that consider to use mHealth apps one aspect has been less pursued so far: Systematic and standardized ways that help about the quality of an app or its medical evidence are mainly missing. The Mobile App Rating Scale (MARS) is a standardized instrument that aims at the systematic and comparable evaluation of the quality of mobile health apps as well as categorizing their goals and functions. It comprises 23 items, which are utilized to calculate a rating scale. Having MARS in mind, a database was developed that is called Mobile Health App Database (MHAD). The latter offers technical features to systematically utilize the MARS for researchers as well as clinicians and end-users that (i) want to evaluate apps as well as (ii) want an interactive and easy-to-use web interface that shows the results of the rating procedure. MHAD comprises a rating platform that supports the conduction of MARS ratings and their release process. With the information platform, a web application was developed that prepares the data stored in the rating platform for being freely viewed and studied by users, patients, and health care providers. The goal of MHAD constitutes to be an open science repository that encourages researchers to release their MARS ratings to a broader audience. Such repositories become more and more important in many fields, especially in the field of mHealth.

Smart devices and low-powered sensors are becoming increasingly ubiquitous and nowadays almost all of these devices are connected, which is a promising foundation for crowdsensing of data related to various environmental phenomena. Resulting data is especially meaningful when it is related to time and location. Interestingly, many existing approaches built their solution on monolithic backends that process data on a per-request basis. However, for many scenarios, such technical setting is not suitable for managing data requests of a large crowd. For example, when dealing with millions of data points, still many challenges arise for modern smartphones if calculations or advanced visualization features must be accomplished directly on the smartphone. Therefore, the work at hand proposes an architectural design for managing geospatial data of tinnitus patients, which combines a cloudnative approach with Big Data concepts used in the Internet of Things. The presented architectural design shall serve as a generic foundation to implement (1) a scalable backend for a platform that covers the aforementioned crowdsensing requirements as well as to provide (2) a sophisticated stream processing concept to calculate and pre-aggregate incoming measurement data of tinnitus patients. Following this, this paper presents a visualization feature to provide users with a comprehensive overview of noise levels in their environment based on noise measurements. This shall help tinnitus or hearing-impaired patients to avoid locations with a burdensome sound level.

Consensus maximization is a key strategy in 3D vision for robust geometric model estimation from measurements with outliers. Generic methods for consensus maximization, such as Random Sampling and Consensus (RANSAC), have played a tremendous role in the success of 3D vision, in spite of the ubiquity of outliers. However, replicating the same generic behaviour in a deeply learned architecture, using supervised approaches, has proven to be difficult. In that context, unsupervised methods have a huge potential to adapt to any unseen data distribution, and therefore are highly desirable. In this paper, we propose for the first time an unsupervised learning framework for consensus maximization, in the context of solving 3D vision problems. For that purpose, we establish a relationship between inlier measurements, represented by an ideal of inlier set, and the subspace of polynomials representing the space of target transformations. Using this relationship, we derive a constraint that must be satisfied by the sought inlier set. This constraint can be tested without knowing the transformation parameters, therefore allows us to efficiently define the geometric model fitting cost. This model fitting cost is used as a supervisory signal for learning consensus maximization, where the learning process seeks for the largest measurement set that minimizes the proposed model fitting cost. Using our method, we solve a diverse set of 3D vision problems, including 3D-3D matching, non-rigid 3D shape matching with piece-wise rigidity and image-to-image matching. Despite being unsupervised, our method outperforms RANSAC in all three tasks for several datasets.

Building on progress in feature representations for image retrieval, image-based localization has seen a surge of research interest. Image-based localization has the advantage of being inexpensive and efficient, often avoiding the use of 3D metric maps altogether. That said, the need to maintain a large amount of reference images as an effective support of localization in a scene, nonetheless calls for them to be organized in a map structure of some kind. The problem of localization often arises as part of a navigation process. We are, therefore, interested in summarizing the reference images as a set of landmarks, which meet the requirements for image-based navigation. A contribution of this paper is to formulate such a set of requirements for the two sub-tasks involved: compact map construction and accurate self localization. These requirements are then exploited for compact map representation and accurate self-localization, using the framework of a network flow problem. During this process, we formulate the map construction and self-localization problems as convex quadratic and second-order cone programs, respectively. We evaluate our methods on publicly available indoor and outdoor datasets, where they outperform existing methods significantly.

We present a self-supervision method for 3D hand pose estimation from depth maps. We begin with a neural network initialized with synthesized data and fine-tune it on real but unlabelled depth maps by minimizing a set of data-fitting terms. By approximating the hand surface with a set of spheres, we design a differentiable hand renderer to align estimates by comparing the rendered and input depth maps. In addition, we place a set of priors including a data-driven term to further regulate the estimate's kinematic feasibility. Our method makes highly accurate estimates comparable to current supervised methods which require large amounts of labelled training samples, thereby advancing state-of-the-art in unsupervised learning for hand pose estimation.

In this paper, we formulate a generic non-minimal solver using the existing tools of Polynomials Optimization Problems (POP) from computational algebraic geometry. The proposed method exploits the well known Shor's or Lasserre's relaxations, whose theoretical aspects are also discussed. Notably, we further exploit the POP formulation of non-minimal solver also for the generic consensus maximization problems in 3D vision. Our framework is simple and straightforward to implement, which is also supported by three diverse applications in 3D vision, namely rigid body transformation estimation, Non-Rigid Structure-from-Motion (NRSfM), and camera autocalibration. In all three cases, both non-minimal and consensus maximization are tested, which are also compared against the state-of-the-art methods. Our results are competitive to the compared methods, and are also coherent with our theoretical analysis. The main contribution of this paper is the claim that a good approximate solution for many polynomial problems involved in 3D vision can be obtained using the existing theory of numerical computational algebra. This claim leads us to reason about why many relaxed methods in 3D vision behave so well? And also allows us to offer a generic relaxed solver in a rather straightforward way. We further show that the convex relaxation of these polynomials can easily be used for maximizing consensus in a deterministic manner. We support our claim using several experiments for aforementioned three diverse problems in 3D vision.

The exploitation of sensor features offered by present smart mobile devices is a trend that becomes increasingly important in various domains. In healthcare, for example, these sensors are used to cheaply gather valuable data for chronic disease management or health care. Regarding the latter, health insurers crave for effective methods that can be offered to their customers. Moreover, smart mobile devices provide many advantages compared to approaches hitherto applied in the aforementioned contexts as they can be easily used in everyday life. Thereby, when taking these advantages properly into account, new mobile application types become possible. Body sensor networks are such an application type that aim at monitoring users in vivo. Furthermore, data gathered with body sensor networks may be a valuable basis to provide user interventions. This paper presents an application that shall support users to walk mindfully. The motivation was to create a mobile tool that can make mindful walking more effective to reduce stress and to target noncommunicable diseases such as diabetes or depression. It is a mobile personalized tool that senses the walking speed and provides haptic feedback thereof. The mindful walking procedure, the technical prototype as well as preliminary study results are presented and discussed in this work. The reported user feedback and the study results indicate promising perspectives for a tool that supports a mindful walking behavior. Altogether, the use of smart mobile device sensors constitutes a promising instrument for realizing mobile applications in the context of health care and disease management.

Treatment of patients with tinnitus is mainly pursued on the basis of screening data, encompassing answers to questionnaires or audiological examinations. Since tinnitus affects the quality of life of patients and is associated with comorbidities like depression, the screening step involves the acquisition of extensive amounts of information, which may contribute to the design of a personalized treatment. Often times, it can be observed that patients give up their treatment before completion (i.e., they constitute “dropout” cases). In this study, we investigate how multi-target classification on the screening data can contribute to characterize patients that will drop out of the study. For our analysis, we base or considerations on the target variable “tinnitus loudness”, i.e. the subjectively perceived loudness of the phantom signal. Following this, we attempt to identify variables that explain the tinnitus loudness together with the likelihood of interrupting the treatment. To be more precise, we report on results from gathered data of 1419 tinnitus patients from the University Hospital of Regensburg.

Presently, mHealth technology is often applied in the context of chronic diseases to gather data that may lead to new and valuable medical insights. As many aspects of chronic diseases are not completely understood, new data sources might be promising. mHealth technology may help in this context as it can be easily used in everyday life. Moreover, the bring your own device principle encourages many patients to use their smartphone to learn more about their disease. The less is known about a disorder (e.g., tinnitus), the more patients crave for new insights and opportunities. Despite the fact that existing mHealth technology like mobile crowdsensing has already gathered data that may help patients, in general, less is known whether and how data gathered with different mobile technologies may differ. In this context, one relevant aspect is the contribution of the mobile operating system itself. For example, are there differences between Android and iOS users that utilize the same mHealth technology for a disease. In the TrackYourTinnitus project, a mobile crowdsensing mHealth platform was developed to gather data for tinnitus patients in order to reveal new insights on this disorder with high economic and patient-related burdens. As many data sets were gathered during the last years that enable us to compare Android and iOS users, the work at hand compares characteristics of these users. Interesting insights like the one that Android users with tinnitus are significantly older than iOS users could be revealed by our study. However, more evaluations are necessary for TrackYourTinnitus in particular and mHealth technology in general to understand how smartphones affect the gathering of data on chronic diseases when using them in the large.

Many studies have revealed that therapeutic homework is beneficial for the efficacy of therapies. Interestingly, the latter have been less supported by IT systems so far and, hence, therapeutic opportunities have been neglected. For example, mobile devices can be used to notify patients about assigned homework and help them to accomplish it in a timely manner. In general, the use of mobile devices as well as their sensors seem to be promising for the support of remote therapeutic interventions. In the Albatros project, we have been developing a framework that enables domain experts to flexibly define the homework required in the context of a remote therapeutic intervention. More precisely, the various tasks of a homework can be specified as a mobile process, which is then run on the mobile device of the respective patient. To realize this vision, a configurator component using a model-driven approach was developed. In particular, the Albatros configurator shall relieve domain experts from complex technical issues when defining a homework. The study presented in this paper investigates whether domain experts are actually able to use the configurator component. In particular, the study revealed three insights. First, basic interventions can be easily defined with an acceptable number of errors. Second, for defining complex interventions (e.g., using a sensor when performing an exercise) several issues could be identified that will contribute to improve the Albatros configurator. Third, additional studies are needed to evaluate the overall mental effort of domain experts when using the configurator. Altogether, the Albatros framework may be a reasonable alley to empower domain experts in creating homework in the context of remote therapeutic interventions.

We present a simple and effective method for 3D hand pose estimation from a single depth frame. As opposed to previous state-of-the-art methods based on holistic 3D regression, our method works on dense pixel-wise estimation. This is achieved by careful design choices in pose parameterization, which leverages both 2D and 3D properties of depth map. Specifically, we decompose the pose parameters into a set of per-pixel estimations, i.e., 2D heat maps, 3D heat maps and unit 3D directional vector fields. The 2D/3D joint heat maps and 3D joint offsets are estimated via multitask network cascades, which is trained end-to-end. The pixel-wise estimations can be directly translated into a vote casting scheme. A variant of mean shift is then used to aggregate local votes while enforcing consensus between the the estimated 3D pose and the pixel-wise 2D and 3D estimations by design. Our method is efficient and highly accurate. On MSRA and NYU hand dataset, our method outperforms all previous state-of-the-art approaches by a large margin. On the ICVL hand dataset, our method achieves similar accuracy compared to the nearly saturated result obtained by [5] and outperforms various other proposed methods. Code is available online1.

Existing depth-based approaches to predicting anthropometric measurements, such as body height, arm span and hip circumference, either directly compute the measurements on 3D point clouds, and thus are sensitive to noise, or fit a model to the observed depth values, which typically is time-consuming. In this paper, we rely on the intuition that, to predict a specific anthropometric measurement, one does not need to have detailed information about the entire body shape. We therefore introduce an approach to anthropometry based on a random regression forest trained from local depth cues. The local predictions are then accumulated into one global, image-level anthropometric measurement prediction. We introduce a forest refinement scheme, whose objective function directly relies on both the image-level prediction, as well as on the local predictions' reliability. The resulting approach has the advantage of being both computationally highly efficient and accurate.

Many symptoms of neuropsychiatric disorders such as tinnitus are subjective and vary over time. Usually, in interviews or self-report questionnaires, patients are asked to report symptoms as well as their severity and duration retrospectively. However, only little is known to what degree such retrospective reports reflect the symptoms experienced in daily life some time ago. Mobile technologies can help to bridge this gap: mobile self-help services allow patients to record their symptoms prospectively when (or shortly after) they occur in daily life. In this study, we present results that we obtained with the mobile crowdsensing platform TrackYourTinnitus to show that there is a discrepancy between the prospective assessment of symptom variability and the retrospective report thereof. To be more precise, we evaluated the real-time entries provided to the platform by individuals experiencing tinnitus. The results indicate that mobile technologies like the TrackYourTinnitus crowdsensing platform may go beyond the role of an assistive service for patients by contributing to more accurate diagnosis and, hence, to a more elaborated treatment.

In the context of therapeutic interventions, smart mobile devices are becoming increasingly important. First, they can properly assist patients in performing their homework - a support required for more efficient therapeutic interventions. Second, mobile applications enable therapists to monitor homework outcomes. From a technical perspective, frequently required changes of the mobile applications supporting therapeutic interventions constitute a major challenge. To tackle the latter for a multitude of remote therapeutic interventions, e.g., in psychotherapy or physiotherapy, we deploy process management technology to smart mobile devices. This paper discusses flexibility issues addressed by the mobile processes. Particularly, the achieved flexibility, in turn, increases the practical benefits of smart mobile devices in the context of remote therapeutic interventions.

The development of information systems, which support homework in the context of therapeutic interventions, has not been sufficiently addressed so far. However, both therapists and patients crave for a mobile assistance managing complex homework procedures. For example, smart mobile devices can automatically inform therapists about corresbonding outcomes, giving them the opportunity to timely adjust homework if required. When realizing information systems that integrate smart mobile devices, the common procedure of therapeutic interventions in general and homework in particular must be carefully captured by the system. Therefore, relevant requirements were elicitated in real-world projects. Based on these requirements, we realized the Albatros platform enabling therapists to manage therapeutic interventions remotely. Using the platform, homework can be created with a web-based component and be performed by patients with the help of smart mobile devices. In this paper, elicitated requirements for realizing the platform as well as its features and architecture are presented. Altogether, the Albatros platform enables therapists as well as patients to manage therapeutic interventions and homework more efficiently.

The identification of subpopulations with particular characteristics with respect to a disease is important for personalized diagnostics and therapy design. For some diseases, the outcome is described by more than one target variable. An example is tinnitus: the perceived loudness of the phantom signal and the level of distress caused by it are both relevant targets for diagnosis and therapy. In this work, we study the potential of multi-target classification for the identification of those screening variables, which separate best among the different subpopulations of patients, paying particular attention to subpopulations with discordant value combinations of loudness and distress. We analyse the screening data of 1344 tinnitus patients from the University Hospital Regensburg, including questions from 7 questionnaires, and report on the performance of our workflow in target separation and in ranking the questionnaires variables on their discriminative power.

Despite the widespread use of process models in healthcare organizations, there are many unresolved issues regarding the reading and comprehension of these models by domain experts. This is aggravated by the fact that there exists a plethora of process modeling languages for the graphical documentation of processes, which are often not used consistently for various reasons. Hence, the identification of those factors fostering the comprehension of process models becomes crucial. We have developed a conceptual framework incorporating measurements and theories from cognitive neuroscience and psychology to unravel factors fostering the comprehension of process models within organizations. We believe that a better comprehension of process models will enhance the support of healthcare processes significantly.

In recent years, skeleton-based action recognition has become a popular 3D classification problem. State-of-the-art methods typically first represent each motion sequence as a high-dimensional trajectory on a Lie group with an additional dynamic time warping, and then shallowly learn favorable Lie group features. In this paper we incorporate the Lie group structure into a deep network architecture to learn more appropriate Lie group features for 3D action recognition. Within the network structure, we design rotation mapping layers to transform the input Lie group features into desirable ones, which are aligned better in the temporal domain. To reduce the high feature dimensionality, the architecture is equipped with rotation pooling layers for the elements on the Lie group. Furthermore, we propose a logarithm mapping layer to map the resulting manifold data into a tangent space that facilitates the application of regular output layers for the final classification. Evaluations of the proposed network for standard 3D human action recognition datasets clearly demonstrate its superiority over existing shallow Lie group feature learning methods as well as most conventional deep learning methods.

State-of-the-art methods for 3D hand pose estimation from depth images require large amounts of annotated training data. We propose modelling the statistical relationship of 3D hand poses and corresponding depth images using two deep generative models with a shared latent space. By design, our architecture allows for learning from unlabeled image data in a semi-supervised manner. Assuming a one-to-one mapping between a pose and a depth map, any given point in the shared latent space can be projected into both a hand pose or into a corresponding depth map. Regressing the hand pose can then be done by learning a discriminator to estimate the posterior of the latent pose given some depth map. To prevent over-fitting and to better exploit unlabeled depth maps, the generator and discriminator are trained jointly. At each iteration, the generator is updated with the back-propagated gradient from the discriminator to synthesize realistic depth maps of the articulated hand, while the discriminator benefits from an augmented training set of synthesized samples and unlabeled depth maps. The proposed discriminator network architecture is highly efficient and runs at 90fps on the CPU with accuracies comparable or better than state-of-art on 3 publicly available benchmarks.

Smart mobile devices are variously used in the health sector. Some mobile applications empower patients to better understand their health problems, others guide them in health behavior. Moreover, smart mobile devices can be used in clinical research. Mobile crowd sensing has proven high usefulness for collecting health data with high ecological validity in this context. As the core idea, individually recorded health data are evaluated and fed back to individuals to better control their symptoms. For this purpose, the Track-YourTinnitus mobile crowd sensing platform was developed to empower patients to cope better with their tinnitus. So far, the platform has solely gathered patient data based on mobile questionnaires. When filling in a questionnaire, however, the analysis of the heartrate might provide novel information to medical experts. As monitoring the heartrate with smart mobile devices is costly, the trend towards wearables offers promising perspectives. Using smartwatches instead of smartphones in TrackYourTinnitus, however, requires questionnaire management on smartwatches. This work presents results of a prestudy related to the feasibility of sophisticated questionnaires on smartwatches. A prototype was developed and evaluated with 24 subjects. The obtained results are promising regarding the use of smartwatches for mobile crowd sensing in the context of chronic disorders.